Skip to main content
APA
Sponsored byCAST AI — Kubernetes cost optimizationBetter Stack — Uptime monitoring and log managementVictoriaMetrics — Fast, open-source time series database and drop-in Prometheus replacement.

Etcd logoEtcd Prometheus Alert Rules

19 Prometheus alerting rules for Etcd.Exported via Embedded exporter.These rules cover critical and warning conditions — copy and paste the YAML into your Prometheus configuration.

⚠️

Alert thresholds depend on the nature of your applications. Some queries may have arbitrary tolerance thresholds. Building an efficient monitoring platform takes time. 😉

7.4.Etcd logoEmbedded exporter(19 rules)

wget https://raw.githubusercontent.com/samber/awesome-prometheus-alerts/refs/heads/master/dist/rules/etcd/embedded-exporter.yml
critical

7.4.1.Etcd no Leader

Etcd cluster have no leader

- alert: EtcdNoLeader
  expr: etcd_server_has_leader == 0
  for: 0m
  labels:
    severity: critical
  annotations:
    summary: Etcd no Leader (instance {{ $labels.instance }})
    description: "Etcd cluster have no leader\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
critical

7.4.2.Etcd insufficient members for quorum

Only {{ $value }} etcd members are reachable, fewer than required for quorum; the cluster will reject writes until quorum is restored.

- alert: EtcdInsufficientMembersForQuorum
  expr: sum(up{job=~".*etcd.*"} == bool 1) without (instance) < ((count(up{job=~".*etcd.*"}) without (instance) + 1) / 2)
  for: 3m
  labels:
    severity: critical
  annotations:
    summary: Etcd insufficient members for quorum (instance {{ $labels.instance }})
    description: "Only {{ $value }} etcd members are reachable, fewer than required for quorum; the cluster will reject writes until quorum is restored.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
critical

7.4.3.Etcd high fsync durations critical

Etcd WAL fsync duration is critically high, 99th percentile is over 1s, which can cause request timeouts and trigger leader elections.

  # 1s threshold is a rough default; fsync duration depends on your disk hardware (NVMe vs spinning disk vs network storage) — adjust based on your baseline.
- alert: EtcdHighFsyncDurationsCritical
  expr: histogram_quantile(0.99, sum(rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m])) by (instance, le)) > 1
  for: 2m
  labels:
    severity: critical
  annotations:
    summary: Etcd high fsync durations critical (instance {{ $labels.instance }})
    description: "Etcd WAL fsync duration is critically high, 99th percentile is over 1s, which can cause request timeouts and trigger leader elections.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
warning

7.4.4.Etcd high fsync durations

Etcd WAL fsync duration increasing, 99th percentile is over 0.5s

  # 0.5s threshold is a rough default; fsync duration depends on your disk hardware (NVMe vs spinning disk vs network storage) — adjust based on your baseline.
- alert: EtcdHighFsyncDurations
  expr: histogram_quantile(0.99, sum(rate(etcd_disk_wal_fsync_duration_seconds_bucket[5m])) by (instance, le)) > 0.5
  for: 2m
  labels:
    severity: warning
  annotations:
    summary: Etcd high fsync durations (instance {{ $labels.instance }})
    description: "Etcd WAL fsync duration increasing, 99th percentile is over 0.5s\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
warning

7.4.5.Etcd high commit durations

Etcd commit duration increasing, 99th percentile is over 0.25s

  # 0.25s threshold is a rough default; commit duration depends on your disk hardware and write workload — adjust based on your baseline.
- alert: EtcdHighCommitDurations
  expr: histogram_quantile(0.99, sum(rate(etcd_disk_backend_commit_duration_seconds_bucket[5m])) by (instance, le)) > 0.25
  for: 2m
  labels:
    severity: warning
  annotations:
    summary: Etcd high commit durations (instance {{ $labels.instance }})
    description: "Etcd commit duration increasing, 99th percentile is over 0.25s\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
warning

7.4.6.Etcd database high fragmentation ratio

Etcd database in-use size is below 50% of its allocated size ({{ $value | humanizePercentage }}), indicating fragmentation; run 'etcdctl defrag' to reclaim disk space.

- alert: EtcdDatabaseHighFragmentationRatio
  expr: (last_over_time(etcd_mvcc_db_total_size_in_use_in_bytes[5m]) / last_over_time(etcd_mvcc_db_total_size_in_bytes[5m])) < 0.5 and etcd_mvcc_db_total_size_in_use_in_bytes > 104857600
  for: 10m
  labels:
    severity: warning
  annotations:
    summary: Etcd database high fragmentation ratio (instance {{ $labels.instance }})
    description: "Etcd database in-use size is below 50% of its allocated size ({{ $value | humanizePercentage }}), indicating fragmentation; run 'etcdctl defrag' to reclaim disk space.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
critical

7.4.7.Etcd insufficient Members

Etcd cluster should have an odd number of members

- alert: EtcdInsufficientMembers
  expr: count(etcd_server_id) % 2 == 0
  for: 0m
  labels:
    severity: critical
  annotations:
    summary: Etcd insufficient Members (instance {{ $labels.instance }})
    description: "Etcd cluster should have an odd number of members\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
critical

7.4.8.Etcd high number of failed GRPC requests critical

More than 5% GRPC request failure detected in Etcd

  # Counts server-side failures only. NotFound, AlreadyExists and Cancelled are normal gRPC
  # responses and are deliberately excluded from the error rate.
- alert: EtcdHighNumberOfFailedGRPCRequestsCritical
  expr: sum(rate(grpc_server_handled_total{grpc_code=~"Unknown|FailedPrecondition|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded"}[1m])) BY (grpc_service, grpc_method) / sum(rate(grpc_server_handled_total[1m])) BY (grpc_service, grpc_method) > 0.05 and sum(rate(grpc_server_handled_total[1m])) BY (grpc_service, grpc_method) > 0
  for: 2m
  labels:
    severity: critical
  annotations:
    summary: Etcd high number of failed GRPC requests critical (instance {{ $labels.instance }})
    description: "More than 5% GRPC request failure detected in Etcd\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
critical

7.4.9.Etcd high number of failed HTTP requests critical

More than 5% HTTP failure detected in Etcd

  # These etcd_http_* metrics are from the etcd v2 API and do not exist in etcd 3.x. Remove these rules if running etcd 3.x.
- alert: EtcdHighNumberOfFailedHTTPRequestsCritical
  expr: sum(rate(etcd_http_failed_total[1m])) BY (method) / sum(rate(etcd_http_received_total[1m])) BY (method) > 0.05 and sum(rate(etcd_http_received_total[1m])) BY (method) > 0
  for: 2m
  labels:
    severity: critical
  annotations:
    summary: Etcd high number of failed HTTP requests critical (instance {{ $labels.instance }})
    description: "More than 5% HTTP failure detected in Etcd\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
warning

7.4.10.Etcd high number of leader changes

Etcd leader changed {{ $value }} times during 10 minutes

- alert: EtcdHighNumberOfLeaderChanges
  expr: increase(etcd_server_leader_changes_seen_total[10m]) > 2
  for: 0m
  labels:
    severity: warning
  annotations:
    summary: Etcd high number of leader changes (instance {{ $labels.instance }})
    description: "Etcd leader changed {{ $value }} times during 10 minutes\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
warning

7.4.11.Etcd high number of failed GRPC requests warning

More than 1% GRPC request failure detected in Etcd

  # Counts server-side failures only. NotFound, AlreadyExists and Cancelled are normal gRPC
  # responses and are deliberately excluded from the error rate.
- alert: EtcdHighNumberOfFailedGRPCRequestsWarning
  expr: sum(rate(grpc_server_handled_total{grpc_code=~"Unknown|FailedPrecondition|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded"}[1m])) BY (grpc_service, grpc_method) / sum(rate(grpc_server_handled_total[1m])) BY (grpc_service, grpc_method) > 0.01 and sum(rate(grpc_server_handled_total[1m])) BY (grpc_service, grpc_method) > 0
  for: 2m
  labels:
    severity: warning
  annotations:
    summary: Etcd high number of failed GRPC requests warning (instance {{ $labels.instance }})
    description: "More than 1% GRPC request failure detected in Etcd\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
warning

7.4.12.Etcd GRPC requests slow

GRPC requests slowing down, 99th percentile is over 0.15s

  # Excludes the Defragment method, which is inherently slow and would otherwise trigger
  # false positives during a manual or automatic etcdctl defrag.
- alert: EtcdGRPCRequestsSlow
  expr: histogram_quantile(0.99, sum(rate(grpc_server_handling_seconds_bucket{grpc_method!="Defragment", grpc_type="unary"}[1m])) by (grpc_service, grpc_method, le)) > 0.15
  for: 2m
  labels:
    severity: warning
  annotations:
    summary: Etcd GRPC requests slow (instance {{ $labels.instance }})
    description: "GRPC requests slowing down, 99th percentile is over 0.15s\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
warning

7.4.13.Etcd high number of failed HTTP requests warning

More than 1% HTTP failure detected in Etcd

  # These etcd_http_* metrics are from the etcd v2 API and do not exist in etcd 3.x. Remove these rules if running etcd 3.x.
- alert: EtcdHighNumberOfFailedHTTPRequestsWarning
  expr: sum(rate(etcd_http_failed_total[1m])) BY (method) / sum(rate(etcd_http_received_total[1m])) BY (method) > 0.01 and sum(rate(etcd_http_received_total[1m])) BY (method) > 0
  for: 2m
  labels:
    severity: warning
  annotations:
    summary: Etcd high number of failed HTTP requests warning (instance {{ $labels.instance }})
    description: "More than 1% HTTP failure detected in Etcd\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
warning

7.4.14.Etcd HTTP requests slow

HTTP requests slowing down, 99th percentile is over 0.15s

  # This etcd_http_* metric is from the etcd v2 API and does not exist in etcd 3.x. Remove this rule if running etcd 3.x.
- alert: EtcdHTTPRequestsSlow
  expr: histogram_quantile(0.99, rate(etcd_http_successful_duration_seconds_bucket[1m])) > 0.15
  for: 2m
  labels:
    severity: warning
  annotations:
    summary: Etcd HTTP requests slow (instance {{ $labels.instance }})
    description: "HTTP requests slowing down, 99th percentile is over 0.15s\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
warning

7.4.15.Etcd member communication slow

Etcd member communication slowing down, 99th percentile is over 0.15s

  # 0.15s threshold is a rough default; round-trip time depends on your network topology between etcd members — adjust based on your baseline.
- alert: EtcdMemberCommunicationSlow
  expr: histogram_quantile(0.99, sum(rate(etcd_network_peer_round_trip_time_seconds_bucket[5m])) by (instance, le)) > 0.15
  for: 2m
  labels:
    severity: warning
  annotations:
    summary: Etcd member communication slow (instance {{ $labels.instance }})
    description: "Etcd member communication slowing down, 99th percentile is over 0.15s\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
warning

7.4.16.Etcd high number of failed proposals

Etcd server got {{ $value }} failed proposals in the past hour

- alert: EtcdHighNumberOfFailedProposals
  expr: increase(etcd_server_proposals_failed_total[1h]) > 5
  for: 2m
  labels:
    severity: warning
  annotations:
    summary: Etcd high number of failed proposals (instance {{ $labels.instance }})
    description: "Etcd server got {{ $value }} failed proposals in the past hour\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
warning

7.4.17.Etcd peer communication failures

Etcd member is experiencing more than 0.01 failed peer sends per second to peer {{ $labels.To }}, which usually indicates the member is unreachable or network-partitioned.

- alert: EtcdPeerCommunicationFailures
  expr: sum(rate(etcd_network_peer_sent_failures_total[1m])) by (To) > 0.01
  for: 2m
  labels:
    severity: warning
  annotations:
    summary: Etcd peer communication failures (instance {{ $labels.instance }})
    description: "Etcd member is experiencing more than 0.01 failed peer sends per second to peer {{ $labels.To }}, which usually indicates the member is unreachable or network-partitioned.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
critical

7.4.18.Etcd database quota low space

Etcd database size is above 95% of the configured storage quota ({{ $value }}%); once the quota is reached, etcd stops accepting writes cluster-wide.

- alert: EtcdDatabaseQuotaLowSpace
  expr: (last_over_time(etcd_mvcc_db_total_size_in_bytes[5m]) / last_over_time(etcd_server_quota_backend_bytes[5m])) * 100 > 95
  for: 10m
  labels:
    severity: critical
  annotations:
    summary: Etcd database quota low space (instance {{ $labels.instance }})
    description: "Etcd database size is above 95% of the configured storage quota ({{ $value }}%); once the quota is reached, etcd stops accepting writes cluster-wide.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
warning

7.4.19.Etcd excessive database growth

Based on the current write rate over the past 4 hours, the etcd database is predicted to exceed the configured storage quota within the next 4 hours.

- alert: EtcdExcessiveDatabaseGrowth
  expr: predict_linear(etcd_mvcc_db_total_size_in_bytes[4h], 4*60*60) > etcd_server_quota_backend_bytes
  for: 10m
  labels:
    severity: warning
  annotations:
    summary: Etcd excessive database growth (instance {{ $labels.instance }})
    description: "Based on the current write rate over the past 4 hours, the etcd database is predicted to exceed the configured storage quota within the next 4 hours.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

More in Orchestrators

KubernetesNomadConsulOpenStack