What is the Prometheus alert rule for "Consul missing master node"?

Numbers of consul raft peers should be 3, in order to preserve quorum. PromQL expression: consul_raft_peers < 3. Severity: critical. Duration: 1m.

What is the Prometheus alert rule for "Consul agent unhealthy"?

A Consul agent is down PromQL expression: consul_health_node_status{status="critical"} == 1. Severity: critical.

What is the Prometheus alert rule for "Consul service healthcheck failed"?

Service: `{{ $labels.service_name }}` Healthcheck: `{{ $labels.service_id }}` PromQL expression: consul_catalog_service_node_healthy == 0. Severity: critical. Duration: 1m.

Consul Prometheus Alert Rules

Q: What is the Prometheus alert rule for "Consul agent unhealthy"?

A Consul agent is down PromQL expression: consul_health_node_status{status="critical"} == 1. Severity: critical.

Q: What is the Prometheus alert rule for "Consul service healthcheck failed"?

Service: `{{ $labels.service_name }}` Healthcheck: `{{ $labels.service_id }}` PromQL expression: consul_catalog_service_node_healthy == 0. Severity: critical. Duration: 1m.

Q: What is the Prometheus alert rule for "Consul has no raft leader"?

The Consul raft cluster has no elected leader (according to {{ $labels.instance }}), meaning the cluster cannot process writes and is effectively unavailable. PromQL expression: consul_raft_leader != 1. Severity: critical. Duration: 1m.

Q: What is the Prometheus alert rule for "Consul exporter query failure"?

The consul_exporter's last query against the local Consul agent failed, meaning the exported Consul metrics on {{ $labels.instance }} are stale or missing even though the exporter itself is still being scraped successfully. PromQL expression: consul_up != 1. Severity: critical. Duration: 1m.

5 Prometheus alerting rules for Consul.Exported via prometheus/consul_exporter.These rules cover critical and warning conditions — copy and paste the YAML into your Prometheus configuration.

⚠️

Alert thresholds depend on the nature of your applications. Some queries may have arbitrary tolerance thresholds. Building an efficient monitoring platform takes time. 😉

groups:
- name: ConsulExporter
  rules:
      # A brief drop below 3 peers is expected during a rolling restart or server migration;
      # the 1m delay tolerates that without masking a genuine quorum loss.
    - alert: ConsulMissingMasterNode
      expr: consul_raft_peers < 3
      for: 1m
      labels:
        severity: critical
      annotations:
        summary: Consul missing master node (instance {{ $labels.instance }})
        description: "Numbers of consul raft peers should be 3, in order to preserve quorum.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
    
    - alert: ConsulAgentUnhealthy
      expr: consul_health_node_status{status="critical"} == 1
      for: 0m
      labels:
        severity: critical
      annotations:
        summary: Consul agent unhealthy (instance {{ $labels.instance }})
        description: "A Consul agent is down\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
    
    - alert: ConsulServiceHealthcheckFailed
      expr: consul_catalog_service_node_healthy == 0
      for: 1m
      labels:
        severity: critical
      annotations:
        summary: Consul service healthcheck failed (instance {{ $labels.instance }})
        description: "Service: `{{ $labels.service_name }}` Healthcheck: `{{ $labels.service_id }}`\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
    
    - alert: ConsulHasNoRaftLeader
      expr: consul_raft_leader != 1
      for: 1m
      labels:
        severity: critical
      annotations:
        summary: Consul has no raft leader (instance {{ $labels.instance }})
        description: "The Consul raft cluster has no elected leader (according to {{ $labels.instance }}), meaning the cluster cannot process writes and is effectively unavailable.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
    
    - alert: ConsulExporterQueryFailure
      expr: consul_up != 1
      for: 1m
      labels:
        severity: critical
      annotations:
        summary: Consul exporter query failure (instance {{ $labels.instance }})
        description: "The consul_exporter's last query against the local Consul agent failed, meaning the exported Consul metrics on {{ $labels.instance }} are stale or missing even though the exporter itself is still being scraped successfully.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

7.3.prometheus/consul_exporter(5 rules)

wget https://raw.githubusercontent.com/samber/awesome-prometheus-alerts/refs/heads/master/dist/rules/consul/consul-exporter.yml

critical

7.3.1.Consul missing master node

Numbers of consul raft peers should be 3, in order to preserve quorum.

  # A brief drop below 3 peers is expected during a rolling restart or server migration;
  # the 1m delay tolerates that without masking a genuine quorum loss.
- alert: ConsulMissingMasterNode
  expr: consul_raft_peers < 3
  for: 1m
  labels:
    severity: critical
  annotations:
    summary: Consul missing master node (instance {{ $labels.instance }})
    description: "Numbers of consul raft peers should be 3, in order to preserve quorum.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

critical

7.3.2.Consul agent unhealthy

A Consul agent is down

- alert: ConsulAgentUnhealthy
  expr: consul_health_node_status{status="critical"} == 1
  for: 0m
  labels:
    severity: critical
  annotations:
    summary: Consul agent unhealthy (instance {{ $labels.instance }})
    description: "A Consul agent is down\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

critical

7.3.3.Consul service healthcheck failed

Service: `{{ $labels.service_name }}` Healthcheck: `{{ $labels.service_id }}`

- alert: ConsulServiceHealthcheckFailed
  expr: consul_catalog_service_node_healthy == 0
  for: 1m
  labels:
    severity: critical
  annotations:
    summary: Consul service healthcheck failed (instance {{ $labels.instance }})
    description: "Service: `{{ $labels.service_name }}` Healthcheck: `{{ $labels.service_id }}`\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

critical

7.3.4.Consul has no raft leader

The Consul raft cluster has no elected leader (according to {{ $labels.instance }}), meaning the cluster cannot process writes and is effectively unavailable.

- alert: ConsulHasNoRaftLeader
  expr: consul_raft_leader != 1
  for: 1m
  labels:
    severity: critical
  annotations:
    summary: Consul has no raft leader (instance {{ $labels.instance }})
    description: "The Consul raft cluster has no elected leader (according to {{ $labels.instance }}), meaning the cluster cannot process writes and is effectively unavailable.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

critical

7.3.5.Consul exporter query failure

The consul_exporter's last query against the local Consul agent failed, meaning the exported Consul metrics on {{ $labels.instance }} are stale or missing even though the exporter itself is still being scraped successfully.

- alert: ConsulExporterQueryFailure
  expr: consul_up != 1
  for: 1m
  labels:
    severity: critical
  annotations:
    summary: Consul exporter query failure (instance {{ $labels.instance }})
    description: "The consul_exporter's last query against the local Consul agent failed, meaning the exported Consul metrics on {{ $labels.instance }} are stale or missing even though the exporter itself is still being scraped successfully.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

More in Orchestrators

Kubernetes Nomad Etcd OpenStack