Alert thresholds depend on the nature of your applications. Some queries may have arbitrary tolerance thresholds. Building an efficient monitoring platform takes time. 😉
3 Prometheus alerting rules for Consul. Exported via prometheus/consul_exporter. These rules cover critical and warning conditions — copy and paste the YAML into your Prometheus configuration.
groups:
- name: ConsulExporter
rules:
- alert: ConsulServiceHealthcheckFailed
expr: consul_catalog_service_node_healthy == 0
for: 1m
labels:
severity: critical
annotations:
summary: Consul service healthcheck failed (instance {{ $labels.instance }})
description: "Service: `{{ $labels.service_name }}` Healthcheck: `{{ $labels.service_id }}`\n VALUE = {{ $value }}\n LABELS = {{ $labels }}"
- alert: ConsulMissingMasterNode
expr: consul_raft_peers < 3
for: 0m
labels:
severity: critical
annotations:
summary: Consul missing master node (instance {{ $labels.instance }})
description: "Numbers of consul raft peers should be 3, in order to preserve quorum.\n VALUE = {{ $value }}\n LABELS = {{ $labels }}"
- alert: ConsulAgentUnhealthy
expr: consul_health_node_status{status="critical"} == 1
for: 0m
labels:
severity: critical
annotations:
summary: Consul agent unhealthy (instance {{ $labels.instance }})
description: "A Consul agent is down\n VALUE = {{ $value }}\n LABELS = {{ $labels }}" wget https://raw.githubusercontent.com/samber/awesome-prometheus-alerts/refs/heads/master/dist/rules/consul/consul-exporter.yml
Service: `{{ $labels.service_name }}` Healthcheck: `{{ $labels.service_id }}`
- alert: ConsulServiceHealthcheckFailed
expr: consul_catalog_service_node_healthy == 0
for: 1m
labels:
severity: critical
annotations:
summary: Consul service healthcheck failed (instance {{ $labels.instance }})
description: "Service: `{{ $labels.service_name }}` Healthcheck: `{{ $labels.service_id }}`\n VALUE = {{ $value }}\n LABELS = {{ $labels }}" Numbers of consul raft peers should be 3, in order to preserve quorum.
- alert: ConsulMissingMasterNode
expr: consul_raft_peers < 3
for: 0m
labels:
severity: critical
annotations:
summary: Consul missing master node (instance {{ $labels.instance }})
description: "Numbers of consul raft peers should be 3, in order to preserve quorum.\n VALUE = {{ $value }}\n LABELS = {{ $labels }}" A Consul agent is down
- alert: ConsulAgentUnhealthy
expr: consul_health_node_status{status="critical"} == 1
for: 0m
labels:
severity: critical
annotations:
summary: Consul agent unhealthy (instance {{ $labels.instance }})
description: "A Consul agent is down\n VALUE = {{ $value }}\n LABELS = {{ $labels }}"