Alert thresholds depend on the nature of your applications. Some queries may have arbitrary tolerance thresholds. Building an efficient monitoring platform takes time. 😉
6 Prometheus alerting rules for Cortex. Exported via Embedded exporter. These rules cover critical and warning conditions — copy and paste the YAML into your Prometheus configuration.
groups:
- name: EmbeddedExporter
rules:
- alert: CortexRulerConfigurationReloadFailure
expr: cortex_ruler_config_last_reload_successful != 1
for: 0m
labels:
severity: warning
annotations:
summary: Cortex ruler configuration reload failure (instance {{ $labels.instance }})
description: "Cortex ruler configuration reload failure (instance {{ $labels.instance }})\n VALUE = {{ $value }}\n LABELS = {{ $labels }}"
- alert: CortexNotConnectedToAlertmanager
expr: cortex_prometheus_notifications_alertmanagers_discovered < 1
for: 0m
labels:
severity: critical
annotations:
summary: Cortex not connected to Alertmanager (instance {{ $labels.instance }})
description: "Cortex not connected to Alertmanager (instance {{ $labels.instance }})\n VALUE = {{ $value }}\n LABELS = {{ $labels }}"
# Threshold of 0.05/s avoids firing on transient single-event spikes.
- alert: CortexNotificationsAreBeingDropped
expr: rate(cortex_prometheus_notifications_dropped_total[5m]) > 0.05
for: 0m
labels:
severity: critical
annotations:
summary: Cortex notifications are being dropped (instance {{ $labels.instance }})
description: "Cortex notifications are being dropped due to errors (instance {{ $labels.instance }}, {{ $value | humanize }}/s).\n VALUE = {{ $value }}\n LABELS = {{ $labels }}"
# Threshold of 0.05/s avoids firing on transient single-event spikes.
- alert: CortexNotificationErrors
expr: rate(cortex_prometheus_notifications_errors_total[5m]) > 0.05
for: 0m
labels:
severity: critical
annotations:
summary: Cortex notification errors (instance {{ $labels.instance }})
description: "Cortex is failing when sending alert notifications (instance {{ $labels.instance }}, {{ $value | humanize }}/s).\n VALUE = {{ $value }}\n LABELS = {{ $labels }}"
- alert: CortexIngesterUnhealthy
expr: cortex_ring_members{state="Unhealthy", name="ingester"} > 0
for: 0m
labels:
severity: critical
annotations:
summary: Cortex ingester unhealthy (instance {{ $labels.instance }})
description: "Cortex has an unhealthy ingester\n VALUE = {{ $value }}\n LABELS = {{ $labels }}"
- alert: CortexFrontendQueriesStuck
expr: sum by (job) (cortex_query_frontend_queue_length) > 0
for: 5m
labels:
severity: critical
annotations:
summary: Cortex frontend queries stuck (instance {{ $labels.instance }})
description: "There are queued up queries in query-frontend.\n VALUE = {{ $value }}\n LABELS = {{ $labels }}" wget https://raw.githubusercontent.com/samber/awesome-prometheus-alerts/refs/heads/master/dist/rules/cortex/embedded-exporter.yml
Cortex ruler configuration reload failure (instance {{ $labels.instance }})
- alert: CortexRulerConfigurationReloadFailure
expr: cortex_ruler_config_last_reload_successful != 1
for: 0m
labels:
severity: warning
annotations:
summary: Cortex ruler configuration reload failure (instance {{ $labels.instance }})
description: "Cortex ruler configuration reload failure (instance {{ $labels.instance }})\n VALUE = {{ $value }}\n LABELS = {{ $labels }}" Cortex not connected to Alertmanager (instance {{ $labels.instance }})
- alert: CortexNotConnectedToAlertmanager
expr: cortex_prometheus_notifications_alertmanagers_discovered < 1
for: 0m
labels:
severity: critical
annotations:
summary: Cortex not connected to Alertmanager (instance {{ $labels.instance }})
description: "Cortex not connected to Alertmanager (instance {{ $labels.instance }})\n VALUE = {{ $value }}\n LABELS = {{ $labels }}" Cortex notifications are being dropped due to errors (instance {{ $labels.instance }}, {{ $value | humanize }}/s).
# Threshold of 0.05/s avoids firing on transient single-event spikes.
- alert: CortexNotificationsAreBeingDropped
expr: rate(cortex_prometheus_notifications_dropped_total[5m]) > 0.05
for: 0m
labels:
severity: critical
annotations:
summary: Cortex notifications are being dropped (instance {{ $labels.instance }})
description: "Cortex notifications are being dropped due to errors (instance {{ $labels.instance }}, {{ $value | humanize }}/s).\n VALUE = {{ $value }}\n LABELS = {{ $labels }}" Cortex is failing when sending alert notifications (instance {{ $labels.instance }}, {{ $value | humanize }}/s).
# Threshold of 0.05/s avoids firing on transient single-event spikes.
- alert: CortexNotificationErrors
expr: rate(cortex_prometheus_notifications_errors_total[5m]) > 0.05
for: 0m
labels:
severity: critical
annotations:
summary: Cortex notification errors (instance {{ $labels.instance }})
description: "Cortex is failing when sending alert notifications (instance {{ $labels.instance }}, {{ $value | humanize }}/s).\n VALUE = {{ $value }}\n LABELS = {{ $labels }}" Cortex has an unhealthy ingester
- alert: CortexIngesterUnhealthy
expr: cortex_ring_members{state="Unhealthy", name="ingester"} > 0
for: 0m
labels:
severity: critical
annotations:
summary: Cortex ingester unhealthy (instance {{ $labels.instance }})
description: "Cortex has an unhealthy ingester\n VALUE = {{ $value }}\n LABELS = {{ $labels }}" There are queued up queries in query-frontend.
- alert: CortexFrontendQueriesStuck
expr: sum by (job) (cortex_query_frontend_queue_length) > 0
for: 5m
labels:
severity: critical
annotations:
summary: Cortex frontend queries stuck (instance {{ $labels.instance }})
description: "There are queued up queries in query-frontend.\n VALUE = {{ $value }}\n LABELS = {{ $labels }}"