What is the Prometheus alert rule for "Kafka topics replicas"?

Kafka topic {{ $labels.topic }} has fewer than 3 in-sync replicas ({{ $value }}), data durability is at risk. PromQL expression: min(kafka_topic_partition_in_sync_replica) by (topic) < 3. Severity: critical.

What is the Prometheus alert rule for "Kafka consumer group lag"?

Kafka consumer group {{ $labels.consumergroup }} is lagging behind ({{ $value }} messages) PromQL expression: sum(kafka_consumergroup_lag) by (consumergroup) > 10000. Severity: warning. Duration: 1m.

What is the Prometheus alert rule for "Kafka topic offset decreased"?

Kafka topic offset has decreased PromQL expression: delta(kafka_burrow_partition_current_offset[1m]) < 0. Severity: warning.

Kafka Prometheus Alert Rules

Q: What is the Prometheus alert rule for "Kafka consumer group lag"?

Kafka consumer group {{ $labels.consumergroup }} is lagging behind ({{ $value }} messages) PromQL expression: sum(kafka_consumergroup_lag) by (consumergroup) > 10000. Severity: warning. Duration: 1m.

Q: What is the Prometheus alert rule for "Kafka consumer group lag increasing"?

Kafka consumer group {{ $labels.consumergroup }} lag on topic {{ $labels.topic }} is continuously increasing over the last 15 minutes, meaning the consumer is falling further behind even though the absolute lag threshold hasn't been crossed yet. PromQL expression: sum by (topic, consumergroup) (delta(kafka_consumergroup_lag{topic!="__consumer_offsets",consumergroup!=""}[5m])) > 0. Severity: warning. Duration: 15m.

Q: What is the Prometheus alert rule for "Kafka consumer lag"?

Kafka consumer has a 30 minutes and increasing lag PromQL expression: kafka_burrow_topic_partition_offset - on(partition, cluster, topic) group_right() kafka_burrow_partition_current_offset >= (kafka_burrow_topic_partition_offset offset 15m - on(partition, cluster, topic) group_right() kafka_burrow_partition_current_offset offset 15m) AND kafka_burrow_topic_partition_offset - on(partition, cluster, topic) group_right() kafka_burrow_partition_current_offset > 0. Severity: warning. Duration: 15m.

5 Prometheus alerting rules for Kafka.Exported via danielqsj/kafka_exporter, linkedin/Burrow.These rules cover critical and warning conditions — copy and paste the YAML into your Prometheus configuration.

⚠️

Alert thresholds depend on the nature of your applications. Some queries may have arbitrary tolerance thresholds. Building an efficient monitoring platform takes time. 😉

groups:
- name: DanielqsjKafkaExporter
  rules:
      # 3 is a rough default matching Kafka's common replication factor; adjust to match your topic's actual configured replication factor.
    - alert: KafkaTopicsReplicas
      expr: min(kafka_topic_partition_in_sync_replica) by (topic) < 3
      for: 0m
      labels:
        severity: critical
      annotations:
        summary: Kafka topics replicas (instance {{ $labels.instance }})
        description: "Kafka topic {{ $labels.topic }} has fewer than 3 in-sync replicas ({{ $value }}), data durability is at risk.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
    
      # 10000 messages is a rough default; scales with your message production and consumption rate — adjust based on your workload.
    - alert: KafkaConsumerGroupLag
      expr: sum(kafka_consumergroup_lag) by (consumergroup) > 10000
      for: 1m
      labels:
        severity: warning
      annotations:
        summary: Kafka consumer group lag (instance {{ $labels.instance }})
        description: "Kafka consumer group {{ $labels.consumergroup }} is lagging behind ({{ $value }} messages)\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
    
    - alert: KafkaConsumerGroupLagIncreasing
      expr: sum by (topic, consumergroup) (delta(kafka_consumergroup_lag{topic!="__consumer_offsets",consumergroup!=""}[5m])) > 0
      for: 15m
      labels:
        severity: warning
      annotations:
        summary: Kafka consumer group lag increasing (instance {{ $labels.instance }})
        description: "Kafka consumer group {{ $labels.consumergroup }} lag on topic {{ $labels.topic }} is continuously increasing over the last 15 minutes, meaning the consumer is falling further behind even though the absolute lag threshold hasn't been crossed yet.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

3.3.1.danielqsj/kafka_exporter(3 rules)

wget https://raw.githubusercontent.com/samber/awesome-prometheus-alerts/refs/heads/master/dist/rules/kafka/danielqsj-kafka-exporter.yml

critical

3.3.1.1.Kafka topics replicas

Kafka topic {{ $labels.topic }} has fewer than 3 in-sync replicas ({{ $value }}), data durability is at risk.

  # 3 is a rough default matching Kafka's common replication factor; adjust to match your topic's actual configured replication factor.
- alert: KafkaTopicsReplicas
  expr: min(kafka_topic_partition_in_sync_replica) by (topic) < 3
  for: 0m
  labels:
    severity: critical
  annotations:
    summary: Kafka topics replicas (instance {{ $labels.instance }})
    description: "Kafka topic {{ $labels.topic }} has fewer than 3 in-sync replicas ({{ $value }}), data durability is at risk.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

warning

3.3.1.2.Kafka consumer group lag

Kafka consumer group {{ $labels.consumergroup }} is lagging behind ({{ $value }} messages)

  # 10000 messages is a rough default; scales with your message production and consumption rate — adjust based on your workload.
- alert: KafkaConsumerGroupLag
  expr: sum(kafka_consumergroup_lag) by (consumergroup) > 10000
  for: 1m
  labels:
    severity: warning
  annotations:
    summary: Kafka consumer group lag (instance {{ $labels.instance }})
    description: "Kafka consumer group {{ $labels.consumergroup }} is lagging behind ({{ $value }} messages)\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

warning

3.3.1.3.Kafka consumer group lag increasing

Kafka consumer group {{ $labels.consumergroup }} lag on topic {{ $labels.topic }} is continuously increasing over the last 15 minutes, meaning the consumer is falling further behind even though the absolute lag threshold hasn't been crossed yet.

- alert: KafkaConsumerGroupLagIncreasing
  expr: sum by (topic, consumergroup) (delta(kafka_consumergroup_lag{topic!="__consumer_offsets",consumergroup!=""}[5m])) > 0
  for: 15m
  labels:
    severity: warning
  annotations:
    summary: Kafka consumer group lag increasing (instance {{ $labels.instance }})
    description: "Kafka consumer group {{ $labels.consumergroup }} lag on topic {{ $labels.topic }} is continuously increasing over the last 15 minutes, meaning the consumer is falling further behind even though the absolute lag threshold hasn't been crossed yet.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

groups:
- name: LinkedinKafkaExporter
  rules:
    - alert: KafkaTopicOffsetDecreased
      expr: delta(kafka_burrow_partition_current_offset[1m]) < 0
      for: 0m
      labels:
        severity: warning
      annotations:
        summary: Kafka topic offset decreased (instance {{ $labels.instance }})
        description: "Kafka topic offset has decreased\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
    
    - alert: KafkaConsumerLag
      expr: kafka_burrow_topic_partition_offset - on(partition, cluster, topic) group_right() kafka_burrow_partition_current_offset >= (kafka_burrow_topic_partition_offset offset 15m - on(partition, cluster, topic) group_right() kafka_burrow_partition_current_offset offset 15m) AND kafka_burrow_topic_partition_offset - on(partition, cluster, topic) group_right() kafka_burrow_partition_current_offset > 0
      for: 15m
      labels:
        severity: warning
      annotations:
        summary: Kafka consumer lag (instance {{ $labels.instance }})
        description: "Kafka consumer has a 30 minutes and increasing lag\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

3.3.2.linkedin/Burrow(2 rules)

wget https://raw.githubusercontent.com/samber/awesome-prometheus-alerts/refs/heads/master/dist/rules/kafka/linkedin-kafka-exporter.yml

warning

3.3.2.1.Kafka topic offset decreased

Kafka topic offset has decreased

- alert: KafkaTopicOffsetDecreased
  expr: delta(kafka_burrow_partition_current_offset[1m]) < 0
  for: 0m
  labels:
    severity: warning
  annotations:
    summary: Kafka topic offset decreased (instance {{ $labels.instance }})
    description: "Kafka topic offset has decreased\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

warning

3.3.2.2.Kafka consumer lag

Kafka consumer has a 30 minutes and increasing lag

- alert: KafkaConsumerLag
  expr: kafka_burrow_topic_partition_offset - on(partition, cluster, topic) group_right() kafka_burrow_partition_current_offset >= (kafka_burrow_topic_partition_offset offset 15m - on(partition, cluster, topic) group_right() kafka_burrow_partition_current_offset offset 15m) AND kafka_burrow_topic_partition_offset - on(partition, cluster, topic) group_right() kafka_burrow_partition_current_offset > 0
  for: 15m
  labels:
    severity: warning
  annotations:
    summary: Kafka consumer lag (instance {{ $labels.instance }})
    description: "Kafka consumer has a 30 minutes and increasing lag\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

More in Message brokers

RabbitMQ Zookeeper Pulsar Nats