Skip to main content
APA
Sponsored byCAST AI — Kubernetes cost optimizationBetter Stack — Uptime monitoring and log managementVictoriaMetrics — Fast, open-source time series database and drop-in Prometheus replacement.

Clickhouse logoClickhouse Prometheus Alert Rules

23 Prometheus alerting rules for Clickhouse.Exported via Embedded Exporter.These rules cover critical and warning conditions — copy and paste the YAML into your Prometheus configuration.

⚠️

Alert thresholds depend on the nature of your applications. Some queries may have arbitrary tolerance thresholds. Building an efficient monitoring platform takes time. 😉

2.14.Clickhouse logoEmbedded Exporter(23 rules)

wget https://raw.githubusercontent.com/samber/awesome-prometheus-alerts/refs/heads/master/dist/rules/clickhouse/embedded-exporter.yml
critical

2.14.1.ClickHouse node down

No metrics received from ClickHouse exporter for over 2 minutes.

  # Adjust the job label to match your Prometheus configuration.
- alert: ClickHouseNodeDown
  expr: up{job="clickhouse"} == 0
  for: 2m
  labels:
    severity: critical
  annotations:
    summary: ClickHouse node down (instance {{ $labels.instance }})
    description: "No metrics received from ClickHouse exporter for over 2 minutes.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
critical

2.14.2.ClickHouse No Available Replicas

No available replicas in ClickHouse.

- alert: ClickHouseNoAvailableReplicas
  expr: increase(ClickHouseErrorMetric_NO_AVAILABLE_REPLICA[5m]) > 0
  for: 3m
  labels:
    severity: critical
  annotations:
    summary: ClickHouse No Available Replicas (instance {{ $labels.instance }})
    description: "No available replicas in ClickHouse.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
critical

2.14.3.ClickHouse No Live Replicas

There are too few live replicas available, risking data loss and service disruption.

- alert: ClickHouseNoLiveReplicas
  expr: increase(ClickHouseErrorMetric_TOO_FEW_LIVE_REPLICAS[5m]) > 0
  for: 3m
  labels:
    severity: critical
  annotations:
    summary: ClickHouse No Live Replicas (instance {{ $labels.instance }})
    description: "There are too few live replicas available, risking data loss and service disruption.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
warning

2.14.4.ClickHouse Interserver Connection Issues

High number of interserver connections may indicate replication or distributed query handling issues.

  # Adjust the threshold based on your cluster size and expected replication traffic.
- alert: ClickHouseInterserverConnectionIssues
  expr: ClickHouseMetrics_InterserverConnection > 50
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: ClickHouse Interserver Connection Issues (instance {{ $labels.instance }})
    description: "High number of interserver connections may indicate replication or distributed query handling issues.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
warning

2.14.5.ClickHouse ZooKeeper Connection Issues

ClickHouse is experiencing issues with ZooKeeper connections, which may affect cluster state and coordination.

- alert: ClickHouseZooKeeperConnectionIssues
  expr: ClickHouseMetrics_ZooKeeperSession != 1
  for: 3m
  labels:
    severity: warning
  annotations:
    summary: ClickHouse ZooKeeper Connection Issues (instance {{ $labels.instance }})
    description: "ClickHouse is experiencing issues with ZooKeeper connections, which may affect cluster state and coordination.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
critical

2.14.6.ClickHouse Memory Usage Critical

Memory usage is critically high, over 90%.

- alert: ClickHouseMemoryUsageCritical
  expr: ClickHouseAsyncMetrics_CGroupMemoryUsed / ClickHouseAsyncMetrics_CGroupMemoryTotal * 100 > 90 and ClickHouseAsyncMetrics_CGroupMemoryTotal > 0
  for: 5m
  labels:
    severity: critical
  annotations:
    summary: ClickHouse Memory Usage Critical (instance {{ $labels.instance }})
    description: "Memory usage is critically high, over 90%.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
warning

2.14.7.ClickHouse Memory Usage Warning

Memory usage is over 80%.

- alert: ClickHouseMemoryUsageWarning
  expr: ClickHouseAsyncMetrics_CGroupMemoryUsed / ClickHouseAsyncMetrics_CGroupMemoryTotal * 100 > 80 and ClickHouseAsyncMetrics_CGroupMemoryTotal > 0
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: ClickHouse Memory Usage Warning (instance {{ $labels.instance }})
    description: "Memory usage is over 80%.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
critical

2.14.8.ClickHouse Disk Space Critical on Default

Disk space on default disk is critically low, below 10%.

- alert: ClickHouseDiskSpaceCriticalOnDefault
  expr: ClickHouseAsyncMetrics_DiskAvailable_default / (ClickHouseAsyncMetrics_DiskAvailable_default + ClickHouseAsyncMetrics_DiskUsed_default) * 100 < 10 and (ClickHouseAsyncMetrics_DiskAvailable_default + ClickHouseAsyncMetrics_DiskUsed_default) > 0
  for: 2m
  labels:
    severity: critical
  annotations:
    summary: ClickHouse Disk Space Critical on Default (instance {{ $labels.instance }})
    description: "Disk space on default disk is critically low, below 10%.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
warning

2.14.9.ClickHouse Disk Space Low on Default

Disk space on default is below 20%.

- alert: ClickHouseDiskSpaceLowOnDefault
  expr: ClickHouseAsyncMetrics_DiskAvailable_default / (ClickHouseAsyncMetrics_DiskAvailable_default + ClickHouseAsyncMetrics_DiskUsed_default) * 100 < 20 and (ClickHouseAsyncMetrics_DiskAvailable_default + ClickHouseAsyncMetrics_DiskUsed_default) > 0
  for: 2m
  labels:
    severity: warning
  annotations:
    summary: ClickHouse Disk Space Low on Default (instance {{ $labels.instance }})
    description: "Disk space on default is below 20%.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
warning

2.14.10.ClickHouse too many parts per partition

The number of data parts per partition on {{ $labels.instance }} is high, an early warning sign of merge pressure before ClickHouse starts rejecting or delaying INSERTs.

  # 100 is a rough default; part count scales with your insert frequency and batch size — adjust based on your workload.
- alert: ClickHouseTooManyPartsPerPartition
  expr: ClickHouseAsyncMetrics_MaxPartCountForPartition > 100
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: ClickHouse too many parts per partition (instance {{ $labels.instance }})
    description: "The number of data parts per partition on {{ $labels.instance }} is high, an early warning sign of merge pressure before ClickHouse starts rejecting or delaying INSERTs.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
warning

2.14.11.ClickHouse high network usage

High network usage. ClickHouse network usage exceeds 100MB/s.

  # Please replace the threshold with an appropriate value
- alert: ClickHouseHighNetworkUsage
  expr: rate(ClickHouseProfileEvents_NetworkSendBytes[1m]) > 100*1024*1024 or rate(ClickHouseProfileEvents_NetworkReceiveBytes[1m]) > 100*1024*1024
  for: 2m
  labels:
    severity: warning
  annotations:
    summary: ClickHouse high network usage (instance {{ $labels.instance }})
    description: "High network usage. ClickHouse network usage exceeds 100MB/s.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
warning

2.14.12.ClickHouse High TCP Connections

High number of TCP connections, indicating heavy client or inter-cluster communication.

  # Please replace the threshold with an appropriate value
- alert: ClickHouseHighTCPConnections
  expr: ClickHouseMetrics_TCPConnection > 400
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: ClickHouse High TCP Connections (instance {{ $labels.instance }})
    description: "High number of TCP connections, indicating heavy client or inter-cluster communication.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
critical

2.14.13.ClickHouse Replica Errors

Critical replica errors detected, either all replicas are stale or lost.

- alert: ClickHouseReplicaErrors
  expr: increase(ClickHouseErrorMetric_ALL_REPLICAS_ARE_STALE[5m]) > 0 or increase(ClickHouseErrorMetric_ALL_REPLICAS_LOST[5m]) > 0
  for: 3m
  labels:
    severity: critical
  annotations:
    summary: ClickHouse Replica Errors (instance {{ $labels.instance }})
    description: "Critical replica errors detected, either all replicas are stale or lost.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
critical

2.14.14.ClickHouse readonly replica

A Replicated table on {{ $labels.instance }} is stuck in readonly mode, usually after losing its ZooKeeper session or starting up without ZooKeeper available.

- alert: ClickHouseReadonlyReplica
  expr: ClickHouseMetrics_ReadonlyReplica > 0
  for: 0m
  labels:
    severity: critical
  annotations:
    summary: ClickHouse readonly replica (instance {{ $labels.instance }})
    description: "A Replicated table on {{ $labels.instance }} is stuck in readonly mode, usually after losing its ZooKeeper session or starting up without ZooKeeper available.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
critical

2.14.15.ClickHouse replicated data loss

A wanted data part does not exist on any replica, including offline ones, indicating actual data loss on {{ $labels.instance }}.

- alert: ClickHouseReplicatedDataLoss
  expr: increase(ClickHouseProfileEvents_ReplicatedDataLoss[1m]) > 0
  for: 1m
  labels:
    severity: critical
  annotations:
    summary: ClickHouse replicated data loss (instance {{ $labels.instance }})
    description: "A wanted data part does not exist on any replica, including offline ones, indicating actual data loss on {{ $labels.instance }}.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
critical

2.14.16.ClickHouse zookeeper hardware exception

Zookeeper hardware exception: network issues communicating with ZooKeeper

- alert: ClickHouseZookeeperHardwareException
  expr: increase(ClickHouseProfileEvents_ZooKeeperHardwareExceptions[1m]) > 0
  for: 1m
  labels:
    severity: critical
  annotations:
    summary: ClickHouse zookeeper hardware exception (instance {{ $labels.instance }})
    description: "Zookeeper hardware exception: network issues communicating with ZooKeeper\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
critical

2.14.17.ClickHouse distributed rejected inserts

INSERTs into Distributed tables rejected due to pending bytes limit.

  # 3 in 5m is a rough default; depends on your configured pending-bytes limit and write volume — adjust based on your workload.
- alert: ClickHouseDistributedRejectedInserts
  expr: increase(ClickHouseProfileEvents_DistributedRejectedInserts[5m]) > 3
  for: 2m
  labels:
    severity: critical
  annotations:
    summary: ClickHouse distributed rejected inserts (instance {{ $labels.instance }})
    description: "INSERTs into Distributed tables rejected due to pending bytes limit.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
warning

2.14.18.ClickHouse replication lag

Replication lag between the freshest replicated data part and the most stale part still to be replicated is high, meaning reads from lagging replicas may return stale data.

  # 300s threshold is a rough default; acceptable lag depends on your network and replication workload — adjust based on your baseline.
- alert: ClickHouseReplicationLag
  expr: ClickHouseAsyncMetrics_ReplicasMaxAbsoluteDelay > 300
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: ClickHouse replication lag (instance {{ $labels.instance }})
    description: "Replication lag between the freshest replicated data part and the most stale part still to be replicated is high, meaning reads from lagging replicas may return stale data.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
warning

2.14.19.ClickHouse rejected insert queries

INSERTs rejected due to too many active data parts. Reduce insert frequency.

  # 2 in 1m is a rough default; depends on your insert frequency and active-parts configuration — adjust based on your workload.
- alert: ClickHouseRejectedInsertQueries
  expr: increase(ClickHouseProfileEvents_RejectedInserts[1m]) > 2
  for: 1m
  labels:
    severity: warning
  annotations:
    summary: ClickHouse rejected insert queries (instance {{ $labels.instance }})
    description: "INSERTs rejected due to too many active data parts. Reduce insert frequency.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
warning

2.14.20.ClickHouse delayed insert queries

INSERTs delayed due to high number of active parts.

  # 10 in 5m is a rough default; depends on your insert frequency and active-parts configuration — adjust based on your workload.
- alert: ClickHouseDelayedInsertQueries
  expr: increase(ClickHouseProfileEvents_DelayedInserts[5m]) > 10
  for: 2m
  labels:
    severity: warning
  annotations:
    summary: ClickHouse delayed insert queries (instance {{ $labels.instance }})
    description: "INSERTs delayed due to high number of active parts.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
warning

2.14.21.ClickHouse Disk Space Low on Backups

Disk space on backups is below 20%.

- alert: ClickHouseDiskSpaceLowOnBackups
  expr: ClickHouseAsyncMetrics_DiskAvailable_backups / (ClickHouseAsyncMetrics_DiskAvailable_backups + ClickHouseAsyncMetrics_DiskUsed_backups) * 100 < 20 and (ClickHouseAsyncMetrics_DiskAvailable_backups + ClickHouseAsyncMetrics_DiskUsed_backups) > 0
  for: 2m
  labels:
    severity: warning
  annotations:
    summary: ClickHouse Disk Space Low on Backups (instance {{ $labels.instance }})
    description: "Disk space on backups is below 20%.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
info

2.14.22.ClickHouse Authentication Failures

Authentication failures detected, indicating potential security issues or misconfiguration.

- alert: ClickHouseAuthenticationFailures
  expr: increase(ClickHouseErrorMetric_AUTHENTICATION_FAILED[5m]) > 3
  for: 0m
  labels:
    severity: info
  annotations:
    summary: ClickHouse Authentication Failures (instance {{ $labels.instance }})
    description: "Authentication failures detected, indicating potential security issues or misconfiguration.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
info

2.14.23.ClickHouse Access Denied Errors

Access denied errors have been logged, which could indicate permission issues or unauthorized access attempts.

  # 3 in 5m is a rough default; denial volume correlates with how many clients/integrations connect to this cluster — adjust based on your baseline.
- alert: ClickHouseAccessDeniedErrors
  expr: increase(ClickHouseErrorMetric_RESOURCE_ACCESS_DENIED[5m]) > 3
  for: 0m
  labels:
    severity: info
  annotations:
    summary: ClickHouse Access Denied Errors (instance {{ $labels.instance }})
    description: "Access denied errors have been logged, which could indicate permission issues or unauthorized access attempts.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

More in Databases

MySQLPostgreSQLSQL ServerOracle DatabasePatroniPGBouncerRedisMemcachedMongoDBElasticsearchOpenSearchMeilisearchCassandraCouchDBSolr