Skip to main content
APA
Sponsored by CAST AI — Kubernetes cost optimization Better Stack — Uptime monitoring and log management
⚠️

Alert thresholds depend on the nature of your applications. Some queries may have arbitrary tolerance thresholds. Building an efficient monitoring platform takes time. 😉

Clickhouse Prometheus Alert Rules

19 Prometheus alerting rules for Clickhouse. Exported via Embedded Exporter. These rules cover critical and warning conditions — copy and paste the YAML into your Prometheus configuration.

2.14. Embedded Exporter (19 rules) 

wget https://raw.githubusercontent.com/samber/awesome-prometheus-alerts/refs/heads/master/dist/rules/clickhouse/embedded-exporter.yml
critical

2.14.1. ClickHouse node down

No metrics received from ClickHouse exporter for over 2 minutes. 

  # Adjust the job label to match your Prometheus configuration.
- alert: ClickHouseNodeDown
  expr: up{job="clickhouse"} == 0
  for: 2m
  labels:
    severity: critical
  annotations:
    summary: ClickHouse node down (instance {{ $labels.instance }})
    description: "No metrics received from ClickHouse exporter for over 2 minutes.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
critical

2.14.2. ClickHouse Memory Usage Critical

Memory usage is critically high, over 90%. 

- alert: ClickHouseMemoryUsageCritical
  expr: ClickHouseAsyncMetrics_CGroupMemoryUsed / ClickHouseAsyncMetrics_CGroupMemoryTotal * 100 > 90 and ClickHouseAsyncMetrics_CGroupMemoryTotal > 0
  for: 5m
  labels:
    severity: critical
  annotations:
    summary: ClickHouse Memory Usage Critical (instance {{ $labels.instance }})
    description: "Memory usage is critically high, over 90%.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
warning

2.14.3. ClickHouse Memory Usage Warning

Memory usage is over 80%. 

- alert: ClickHouseMemoryUsageWarning
  expr: ClickHouseAsyncMetrics_CGroupMemoryUsed / ClickHouseAsyncMetrics_CGroupMemoryTotal * 100 > 80 and ClickHouseAsyncMetrics_CGroupMemoryTotal > 0
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: ClickHouse Memory Usage Warning (instance {{ $labels.instance }})
    description: "Memory usage is over 80%.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
warning

2.14.4. ClickHouse Disk Space Low on Default

Disk space on default is below 20%. 

- alert: ClickHouseDiskSpaceLowOnDefault
  expr: ClickHouseAsyncMetrics_DiskAvailable_default / (ClickHouseAsyncMetrics_DiskAvailable_default + ClickHouseAsyncMetrics_DiskUsed_default) * 100 < 20 and (ClickHouseAsyncMetrics_DiskAvailable_default + ClickHouseAsyncMetrics_DiskUsed_default) > 0
  for: 2m
  labels:
    severity: warning
  annotations:
    summary: ClickHouse Disk Space Low on Default (instance {{ $labels.instance }})
    description: "Disk space on default is below 20%.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
critical

2.14.5. ClickHouse Disk Space Critical on Default

Disk space on default disk is critically low, below 10%. 

- alert: ClickHouseDiskSpaceCriticalOnDefault
  expr: ClickHouseAsyncMetrics_DiskAvailable_default / (ClickHouseAsyncMetrics_DiskAvailable_default + ClickHouseAsyncMetrics_DiskUsed_default) * 100 < 10 and (ClickHouseAsyncMetrics_DiskAvailable_default + ClickHouseAsyncMetrics_DiskUsed_default) > 0
  for: 2m
  labels:
    severity: critical
  annotations:
    summary: ClickHouse Disk Space Critical on Default (instance {{ $labels.instance }})
    description: "Disk space on default disk is critically low, below 10%.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
warning

2.14.6. ClickHouse Disk Space Low on Backups

Disk space on backups is below 20%. 

- alert: ClickHouseDiskSpaceLowOnBackups
  expr: ClickHouseAsyncMetrics_DiskAvailable_backups / (ClickHouseAsyncMetrics_DiskAvailable_backups + ClickHouseAsyncMetrics_DiskUsed_backups) * 100 < 20 and (ClickHouseAsyncMetrics_DiskAvailable_backups + ClickHouseAsyncMetrics_DiskUsed_backups) > 0
  for: 2m
  labels:
    severity: warning
  annotations:
    summary: ClickHouse Disk Space Low on Backups (instance {{ $labels.instance }})
    description: "Disk space on backups is below 20%.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
critical

2.14.7. ClickHouse Replica Errors

Critical replica errors detected, either all replicas are stale or lost. 

- alert: ClickHouseReplicaErrors
  expr: ClickHouseErrorMetric_ALL_REPLICAS_ARE_STALE == 1 or ClickHouseErrorMetric_ALL_REPLICAS_LOST == 1
  for: 0m
  labels:
    severity: critical
  annotations:
    summary: ClickHouse Replica Errors (instance {{ $labels.instance }})
    description: "Critical replica errors detected, either all replicas are stale or lost.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
critical

2.14.8. ClickHouse No Available Replicas

No available replicas in ClickHouse. 

- alert: ClickHouseNoAvailableReplicas
  expr: ClickHouseErrorMetric_NO_AVAILABLE_REPLICA == 1
  for: 0m
  labels:
    severity: critical
  annotations:
    summary: ClickHouse No Available Replicas (instance {{ $labels.instance }})
    description: "No available replicas in ClickHouse.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
critical

2.14.9. ClickHouse No Live Replicas

There are too few live replicas available, risking data loss and service disruption. 

- alert: ClickHouseNoLiveReplicas
  expr: ClickHouseErrorMetric_TOO_FEW_LIVE_REPLICAS == 1
  for: 0m
  labels:
    severity: critical
  annotations:
    summary: ClickHouse No Live Replicas (instance {{ $labels.instance }})
    description: "There are too few live replicas available, risking data loss and service disruption.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
warning

2.14.10. ClickHouse High TCP Connections

High number of TCP connections, indicating heavy client or inter-cluster communication. 

  # Please replace the threshold with an appropriate value
- alert: ClickHouseHighTCPConnections
  expr: ClickHouseMetrics_TCPConnection > 400
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: ClickHouse High TCP Connections (instance {{ $labels.instance }})
    description: "High number of TCP connections, indicating heavy client or inter-cluster communication.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
warning

2.14.11. ClickHouse Interserver Connection Issues

High number of interserver connections may indicate replication or distributed query handling issues. 

  # Adjust the threshold based on your cluster size and expected replication traffic.
- alert: ClickHouseInterserverConnectionIssues
  expr: ClickHouseMetrics_InterserverConnection > 50
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: ClickHouse Interserver Connection Issues (instance {{ $labels.instance }})
    description: "High number of interserver connections may indicate replication or distributed query handling issues.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
warning

2.14.12. ClickHouse ZooKeeper Connection Issues

ClickHouse is experiencing issues with ZooKeeper connections, which may affect cluster state and coordination. 

- alert: ClickHouseZooKeeperConnectionIssues
  expr: ClickHouseMetrics_ZooKeeperSession != 1
  for: 3m
  labels:
    severity: warning
  annotations:
    summary: ClickHouse ZooKeeper Connection Issues (instance {{ $labels.instance }})
    description: "ClickHouse is experiencing issues with ZooKeeper connections, which may affect cluster state and coordination.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
info

2.14.13. ClickHouse Authentication Failures

Authentication failures detected, indicating potential security issues or misconfiguration. 

- alert: ClickHouseAuthenticationFailures
  expr: increase(ClickHouseErrorMetric_AUTHENTICATION_FAILED[5m]) > 3
  for: 0m
  labels:
    severity: info
  annotations:
    summary: ClickHouse Authentication Failures (instance {{ $labels.instance }})
    description: "Authentication failures detected, indicating potential security issues or misconfiguration.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
info

2.14.14. ClickHouse Access Denied Errors

Access denied errors have been logged, which could indicate permission issues or unauthorized access attempts. 

- alert: ClickHouseAccessDeniedErrors
  expr: increase(ClickHouseErrorMetric_RESOURCE_ACCESS_DENIED[5m]) > 3
  for: 0m
  labels:
    severity: info
  annotations:
    summary: ClickHouse Access Denied Errors (instance {{ $labels.instance }})
    description: "Access denied errors have been logged, which could indicate permission issues or unauthorized access attempts.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
warning

2.14.15. ClickHouse rejected insert queries

INSERTs rejected due to too many active data parts. Reduce insert frequency. 

- alert: ClickHouseRejectedInsertQueries
  expr: increase(ClickHouseProfileEvents_RejectedInserts[1m]) > 2
  for: 1m
  labels:
    severity: warning
  annotations:
    summary: ClickHouse rejected insert queries (instance {{ $labels.instance }})
    description: "INSERTs rejected due to too many active data parts. Reduce insert frequency.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
warning

2.14.16. ClickHouse delayed insert queries

INSERTs delayed due to high number of active parts. 

- alert: ClickHouseDelayedInsertQueries
  expr: increase(ClickHouseProfileEvents_DelayedInserts[5m]) > 10
  for: 2m
  labels:
    severity: warning
  annotations:
    summary: ClickHouse delayed insert queries (instance {{ $labels.instance }})
    description: "INSERTs delayed due to high number of active parts.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
critical

2.14.17. ClickHouse zookeeper hardware exception

Zookeeper hardware exception: network issues communicating with ZooKeeper 

- alert: ClickHouseZookeeperHardwareException
  expr: increase(ClickHouseProfileEvents_ZooKeeperHardwareExceptions[1m]) > 0
  for: 1m
  labels:
    severity: critical
  annotations:
    summary: ClickHouse zookeeper hardware exception (instance {{ $labels.instance }})
    description: "Zookeeper hardware exception: network issues communicating with ZooKeeper\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
warning

2.14.18. ClickHouse high network usage

High network usage. ClickHouse network usage exceeds 100MB/s. 

  # Please replace the threshold with an appropriate value
- alert: ClickHouseHighNetworkUsage
  expr: rate(ClickHouseProfileEvents_NetworkSendBytes[1m]) > 100*1024*1024 or rate(ClickHouseProfileEvents_NetworkReceiveBytes[1m]) > 100*1024*1024
  for: 2m
  labels:
    severity: warning
  annotations:
    summary: ClickHouse high network usage (instance {{ $labels.instance }})
    description: "High network usage. ClickHouse network usage exceeds 100MB/s.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
critical

2.14.19. ClickHouse distributed rejected inserts

INSERTs into Distributed tables rejected due to pending bytes limit. 

- alert: ClickHouseDistributedRejectedInserts
  expr: increase(ClickHouseProfileEvents_DistributedRejectedInserts[5m]) > 3
  for: 2m
  labels:
    severity: critical
  annotations:
    summary: ClickHouse distributed rejected inserts (instance {{ $labels.instance }})
    description: "INSERTs into Distributed tables rejected due to pending bytes limit.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

More in Databases

MySQL PostgreSQL SQL Server Oracle Database Patroni PGBouncer Redis Memcached MongoDB Elasticsearch OpenSearch Meilisearch Cassandra CouchDB Solr