warning
4.3.1.1.HAProxy HTTP slowing down
HAProxy backend max total time is above 1s on {{ $labels.proxy }} - {{ $value | printf "%.2f"}}s
# 1s threshold is a rough default; acceptable latency varies by backend and your SLO — adjust based on your baseline.
- alert: HAProxyHTTPSlowingDown
expr: avg by (instance, proxy) (haproxy_backend_max_total_time_seconds) > 1
for: 1m
labels:
severity: warning
annotations:
summary: HAProxy HTTP slowing down (instance {{ $labels.instance }})
description: "HAProxy backend max total time is above 1s on {{ $labels.proxy }} - {{ $value | printf \"%.2f\"}}s\n VALUE = {{ $value }}\n LABELS = {{ $labels }}"warning
4.3.1.2.HAProxy dropping logs
HAProxy {{ $labels.instance }} is dropping log messages, which usually indicates the syslog/logging backend cannot keep up or is unreachable.
- alert: HAProxyDroppingLogs
expr: rate(haproxy_process_dropped_logs_total[5m]) != 0
for: 0m
labels:
severity: warning
annotations:
summary: HAProxy dropping logs (instance {{ $labels.instance }})
description: "HAProxy {{ $labels.instance }} is dropping log messages, which usually indicates the syslog/logging backend cannot keep up or is unreachable.\n VALUE = {{ $value }}\n LABELS = {{ $labels }}"warning
4.3.1.3.HAProxy backend healthcheck flapping
Backend {{ $labels.proxy }} on {{ $labels.instance }} has servers repeatedly transitioning between UP and DOWN health-check states, which points to unstable health checks or a flaky network path rather than a hard failure.
- alert: HAProxyBackendHealthcheckFlapping
expr: rate(haproxy_backend_check_up_down_total[5m]) != 0
for: 0m
labels:
severity: warning
annotations:
summary: HAProxy backend healthcheck flapping (instance {{ $labels.instance }})
description: "Backend {{ $labels.proxy }} on {{ $labels.instance }} has servers repeatedly transitioning between UP and DOWN health-check states, which points to unstable health checks or a flaky network path rather than a hard failure.\n VALUE = {{ $value }}\n LABELS = {{ $labels }}"warning
4.3.1.4.HAProxy server healthcheck flapping
Server {{ $labels.server }} on {{ $labels.instance }} is repeatedly transitioning between UP and DOWN health-check states, which may indicate an unstable backend node or overly sensitive health-check thresholds.
- alert: HAProxyServerHealthcheckFlapping
expr: rate(haproxy_server_check_up_down_total[5m]) != 0
for: 0m
labels:
severity: warning
annotations:
summary: HAProxy server healthcheck flapping (instance {{ $labels.instance }})
description: "Server {{ $labels.server }} on {{ $labels.instance }} is repeatedly transitioning between UP and DOWN health-check states, which may indicate an unstable backend node or overly sensitive health-check thresholds.\n VALUE = {{ $value }}\n LABELS = {{ $labels }}"critical
4.3.1.5.HAProxy high HTTP 4xx error rate backend
Too many HTTP requests with status 4xx (> 5%) on backend {{ $labels.proxy }}
# 5% 4xx rate is a rough default. Client-error rates vary widely by application (bots, API misuse, rate limits, short-lived token expiry, malformed clients) — adjust based on your baseline.
- alert: HAProxyHighHTTP4xxErrorRateBackend
expr: ((sum by (proxy) (rate(haproxy_server_http_responses_total{code="4xx"}[1m])) / sum by (proxy) (rate(haproxy_server_http_responses_total[1m]))) * 100) > 5 and sum by (proxy) (rate(haproxy_server_http_responses_total[1m])) > 0
for: 1m
labels:
severity: critical
annotations:
summary: HAProxy high HTTP 4xx error rate backend (instance {{ $labels.instance }})
description: "Too many HTTP requests with status 4xx (> 5%) on backend {{ $labels.proxy }}\n VALUE = {{ $value }}\n LABELS = {{ $labels }}"critical
4.3.1.6.HAProxy high HTTP 5xx error rate backend
Too many HTTP requests with status 5xx (> 5%) on backend {{ $labels.proxy }}
- alert: HAProxyHighHTTP5xxErrorRateBackend
expr: ((sum by (proxy) (rate(haproxy_server_http_responses_total{code="5xx"}[1m])) / sum by (proxy) (rate(haproxy_server_http_responses_total[1m]))) * 100) > 5 and sum by (proxy) (rate(haproxy_server_http_responses_total[1m])) > 0
for: 1m
labels:
severity: critical
annotations:
summary: HAProxy high HTTP 5xx error rate backend (instance {{ $labels.instance }})
description: "Too many HTTP requests with status 5xx (> 5%) on backend {{ $labels.proxy }}\n VALUE = {{ $value }}\n LABELS = {{ $labels }}"critical
4.3.1.7.HAProxy high HTTP 4xx error rate server
Too many HTTP requests with status 4xx (> 5%) on server {{ $labels.server }}
# 5% 4xx rate is a rough default. Client-error rates vary widely by application (bots, API misuse, rate limits, short-lived token expiry, malformed clients) — adjust based on your baseline.
- alert: HAProxyHighHTTP4xxErrorRateServer
expr: ((sum by (server) (rate(haproxy_server_http_responses_total{code="4xx"}[1m])) / sum by (server) (rate(haproxy_server_http_responses_total[1m]))) * 100) > 5 and sum by (server) (rate(haproxy_server_http_responses_total[1m])) > 0
for: 1m
labels:
severity: critical
annotations:
summary: HAProxy high HTTP 4xx error rate server (instance {{ $labels.instance }})
description: "Too many HTTP requests with status 4xx (> 5%) on server {{ $labels.server }}\n VALUE = {{ $value }}\n LABELS = {{ $labels }}"critical
4.3.1.8.HAProxy high HTTP 5xx error rate server
Too many HTTP requests with status 5xx (> 5%) on server {{ $labels.server }}
- alert: HAProxyHighHTTP5xxErrorRateServer
expr: ((sum by (server) (rate(haproxy_server_http_responses_total{code="5xx"}[1m])) / sum by (server) (rate(haproxy_server_http_responses_total[1m]))) * 100) > 5 and sum by (server) (rate(haproxy_server_http_responses_total[1m])) > 0
for: 1m
labels:
severity: critical
annotations:
summary: HAProxy high HTTP 5xx error rate server (instance {{ $labels.instance }})
description: "Too many HTTP requests with status 5xx (> 5%) on server {{ $labels.server }}\n VALUE = {{ $value }}\n LABELS = {{ $labels }}"critical
4.3.1.9.HAProxy server response errors
Too many response errors to {{ $labels.server }} server (> 5%).
- alert: HAProxyServerResponseErrors
expr: (sum by (server) (rate(haproxy_server_response_errors_total[1m])) / sum by (server) (rate(haproxy_server_http_responses_total[1m]))) * 100 > 5 and sum by (server) (rate(haproxy_server_http_responses_total[1m])) > 0
for: 1m
labels:
severity: critical
annotations:
summary: HAProxy server response errors (instance {{ $labels.instance }})
description: "Too many response errors to {{ $labels.server }} server (> 5%).\n VALUE = {{ $value }}\n LABELS = {{ $labels }}"critical
4.3.1.10.HAProxy backend connection errors
Too many connection errors to {{ $labels.proxy }} backend (> 100 req/s). Request throughput may be too high.
# 100 req/s is a rough default; scales with your request throughput — adjust based on your workload.
- alert: HAProxyBackendConnectionErrors
expr: (sum by (proxy) (rate(haproxy_backend_connection_errors_total[1m]))) > 100
for: 1m
labels:
severity: critical
annotations:
summary: HAProxy backend connection errors (instance {{ $labels.instance }})
description: "Too many connection errors to {{ $labels.proxy }} backend (> 100 req/s). Request throughput may be too high.\n VALUE = {{ $value }}\n LABELS = {{ $labels }}"critical
4.3.1.11.HAProxy server connection errors
Too many connection errors to {{ $labels.proxy }} (> 100 req/s). Request throughput may be too high.
# 100 req/s is a rough default; scales with your request throughput — adjust based on your workload.
- alert: HAProxyServerConnectionErrors
expr: (sum by (proxy) (rate(haproxy_server_connection_errors_total[1m]))) > 100
for: 0m
labels:
severity: critical
annotations:
summary: HAProxy server connection errors (instance {{ $labels.instance }})
description: "Too many connection errors to {{ $labels.proxy }} (> 100 req/s). Request throughput may be too high.\n VALUE = {{ $value }}\n LABELS = {{ $labels }}"warning
4.3.1.12.HAProxy backend max active session > 80%
Session limit from backend {{ $labels.proxy }} reached 80% of limit - {{ $value | printf "%.2f"}}%
- alert: HAProxyBackendMaxActiveSession>80%
expr: (haproxy_backend_current_sessions / haproxy_backend_limit_sessions * 100) > 80 and haproxy_backend_limit_sessions > 0
for: 2m
labels:
severity: warning
annotations:
summary: HAProxy backend max active session > 80% (instance {{ $labels.instance }})
description: "Session limit from backend {{ $labels.proxy }} reached 80% of limit - {{ $value | printf \"%.2f\"}}%\n VALUE = {{ $value }}\n LABELS = {{ $labels }}"warning
4.3.1.13.HAProxy pending requests
Some HAProxy requests are pending on {{ $labels.proxy }} - {{ $value | printf "%.2f"}}
- alert: HAProxyPendingRequests
expr: sum by (proxy) (haproxy_backend_current_queue) > 0
for: 2m
labels:
severity: warning
annotations:
summary: HAProxy pending requests (instance {{ $labels.instance }})
description: "Some HAProxy requests are pending on {{ $labels.proxy }} - {{ $value | printf \"%.2f\"}}\n VALUE = {{ $value }}\n LABELS = {{ $labels }}"warning
4.3.1.14.HAProxy retry high
High rate of retry on {{ $labels.proxy }} - {{ $value | printf "%.2f"}}
# 10 is a rough default; retry rate correlates with backend health and traffic volume — adjust based on your workload.
- alert: HAProxyRetryHigh
expr: sum by (proxy) (rate(haproxy_backend_retry_warnings_total[1m])) > 10
for: 2m
labels:
severity: warning
annotations:
summary: HAProxy retry high (instance {{ $labels.instance }})
description: "High rate of retry on {{ $labels.proxy }} - {{ $value | printf \"%.2f\"}}\n VALUE = {{ $value }}\n LABELS = {{ $labels }}"warning
4.3.1.15.HAProxy server healthcheck failure
Some server healthcheck are failing on {{ $labels.server }} ({{ $value }} in the last 1m)
- alert: HAProxyServerHealthcheckFailure
expr: increase(haproxy_server_check_failures_total[1m]) > 2
for: 0m
labels:
severity: warning
annotations:
summary: HAProxy server healthcheck failure (instance {{ $labels.instance }})
description: "Some server healthcheck are failing on {{ $labels.server }} ({{ $value }} in the last 1m)\n VALUE = {{ $value }}\n LABELS = {{ $labels }}"critical
4.3.1.16.HAproxy has no alive backends
HAProxy has no alive active or backup backends for {{ $labels.proxy }}
- alert: HAproxyHasNoAliveBackends
expr: haproxy_backend_active_servers + haproxy_backend_backup_servers == 0
for: 0m
labels:
severity: critical
annotations:
summary: HAproxy has no alive backends (instance {{ $labels.instance }})
description: "HAProxy has no alive active or backup backends for {{ $labels.proxy }}\n VALUE = {{ $value }}\n LABELS = {{ $labels }}"warning
4.3.1.17.HAProxy frontend security blocked requests
HAProxy is blocking requests for security reason
# 10 is a rough default; blocked-request volume correlates with your exposure to scans/attacks and the false-positive rate of your security rules — adjust based on your baseline.
- alert: HAProxyFrontendSecurityBlockedRequests
expr: sum by (proxy) (rate(haproxy_frontend_denied_connections_total[2m])) > 10
for: 2m
labels:
severity: warning
annotations:
summary: HAProxy frontend security blocked requests (instance {{ $labels.instance }})
description: "HAProxy is blocking requests for security reason\n VALUE = {{ $value }}\n LABELS = {{ $labels }}"
HaProxy Prometheus Alert Rules