What is the Prometheus alert rule for "MySQL down"?

MySQL instance is down on {{ $labels.instance }} PromQL expression: mysql_up == 0. Severity: critical. Duration: 1m.

What is the Prometheus alert rule for "MySQL slow queries"?

MySQL server has some new slow queries ({{ $value }} in the last minute). PromQL expression: delta(mysql_global_status_slow_queries[1m]) > 0. Severity: warning. Duration: 2m.

What is the Prometheus alert rule for "MySQL InnoDB log waits"?

MySQL innodb log writes stalling ({{ $value }} waits/s) PromQL expression: deriv(mysql_global_status_innodb_log_waits[15m]) > 10. Severity: warning.

What is the Prometheus alert rule for "MySQL restarted"?

MySQL has just been restarted, less than one minute ago on {{ $labels.instance }}. PromQL expression: mysql_global_status_uptime < 60. Severity: info.

What is the Prometheus alert rule for "MySQL High QPS"?

MySQL is being overload with unusual QPS (> 10k QPS). PromQL expression: deriv(mysql_global_status_questions[1m]) > 10000. Severity: info. Duration: 2m.

What is the Prometheus alert rule for "MySQL InnoDB Force Recovery is enabled"?

MySQL InnoDB force recovery is enabled on {{ $labels.instance }} PromQL expression: mysql_global_variables_innodb_force_recovery != 0. Severity: warning. Duration: 2m.

What is the Prometheus alert rule for "MySQL InnoDB history_len too long"?

MySQL history_len (undo log) too long on {{ $labels.instance }} PromQL expression: mysql_info_schema_innodb_metrics_transaction_trx_rseg_history_len > 50000. Severity: warning. Duration: 2m.

⚠️

Alert thresholds depend on the nature of your applications. Some queries may have arbitrary tolerance thresholds. Building an efficient monitoring platform takes time. 😉

MySQL Prometheus Alert Rules

Q: What is the Prometheus alert rule for "MySQL Slave replication lag"?

MySQL replication lag on {{ $labels.instance }} PromQL expression: ( (mysql_slave_status_seconds_behind_master - mysql_slave_status_sql_delay) and ON (instance) mysql_slave_status_master_server_id > 0 ) > 30. Severity: critical. Duration: 1m.

14 Prometheus alerting rules for MySQL. Exported via prometheus/mysqld_exporter. These rules cover critical and warning conditions — copy and paste the YAML into your Prometheus configuration.

groups:
- name: MysqldExporter
  rules:
      # 1m delay allows a restart without triggering an alert.
    - alert: MySQLDown
      expr: mysql_up == 0
      for: 1m
      labels:
        severity: critical
      annotations:
        summary: MySQL down (instance {{ $labels.instance }})
        description: "MySQL instance is down on {{ $labels.instance }}\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
    
    - alert: MySQLTooManyConnections(>80%)
      expr: max_over_time(mysql_global_status_threads_connected[1m]) / mysql_global_variables_max_connections * 100 > 80 and mysql_global_variables_max_connections > 0
      for: 2m
      labels:
        severity: warning
      annotations:
        summary: MySQL too many connections (> 80%) (instance {{ $labels.instance }})
        description: "More than 80% of MySQL connections are in use on {{ $labels.instance }}\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
    
    - alert: MySQLHighPreparedStatementsUtilization(>80%)
      expr: max_over_time(mysql_global_status_prepared_stmt_count[1m]) / mysql_global_variables_max_prepared_stmt_count * 100 > 80 and mysql_global_variables_max_prepared_stmt_count > 0
      for: 2m
      labels:
        severity: warning
      annotations:
        summary: MySQL high prepared statements utilization (> 80%) (instance {{ $labels.instance }})
        description: "High utilization of prepared statements (>80%) on {{ $labels.instance }}\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
    
    - alert: MySQLHighThreadsRunning
      expr: max_over_time(mysql_global_status_threads_running[1m]) / mysql_global_variables_max_connections * 100 > 60 and mysql_global_variables_max_connections > 0
      for: 2m
      labels:
        severity: warning
      annotations:
        summary: MySQL high threads running (instance {{ $labels.instance }})
        description: "More than 60% of MySQL connections are in running state on {{ $labels.instance }}\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
    
      # 1m delay allows a restart without triggering an alert.
    - alert: MySQLSlaveIOThreadNotRunning
      expr: ( mysql_slave_status_slave_io_running and ON (instance) mysql_slave_status_master_server_id > 0 ) == 0
      for: 1m
      labels:
        severity: critical
      annotations:
        summary: MySQL Slave IO thread not running (instance {{ $labels.instance }})
        description: "MySQL Slave IO thread not running on {{ $labels.instance }}\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
    
      # 1m delay allows a restart without triggering an alert.
    - alert: MySQLSlaveSQLThreadNotRunning
      expr: ( mysql_slave_status_slave_sql_running and ON (instance) mysql_slave_status_master_server_id > 0) == 0
      for: 1m
      labels:
        severity: critical
      annotations:
        summary: MySQL Slave SQL thread not running (instance {{ $labels.instance }})
        description: "MySQL Slave SQL thread not running on {{ $labels.instance }}\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
    
    - alert: MySQLSlaveReplicationLag
      expr: ( (mysql_slave_status_seconds_behind_master - mysql_slave_status_sql_delay) and ON (instance) mysql_slave_status_master_server_id > 0 ) > 30
      for: 1m
      labels:
        severity: critical
      annotations:
        summary: MySQL Slave replication lag (instance {{ $labels.instance }})
        description: "MySQL replication lag on {{ $labels.instance }}\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
    
      # mysqld_exporter exposes SHOW GLOBAL STATUS variables as untyped/gauge, so delta() is used instead of increase().
    - alert: MySQLSlowQueries
      expr: delta(mysql_global_status_slow_queries[1m]) > 0
      for: 2m
      labels:
        severity: warning
      annotations:
        summary: MySQL slow queries (instance {{ $labels.instance }})
        description: "MySQL server has some new slow queries ({{ $value }} in the last minute).\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
    
      # mysqld_exporter exposes SHOW GLOBAL STATUS variables as untyped/gauge, so deriv() is used instead of rate().
    - alert: MySQLInnoDBLogWaits
      expr: deriv(mysql_global_status_innodb_log_waits[15m]) > 10
      for: 0m
      labels:
        severity: warning
      annotations:
        summary: MySQL InnoDB log waits (instance {{ $labels.instance }})
        description: "MySQL innodb log writes stalling ({{ $value }} waits/s)\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
    
    - alert: MySQLRestarted
      expr: mysql_global_status_uptime < 60
      for: 0m
      labels:
        severity: info
      annotations:
        summary: MySQL restarted (instance {{ $labels.instance }})
        description: "MySQL has just been restarted, less than one minute ago on {{ $labels.instance }}.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
    
      # mysqld_exporter exposes SHOW GLOBAL STATUS variables as untyped/gauge, so deriv() is used instead of irate().
    - alert: MySQLHighQPS
      expr: deriv(mysql_global_status_questions[1m]) > 10000
      for: 2m
      labels:
        severity: info
      annotations:
        summary: MySQL High QPS (instance {{ $labels.instance }})
        description: "MySQL is being overload with unusual QPS (> 10k QPS).\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
    
    - alert: MySQLTooManyOpenFiles
      expr: mysql_global_status_innodb_num_open_files / mysql_global_variables_open_files_limit * 100 > 75 and mysql_global_variables_open_files_limit > 0
      for: 2m
      labels:
        severity: warning
      annotations:
        summary: MySQL too many open files (instance {{ $labels.instance }})
        description: "MySQL has too many open files, consider increase variables open_files_limit on {{ $labels.instance }}.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
    
    - alert: MySQLInnoDBForceRecoveryIsEnabled
      expr: mysql_global_variables_innodb_force_recovery != 0
      for: 2m
      labels:
        severity: warning
      annotations:
        summary: MySQL InnoDB Force Recovery is enabled (instance {{ $labels.instance }})
        description: "MySQL InnoDB force recovery is enabled on {{ $labels.instance }}\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
    
    - alert: MySQLInnoDBHistory_lenTooLong
      expr: mysql_info_schema_innodb_metrics_transaction_trx_rseg_history_len > 50000
      for: 2m
      labels:
        severity: warning
      annotations:
        summary: MySQL InnoDB history_len too long (instance {{ $labels.instance }})
        description: "MySQL history_len (undo log) too long on {{ $labels.instance }}\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

2.1. prometheus/mysqld_exporter (14 rules)

wget https://raw.githubusercontent.com/samber/awesome-prometheus-alerts/refs/heads/master/dist/rules/mysql/mysqld-exporter.yml

critical

2.1.1. MySQL down

MySQL instance is down on {{ $labels.instance }}

  # 1m delay allows a restart without triggering an alert.
- alert: MySQLDown
  expr: mysql_up == 0
  for: 1m
  labels:
    severity: critical
  annotations:
    summary: MySQL down (instance {{ $labels.instance }})
    description: "MySQL instance is down on {{ $labels.instance }}\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

warning

2.1.2. MySQL too many connections (> 80%)

More than 80% of MySQL connections are in use on {{ $labels.instance }}

- alert: MySQLTooManyConnections(>80%)
  expr: max_over_time(mysql_global_status_threads_connected[1m]) / mysql_global_variables_max_connections * 100 > 80 and mysql_global_variables_max_connections > 0
  for: 2m
  labels:
    severity: warning
  annotations:
    summary: MySQL too many connections (> 80%) (instance {{ $labels.instance }})
    description: "More than 80% of MySQL connections are in use on {{ $labels.instance }}\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

warning

2.1.3. MySQL high prepared statements utilization (> 80%)

High utilization of prepared statements (>80%) on {{ $labels.instance }}

- alert: MySQLHighPreparedStatementsUtilization(>80%)
  expr: max_over_time(mysql_global_status_prepared_stmt_count[1m]) / mysql_global_variables_max_prepared_stmt_count * 100 > 80 and mysql_global_variables_max_prepared_stmt_count > 0
  for: 2m
  labels:
    severity: warning
  annotations:
    summary: MySQL high prepared statements utilization (> 80%) (instance {{ $labels.instance }})
    description: "High utilization of prepared statements (>80%) on {{ $labels.instance }}\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

warning

2.1.4. MySQL high threads running

More than 60% of MySQL connections are in running state on {{ $labels.instance }}

- alert: MySQLHighThreadsRunning
  expr: max_over_time(mysql_global_status_threads_running[1m]) / mysql_global_variables_max_connections * 100 > 60 and mysql_global_variables_max_connections > 0
  for: 2m
  labels:
    severity: warning
  annotations:
    summary: MySQL high threads running (instance {{ $labels.instance }})
    description: "More than 60% of MySQL connections are in running state on {{ $labels.instance }}\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

critical

2.1.5. MySQL Slave IO thread not running

MySQL Slave IO thread not running on {{ $labels.instance }}

  # 1m delay allows a restart without triggering an alert.
- alert: MySQLSlaveIOThreadNotRunning
  expr: ( mysql_slave_status_slave_io_running and ON (instance) mysql_slave_status_master_server_id > 0 ) == 0
  for: 1m
  labels:
    severity: critical
  annotations:
    summary: MySQL Slave IO thread not running (instance {{ $labels.instance }})
    description: "MySQL Slave IO thread not running on {{ $labels.instance }}\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

critical

2.1.6. MySQL Slave SQL thread not running

MySQL Slave SQL thread not running on {{ $labels.instance }}

  # 1m delay allows a restart without triggering an alert.
- alert: MySQLSlaveSQLThreadNotRunning
  expr: ( mysql_slave_status_slave_sql_running and ON (instance) mysql_slave_status_master_server_id > 0) == 0
  for: 1m
  labels:
    severity: critical
  annotations:
    summary: MySQL Slave SQL thread not running (instance {{ $labels.instance }})
    description: "MySQL Slave SQL thread not running on {{ $labels.instance }}\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

critical

2.1.7. MySQL Slave replication lag

MySQL replication lag on {{ $labels.instance }}

- alert: MySQLSlaveReplicationLag
  expr: ( (mysql_slave_status_seconds_behind_master - mysql_slave_status_sql_delay) and ON (instance) mysql_slave_status_master_server_id > 0 ) > 30
  for: 1m
  labels:
    severity: critical
  annotations:
    summary: MySQL Slave replication lag (instance {{ $labels.instance }})
    description: "MySQL replication lag on {{ $labels.instance }}\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

warning

2.1.8. MySQL slow queries

MySQL server has some new slow queries ({{ $value }} in the last minute).

  # mysqld_exporter exposes SHOW GLOBAL STATUS variables as untyped/gauge, so delta() is used instead of increase().
- alert: MySQLSlowQueries
  expr: delta(mysql_global_status_slow_queries[1m]) > 0
  for: 2m
  labels:
    severity: warning
  annotations:
    summary: MySQL slow queries (instance {{ $labels.instance }})
    description: "MySQL server has some new slow queries ({{ $value }} in the last minute).\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

warning

2.1.9. MySQL InnoDB log waits

MySQL innodb log writes stalling ({{ $value }} waits/s)

  # mysqld_exporter exposes SHOW GLOBAL STATUS variables as untyped/gauge, so deriv() is used instead of rate().
- alert: MySQLInnoDBLogWaits
  expr: deriv(mysql_global_status_innodb_log_waits[15m]) > 10
  for: 0m
  labels:
    severity: warning
  annotations:
    summary: MySQL InnoDB log waits (instance {{ $labels.instance }})
    description: "MySQL innodb log writes stalling ({{ $value }} waits/s)\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

info

2.1.10. MySQL restarted

MySQL has just been restarted, less than one minute ago on {{ $labels.instance }}.

- alert: MySQLRestarted
  expr: mysql_global_status_uptime < 60
  for: 0m
  labels:
    severity: info
  annotations:
    summary: MySQL restarted (instance {{ $labels.instance }})
    description: "MySQL has just been restarted, less than one minute ago on {{ $labels.instance }}.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

info

2.1.11. MySQL High QPS

MySQL is being overload with unusual QPS (> 10k QPS).

  # mysqld_exporter exposes SHOW GLOBAL STATUS variables as untyped/gauge, so deriv() is used instead of irate().
- alert: MySQLHighQPS
  expr: deriv(mysql_global_status_questions[1m]) > 10000
  for: 2m
  labels:
    severity: info
  annotations:
    summary: MySQL High QPS (instance {{ $labels.instance }})
    description: "MySQL is being overload with unusual QPS (> 10k QPS).\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

warning

2.1.12. MySQL too many open files

MySQL has too many open files, consider increase variables open_files_limit on {{ $labels.instance }}.

- alert: MySQLTooManyOpenFiles
  expr: mysql_global_status_innodb_num_open_files / mysql_global_variables_open_files_limit * 100 > 75 and mysql_global_variables_open_files_limit > 0
  for: 2m
  labels:
    severity: warning
  annotations:
    summary: MySQL too many open files (instance {{ $labels.instance }})
    description: "MySQL has too many open files, consider increase variables open_files_limit on {{ $labels.instance }}.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

warning

2.1.13. MySQL InnoDB Force Recovery is enabled

MySQL InnoDB force recovery is enabled on {{ $labels.instance }}

- alert: MySQLInnoDBForceRecoveryIsEnabled
  expr: mysql_global_variables_innodb_force_recovery != 0
  for: 2m
  labels:
    severity: warning
  annotations:
    summary: MySQL InnoDB Force Recovery is enabled (instance {{ $labels.instance }})
    description: "MySQL InnoDB force recovery is enabled on {{ $labels.instance }}\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

warning

2.1.14. MySQL InnoDB history_len too long

MySQL history_len (undo log) too long on {{ $labels.instance }}

- alert: MySQLInnoDBHistory_lenTooLong
  expr: mysql_info_schema_innodb_metrics_transaction_trx_rseg_history_len > 50000
  for: 2m
  labels:
    severity: warning
  annotations:
    summary: MySQL InnoDB history_len too long (instance {{ $labels.instance }})
    description: "MySQL history_len (undo log) too long on {{ $labels.instance }}\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

More in Databases

PostgreSQL SQL Server Oracle Database Patroni PGBouncer Redis Memcached MongoDB Elasticsearch OpenSearch Meilisearch Cassandra Clickhouse CouchDB Solr