What is the Prometheus alert rule for "OpenSearch host CPU usage high"?

System CPU usage on OpenSearch node {{ $labels.node }} (cluster {{ $labels.cluster }}) is above 90%, which may cause search and indexing latency. PromQL expression: sum by (cluster, instance, node) (opensearch_os_cpu_percent) > 90. Severity: warning. Duration: 5m.

What is the Prometheus alert rule for "OpenSearch process CPU usage high"?

CPU usage of the OpenSearch process on node {{ $labels.node }} (cluster {{ $labels.cluster }}) is above 90%, indicating heavy load on the JVM process. PromQL expression: sum by (cluster, instance, node) (opensearch_process_cpu_percent) > 90. Severity: warning. Duration: 5m.

What is the Prometheus alert rule for "OpenSearch high heap usage"?

OpenSearch heap usage on cluster {{ $labels.cluster }} is too high PromQL expression: opensearch_jvm_mem_heap_used_percent > 75. Severity: warning. Duration: 10m.

What is the Prometheus alert rule for "OpenSearch disk high watermark reached"?

Disk usage on OpenSearch node {{ $labels.node }} (cluster {{ $labels.cluster }}) is above the high watermark (90%): OpenSearch will start relocating existing shards away from this node. PromQL expression: sum by (cluster, instance, node) (round((1 - (opensearch_fs_path_available_bytes / opensearch_fs_path_total_bytes)) * 100, 0.001)) > 90. Severity: critical. Duration: 5m.

What is the Prometheus alert rule for "OpenSearch disk low watermark reached"?

Disk usage on OpenSearch node {{ $labels.node }} (cluster {{ $labels.cluster }}) is above the low watermark (85%): OpenSearch will stop allocating new shards to this node. PromQL expression: sum by (cluster, instance, node) (round((1 - (opensearch_fs_path_available_bytes / opensearch_fs_path_total_bytes)) * 100, 0.001)) > 85. Severity: warning. Duration: 5m.

What is the Prometheus alert rule for "OpenSearch is red"?

OpenSearch cluster {{ $labels.cluster }} is RED: some primary shards are unassigned PromQL expression: opensearch_cluster_status == 2. Severity: critical. Duration: 2m.

What is the Prometheus alert rule for "OpenSearch is yellow"?

OpenSearch cluster {{ $labels.cluster }} is YELLOW: some replica shards are unassigned PromQL expression: opensearch_cluster_status == 1. Severity: warning. Duration: 5m.

What is the Prometheus alert rule for "OpenSearch circuitbreaker tripped"?

The circuitbreaker on OpenSearch cluster {{ $labels.cluster }} has tripped to prevent Java OutOfMemoryError PromQL expression: increase(opensearch_circuitbreaker_tripped_count[5m]) > 0. Severity: warning. Duration: 5m.

What is the Prometheus alert rule for "OpenSearch has pending tasks"?

OpenSearch cluster {{ $labels.cluster }} has pending tasks PromQL expression: opensearch_cluster_pending_tasks_number > 0. Severity: warning. Duration: 5m.

What is the Prometheus alert rule for "OpenSearch indexing is throttled"?

Indexing on OpenSearch cluster {{ $labels.cluster }} is throttled PromQL expression: opensearch_indices_indexing_is_throttled_bool > 0. Severity: warning. Duration: 5m.

What is the Prometheus alert rule for "OpenSearch has inactive shards"?

OpenSearch cluster {{ $labels.cluster }} has inactive shards PromQL expression: opensearch_cluster_shards_active_percent < 100.0. Severity: warning. Duration: 5m.

What is the Prometheus alert rule for "OpenSearch high bulk rejection rate"?

The ratio of rejected to completed bulk indexing requests on OpenSearch node {{ $labels.node }} is {{ $value }}%, meaning the node cannot keep up with bulk indexing throughput. PromQL expression: round((rate(opensearch_threadpool_threads_count{name="bulk",type="rejected"}[5m]) / rate(opensearch_threadpool_threads_count{name="bulk",type="completed"}[5m])) * 100, 0.001) > 5. Severity: warning. Duration: 5m.

OpenSearch Prometheus Alert Rules

12 Prometheus alerting rules for OpenSearch.Exported via opensearch-project/opensearch-prometheus-exporter.These rules cover critical and warning conditions — copy and paste the YAML into your Prometheus configuration.

⚠️

Alert thresholds depend on the nature of your applications. Some queries may have arbitrary tolerance thresholds. Building an efficient monitoring platform takes time. 😉

groups:
- name: OpensearchProjectOpensearchPrometheusExporter
  rules:
    - alert: OpenSearchHostCPUUsageHigh
      expr: sum by (cluster, instance, node) (opensearch_os_cpu_percent) > 90
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: OpenSearch host CPU usage high (instance {{ $labels.instance }})
        description: "System CPU usage on OpenSearch node {{ $labels.node }} (cluster {{ $labels.cluster }}) is above 90%, which may cause search and indexing latency.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
    
    - alert: OpenSearchProcessCPUUsageHigh
      expr: sum by (cluster, instance, node) (opensearch_process_cpu_percent) > 90
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: OpenSearch process CPU usage high (instance {{ $labels.instance }})
        description: "CPU usage of the OpenSearch process on node {{ $labels.node }} (cluster {{ $labels.cluster }}) is above 90%, indicating heavy load on the JVM process.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
    
    - alert: OpenSearchHighHeapUsage
      expr: opensearch_jvm_mem_heap_used_percent > 75
      for: 10m
      labels:
        severity: warning
      annotations:
        summary: OpenSearch high heap usage (instance {{ $labels.instance }})
        description: "OpenSearch heap usage on cluster {{ $labels.cluster }} is too high\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
    
    - alert: OpenSearchDiskHighWatermarkReached
      expr: sum by (cluster, instance, node) (round((1 - (opensearch_fs_path_available_bytes / opensearch_fs_path_total_bytes)) * 100, 0.001)) > 90
      for: 5m
      labels:
        severity: critical
      annotations:
        summary: OpenSearch disk high watermark reached (instance {{ $labels.instance }})
        description: "Disk usage on OpenSearch node {{ $labels.node }} (cluster {{ $labels.cluster }}) is above the high watermark (90%): OpenSearch will start relocating existing shards away from this node.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
    
    - alert: OpenSearchDiskLowWatermarkReached
      expr: sum by (cluster, instance, node) (round((1 - (opensearch_fs_path_available_bytes / opensearch_fs_path_total_bytes)) * 100, 0.001)) > 85
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: OpenSearch disk low watermark reached (instance {{ $labels.instance }})
        description: "Disk usage on OpenSearch node {{ $labels.node }} (cluster {{ $labels.cluster }}) is above the low watermark (85%): OpenSearch will stop allocating new shards to this node.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
    
    - alert: OpenSearchIsRed
      expr: opensearch_cluster_status == 2
      for: 2m
      labels:
        severity: critical
      annotations:
        summary: OpenSearch is red (instance {{ $labels.instance }})
        description: "OpenSearch cluster {{ $labels.cluster }} is RED: some primary shards are unassigned\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
    
    - alert: OpenSearchIsYellow
      expr: opensearch_cluster_status == 1
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: OpenSearch is yellow (instance {{ $labels.instance }})
        description: "OpenSearch cluster {{ $labels.cluster }} is YELLOW: some replica shards are unassigned\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
    
    - alert: OpenSearchCircuitbreakerTripped
      expr: increase(opensearch_circuitbreaker_tripped_count[5m]) > 0
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: OpenSearch circuitbreaker tripped (instance {{ $labels.instance }})
        description: "The circuitbreaker on OpenSearch cluster {{ $labels.cluster }} has tripped to prevent Java OutOfMemoryError\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
    
    - alert: OpenSearchHasPendingTasks
      expr: opensearch_cluster_pending_tasks_number > 0
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: OpenSearch has pending tasks (instance {{ $labels.instance }})
        description: "OpenSearch cluster {{ $labels.cluster }} has pending tasks\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
    
    - alert: OpenSearchIndexingIsThrottled
      expr: opensearch_indices_indexing_is_throttled_bool > 0
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: OpenSearch indexing is throttled (instance {{ $labels.instance }})
        description: "Indexing on OpenSearch cluster {{ $labels.cluster }} is throttled\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
    
    - alert: OpenSearchHasInactiveShards
      expr: opensearch_cluster_shards_active_percent < 100.0
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: OpenSearch has inactive shards (instance {{ $labels.instance }})
        description: "OpenSearch cluster {{ $labels.cluster }} has inactive shards\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
    
      # 5% rejection ratio is a rough default; depends on your indexing throughput relative to the configured bulk thread pool/queue size — adjust based on your workload.
    - alert: OpenSearchHighBulkRejectionRate
      expr: round((rate(opensearch_threadpool_threads_count{name="bulk",type="rejected"}[5m]) / rate(opensearch_threadpool_threads_count{name="bulk",type="completed"}[5m])) * 100, 0.001) > 5
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: OpenSearch high bulk rejection rate (instance {{ $labels.instance }})
        description: "The ratio of rejected to completed bulk indexing requests on OpenSearch node {{ $labels.node }} is {{ $value }}%, meaning the node cannot keep up with bulk indexing throughput.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

2.11.opensearch-project/opensearch-prometheus-exporter(12 rules)

wget https://raw.githubusercontent.com/samber/awesome-prometheus-alerts/refs/heads/master/dist/rules/opensearch/opensearch-project-opensearch-prometheus-exporter.yml

warning

2.11.1.OpenSearch host CPU usage high

System CPU usage on OpenSearch node {{ $labels.node }} (cluster {{ $labels.cluster }}) is above 90%, which may cause search and indexing latency.

- alert: OpenSearchHostCPUUsageHigh
  expr: sum by (cluster, instance, node) (opensearch_os_cpu_percent) > 90
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: OpenSearch host CPU usage high (instance {{ $labels.instance }})
    description: "System CPU usage on OpenSearch node {{ $labels.node }} (cluster {{ $labels.cluster }}) is above 90%, which may cause search and indexing latency.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

warning

2.11.2.OpenSearch process CPU usage high

CPU usage of the OpenSearch process on node {{ $labels.node }} (cluster {{ $labels.cluster }}) is above 90%, indicating heavy load on the JVM process.

- alert: OpenSearchProcessCPUUsageHigh
  expr: sum by (cluster, instance, node) (opensearch_process_cpu_percent) > 90
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: OpenSearch process CPU usage high (instance {{ $labels.instance }})
    description: "CPU usage of the OpenSearch process on node {{ $labels.node }} (cluster {{ $labels.cluster }}) is above 90%, indicating heavy load on the JVM process.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

warning

2.11.3.OpenSearch high heap usage

OpenSearch heap usage on cluster {{ $labels.cluster }} is too high

- alert: OpenSearchHighHeapUsage
  expr: opensearch_jvm_mem_heap_used_percent > 75
  for: 10m
  labels:
    severity: warning
  annotations:
    summary: OpenSearch high heap usage (instance {{ $labels.instance }})
    description: "OpenSearch heap usage on cluster {{ $labels.cluster }} is too high\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

critical

2.11.4.OpenSearch disk high watermark reached

Disk usage on OpenSearch node {{ $labels.node }} (cluster {{ $labels.cluster }}) is above the high watermark (90%): OpenSearch will start relocating existing shards away from this node.

- alert: OpenSearchDiskHighWatermarkReached
  expr: sum by (cluster, instance, node) (round((1 - (opensearch_fs_path_available_bytes / opensearch_fs_path_total_bytes)) * 100, 0.001)) > 90
  for: 5m
  labels:
    severity: critical
  annotations:
    summary: OpenSearch disk high watermark reached (instance {{ $labels.instance }})
    description: "Disk usage on OpenSearch node {{ $labels.node }} (cluster {{ $labels.cluster }}) is above the high watermark (90%): OpenSearch will start relocating existing shards away from this node.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

warning

2.11.5.OpenSearch disk low watermark reached

Disk usage on OpenSearch node {{ $labels.node }} (cluster {{ $labels.cluster }}) is above the low watermark (85%): OpenSearch will stop allocating new shards to this node.

- alert: OpenSearchDiskLowWatermarkReached
  expr: sum by (cluster, instance, node) (round((1 - (opensearch_fs_path_available_bytes / opensearch_fs_path_total_bytes)) * 100, 0.001)) > 85
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: OpenSearch disk low watermark reached (instance {{ $labels.instance }})
    description: "Disk usage on OpenSearch node {{ $labels.node }} (cluster {{ $labels.cluster }}) is above the low watermark (85%): OpenSearch will stop allocating new shards to this node.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

critical

2.11.6.OpenSearch is red

OpenSearch cluster {{ $labels.cluster }} is RED: some primary shards are unassigned

- alert: OpenSearchIsRed
  expr: opensearch_cluster_status == 2
  for: 2m
  labels:
    severity: critical
  annotations:
    summary: OpenSearch is red (instance {{ $labels.instance }})
    description: "OpenSearch cluster {{ $labels.cluster }} is RED: some primary shards are unassigned\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

warning

2.11.7.OpenSearch is yellow

OpenSearch cluster {{ $labels.cluster }} is YELLOW: some replica shards are unassigned

- alert: OpenSearchIsYellow
  expr: opensearch_cluster_status == 1
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: OpenSearch is yellow (instance {{ $labels.instance }})
    description: "OpenSearch cluster {{ $labels.cluster }} is YELLOW: some replica shards are unassigned\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

warning

2.11.8.OpenSearch circuitbreaker tripped

The circuitbreaker on OpenSearch cluster {{ $labels.cluster }} has tripped to prevent Java OutOfMemoryError

- alert: OpenSearchCircuitbreakerTripped
  expr: increase(opensearch_circuitbreaker_tripped_count[5m]) > 0
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: OpenSearch circuitbreaker tripped (instance {{ $labels.instance }})
    description: "The circuitbreaker on OpenSearch cluster {{ $labels.cluster }} has tripped to prevent Java OutOfMemoryError\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

warning

2.11.9.OpenSearch has pending tasks

OpenSearch cluster {{ $labels.cluster }} has pending tasks

- alert: OpenSearchHasPendingTasks
  expr: opensearch_cluster_pending_tasks_number > 0
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: OpenSearch has pending tasks (instance {{ $labels.instance }})
    description: "OpenSearch cluster {{ $labels.cluster }} has pending tasks\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

warning

2.11.10.OpenSearch indexing is throttled

Indexing on OpenSearch cluster {{ $labels.cluster }} is throttled

- alert: OpenSearchIndexingIsThrottled
  expr: opensearch_indices_indexing_is_throttled_bool > 0
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: OpenSearch indexing is throttled (instance {{ $labels.instance }})
    description: "Indexing on OpenSearch cluster {{ $labels.cluster }} is throttled\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

warning

2.11.11.OpenSearch has inactive shards

OpenSearch cluster {{ $labels.cluster }} has inactive shards

- alert: OpenSearchHasInactiveShards
  expr: opensearch_cluster_shards_active_percent < 100.0
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: OpenSearch has inactive shards (instance {{ $labels.instance }})
    description: "OpenSearch cluster {{ $labels.cluster }} has inactive shards\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

warning

2.11.12.OpenSearch high bulk rejection rate

The ratio of rejected to completed bulk indexing requests on OpenSearch node {{ $labels.node }} is {{ $value }}%, meaning the node cannot keep up with bulk indexing throughput.

  # 5% rejection ratio is a rough default; depends on your indexing throughput relative to the configured bulk thread pool/queue size — adjust based on your workload.
- alert: OpenSearchHighBulkRejectionRate
  expr: round((rate(opensearch_threadpool_threads_count{name="bulk",type="rejected"}[5m]) / rate(opensearch_threadpool_threads_count{name="bulk",type="completed"}[5m])) * 100, 0.001) > 5
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: OpenSearch high bulk rejection rate (instance {{ $labels.instance }})
    description: "The ratio of rejected to completed bulk indexing requests on OpenSearch node {{ $labels.node }} is {{ $value }}%, meaning the node cannot keep up with bulk indexing throughput.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

More in Databases

MySQL PostgreSQL SQL Server Oracle Database Patroni PGBouncer Redis Memcached MongoDB Elasticsearch Meilisearch Cassandra Clickhouse CouchDB Solr