What is the Prometheus alert rule for "Provider failed because net_version failed"?

Failed net_version for Provider `{{$labels.provider}}` in Graph node `{{$labels.instance}}` PromQL expression: eth_rpc_status == 1. Severity: critical.

What is the Prometheus alert rule for "Provider failed because get genesis failed"?

Failed to get genesis for Provider `{{$labels.provider}}` in Graph node `{{$labels.instance}}` PromQL expression: eth_rpc_status == 2. Severity: critical.

What is the Prometheus alert rule for "Provider failed because net_version timeout"?

net_version timeout for Provider `{{$labels.provider}}` in Graph node `{{$labels.instance}}` PromQL expression: eth_rpc_status == 3. Severity: critical.

What is the Prometheus alert rule for "Provider failed because get genesis timeout"?

Timeout to get genesis for Provider `{{$labels.provider}}` in Graph node `{{$labels.instance}}` PromQL expression: eth_rpc_status == 4. Severity: critical.

What is the Prometheus alert rule for "Store connection very slow"?

Store connection is very slow to `{{$labels.pool}}` pool, `{{$labels.shard}}` shard in Graph node `{{$labels.instance}}` PromQL expression: store_connection_wait_time_ms > 20. Severity: critical.

What is the Prometheus alert rule for "Store connection slow"?

Store connection is too slow to `{{$labels.pool}}` pool, `{{$labels.shard}}` shard in Graph node `{{$labels.instance}}` PromQL expression: store_connection_wait_time_ms > 10. Severity: warning.

What is the Prometheus alert rule for "Subgraph indexing lag behind chain head"?

Subgraph deployment `{{$labels.deployment}}` is more than 100 blocks behind the `{{$labels.network}}` chain head in Graph node `{{$labels.instance}}`, indicating indexing is stalled or too slow to keep up with new blocks. PromQL expression: ethereum_chain_head_number - ignoring(deployment,shard) group_right(deployment,shard) deployment_head > 100. Severity: warning. Duration: 10m.

Graph Node Prometheus Alert Rules

8 Prometheus alerting rules for Graph Node.Exported via Embedded exporter.These rules cover critical and warning conditions — copy and paste the YAML into your Prometheus configuration.

⚠️

Alert thresholds depend on the nature of your applications. Some queries may have arbitrary tolerance thresholds. Building an efficient monitoring platform takes time. 😉

groups:
- name: EmbeddedExporter
  rules:
    - alert: ProviderFailedBecauseNet_versionFailed
      expr: eth_rpc_status == 1
      for: 0m
      labels:
        severity: critical
      annotations:
        summary: Provider failed because net_version failed (instance {{ $labels.instance }})
        description: "Failed net_version for Provider `{{$labels.provider}}` in Graph node `{{$labels.instance}}`\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
    
    - alert: ProviderFailedBecauseGetGenesisFailed
      expr: eth_rpc_status == 2
      for: 0m
      labels:
        severity: critical
      annotations:
        summary: Provider failed because get genesis failed (instance {{ $labels.instance }})
        description: "Failed to get genesis for Provider `{{$labels.provider}}` in Graph node `{{$labels.instance}}`\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
    
    - alert: ProviderFailedBecauseNet_versionTimeout
      expr: eth_rpc_status == 3
      for: 0m
      labels:
        severity: critical
      annotations:
        summary: Provider failed because net_version timeout (instance {{ $labels.instance }})
        description: "net_version timeout for Provider `{{$labels.provider}}` in Graph node `{{$labels.instance}}`\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
    
    - alert: ProviderFailedBecauseGetGenesisTimeout
      expr: eth_rpc_status == 4
      for: 0m
      labels:
        severity: critical
      annotations:
        summary: Provider failed because get genesis timeout (instance {{ $labels.instance }})
        description: "Timeout to get genesis for Provider `{{$labels.provider}}` in Graph node `{{$labels.instance}}`\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
    
      # Threshold of 20ms. Adjust based on your expected database latency.
    - alert: StoreConnectionVerySlow
      expr: store_connection_wait_time_ms > 20
      for: 0m
      labels:
        severity: critical
      annotations:
        summary: Store connection very slow (instance {{ $labels.instance }})
        description: "Store connection is very slow to `{{$labels.pool}}` pool, `{{$labels.shard}}` shard in Graph node `{{$labels.instance}}`\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
    
      # Requires graph-node >= 0.37.0. The deployment_status gauge (1=starting, 2=running,
      # 3=stopped, 4=failed) replaced the older deployment_failed metric, which was removed.
    - alert: SubgraphDeploymentFailed
      expr: deployment_status == 4
      for: 0m
      labels:
        severity: critical
      annotations:
        summary: Subgraph deployment failed (instance {{ $labels.instance }})
        description: "Subgraph deployment `{{$labels.deployment}}` has failed and indexing has stopped in Graph node `{{$labels.instance}}`; check the graph-node logs for the deployment to find the root cause.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
    
      # Threshold of 10ms. Adjust based on your expected database latency.
    - alert: StoreConnectionSlow
      expr: store_connection_wait_time_ms > 10
      for: 0m
      labels:
        severity: warning
      annotations:
        summary: Store connection slow (instance {{ $labels.instance }})
        description: "Store connection is too slow to `{{$labels.pool}}` pool, `{{$labels.shard}}` shard in Graph node `{{$labels.instance}}`\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
    
      # 100 blocks is a rough default; real-world lag varies enormously by chain (block time ranges from ~2s to ~10min across networks) — adjust based on the specific network's block time.
    - alert: SubgraphIndexingLagBehindChainHead
      expr: ethereum_chain_head_number - ignoring(deployment,shard) group_right(deployment,shard) deployment_head > 100
      for: 10m
      labels:
        severity: warning
      annotations:
        summary: Subgraph indexing lag behind chain head (instance {{ $labels.instance }})
        description: "Subgraph deployment `{{$labels.deployment}}` is more than 100 blocks behind the `{{$labels.network}}` chain head in Graph node `{{$labels.instance}}`, indicating indexing is stalled or too slow to keep up with new blocks.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

13.2.Embedded exporter(8 rules)

wget https://raw.githubusercontent.com/samber/awesome-prometheus-alerts/refs/heads/master/dist/rules/graph-node/embedded-exporter.yml

critical

13.2.1.Provider failed because net_version failed

Failed net_version for Provider `{{$labels.provider}}` in Graph node `{{$labels.instance}}`

- alert: ProviderFailedBecauseNet_versionFailed
  expr: eth_rpc_status == 1
  for: 0m
  labels:
    severity: critical
  annotations:
    summary: Provider failed because net_version failed (instance {{ $labels.instance }})
    description: "Failed net_version for Provider `{{$labels.provider}}` in Graph node `{{$labels.instance}}`\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

critical

13.2.2.Provider failed because get genesis failed

Failed to get genesis for Provider `{{$labels.provider}}` in Graph node `{{$labels.instance}}`

- alert: ProviderFailedBecauseGetGenesisFailed
  expr: eth_rpc_status == 2
  for: 0m
  labels:
    severity: critical
  annotations:
    summary: Provider failed because get genesis failed (instance {{ $labels.instance }})
    description: "Failed to get genesis for Provider `{{$labels.provider}}` in Graph node `{{$labels.instance}}`\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

critical

13.2.3.Provider failed because net_version timeout

net_version timeout for Provider `{{$labels.provider}}` in Graph node `{{$labels.instance}}`

- alert: ProviderFailedBecauseNet_versionTimeout
  expr: eth_rpc_status == 3
  for: 0m
  labels:
    severity: critical
  annotations:
    summary: Provider failed because net_version timeout (instance {{ $labels.instance }})
    description: "net_version timeout for Provider `{{$labels.provider}}` in Graph node `{{$labels.instance}}`\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

critical

13.2.4.Provider failed because get genesis timeout

Timeout to get genesis for Provider `{{$labels.provider}}` in Graph node `{{$labels.instance}}`

- alert: ProviderFailedBecauseGetGenesisTimeout
  expr: eth_rpc_status == 4
  for: 0m
  labels:
    severity: critical
  annotations:
    summary: Provider failed because get genesis timeout (instance {{ $labels.instance }})
    description: "Timeout to get genesis for Provider `{{$labels.provider}}` in Graph node `{{$labels.instance}}`\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

critical

13.2.5.Store connection very slow

Store connection is very slow to `{{$labels.pool}}` pool, `{{$labels.shard}}` shard in Graph node `{{$labels.instance}}`

  # Threshold of 20ms. Adjust based on your expected database latency.
- alert: StoreConnectionVerySlow
  expr: store_connection_wait_time_ms > 20
  for: 0m
  labels:
    severity: critical
  annotations:
    summary: Store connection very slow (instance {{ $labels.instance }})
    description: "Store connection is very slow to `{{$labels.pool}}` pool, `{{$labels.shard}}` shard in Graph node `{{$labels.instance}}`\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

critical

13.2.6.Subgraph deployment failed

Subgraph deployment `{{$labels.deployment}}` has failed and indexing has stopped in Graph node `{{$labels.instance}}`; check the graph-node logs for the deployment to find the root cause.

  # Requires graph-node >= 0.37.0. The deployment_status gauge (1=starting, 2=running,
  # 3=stopped, 4=failed) replaced the older deployment_failed metric, which was removed.
- alert: SubgraphDeploymentFailed
  expr: deployment_status == 4
  for: 0m
  labels:
    severity: critical
  annotations:
    summary: Subgraph deployment failed (instance {{ $labels.instance }})
    description: "Subgraph deployment `{{$labels.deployment}}` has failed and indexing has stopped in Graph node `{{$labels.instance}}`; check the graph-node logs for the deployment to find the root cause.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

warning

13.2.7.Store connection slow

Store connection is too slow to `{{$labels.pool}}` pool, `{{$labels.shard}}` shard in Graph node `{{$labels.instance}}`

  # Threshold of 10ms. Adjust based on your expected database latency.
- alert: StoreConnectionSlow
  expr: store_connection_wait_time_ms > 10
  for: 0m
  labels:
    severity: warning
  annotations:
    summary: Store connection slow (instance {{ $labels.instance }})
    description: "Store connection is too slow to `{{$labels.pool}}` pool, `{{$labels.shard}}` shard in Graph node `{{$labels.instance}}`\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

warning

13.2.8.Subgraph indexing lag behind chain head

Subgraph deployment `{{$labels.deployment}}` is more than 100 blocks behind the `{{$labels.network}}` chain head in Graph node `{{$labels.instance}}`, indicating indexing is stalled or too slow to keep up with new blocks.

  # 100 blocks is a rough default; real-world lag varies enormously by chain (block time ranges from ~2s to ~10min across networks) — adjust based on the specific network's block time.
- alert: SubgraphIndexingLagBehindChainHead
  expr: ethereum_chain_head_number - ignoring(deployment,shard) group_right(deployment,shard) deployment_head > 100
  for: 10m
  labels:
    severity: warning
  annotations:
    summary: Subgraph indexing lag behind chain head (instance {{ $labels.instance }})
    description: "Subgraph deployment `{{$labels.deployment}}` is more than 100 blocks behind the `{{$labels.network}}` chain head in Graph node `{{$labels.instance}}`, indicating indexing is stalled or too slow to keep up with new blocks.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

More in Other

APC UPS LiteLLM