Alert thresholds depend on the nature of your applications. Some queries may have arbitrary tolerance thresholds. Building an efficient monitoring platform takes time. 😉
6 Prometheus alerting rules for Graph Node. Exported via Embedded exporter. These rules cover critical and warning conditions — copy and paste the YAML into your Prometheus configuration.
groups:
- name: EmbeddedExporter
rules:
- alert: ProviderFailedBecauseNet_versionFailed
expr: eth_rpc_status == 1
for: 0m
labels:
severity: critical
annotations:
summary: Provider failed because net_version failed (instance {{ $labels.instance }})
description: "Failed net_version for Provider `{{$labels.provider}}` in Graph node `{{$labels.instance}}`\n VALUE = {{ $value }}\n LABELS = {{ $labels }}"
- alert: ProviderFailedBecauseGetGenesisFailed
expr: eth_rpc_status == 2
for: 0m
labels:
severity: critical
annotations:
summary: Provider failed because get genesis failed (instance {{ $labels.instance }})
description: "Failed to get genesis for Provider `{{$labels.provider}}` in Graph node `{{$labels.instance}}`\n VALUE = {{ $value }}\n LABELS = {{ $labels }}"
- alert: ProviderFailedBecauseNet_versionTimeout
expr: eth_rpc_status == 3
for: 0m
labels:
severity: critical
annotations:
summary: Provider failed because net_version timeout (instance {{ $labels.instance }})
description: "net_version timeout for Provider `{{$labels.provider}}` in Graph node `{{$labels.instance}}`\n VALUE = {{ $value }}\n LABELS = {{ $labels }}"
- alert: ProviderFailedBecauseGetGenesisTimeout
expr: eth_rpc_status == 4
for: 0m
labels:
severity: critical
annotations:
summary: Provider failed because get genesis timeout (instance {{ $labels.instance }})
description: "Timeout to get genesis for Provider `{{$labels.provider}}` in Graph node `{{$labels.instance}}`\n VALUE = {{ $value }}\n LABELS = {{ $labels }}"
# Threshold of 10ms. Adjust based on your expected database latency.
- alert: StoreConnectionSlow
expr: store_connection_wait_time_ms > 10
for: 0m
labels:
severity: warning
annotations:
summary: Store connection slow (instance {{ $labels.instance }})
description: "Store connection is too slow to `{{$labels.pool}}` pool, `{{$labels.shard}}` shard in Graph node `{{$labels.instance}}`\n VALUE = {{ $value }}\n LABELS = {{ $labels }}"
# Threshold of 20ms. Adjust based on your expected database latency.
- alert: StoreConnectionVerySlow
expr: store_connection_wait_time_ms > 20
for: 0m
labels:
severity: critical
annotations:
summary: Store connection very slow (instance {{ $labels.instance }})
description: "Store connection is very slow to `{{$labels.pool}}` pool, `{{$labels.shard}}` shard in Graph node `{{$labels.instance}}`\n VALUE = {{ $value }}\n LABELS = {{ $labels }}" wget https://raw.githubusercontent.com/samber/awesome-prometheus-alerts/refs/heads/master/dist/rules/graph-node/embedded-exporter.yml
Failed net_version for Provider `{{$labels.provider}}` in Graph node `{{$labels.instance}}`
- alert: ProviderFailedBecauseNet_versionFailed
expr: eth_rpc_status == 1
for: 0m
labels:
severity: critical
annotations:
summary: Provider failed because net_version failed (instance {{ $labels.instance }})
description: "Failed net_version for Provider `{{$labels.provider}}` in Graph node `{{$labels.instance}}`\n VALUE = {{ $value }}\n LABELS = {{ $labels }}" Failed to get genesis for Provider `{{$labels.provider}}` in Graph node `{{$labels.instance}}`
- alert: ProviderFailedBecauseGetGenesisFailed
expr: eth_rpc_status == 2
for: 0m
labels:
severity: critical
annotations:
summary: Provider failed because get genesis failed (instance {{ $labels.instance }})
description: "Failed to get genesis for Provider `{{$labels.provider}}` in Graph node `{{$labels.instance}}`\n VALUE = {{ $value }}\n LABELS = {{ $labels }}" net_version timeout for Provider `{{$labels.provider}}` in Graph node `{{$labels.instance}}`
- alert: ProviderFailedBecauseNet_versionTimeout
expr: eth_rpc_status == 3
for: 0m
labels:
severity: critical
annotations:
summary: Provider failed because net_version timeout (instance {{ $labels.instance }})
description: "net_version timeout for Provider `{{$labels.provider}}` in Graph node `{{$labels.instance}}`\n VALUE = {{ $value }}\n LABELS = {{ $labels }}" Timeout to get genesis for Provider `{{$labels.provider}}` in Graph node `{{$labels.instance}}`
- alert: ProviderFailedBecauseGetGenesisTimeout
expr: eth_rpc_status == 4
for: 0m
labels:
severity: critical
annotations:
summary: Provider failed because get genesis timeout (instance {{ $labels.instance }})
description: "Timeout to get genesis for Provider `{{$labels.provider}}` in Graph node `{{$labels.instance}}`\n VALUE = {{ $value }}\n LABELS = {{ $labels }}" Store connection is too slow to `{{$labels.pool}}` pool, `{{$labels.shard}}` shard in Graph node `{{$labels.instance}}`
# Threshold of 10ms. Adjust based on your expected database latency.
- alert: StoreConnectionSlow
expr: store_connection_wait_time_ms > 10
for: 0m
labels:
severity: warning
annotations:
summary: Store connection slow (instance {{ $labels.instance }})
description: "Store connection is too slow to `{{$labels.pool}}` pool, `{{$labels.shard}}` shard in Graph node `{{$labels.instance}}`\n VALUE = {{ $value }}\n LABELS = {{ $labels }}" Store connection is very slow to `{{$labels.pool}}` pool, `{{$labels.shard}}` shard in Graph node `{{$labels.instance}}`
# Threshold of 20ms. Adjust based on your expected database latency.
- alert: StoreConnectionVerySlow
expr: store_connection_wait_time_ms > 20
for: 0m
labels:
severity: critical
annotations:
summary: Store connection very slow (instance {{ $labels.instance }})
description: "Store connection is very slow to `{{$labels.pool}}` pool, `{{$labels.shard}}` shard in Graph node `{{$labels.instance}}`\n VALUE = {{ $value }}\n LABELS = {{ $labels }}"