Skip to content

PromQL cookbook

Copy-paste PromQL for this cluster. Every metric name below was verified either against live Prometheus (/api/v1/label/__name__/values) or against a PrometheusRule expr tracked in this repo. Queries whose metric or label selector could not be confirmed were removed rather than guessed.

Most entries are the bare selector form of a shipped alert rule — the alert adds a threshold and a for: duration. The source rule is linked per section so the threshold stays in one authoritative place.

Read-only by definition: querying Prometheus mutates nothing. All cluster changes still go through Git → ArgoCD.

Where to run these

Prometheus lives in ns monitoring as service kube-prometheus-stack-prometheus:9090. Read-only query via the API server proxy, no port-forward needed:

kubectl get --raw \
  '/api/v1/namespaces/monitoring/services/kube-prometheus-stack-prometheus:9090/proxy/api/v1/query?query=up'

List every metric name the cluster currently exposes:

kubectl get --raw \
  '/api/v1/namespaces/monitoring/services/kube-prometheus-stack-prometheus:9090/proxy/api/v1/label/__name__/values'

Model serving — OVMS

Source rule: apps/ai/ovms/templates/prometheusrule.yaml.

OVMS predictors scale to zero, so these return empty while no model is warm. That is expected and is not evidence the metric is wrong — the exported metric set is pinned by --metrics_list in apps/ai/ovms/templates/servingruntime.yaml.

Question Query
Request rejection rate by model sum by (name) (rate(ovms_requests_rejected[5m])) — BLOCKED: no ovms_* series exist; the ovms-predictors ServiceMonitor targets are relabel-dropped (BUG-52).
Request failure rate by model sum by (name) (rate(ovms_requests_fail[5m])) — BLOCKED: no ovms_* series exist; the ovms-predictors ServiceMonitor targets are relabel-dropped (BUG-52).
Max inference queue size by model max by (name) (ovms_infer_req_queue_size) — BLOCKED: no ovms_* series exist; the ovms-predictors ServiceMonitor targets are relabel-dropped (BUG-52).

ovms_requests_rejected counts MediaPipe packet-creation failures, not HTTP 4xx. ovms_requests_fail is the trustworthy failure signal — see docs/runbooks/alerts/ovms.md.

Model serving — LiteLLM and llama-swap

All inference traffic goes through the LiteLLM proxy (ADR-0030); these two deployments are the front door.

Question Query
LiteLLM available replicas kube_deployment_status_replicas_available{namespace="ai", deployment="litellm"}
LiteLLM scrape target up up{job="litellm", namespace="ai"}
llama-swap available replicas kube_deployment_status_replicas_available{namespace="ai", deployment="llama-swap"}
llama-swap scrape target up up{job="llama-swap", namespace="ai"}
Available replicas for every ns ai deployment kube_deployment_status_replicas_available{namespace="ai"}

Scrape health

Question Query
All scrape targets and their state up
Targets currently down up == 0
Scrape pools that blew their target limit prometheus_target_scrape_pool_exceeded_target_limit_total
Grafana target up up{job=~".*grafana.*"}

The Grafana job is kube-prometheus-stack-grafana; the regex above is what the shipped rules use.

Tetragon

Source rule: cluster/monitoring/tetragon-alerts.yaml. Runbook: docs/runbooks/alerts/tetragon.md.

Question Query
Ring-buffer events lost rate(tetragon_observer_ringbuf_events_lost_total[5m])
BPF missed events rate(tetragon_missed_prog_probes_total[5m])
Missed events broken down by error kind sum by (error) (rate(tetragon_missed_prog_probes_total[5m]))
TracingPolicies in an error state tetragon_tracingpolicy_loaded{state=~"error\|load_error"}
Agent DaemonSet targets up up{service="tetragon"}
Operator metrics target up up{service="tetragon-operator-metrics"}
Seconds since last event export time() - tetragon_events_last_exported_timestamp
Process cache eviction rate rate(tetragon_process_cache_evictions_total[5m])

tetragon_tracingpolicy_loaded carries a state label with values enabled, disabled, error and load_error. A non-zero series for the latter two is the condition worth acting on.

Cilium and Hubble

Source rule: cluster/monitoring/cilium-alerts.yaml.

Question Query
Cilium agents up (operator excluded) up{job=~".*cilium.*", job!~".*operator.*"}
BPF map pressure, all maps cilium_bpf_map_pressure
Conntrack map pressure cilium_bpf_map_pressure{map_name=~"ct.*"}
Failed endpoint regenerations rate(cilium_endpoint_regenerations_total{outcome="fail"}[10m])
Unreachable nodes max(cilium_unreachable_nodes)
Operator entirely absent absent(up{job=~".*cilium.*operator.*"} == 1)
Operator restarts in the last hour increase(kube_pod_container_status_restarts_total{namespace="kube-system", pod=~"cilium-operator.*"}[1h])
Hubble relay target up up{job=~".*hubble-relay.*"}
Total Hubble drop rate sum(rate(hubble_drop_total[10m]))

Two label facts worth knowing before you edit these:

  • Conntrack maps are named ct4_global and ct_any4_global. A selector of map_name=~"cilium_ct.*" matches nothing — the cilium_ prefix belongs to the metric name, not the map name.
  • Cilium and Hubble scrape targets live in ns kube-system, with jobs cilium-agent, cilium-operator, hubble-metrics and hubble-relay-metrics.

Note the brace placement in the absent(...) query: the == 1 comparison sits outside the label selector. absent(up{...} == 1) is valid; moving == 1 inside the braces is a parse error.

Velero

Question Query
Backup failures, last 24h increase(velero_backup_failure_total[24h])
Partial backup failures, last 24h increase(velero_backup_partial_failure_total[24h])
Backup validation failures, last 6h increase(velero_backup_validation_failure_total[6h])

These series carry a schedule label, so sum by (schedule) (...) splits the result per backup schedule.

Loki log alerts are LogQL, not PromQL

cluster/monitoring/loki-log-alerts.yaml contains rules such as absent_over_time({flow_log="hubble"}[10m]). Those are LogQL, evaluated by the Loki ruler against log streams — they are deliberately excluded from this cookbook. Pasting them into Prometheus does not error; it silently returns a meaningless result, because Prometheus has no flow_log label. Query them through Loki instead (/loki-query).