Skip to content

OVMS B60 saturation alerts

Source rule: apps/ai/ovms/templates/prometheusrule.yaml — PrometheusRule/ovms-b60-saturation, namespace ai, group ovms-b60-saturation.

Metrics arrive via ServiceMonitor/ovms-predictors (apps/ai/ovms/templates/servicemonitor.yaml): port 8080, path /metrics, interval 30s, honorLabels: true. The exported metric set is fixed by --metrics_list in templates/servingruntime.yaml and templates/servingruntime-rerank.yaml.

Predictors scale to zero. Targets come and go by design. Absent series for a sleeping model are expected and are not an incident.

Metric-semantics warnings (read before acting)

Verified against the upstream OVMS metrics doc. Two of the three alerts rest on metrics whose upstream meaning differs from the rule's own annotation text. This matters when you are deciding whether an alert is real.

  • ovms_requests_rejected counts failures in the MediaPipe packet-creation phase (for example a malformed image). Upstream states explicitly that for the V3 API the LLM request is validated at graph-node level, so packet creation always succeeds — the qwen3 / gpt-oss chat endpoints serve V3, so this counter is expected to stay flat for LLM traffic regardless of load. The rule's annotation attributing it to max_tokens_limit / queue guardrails is not what the metric measures. Treat a firing OVMSRequestsRejected on an LLM model as unexplained and investigate before assuming saturation. The real LLM-graph error counter is ovms_graph_error, which this deployment does not currently scrape (see Gaps).
  • ovms_infer_req_queue_size is a gauge of the configured inference-request queue size (nireq) — not a live backlog depth. It is near-static per model. OVMSInferQueueBacklog therefore effectively tests "is this model's configured nireq greater than 8", so it tends to fire continuously or never, driven by configuration rather than by load. Do not read it as "N requests are waiting". The genuine queue-pressure signals are ovms_wait_for_infer_req_time_us (time a request waits for resources) and ovms_infer_req_active.
  • ovms_infer_req_queue_size, ovms_infer_req_active and ovms_streams are OpenVINO-backend model metrics. Confirm the series actually exists for the model in question before drawing conclusions from it.

OVMSRequestsRejected

Severity: warning · Labels: component: ovms-b60

Fires when: sum by (name) (rate(ovms_requests_rejected[5m])) > 0 holds for for: 5m. That is: any non-zero 5-minute rejection rate for any model name, sustained 5 minutes.

Likely causes

  1. Malformed or unsupported input at packet creation — bad image format on the vision detectors (nsfw, nudenet, rtmpose) is the canonical upstream case. This is the most probable cause given the metric's real semantics.
  2. A non-V3 (KServe/TFS) caller sending input the graph cannot accept.
  3. Only if the above are excluded: genuine guardrail rejection as the rule annotation assumes. Verify against pod logs — do not assume it.

Diagnose (read-only)

  1. Which model is rejecting, and how it compares to accepted traffic:
  2. sum by (name) (rate(ovms_requests_rejected[5m]))
  3. sum by (name) (rate(ovms_requests_accepted[5m]))
  4. List predictor pods (KServe labels predictor pods component=predictor; this is the same selector the ServiceMonitor uses): kubectl -n ai get pods -l component=predictor
  5. Read the rejection reason from the container log — this is the authoritative source, since the metric alone cannot tell you why: kubectl -n ai logs <predictor-pod> --tail=200
  6. Confirm the model is otherwise healthy: kubectl -n ai describe pod <predictor-pod>

Resolution

Nothing here is fixed live. If the cause is a bad client payload, fix the calling workload. If it is model or runtime configuration (max_num_seqs, token limits, model placement), that is a Git change to apps/ai/ovms/values.yaml or the relevant ServingRuntime / InferenceService template, merged and synced by ArgoCD. Do not patch the running ServingRuntime or ConfigMap.

OVMSRequestFailures

Severity: warning · Labels: component: ovms-b60

Fires when: sum by (name) (rate(ovms_requests_fail[5m])) > 0.05 holds for for: 5m. That is: more than 0.05 failed requests per second (roughly one failure every 20 seconds) for a model name, sustained 5 minutes.

This is the most trustworthy of the three alerts — ovms_requests_fail is a plain counter of failed requests to a model or DAG, and means what it says.

Likely causes

  1. GPU hang or driver error on the B60 host (xe engine reset, GuC timeout) — the 2026-07-05 CCS VRAM-overcommit hang class would surface here.
  2. VRAM exhaustion / OOM inside the inference engine under co-residency.
  3. Model load failure or a bad model artifact after a sync job.

Diagnose (read-only)

  1. Failure rate and share of total traffic per model:
  2. sum by (name) (rate(ovms_requests_fail[5m]))
  3. sum by (name) (rate(ovms_requests_fail[5m])) / clamp_min(sum by (name) (rate(ovms_requests_fail[5m]) + rate(ovms_requests_success[5m])), 1e-9)
  4. Container logs for stack traces / OOM / model-load errors: kubectl -n ai logs <predictor-pod> --tail=200 kubectl -n ai logs <predictor-pod> --previous --tail=200 (if it restarted)
  5. Restart count and last termination reason: kubectl -n ai get pods -l component=predictor -o wide kubectl -n ai describe pod <predictor-pod>
  6. Host kernel log for GPU resets. The rule annotation points at worker-home-2's host — check from the PVE host, not from inside the container (a container dmesg will not show host kmsg): qm guest exec <vmid> -- dmesg | grep -iE 'xe|guc|reset|coredump' (home PVE hosts: 192.168.1.10 / 192.168.1.11.)
  7. Node-level pressure: kubectl top nodes and kubectl top pods -n ai.

There is no GPU exporter deployed in this cluster — node_gpu_utilization and node_gpu_memory_used_bytes do not exist. VRAM residency must be read from the host (xpu-smi / intel_gpu_top on the PVE host), not from Prometheus.

Resolution

A transient GPU hang may clear when the pod is rescheduled, but do not delete or restart pods by hand — let the liveness probe or ArgoCD handle it. Anything durable (resource requests/limits, model co-residency, switching a model to the B580, changing the runtime image) is a Git change under apps/ai/ovms/, merged and synced by ArgoCD. If the host GPU itself is wedged, that is node maintenance, not a manifest change — follow the kured / worker-maintenance runbook.

OVMSInferQueueBacklog

Severity: info · Labels: component: ovms-b60

Fires when: max by (name) (ovms_infer_req_queue_size) > 8 holds for for: 10m. That is: the maximum configured inference-request queue size (nireq) for a model name exceeds 8, sustained 10 minutes.

Read the metric-semantics warning above first. This alert is info-severity and, because ovms_infer_req_queue_size reports configured nireq rather than live backlog, it is largely a statement about configuration. It is not by itself evidence that requests are queuing.

Likely causes

  1. The model is simply configured with nireq > 8 — the common, benign case.
  2. Genuine throughput saturation of the single B60, which you must confirm with the wait-time and active-request metrics below, not with this alert.

Diagnose (read-only)

  1. What the alert is actually reading (configured queue size per model): max by (name) (ovms_infer_req_queue_size)
  2. Whether requests are genuinely waiting — the real backlog signal, mean wait in microseconds: rate(ovms_wait_for_infer_req_time_us_sum[5m]) / rate(ovms_wait_for_infer_req_time_us_count[5m])
  3. Concurrency actually in flight:
  4. sum by (name) (ovms_infer_req_active)
  5. sum by (name) (ovms_current_requests)
  6. End-to-end mean request latency, microseconds: rate(ovms_request_time_us_sum[5m]) / rate(ovms_request_time_us_count[5m])
  7. Backend inference time, to separate queueing from compute: rate(ovms_inference_time_us_sum[5m]) / rate(ovms_inference_time_us_count[5m])

If wait time is flat and ovms_infer_req_active is well under the queue size, this alert is noise — record it and move on.

Resolution

If saturation is confirmed, tuning max_num_seqs / nireq / batch size, or moving a model to the B580, is a Git change to apps/ai/ovms/values.yaml or the ServingRuntime / InferenceService templates. Do not scale replicas live; on a single-GPU bottleneck extra replicas contend for the same card anyway. If the alert is confirmed noise, the durable fix is also a Git change — rewrite the expression against ovms_wait_for_infer_req_time_us or ovms_infer_req_active.

Gaps worth closing (each a Git change)

  • ovms_graph_error is not scraped. It is the real error counter for V3 LLM MediaPipe graphs and is the metric OVMSRequestsRejected was probably reaching for. Add it to --metrics_list in both ServingRuntime templates.
  • ovms_graph_request_latency_us / ovms_graph_processing_time_us are also unscraped and are the correct latency signals for the LLM graph path.
  • OVMSInferQueueBacklog targets a configuration gauge. Re-express against a genuine load signal.
  • No xe engine-reset / coredump alert exists. The source rule's header comment notes the kmsg alert is a separate Loki rule and still a follow-up; the 2026-07-05 hang was found only by hand-grepping kern.log.