Skip to content

xe GPU alerts

Source rule: cluster/monitoring/xe-gpu-alerts.yaml (PrometheusRule xe-gpu-alerts, namespace monitoring, group xe-gpu-eviction).

Metrics are produced by the xe-textfile-exporter systemd timer (ansible/roles/xe-textfile-exporter, 30s interval) and surfaced through the node_exporter textfile collector on the Intel xe (Battlemage) nodes. All instance labels are therefore node_exporter targets (<host>:9100), not Kubernetes node names — map one to the other with kubectl get nodes -o wide before filtering pods.

Available series and their labels:

Metric Labels Meaning
xe_client_spilled_kib (none) Sum of drm-resident-gtt + drm-resident-system KiB across all renderD clients
xe_dmem_used_bytes region VRAM charge per region, from cgroup dmem.current
xe_dmem_capacity_bytes region VRAM capacity per region, from cgroup dmem.capacity
xe_dmem_pod_used_bytes pod_uid, region Per-pod VRAM charge (non-zero only)
xe_gt_throttle_status card, tile, gt 1 = GT frequency-throttled
xe_gt_throttle_reason card, tile, gt, reason Per-reason throttle flag
xe_gt_act_freq_mhz card, tile, gt Actual GT frequency in MHz

Staleness caveat. These are textfile-collector gauges. If the exporter timer stops, the series go stale and every alert below silently stops firing — absence of alerts is not evidence of health. Confirm freshness with node_textfile_mtime_seconds and node_textfile_scrape_error for the affected instance.

GitOps. This cluster is mutation-free outside Git. Every command in this runbook is read-only. All fixes land as a PR that ArgoCD reconciles — never kubectl apply/patch/scale/delete/cordon/drain a live object to resolve one of these alerts.

XeVramEvictCliff

Severity: critical

Fires when: xe_client_spilled_kib > 262144 sustained for: 3m — i.e. total GTT + system-RAM residency across all renderD clients exceeds 256 MiB for three minutes. Healthy baseline is roughly 40 MiB.

Why this metric exists: when VRAM fills, the xe driver evicts buffers to GTT/system RAM and the dmem cgroup uncharges on evict. No VRAM-usage gauge ever shows the cliff, so this fdinfo residency sum is the only direct signal. The throughput penalty is roughly 16x.

Likely causes

  1. A model, context length, or batch size on this node exceeds the VRAM that is actually free.
  2. Co-resident workloads that individually fit but collectively do not (a second model tier got scheduled or scaled up from zero).
  3. A GPU-bound process leaking resident buffers over time.

Diagnose (read-only)

  • Confirm the spill is persistent, not a scrape artifact: xe_client_spilled_kib{instance="<instance>"}[30m]
  • Attribute it to a pod — this is the metric that actually answers "who": topk(5, xe_dmem_pod_used_bytes{instance="<instance>"})
  • Resolve the pod_uid from that result to a real pod: kubectl get pods -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,UID:.metadata.uid | grep <pod_uid>
  • Check headroom at the same timestamp: xe_dmem_used_bytes / xe_dmem_capacity_bytes
  • Inspect the identified workload for load-time OOM or fallback-to-host messages: kubectl logs -n <namespace> <pod> --tail=200

Resolution — requires a Git change. Reduce the GPU-side footprint of the identified workload in the repo: smaller quantization, lower context length, smaller batch, or fewer concurrently warm model tiers. Commit the change and let ArgoCD reconcile. Deleting or restarting the pod live clears the symptom without changing the sizing, and the alert returns on the next load. Consult .workingdir2/gpu-unified-backend-design.md before changing which tiers are co-resident — the B60 cannot hold more than two warm tiers.

XeVramNearFull

Severity: warning

Fires when: xe_dmem_used_bytes / xe_dmem_capacity_bytes > 0.93 sustained for: 5m. The division matches on the region label, so this evaluates per VRAM region.

Interpretation: dmem never exceeds capacity because it uncharges on evict, which makes >93% the last warning before XeVramEvictCliff. Treat this as the actionable one — by the time the cliff fires, throughput is already gone.

Likely causes

  1. Gradual growth in resident buffers from a long-running inference workload.
  2. A newly scheduled or scaled-from-zero GPU workload consuming the remaining headroom.
  3. Allocator fragmentation leaving usable-but-unallocatable VRAM.

Diagnose (read-only)

  • Current utilisation per region: xe_dmem_used_bytes / xe_dmem_capacity_bytes
  • Per-pod breakdown of the charge: sort_desc(xe_dmem_pod_used_bytes{instance="<instance>"})
  • Map pod_uid values to pods: kubectl get pods -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,UID:.metadata.uid | grep <pod_uid>
  • Check whether spill has already begun: xe_client_spilled_kib{instance="<instance>"}
  • List GPU-requesting pods on the node (translate <instance> to a node name first): kubectl get pods -A -o wide --field-selector spec.nodeName=<node>

Resolution — requires a Git change. Adjust the workload footprint or the set of co-resident tiers in the repo, exactly as for XeVramEvictCliff. If the fix is to stop scheduling further GPU work here, express that through the workload's own manifests in Git; do not cordon or drain the node by hand.

XeGtSustainedThrottle

Severity: warning

Fires when: xe_gt_throttle_status == 1 sustained for: 10m — a GT has been frequency-throttled continuously for ten minutes. Labels identify the specific card/tile/gt.

Likely causes

  1. Thermal ceiling — inadequate airflow or high ambient temperature at the site.
  2. Power/current limit under sustained load.
  3. PROCHOT asserted by another component in the chassis.

Diagnose (read-only)

  • Identify why it is throttling — this distinguishes all three causes above: xe_gt_throttle_reason{instance="<instance>"} == 1
  • Measure the actual cost in clock terms: xe_gt_act_freq_mhz{instance="<instance>"}
  • Correlate with load: xe_dmem_used_bytes{instance="<instance>"} over the same window.
  • Confirm the exporter is still fresh before trusting a flat line: node_textfile_mtime_seconds{instance="<instance>"}

Resolution — depends on the reason label

  • Thermal or PROCHOT: physical. Check chassis airflow, fan health, and ambient temperature at the affected site. Host-level inspection goes through Proxmox (qm guest exec on the PVE host owning the VM), not a separate management interface.
  • Power limit: if the card is being held at a configured power ceiling, that ceiling lives in the node's Ansible/host configuration — changing it is a Git change, applied via the repo, not an ad-hoc sysfs write that a reboot discards.
  • A short throttle under genuine peak load is the hardware protecting itself and is not a defect. Escalate only when it is sustained across an otherwise normal workload, or when xe_gt_act_freq_mhz shows a materially degraded clock.