xe-dmem enforcement alerts¶

Source rule: apps/infra/xe-dmem-nri/templates/prometheusrule.yaml (PrometheusRule xe-dmem-enforcement, namespace monitoring, group xe-dmem-enforcement; rules live in files/enforcement-rules.tpl).

These cover the per-container VRAM ceiling injected by apps/infra/xe-dmem-nri, a containerd NRI plugin that writes cgroup-v2 dmem.max into annotated Intel Arc (xe) GPU containers.

The thing to internalise before reading further: dmem.max does not fail an allocation at the cap. The xe driver evicts to system RAM instead — roughly a 16× throughput collapse — and the dmem cgroup uncharges on evict, so dmem.current never exceeds capacity and no VRAM gauge shows the cliff. "The cap is obeyed" and "the workload just died" are simultaneously true. That is why enforcement and thrashing are separate alerts.

Metric	Source	Meaning
`xe_dmem_nri_ready`	plugin `/metrics` (DaemonSet, ns `kube-system`)	plugin registered on the NRI socket
`xe_dmem_nri_adjustments_total{result,reason}`	plugin	every `CreateContainer` decision
`xe_dmem_pod_max_bytes`	host `xe-textfile-exporter`	the cap read back off the container cgroup
`xe_dmem_pod_peak_bytes`	host exporter	high-water charge (`dmem.peak`)
`xe_dmem_pod_used_bytes`	host exporter	current charge (`dmem.current`)
`xe_client_spilled_kib`	host exporter	fdinfo GTT+system residency — the only direct evict signal

GitOps. Every command here is read-only. Fixes land as a PR that ArgoCD reconciles. Never kubectl apply/patch/annotate/delete a live object to resolve one of these. Rollback of a cap is deleting the pod annotation in Git and letting the workload recreate.

Never auto-restart on these alerts. A restart clears the symptom and destroys the evidence of which cap was wrong.

XeDmemNriNotRegistered¶

Severity: warning

Fires when: xe_dmem_nri_ready == 0 for 10m.

The plugin process is up and scrapeable but not attached to containerd's NRI socket. This is not an outage — containers are created normally while the plugin is away. It is a silent loss of enforcement: new GPU containers come up uncapped, with no error anywhere.

Checks¶

kubectl -n kube-system logs ds/xe-dmem-nri --tail=50
kubectl -n kube-system get pod -l app.kubernetes.io/name=xe-dmem-nri -o wide

On the node (read-only):

ctr plugins ls | grep nri        # expect io.containerd.nri.v1.nri  ok
stat /var/run/nri/nri.sock
containerd config dump | grep -A8 "io.containerd.nri.v1.nri"

Likely causes¶

containerd restarted. The plugin reconnects every 5s on its own; if this cleared itself, no action.
NRI disabled in /etc/containerd/config.toml. The containerd role pins and asserts disable = false (ansible/roles/containerd/tasks/main.yml) — a drift here means the role has not run since the config was regenerated.
The socket path moved. Inventory pin is containerd_nri_socket_path in ansible/inventory/group_vars/all.yml; the chart's nri.socketPath must match.

Already-running capped containers keep their caps. Only newly created ones are affected.

XeDmemCapNotApplied¶

Severity: warning

Fires when: increase(xe_dmem_nri_adjustments_total{result!="applied",reason!~"no_annotation|container_not_selected"}[15m]) > 0 for 5m.

Fail-open fired. An annotated GPU container was created unmodified. This is the designed behaviour — the plugin never blocks container creation — but the workload declares a cap it does not have.

Read $labels.reason:

reason	meaning	fix
`malformed_annotation`	JSON invalid, wrong `version`, unknown field, or `maxBytes` below 256 MiB	fix the annotation; the contract is in the chart README
`no_gpu_device`	annotated container has no DRM render node	it never got a `gpu.intel.com/xe` device — check the request and the device plugin
`multiple_gpu_devices`	more than one render node in the container	v1 caps exactly one card per container
`device_resolution_failed`	sysfs/cgroup path unreadable, or the card is not on the `xe` driver	check the plugin's hostPath mounts and `cat /sys/fs/cgroup/dmem.capacity`
`exceeds_capacity`	declared `maxBytes` > the region's `dmem.capacity`	lower the annotation; the plugin refuses rather than clamping, deliberately
`panic`	recovered panic in the handler	escalate — capture logs, this is a plugin bug, not a config error

kubectl -n kube-system logs ds/xe-dmem-nri --tail=100 | grep -v '"level":"INFO"'

XeDmemPodNearCap¶

Severity: warning

Fires when: xe_dmem_pod_used_bytes / on(instance, pod_uid, region) xe_dmem_pod_max_bytes > 0.93 for 5m.

The workload is within 7% of its own cap. The next resident buffer evicts to system RAM and throughput falls off the cliff — this is the last warning before XeDmemThrashing.

Because xe_dmem_pod_max_bytes is read back from the container cgroup rather than from the annotation, this firing also proves the cap is really in place.

Response¶

Compare xe_dmem_pod_peak_bytes against xe_dmem_pod_max_bytes for that pod_uid — has it always run this close, or is this new?
If the cap is genuinely too tight, raise maxBytes in the pod annotation via PR. Headroom below the card's dmem.capacity is the constraint.
If the workload grew unexpectedly, shrink the working set (model size, context length, concurrent streams) instead.

Do not raise the cap above dmem.capacity — the plugin refuses it and the pod silently runs uncapped.

XeDmemThrashing¶

Severity: critical

Fires when: the retained combined detector holds for 2m:

xe_client_spilled_kib > 262144
and on(instance)
(max by (instance) (xe_dmem_used_bytes / on(instance, region) xe_dmem_capacity_bytes) > 0.93)

Spill above 256 MiB (healthy baseline ~40 MiB) and VRAM above 93%. GPU buffers are resident in system RAM. Expect roughly 16× slower inference/transcode with no errors logged anywhere.

This can fire while every cap is being obeyed perfectly. A correct cap that is too small produces exactly this.

Triage¶

xe_dmem_pod_max_bytes                     # who is capped, and at what
xe_dmem_pod_peak_bytes                    # who actually hit their ceiling
xe_dmem_pod_used_bytes                    # current charge per pod
xe_client_spilled_kib                     # magnitude of the spill
sum by (region) (xe_dmem_used_bytes)      # is the whole card full, or one tenant

One capped pod at its ceiling, card not full → its cap is too tight. Raise maxBytes by PR, or move the workload.
Card full, several tenants → this is co-tenancy overcommit, not a cap bug. The B580 (12 GB, jellyfin + qwen3-8b-b580-predictor) is the known-tight case. Either lower one tenant's footprint or move it off the card.
Spill high, no pod near its cap → an uncapped tenant is the cause. Caps contain their own pod; they cannot stop a neighbour's eviction pressure from degrading everyone on the card.

Escalation¶

Include in the page: pod_uid, namespace, pod, container, region, declared cap, current, peak, spill, and the rollback PR (annotation deletion).

Stop immediately and escalate if you also see an xe engine reset or CAT error in dmesg — B60-class reset risk is tracked separately as BUG-39, and it is not caused by dmem.

What this alert does not mean¶

dmem constrains charged device residency. It does not arbitrate GPU execution time, memory bandwidth, engine scheduling, or cross-tenant fault isolation, and it does not replace llama-swap's model-residency logic — see the chart README, "llama-swap interaction".