Skip to content

llama-swap alert runbook

Source rule: apps/ai/llama-swap/templates/prometheusrule.yaml (PrometheusRule llama-swap, namespace ai, group llama-swap).

Rendered only when enabled and monitoring.enabled are true in apps/ai/llama-swap/values.yaml.

Context

llama-swap is Shape A: a single pod (replicas: 1, strategy: Recreate) that owns the B60 gpu.intel.com/xe share on node k8s-worker-home-2 and swaps OVMS children in-process. There is no redundancy by design — a surge pod would double-claim the xe share, so any pod-level outage is a full outage of llama-swap-only models. LiteLLM fallbacks absorb some aliases but not all.

The pinned v238 exporter exposes host CPU/memory/swap/load/network series only — no stable per-model load, eviction, queue, child-process, or swap-duration metrics. Do not expect per-model alerting here; backend request failures are covered at the LiteLLM layer.
Deployment / Service / PVC llama-swap / llama-swap / llama-swap-models
Service port → target 80 → container port 8080 (named http)
Node pin k8s-worker-home-2 (B60)
Storage longhorn-home-v2, RWO
ServiceMonitor port http, path /metrics, interval 30s
Init container hydrate-models — syncs IR from minio-home.media.svc.cluster.local:9000

Cold start is slow by design: a 30b/27b load is ~16 GB of IR plus an OpenVINO GPU compile, which is why healthCheckTimeout is 1800s. A pod that has been up for only a few minutes is not necessarily broken.

LlamaSwapDown

Severity: critical · Component: llama-swap

Fires when: kube_deployment_status_replicas_available{namespace="ai", deployment="llama-swap"} == 0 holds continuously for 5m.

Because replicas: 1, this metric is 0 whenever the single pod is not Ready — including during a legitimate Recreate rollout or a long cold model load.

Likely causes

  1. Init container hydrate-models failing or still running — MinIO unreachable, bad s3.secretName credentials, or a large first-time IR sync.
  2. Pod cannot schedule — node k8s-worker-home-2 down/cordoned, or gpu.intel.com/xe unavailable because the Intel device plugin is not advertising the resource.
  3. Image pull failure. Note the 2026-07-14 regression: a digest was folded into image.tag, producing tag@digest@digest → InvalidImageName → 2.5h outage. The digest belongs on image.digest, never in the tag.
  4. Longhorn RWO volume llama-swap-models stuck attached to a previous pod, blocking mount.
  5. Container OOMKilled, or cordana4 fatal lifecycle quarantine: an OVMS child did not reap after SIGKILL, /health returned 503, and liveness failed 3 consecutive 10-second probes before giving CRI a 25-second termination grace.

Diagnose (all read-only)

kubectl -n ai get pods -l app.kubernetes.io/name=llama-swap -o wide
kubectl -n ai describe deploy llama-swap
kubectl -n ai describe pod -l app.kubernetes.io/name=llama-swap   # events: scheduling, image, mount
kubectl -n ai logs deploy/llama-swap -c hydrate-models            # init: S3 hydration
kubectl -n ai logs deploy/llama-swap -c llama-swap --previous     # last crash
kubectl -n ai get pvc llama-swap-models
kubectl get node k8s-worker-home-2 -o wide
kubectl get node k8s-worker-home-2 -o jsonpath='{.status.allocatable}'   # confirm gpu.intel.com/xe

PromQL:

kube_deployment_status_replicas_available{namespace="ai", deployment="llama-swap"}
kube_pod_container_status_terminated_reason{namespace="ai", container="llama-swap"}

Resolution

  • First, rule out a normal slow start. If the init container is mid-sync or the pod was recreated after fatal quarantine and a model is compiling, the alert is expected to clear on its own. Ordinary model loading does not make the manager /health endpoint fail.
  • Any actual fix is a Git change. Image digest/tag, resources, node pin, storage class, model list, and probe timings all live in apps/ai/llama-swap/values.yaml and the chart templates. Edit them in a PR and let ArgoCD sync. This cluster is strictly GitOps: do not kubectl edit, patch, scale, rollout restart, or delete the pod to "kick" it.
  • Longhorn stale attachment is the one class of problem Git cannot express. Confirm it first from the Longhorn UI / volume status, then handle it as a deliberate storage operation — see the Longhorn notes in .workingdir2/; do not cordon or force-delete as a reflex.
  • If the node itself is the problem, treat it as a node incident (k8s-worker-home-2), not a llama-swap incident.

LlamaSwapNoMetrics

Severity: warning · Component: llama-swap

Fires when: absent(up{job="llama-swap", namespace="ai"} == 1) holds continuously for 10m.

absent() fires when the series does not exist at all — that is, either the target is missing from Prometheus entirely (discovery failure) or every scrape is failing (up == 0). A healthy scraped target yields up == 1 and the alert stays silent. This commonly fires alongside LlamaSwapDown, in which case the pod outage is the cause and this alert is downstream noise.

Likely causes

  1. Pod not running / not Ready, so the Service has no ready endpoints (check LlamaSwapDown first).
  2. ServiceMonitor not discovered by kube-prometheus-stack — the release: kube-prometheus-stack label is missing or the operator's serviceMonitorSelector does not match.
  3. Selector drift: the ServiceMonitor matches app.kubernetes.io/name: llama-swap on port name http. Renaming the Service port or labels silently breaks discovery.
  4. monitoring.enabled set false in values — the ServiceMonitor and this rule are both gated on it, so a target that vanished after a deploy is usually this.
  5. /metrics erroring or timing out while the process is saturated by a model load.

Diagnose (all read-only)

kubectl -n ai get servicemonitor llama-swap -o yaml
kubectl -n ai get svc llama-swap -o yaml
kubectl -n ai get endpointslices -l kubernetes.io/service-name=llama-swap
kubectl -n ai get pods -l app.kubernetes.io/name=llama-swap
kubectl -n ai logs deploy/llama-swap -c llama-swap --tail=200

PromQL — distinguish "target gone" from "scrape failing":

up{job="llama-swap", namespace="ai"}
scrape_samples_scraped{job="llama-swap", namespace="ai"}

An empty result for the first query means discovery failed (cause 2/3/4). A result of 0 means the target is discovered but scrapes are failing (cause 1/5).

Optional local read of the endpoint — a read-only tunnel, it changes no cluster state:

kubectl -n ai port-forward svc/llama-swap 8080:80
# then, in another shell: curl -s localhost:8080/metrics | head

Resolution

  • If LlamaSwapDown is also firing, resolve that first; this alert follows.
  • Discovery/labelling problems are Git changes. The ServiceMonitor labels, Service port name, and monitoring.enabled are all chart/values content under apps/ai/llama-swap/. Fix in a PR and let ArgoCD reconcile — do not patch the live ServiceMonitor.
  • If the target is discovered and scrapes fail only while a large model loads, this is expected transient behaviour on a cold start; confirm it clears once the pod is Ready before changing anything.