Skip to content

LiteLLM gateway alerts

Source rule: apps/ai/litellm/templates/prometheusrule.yaml (PrometheusRule litellm-gateway, ns ai, group litellm-gateway).

LiteLLM is the single OpenAI-compatible front door for every local model. When it is down, backends stay healthy but every /v1 consumer (paperclip fleet, apps) fails. Metrics arrive via the litellm ServiceMonitor (:4000/metrics, Bearer-authenticated).

All cluster changes go through Git → ArgoCD. Every command below is read-only; do not apply, patch, scale, delete, edit, or annotate anything while working these alerts.

LiteLLMGatewayDown

Severity: critical · component: litellm

Fires when: kube_deployment_status_replicas_available{namespace="ai", deployment="litellm"} == 0 holds continuously for: 5m — i.e. the Deployment has zero available replicas for five minutes.

Likely causes

  1. All pods in CrashLoopBackOff — commonly a failed Prisma migration on boot, or a missing/rotated master-key or database secret.
  2. OOMKilled or evicted pods across all replicas (node pressure, resources.limits.memory too low).
  3. Readiness probe failing, so replicas exist but none are counted available.
  4. Cilium network policy blocking pod startup dependencies (DB, Redis).

Diagnose (read-only)

kubectl -n ai get pods -l app.kubernetes.io/name=litellm -o wide
kubectl -n ai describe deploy litellm
kubectl -n ai logs -l app.kubernetes.io/name=litellm --tail=100
kubectl -n ai logs -l app.kubernetes.io/name=litellm --previous --tail=100   # last crash
kubectl -n ai get events --sort-by=.lastTimestamp | tail -30

PromQL — confirm which pods are non-Running (kube_pod_status_phase is a 0/1 gauge with a phase label, not a numeric phase code):

kube_pod_status_phase{namespace="ai", pod=~"litellm-.*"} == 1
kube_deployment_status_replicas_available{namespace="ai", deployment="litellm"}

Resolution: almost always a Git change. Identify the cause from logs, then fix it in apps/ai/litellm/ (secret references, resources limits, config/migration settings) and let ArgoCD reconcile. Do not hand-restart or scale pods — a manual restart that "fixes" it hides the real defect and drifts from Git. The one exception is an infrastructure fault outside this chart (e.g. the CNPG database being down), which is handled in that component's own runbook.

LiteLLMGatewayNoMetrics

Severity: critical · component: litellm

Fires when: absent(up{job="litellm", namespace="ai"} == 1) holds for: 5m — the alert fires when no litellm scrape target reports up == 1 for five minutes. Note this also fires if the target vanishes from Prometheus entirely (ServiceMonitor mismatch, no Service endpoints), not only when a scrape fails.

Likely causes

  1. Pods are not ready, so the Service has no endpoints and the target disappears.
  2. The prometheus callback is disabled in the LiteLLM config, so /metrics returns nothing useful or 404s.
  3. ServiceMonitor selector/port mismatch — it selects app.kubernetes.io/name: litellm on Service port name http (4000), path /metrics.
  4. Bearer credential failure: the ServiceMonitor authenticates with key LITELLM_MASTER_KEY from the master-key Secret. If that secret was rotated without a rollout, scrapes 401.
  5. Cilium network policy blocking Prometheus → pod :4000.

Diagnose (read-only)

kubectl -n ai get endpoints litellm
kubectl -n ai describe svc litellm
kubectl -n ai get servicemonitor litellm -o yaml
kubectl -n ai get pods -l app.kubernetes.io/name=litellm

Test /metrics from inside a pod (needs the Bearer token, already in the pod env — a bare curl returns 401):

kubectl -n ai exec deploy/litellm -- \
  sh -c 'curl -sS -H "Authorization: Bearer $LITELLM_MASTER_KEY" http://localhost:4000/metrics | head -20'

PromQL — check whether any target series exists at all:

up{job="litellm"}

Also check the Prometheus Targets page for ai/litellm and read the scrape error text.

Resolution: Git change in every normal case. Callback disabled → enable litellm_settings.callbacks: ["prometheus"] in templates/configmap.yaml. Selector/port/path drift → fix templates/servicemonitor.yaml. Network policy → fix the policy manifest. Rotated master key → reseal the secret first, then let the rollout follow (env secretKeyRef is snapshotted at pod create, so the pods must be recreated by ArgoCD after the sealed secret lands).

LiteLLMBackendModelHighErrorRate

Severity: warning · component: litellm

Fires when: for any single litellm_model_name, the failure share of gateway calls exceeds 0.25 (25%) continuously for: 10m:

sum by (litellm_model_name) (rate(litellm_deployment_failure_responses_total[10m]))
  /
(
  sum by (litellm_model_name) (rate(litellm_deployment_failure_responses_total[10m]))
    +
  sum by (litellm_model_name) (rate(litellm_deployment_success_responses_total[10m]))
) > 0.25

It is a ratio of rates, so a model with no traffic produces no series and cannot false-fire; a model with only failures reads 1.0. This is per-model, not gateway-wide — the gateway itself may be perfectly healthy.

Likely causes

  1. The backing service (llama-swap, OVMS router, Ollama, LM Studio on the workstation) is unhealthy, overloaded, or returning 5xx.
  2. llama-swap swap-in failure or timeout — the model could not be loaded into VRAM in time (tier thrash on the B60).
  3. The backend is simply off/scaled to zero (workstation-hosted models) while requests keep arriving.
  4. Network or DNS failure between the gateway and the backend pod.
  5. Consumers sending payloads the backend rejects consistently (bad model params, oversized context).

Diagnose (read-only)

# Which models are failing, and how badly
sum by (litellm_model_name) (rate(litellm_deployment_failure_responses_total[10m]))

# Is any traffic still succeeding (partial vs total failure)?
sum by (litellm_model_name) (rate(litellm_deployment_success_responses_total[10m]))

kubectl -n ai logs -l app.kubernetes.io/name=litellm --tail=200 | grep -i -E 'error|cooldown|fallback|timeout'
kubectl -n ai get pods -o wide          # locate the backend serving the failing model
kubectl -n ai logs <backend-pod> --tail=100
kubectl -n ai describe pod <backend-pod>
kubectl top pod -n ai

Map litellm_model_name → backend via the gateway config: apps/ai/litellm/templates/configmap.yaml (model_list).

Resolution: depends on the cause, but the durable fix is a Git change.

  • Backend misconfigured, under-resourced, or missing a fallback → fix model_list / litellm_settings.fallbacks / the backend's own resources in Git; ArgoCD reconciles.
  • Backend pod genuinely crashed → treat it under that component's runbook. Do not manually restart pods to clear the alert; find why it crashed first.
  • Transient swap-in contention that self-clears → no action beyond confirming recovery, but if it recurs, the fix is capacity/tier configuration in Git, not a live change.

Because severity is warning, this alert does not by itself justify an emergency intervention — confirm consumer impact (are fallbacks absorbing it?) before escalating.