Skip to content

Cilium alert runbook

Source rule file: cluster/monitoring/cilium-alerts.yaml (PrometheusRule cilium-alerts, namespace monitoring).

Covers all 10 alerts in that rule across groups cilium-agent, cilium-operator, cilium-gateway, cilium-hubble. Cilium is pinned at chart 1.19.6 in argocd-apps/infra/cilium.yaml.

All commands below are read-only. This cluster is strictly GitOps: every configuration fix lands as a commit to argocd-apps/infra/cilium.yaml (or the relevant manifest) and reaches the cluster via ArgoCD. Do not kubectl apply/patch/delete/scale/edit to resolve any of these alerts.

The in-agent CLI is cilium-dbg (renamed upstream; cilium is the separate client-side CLI).

CiliumAgentDown

Severity: critical Fires when: up{job=~".*cilium.*", job!~".*operator.*"} == 0 for 5m — an agent scrape target is down. The operator job is excluded by the second matcher.

Likely causes

  1. Cilium DaemonSet pod CrashLooping or Evicted on the node.
  2. Node itself NotReady or unreachable, so the target stops being scraped.
  3. Agent alive but its metrics port is not serving.

Diagnose

  • kubectl get pods -n kube-system -l k8s-app=cilium -o wide — which node's agent is affected.
  • kubectl get nodes — distinguish "agent broken" from "node down".
  • kubectl describe node <node> — conditions and recent events.
  • kubectl logs -n kube-system <cilium-pod> --previous — if it is restarting.
  • PromQL: up{job=~".*cilium.*", job!~".*operator.*"} == 0 — confirm the exact instance.

Resolution: Usually not a Git change. If the node is down, recover the node (see PVE/qm guest exec path for unreachable nodes); the agent returns with it. Datapath and policy enforcement on that node are unmanaged until it does. Only if the agent is failing due to configuration (bad values, image, resources) is this a Git change to argocd-apps/infra/cilium.yaml.

CiliumBPFMapPressureHigh

Severity: warning Fires when: cilium_bpf_map_pressure > 0.85 for 15m — any BPF map on any agent is over 85% full.

Likely causes

  1. Connection/endpoint/policy growth exceeding the pre-allocated map size.
  2. Map entry leak from policy churn.
  3. Map sized for a lighter workload than currently scheduled.

bpf.preallocateMaps: true is set in argocd-apps/infra/cilium.yaml (chart default is false), so map size is a hard ceiling — at pressure 1.0 entries are dropped rather than the map growing.

Diagnose

  • PromQL: topk(5, cilium_bpf_map_pressure) — which map and pod.
  • PromQL: cilium_bpf_map_pressure > 0.85 — full affected set.
  • kubectl exec -n kube-system <cilium-pod> -- cilium-dbg bpf map list — live map sizes and usage.
  • kubectl top pods -n kube-system -l k8s-app=cilium — correlate memory growth.

Resolution: Git change. Raise the relevant map ceiling under the bpf: block in argocd-apps/infra/cilium.yaml (ctTcpMax, ctAnyMax, natMax, policyMapMax, or mapDynamicSizeRatio), then let ArgoCD sync. Note that raising pre-allocated maps raises per-node memory; check node headroom before committing.

CiliumConntrackMapNearFull

Severity: warning Fires when: cilium_bpf_map_pressure{map_name=~"cilium_ct.*"} > 0.9 for 10m — same metric as above, restricted to conntrack maps, at a higher threshold.

Likely causes

  1. Connection-tracking table exhaustion from high connection churn or many long-lived flows.
  2. Traffic spike on one node.
  3. GC not reclaiming entries fast enough relative to churn.

Diagnose

  • PromQL: cilium_bpf_map_pressure{map_name=~"cilium_ct.*"} — which CT map, which pod.
  • PromQL: sum(rate(cilium_drop_count_total[5m])) by (reason) — whether drops have started, and why.
  • PromQL: cilium_datapath_conntrack_gc_entries — GC entries alive vs deleted per run.
  • kubectl exec -n kube-system <cilium-pod> -- cilium-dbg bpf ct list global — inspect live CT entries.

Resolution: Git change for capacity: raise bpf.ctTcpMax / bpf.ctAnyMax in argocd-apps/infra/cilium.yaml. If instead a single workload is generating the churn, the fix is that workload's manifest (connection pooling / replica spread), also via Git. This alert precedes real connection drops — treat 0.9 as the warning shoulder, not the failure.

CiliumEndpointRegenerationFailing

Severity: warning Fires when: rate(cilium_endpoint_regenerations_total{outcome="fail"}[10m]) > 0 for 15m — any sustained failure rate over a 10m rate window.

Likely causes

  1. A NetworkPolicy / CiliumNetworkPolicy that fails to compile into the datapath.
  2. Resource pressure on the node causing regeneration timeouts.
  3. Kernel/datapath incompatibility (this cluster runs bpf.datapathMode: netkit, which needs kernel 6.8+).

Diagnose

  • PromQL: rate(cilium_endpoint_regenerations_total{outcome="fail"}[10m]) > 0 — which pods.
  • kubectl logs -n kube-system <cilium-pod> | grep -i regenerat — the actual failure text.
  • kubectl get cnp,ccnp -A — recently changed policies.
  • kubectl exec -n kube-system <cilium-pod> -- cilium-dbg endpoint list — endpoints not in ready state.

Resolution: Git change in almost all cases — correct the offending policy manifest, or adjust agent resources in argocd-apps/infra/cilium.yaml. Important: while regeneration is failing, network policy on that node is stale, so enforcement may not match what is committed in Git. Treat it as a security-relevant drift, not just a noisy warning.

CiliumUnreachableNodes

Severity: warning Fires when: max(cilium_unreachable_nodes) > 0 for 10m — the health probe reports at least one node it cannot reach. Note this is a cluster-wide max(), so it does not identify the reporter by itself.

Likely causes

  1. Cross-site link degradation between home and office (BUG-14 class).
  2. A node down or rebooting.
  3. Agent health subsystem degraded on the reporting node.

Diagnose

  • kubectl get nodes -o wide — node status across both sites.
  • PromQL: cilium_unreachable_nodes > 0 — without the max(), to find which agents are reporting.
  • kubectl exec -n kube-system <cilium-pod> -- cilium-dbg status --all-health — per-node health from that agent's view.
  • kubectl exec -n kube-system <cilium-pod> -- cilium-dbg-health status — health-probe detail if available.

Resolution: Usually not a Git change. If a node is down, recover the node. If it is cross-site, the path is the UniFi gateway tunnel between sites — investigate the link itself. Do not go looking for firewall ACLs to "fix" this: VLANs are open gateway-side except wlan-iot / wlan-guest, so a blocked-ACL theory is almost always the wrong lead. Only health-probe configuration changes are a Git change.

CiliumOperatorDown

Severity: critical Fires when: absent(up{job=~".*cilium.*operator.*"} == 1) for 10m — no operator replica is reporting up. Because this uses absent(), it fires only when every replica is gone; one surviving replica keeps it silent.

Likely causes

  1. All operator replicas CrashLooping or Evicted.
  2. Replicas Pending — the operator is pinned to the office zone via operator.nodeSelector, so office-side scheduling pressure can strand it.
  3. Operator-level bug (BUG-03 territory: GW-API CRD-discovery silent disable, fixed in 1.19.4+).

Diagnose

  • kubectl get pods -n kube-system -l io.cilium/app=operator -o wide — status and node.
  • kubectl describe pods -n kube-system -l io.cilium/app=operator — OOMKilled, ImagePullBackOff, unschedulable.
  • kubectl logs -n kube-system <operator-pod> --previous — crash cause.
  • PromQL: up{job=~".*cilium.*operator.*"} — confirm no replica is up.

Resolution: With the operator down, Gateway API translation, LB-IPAM, and identity GC are stalled — existing traffic keeps flowing, but new Gateways and LB IPs will not be provisioned. If pods are Pending for capacity, recover office-side capacity. Anything config-shaped (resources, nodeSelector, version rollback) is a Git change to argocd-apps/infra/cilium.yaml.

CiliumOperatorRestartingRepeatedly

Severity: warning Fires when: increase(kube_pod_container_status_restarts_total{namespace="kube-system", pod=~"cilium-operator.*"}[1h]) > 3 for 5m — more than 3 restarts in the trailing hour, sustained 5m.

Likely causes

  1. Leader-election churn (2026-05-31 class). The operator is pinned to office so its lease renewals stay local; if that pinning is lost, cross-site latency can time out renewals.
  2. OOMKills or aggressive liveness probes.
  3. Startup bug loop — repeated restarts risk the BUG-03 GW-API CRD-discovery stall.

Diagnose

  • PromQL: increase(kube_pod_container_status_restarts_total{namespace="kube-system", pod=~"cilium-operator.*"}[1h]) — confirm the count.
  • kubectl get events -n kube-system --sort-by=.lastTimestamp | grep cilium-operator — recent events. (Do not use --field-selector involvedObject.name=~...; field selectors do not support regex.)
  • kubectl logs -n kube-system <operator-pod> --previous — panic or lease-renewal errors.
  • kubectl describe pod -n kube-system <operator-pod> — last state, exit code, OOM flag.

Resolution: Git change for probe timeouts, resource limits, or operator.nodeSelector in argocd-apps/infra/cilium.yaml. Verify the office-zone pinning is still in place before tuning anything else — a lost nodeSelector is the cheapest explanation for lease churn.

CiliumGatewayEnvoyNotReady

Severity: warning Fires when: up{namespace="networking", endpoint="http"} == 0 for 5m.

Read the expression carefully: this matches any scrape target in the networking namespace with endpoint="http" — it is not restricted to Envoy or to Gateway pods by any label. The alert is named for Envoy because that is the intended target, but during an incident confirm which target actually went down before assuming it is a gateway.

Likely causes

  1. Gateway Envoy pod CrashLooping or failing its readiness scrape.
  2. A different networking-namespace http target down (see caveat above).
  3. Node-level resource exhaustion where the gateway pod runs.

Diagnose

  • PromQL: up{namespace="networking", endpoint="http"} == 0 — identify the actual down target first.
  • kubectl get pods -n networking -o wide — pod status.
  • kubectl describe pod -n networking <pod> — events and container states.
  • kubectl logs -n networking <gateway-pod> — startup/config errors.
  • kubectl get gateway,httproute -A — whether Gateways are programmed.

Resolution: External ingress for the affected gateway may be impaired. Fixes to Gateway/GatewayClass/HTTPRoute definitions or Cilium gateway values are Git changes. If the operator is also down, fix CiliumOperatorDown first — gateway programming depends on it.

HubbleRelayDown

Severity: warning Fires when: up{job=~".*hubble-relay.*"} == 0 for 10m.

Likely causes

  1. Relay pod CrashLooping or Evicted.
  2. Relay cannot reach agent Hubble endpoints (TLS is cert-manager-issued here — check for expiry).
  3. Resource pressure making the relay unresponsive.

Diagnose

  • kubectl get pods -n kube-system -o wide | grep hubble-relay — pod status (name-match; avoids depending on a label selector).
  • kubectl logs -n kube-system deploy/hubble-relay — connection or TLS errors.
  • PromQL: up{job=~".*hubble-relay.*"} — confirm which instance.
  • kubectl get certificate -n kube-system — Hubble TLS is issued via cert-manager (certValidityDuration: 365); check it is not expired.

Resolution: Flow observability only — the datapath is unaffected, Hubble UI/CLI and flow export lose data while relay is down. Config, resources, and TLS issuer changes are Git changes. Relevant skills: /verify-hubble-tls, /check-hubble-export, /hubble-port-forward.

HubbleFlowDropSpike

Severity: warning Fires when: sum(rate(hubble_drop_total[10m])) > 50 for 30m — cluster-wide summed drop rate above 50/s, sustained 30 minutes. Because of the sum(), this is a cluster aggregate with no per-node breakdown in the alert itself.

Likely causes

  1. Policy denials from a newly applied or overly broad NetworkPolicy.
  2. Datapath drops (correlate with BPF/CT map pressure alerts above).
  3. A single noisy workload generating rejected traffic.

Diagnose

  • PromQL: sum(rate(hubble_drop_total[10m])) by (reason) — the drop reason is the single most useful split.
  • PromQL: topk(10, sum(rate(hubble_drop_total[10m])) by (pod, reason)) — where the drops concentrate.
  • PromQL: sum(rate(cilium_drop_count_total[5m])) by (reason) — agent-side corroboration.
  • hubble observe --type drop --last 200 — inspect actual dropped flows (via /hubble-port-forward).

Resolution: Depends entirely on the drop reason. POLICY_DENIED means a policy is doing its job or is too broad — either way the fix is a Git change to the offending NetworkPolicy/CiliumNetworkPolicy. Datapath drop reasons point back at map sizing (CiliumBPFMapPressureHigh / CiliumConntrackMapNearFull), also a Git change. Confirm the reason before changing any policy; a sustained drop spike is sometimes correct enforcement of an intended deny.