Skip to content

Tetragon alerts

Source rule: cluster/monitoring/tetragon-alerts.yaml (PrometheusRule tetragon-alerts, ns monitoring, group tetragon-health).

Tetragon is the eBPF runtime-security observer: a privileged DaemonSet (hostNetwork, host PID) plus a single operator replica that reconciles TracingPolicy CRDs. Posture is visibility-only (decision T2) — no Sigkill/Override/Signal. When these alerts fire the cluster is not less safe in the enforcement sense; it is blind, which is what makes them worth working.

Where things live

Namespace note: the Application destination is kube-system and has been since the component landed. apps/security/tetragon/AGENTS.md prose says security — that prose is stale; the Application is authoritative. Confirm live with kubectl get ds -A | grep tetragon before pasting the commands below, and use whatever that returns.

Events reach Loki via the export-stdout sidecar → Alloy → Loki, tagged flow_log="tetragon".

All cluster changes go through Git → ArgoCD. Every command below is read-only; do not apply, patch, scale, delete, edit, annotate, cordon, or drain anything while working these alerts.

TetragonRingbufEventLoss

Severity: warning

Fires when: rate(tetragon_observer_ringbuf_events_lost_total[5m]) > 0 holds for: 10m — i.e. any non-zero loss rate, sustained ten minutes. There is no tolerance band; a single steady trickle of dropped events trips it.

Likely causes

  1. Kernel ringbuf overflowing because the agent cannot drain it fast enough — usually a node with heavy process churn (media + ai stacks).
  2. Tetragon pod CPU-throttled against its limits.cpu: "1".
  3. A burst workload (batch job, image build, CI) producing exec events far above baseline.

Diagnose (read-only)

kubectl get ds -A | grep tetragon                                  # confirm namespace first
kubectl -n kube-system get pods -o wide | grep tetragon
kubectl -n kube-system logs <tetragon-pod> -c tetragon --tail=100
kubectl top nodes
kubectl top pods -n kube-system | grep tetragon

rate(tetragon_observer_ringbuf_events_lost_total[5m])
rate(tetragon_observer_ringbuf_events_lost_total[5m]) > 0            # which instances

Resolution: identify the node and the workload driving the churn. If the agent is CPU-starved, the fix is a Git change to resources in apps/security/tetragon/values.yaml. Note that the ringbuf size itself is not currently set in our values — adding a size override is also a Git change, and should be justified rather than reflexive. If a single transient batch job caused it, the alert resolving on its own is the expected outcome; no action.

TetragonBPFMissedEvents

Severity: warning

Fires when: rate(tetragon_missed_prog_probes_total[5m]) > 0 holds for: 10m. Same shape as the ringbuf alert — any sustained non-zero rate. The alert summary carries an error label identifying the failure kind.

Likely causes

  1. Kernel-side event sends failing under buffer pressure — frequently correlated with TetragonRingbufEventLoss on the same instance.
  2. Exec/syscall rate exceeding what the BPF programs can hand off.
  3. Kernel / eBPF incompatibility after a node kernel change (relevant here: the home nodes have moved kernels for the Arc GPU work).

Diagnose (read-only)

kubectl -n kube-system logs <tetragon-pod> -c tetragon --tail=200
kubectl describe node <node>                       # kernel version, pressure conditions
kubectl get nodes -o wide                          # kernel/OS image across the fleet

rate(tetragon_missed_prog_probes_total[5m])
sum by (error) (rate(tetragon_missed_prog_probes_total[5m]))          # break down by error kind

Resolution: first check whether this is fleet-wide or isolated to nodes on a particular kernel — that distinguishes load pressure from a kernel/agent-version mismatch. Load pressure is a Git change to Tetragon resources; a version mismatch is a Git change to targetRevision in the Application. Neither is a live action.

TetragonPolicyLoadError

Severity: critical

Fires when: tetragon_tracingpolicy_loaded{state=~"error|load_error"} > 0 holds for: 5m — a policy is reporting the error or load_error state. Critical because a policy in this state is neither observing nor enforcing; the detection it represents is simply absent.

Likely causes

  1. Invalid TracingPolicy spec — bad selector, malformed matchArgs, unsupported field.
  2. Kernel lacks a hook or eBPF helper the policy needs (kprobe symbol missing on that kernel).
  3. Policy references a symbol or path that does not exist on the node.

Diagnose (read-only)

kubectl get tracingpolicy -A
kubectl get tracingpolicynamespaced -A
kubectl describe tracingpolicy <policy>
kubectl -n kube-system logs <tetragon-pod> -c tetragon --tail=200 | grep -i policy
kubectl -n kube-system logs deploy/tetragon-operator --tail=100

tetragon_tracingpolicy_loaded{state=~"error|load_error"}
tetragon_tracingpolicy_loaded                                        # full state picture

Resolution: always a Git change. The five policies live in cluster/tetragon/ (fileless-exec, kernel-module-load, privilege-escalation, sensitive-file-access, writable-dir-exec). Fix the manifest there and let ArgoCD reconcile — do not delete or re-apply the CRD live to "reload" it. Use /verify-tetragon-policies to lint before opening the PR. If the policy is valid but the kernel lacks the hook, the decision is whether to scope the policy by node or drop it; that warrants an ADR note rather than a silent edit.

TetragonAgentDown

Severity: critical

Fires when: up{service="tetragon"} == 0 holds for: 5m — the scrape target for the agent on some instance has been failing for five minutes. Per-instance: one node's agent being down fires this while the rest of the fleet is fine.

Likely causes

  1. Agent pod crashing (CrashLoopBackOff) or OOMKilled against limits.memory: 1Gi.
  2. The node itself is down, NotReady, or rebooting (kured maintenance window).
  3. Metrics endpoint :2112 unreachable — the DaemonSet is hostNetwork, so a node-level port conflict can do this.

Diagnose (read-only)

kubectl -n kube-system get pods -o wide | grep tetragon
kubectl -n kube-system describe pod <tetragon-pod>          # OOMKilled / CrashLoopBackOff / last state
kubectl -n kube-system logs <tetragon-pod> -c tetragon --previous --tail=100
kubectl get nodes -o wide                                   # is the node itself NotReady?
kubectl -n kube-system get events --sort-by=.lastTimestamp | tail -30

up{service="tetragon"}
up{service="tetragon"} == 0                                 # which instances

Resolution: if the node is down or mid-reboot, this is the node's incident, not Tetragon's — work that first and expect this to clear. If the pod is crashing, read the previous-container logs and fix the cause in Git (apps/security/tetragon/values.yaml for resources/config, or the Application for a chart version). Do not delete the pod to get it rescheduled; that hides the defect and the DaemonSet will reproduce it. Do not cordon or drain the node for this alert alone.

TetragonOperatorDown

Severity: warning

Fires when: up{service="tetragon-operator-metrics"} == 0 holds for: 10m — the operator's metrics target (:2113) has been down for ten minutes. Warning rather than critical because already-loaded policies keep running; what stops is reconciliation of TracingPolicy CRDs and CRD installation (crds.installMethod: operator).

Likely causes

  1. The single operator replica (replicas: 1) is down, crashing, or Pending — there is no redundancy here by design.
  2. OOMKilled against the tight limits.memory: 128Mi.
  3. Pod unschedulable due to node resource pressure.

Diagnose (read-only)

kubectl -n kube-system get deploy | grep tetragon
kubectl -n kube-system describe deploy tetragon-operator
kubectl -n kube-system get pods -o wide | grep tetragon-operator
kubectl -n kube-system logs deploy/tetragon-operator --tail=100
kubectl -n kube-system logs deploy/tetragon-operator --previous --tail=100

up{service="tetragon-operator-metrics"}

Resolution: a crashing or OOMKilled operator is a Git change to tetragonOperator.resources in apps/security/tetragon/values.yaml — the memory limit is 128Mi and is the first thing to check. A Pending pod is a scheduling/capacity problem on the cluster, handled as its own incident. Because policies continue to run while the operator is down, there is no justification for a live restart to "fix" the metric.

TetragonExportStalled

Severity: warning

Fires when: time() - tetragon_events_last_exported_timestamp > 1800 holds for: 5m — no event has been exported for more than 30 minutes (1800s), sustained five minutes.

Read the threshold history before acting. The rule carries an in-file comment: this was originally 5 minutes and fired daily on k8s-cp-home-1 and k8s-cp-office-2 while the export pipeline was verifiably healthy. Control-plane nodes run etcd / apiserver / scheduler / controller-manager, which rarely fork, so they legitimately produce dead air. 30 minutes was chosen to keep catching real stalls (a worker node would normally export within seconds) without the CP false positive. Do not "fix" a recurrence by raising the threshold again — at 30 minutes on a worker node this is a real signal.

Likely causes

  1. The export-stdout sidecar is stuck, crashed, or cannot read the JSON log (file perms are 640 precisely so the sidecar's group can read it).
  2. The agent stopped producing events at all — cross-check TetragonAgentDown on the same instance.
  3. Genuine dead air on a quiet control-plane node, now much rarer at the 30-minute threshold but still possible.
  4. The export deny-list is dropping everything the node produces (all its traffic is from denied namespaces such as kube-system, cilium, argocd, monitoring).

Diagnose (read-only)

kubectl -n kube-system get pods -o wide | grep tetragon
kubectl -n kube-system logs <tetragon-pod> -c export-stdout --tail=100    # sidecar is `export-stdout`
kubectl -n kube-system describe pod <tetragon-pod>                        # sidecar restarts?
kubectl get nodes -o wide                                                 # is the instance a control-plane node?

time() - tetragon_events_last_exported_timestamp                          # staleness per instance, ranked
tetragon_events_last_exported_timestamp
up{service="tetragon"}                                                    # rule out agent-down as the cause

Loki — confirm whether anything is actually landing from that node:

{flow_log="tetragon"} | json

Resolution: first establish whether the instance is a control-plane node (expected quiet) or a worker (real problem). On a worker, work it as a broken export path: sidecar crash, log-file permission drift, or the deny-list swallowing everything. Every one of those fixes is a Git change to apps/security/tetragon/values.yaml (exportFilePerm, exportDenyList, export.mode) or to the Alloy pipeline in argocd-apps/monitoring/alloy.yaml. Use /check-tetragon-export to verify the deny-list and Loki path end to end. Restarting the sidecar live is not the fix and destroys the evidence.

TetragonProcessCacheEvictionsHigh

Severity: warning

Fires when: rate(tetragon_process_cache_evictions_total[5m]) > 30 holds for: 15m — sustained eviction rate above 30/s for fifteen minutes.

Read the threshold history before acting. The rule comment records that this was > 10, and that busy worker nodes running the media + ai stacks sustained 24–39/s even after processCacheSize was raised 65536 → 131072 (PR #849) → 262144 (PR #866). The threshold was moved to 30 because, at the current 4x cache size, anything above that sustained for 15 minutes indicates real pressure.

This matters for the resolution: "just raise processCacheSize" has already been tried twice and did not resolve it. Do not open a third bump PR without evidence that the workload profile has changed.

Likely causes

  1. Genuine high process churn on the node — media stack and ai workloads forking heavily.
  2. A new or misbehaving workload creating short-lived processes at an unusual rate (a hot-looping CronJob, a crashlooping pod restarting continuously).
  3. Cache pressure genuinely exceeding 262144 entries because the node's workload set grew.

Diagnose (read-only)

kubectl -n kube-system get pods -o wide | grep tetragon
kubectl top pods -n kube-system | grep tetragon
kubectl top nodes
kubectl get pods -A --field-selector spec.nodeName=<node> -o wide         # what actually runs there
kubectl get pods -A --sort-by=.status.containerStatuses[0].restartCount | tail -20   # crashloopers

rate(tetragon_process_cache_evictions_total[5m])
topk(5, rate(tetragon_process_cache_evictions_total[5m]))                 # worst instances
rate(tetragon_process_cache_evictions_total[5m]) > 30

Resolution: identify the workload driving the churn before touching Tetragon config — a crashlooping pod or a runaway job is the more common root cause, and fixing that is the real resolution. Only if the node's legitimate steady-state workload has genuinely grown is another processCacheSize increase justified, and that is a Git change to apps/security/tetragon/values.yaml with a note on why the previous two bumps were insufficient. Evictions alone do not mean lost events; they mean process metadata may be incomplete on some events. Treat this as a tuning signal, not an outage.