Runbooks¶
Operational runbooks. Reference when doing a specific task; not first-read material.
Alert runbooks¶
One file per alert rule group, in alerts/. Each documents every alert in its source rule: fire condition transcribed from the live expr/for, likely causes, read-only diagnostics, and the Git-side fix. All diagnostic commands are read-only — fixes land as a PR that ArgoCD reconciles.
| Runbook | Covers | Source rule |
|---|---|---|
authentik.md |
Authentik IdP availability, workers, tasks/blueprints, proxy outposts, enterprise licence. | cluster/monitoring/authentik-alerts.yaml |
chrony-ptb.md |
Node clock conformance: four PTB sources, NTS-only, no Ubuntu/DHCP fallback. Fixed via the Ansible role, not ArgoCD. | cluster/monitoring/chrony-ptb-alerts.yaml |
cilium.md |
Cilium agent, operator, Gateway API and Hubble health. | cluster/monitoring/cilium-alerts.yaml |
cluster.md |
Cluster-wide: pod/node health, storage, certs, databases, deployments, ArgoCD, Trivy, cert-manager, etcd, apiserver, Kyverno, Argo Workflows. | cluster/monitoring/alerting-rules.yaml |
gateway.md |
Gateway LoadBalancer IP rules. Both are currently non-functional — kube-state-metrics emits an info-style gauge, so neither can fire. Read the caveat before treating silence as health. | cluster/gateway/monitoring.yaml |
kps-self-alerts.md |
kube-prometheus-stack self-monitoring: PrometheusTargetDown, PrometheusScrapePoolExceedTarget, AlertmanagerConfigInconsistent, GrafanaDown. |
cluster/monitoring/kps-self-alerts.yaml |
litellm-gateway.md |
LiteLLM, the OpenAI-compatible front door for local models in ns ai. |
apps/ai/litellm/templates/prometheusrule.yaml |
llama-swap.md |
Single-pod llama-swap owning the B60 gpu.intel.com/xe share on k8s-worker-home-2. No redundancy by design. |
apps/ai/llama-swap/templates/prometheusrule.yaml |
loki-log-alerts.md |
LogQL rules over Hubble and Tetragon flow logs, evaluated by the Loki ruler — not PrometheusRule alerts. Query against the Loki datasource. | cluster/monitoring/loki-log-alerts.yaml |
longhorn.md |
Volume faulted/degraded, read-only filesystem, node and disk full, backup failure and staleness, instance-manager CPU. | cluster/monitoring/longhorn-alerts.yaml |
media-library.md |
JellyfinHighErrorRate, ArrMissingBacklogStale, and DataMediaZfsHigh (ZFS pool on the PVE host, via node_exporter — not a pod). |
cluster/monitoring/media-library-alerts.yaml |
ovms.md |
OVMS B60 saturation. Predictors scale to zero, so absent series for a sleeping model are expected. | apps/ai/ovms/templates/prometheusrule.yaml |
tetragon.md |
Tetragon observer health. Posture is visibility-only — these alerts mean the cluster is blind, not unprotected. | cluster/monitoring/tetragon-alerts.yaml |
velero.md |
Backup health, backup freshness, and infra health (BSLs) across the office and home MinIO targets. | cluster/monitoring/velero-alerts.yaml |
xe-gpu.md |
XeVramEvictCliff, XeVramNearFull, XeGtSustainedThrottle on Intel xe (Battlemage) nodes, from the node_exporter textfile collector. |
cluster/monitoring/xe-gpu-alerts.yaml |
Several alert runbooks carry staleness caveats: textfile-collector and worker-scraped series go stale silently, so absence of alerts is not evidence of health. The caveats are in the individual files.
Procedure runbooks¶
| Runbook | When to use |
|---|---|
synology-nas-setup.md |
First-time provisioning of the Synology NAS at 10.2.10.19 for k8s. |
secret-rotation-2026-04.md |
OPEN — rotate the secrets found by the gitleaks historical scan of 2026-04-15 (16 findings across 934 commits). HEAD is clean; historical commits still hold valid credentials until rotated. |
litellm-db-credential-rotation.md |
OPEN (#2886) — rotate the leaked LiteLLM DB password across the two owning SealedSecrets (ai/litellm-database consumer + databases/pg-cluster-litellm CNPG role). Argv exposure already closed (PR #2888); credential still needs a live-cluster rotation. |
etcd-restore.md |
Disaster recovery — restore the 4-node etcd cluster from a MinIO Office snapshot when the data store is lost or corrupted. ~30–60 min window. |
jellyfin-restore.md |
Recover the home-zone Jellyfin server when the config PVC is lost or corrupted (hybrid GitOps, PRs #436/#441). |
k8s-1.36-upgrade.md |
Live kubeadm upgrade of every node (4 CPs + 5 workers), 1.35.3 → 1.36.2, no VM rebuild. |
kured-home-worker-maintenance.md |
Fail-closed home-worker reboot gate and canary requirements while issue #2573 is open. |
ubuntu-26-04-rotation.md |
Root-only 50 GiB scsi0 replacement on the two live home worker VMs (9003, 9004) from the verified UEFI template, preserving VM identity and non-root disks (ADR-0007). |
paperless3-rebuild.md |
paperless-ngx 3 reset & cutover (#2106) — NAS prep, DB reset, PVC swap to static NFS + longhorn, readback checks. |
nextcloud-rebuild.md |
Fresh reset install of office Nextcloud onto the static NFS data PV (#2108) — freeze, dump/snapshot, reset, sync, verification. |
chrony-ptb-nts.md |
Audit or enforce the cluster-node clock policy: four PTB hostnames with NTS authentication, no Ubuntu/DHCP fallback, and two-source agreement. |
Adding a runbook¶
- Use a kebab-case filename describing the task:
<subject>-<action>.md. - Keep it operational: prerequisites → numbered steps → verification → what-can-go-wrong.
- Include only the commands you'd actually paste; no narrative.
- If the runbook will be re-run frequently, prefer turning it into a script under
../../scripts/. - Reference related ADRs and AGENTS.md for context.
Adding an alert runbook¶
- One file per rule group, named after the group or component:
alerts/<component>.md. - Open with the source rule path, PrometheusRule name, namespace and groups.
- Cover every alert in the rule. Transcribe
exprandforfrom the file — do not paraphrase the threshold. - State explicitly whether a resolution is a Git change or an investigation step; keep all pasted commands read-only.
- Record verification caveats (metrics that go stale, rules that cannot fire) up front, so silence is never mistaken for health.