Skip to content

0005 — Control-plane hardening (etcd + kube-apiserver)

  • Status: proposed
  • Status note (2026-07-26): most of the decision has shipped, but the live control plane diverges from the binding apiserver items, so this stays proposed rather than flipping to accepted. Shipped and verified: E1 (cluster/monitoring/etcd-snapshot-cronjob.yaml + ansible/playbooks/etcd-snapshot-install.yml; live etcd-snapshot CronJob on the 6 h schedule), E2 (cluster/monitoring/servicemonitor-control-plane.yaml), E3 (live --quota-backend-bytes=8589934592 — later doubled past the decided 4 GiB), A4 (cluster/monitoring/priority-fairness.yaml; FlowSchemas live), M (all five custom alerts in cluster/monitoring/alerting-rules.yaml), S1 (terraform/cloudflare.tf Access application + policy for both API hostnames), R (six skills, ansible/playbooks/control-plane/AGENTS.md, docs/runbooks/etcd-restore.md). Diverged: the entire T2 flag tier is staged but not applied — ansible/playbooks/tune-apiserver-{audit,oidc,encryption,anonymous-auth}.yml exist in-tree, yet all three live apiservers run zero --audit-*, --oidc-*, --encryption-provider-config, and --anonymous-auth flags (verified against the static-pod specs 2026-07-26). A1/A2/A3/A5 therefore remain decisions without live effect until an attended T2 rolling-restart window executes those playbooks.
  • Date: 2026-04-18
  • Deciders: lusoris

Context

The etcd + kube-apiserver deep-dive (.workingdir2/research/18-etcd-apiserver/) flagged nine material gaps on the 4-node kubeadm control plane: no etcd snapshot backups (defrag-only, manual recovery path), no kube-apiserver audit trail, no OIDC flow (kubeconfigs are hand-distributed), no encryption-at-rest for Secrets, --anonymous-auth left at the k8s ≤1.31 default, no dedicated Grafana dashboard or custom PrometheusRules for the control plane, no ServiceMonitor pinning scrape targets to real etcd members with mTLS, no Priority-and-Fairness tuning for known-noisy clients (argocd-repo-server, kyverno reports-controller, hubble-relay), and the public API endpoints (k8s-home.infra.cauda.dev, k8s-office.infra.cauda.dev) are reachable through cloudflared without a Cloudflare Access policy. This ADR records the decisions taken to close those gaps in one coordinated change set, aligned with the established deep-dive pattern (config, dashboards, alerts, skills, hooks, AGENTS.md, runbook) used by the ArgoCD, Hubble, Tetragon, and Kyverno sweeps.

Decision

The control plane is hardened along four axes. Every item below is binding; no hedging.

etcd:

  • E1 — Backups: ship etcd snapshots every 6 h, retain 14 days, target is MinIO Office (Synology-backed S3 endpoint) at s3://etcd-backups/scheduled/etcd-snapshot-<RFC3339>.db. Same pattern as Longhorn / Loki / CNPG. Not direct NFS — the Synology only speaks NFSv3 and the S3 gateway gives versioning + lifecycle for free.
  • E2 — Scrape target: declare an explicit ServiceMonitor for etcd (port 2381) with mTLS client certs mounted from the kubeadm PKI. Replaces reliance on kube-prometheus-stack chart defaults.
  • E3 — quota-backend-bytes: bump to 4 GiB (4294967296). Preemptive headroom for the growing CRD footprint; restore time impact acceptable at that size.
  • Storage tier: unchanged. The per-node SSDs are fast enough; a dedicated partition is not on the critical path.

kube-apiserver:

  • A1 — Audit logging: layered policy — Metadata default, RequestResponse for the sensitive verb set (secrets, configmaps, roles, rolebindings, clusterrolebindings, serviceaccounts/token). Log to node disk, ship to Loki via the existing Alloy DaemonSet.
  • A2 — OIDC: wire Authentik as OIDC provider for the apiserver (--oidc-issuer-url, --oidc-client-id, --oidc-username-claim=email, --oidc-groups-claim=groups). Ship a /kubeconfig-oidc skill that generates a per-user kubeconfig using kubectl oidc-login. Authentik Application + Provider declared as an in-repo blueprint.
  • A3 — Encryption at rest: EncryptionConfiguration with AES-GCM on secrets only (keeps the plugin-free path; no KMS dependency). Key distributed to CP nodes via Ansible, sealed via sealed-secrets.
  • A4 — Priority & Fairness: declare FlowSchema + PriorityLevelConfiguration for argocd-repo-server, kyverno reports-controller, and hubble-relay so they can't starve the default pool during backlog catch-up.
  • A5 — anonymous-auth: --anonymous-auth=false. Health probes are switched to authz'd endpoints during the rollout.

Monitoring:

  • M — Full package: enable the chart-bundled Grafana dashboards (etcd, kubernetes-system-apiserver) via kube-prometheus-stack values, add cauda-custom PrometheusRules (EtcdNoLeader, EtcdDBSizeApproachingQuota at 80 %, EtcdBackupMissing, ApiserverAuditLogBackpressure, ApiserverRequestThrottled), and declare an Alertmanager inhibit hierarchy — EtcdMembersDown inhibits ApiserverRequestErrors; ApiserverAllDown inhibits downstream workload alerts.

Secure access:

  • S1 — Cloudflare Access: add a cloudflare_zero_trust_access_application plus a cloudflare_zero_trust_access_policy resource in terraform/cloudflare.tf for both k8s-home.infra.cauda.dev and k8s-office.infra.cauda.dev, with Authentik as the SAML/OIDC IDP on the Access side. Audit of the current Terraform state confirmed neither resource exists today.

Runbook / skills / AGENTS.md:

  • R — Full: six skills (/etcd-snapshot, /etcd-restore, /etcd-member-status, /apiserver-flags-audit, /cp-upgrade-checklist, /kubeconfig-oidc), an AGENTS.md at ansible/playbooks/control-plane/ consolidating tune-etcd / fix-home-kubelet-apiserver / the new snapshot cron and upgrade sequence, and a restore runbook at docs/runbooks/etcd-restore.md.

Implementation tiers

  • T0 — manifests-only, ArgoCD-sync-safe: ServiceMonitor, FlowSchema/PriorityLevelConfiguration, custom PrometheusRules, kube-prometheus-stack values update (dashboards + inhibits), etcd snapshot CronJob + MinIO user + sealed creds, Authentik OIDC Application blueprint.
  • T1 — Ansible node-side, no apiserver flag change: etcd quota-backend-bytes bump, audit policy file distribution, EncryptionConfiguration + key distribution.
  • T2 — Ansible apiserver-flag changes, requires a rolling CP restart: --audit-*, --oidc-*, --encryption-provider-config, --anonymous-auth=false.
  • T3 — Terraform Cloudflare + Authentik IDP coordination: Access Application + Policy resources.

Consequences

Positive:

  • First genuine etcd recovery path. MTTR moves from "unbounded, manual" to a documented 30–60 min restore for the whole 4-node cluster.
  • Every mutation to the cluster becomes attributable through the audit trail; incident triage gains a searchable source of truth via Loki.
  • SSO replaces long-lived static kubeconfigs; offboarding a user becomes a single Authentik action instead of tracking down credential copies.
  • Secrets gain a second encryption barrier on top of disk encryption.
  • Noisy clients can no longer starve the apiserver request queue.
  • Dashboards and inhibit hierarchy shorten alert-to-diagnosis time on CP incidents and suppress the downstream alert storm.
  • Public API endpoints gain an identity-bound gate; a stolen kubeconfig is no longer sufficient to reach the apiserver from the internet.

Negative:

  • Every T2 flag change is a rolling restart of all four CP nodes — ~15 min of degraded-but-not-down apiserver per pass, serialised across nodes. A maintenance window is required.
  • The audit log consumes node disk at non-trivial rate (RequestResponse on sensitive verbs writes bodies). Node disk pressure becomes a monitored quantity; rotation policy must be tuned.
  • The OIDC rollout has an adoption curve — every existing kubeconfig user must switch to kubectl oidc-login within a grace period before static kubeconfigs are retired. Scripts, CI jobs, and iOS (Nautik) configs all need a migration pass.
  • Cloudflare Access requires a working Authentik IDP hookup on the CF side; if Authentik is down, the public API endpoints are unreachable (site-local 10.{1,2}.20.10:6443 remains available to admins).
  • Six new skills, a new AGENTS.md, dashboards, and rules broaden the surface the deep-dive pattern asks future contributors to keep in sync.

Alternatives considered

  • Migrate to Talos. Rejected for this pass — scope is unbounded and the kubeadm cluster is stable. Revisit once the T2 work is stable.
  • Use Velero for etcd backups. Rejected — Velero snapshots k8s resources and PVs via the API, not the etcd data store itself. Wrong tool for a control-plane disaster where the apiserver is the thing that may be unavailable. etcdctl snapshot save is the correct mechanism.
  • Defer everything and accept the risk. Rejected — the inventory identified nine gaps that compound. In particular, "no etcd backup" is a single-incident-away-from-rebuild failure mode.

References

  • .workingdir2/research/18-etcd-apiserver/01-inventory.md — current-state inventory.
  • .workingdir2/research/18-etcd-apiserver/02-decision-log.md — locked decisions (round 1 + round 2).
  • .workingdir2/research/18-etcd-apiserver/99-decision-matrix.md — full option set.
  • docs/adr/0001-record-architecture-decisions.md — ADR process.
  • docs/adr/0002-dual-cloudflared-split-dns.md — cloudflared tunnel topology that fronts the public apiserver hostnames.
  • docs/adr/0003-ipv6-phased-rollout.md — dual-stack Service CIDR kubeadm patch that will land alongside A5's rolling CP restart.
  • docs/adr/0004-latest-tag-remediation.md — pinning discipline that applies to every chart/image this ADR touches.
  • kube-prometheus-stack chart — bundled etcd + apiserver dashboards and default rule groups.
  • kubeadm reference — control-plane configuration, patches, and upgrade flow.
  • int128/kubelogin — kubectl oidc-login plugin backing the /kubeconfig-oidc skill.
  • Authentik — Kubernetes provider — Application + Provider configuration for the apiserver OIDC flow.