Skip to content

0010. Defer multi-site Kyverno split

  • Status: accepted (defer)
  • Date: 2026-05-25
  • Deciders: lusoris
  • Related: issue lusoris/k8s#56 (K1.D — multi-site Kyverno), ADR-0006 (office 3-member etcd interim).

Context

cauda runs a single Kyverno install (one Helm release in the kyverno namespace) with admission controllers fronted by the cluster apiserver. Issue #56 proposed splitting Kyverno across the home + office sites so admission decisions for office-pinned workloads are made by office-side replicas, and vice-versa, reducing apiserver→Kyverno latency over the cross-site link.

The link itself is the UniFi gateway tunnel (see reference_crosssite_tunnel memory) — stable but with periodic ~100 ms RTT spikes that show up as admission latency budget burns (KyvernoAdmissionLatencyP99High).

Decision

Defer, with the issue staying open as a roadmap entry.

The cost of splitting is high:

  • Two Helm releases means two CRD sets, and Kyverno's policy CRs are cluster-scoped. We'd either run them with different policy subsets (split by namespace selector — duplicates the policy authoring layer) or maintain identical sets and trust both replicas to converge.
  • Webhook configuration becomes a routing problem: which ValidatingWebhookConfiguration does the apiserver call? The native webhook contract is per-cluster, not per-site. We'd need an apiserver proxy that picks the closest Kyverno based on the request's source — significant infra.
  • The current p99 admission latency is well within the 8-second webhook timeout. The actual KyvernoAdmissionLatencyP99High alert fires sporadically and is dominated by single-policy hot paths (require-resource-limits, disallow-default-namespace), not link RTT.

The expected benefit is small in absolute terms (~50 ms latency cut on a path the user never observes directly) and the operational surface explosion is real (twice the policy-test fixtures, twice the admission webhook trouble-shooting paths, twice the helm bumps).

Alternatives considered

  • Split status quo + add load balancer routing. Doable but the apiserver doesn't know to route admission calls by site; we'd be inventing a kube proxy layer.
  • Two Kyverno releases with namespace selectors. Policy duplication. Easy to drift. No.
  • Use policies.kyverno.io/v1alpha1 ValidatingPolicy (CEL) — CEL policies run in-process on the apiserver, removing the webhook hop entirely. This is the long-game answer. Tracked in #65 (CEL migration).

Consequences

  • Issue #56 stays open as a deferred roadmap item; the policy CRs continue to live in a single Kyverno install in office.
  • We pay for the cross-site latency on every admission decision for home-side workloads (small in practice — workload create is rare).
  • The CEL migration tracked in #65 supersedes most of #56's value; once the bulk of policies are CEL the admission hop disappears.
  • If admission p99 ever pegs at the webhook timeout regularly, revisit and consider implementing the split. The KubeAPIDown inhibit rule already mutes downstream noise during such an outage.

References

  • Memory: reference_crosssite_tunnel.md — cross-site link is UniFi gateway tunnel, not IPsec; stable but sometimes slow.
  • cluster/kyverno/restrict-node-port-cel.yaml — proof-of-pattern CEL conversion (issue #65).
  • argocd-apps/infra/kyverno.yaml — current single-install Kyverno.