Skip to content

0006. Office CP quiesced: 3-member etcd + home-only apiserver-office-2 (interim)

  • Status: superseded — reverted to 4-member cluster 2026-05-05
  • Date: 2026-04-19
  • Deciders: lusoris
  • Blocked by: BUG-14 (etcd raft intolerant of cross-site UniFi-tunnel latency/jitter episodes)

Context

For several hours on 2026-04-19 the etcd-k8s-cp-office-1 member flapped continuously. Root-cause signals observed during the incident:

  • etcd-k8s-cp-office-1 logs: failed to commit proposal: context deadline exceeded, agreement among raft nodes before linearized reading, ReadIndex response took too long, publish error: context deadline.
  • Cross-site raft peer TCP stream k8s-cp-home-1 ↔ k8s-cp-office-1: unexpected EOF / connection reset by peer every 30–90 s, consistent with follower replication timing out during slow episodes on the cross-site UniFi gateway tunnel.
  • Cross-site RTT 20–81 ms with 15 ms jitter spikes (the UniFi gateway-to-gateway tunnel is stable link-wise but bursts into high-latency/jitter under load).
  • kube-apiserver-k8s-cp-office-1 crashlooped on PostStartHook "start-service-ip-repair-controllers" failed: unable to perform initial IP and Port allocation check — the 60 s watch-cache warmup couldn't complete because the local etcd member was raft-stalled.
  • CNPG operator + plugin-barman-cloud leader-election flapped in cnpg-system every ~5 min (the same crash-loop called out as BUG-14 candidate in the 22-cnpg deep-dive research).
  • Roughly half of the ArgoCD apps reached Progressing or Degraded.

The 4-node kubeadm cluster was designed with two etcd + apiserver peers per site, assuming the cross-site UniFi tunnel is fast enough for etcd raft heartbeats at all times. The BUG-14 evidence above says this assumption no longer holds: raft is very sensitive to the 15 ms jitter episodes that the tunnel produces under load, and the office members (office-1, office-2) were both acting as flaky followers for a home-elected leader.

BUG-14 is an etcd-side sensitivity problem, not a network-layer bug: the UniFi gateway-to-gateway tunnel itself stays up, it just slows down occasionally. Fixing it by relaxing raft's expectations (heartbeat / election timeouts) is a separate investigation and cannot be done safely in the same change window without user-visible downtime. An interim topology was chosen that stabilises the cluster immediately and keeps the door open for office-1 to rejoin later.

Decision

Operate the control plane in a reduced, stable configuration until BUG-14 is resolved:

etcd cluster: 3 members (home-1, home-2, office-2), leader pinned to home-1.

  • k8s-cp-office-1 etcd member removed via etcdctl member remove. Member ID e06820f549956bbc at time of removal. Data directory on k8s-cp-office-1 untouched.
  • Quorum math unchanged (still tolerates loss of 1 member). Pre-incident 4-member cluster also tolerated exactly 1 loss — no worse than before.
  • Rejoin procedure documented in docs/runbooks/etcd-restore.md (covers both full-restore and add-existing-member flows).

Control plane: office-1 kubelet-managed static pods quiesced.

  • /etc/kubernetes/manifests/etcd.yaml → /root/etcd.yaml.quiesced-2026-04-19
  • /etc/kubernetes/manifests/kube-apiserver.yaml → /root/kube-apiserver.yaml.quiesced-2026-04-19
  • The host otherwise runs normally (kubelet, kube-proxy, Cilium agent). The kube-vip static pod remains (harmless on office-1; it doesn't hold the VIP because its local 127.0.0.1:6443 probe fails).

apiserver-office-2: home-only etcd endpoints.

  • --etcd-servers=https://10.1.20.200:2379,https://10.1.20.201:2379 (was https://127.0.0.1:2379).
  • Rationale: apiserver round-robins the etcd endpoint list. If one endpoint stalls (as local office-2 etcd does today), a large fraction of apiserver calls stall with it; in particular, the 60 s watch-cache warmup at startup fails. Pointing exclusively at the stable home pair trades ~50 ms cross-site read latency for reliable startup and reliable admission.
  • Confirmed in the incident: same manifest with 127.0.0.1 in the list (even last) still tripped PostStartHook "start-service-ip-repair-controllers" at attempt 5. Removing local entirely was necessary, not just re-ordering.

kubernetes Service endpoints: 10.1.20.200, 10.1.20.201, 10.2.20.201. Office-1 self-removes as an apiserver-endpoint participant the moment its static pod goes down, so cluster-internal 10.96.0.1 traffic never lands on the dead node.

kube-vip: office VIP 10.2.20.10 now elected and held by office-2 permanently (no failover peer at office until office-1 rejoins). Home VIP 10.1.20.10 unchanged (home-1 + home-2 peers).

Alternatives considered

  • Leave the cluster as-is and debug the network live. Rejected. Half the ArgoCD estate was Progressing or Degraded, user directive "fix the actual problem, don't stall on diagnostics" (saved as feedback memory). Network-layer investigation of the UniFi tunnel latency profile is a separate multi-hour project and the cascade was actively worsening.
  • Keep office-1 etcd member, just quiesce its apiserver. Rejected. The flaky etcd member still blocks raft consensus — home-1's proposals time out waiting for follower ack. Removing the flaky member was the load-bearing fix for etcd; quiescing its apiserver alone would have left the original symptom intact.
  • Drop to 2-member home-only etcd (also remove office-2). Rejected. 2-member quorum tolerates zero losses (both must be up for writes). The home pair is on the same L2 segment and rarely fails simultaneously, but any single-node maintenance (kured reboot, chart bump requiring drain) would freeze writes. 3-member stays 1-tolerant.
  • Bake Raft ping / pre-vote / shorter heartbeat-timeout tuning into etcd. Deferred to BUG-14. The network jitter is large enough that no etcd-side tuning short of a full storage-tier lift solves it; raft is the wrong layer to paper over a network problem.
  • Switch apiserver-office-2 back to 3-endpoint home-1, home-2, 127.0.0.1 list with --etcd-healthcheck-timeout raised. Rejected at T+. The endpoint-round-robin behaviour is internal to apiserver's storage layer and doesn't honour per-endpoint health during the first fill; the 60 s start-service-ip-repair-controllers hook doesn't care why the watch is late. Keeping only healthy endpoints is the straightforward fix.

Consequences

Positive:

  • Office VIP stable, office-pinned workloads (CNPG operator, plugin-barman-cloud, pg-cluster, grafana-oncall-engine when fixed) schedule and run. ArgoCD app health drains back to Healthy.
  • Only one raft replication flow crosses the UniFi tunnel now (office-2 as follower); with two flows (office-1 + office-2 each replicating from home-1) the tunnel's slow episodes were enough to tip both into ReadIndex timeouts at once.
  • CNPG operator leader-election lease churn stops (BUG-14 crash-loop candidate confirmed as root-caused by apiserver availability, not by the plugin).

Negative:

  • Office-scoped apiserver reads pay ~50 ms cross-site for every admission, conversion, webhook callback, and informer relist. Expected impact: controller reconcile latency up by roughly that factor. No SLO breach observed at this scale, but chart bumps / mass-apply scenarios will feel slower.
  • Office loses apiserver HA. If office-2 apiserver crashes, office VIP dies with it. Home continues to serve (2 apiservers), but office kubelets lose their nearest apiserver. Mitigation: alert on up{job="kube-apiserver",instance="10.2.20.201:6443"}==0 (to be added in a follow-up PR on alerting-rules.yaml).
  • Office-1 node remains a kubelet worker but is not scheduled anything control-plane-relevant. Its kubelet + CNI stay up; this ADR doesn't cordon office-1 (it's still a potential scheduling target for pods that tolerate the node-role.kubernetes.io/control-plane taint).

Debt accepted (tracked in STATE.md 2026-04-19 incident block):

  • Ansible backfill: closed 2026-04-20. Three pieces had to land: (a) apiserver-office-2 --etcd-servers=https://10.1.20.200:2379,https://10.1.20.201:2379 pinned so a control-plane role re-run cannot revert it to 127.0.0.1:2379; (b) office-1 static-pod manifest pause encoded so tune-etcd / tune-apiserver playbooks skip the quiesced host; (c) kube-vip kubeconfig mount switched from admin.conf (server: site VIP) to kube-vip.conf (server: 127.0.0.1:6443) to break the restart-bootstrap chicken-and-egg that became fatal once office-2 lost its VIP-failover peer, plus lease / renewDeadline / retryPeriod relaxed from 15/10/2 to 60/40/5 to tolerate cross-site read excursions. (a) and (b) landed via PR #87 as ansible/playbooks/apply-adr0006-interim.yml driven by the new cp_quiesced_nodes and apiserver_etcd_servers_override vars in group_vars/all.yml. (c) landed via PR #118 in ansible/playbooks/deploy-kube-vip.yml.
  • BUG-14 triage: separate dossier under .workingdir2/bugs/. The question is how to make etcd tolerate the UniFi tunnel's slow episodes — likely knobs: raise heartbeat-interval / election-timeout further (already bumped to 1000/10000 for jitter); consider a --max-request-bytes / snapshot-rate trade-off; or deliberately pin the etcd leader to a home member to keep the cross-site flow in one direction only. Until BUG-14 is closed, this ADR stays proposed — it's an operational interim, not an accepted permanent architecture.
  • Office-1 rejoin runbook: extend docs/runbooks/etcd-restore.md with the "rejoin an existing member with untouched data-dir" path. The data dir on office-1 still carries a raft log up to just-before-removal; it must be wiped before rejoin (member removal invalidates the cluster ID).

Rejoin plan (to execute once BUG-14 is resolved)

  1. Confirm cross-site raft flow on office-2 stays stable for 1 h — no ReadIndex timeouts, no :2380 connection resets in etcd logs.
  2. Wipe office-1 etcd data dir: rm -rf /var/lib/etcd on k8s-cp-office-1.
  3. On home-1 leader: etcdctl member add k8s-cp-office-1 --peer-urls=https://10.2.20.200:2380.
  4. On office-1: restore /etc/kubernetes/manifests/etcd.yaml with --initial-cluster-state=existing and the cluster string returned by member add.
  5. Verify 4-member quorum, leader still home-1, all raftIndex equal after 60 s.
  6. Restore /etc/kubernetes/manifests/kube-apiserver.yaml on office-1.
  7. Flip apiserver-office-2 --etcd-servers back to multi-endpoint with local included (via Ansible, not live sed).
  8. Update this ADR status → superseded-by or accepted, depending on whether the Ansible state now matches ADR-0005 (pre-incident) exactly.

References

  • STATE.md 2026-04-19 incident block — live-action log.
  • feedback_fix_the_actual_problem.md — the "stop stalling on diagnostics" correction that drove the decision cadence during the incident.
  • 0005-control-plane-hardening.md — the pre-incident target architecture that this ADR is a temporary deviation from.
  • .workingdir2/research/22-cnpg/ — BUG-14 crash-loop evidence gathered during the CNPG deep-dive (predates this incident).
  • docs/runbooks/etcd-restore.md — member-rejoin + full-restore runbook.