Skip to content

0036. etcd Snapshot Count and Upgrade Playbook Robustness

  • Status: accepted
  • Date: 2026-07-13
  • Deciders: lusoris
  • Related: PR #2401

Context

etcd was configured with snapshot-count: 1000000, far above the etcd 3.6 default of 10000. That grew the write-ahead log to roughly 3.3 GB and drove etcd to an ~11 GB peak that OOM-killed the member on restart during the Kubernetes 1.36.2 upgrade. snapshot-catchup-entries has a separate purpose: it is the post-compaction raft-log window available to a slow follower. Separately, the upgrade playbook drove kubeadm through systemd-run, whose exit status did not reliably propagate, so the playbook could report success or failure incorrectly.

Decision

Set etcd_snapshot_count from 1000000 to 10000, matching the etcd 3.6 default. Initially set etcd_snapshot_catchup_entries to 10000. Make the upgrade playbook robust by capturing the kubeadm exit code to a file and polling that file for accurate status, so a systemd-run failure surfaces as a real failure rather than a false success.

Amendment — 2026-07-15

Raise etcd_snapshot_catchup_entries from 10000 to 100000 while keeping etcd_snapshot_count at 10000. Live evidence showed that a 399 MB database snapshot took 29m15s to cross the lossy home-office link. During that transfer, the leader advanced 40,394 raft entries. The 10,000-entry catch-up window had already been compacted away when home applied the snapshot, so the leader immediately sent another full database snapshot. A 100,000-entry window is more than twice the measured gap and gives approximately 64 minutes of retention at the observed ~26 entries/s leader proposal rate.

The two values are deliberately independent. Keeping snapshot-count=10000 continues to bound WAL size and restart memory; raising only the retained catch-up window does not restore the previous snapshot-count=1000000 failure mode. The added memory cost on each member is accepted to make cross-site slow-follower recovery converge.

Alternatives considered

  • Raise the etcd memory limit: masks unbounded WAL growth instead of fixing it; the OOM returns at scale.
  • Keep the large snapshot-count: the WAL bloat and restart OOM persist.
  • Keep a 10,000-entry catch-up window: rejected after it caused repeat full snapshots on the measured cross-site path.
  • Purge and re-add the home member: rejected because a replacement member also needs a full leader snapshot over the same path and does not fix retention.
  • Trust systemd-run exit propagation: already shown to misreport status, which is what motivated the change.

Consequences

  • WAL size and restart RAM are bounded on every member, including the laggy home member.
  • Slow followers can resume from retained raft entries after a long snapshot transfer instead of immediately starting another full snapshot.
  • Each member retains a larger in-memory raft window; monitor RSS and proposal lag, and size the window from observed transfer time and write rate.
  • Upgrade runs report accurate status, removing a class of silent false-success/false-failure during control-plane upgrades.
  • Values live in Ansible group_vars; the playbook change is reusable for future kubeadm upgrades.

References

  • PR #2401 — fix(k8s): etcd snapshot-count OOM fix + 1.36.2 upgrade playbook robustness
  • etcd 3.6 configuration options — snapshot-count triggers on-disk snapshots; snapshot-catchup-entries retains raft entries for slow followers after compaction.
  • etcd 3.6 tuning — latency and packet loss directly affect multi-site raft behavior.
  • ansible/inventory/group_vars/all.yml (etcd_snapshot_count, etcd_snapshot_catchup_entries), ansible/playbooks/