etcd Restore — 4-node dual-site cluster¶

Restore etcd from a MinIO Office snapshot when the cluster's data store is lost or corrupted beyond repair. Applies to the 4-node control plane (k8s-cp-home-{1,2}, k8s-cp-office-{1,2}). Expected recovery window: 30–60 min end-to-end.

Related: ADR-0005 defines the backup target; ansible/playbooks/control-plane/AGENTS.md covers the snapshot CronJob that produces these files.

Preflight¶

Do not start a restore without answering all of these.

Is a restore actually required?
Can the apiserver still serve reads? A partial etcd degradation (one member down, quorum intact) is a member-replace, not a restore. Use /etcd-member-status first.
Has a rogue ArgoCD sync wiped a namespace? That's a Git revert, not an etcd restore.
Scope of damage. Record:
Which members report unhealthy (etcdctl endpoint status on any surviving CP node).
Approximate blast radius — one object, one namespace, or cluster-wide.
Snapshot availability. Confirm at least one snapshot from the last 6 h is present on MinIO Office before touching anything on the CP nodes. See "Fetch snapshot" below.
Coordinate.
Silence the etcd* + kube-apiserver* Alertmanager routes for the expected window (use the ntfy-silence skill or the Alertmanager UI).
Freeze ArgoCD sync: set syncPolicy.automated: null on the app-of-apps or use the argocd app set --sync-policy none burst script. Restore-time drift would cascade.
Notify users of the public API endpoints (k8s-home.infra.cauda.dev, k8s-office.infra.cauda.dev) that the cluster is in maintenance. If Cloudflare Access is live, they'll see an identity challenge regardless, but the apiserver behind it will be unavailable.
Have the kubeadm ConfigMaps on hand. The last-known-good kubeadm-config + kubelet-config + cluster-info ConfigMaps are the fallback if the restore path itself fails — see "If restore fails" at the bottom.

Fetch the snapshot¶

On any workstation or CP node with MinIO creds configured:

# List recent snapshots
mc ls s3/etcd-backups/scheduled/ | tail -20

# Pick the freshest (or a specific date) and pull it
SNAP=etcd-snapshot-2026-04-18T120000.db
mc cp s3/etcd-backups/scheduled/"$SNAP" /tmp/"$SNAP"

Verify integrity before proceeding. A truncated snapshot will silently restore a broken data-dir.

etcdctl snapshot status --write-out=table /tmp/"$SNAP"
# Expect: non-zero hash, non-zero revision, size matches MinIO object size.

Copy the snapshot to k8s-cp-home-1:/tmp/ — this node is the designated restore seed.

scp /tmp/"$SNAP" k8s-cp-home-1:/tmp/

Scale down the control plane¶

All steps below run as root on the CP nodes. Use SSH-as-root or sudo -i.

1. Stop apiserver, controller-manager, scheduler on every CP node¶

On each of k8s-cp-home-1, k8s-cp-home-2, k8s-cp-office-1, k8s-cp-office-2:

cd /etc/kubernetes/manifests
mkdir -p /etc/kubernetes/manifests.disabled
mv kube-apiserver.yaml kube-controller-manager.yaml kube-scheduler.yaml \
   /etc/kubernetes/manifests.disabled/

Kubelet will stop the static pods within ~10 s. Verify:

crictl ps | grep -E 'kube-apiserver|kube-controller|kube-scheduler'
# Expect: no rows.

2. Stop etcd on every CP node¶

Same loop, with etcd:

mv /etc/kubernetes/manifests/etcd.yaml \
   /etc/kubernetes/manifests.disabled/etcd.yaml

# Confirm etcd container has been reaped
crictl ps | grep etcd
# Expect: no rows.

Only proceed once crictl ps returns no etcd container on all four CP nodes.

Restore on the seed node (k8s-cp-home-1)¶

3. Backup the existing data-dir, wipe it¶

mv /var/lib/etcd /var/lib/etcd.bak-$(date +%s)
mkdir -p /var/lib/etcd
chown root:root /var/lib/etcd
chmod 700 /var/lib/etcd

4. Run `etcdctl snapshot restore` — single-member bootstrap¶

We restore into a new cluster-of-one on the seed node. The other three members are re-added after etcd comes up.

ETCDCTL_API=3 etcdctl snapshot restore /tmp/"$SNAP" \
  --name k8s-cp-home-1 \
  --initial-cluster k8s-cp-home-1=https://10.1.20.200:2380 \
  --initial-cluster-token cauda-etcd-restore \
  --initial-advertise-peer-urls https://10.1.20.200:2380 \
  --data-dir /var/lib/etcd

(IPs follow the established CP addressing: k8s-cp-home-1=10.1.20.200, k8s-cp-home-2=10.1.20.201, k8s-cp-office-1=10.2.20.200, k8s-cp-office-2=10.2.20.201. Adjust if the inventory has drifted.)

5. Patch the etcd static-pod manifest to match the new token¶

Edit the copy in /etc/kubernetes/manifests.disabled/etcd.yaml:

--initial-cluster=k8s-cp-home-1=https://10.1.20.200:2380
--initial-cluster-state=existing
--initial-cluster-token=cauda-etcd-restore

Move it back to /etc/kubernetes/manifests/ and wait for kubelet to start the pod.

mv /etc/kubernetes/manifests.disabled/etcd.yaml /etc/kubernetes/manifests/
sleep 15
crictl ps | grep etcd

6. Confirm etcd is serving¶

ETCDCTL_API=3 etcdctl \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  --endpoints=https://10.1.20.200:2379 \
  member list --write-out=table
# Expect: one member, STATUS=started, learner=false.

Re-add the other three members¶

For each remaining node, in this order — k8s-cp-home-2, k8s-cp-office-1, k8s-cp-office-2:

7. Announce the new member from the seed node¶

ETCDCTL_API=3 etcdctl \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  --endpoints=https://10.1.20.200:2379 \
  member add k8s-cp-home-2 \
  --peer-urls=https://10.1.20.201:2380

Note the --initial-cluster and ETCD_INITIAL_CLUSTER_* values etcdctl prints — they're needed on the joining node.

8. Prepare the joining node¶

On the joining node (example: k8s-cp-home-2):

mv /var/lib/etcd /var/lib/etcd.bak-$(date +%s) || true
mkdir -p /var/lib/etcd
chown root:root /var/lib/etcd
chmod 700 /var/lib/etcd

Edit /etc/kubernetes/manifests.disabled/etcd.yaml on that node:

--name=k8s-cp-home-2
--initial-cluster=<string printed by etcdctl member add>
--initial-cluster-state=existing
--initial-cluster-token=cauda-etcd-restore
--initial-advertise-peer-urls=https://10.1.20.201:2380

Move it back:

mv /etc/kubernetes/manifests.disabled/etcd.yaml /etc/kubernetes/manifests/
sleep 20

9. Verify the member joined¶

From the seed node:

etcdctl ... member list --write-out=table
# Expect: two members, both STATUS=started.
etcdctl ... endpoint health --cluster
# Expect: both endpoints healthy.

Repeat steps 7–9 for k8s-cp-office-1 and k8s-cp-office-2. Wait for the previous join to reach STATUS=started before adding the next.

Bring the apiserver layer back¶

10. Restore the control-plane static pods on every CP node¶

On each of the four CP nodes:

mv /etc/kubernetes/manifests.disabled/kube-apiserver.yaml \
   /etc/kubernetes/manifests.disabled/kube-controller-manager.yaml \
   /etc/kubernetes/manifests.disabled/kube-scheduler.yaml \
   /etc/kubernetes/manifests/

Kubelet will start them within ~15 s. Watch for:

crictl ps | grep -E 'kube-apiserver|kube-controller|kube-scheduler'

Post-restore verification¶

From a workstation with a site-local kubeconfig (bypass cloudflared — the public endpoint may have been identity-gated or rate-limited during the incident):

kubectl --server=https://10.1.20.10:6443 get nodes
# Expect: 4 CP + all workers Ready.

kubectl get pods -A | grep -vE 'Running|Completed'
# Expect: empty after ~2 min.

kubectl -n kube-system get ep kube-controller-manager kube-scheduler
# Expect: endpoints populated (one per active leader).

etcdctl ... endpoint status --cluster --write-out=table
# Expect: 4 members, one leader, DB size sane, revisions aligned.

Check ArgoCD is not auto-syncing — drift during post-restore is expected (audit trails, volume attachments, transient pods). Unfreeze syncs only after a full kubectl get pods -A sweep is clean:

# Re-enable auto-sync per ArgoCD application (app-of-apps pattern):
argocd app set <root-app> --sync-policy automated --auto-prune --self-heal

Finally, lift the Alertmanager silence and notify users.

If restore fails¶

The fallback is a kubeadm rebuild. This is slower (~2–3 h) but deterministic.

Keep the snapshot copy on k8s-cp-home-1 — it stays usable.
On every CP node, move every static-pod manifest out of /etc/kubernetes/manifests/ and stop kubelet: systemctl stop kubelet.
Wipe /var/lib/etcd and /etc/kubernetes/pki on every CP node (after backing them up to /root/).
Re-run ansible-playbook ansible/playbooks/init-cluster.yml on the designated init node. This re-bootstraps etcd + apiserver + controller-manager + scheduler with fresh PKI.
Re-run ansible-playbook ansible/playbooks/join-control-planes.yml to join the other three CP nodes.
Re-run ansible-playbook ansible/playbooks/join-nodes.yml for workers.
Re-apply the last-known-good kubeadm ConfigMaps (kubeadm-config, kubelet-config, cluster-info) from the snapshot — extract via etcdctl get /registry/configmaps/kube-system/... against a side-loaded etcd brought up just against the snapshot, or from a Velero dump if one exists.
Restore workloads via ArgoCD by unfreezing the root app-of-apps.
Restore PV data as needed (Longhorn backups, CNPG PITR from MinIO).

Document the incident and the restore path used in STATE.md.

Rejoin a removed member with a stale data-dir¶

Separate flow from the full-restore path above. Use this when one member was deliberately removed via etcdctl member remove (e.g. to stabilise a raft cascade under cross-site network stress) while the remaining members still serve a healthy quorum, and you now want to bring the removed host back. The host's /var/lib/etcd still carries the raft log up to the moment of removal; that log is no longer valid against the surviving cluster because member remove invalidated this host's member ID.

Expected recovery window: 15–30 min end-to-end. No apiserver downtime cluster-wide; only the rejoining host's static pods are touched.

Active example as of 2026-04-20: k8s-cp-office-1 is the removed member, tracked by ADR-0006. The steps below parameterise on REJOIN_NAME + REJOIN_PEER_URL; the office-1 values are noted inline where relevant.

R0. Preflight¶

Quorum is healthy, and the rejoin is not being used to recover from a quorum loss — that's the full-restore path above. Run on any surviving CP:

etcdctl ... endpoint status --cluster --write-out=table
# Expect: 3 members started, one leader, revisions aligned.

The removed member's host is reachable (kubelet Ready, SSH works) and you have root on it. For office-1 via PVE: use the PVE host's qm guest exec if SSH is broken — see the session memory entry "Use PVE + qm guest exec for unreachable nodes" for the command shape.
Static-pod manifests are where ADR-0006 left them. Per that ADR, the quiesced manifests live under /root/etcd.yaml.quiesced-2026-04-19 and /root/kube-apiserver.yaml.quiesced-2026-04-19 on the rejoining host. Confirm with ls /root/*.quiesced-*.yaml before wiping anything.
Ansible state matches intent. If you're about to rejoin office-1, you must also plan to remove its entry from cp_quiesced_nodes in ansible/inventory/group_vars/all.yml and, for the full pre-incident topology, revert apiserver_etcd_servers_override for k8s-cp-office-2 — do that as a separate PR after the rejoin has been stable for the soak window defined in ADR-0006 step 1. See the ADR's "Rejoin plan" for the sequencing contract.
Alertmanager noise. Silence etcd* + kube-apiserver* routes for the rejoining host for ~30 min so the expected crash-loop during member startup doesn't page.

R1. Wipe the stale data-dir on the rejoining host¶

On REJOIN_NAME (example: k8s-cp-office-1):

# Keep the old dir around in case the rejoin fails and you need forensics.
mv /var/lib/etcd /var/lib/etcd.bak-$(date +%s)
mkdir -p /var/lib/etcd
chown root:root /var/lib/etcd
chmod 700 /var/lib/etcd

Confirm no etcd container is running (expected — the manifest is quiesced):

crictl ps --name etcd -q | wc -l
# Expect: 0

R2. Announce the new member from a healthy CP¶

Pick any healthy CP (the leader is conventional but not required). For the office-1 case, home-1 at 10.1.20.200 is the leader per ADR-0006.

# Values below come from ADR-0006 for the office-1 example.
REJOIN_NAME=k8s-cp-office-1
REJOIN_PEER_URL=https://10.2.20.200:2380

ETCDCTL_API=3 etcdctl \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  --endpoints=https://10.1.20.200:2379 \
  member add "$REJOIN_NAME" --peer-urls="$REJOIN_PEER_URL"

etcdctl member add prints a block of ETCD_* environment variables. Capture them — the ETCD_INITIAL_CLUSTER string is the authoritative input for the rejoining host's static pod in the next step. The leader now shows the new member in learner or unstarted state:

etcdctl ... member list --write-out=table
# Expect: 4 rows; the new row has STARTED=false.

R3. Patch the quiesced etcd manifest on the rejoining host¶

Edit /root/etcd.yaml.quiesced-2026-04-19 on REJOIN_NAME. The kubeadm template already carries the correct --name, --cert-file, data-dir, and peer/client URLs; only three flags change vs. the quiesced copy:

--initial-cluster=... — set to the string printed by etcdctl member add in R2. For office-1 that lists all 4 members (home-1,home-2,office-1,office-2).
--initial-cluster-state=existing (was new in the original kubeadm bootstrap; must be existing for a member joining an established cluster).
--initial-advertise-peer-urls=<REJOIN_PEER_URL> (should already be correct from kubeadm; verify).

Sanity-check the edited file — kubeadm-generated etcd manifests are opinionated and broken flag values silently crash-loop the pod:

grep -E 'initial-cluster|initial-advertise-peer-urls' \
  /root/etcd.yaml.quiesced-2026-04-19

R4. Reactivate the etcd static pod¶

Move the patched manifest back into the kubelet watch path:

mv /root/etcd.yaml.quiesced-2026-04-19 /etc/kubernetes/manifests/etcd.yaml

Kubelet picks up the file within ~15 s. Watch the new member come up:

# On the rejoining host:
crictl ps --name etcd
# Expect: etcd container running within 30 s.

# On a healthy CP:
etcdctl ... member list --write-out=table
# Expect: 4 rows, all STARTED=true, within 60 s of the manifest move.

etcdctl ... endpoint health --cluster
# Expect: all 4 endpoints healthy.

etcdctl ... endpoint status --cluster --write-out=table
# Expect: revisions aligned across all 4 members. A brief raftIndex lag
# (< 1000 entries) is normal while the new member replays from the
# leader's snapshot; it should close inside the soak window.

If the new member fails to start (crash-loops with error validating peerURLs ... unmatched member while checking PeerURLs or similar), the most common causes are:

--initial-cluster has a typo vs. the member add output. Re-copy.
/var/lib/etcd wasn't actually empty. Re-wipe and retry.
The new member's peer cert SAN doesn't cover REJOIN_PEER_URL's hostname or IP. Regenerate with kubeadm init phase certs etcd-peer --config <kubeadm-config> on the rejoining host and retry.

R5. Reactivate the kube-apiserver static pod¶

Same pattern for the apiserver manifest, if it was quiesced too (per ADR-0006 both are). Before moving it back, confirm the apiserver's --etcd-servers list matches the intended end-state:

Local-only (pre-ADR-0006 default): --etcd-servers=https://127.0.0.1:2379. Use this when the rejoined member is fully healthy and expected to serve local reads.
Multi-endpoint: --etcd-servers=https://127.0.0.1:2379,https://<other>:2379,.... Use this only if you want endpoint round-robin across sites; remember the lesson from ADR-0006's "Alternatives considered" — a single slow endpoint in the list stalls the apiserver's startup watch-cache fill.

For office-1 rejoin: the pre-incident manifest had 127.0.0.1:2379 only; restore it as-is (the ADR's "Rejoin plan" step 6).

mv /root/kube-apiserver.yaml.quiesced-2026-04-19 \
   /etc/kubernetes/manifests/kube-apiserver.yaml

Watch the pod come up:

crictl ps --name kube-apiserver
# Expect: running within 30 s.

curl -sk https://<REJOIN_HOST_IP>:6443/healthz
# Expect: ok, after ~60 s (service-ip-repair-controllers hook window).

R6. Un-quiesce in Ansible, reset the site apiserver list¶

After the rejoined member has been stable for ≥ 1 h under representative cross-site load (watch for ReadIndex response took too long or connection reset on the raft stream between sites):

Remove the rejoined host from cp_quiesced_nodes in ansible/inventory/group_vars/all.yml.
If the sibling site's apiserver was pointed at a restricted endpoint list during the interim (ADR-0006 apiserver_etcd_servers_override for office-2), plan a revert to the pre-incident list. Do this via the ansible/playbooks/control-plane/ role, not by live-editing the static manifest on the host, so the Ansible state matches what the cluster is actually running.
Flip the ADR status from proposed to accepted (or superseded-by if a newer ADR now captures the permanent architecture).

Do steps 1–2 as a single PR; ArgoCD is not in the loop here (this is static-pod land), but having the PR as the forcing function keeps the Git history and STATE.md aligned with what's live.

R7. Drop the Alertmanager silence + notify¶

Lift the silence from R0 step 5. Post-rejoin monitoring should be quiet: any etcd_server_proposals_failed_total increments beyond the first minute indicate the rejoined member is still unstable and the soak window should be extended before step R6.

Tracking¶

Owner: @lusoris (the only human operator).
First drill target: once the snapshot CronJob is live (ADR-0005 T0), run a scheduled dry-run restore against a disposable VM pair to validate this runbook's step-by-step against the real artifact.
Rejoin-path drill: BUG-14 closure will produce the first real execution of "Rejoin a removed member with a stale data-dir" against k8s-cp-office-1. Treat the first run as a drill — walk through each step with the ADR-0006 "Rejoin plan" section open side-by-side, not in parallel.