Skip to content

Kubernetes 1.35.3 → 1.36.2 upgrade runbook

Target: every node (4 CPs + 5 workers) live-upgrades via kubeadm without VM rebuild. OS migration (24.04 → 26.04) is separate (see ADR-0007).

Historical baseline: all nodes started on v1.35.3 (kubelet + apiserver). Current target is v1.36.2.

Pre-flight (do once)

  1. Skim the upstream 1.36 release notes for deprecated APIs hit by any CRD/chart we run. The cluster's worth checking:
  2. policies.kyverno.io/v1alpha1 (ValidatingPolicy) — confirm GA path
  3. gateway.networking.k8s.io/v1 — stable
  4. autoscaling.k8s.io/v1 (VPA) — out-of-tree, unaffected
  5. Take an etcd snapshot before touching anything:
kubectl exec -n kube-system etcd-k8s-cp-home-1 -- etcdctl \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot save /var/lib/etcd-backups/pre-1.36-upgrade-$(date +%F).db
  1. Confirm group_vars/all.yml is at kubernetes_version: "1.36", kubernetes_patch_version: "1.36.2", and kubernetes_package_revision: "2.1".

Rotation order

kubeadm allows kubelet within ±1 minor of apiserver. Apiserver goes first on the first CP, then the rest of the CPs, then workers.

1. k8s-cp-office-1    (kubeadm upgrade apply, first_cp=true)
2. k8s-cp-office-2    (kubeadm upgrade node)
3. k8s-cp-home-1      (kubeadm upgrade node)
4. k8s-cp-home-2      (kubeadm upgrade node)
5. k8s-worker-home-1  (kubeadm upgrade node)
6. k8s-worker-home-2  (kubeadm upgrade node)
7. k8s-worker-office-1
8. k8s-worker-office-2
9. k8s-worker-home-3

Office CPs first because of BUG-14 (their etcd member is more fragile; upgrade them while quorum is healthiest).

Per-node commands

For step 1 (first CP only):

ansible-playbook -i ansible/inventory \
  ansible/playbooks/upgrade-k8s-minor.yml \
  -l k8s-cp-office-1 \
  -e first_cp=true

For steps 2–9 (each subsequent node):

ansible-playbook -i ansible/inventory \
  ansible/playbooks/upgrade-k8s-minor.yml \
  -l <node-name>

Per-node verification (do not skip)

After each node finishes:

kubectl get nodes -o wide                # node Ready, kubelet at v1.36.2
kubectl get pods -A | grep -v Running    # no orphaned terminating pods
kubectl logs -n kube-system etcd-k8s-cp-home-1 --tail=5  # CP nodes only

If anything looks wrong, stop. Subsequent kubelets need an apiserver no more than one minor behind — but a CP that fails the upgrade can be rebooted from the etcd snapshot taken in pre-flight.

After all 9 nodes done

kubectl version                          # both Client + Server v1.36.2
kubectl get nodes                        # all Ready, all at v1.36.2
kubectl get componentstatuses            # cs healthy (deprecated but informative)

Run a smoke pass on the critical paths: - ArgoCD reconciles a synthetic Application - A Renovate PR rebases against fresh main - Cilium-Envoy idle_timeout fix (cluster/authentik/outpost-upstream-idle.yaml) still effective — qbit UI responsive

Rollback (if needed)

kubeadm upgrade plan --dry-run doesn't auto-roll. Per-node rollback path:

  1. apt-mark unhold kubeadm → apt install kubeadm=1.35.5-1.1 → apt-mark hold kubeadm.
  2. Same for kubelet + kubectl.
  3. systemctl restart kubelet.
  4. Confirm the node rejoins as v1.35.5.

Don't try to mix minors across the whole cluster. If 1.36 introduces a blocking regression, roll back all 9 to 1.35.5 (the safe interim patch) and revisit.

After all nodes are at 1.36.2

Update STATE.md with the cutover date + the etcd snapshot path. Issue still tracking the 26.04 OS rebuild (separate from this work).