Skip to content

0007. Ubuntu 24.04 LTS → 26.04 LTS migration

  • Status: accepted
  • Date: 2026-05-22
  • Deciders: lusoris
  • Related: ADR-0005 (control-plane hardening), ADR-0006 (etcd interim). Supersedes the partial 26.04 metadata changes already in packer/.

Context

All ten cluster VMs run Ubuntu 24.04.4 LTS (Noble) on kernel 6.17.0-29-generic (HWE), containerd 2.2.x, k8s v1.35.3 (kubectl get nodes, 2026-05-22).

Ubuntu 26.04 LTS ("Resolute") released 2026-04-23. Standard LTS support window is 5 years. The repo's Packer template (packer/ubuntu-cloud.pkr.hcl + variables.pkr.hcl) was partially edited in a previous session to target Resolute:

  • vm_name = "ubuntu-2604-cloud" ✓
  • iso_url = "https://cloud-images.ubuntu.com/resolute/current/resolute-server-cloudimg-amd64.img" ✓
  • Header comment says "Ubuntu 26.04 LTS (resolute)" ✓
  • But: template_description still says "Ubuntu 24.04 cloud image" (line 42 — stale).
  • And: none of the live nodes were rebuilt yet.

There's nothing risky-but-tested in the migration; the gap is the per-node rotation and a clean package allow-list.

Decision

Complete the migration in two phases:

  1. Build the 26.04 template on each PVE host (pve-home-{01,02}, pve-office-{01,02}), so cloning is local to each host.
  2. Rotate one node at a time — destroy old VM, clone new template, re-init via Ansible. Order minimises cluster impact:
  3. First: a single home worker (cheapest, least cross-site).
  4. Then: alternate worker per zone until all workers are on 26.04.
  5. Then: control planes one at a time (etcd quorum must hold; BUG-14 means office cp's still fragile).

No in-place dist-upgrade — Ubuntu's do-release-upgrade from 24.04 → 26.04 is supported but adds debugging surface area. Greenfield clone is faster + cleaner.

Exact package set

The template builds from cloud-images.ubuntu.com/resolute/current/resolute-server-cloudimg-amd64.img (minimal cloud image, ~250 MB compressed). Provisioner installs only:

qemu-guest-agent          # PVE host integration: VM info, IP reporting, shutdown
ca-certificates           # required for any HTTPS apt/curl fetch
curl                      # used by the install-* shell snippets in roles/
gnupg                     # apt keyring management (kubernetes repo, etc.)
lsb-release               # codename queries (used by repo-add helpers)
linux-generic             # mainline 7.x kernel package for Resolute
unattended-upgrades       # security-only auto-patching (see policy below)

Nothing else gets installed at build time. The Ansible kubernetes + containerd roles install kubeadm/kubelet/kubectl + containerd.io on first prepare-nodes.yml run.

Purges (kept from the existing template, no additions)

The current template already strips these — keep as-is:

snapd                     # ~500 MB, kubernetes nodes don't run snaps
lxd-installer             # pulls snapd
ubuntu-pro-client         # subscription / advertising surface
landscape-common          # Canonical telemetry
popularity-contest        # apt-popcon
motd-news-config          # MOTD network calls
plymouth                  # graphical boot splash (never visible on a server)
plymouth-theme-ubuntu-text

Plus the dpkg path-excludes (kept):

path-exclude=/usr/share/doc/*
path-exclude=/usr/share/man/*
path-exclude=/usr/share/info/*
path-exclude=/usr/share/locale/*
path-include=/usr/share/locale/en*

Plus the systemd timer disables (kept):

snapd.service / snapd.socket / snapd.seeded.service
apt-news.service / esm-cache.service / motd-news.timer

Kernel

Resolute ships mainline kernel 7.x by default (per Ubuntu release notes). Drop the -hwe-24.04 HWE suffix — the existing Packer template already does this (linux-generic). No backport packages.

containerd

Resolute's main repository ships containerd 2.4.x. Verify on first template build:

apt-cache madison containerd | head -5

The Ansible containerd role currently installs from the distro repo (no docker.com / docker-ce repo override). Resolute's 2.4.x is compatible with CRI v1 and the cluster's containerd://2.2.x runtime endpoint — no config changes required in containerd/files/.

If Resolute ends up shipping containerd 2.5+ that drops runc 1.x support, switch to docker.com's containerd.io repo and pin to a 2.4.x build. Track on first template build.

The current ansible/inventory/group_vars/all.yml value containerd_version: "2.2.*" (just bumped from 1.7.* in commit 512e06c3) needs to be re-evaluated against Resolute's repo on first template build — likely "2.4.*" or removed entirely (the var isn't currently referenced by any task; it's documentation drift).

Cloud-init datasources

Keep the existing datasource_list: [NoCloud, ConfigDrive, None] in /etc/cloud/cloud.cfg.d/99_datasource.cfg. Resolute's cloud-init is 24.x which supports both.

Auto-upgrade policy

Keep the existing unattended-upgrades blacklist verbatim:

linux-image-*       # kernel — Ansible drives reboots
linux-headers-*
containerd*         # runtime — pinned, upgraded explicitly
kubelet
kubeadm
kubectl

Only ${distro_id}:${distro_codename}-security repo gets unattended-upgraded. The -updates line is enabled per the current template.

sysctls + kernel modules (kept)

/etc/sysctl.d/99-k8s.conf:

net.bridge.bridge-nf-call-iptables=1
net.bridge.bridge-nf-call-ip6tables=1
net.ipv4.ip_forward=1
net.ipv6.conf.all.forwarding=1
fs.inotify.max_user_instances=8192
fs.inotify.max_user_watches=524288

/etc/modules-load.d/k8s.conf:

overlay
br_netfilter

Swap disabled via /etc/fstab strip + systemctl mask swap.target.

Rollout plan (operator runbook)

Phase 0 — template per PVE host

For each of pve-home-01, pve-home-02, pve-office-01, pve-office-02:

  1. Download the Resolute cloud image to the PVE host:
wget -O /var/lib/vz/template/cache/resolute-server-cloudimg-amd64.img \
  https://cloud-images.ubuntu.com/resolute/current/resolute-server-cloudimg-amd64.img
  1. Bootstrap a base VM (one-time):
qm create 9000 --name ubuntu-2604-base --memory 2048 --net0 virtio,bridge=vmbr0
qm importdisk 9000 /var/lib/vz/template/cache/resolute-server-cloudimg-amd64.img local-lvm
qm set 9000 --scsihw virtio-scsi-pci --scsi0 local-lvm:vm-9000-disk-0 --ide2 local-lvm:cloudinit
qm set 9000 --boot c --bootdisk scsi0 --serial0 socket --agent enabled=1
qm template 9000
  1. Run Packer to produce the maintained template:
cd packer/
packer build -var-file=$(hostname).pkrvars.hcl ubuntu-cloud.pkr.hcl
  1. Confirm qm list | grep ubuntu-2604-cloud shows the new template on each host.

Phase 1 — fix the stale template_description

One-line fix in packer/ubuntu-cloud.pkr.hcl:42: change "Ubuntu 24.04 cloud image - …" → "Ubuntu 26.04 cloud image - …". Commit alongside this ADR.

Phase 2 — rotate one worker (canary)

Pick k8s-worker-home-2 (newest home worker, least state attached). For each candidate node:

  1. Drain the node:
kubectl drain k8s-worker-home-2 \
  --ignore-daemonsets --delete-emptydir-data --force --timeout=15m

Watch ArgoCD for any "Degraded" — workloads should reschedule to peer node within ~2 min. 2. Remove from the cluster (don't reuse the cordoned object):

kubectl delete node k8s-worker-home-2
  1. Destroy the VM on PVE:
ssh root@pve-home-02 "qm stop <vmid> && qm destroy <vmid>"
  1. Clone from the new template (use Terraform if cluster/terraform/proxmox-vm-* is wired; otherwise qm clone):
ssh root@pve-home-02 "qm clone <template-vmid> <new-vmid> --name k8s-worker-home-2 --full"
# set network MAC, disk size, vlan tag etc. to match the old VM
qm set <new-vmid> --ipconfig0 ip=10.1.20.101/24,gw=10.1.20.1
qm start <new-vmid>
  1. Prepare + join via Ansible:
ansible-playbook -i inventory prepare-nodes.yml -l k8s-worker-home-2
ansible-playbook -i inventory join-nodes.yml -l k8s-worker-home-2
  1. Verify: kubectl get nodes k8s-worker-home-2 -o wide shows Resolute + kernel 7.x + containerd 2.4.x + Ready.
  2. Bake-in window: leave the cluster running on a mixed-OS state for 24 h. Watch kubectl top nodes, kubectl get events -A --field-selector type=Warning, and ArgoCD app health.

Phase 3 — roll the other workers

Same routine, alternating zones: k8s-worker-office-3, k8s-worker-home-1, k8s-worker-office-2, k8s-worker-office-1 (last because of BUG-15 fragility per the office-1 memory).

Phase 4 — roll control planes

Order: home-1, office-2, home-2, office-1.

For each:

  1. Drain (kubectl drain + same flags).
  2. On a CP: ALSO kubectl get pods -n kube-system -l component=etcd and confirm etcd quorum is healthy on the remaining 3 members BEFORE destroying.
  3. Destroy + clone + IP + start (as Phase 2).
  4. ansible-playbook prepare-nodes.yml -l <cp-name>
  5. ansible-playbook join-control-planes.yml -l <cp-name>
  6. Verify kubectl get nodes shows control-plane + Ready.
  7. kubectl exec -n kube-system etcd-<name> -- etcdctl endpoint health --cluster — all members healthy.

Phase 5 — clean up

  • Delete the 24.04 template VMs from each PVE host.
  • Verify kubectl get nodes -o wide shows ALL nodes on Ubuntu 26.04 LTS.
  • Update ansible/inventory/group_vars/all.yml containerd_version to whatever Resolute ships ("2.4.*" likely).
  • Append to STATE.md with the rotation completion timestamps.
  • Consider squashing the dual-template scripts/setup/linux-apt-mise.sh if any 24.04-specific blocks remain (likely none).

Risks

# Risk Mitigation
R1 Resolute ships containerd / runc that breaks kubelet 1.35.x Pin containerd via docker.com repo if needed; cluster's kubelet runs CRI v1 which is stable. Test on canary worker first.
R2 Kernel 7.x has different eBPF behavior → Cilium agent regression Cilium 1.19.4 (just rolled) supports kernels through 7.x per upstream test matrix. Canary worker catches this in Phase 2.
R3 etcd quorum drops below 3 during CP roll Strict one-at-a-time, etcd health check between each.
R4 BUG-15 office-1 fragility worsens on rebuild Roll office-1 last. If anything goes wrong, restore from PVE snapshot.
R5 Cloud-init datasource changes between 24.04 (cloud-init 24.x) and 26.04 (cloud-init 26.x) Datasource list explicitly set; both versions support NoCloud + ConfigDrive. Verify on canary.
R6 Longhorn replicas pinned to specific node → rebuild loses replica Longhorn auto-recreates replicas on peer nodes; volume health stays Green during single-node loss.

Rollback

PVE keeps the old VM's disk in local-lvm until qm destroy runs. Don't destroy the old VM until Phase 2 verify is green. If verify fails:

  1. qm stop <new-vmid> && qm destroy <new-vmid>
  2. qm start <old-vmid> (still there)
  3. kubectl uncordon <node> (it'll re-join since kubeconfig and CRI are unchanged)

Total rollback time: ~3 min per node.

Open questions / decisions deferred

  • Should the rotation also bump k8s 1.35.x → 1.36.x in the same window? Probably no — pair each migration with a single variable change. Decide separately via a follow-up ADR or just bump k8s patch (already pinned to 1.35.5 in upgrade-k8s-patch.yml) before rotation.
  • Should we use Terraform to manage the cloned VMs? The repo has terraform/modules/proxmox-vm partially wired. Worth confirming whether it's the canonical cloning path before Phase 2 starts. If yes, the rollout becomes terraform apply per node instead of qm clone.
  • Should linux-image-* join the unattended-upgrade allow-list once we're on 26.04? Default is to keep it blacklisted (Ansible drives reboots). Leave the policy as-is unless we adopt kured or another reboot operator on 26.04.

References

  • Ubuntu 26.04 LTS release notes (2026-04-23 GA).
  • packer/ubuntu-cloud.pkr.hcl — partially-migrated template (this ADR ratifies + completes).
  • ansible/playbooks/upgrade-k8s-patch.yml — sibling pattern for kubeadm patch upgrades (added in commit 5b801b4e).
  • ADR-0005 — control-plane hardening (relevant for Phase 4 ordering).
  • ADR-0006 — office-cp etcd interim (informs office-1 last-in-rotation).
  • BUG-15 — office-1 default-route flap (per .workingdir2/bugs/08-known-bugs.md / memory project_office1_unreachable_from_home); office-1 rotation should include a netplan sanity check post-boot.