Skip to content

Home worker 50 GiB root-only replacement

This runbook replaces only scsi0 on the two live home worker VMs with a fresh Ubuntu 26.04 root. It is not an in-place filesystem shrink and it is not a whole-VM rotation.

Current order:

  1. VM 9003 / k8s-worker-home-2
  2. VM 9004 / k8s-worker-home-3, only after 9003 fully converges

The active Longhorn v2 device is scsi1 and must remain byte-for-byte unchanged. VM 9003 also has an active 30 GiB scsi2 ext4 device mounted at /var/lib/prometheus-local; preserve that disk byte-for-byte too. Each fresh root additionally needs a new thin-provisioned local-lvm containerd disk:

  • VM 9003: new 100 GiB scsi3;
  • VM 9004: new 200 GiB scsi2.

The dedicated runtime disks are required, not an optional optimization. Measured containerd data is about 23.5 GB on home-2 and 204.5 GB on home-3; home-3's active snapshot closure alone is about 44.0 GB. Putting that runtime state back on a 50 GiB OS filesystem would make the rebuild invalid before normal root usage and image growth. Do not run Terraform, destroy either VM, cordon/drain either node, or run ansible/playbooks/drain-and-prep-rotation.yml for this operation.

Required gate

Before staging the first root:

  • prove there are zero v1 Longhorn scheduledReplica entries on both home workers;
  • run the default report-only ansible/playbooks/retire-home-longhorn-v1-disks.yml preflight, then use its explicit mode=apply path to disable and remove the filesystem-backed Longhorn disk from both Node CRs;
  • verify both active scsi1 devices are healthy v2 block disks of exactly 1 TiB;
  • prove no local PV, hostPath workload, or undeclared host data still depends on the old root; VM 9003's prometheus-local-home-2 local PV deliberately depends on scsi2, not the root, so the complete scsi2 line is a required preservation gate;
  • capture VM 9003 scsi2 as the exact 30 GiB ext4 filesystem with label prometheus-local and UUID b66a7758-8337-4f75-b867-22ad17670ad6; the replacement root must reconstruct its /var/lib/prometheus-local UUID mount and must never format it;
  • capture qm config, qm pending, boot order, EFI/cloud-init disks, NIC/MAC/VLAN, CPU/RAM, hostpci, and every active disk line;
  • capture the Kubernetes Node's current .spec.unschedulable value; this root-only flow does not cordon or uncordon and must preserve any pre-existing operator cordon;
  • prove VM 9003 scsi3 and VM 9004 scsi2 are unused, and prove local-lvm has enough free thin-pool capacity for the new 100/200 GiB runtime disks;
  • verify the live OVMF-compatible template VM 9102, ubuntu-2604-uefi; do not substitute the legacy BIOS cloud template for an existing UEFI worker;
  • confirm the template root is no larger than 50 GiB;
  • append the final preflight to STATE.md, run the repo's narrow gate, and obtain explicit operator confirmation for the destructive PVE boundary.

Run the Longhorn disk preflight through the healthy office API endpoint. The default invocation performs only Kubernetes reads and local JSON validation:

ANSIBLE_CONFIG=ansible/ansible.cfg ansible-playbook \
  -i localhost, \
  ansible/playbooks/retire-home-longhorn-v1-disks.yml \
  -e retirement_kube_api_server=https://10.2.20.200:6443

It requires exactly one /var/lib/longhorn/ filesystem disk and one preserved block disk on each worker, matching spec/status keys, no scheduled storage or objects that reference either filesystem disk, no zone-home v1 volume, and a Ready v2 disk. The analyzer hard-gates the captured live identities instead of accepting any otherwise-valid replacement object:

  • home-2 Kubernetes Node UID efc1e3d0-f9b8-48f2-ae25-82fbc24fbbaa, Longhorn Node UID d0514221-1ec8-4fd3-a8d6-b9c9767796df, v2 disk UUID 4a0ff0b7-c131-4f26-bdf8-f683f3d273b2;
  • home-3 Kubernetes Node UID e1386ebd-775f-46c6-8494-1aacccc36810, Longhorn Node UID ab4110c2-d1b3-412d-b31e-898d24910fbe, v2 disk UUID 9022e914-f15b-4656-9f23-1cd0a0554263;
  • both v2 paths /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi1.

Only after this report passes, run the same command with the explicit live mode:

ANSIBLE_CONFIG=ansible/ansible.cfg ansible-playbook \
  -i localhost, \
  ansible/playbooks/retire-home-longhorn-v1-disks.yml \
  -e retirement_kube_api_server=https://10.2.20.200:6443 \
  -e home_longhorn_v1_disk_retirement_mode=apply

Apply mode disables both filesystem disks before removing either, waits for each exact allowScheduling spec flag to become false, repeats all gates from fresh API reads, and uses JSON Patch tests so an identity or spec race aborts the removal. Longhorn's Schedulable condition describes disk capability and can correctly remain True after that operator toggle is false. The final gate proves the Kubernetes and Longhorn Node UIDs and the v2 disk key, path, UUID, spec, and Ready condition did not change. Do not proceed to PVE if the playbook stops after disabling scheduling or fails final verification.

VM 9003's active 30 GiB scsi2 backs /var/lib/prometheus-local; preserve it. VM 9004 already has a separate 700 GiB unused0; preserve that too. The old 800 GiB root will therefore probably become another unused slot; discover the actual slot rather than assuming its number.

Stage a 50 GiB root

Run on the PVE host while the target worker is still online. Select exactly one parameter block; the stage IDs and storage pool are part of the audited plan, not free-form placeholders:

set -euo pipefail

# First worker.
TARGET_VMID=9003
STAGE_VMID=9903
RUNTIME_SLOT=scsi3
RUNTIME_GIB=100
OLD_ROOT_VOL=local-lvm:vm-9003-disk-0

# Second worker only after VM 9003 fully converges; replace the five values:
# TARGET_VMID=9004
# STAGE_VMID=9904
# RUNTIME_SLOT=scsi2
# RUNTIME_GIB=200
# OLD_ROOT_VOL=local-lvm:vm-9004-disk-1

TEMPLATE_VMID=9102
ROOT_POOL=local-lvm
PVE_NODE="$(hostname -s)"
ROTATION_STATE=/root/cauda-vm-root-rotation-${TARGET_VMID}.state

config_digest() {
  pvesh get "/nodes/$PVE_NODE/qemu/$1/config" --output-format json |
    jq -er .digest
}

test "$(qm status "$TEMPLATE_VMID")" = "status: stopped"
TEMPLATE_CONFIG=$(qm config "$TEMPLATE_VMID")
grep -Fx 'name: ubuntu-2604-uefi' <<<"$TEMPLATE_CONFIG"
grep -Fx 'template: 1' <<<"$TEMPLATE_CONFIG"
grep -Fx 'bios: ovmf' <<<"$TEMPLATE_CONFIG"
TEMPLATE_ROOT_VOL=$(sed -n 's/^scsi0: \([^,]*\).*/\1/p' <<<"$TEMPLATE_CONFIG")
test "$(blockdev --getsize64 "$(pvesm path "$TEMPLATE_ROOT_VOL")")" \
  -eq 8589934592
for SNAPSHOT_VMID in "$TEMPLATE_VMID" "$TARGET_VMID"; do
  SNAPSHOT_OUTPUT=$(qm listsnapshot "$SNAPSHOT_VMID")
  test -n "$SNAPSHOT_OUTPUT"
  test "$(printf '%s\n' "$SNAPSHOT_OUTPUT" |
    awk '$2 != "current" { n++ } END { print n + 0 }')" -eq 0
done
if ! PENDING_OUTPUT=$(qm pending "$TARGET_VMID"); then
  echo "failed to read pending config for VM $TARGET_VMID" >&2
  exit 1
fi
test -n "$PENDING_OUTPUT"
test "$(printf '%s\n' "$PENDING_OUTPUT" |
  awk '$1 != "cur" { n++ } END { print n + 0 }')" -eq 0
test ! -e "/etc/pve/qemu-server/$STAGE_VMID.conf"
STAGE_LOCAL_LVM_VOLUMES=$(pvesm list local-lvm --vmid "$STAGE_VMID")
STAGE_PVE_STORAGE_VOLUMES=$(pvesm list pve-storage --vmid "$STAGE_VMID")
test -z "$(printf '%s\n' "$STAGE_LOCAL_LVM_VOLUMES" | tail -n +2)"
test -z "$(printf '%s\n' "$STAGE_PVE_STORAGE_VOLUMES" | tail -n +2)"
TARGET_CONFIG_CURRENT=$(qm config "$TARGET_VMID")
test "$(qm status "$TARGET_VMID")" = "status: running"
grep -Fx 'bios: ovmf' <<<"$TARGET_CONFIG_CURRENT"
grep -Fx 'machine: q35' <<<"$TARGET_CONFIG_CURRENT"
if grep -q "^${RUNTIME_SLOT}:" <<<"$TARGET_CONFIG_CURRENT"; then
  echo "$RUNTIME_SLOT is already occupied on VM $TARGET_VMID" >&2
  exit 1
fi
test "$TARGET_VMID" != 9003 ||
  grep -Fx 'scsi1: local-lvm:vm-9003-disk-2,size=1T' \
    <<<"$TARGET_CONFIG_CURRENT"
test "$TARGET_VMID" != 9003 ||
  grep -Fx 'scsi2: pve-storage:vm-9003-disk-0,discard=on,size=30G,ssd=1' \
    <<<"$TARGET_CONFIG_CURRENT"
test "$TARGET_VMID" != 9004 ||
  grep -Fx 'scsi1: pve-storage:vm-9004-disk-0,discard=on,size=1T,ssd=1' \
    <<<"$TARGET_CONFIG_CURRENT"
test "$TARGET_VMID" != 9004 ||
  grep -Fx 'unused0: local-lvm:vm-9004-disk-2' <<<"$TARGET_CONFIG_CURRENT"

LOCAL_LVM_FREE_KIB=$(pvesm status --storage "$ROOT_POOL" |
  awk -v pool="$ROOT_POOL" '$1 == pool { print $6 }')
test "$LOCAL_LVM_FREE_KIB" -ge \
  "$(((RUNTIME_GIB + 150) * 1024 * 1024))"

TARGET_CONFIG_BEFORE=/root/vm-${TARGET_VMID}-before.conf
qm config "$TARGET_VMID" >"$TARGET_CONFIG_BEFORE"
sha256sum "$TARGET_CONFIG_BEFORE"

OLD_ROOT_LINE=$(sed -n 's/^scsi0: //p' "$TARGET_CONFIG_BEFORE")
test "${OLD_ROOT_LINE%%,*}" = "$OLD_ROOT_VOL"
TARGET_BOOT_LINE=$(sed -n 's/^boot: //p' "$TARGET_CONFIG_BEFORE")
test "$TARGET_BOOT_LINE" = 'order=scsi0'

# The original 700 GiB VM 9004 unused disk is protected rollback material.
test "$TARGET_VMID" != 9004 ||
  grep -Fx 'unused0: local-lvm:vm-9004-disk-2' "$TARGET_CONFIG_BEFORE"

umask 077
{
  printf 'TARGET_VMID=%q\n' "$TARGET_VMID"
  printf 'STAGE_VMID=%q\n' "$STAGE_VMID"
  printf 'RUNTIME_SLOT=%q\n' "$RUNTIME_SLOT"
  printf 'RUNTIME_GIB=%q\n' "$RUNTIME_GIB"
  printf 'OLD_ROOT_VOL=%q\n' "$OLD_ROOT_VOL"
  printf 'TEMPLATE_VMID=%q\n' "$TEMPLATE_VMID"
  printf 'ROOT_POOL=%q\n' "$ROOT_POOL"
  printf 'PVE_NODE=%q\n' "$PVE_NODE"
  printf 'ROTATION_STATE=%q\n' "$ROTATION_STATE"
  printf 'TARGET_CONFIG_BEFORE=%q\n' "$TARGET_CONFIG_BEFORE"
  printf 'OLD_ROOT_LINE=%q\n' "$OLD_ROOT_LINE"
  printf 'TARGET_BOOT_LINE=%q\n' "$TARGET_BOOT_LINE"
  declare -f config_digest
} >"$ROTATION_STATE"

qm clone "$TEMPLATE_VMID" "$STAGE_VMID" \
  --name "root-seed-$TARGET_VMID" \
  --full 1 \
  --storage "$ROOT_POOL"

test "$(qm status "$STAGE_VMID")" = "status: stopped"
STAGE_SNAPSHOT_OUTPUT=$(qm listsnapshot "$STAGE_VMID")
test -n "$STAGE_SNAPSHOT_OUTPUT"
test "$(printf '%s\n' "$STAGE_SNAPSHOT_OUTPUT" |
  awk '$2 != "current" { n++ } END { print n + 0 }')" -eq 0
qm disk resize "$STAGE_VMID" scsi0 50G

STAGE_ROOT_VOL=$(qm config "$STAGE_VMID" |
  sed -n 's/^scsi0: \([^,]*\).*/\1/p')
STAGE_ROOT_PATH=$(pvesm path "$STAGE_ROOT_VOL")
test "$(blockdev --getsize64 "$STAGE_ROOT_PATH")" -eq 53687091200

STAGE_CONFIG_BEFORE=/root/vm-${STAGE_VMID}-before.conf
qm config "$STAGE_VMID" >"$STAGE_CONFIG_BEFORE"
sha256sum "$STAGE_CONFIG_BEFORE"

{
  printf 'STAGE_CONFIG_BEFORE=%q\n' "$STAGE_CONFIG_BEFORE"
} >>"$ROTATION_STATE"

The clone must remain stopped. qm disk resize only enlarges a block device; it cannot shrink an oversized source filesystem. If the template or clone is already larger than 50 GiB, stop and build a correctly sized UEFI source.

Do not create either runtime disk while doing read-only discovery. The following allocation is part of the operator-approved destructive PVE boundary in the next section.

All later blocks use the variables above. If the SSH shell disconnects, set TARGET_VMID=9003 (or 9004) and source /root/cauda-vm-root-rotation-${TARGET_VMID}.state before continuing. Abort if that mode-0600 state file or either captured config is missing; do not recreate values from memory.

Root-only cutover

Refresh the captured configs immediately before shutdown and abort if either VM has changed since the approved preflight. Then shut down the target cleanly:

set -euo pipefail

test "$(qm config "$TARGET_VMID" | sha256sum | cut -d' ' -f1)" = \
  "$(sha256sum "$TARGET_CONFIG_BEFORE" | cut -d' ' -f1)"
test "$(qm config "$STAGE_VMID" | sha256sum | cut -d' ' -f1)" = \
  "$(sha256sum "$STAGE_CONFIG_BEFORE" | cut -d' ' -f1)"

qm shutdown "$TARGET_VMID" --timeout 180
test "$(qm status "$TARGET_VMID")" = "status: stopped"
test "$(qm status "$STAGE_VMID")" = "status: stopped"

If it does not stop, diagnose the guest. Do not automatically force-stop it. Do not add --forceStop, use qm stop, or bypass a lock. Unlink only the old root; without --force, Proxmox retains the volume as an unused rollback disk:

ACTIVE_ROOT_LINE=$(qm config "$TARGET_VMID" | sed -n 's/^scsi0: //p')
if [[ "${ACTIVE_ROOT_LINE%%,*}" != "$OLD_ROOT_VOL" ]]; then
  echo "scsi0 is no longer the captured old root; use rollback/recovery" >&2
  exit 1
fi

TARGET_DIGEST=$(config_digest "$TARGET_VMID")
qm set "$TARGET_VMID" --digest "$TARGET_DIGEST" --delete scsi0
TARGET_CONFIG_WITHOUT_ROOT=$(qm config "$TARGET_VMID")
if grep -q '^scsi0:' <<<"$TARGET_CONFIG_WITHOUT_ROOT"; then
  echo "scsi0 remained attached after unlink" >&2
  exit 1
fi

OLD_UNUSED_KEY=$(
  printf '%s\n' "$TARGET_CONFIG_WITHOUT_ROOT" |
    awk -F': ' -v vol="$OLD_ROOT_VOL" \
      '$1 ~ /^unused[0-9]+$/ && $2 == vol { print $1 }'
)
test -n "$OLD_UNUSED_KEY"
printf 'OLD_UNUSED_KEY=%q\n' "$OLD_UNUSED_KEY" >>"$ROTATION_STATE"

test "$TARGET_VMID" != 9004 ||
  qm config 9004 | grep -Fx 'unused0: local-lvm:vm-9004-disk-2'

test "$(qm status "$STAGE_VMID")" = "status: stopped"
test "$(qm status "$TARGET_VMID")" = "status: stopped"
STAGE_ROOT_LINE=$(sed -n 's/^scsi0: //p' "$STAGE_CONFIG_BEFORE")
CURRENT_STAGE_ROOT_LINE=$(qm config "$STAGE_VMID" | sed -n 's/^scsi0: //p')
if [[ -z "$STAGE_ROOT_LINE" || "$CURRENT_STAGE_ROOT_LINE" != "$STAGE_ROOT_LINE" ]]; then
  echo "staged scsi0 changed after capture; refusing the cross-VM move" >&2
  exit 1
fi
SOURCE_DIGEST=$(config_digest "$STAGE_VMID")
TARGET_DIGEST=$(config_digest "$TARGET_VMID")

if ! qm disk move "$STAGE_VMID" scsi0 \
  --target-vmid "$TARGET_VMID" \
  --target-disk scsi0 \
  --digest "$SOURCE_DIGEST" \
  --target-digest "$TARGET_DIGEST"; then
  echo "cross-VM root move failed; inspect for a renamed unreferenced volume" >&2
  qm disk rescan --dryrun 1 --vmid "$TARGET_VMID" >&2 || true
  pvesm list "$ROOT_POOL" --vmid "$TARGET_VMID" >&2 || true
  exit 1
fi

# Removing the old root also removed scsi0 from the target boot order. PVE
# does not add a non-CD drive back automatically, so restore the exact capture.
TARGET_DIGEST=$(config_digest "$TARGET_VMID")
qm set "$TARGET_VMID" \
  --digest "$TARGET_DIGEST" \
  --boot "$TARGET_BOOT_LINE"

STAGE_CONFIG_AFTER_MOVE=$(qm config "$STAGE_VMID")
if grep -q '^scsi0:' <<<"$STAGE_CONFIG_AFTER_MOVE"; then
  echo "staged scsi0 remained attached after move" >&2
  exit 1
fi

NEW_ROOT_VOL=$(qm config "$TARGET_VMID" |
  sed -n 's/^scsi0: \([^,]*\).*/\1/p')
test "$NEW_ROOT_VOL" != "$OLD_ROOT_VOL"
test "$(blockdev --getsize64 "$(pvesm path "$NEW_ROOT_VOL")")" \
  -eq 53687091200

{
  printf 'NEW_ROOT_VOL=%q\n' "$NEW_ROOT_VOL"
} >>"$ROTATION_STATE"

The installed backend removes the moved scsi0 reference from the stage VM's boot order. It also removes an unlinked target scsi0 from the target boot order and does not restore non-CD drives when they are added or reassigned. The command above therefore restores the target's exact captured boot: order=scsi0. Apart from scsi0 and the source boot-order reference, the stopped stage config must remain unchanged. Do not add --storage or --delete to the cross-VM move.

Only after the new root verifies, allocate the current VM's reconstructable containerd cache disk. The target remains stopped and the runtime slot must still be absent:

TARGET_CONFIG_BEFORE_RUNTIME=$(qm config "$TARGET_VMID")
if grep -q "^${RUNTIME_SLOT}:" <<<"$TARGET_CONFIG_BEFORE_RUNTIME"; then
  echo "$RUNTIME_SLOT became occupied before runtime allocation" >&2
  exit 1
fi
TARGET_DIGEST=$(config_digest "$TARGET_VMID")

qm set "$TARGET_VMID" \
  --digest "$TARGET_DIGEST" \
  "--$RUNTIME_SLOT" \
  "local-lvm:${RUNTIME_GIB},backup=0,discard=on,ssd=1"

RUNTIME_VOL=$(qm config "$TARGET_VMID" |
  sed -n "s/^${RUNTIME_SLOT}: \\([^,]*\\).*/\\1/p")
RUNTIME_PATH=$(pvesm path "$RUNTIME_VOL")
RUNTIME_BYTES=$((RUNTIME_GIB * 1024 * 1024 * 1024))

test "$(blockdev --getsize64 "$RUNTIME_PATH")" -eq "$RUNTIME_BYTES"
RUNTIME_SIGNATURES=$(wipefs --no-act --noheadings --output TYPE "$RUNTIME_PATH")
test -z "$RUNTIME_SIGNATURES"

{
  printf 'RUNTIME_VOL=%q\n' "$RUNTIME_VOL"
  printf 'RUNTIME_PATH=%q\n' "$RUNTIME_PATH"
  printf 'RUNTIME_BYTES=%q\n' "$RUNTIME_BYTES"
} >>"$ROTATION_STATE"

TARGET_CONFIG_AFTER=/root/vm-${TARGET_VMID}-after-cutover.conf
STAGE_CONFIG_AFTER=/root/vm-${STAGE_VMID}-after-move.conf
qm config "$TARGET_VMID" >"$TARGET_CONFIG_AFTER"
qm config "$STAGE_VMID" >"$STAGE_CONFIG_AFTER"
{
  printf 'TARGET_CONFIG_AFTER=%q\n' "$TARGET_CONFIG_AFTER"
  printf 'STAGE_CONFIG_AFTER=%q\n' "$STAGE_CONFIG_AFTER"
} >>"$ROTATION_STATE"

# Remove only the three approved target deltas, then require every other line
# (EFI/cloud-init, boot, NIC, CPU/RAM, hostpci, SMBIOS, vmgenid, and preserved
# disks/unused entries) to remain byte-identical.
awk -F': ' -v runtime="$RUNTIME_SLOT" \
  '$1 == "scsi0" { next } { print }' \
  "$TARGET_CONFIG_BEFORE" >/root/target-before.normalized
awk -F': ' -v runtime="$RUNTIME_SLOT" -v oldkey="$OLD_UNUSED_KEY" \
  -v oldvol="$OLD_ROOT_VOL" \
  '$1 == "scsi0" || $1 == runtime || ($1 == oldkey && $2 == oldvol) { next }
   { print }' \
  "$TARGET_CONFIG_AFTER" >/root/target-after.normalized
diff -u /root/target-before.normalized /root/target-after.normalized

# The source loses only its moved root and the corresponding boot reference.
awk -F': ' '$1 != "scsi0" && $1 != "boot" { print }' \
  "$STAGE_CONFIG_BEFORE" >/root/stage-before.normalized
awk -F': ' '$1 != "scsi0" && $1 != "boot" { print }' \
  "$STAGE_CONFIG_AFTER" >/root/stage-after.normalized
diff -u /root/stage-before.normalized /root/stage-after.normalized

grep -Fx "${OLD_UNUSED_KEY}: ${OLD_ROOT_VOL}" "$TARGET_CONFIG_AFTER"

test "$TARGET_VMID" != 9003 ||
  grep -Fx 'scsi1: local-lvm:vm-9003-disk-2,size=1T' "$TARGET_CONFIG_AFTER"
test "$TARGET_VMID" != 9003 ||
  grep -Fx 'scsi2: pve-storage:vm-9003-disk-0,discard=on,size=30G,ssd=1' \
    "$TARGET_CONFIG_AFTER"
test "$TARGET_VMID" != 9004 ||
  grep -Fx 'scsi1: pve-storage:vm-9004-disk-0,discard=on,size=1T,ssd=1' \
    "$TARGET_CONFIG_AFTER"
test "$TARGET_VMID" != 9004 ||
  grep -Fx 'unused0: local-lvm:vm-9004-disk-2' "$TARGET_CONFIG_AFTER"

Do not add iothread=1: both workers use virtio-scsi-pci, while per-disk I/O threads require virtio-scsi-single. Changing the controller is outside this operation. backup=0 is deliberate because this disk contains reconstructable container image/cache data.

Before starting, compare mechanically with the saved config:

  • scsi0 is the new 50 GiB root;
  • the complete scsi1 line is unchanged;
  • for VM 9003, the complete active scsi2 line is unchanged;
  • VM 9003 has only the new 100 GiB local-lvm runtime disk at scsi3, or VM 9004 has only the new 200 GiB local-lvm runtime disk at scsi2;
  • the old 800 GiB root exists as an unused rollback disk;
  • VM 9004's pre-existing 700 GiB unused0 still exists;
  • boot order, EFI disk, cloud-init disk, NIC/MAC/VLAN, CPU, RAM, and hostpci are unchanged.

Then start the existing VM and wait for cloud-init inside the guest before running Ansible:

set -euo pipefail
qm start "$TARGET_VMID"

for _attempt in $(seq 1 60); do
  qm guest cmd "$TARGET_VMID" ping >/dev/null 2>&1 && break
  sleep 5
done
qm guest cmd "$TARGET_VMID" ping >/dev/null

# The single-quoted scripts expand inside the guest, not on the PVE host.
# shellcheck disable=SC2016
ROOT_CHECK_JSON=$(qm guest exec "$TARGET_VMID" --timeout 300 -- \
  bash -lc '
    set -eu
    cloud_init_rc=0
    cloud-init status --wait || cloud_init_rc=$?
    test "$cloud_init_rc" -eq 0 || test "$cloud_init_rc" -eq 2
    cloud-init status --format=json | python3 -c '\''
import json
import sys

status = json.load(sys.stdin)
assert status["status"] == "done"
assert status.get("errors", []) == []
recoverable = status.get("recoverable_errors", {})
assert set(recoverable) <= {"DEPRECATED"}
assert all(
    "user" in message and "of type string is deprecated" in message
    for message in recoverable.get("DEPRECATED", [])
)
'\''
    root_disk=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi0
    test "$(blockdev --getsize64 "$root_disk")" -eq 53687091200
    root_partition=$(readlink -f "$(findmnt -nro SOURCE /)")
    test "$(blockdev --getsize64 "$root_partition")" \
      -gt $((48 * 1024 * 1024 * 1024))
    test "$(findmnt -nro FSTYPE /)" = ext4
    # A normal 50 GiB ext4 root has about 47.35 GiB usable after filesystem
    # metadata; the raw disk and partition checks above enforce the real size.
    test "$(df -B1 --output=size / | tail -n 1)" \
      -gt $((47 * 1024 * 1024 * 1024))
  ')
jq -e '.exitcode == 0' <<<"$ROOT_CHECK_JSON"

RUNTIME_BY_ID=/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-${RUNTIME_SLOT}
# shellcheck disable=SC2016
RUNTIME_CHECK_JSON=$(qm guest exec "$TARGET_VMID" --timeout 60 -- \
  bash -c '
    set -eu
    test -b "$1"
    test "$(blockdev --getsize64 "$1")" -eq "$2"
    signatures=$(wipefs --no-act --noheadings --output TYPE "$1")
    test -z "$signatures"
  ' bash "$RUNTIME_BY_ID" "$RUNTIME_BYTES")
jq -e '.exitcode == 0' <<<"$RUNTIME_CHECK_JSON"

Both JSON checks must report exit code zero. The runtime disk must still be blank and exact-sized before Ansible formats it. If cloud-init, root growth, or either device identity check fails, stop before Ansible; containerd must never populate the undersized OS root.

Rebuild the guest root

From the repository root, without --limit:

ansible-galaxy collection install --force \
  -r ansible/requirements.yml \
  -p ansible/collections

for MODULE_FILE in \
  ansible/collections/ansible_collections/ansible/posix/plugins/modules/mount.py \
  ansible/collections/ansible_collections/ansible/posix/plugins/modules/sysctl.py \
  ansible/collections/ansible_collections/community/general/plugins/modules/filesystem.py \
  ansible/collections/ansible_collections/community/general/plugins/modules/modprobe.py \
  ansible/collections/ansible_collections/community/general/plugins/modules/timezone.py; do
  test -s "$MODULE_FILE"
  ! grep -q 'mocked Ansible module generated by ansible-lint' "$MODULE_FILE"
done

ANSIBLE_CONFIG=ansible/ansible.cfg ansible-playbook \
  -i ansible/inventory/home.yml \
  ansible/playbooks/rejoin-node-26-04.yml \
  -e target_node=k8s-worker-home-2

Before installing containerd, the playbook reconstructs home-2's preserved Prometheus mount and prepares the inventory-pinned runtime disk. The preserved disk role checks exact by-id, size, ext4 type, label, and UUID and never formats the device. The runtime role accepts only a blank whole disk or its own exact previous ext4 mount, creates ext4 without force, persists it by UUID with noatime, and verifies the mounted source and size. Any mismatch stops before containerd can write to the OS root. The playbook then installs the base roles and longhorn-v2-node. Before running kubeadm join, it requires the preserved Kubernetes Node UID and waits for the stopped old kubelet's Ready condition to become False or Unknown. Kubeadm 1.36 refuses a same-name join while the existing Node remains Ready; waiting preserves the Node object without deleting it. The playbook also rejects bootstrap files left by an interrupted join instead of guessing that the root is clean. If the home API endpoint is not healthy at this boundary, stop and repair or explicitly update the kubelet API endpoint procedure first; do not improvise during the outage.

Validation before the second VM

  • guest root reports 50 GiB and PVE still reports scsi1 as 1 TiB;
  • /var/lib/containerd is an ext4 mount from the inventory-pinned by-id and is persisted in /etc/fstab by filesystem UUID with noatime; its raw block device reports exactly 100 GiB on home-2 or 200 GiB on home-3 (the ext4 filesystem is slightly smaller because of metadata);
  • the containerd systemd drop-in requires and verifies that runtime mount before containerd starts; on home-2, the kubelet drop-in likewise requires /var/lib/prometheus-local, so a later missing disk cannot silently fall through to the 50 GiB root;
  • on VM 9003, scsi2 remains the same 30 GiB device and /var/lib/prometheus-local mounts from UUID b66a7758-8337-4f75-b867-22ad17670ad6 with label prometheus-local, ext4, and noatime;
  • the node is Ready under its original name/IP with expected labels and taints, and its pre-cutover schedulability state is unchanged;
  • the Longhorn block disk is Ready with the same disk UUID;
  • nvme_tcp, uio_pci_generic, and vfio_pci are loaded and at least 1024 x 2 MiB hugepages are available;
  • the expected Intel GPU and /dev/dri are present;
  • Cilium, Longhorn manager/CSI, and node-local system pods are healthy;
  • all Longhorn v2 volumes and all ArgoCD Applications have converged;
  • the old 800 GiB root remains untouched as rollback storage.

Only after those gates pass should the same procedure be repeated for VM 9004 and k8s-worker-home-3.

Stage cleanup after validation

Keep the stopped stage VM until cloud-init, Ansible, Kubernetes, Cilium, Longhorn, mounts, and applications have passed the validation above. Then prove the moved root belongs to the target before deleting only the now-empty stage VM:

set -euo pipefail
test "$(qm status "$STAGE_VMID")" = "status: stopped"
STAGE_CONFIG_CLEANUP=$(qm config "$STAGE_VMID")
if grep -q '^scsi0:' <<<"$STAGE_CONFIG_CLEANUP"; then
  echo "refusing stage cleanup while scsi0 remains attached" >&2
  exit 1
fi
test "$(printf '%s\n' "$STAGE_CONFIG_CLEANUP" | sha256sum | cut -d' ' -f1)" = \
  "$(sha256sum "$STAGE_CONFIG_AFTER" | cut -d' ' -f1)"
test "$(qm config "$TARGET_VMID" |
  sed -n 's/^scsi0: \([^,]*\).*/\1/p')" = "$NEW_ROOT_VOL"

qm destroy "$STAGE_VMID" --purge 1

Never add --destroy-unreferenced-disks. The old 800 GiB root and VM 9004's pre-existing unused0 remain protected through the bake-in period.

Rollback

Stop the VM, retain the new scsi0, and retain the newly added runtime slot (scsi3 on VM 9003 or scsi2 on VM 9004) when that allocation completed. Each existing replacement volume must remain as an unused rollback/retry disk; do not delete it. Reattach the captured old root volume with its exact previous disk options as scsi0, verify that scsi1 and VM 9003's preserved scsi2 are still unchanged, and start the VM. The old root keeps its original containerd data and fstab, so the detached new runtime disk is not needed by the rollback boot. Keep both old roots and both new runtime disks until both workers have completed their bake-in period.

set -euo pipefail

# Select the worker, then restore the exact state persisted during staging.
TARGET_VMID=9003
ROTATION_STATE=/root/cauda-vm-root-rotation-${TARGET_VMID}.state
test -r "$ROTATION_STATE"
# shellcheck disable=SC1090
source "$ROTATION_STATE"

if [[ "${ROLLBACK_COMPLETE:-0}" == 1 ]]; then
  echo "rollback already completed for VM $TARGET_VMID"
  exit 0
fi

case "$(qm status "$TARGET_VMID")" in
  'status: running') qm shutdown "$TARGET_VMID" --timeout 180 ;;
  'status: stopped') ;;
  *) echo "VM $TARGET_VMID is neither cleanly running nor stopped" >&2; exit 1 ;;
esac
test "$(qm status "$TARGET_VMID")" = "status: stopped"

ACTIVE_ROOT_VOL=$(qm config "$TARGET_VMID" |
  sed -n 's/^scsi0: \([^,]*\).*/\1/p')
ACTIVE_RUNTIME_VOL=$(qm config "$TARGET_VMID" |
  sed -n "s/^${RUNTIME_SLOT}: \\([^,]*\\).*/\\1/p")

# Persist failed/retry volume IDs before the first rollback mutation. On a
# resumed rollback the active slots may already be empty or the old root may
# already be back on scsi0, so the state file is the authoritative identity.
if ! grep -q '^ROLLBACK_FAILED_NEW_ROOT_VOL=' "$ROTATION_STATE"; then
  ROLLBACK_FAILED_NEW_ROOT_VOL="$ACTIVE_ROOT_VOL"
  if [[ "$ROLLBACK_FAILED_NEW_ROOT_VOL" == "$OLD_ROOT_VOL" ]]; then
    ROLLBACK_FAILED_NEW_ROOT_VOL="${NEW_ROOT_VOL:-}"
  fi
  printf 'ROLLBACK_FAILED_NEW_ROOT_VOL=%q\n' \
    "$ROLLBACK_FAILED_NEW_ROOT_VOL" >>"$ROTATION_STATE"
fi
if ! grep -q '^ROLLBACK_FAILED_RUNTIME_VOL=' "$ROTATION_STATE"; then
  ROLLBACK_FAILED_RUNTIME_VOL="$ACTIVE_RUNTIME_VOL"
  if [[ -z "$ROLLBACK_FAILED_RUNTIME_VOL" ]]; then
    ROLLBACK_FAILED_RUNTIME_VOL="${RUNTIME_VOL:-}"
  fi
  printf 'ROLLBACK_FAILED_RUNTIME_VOL=%q\n' \
    "$ROLLBACK_FAILED_RUNTIME_VOL" >>"$ROTATION_STATE"
fi

FAILED_NEW_ROOT_VOL="$ROLLBACK_FAILED_NEW_ROOT_VOL"
FAILED_RUNTIME_VOL="$ROLLBACK_FAILED_RUNTIME_VOL"
if [[ -n "$FAILED_NEW_ROOT_VOL" && "$FAILED_NEW_ROOT_VOL" == "$OLD_ROOT_VOL" ]]; then
  echo "persisted failed-root identity incorrectly names the old root" >&2
  exit 1
fi
if [[ -n "$ACTIVE_ROOT_VOL" && "$ACTIVE_ROOT_VOL" != "$OLD_ROOT_VOL" &&
      "$ACTIVE_ROOT_VOL" != "$FAILED_NEW_ROOT_VOL" ]]; then
  echo "active scsi0 is neither the old root nor the persisted failed root" >&2
  exit 1
fi
if [[ -n "$ACTIVE_RUNTIME_VOL" &&
      "$ACTIVE_RUNTIME_VOL" != "$FAILED_RUNTIME_VOL" ]]; then
  echo "active runtime slot differs from the persisted rollback identity" >&2
  exit 1
fi

# Preserve failed/retry disks as unused volumes. These are active slot names,
# not unusedN keys. The runtime branch is optional if allocation never happened.
FAILED_NEW_ROOT_UNUSED_KEY=
if [[ -n "$FAILED_NEW_ROOT_VOL" && "$ACTIVE_ROOT_VOL" == "$FAILED_NEW_ROOT_VOL" ]]; then
  TARGET_DIGEST=$(config_digest "$TARGET_VMID")
  qm set "$TARGET_VMID" --digest "$TARGET_DIGEST" --delete scsi0
fi
if [[ -n "$FAILED_RUNTIME_VOL" && "$ACTIVE_RUNTIME_VOL" == "$FAILED_RUNTIME_VOL" ]]; then
  TARGET_DIGEST=$(config_digest "$TARGET_VMID")
  qm set "$TARGET_VMID" --digest "$TARGET_DIGEST" --delete "$RUNTIME_SLOT"
fi

if [[ -n "$FAILED_NEW_ROOT_VOL" ]]; then
  FAILED_NEW_ROOT_UNUSED_KEY=$(
    qm config "$TARGET_VMID" |
      awk -F': ' -v vol="$FAILED_NEW_ROOT_VOL" \
        '$1 ~ /^unused[0-9]+$/ && $2 == vol { print $1 }'
  )
  if [[ -z "$FAILED_NEW_ROOT_UNUSED_KEY" ]]; then
    echo "failed root is neither active nor retained as unused; inspect rescan" >&2
    qm disk rescan --dryrun 1 --vmid "$TARGET_VMID" >&2 || true
    pvesm list "$ROOT_POOL" --vmid "$TARGET_VMID" >&2 || true
    exit 1
  fi
fi

FAILED_RUNTIME_UNUSED_KEY=
if [[ -n "$FAILED_RUNTIME_VOL" ]]; then
  FAILED_RUNTIME_UNUSED_KEY=$(
    qm config "$TARGET_VMID" |
      awk -F': ' -v vol="$FAILED_RUNTIME_VOL" \
        '$1 ~ /^unused[0-9]+$/ && $2 == vol { print $1 }'
  )
  test -n "$FAILED_RUNTIME_UNUSED_KEY"
fi

test "$TARGET_VMID" != 9004 ||
  qm config 9004 | grep -Fx 'unused0: local-lvm:vm-9004-disk-2'

OLD_ROOT_LINE=$(sed -n 's/^scsi0: //p' "$TARGET_CONFIG_BEFORE")
test "${OLD_ROOT_LINE%%,*}" = "$OLD_ROOT_VOL"
ACTIVE_ROOT_VOL=$(qm config "$TARGET_VMID" |
  sed -n 's/^scsi0: \([^,]*\).*/\1/p')
if [[ "$ACTIVE_ROOT_VOL" != "$OLD_ROOT_VOL" ]]; then
  test -z "$ACTIVE_ROOT_VOL"
  OLD_ROOT_UNUSED_KEY=$(
    qm config "$TARGET_VMID" |
      awk -F': ' -v vol="$OLD_ROOT_VOL" \
        '$1 ~ /^unused[0-9]+$/ && $2 == vol { print $1 }'
  )
  test -n "$OLD_ROOT_UNUSED_KEY"
  TARGET_DIGEST=$(config_digest "$TARGET_VMID")

  qm set "$TARGET_VMID" \
    --digest "$TARGET_DIGEST" \
    --scsi0 "$OLD_ROOT_LINE"
fi

# Reattaching a non-CD root does not restore the boot order removed above.
TARGET_DIGEST=$(config_digest "$TARGET_VMID")
qm set "$TARGET_VMID" \
  --digest "$TARGET_DIGEST" \
  --boot "$TARGET_BOOT_LINE"

qm config "$TARGET_VMID" | grep -Fx "scsi0: $OLD_ROOT_LINE"
qm config "$TARGET_VMID" | grep -Fx "boot: $TARGET_BOOT_LINE"
qm config "$TARGET_VMID" | grep -F "scsi1:"
test "$TARGET_VMID" != 9003 ||
  qm config 9003 |
    grep -Fx 'scsi2: pve-storage:vm-9003-disk-0,discard=on,size=30G,ssd=1'
test "$TARGET_VMID" != 9004 ||
  qm config 9004 | grep -Fx 'unused0: local-lvm:vm-9004-disk-2'

# Remove only the two newly retained failed/retry volumes and require every
# pre-cutover config line to be restored byte-for-byte before starting.
ROLLBACK_CONFIG=/root/vm-${TARGET_VMID}-rollback.conf
qm config "$TARGET_VMID" >"$ROLLBACK_CONFIG"
awk -F': ' \
  -v rootkey="$FAILED_NEW_ROOT_UNUSED_KEY" -v rootvol="$FAILED_NEW_ROOT_VOL" \
  -v runtimekey="$FAILED_RUNTIME_UNUSED_KEY" -v runtimevol="$FAILED_RUNTIME_VOL" \
  '(rootkey != "" && $1 == rootkey && $2 == rootvol) ||
   (runtimekey != "" && $1 == runtimekey && $2 == runtimevol) { next }
   { print }' \
  "$ROLLBACK_CONFIG" >/root/rollback.normalized
diff -u "$TARGET_CONFIG_BEFORE" /root/rollback.normalized

qm start "$TARGET_VMID"
test "$(qm status "$TARGET_VMID")" = "status: running"
printf 'ROLLBACK_COMPLETE=1\n' >>"$ROTATION_STATE"

Never pass an unusedN key to qm disk unlink: Proxmox physically removes an already-unused volume even without --force. Reattach using the captured volume ID and exact disk options; do not delete an unused entry as a shortcut.

The relevant Proxmox semantics are documented in the official qm manual: unlinking a disk without force preserves it as unused storage, and qm disk move can reassign a disk with --target-vmid and --target-disk.