Skip to content

Kured home-worker maintenance gate — issue #2573

Use this runbook while automatic home-worker reboots are blocked by the one-replica Longhorn V2 and Gitea Valkey disruption contracts.

Active fail-closed gate

Kured 1.23.0 checks two node-local pod selectors before acquiring its lock, cordoning or draining:

  • the Longhorn V2 instance manager on a home-zone node;
  • a Gitea Valkey cluster member.

The check period is one hour, a drain attempt is bounded at five minutes, and forceReboot is explicitly false. skipWaitForDeleteTimeout: 60 only stops waiting for a pod that was already terminating for 60 seconds; it never bypasses a PodDisruptionBudget.

Verify the source and exact upstream-chart render:

python3 .githooks/check-kured-render.py --render

Verify live ownership without changing the cluster:

kubectl -n kube-system get daemonset kured \
  -o jsonpath='{.spec.template.spec.containers[0].args}'
kubectl get node k8s-worker-home-2 k8s-worker-home-3
kubectl -n dev get pod,pdb -l app.kubernetes.io/instance=gitea -o wide
kubectl -n longhorn-system get settings.longhorn.io node-drain-policy -o yaml

Removal gates

Do not remove either selector until all of these are true:

  1. Stable Longhorn 1.12.1 or newer is installed; RC upgrades are unsupported.
  2. The V2 snapshot/backup fix for Longhorn issue #13331 passes an attached, writable canary and a two-home-worker detach/reattach canary.
  3. Gitea Valkey is healthy and its placement/PDB contract permits loss of every single home worker. Three masters on two workers cannot meet this condition when two members share the worker being drained.
  4. Both viable official node-drain policies pass separate post-upgrade canaries before one is selected:
  5. allow-if-replica-is-stopped avoids a replica copy for same-node in-place reboot, but the only replica must actually stop, remote reattachment must not race the drain, and loss of the returning node would lose the data;
  6. block-for-eviction is Longhorn's one-replica automation lane, but it must prove same-zone target capacity and needs a much longer bounded timeout to copy every local replica without cross-site movement. Never use always-allow. Do not preselect either viable policy before these canaries produce evidence for this cluster.
  7. One controlled worker canary drains, reboots, returns Ready, and uncordons with every affected filesystem writable and every Longhorn volume healthy.

Removing the gate or running the canary changes production state and requires explicit operator approval. Keep all configuration changes in Git and let ArgoCD apply them.