Skip to content

0012. Two Cluster Rescue Workloads

  • Status: accepted
  • Date: 2026-05-26
  • Deciders: lusoris
  • Related: PR #258

Context

PR #258 added two small, scheduled self-healing workloads to recover from specific cluster failure modes that otherwise need manual kubectl surgery. This ADR was auto-drafted by the adr-suggester and its Decision was left as the degenerate LLM-judge placeholder true; this records the actual decision.

Decision

Deploy two cluster-rescue workloads in ns ai (apps/ai/), each a scheduled CronWorkflow that acts only on clearly-wedged objects:

  • pvc-mount-failure-detector — detects pods stuck on PVC mount/attach failures (apps/ai/pvc-mount-failure-detector/).
  • finalizer-stuck-pod-rescuer — clears pods stuck terminating on unresolved finalizers (apps/ai/finalizer-stuck-pod-rescuer/).

Alternatives considered

  • Manual remediation only — rejected: these failure modes recur and each incident needed hands-on kubectl intervention.
  • A single combined rescue workload — rejected: the two failure modes have different detection signals and blast radii; keeping them separate bounds each one's authority.

Consequences

  • Two more scheduled workloads to monitor, but each is narrowly scoped and acts only on unambiguously stuck objects.
  • Recurring PVC-mount and stuck-finalizer incidents self-heal without manual intervention.

References

  • PR #258 — feat(ai): two cluster-rescue workloads (batch 15a)