Skip to content

0044. Cordana governed compute plane

  • Status: superseded by ADR-0045 — governance-plane split stands, whole-card ownership reversed
  • Date: 2026-07-20
  • Deciders: lusoris
  • Related: ADR-0030, ADR-0041, ADR-0042

Context

ADR-0030 requires that AI inference go through the managing proxy rather than direct-to-card. That rule was written around inference because inference was the workload that happened to have a proxy in front of it. Every other heavy workload in the cluster is ungoverned: video transcode (jellyfin, whisper-asr), vision and embedding (immich ML, NSFW detectors, qwen3-embed), simulation (engineering sim-runners), rendering (northlight video-runner) and media processing (romm-curator).

The contention is already real. Four pods claim gpu.intel.com/xe: 1 — llama-swap, qwen3-8b-b580-predictor and qwen3-embed-predictor in ai, plus jellyfin in media. A media transcode therefore competes with LLM inference for the same device class with no arbiter, no queue and no priority. llama-swap is documented as needing exclusive ownership of the B60 share, so exclusivity today is maintained by convention rather than by a mechanism.

Much of this work is inherently async and batchable — transcoding a library, running a simulation, embedding a corpus, ROM curation, vision tagging — and is currently competing with interactive, latency-sensitive work for the same silicon. Deferring async work to idle windows is free capacity.

The operator's target is enterprise-grade workload management across all workload kinds, explicitly including on consumer hardware that normally does not support it, and named CUDA as a real gap: the RTX 4090 lives on the workstation and is not a cluster node, so it is invisible to any in-cluster scheduler.

Decision

Extend ADR-0030's principle to all heavy compute, and adopt the split that the hardware actually permits.

1. Separate the governance plane from device isolation. An enterprise-grade scheduling and governance plane — queueing, fair-share, per-tenant quota, priority with preemption, topology-aware placement, accounting — is achievable on this hardware. Datacenter-grade GPU isolation is not, on either card. These are independent decisions and must not be traded against each other.

2. Whole-card ownership is the unit. Neither card supports partitioning that the kernel or driver will enforce. sharedDevNum replicates access, it does not partition; MIG is absent from the RTX 4090's supported products; time-slicing provides no isolation; MPS is marked experimental by the NVIDIA plugin and keeps all clients in one fault domain. Advertising five Arc shares therefore states a capacity that does not exist, and any quota built on it is bookkeeping rather than a limit. The production Arc flavor moves to sharedDevNum: 1.

3. Multiplexing happens in userspace, per card, above the device. Where consumers can be made to speak a protocol, one arbiter owns the card and serialises access — the shape llama-swap already implements for LLM serving. Where a consumer needs the device node directly (VA-API transcode needs /dev/dri), no userspace arbiter can mediate it and the only lever is scheduling. That line, not workload type, determines the device classes.

4. Batch is queued; serving is scheduled. Kueue governs Jobs, JobSets and its supported integrations. Long-running Deployments and KServe InferenceServices are not Jobs and Kueue does not govern them. The governed plane is therefore two mechanisms, not one, and claiming otherwise would leave the serving half unmanaged while appearing solved.

5. Non-node compute is reached by a queue-worker, not a fake node. The workstation GPU is consumed by a worker that pulls from an in-cluster queue. Virtual-kubelet-family options can present it as a Node, but faithfully reproducing kubelet semantics for a single card is unjustified, and the synthetic Node lies whenever the workstation is off.

6. Preemption requires a checkpoint contract. No transparent GPU checkpoint or migration exists for CUDA or Level Zero. A workload is eligible for preemption only if it checkpoints to durable storage and resumes idempotently; everything else is preemptionPolicy: Never or lives in a protected queue.

Consequences

Async work becomes deferrable to idle windows, which is the capacity win that motivated this. Tenants gain real quota and fair-share, and accounting becomes possible per workload.

sharedDevNum: 1 means one pod per card. The B580 currently hosts both jellyfin and qwen3-8b-b580-predictor, so that co-tenancy must be resolved — by scale-to-zero, by moving one consumer, or by accepting serialisation — before the change lands. That decision is the operator's and is not made here.

Preemption is not enabled until the checkpoint contract exists. Enabling it first would silently destroy work.

The xe-dmem-nri plugin stays experimental and disabled. It can cap VRAM residency, but spill, failure, hierarchy and cleanup behaviour are unproven on this kernel, and it does not cap compute.

We accept that this cluster will not have MIG-equivalent partitioning, transparent GPU checkpointing, or per-tenant compute-bandwidth fairness on the Arc cards. The nearest dependable equivalents are whole-card ownership, queue-level multiplexing and application-controlled checkpoint/restart. Stating this plainly is preferable to configuring something that appears to enforce and does not.

Alternatives considered

Enable the Intel plugin's resource manager. Removed upstream in intel-device-plugins 0.34; the setting is inert on the deployed 0.36.0 chart. Its GPU-Aware-Scheduling prerequisite is a scheduler extender whose maintenance does not justify a dependency on Kubernetes 1.36.

Volcano or YuniKorn instead of Kueue. Volcano has no Kubernetes 1.36 support and YuniKorn is capped at 1.34. Both would gate the cluster's own upgrade path.

Make the workstation a real node. Technically the strongest option and it remains open if the workstation becomes always-on. Today it sleeps and reboots on the operator's schedule, so its Pods would be evicted and recreated on a cadence set by desktop use.

MPS on the RTX 4090 for memory and compute caps. Deferred, not rejected. It can impose limits, but the plugin calls it experimental, clients share a fault domain, and accounting attributes work to the MPS server rather than the tenant. It may be promoted to a separately labelled experimental flavor after node-local validation.

Do nothing beyond inference. Rejected. Contention is present now, and the convention protecting llama-swap's card is unenforced.

References

  • .workingdir2/cache/cc-enterprise-batch-gap.md — capability matrix per HPC property against stock Kubernetes 1.36, Kueue 0.18.3, Intel xe and the RTX 4090
  • .workingdir2/cache/cc-nonnode-compute-adapters.md — ranked evaluation of virtual-kubelet, interLink, MultiKueue, queue-worker and real-node options
  • .workingdir2/cache/cr-resourcemanager-reality.md — Intel resourceManager removed in 0.34
  • .workingdir2/cache/cr-xe-cgroup-enforcement.md — dmem cgroup as the only VRAM ceiling available on Arc
  • .workingdir2/gpu-unified-backend-design.md — the prior inference-only design this generalises