Skip to content

0041. Node Feature Discovery + Intel GPU NodeFeatureRule for GPU node labeling

  • Status: accepted
  • Date: 2026-07-14
  • Deciders: lusoris
  • Related: PR #2467, PR #2466

Context

intel-gpu-plugin schedules only onto nodes carrying intel.feature.node.kubernetes.io/gpu=true. The GPU scheduling audit (.workingdir2/gpu-scheduling-audit.md, PR #2466) established that Node Feature Discovery was genuinely not deployed — zero nfd pods, and none of the richer gpu.intel.com/* capability labels present — and that the plugin's required label was being faked by a manual kubectl label node ... --overwrite in ansible/playbooks/join-nodes.yml.

That is a silent footgun rather than an outage: with one card per node, GPU workloads already schedule correctly via hostname and zone selectors. The failure mode is additive — add a GPU node, forget the Ansible loop entry, and the plugin quietly does not schedule there with no error surfaced.

The audit graded the value as medium and was explicit that this fixes nothing currently broken. It also rendered the Intel chart's output to verify the mechanism rather than assume it: chart 0.36.0 with nodeFeatureRule: true emits a NodeFeatureRule at nfd.k8s-sigs.io/v1alpha1, matching pci.vendor In [8086] and class In [0300,0380], which produces exactly the label the plugin's nodeSelector needs. It additionally produces gpu.intel.com/device-id.0300-e211 (B60) and ...-e20b (B580), which distinguish the two Battlemage cards semantically — today they are separable only by hostname. gpu.intel.com/product and family labels are emitted only for datacenter Flex/Max parts, so client Arc gets none and device-id is the differentiator available to us.

Compatibility was verified against the live charts on 2026-07-14: NFD 0.19.0 serves the nfd.k8s-sigs.io/v1alpha1 CRD the Intel chart targets, and NFD 0.19.0 renders cleanly into kube-system with resources set on master, worker and gc — required because require-resource-limits is Enforce.

The PR was opened gated ("do not merge without a deploy decision") because the app-of-apps recurses argocd-apps/, so merging it deploys NFD cluster-wide.

Decision

Deploy Node Feature Discovery as an ArgoCD Application (argocd-apps/infra/node-feature-discovery.yaml), chart 0.19.0, into kube-system, and flip nodeFeatureRule: false → true on argocd-apps/infra/intel-gpu-plugin.yaml so the Intel NodeFeatureRule ships and NFD produces the GPU labels automatically. The NFD chart repo is added to sourceRepos on the infra ArgoCD project.

Ordering is handled by sync waves: NFD at -2 (creating the CRD), the Intel operator at -1, the plugin at 0. ArgoCD retry absorbs any transient "CRD not found" on first convergence.

postDeleteCleanup: false is set because the chart's Helm post-delete prune Job is un-resourced and would trip the require-resource-limits Kyverno Enforce policy; CRD cleanup on the rare uninstall becomes a manual step instead.

NFD here is labelling only. It does not re-enable fractional or GPU-aware scheduling — the Intel GPU-aware scheduling extensions were removed upstream after v0.32.0 and remain off.

The manual Ansible label was deliberately kept as belt-and-braces in this PR, with the intent to retire it after NFD was confirmed to label both GPU nodes. Removing it before that confirmation would have broken GPU scheduling.

Alternatives considered

  • Status quo — keep the manual Ansible label. This is the option the audit weighed the change against, and it is why the value was graded medium rather than high: one-card-per-node scheduling already worked. It was rejected for automation, richer labels and upstream alignment, at the cost of a cluster-wide nfd-worker DaemonSet plus master, gc and CRDs.
  • Bringing the 4090 workstation into the cluster and reusing this NFD for nvidia-device-plugin with gfd.enabled=true / nfd.enabled=false. The audit documents this as the turnkey pattern for a future dedicated, headless NVIDIA box but explicitly rejects it for the current workstation: it conflicts with that card's training and model-prep role, and the card is already routable through LiteLLM. Documented, not scaffolded — no such box exists.

Consequences

  • Adding a GPU node no longer requires remembering to label it; the plugin's nodeSelector is satisfied from detected PCI hardware.
  • Device-id labels now distinguish B60 from B580, which is available for future model-placement and observability work but is not yet consumed by any manifest.
  • A cluster-wide nfd-worker DaemonSet plus master and gc pods is added to kube-system on every node, not just GPU nodes.
  • The planned post-merge cutover was never completed. The manual label task is still present at ansible/playbooks/join-nodes.yml:73, and its loop still contains only k8s-worker-home-1 — which is not where the current GPU cards live. So the Ansible fallback is now both redundant and stale: it no longer provides the belt-and-braces coverage it was retained for, while NFD does cover the actual GPU nodes. This is exactly the drift the change was meant to eliminate, and retiring that task is outstanding.
  • Uninstalling NFD leaves its CRDs behind, by the postDeleteCleanup: false choice above.

References

  • PR #2467 — feat(gpu): NFD + Intel GPU NodeFeatureRule for auto GPU node labels [GATED — do not merge without deploy decision]
  • PR #2466 — the GPU scheduling audit doc
  • .workingdir2/gpu-scheduling-audit.md — assessment area 1 (NFD), the rendered NodeFeatureRule output, ranked recommendation R2, and the R3 rejection of in-clustering the 4090
  • argocd-apps/infra/node-feature-discovery.yaml, argocd-apps/infra/intel-gpu-plugin.yaml