AI inference optimization matrix¶
Date: 2026-07-10
Scope: the seven techniques in Red Hat's April 2026 AI-optimization article, mapped to this cluster's actual runtimes and hardware. The article describes general production patterns; implementation decisions below use the live OVMS 2026.2/OpenVINO stack rather than assuming vLLM or NVIDIA controls exist.
Decision matrix¶
| Technique | Current state | Decision |
|---|---|---|
| Quantization | All production OVMS LLM weights are pre-converted INT4. LLM KV caches are explicitly u8. |
Keep. Accuracy-sensitive model changes still require task-corpus evaluation before replacement. |
| Automatic prefix caching | Explicitly enabled on the 8B, B580 8B, 30B, gpt-oss, 27B, and 4B text services. Paperclip agents reuse large stable instruction prefixes. | Keep. Put stable instructions/context before per-run variable content when authoring prompts. |
| Disaggregated prefill/decode | Not deployed. The fleet has one B60, two B580s, and one RTX 4090, but they use different runtimes/models across cluster and workstation boundaries. | Do not deploy now. Prefill/decode requires a compatible target model and KV-transfer path across both pools; the present heterogeneous endpoints cannot be treated as interchangeable halves. |
| Flash Attention | Not exposed as an OVMS task flag. OpenVINO selects its supported GPU attention kernels; CUDA/xFormers/Triton flags do not apply to Intel Arc. | No config change. Track OpenVINO's native XAttention/attention-kernel releases instead of importing vLLM/CUDA settings. |
| Continuous batching | OVMS text-generation continuous batching is active. Services explicitly bound max_num_seqs=16 and max_num_batched_tokens=2048; Paperclip adds backend/global admission caps. |
Keep and tune from measurements only. Raising defaults previously contributed to B60 overcommit/hangs. |
| KV-cache management | Static 2 GiB pools on 8B/B580 and 4 GiB on 30B, U8 precision, bounded concurrency, queue/KV-related alerts, and demand-based per-GPU eviction are deployed. | Keep. Prefer measured cache/queue tuning over increasing utilization toward 100%; reliability headroom is intentional. |
| Speculative decoding | Not deployed. OVMS 2026.2 supports EAGLE-3 on Intel CPU/GPU and reports validation with Qwen3-8B. Benefit depends on draft/target alignment and workload predictability. | Benchmark on the B580 before production. Track in #2244; reject if draft-model VRAM harms Jellyfin coexistence or gains are negligible. |
Workload coverage¶
- Paperclip / Open WebUI text: all immediately applicable baseline techniques are enabled. Paperclip's stable instruction bundles are naturally prefix-cache friendly; global/backend dispatcher caps protect the Node control plane and GPUs.
- Embedding and vision models: quantized OpenVINO artifacts and GPU routing apply, but autoregressive prefix caching, speculative decoding, and LLM KV tuning do not.
- ComfyUI / diffusion: the article's LLM-specific prefill, decode, prefix, and KV recommendations do not apply. It remains governed by the B60 capacity scheduler and workload-specific memory limits.
- Whisper ASR and CPU classifiers: use the runtime-appropriate OpenVINO/XPU path; LLM continuous batching and speculative decoding are not applicable.
- External workstations: the home RTX 4090/Ollama coder lane and office B580/LM Studio Gemma lane are separately availability-gated. Runtime settings live on their hosts; tune them only against supported controls and measured workloads.
Measurement contract¶
Any further inference optimization must record baseline and candidate TTFT, decode tokens/second, aggregate throughput, p95 latency, error/OOM rate, VRAM, power, and task-quality results. A throughput improvement that reintroduces GPU wedges, Paperclip retry storms, or poor tool-call output is a regression.
Sources¶
- Red Hat, AI optimization: 7 powerful techniques you can use today!, 2026-04-16.
- OpenVINO 2026 release notes and OVMS performance-tuning/speculative-decoding documentation, checked 2026-07-10.