Skip to content

0048. Restoring Qwen3-30B to Unordered Backend Pool

  • Status: accepted
  • Date: 2026-07-24
  • Deciders: lusoris
  • Related: PR #3005

Context

An earlier correction on issue #2959 removed the B60 llama-swap Qwen3 30B physical leg from normal LiteLLM routing after OVMS 2026.2.1 cold-start and child-reaping failures, leaving the canonical cordana/qwen3-30b group served only through the CUDA-capable deployment. The superseded state entry records that this removal incorrectly coupled the working route's exclusion to an independent LM Studio Qwen3 8B context correction (Paperclip advertising a 40,960-token context against a 32,768-token live deployment). PR #3005 undoes the exclusion while keeping the context correction.

Decision

Restore the llama-swap Qwen3 30B route (openai/qwen3-30b at http://llama-swap.ai.svc.cluster.local/upstream/qwen3-30b/v3) to the canonical cordana/qwen3-30b pool alongside the existing CUDA-capable route, as compatible physical legs of one unordered pool. Neither leg has a configured order or a consumer-visible hardware identity: LiteLLM selects an eligible backend and llama-swap loads models on demand. The restored leg carries a 180-second request circuit breaker grounded in measured healthy cold starts of 41–84 seconds, so LiteLLM's typed retries can fail over to another compatible route instead of holding a request on a wedged child. The standing AGENTS.md instruction to keep the B60 Qwen3 30B deployment out of the pool is removed, and the independent LM Studio/Hindsight Qwen3 8B context correction is retained unchanged.

Alternatives considered

  • Keep the leg quarantined and serve cordana/qwen3-30b from the CUDA deployment only: rejected — the exclusion had been coupled to an unrelated context correction rather than justified on its own evidence, and it left the canonical group with a single backend.
  • Restore the leg with a configured routing order or a hardware-specific public alias: rejected — consumers must never select hardware or perform failover; ordering and hardware identity stay internal to LiteLLM's unordered pool routing.

Consequences

  • cordana/qwen3-30b regains a second eligible backend; a busy or wedged leg fails over via the existing same-group retry policy instead of depending on one deployment.
  • The 180-second circuit breaker bounds the restored leg; requests that legitimately need longer on that leg are retried elsewhere in the pool, and raising the bound requires new cold-load and completion evidence.
  • Chart tests now assert the llama-swap leg's presence, timeout, chat-completions bridging, and cache policy on the new upstream; the model-deployment projections were regenerated for the changed LiteLLM values.
  • Paperclip continues to advertise the measured 32,768-token Qwen3 8B context independently of this routing change.

References

  • PR #3005 — fix(ai): restore dynamic qwen3 30b routing