Skip to content

ADR-0030: All AI/GPU consumers route through LiteLLM — never directly to a backend

  • Status: Accepted
  • Date: 2026-07-13

Context

The paperclip fleet's opencode providers connected directly to each GPU backend (llama-swap on the B60, the OVMS B580 router, workstation Ollama on the 4090, office LM Studio) with static keys and no timeouts, and reimplemented routing/health/capacity in an app-side admission policy (globalMaxRuns:5 + per-backend caps). Meanwhile LiteLLM — deployed as the unified gateway (ADR context: .workingdir2/gpu-unified-backend-design.md) — already served every one of those models with routing, load-balancing, fallbacks, cooldowns, and timeouts.

Observed consequences of the bypass (2026-07-13): runs hung queued/ running for 4+ hours (direct calls with no timeout never fail), agents stayed pinned to dead runs (executor deadlock), the healthy B580 sat idle with 9 runs queued while the B60 thrashed, and the global cap of 5 sat below the ~10-run combined capacity of the four cards — three of four GPUs idle while the fleet starved. Per user direction: every AI, encode/decode, or compute job must go through a proxy that manages jobs, never directly to a card.

Decision

  1. Every consumer of local AI targets LiteLLM only (http://litellm.ai.svc.cluster.local:4000/v1, key from the litellm-master-key Secret). No app may configure a backend/card URL directly. The proxy owns routing, load-balancing, failover, cooldowns, and timeouts.
  2. llama-swap remains the per-host model-swap manager under LiteLLM for cards that serve multiple swappable models (today: the B60). It cannot span hosts or proxy remote endpoints (llama-swap main docs: cmd is required per model), so single-model cards (B580 OVMS router) and remote cards (4090 Ollama, office LM Studio) are plain LiteLLM deployments.
  3. The paperclip fleet's providers keep their names and model ids (no agent reconfiguration) but transport via LiteLLM; LiteLLM carries bare-name alias entries mirroring the canonical cauda/* entries.
  4. App-side admission remains only as a concurrency bound (protects the single-threaded paperclip pod), not as a router: globalMaxRuns raised 5→10 to match the real per-lane capacity of the four cards.

Consequences

  • A stalled backend now returns a timeout/fallback via LiteLLM instead of wedging runs for hours; a down card cools down and traffic fails over.
  • New GPU workload = one LiteLLM model_list entry; consumers never change.
  • The bare-name aliases in apps/ai/litellm/values.yaml must stay in sync with their canonical cauda/* entries (flagged inline).
  • Virtual keys per consumer (budgets/scoping) are the follow-up; the master key is the documented bridge.

References

  • apps/ai/litellm/values.yaml, apps/ai/paperclip/templates/deployment.yaml, apps/ai/paperclip-dispatcher/values.yaml
  • docs.litellm.ai: routing, proxy/load_balancing, proxy/reliability, proxy/timeout, proxy/health; github.com/mostlygeek/llama-swap (README + docs/configuration.md, v239)
  • .workingdir2/research/22-litellm-overhaul.md (docs-grounded synthesis)