Skip to content

ADR-0039: Easy/hard difficulty routing uses LiteLLM's native auto-router, not a standalone semantic-router service

  • Status: Accepted
  • Date: 2026-07-14

Context

Trivial agent turns (greetings, one-line acknowledgements, tiny lookups) were being sent to heavy tiers — in particular the 27B dense model — the same way substantive reasoning turns were, because callers pick a fixed model name. On the contended Arc B60 this wakes a swap tier for work an 8B could finish in a fraction of the time (the "B60 can't keep >2 warm tiers" thrash). The ecosystem gap-scan (.workingdir2/research/ecosystem-gap-scan.md, item 4) flagged cheap embedding-based difficulty routing as a real, low-effort win and named aurelio-labs semantic-router as a candidate.

The open question was whether difficulty routing needs a NEW component (a standalone semantic-router service in front of the proxy) or whether the in-stack gateway can already do it. ADR-0030 mandates that every AI consumer routes through LiteLLM at litellm.ai.svc:4000 — adding a second router in the request path would fight that decision.

Verification against the installed proxy (LiteLLM v1.92.0, source read at the v1.92.0 tag):

  • LiteLLM ships a native auto-router (litellm/router_strategy/auto_router/), introduced in v1.74.9 and powered by the semantic-router library — the exact project the gap-scan proposed to deploy separately. It is present in our pinned version.
  • A model_list entry of the form model: auto_router/<name> with an inline auto_router_config (routes: name → target model, utterances, score threshold), an auto_router_default_model, and an auto_router_embedding_model turns the last user message into an embedding, matches it against the routes, and forwards the call to the best route above threshold — or to the default model when nothing matches. No extra LLM call; a deterministic vector match.
  • The encoder is LiteLLM's own LiteLLMRouterEncoder, so the embedding call goes back through the proxy's own embedding model — we already serve cauda/qwen3-embed. Embeddings never leave litellm:4000, satisfying ADR-0030.
  • semantic-router is an OPTIONAL LiteLLM extra (semantic-router>=0.1.15) and is NOT in the base litellm-non_root image; enabling the feature requires a thin image overlay that installs it.

So the capability the gap-scan wanted already lives inside the gateway. A separate service is unnecessary and would violate ADR-0030.

Decision

  1. Difficulty routing is implemented with LiteLLM's native auto-router, not a standalone semantic-router deployment. It is one gated model_list entry (cauda/auto) in apps/ai/litellm/values.yaml under autoRouter.
  2. The router embeds via the proxy's own cauda/qwen3-embed (encoder_type=litellm), so all inference AND embedding traffic stays on litellm:4000 (ADR-0030). No new component, no second router in the path.
  3. The routes map hard/reasoning turns to cordana/qwen3-6-27b and coding turns to cauda/qwen3-coder-30b; everything else (the trivial-turn majority) falls through to the cheap cauda/qwen3-8b-any default. The heavy tiers stay registered as ordinary models, so nothing is removed as a fallback.
  4. The feature ships INERT: autoRouter.enabled=false renders no model_list entry, so live routing is byte-identical to before. Arming is a two-step operator action — build/pin an image overlay that installs semantic-router, then set the flag and point a chosen caller at cauda/auto. Callers are opt-in; nothing existing is re-wired.

Alternatives considered

  • Standalone aurelio-labs semantic-router service in front of LiteLLM (the gap-scan's fallback option). Rejected: it duplicates a capability the proxy already has, adds a second router hop, and contradicts ADR-0030's single-front-door rule.
  • LiteLLM's newer complexity_router (a heuristic/LLM/keyword classifier with SIMPLE/MEDIUM/COMPLEX/REASONING tiers). It lands in v1.94.x+; our proxy is pinned at v1.92.0, which predates it. Revisit when the proxy is bumped — its zero-embedding heuristic scorer would remove even the embedding call.
  • Tag/model-group routing (caller passes a tags hint). Rejected as the primary mechanism: it needs every caller to classify its own request, which is exactly the judgement we want the router to make; it stays available for explicit overrides.

Consequences

  • No new Deployment, Service, or image beyond a thin semantic-router overlay on the existing litellm image. The whole feature is one values block plus a gated append in templates/configmap.yaml.
  • Arming requires the image overlay first — the base image ImportErrors on the optional dependency if the flag is flipped without it. This is documented in apps/ai/litellm/README.md and enforced by the default-off flag.
  • When the proxy is bumped to v1.94.x+, evaluate migrating cauda/auto to the heuristic complexity_router to drop the per-request embedding call.
  • Route quality (utterance sets, score thresholds) is unmeasured against real qwen3-embed vectors; tune during a shadow before pointing production callers at cauda/auto.

References

  • apps/ai/litellm/values.yaml (autoRouter), apps/ai/litellm/templates/configmap.yaml, apps/ai/litellm/README.md
  • docs.litellm.ai/docs/proxy/auto_routing_semantic (semantic auto-router, config schema); release_notes/v1.74.9-stable ("auto-routing powered by semantic-router")
  • LiteLLM source @ v1.92.0: litellm/router_strategy/auto_router/auto_router.py, litellm/router_strategy/auto_router/litellm_encoder.py, pyproject.toml ([project.optional-dependencies] semantic-router)
  • ADR-0030 (all AI consumers route through LiteLLM)
  • .workingdir2/research/ecosystem-gap-scan.md (item 4, semantic-router)