Designing an LLM Gateway: Multi-Provider Routing and Fallback Strategies

Relying on OpenAI alone for a production LLM service is risky. When OpenAI went down in November 2023, countless services went down with it. An LLM gateway puts multiple providers behind a single interface and centrally manages routing, fallback, and cost optimization.

What an LLM Gateway Does

CODE

Client
    ↓
LLM Gateway (routing / fallback / caching / logging)
    ↓        ↓        ↓
  OpenAI  Anthropic  Google

Routing: Choose which model to send each request to
Fallback: Automatically switch to another provider on failure
Rate limiting: Manage per-provider quotas
Cost tracking: Unified cost monitoring

Getting Started Quickly with LiteLLM

LiteLLM unifies 100+ LLMs behind an OpenAI-compatible interface.

Python

from litellm import completion

#
def llm_call(model, messages):
    response = completion(model=model, messages=messages)
    return response.choices[0].message.content

llm_call("gpt-4o", [{"role": "user", "content": ""}])
llm_call("claude-sonnet-4-6", [{"role": "user", "content": ""}])
llm_call("gemini/gemini-1.5-pro",[{"role": "user", "content": ""}])

Implementing a Fallback Chain

Python

from litellm import completion

FALLBACK_CHAIN = [
    "gpt-4o",
    "claude-sonnet-4-6",
    "gemini/gemini-1.5-pro"
]

def resilient_llm_call(messages, preferred_model=None):
    chain = ([preferred_model] + FALLBACK_CHAIN) if preferred_model else FALLBACK_CHAIN
    chain = list(dict.fromkeys(chain))

    last_error = None
    for model in chain:
        try:
            response = completion(model=model, messages=messages, timeout=10, num_retries=1)
            return {
                "content":      response.choices[0].message.content,
                "model_used":   model,
                "was_fallback": model != chain[0]
            }
        except Exception as e:
            last_error = e
 print(f"[WARN] {model} failed, when")

 raise RuntimeError(f" failed: {last_error}")

Cost-Based Routing

Python

ROUTING_TABLE = {
    "low":    {"primary": "gpt-4o-mini",       "fallback": "claude-haiku-4-5-20251001"},
    "medium": {"primary": "claude-sonnet-4-6", "fallback": "gpt-4o"},
    "high":   {"primary": "claude-opus-4-7",   "fallback": "gpt-4o"},
}

def classify_task(messages):
    user_text = " ".join(m["content"] for m in messages if m["role"] == "user")
 if any(kw in user_text for kw in ["", "", "architecture", ""]):
        return "high"
 if any(kw in user_text for kw in ["", "", "", ""]):
        return "low"
    return "medium"

def smart_route(messages):
    complexity = classify_task(messages)
    route = ROUTING_TABLE[complexity]
    return resilient_llm_call(messages, preferred_model=route["primary"])

Deploying a LiteLLM Proxy Server

This lets the entire team share a single endpoint.

YAML

# litellm_config.yaml
model_list:
  - model_name: gpt-4o
    litellm_params:
      model: gpt-4o
      api_key: os.environ/OPENAI_API_KEY

  - model_name: claude-sonnet
    litellm_params:
      model: claude-sonnet-4-6
      api_key: os.environ/ANTHROPIC_API_KEY

router_settings:
  routing_strategy: cost-based-routing
  fallbacks:
    - {"gpt-4o": ["claude-sonnet", "gemini-pro"]}

general_settings:
  master_key: sk-my-master-key
  database_url: os.environ/DATABASE_URL

Bash

litellm --config litellm_config.yaml --port 4000

From then on, anyone on the team can use http://localhost:4000 as if it were an OpenAI endpoint. Key management, cost tracking, and fallbacks are all centralized.

When Fallback Doesn't Trigger, or Triggers Too Much

Gateway fallback settings are only exercised during outages, so misconfigurations go unnoticed in normal times.

Causes

Fallback triggers only on 5xx, while timeouts and 429 (rate limit) just retry repeatedly
Falling back on every error sends even our own bugs, like malformed requests (400), to another provider and only raises cost
The fallback model handles output format (JSON mode, tool calling) differently from the primary

Reproduce: fault-injection test In staging, deliberately fail calls to the primary provider and verify the fallback path. With LiteLLM you can configure the primary with an invalid API key, or block the provider's domain with a network policy.

Bash

# send a request through the proxy and see which model actually answered
curl -s http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer $LITELLM_KEY" -H "Content-Type: application/json" \
  -d '{"model":"default-chat","messages":[{"role":"user","content":"ping"}]}' | jq '.model'

Fix

Classify errors explicitly: fall back on timeouts, 429, and 5xx; fail immediately on request/auth errors such as 400 and 401.
Run regression tests confirming the fallback model also passes output schema validation.

Prevention

Record fallback counts as metrics by model and cause and alert on spikes. Repeat fault-injection tests every quarter.

#LLM #Architecture #LiteLLM

✦ ✦ ✦

Sources and verification

Sources and checks vary by article. Only articles with a recorded check date, scope and sources below were checked, and only within that scope. For other articles, verify against the current docs for your version before applying anything, and email us about errors. Editorial policy · email us

This article has no recorded human check (date and scope) yet.

Published · May 20, 2026

Topic-related official docs (auto-linked · not per-claim sources)OpenAI documentation ↗Anthropic documentation ↗

How we write and review →Editorial policy →

Was this helpful?

Comments

Be the first to comment.