One Client, Many Models: LiteLLM Gateway in Python (2026) — stop rewriting provider SDKs. Use one OpenAI-shaped API across 100+ models, with retries and fallbacks.
If you already wire agents with Pydantic AI or ship tools via FastMCP, LiteLLM is the swap layer underneath: change the model string, keep the call site.
TL;DR
- SDK:
litellm.completion(...)— OpenAI-compatible arguments Router— load balance, retries, fallbacks across deployments- Proxy (optional) — shared HTTP gateway with virtual keys / budgets
Install
pip install litellm
# or: uv add litellm
Example 1 — one completion API
import os
from litellm import completion
# model strings look like "provider/model"
resp = completion(
model="openai/gpt-4o-mini", # or anthropic/..., ollama/..., etc.
messages=[{"role": "user", "content": "Say hello in one sentence."}],
api_key=os.getenv("OPENAI_API_KEY"),
)
print(resp.choices[0].message.content)
Example 2 — Router with fallbacks
import os
from litellm import Router
router = Router(
model_list=[
{
"model_name": "chat",
"litellm_params": {
"model": "openai/gpt-4o-mini",
"api_key": os.environ["OPENAI_API_KEY"],
},
},
{
"model_name": "chat",
"litellm_params": {
"model": "anthropic/claude-sonnet-4-20250514",
"api_key": os.environ["ANTHROPIC_API_KEY"],
},
},
],
num_retries=2,
timeout=30,
)
resp = router.completion(
model="chat",
messages=[{"role": "user", "content": "Summarize LiteLLM in 12 words."}],
)
print(resp.choices[0].message.content)
Optional: LiteLLM Proxy
Run a gateway process, then point any OpenAI client at it:
from openai import OpenAI
client = OpenAI(api_key="sk-your-proxy-key", base_url="http://127.0.0.1:4000")
print(client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": "ping"}],
).choices[0].message.content)
When to use which
- SDK / Router — inside one app or agent process
- Proxy — platform team shared gateway, keys, spend caps
For streaming tokens to browsers, pair with FastAPI SSE.
Production tips
- Keep secrets in env vars; never hardcode keys
- Define explicit fallback chains for outages
- Track cost/latency per model alias
- Prefer stable model IDs; pin versions in config
Wrap-up
LiteLLM is the boring, useful glue of 2026 LLM stacks: one call shape, many backends, Router when you need resilience, Proxy when you need a company gateway.