LiteLLM alternatives, and when you should not switch

At a glance

Why people go looking for a LiteLLM alternative

LiteLLM is the default open-source LLM gateway for a reason: one OpenAI-compatible endpoint in front of OpenAI, Anthropic, Gemini, Bedrock, Azure, Groq, local servers and a long tail of others, with virtual keys, per-key budgets, fallbacks and a spend log. The complaints come after it goes into production, and they fall into three groups.

Operating it. A real deployment is the proxy plus Postgres. Releases land several times a week, each can carry database migrations, and a careless upgrade on a Friday is how a gateway that every application depends on goes down. There is also LITELLM_SALT_KEY: it encrypts the provider keys stored in the database and cannot be rotated once models exist. Lose it, and every stored credential has to be entered again. Log full prompts and responses and the database grows faster than anyone planned for.

The enterprise line. SSO is free for up to five users; beyond that, and for the other features LiteLLM marks as enterprise, you need their licence. For a 40-person company that wants Okta login on the admin UI, that is the moment someone opens a search tab.

Overhead at scale. A Python proxy adds a few milliseconds per request. For a few hundred requests per second nobody notices — the model takes seconds to answer. For a high-volume service with a tight latency budget, gateways written in Go or Rust are measurably leaner.

Most of these problems are not about the software

Look at that list again. Migrations, upgrades, the salt key, database growth, backups — none of that goes away by switching to another self-hosted gateway. Bifrost, TensorZero and Portkey all need a server, a database or config store, upgrades and monitoring too. You would trade a well-known set of operating tasks for a less-known one, and rewrite your routing config on the way.

If the honest reason you are looking is “we are tired of running this”, the alternative is the same LiteLLM, run by someone whose job it is.

What managed LiteLLM looks like here

Each customer gets a dedicated instance: the LiteLLM proxy on port 4000, its own Postgres 16, TLS on your domain or ours. Nothing is shared with other customers, and we do not read the spend log or the prompts in it.

Keys handled properly. LITELLM_MASTER_KEY and LITELLM_SALT_KEY are generated at setup and kept with your instance secrets. The salt never changes during upgrades, so stored provider credentials keep decrypting.

Upgrades without surprises. We follow the main-stable image, run migrations before the new version starts serving, and keep virtual keys, teams, budgets and spend history in Postgres across versions. A release that breaks something is not rolled out underneath you.

Nightly off-box backups of the database, so a bad migration or a deleted team is a restore, not a rebuild.

Plans: Starter, 2 GB, $5.99 — one team routing a handful of providers. Plus, 4 GB, $9.99 — several teams with spend logging on every call. Pro, 8 GB, $16.99 — heavy traffic or long request logs. Provider tokens are billed by your provider at list price; we add nothing to them.

When another gateway really is the better choice

Bifrost if throughput is the actual problem: thousands of requests per second where the gateway’s own overhead shows up in your latency graphs. Measure first — in most stacks the provider is the slow part, not the proxy.

TensorZero if you want the gateway to be where experiments happen: routing a share of traffic to a new prompt or model, collecting feedback, and improving from it.

Kong AI Gateway if your platform team already runs Kong. Adding AI plugins to a gateway you operate anyway beats introducing a second one.

Portkey if you want its guardrails and prompt management and are happy to buy the hosted platform that sits on top of the open-source gateway.

OpenRouter if you want zero infrastructure and access to models you have no account with. Our LiteLLM vs OpenRouter page covers that trade in detail — and the two combine well, with OpenRouter as one provider behind LiteLLM.

We do not host these five. If one of them is the right answer for you, use it.

Moving an existing LiteLLM deployment

Send your config.yaml and we mount it, so model_list entries and router settings work as before. To keep virtual keys, teams and spend history, add a pg_dump of the LiteLLM database and the same LITELLM_SALT_KEY — without the matching salt, stored provider credentials will not decrypt.

Your applications only change one thing: the base URL. Virtual keys issued before the move keep working once the database is restored, so teams do not have to be re-issued keys.

What to put next to the gateway

Langfuse for traces and cost per feature — LiteLLM can send every call to it with one callback, so you get full coverage without instrumenting each application. Opik if you need evaluations that fail a build when quality drops.

Ollama behind the same endpoint for cheap, high-volume or sensitive calls, while a frontier model handles the hard ones. LiteLLM routes by model name, so applications do not know or care which is which. Our guide to cutting LLM costs shows the routing patterns that save the most.

The verdict

If LiteLLM does what you need and the pain is running it, do not switch — hand it off. Managed LiteLLM on a dedicated instance starts at $5.99 a month, and moving an existing deployment keeps your keys, teams and spend history.

Switch software only for a reason LiteLLM cannot fix: extreme throughput (Bifrost), experimentation in the gateway (TensorZero), an existing Kong estate, or no infrastructure at all (OpenRouter).

Mentioned on this page

LiteLLM

One OpenAI-compatible endpoint in front of every model provider you use.

from $5.99/moDetails →

Langfuse

Tracing, evaluation and prompt management for LLM applications.

from $11.99/moDetails →

Ollama

Your private LLM server — pull a model, get an API, keep every token on your own box.

from $16.99/moDetails →

Opik

Open-source evaluation and monitoring for LLM applications.

from $11.99/moDetails →