Gateways & LLM ops
Route every provider through one endpoint and see what it costs.
An LLM gateway is a proxy presenting one OpenAI-compatible endpoint in front of every provider you use, adding virtual keys, per-key budgets, fallbacks and a spend log. LiteLLM is the standard choice; Langfuse and Opik add tracing and evaluation.
Once more than one team is calling more than one provider, three problems appear on schedule: keys spread into places nobody tracks, spend cannot be attributed to a feature, and a provider outage takes everything with it. A gateway solves all three without touching application code, because the base URL never changes while what sits behind it does.
The observability half matters just as much when agents are involved. One user request can become dozens of model calls, and without traces the only debugging tool available is a guess. Running both yourself means the prompt log — often the most sensitive record a company has — stays on your instance.
Gateways & LLM ops we host
Choosing between these
Straight answers
Does a gateway add latency?
Can I put local and commercial models behind one endpoint?
Put your AI stack on your own box
Pick an app, pick a size, and have it running today. Month to month, cancel whenever.