Cutting an LLM bill without cutting quality

An LLM bill that surprises you is almost never caused by the thing you suspect. Here is where the money actually goes, in rough order of size, and what to do about each.

First, find out where it goes

You cannot cut what you cannot attribute. Before changing anything, route traffic through a gateway that logs per key and per model, and give each application its own virtual key. Within a week you will have the answer, and it is usually not the feature anyone suspected — it is a background job that retries, or an agent loop nobody bounded.

LiteLLM does this for $5.99 a month. The spend log alone tends to pay for it in the first fortnight.

1. Route by difficulty, not by habit

Most production traffic is not hard. Classification, extraction, routing, short summaries and reformatting are all comfortably handled by a small model. Sending them to a frontier model because that is what the prototype used is the single most common source of waste.

Route by model name at the gateway and the application never changes.

2. Move the volume work local

Once high-volume, low-difficulty calls are identified, a local model absorbs them at flat cost. An Ollama instance at $16.99 replaces a lot of metered tokens, and the CPU performance figures tell you honestly which jobs it handles.

3. Cut the context, not the model

Input tokens are billed too, and context is where they hide: entire documents pasted where a retrieved chunk would do, conversation history resent in full on every turn, system prompts that grew for a year without review.

Retrieval is a cost control as much as a quality feature — sending three relevant paragraphs instead of forty pages changes the arithmetic completely.

4. Cache the repeated calls

Identical or near-identical requests are more common than teams expect, particularly behind automation and batch jobs. A cache at the gateway turns repeats into nothing. Where a provider offers prompt caching for long shared prefixes, use it — it targets exactly the system-prompt weight described above.

5. Bound your agents

Agent loops are the failure mode that produces alarming invoices: a plan-act-observe cycle with no step limit, retrying a tool that keeps failing. Cap iterations, cap retries, and set a hard budget per key so the worst case is a stopped job rather than an open tab.

6. Watch for the retry you did not write

SDKs retry. Queues retry. Workflow platforms retry. A failing call can quietly cost three times what it appears to, and none of it shows up in application logs. Traces make this visible in minutes; guesswork never finds it.

7. Reconsider what needs a model at all

A surprising share of production LLM calls are doing work a regular expression, a lookup table or a small classifier would do faster, cheaper and more predictably. It is worth one afternoon of review per quarter.

Related reading

LiteLLM

One OpenAI-compatible endpoint in front of every model provider you use.

from $5.99/moDetails →

Ollama

Your private LLM server — pull a model, get an API, keep every token on your own box.

from $16.99/moDetails →

Langfuse

Tracing, evaluation and prompt management for LLM applications.

from $11.99/moDetails →