Running an LLM on CPU: what actually works

Everyone asks whether you can run a language model without a GPU. You can. The useful question is which jobs it is fast enough for — and this page answers that with numbers rather than enthusiasm.

The numbers, up front

On dedicated modern server cores, a quantized 7-8B model runs at roughly 5-15 tokens per second. A 13B quantized model on twelve gigabytes is slower again, usable for batch work rather than conversation. Anything at 70B is not a CPU proposition.

For scale: average reading speed is around 4-5 words per second, and a token is roughly three-quarters of a word. So 10 tokens/sec is about reading pace — fine when someone is reading the answer, frustrating when they are waiting for it to finish before acting.

Memory: how big a model fits

The rule of thumb is model file size plus room for context. A 7-8B model quantized to four bits is roughly 4-5 GB on disk and wants similar in memory, so 8 GB holds it with context headroom. A 13B at the same quantization wants 12 GB.

Context length costs memory too, and it is the setting people forget. Asking for a very long context on a small plan is a common cause of a model that loads and then dies mid-answer.

Quantization is the lever that makes any of this practical. Four-bit builds lose a little quality against the full-precision original — usually invisible in summarising, extraction and classification, occasionally visible in careful reasoning.

Which workloads fit comfortably

Classification and extraction. Short prompts, short answers, high volume — CPU is excellent here and the cost model is unbeatable, because nothing is metered.

RAG answers. Retrieval does the heavy lifting; the model writes a paragraph from the chunks. Perfectly comfortable.

Automation steps. A workflow that summarises an email or decides a category does not care about latency.

Internal chat. Acceptable, with the honest caveat that people used to a frontier model will notice both the speed and the capability gap.

Drafting long documents. Works, but the wait is real. Better as a background job than an interactive one.

Which workloads do not

Anything customer-facing where someone waits on the first token. Anything needing a large model’s reasoning. Anything with real concurrency — CPU inference does not batch the way a GPU does, so ten simultaneous users means roughly ten times the wait.

If that is your situation, rent GPU capacity from a specialist. We would rather say so than sell you a plan that disappoints, and it is why our catalogue lists vLLM without selling it.

A sensible hybrid

The arrangement that works for most teams is not all-local or all-cloud. Put a gateway such as LiteLLM in front, route high-volume and sensitive calls to a local Ollama endpoint, and send the genuinely hard questions to a frontier model. Applications see one endpoint and never need to know.

That way the flat-cost instance absorbs the traffic that would otherwise dominate the bill, and the expensive model is reserved for the work that justifies it.

Related reading

Ollama

Your private LLM server — pull a model, get an API, keep every token on your own box.

from $16.99/moDetails →

llama.cpp

The reference GGUF inference server, with nothing between it and the CPU.

from $13.99/moDetails →

LocalAI

A drop-in OpenAI replacement for text, images, audio and embeddings.

from $16.99/moDetails →

Open WebUI

A ChatGPT-style workspace for your team, pointed at any model you like.

from $5.99/moDetails →