Token counter and estimator

Language models do not read characters or words; they read tokens, which are fragments produced by a tokenizer specific to each model family. That is why the same paragraph counts differently on different providers, and why an exact number requires running that provider's own tokenizer.

This is an estimator and says so. It applies well-established ratios — around four characters per token for ordinary English prose, more tokens per word for code, markup, names and non-English text — and shows a range. For billing decisions, treat it as a planning figure; for the exact count, use your provider's tokenizer.

Why the number changes per model

Tokenizers are trained alongside the model on a particular corpus. A vocabulary trained mostly on English splits Turkish, Japanese or Arabic into far more pieces — two to three times as many tokens for the same meaning is common. Code sits between the two: frequent keywords compress well, long identifiers do not.

This matters commercially. A product serving a non-English market pays noticeably more per sentence than the same product in English, and no amount of prompt tuning changes that.

Using this to size a context window

The other everyday use of a token count is fitting things into a context window. Take the estimate, add your system prompt, add whatever retrieval will inject, and leave room for the answer — the output shares the same budget on most models.

If the total is uncomfortably close to the limit, the fix is almost always better retrieval rather than a bigger window: three relevant paragraphs beat forty pages, and they cost a fraction as much. That is the argument for a proper vector store over pasting documents in whole.