Token
The unit an LLM reads and writes: a word or word fragment. Context limits, latency and price are all metered in tokens, not characters.
A token is the small chunk of text a language model actually works with. Common words are usually one token ('invoice'), rarer words are split into pieces ('tokenization' might become 'token' + 'ization'), and spaces and punctuation count too.
Everything is measured in tokens: how much fits in the Context window, how long a reply takes, and how much you pay.
LEGO bricks for text: the model never sees whole sentences, only the bricks they were snapped together from.
Budgeting with 1 token ≈ 1 word or chars ÷ 4 everywhere. The ratio swings 2–3× by language and format, and budgets built on it break the first week of real traffic.
Tokenizers are byte-level BPE or SentencePiece models learned from a corpus: the most frequent byte sequences get merged into single vocabulary entries. Vocabulary sizes in current families: Llama 3 uses 128,256 entries, OpenAI's o200k_base about 200k, Gemma 256k. A bigger vocabulary means fewer tokens per sentence, especially for non-English text, at the cost of a larger embedding table.
For English prose expect ~4 characters or ~0.75 words per token. Source code runs ~3–3.5 characters per token, JSON full of UUIDs and numbers ~2–2.5, and Indic, Thai or Arabic scripts need 1.3–3× the tokens of the same meaning in English depending on the tokenizer, which raises cost and latency for those users. Tokenization explains classic quirks: weak letter counting, unstable arithmetic on long numbers, and sensitivity to leading spaces. Counts are not portable across model families, so always count with the target model's tokenizer.
Count with the provider's token-counting endpoint or the model's own tokenizer (tiktoken for OpenAI models, the Hugging Face tokenizer for open weights). Size chunks in tokens using the embedding model's tokenizer, not characters. Log input and output token counts per request, tagged by locale and route.
A 10-K annual report of ~60,000 words is ~80,000 tokens with an o200k-class tokenizer. Sending it whole at an assumed $3 / M input costs $0.24 per call. The same filing's financial tables exported as JSON with full key names come to ~2× the tokens of the equivalent CSV, so a pipeline that reads 5,000 filings a night spends ~$1,200 instead of ~$600 for nothing but key repetition.
In any estimation round, convert requirements into tokens first: tokens per request × requests per second × price. Say you would measure on 500 real samples with the production tokenizer, because multilingual and JSON-heavy traffic can be 2× the 'chars ÷ 4' estimate. The follow-up is usually 'what changes for a Hindi or Japanese market?': answer with a per-locale token budget and a tokenizer comparison across candidate models.
| Content | Chars per token | Tokens vs English prose |
|---|---|---|
| English prose | ≈ 4.0 | 1.0× |
| Python source | ≈ 3.2 | 1.25× |
| JSON with UUIDs | ≈ 2.3 | 1.7× |
| German / French prose | ≈ 3.3 | 1.2× |
| Hindi (Devanagari) | varies | 1.3–3× |
Numbers worth knowing
- English prose
- ≈ 4 chars or 0.75 words per token100k–200k vocabulary BPE tokenizers
- Non-English multiplier
- 1.3–3× tokens for the same meaningIndic, Thai, Arabic scripts; lower end on newer 200k+ vocabularies
- Vocabulary size
- 128k–256k entriesLlama 3: 128,256; `o200k_base`: ~200k; Gemma: 256k
Try the numbers
Illustrative list prices — check your provider. Assumes cache reads at 0.1× input and misses re-writing the prefix at 1.25× (a 5-minute TTL); batch APIs typically take ~50% off on top.
How to decide
| When | Do this |
|---|---|
| Sizing a new feature's cost or context budget | Tokenize 500 real samples with the target model; never use chars ÷ 4 for anything that ships. |
| More than 20% of traffic is non-English | Budget and rate-limit per locale, and compare tokenizers across candidate models before choosing one. |
| The model reads large structured payloads | Send CSV or compact JSON with short keys and measure; switching formats often cuts tokens 30–50%. |
Cost maths
Monthly cost = requests × (input_tokens × p_in + output_tokens × p_out). Assume $3 / M input (illustrative, Oct 2026). A support route at 1M requests/month with 2,000 English input tokens costs $6,000/month in input. Launch the same flow in a market where the tokenizer needs 1.6× the tokens and input becomes $9,600/month: the same budget serves 38% fewer conversations.
Operating it
- Tokens per request p95 grows more than 30% after a new locale or data source launches
- Embedding-time truncation count is non-zero
- Requests rejected for exceeding the context window above 0.1%
- Input tokens per request p95 by locale and route: alert on +30% week over week
- Chunks truncated at embedding time: alert on any non-zero count
- Context-length-exceeded errors: alert above 0.1% of requests
When it goes wrong
Recall on Japanese developer-docs queries was 0.41 against 0.83 for English, with no errors in any log.
Per-language recall@10 on the golden set, added after support escalations from the Tokyo office.
The chunker split at 2,000 characters assuming ~500 tokens; Japanese chunks came out at ~1,300 tokens and the 512-token embedding model silently truncated the last 60% of each one.
Token-based chunking with the embedding model's own tokenizer, an ingest-time assertion that every chunk is ≤ max sequence length, and a truncation counter on the dashboard.
Where it shows up
Sources
- Neural Machine Translation of Rare Words with Subword Units — Sennrich et al., 2015paper
- tiktoken: fast BPE tokeniser for OpenAI models — OpenAI, 2023docs
Check yourself
Three questions. Every option is explained once you answer.