What is a token, and why should an engineer care about tokens rather than words?
- 1 min — what BPE does and vocabulary sizes
- 2 min — why billing, limits and latency are all per token
- 2 min — language/code ratios and how to measure them
- 2 min — cost formula and per-locale reporting
A token is an entry in the model's subword vocabulary, produced by byte-pair encoding (BPE) or a variant: frequent strings merge into single tokens, rare ones split into pieces. Vocabularies run 32k (Llama 2) to 128k (Llama 3) to ~200k (OpenAI o200k). Everything is billed, limited and timed in tokens: context window, max output, price, rate limits and decode latency.
English prose runs ~1.3 tokens per word (0.75 words/token). Minified JSON, code with long identifiers, UUIDs and non-Latin scripts cost more: on older 32k–100k vocabularies, Hindi or Thai text ran 3–5× the English token count for the same meaning; larger multilingual vocabularies narrow that gap but do not close it. Measure your own payloads.
An e-commerce catalog team generates product descriptions in 6 locales. Finance modelled 220 output tokens per description from English. A 500-item sample per locale through the production tokenizer (illustrative measurement) shows English 220, German 290, Japanese 380, Thai 510. At $15 / M output tokens and 2M descriptions/month, Thai costs $15.30k/month vs the $6.60k modelled — a 2.3× miss caught before launch by a one-hour script.
- Count with the target model's tokenizer (tiktoken for OpenAI, the provider's count-tokens endpoint for Anthropic,
AutoTokenizerfor open weights). A model swap changes the tokenizer, so it silently changes cost, limits and truncation points. - Tokens explain odd failures: letter-counting, spelling, reversing strings and long-number arithmetic are hard because the model never sees characters. Route those to code or a tool.
- Report cost per query by locale and feature: input_tokens × input_price + output_tokens × output_price, with p50 and p95, not a single mean.
- Org scope: the gateway team owns a shared token-accounting library so every product team counts the same way; finance gets per-locale cost from the same source. Never build your own tokenizer — buy/borrow the provider's.
- Where it breaks: character limits used as a proxy for token limits. A 4,000-character cap is ~1,000 English tokens and ~2,500 Thai tokens on an older vocabulary.
2M descriptions/month × output tokens × $15 / M: English 220 → $6.60k; Thai 510 → $15.30k. Input side (800-token prompt × $3 / M × 2M) adds $4.80k per locale regardless of language, so the language effect is concentrated in output.
What the answer sounds like at each level
Knows tokens are subwords, ~0.75 English words per token, and that pricing and limits are in tokens.
Measures per-locale and per-payload token ratios with the real tokenizer and catches the cost impact before launch.
Makes token accounting a shared platform library with per-feature, per-locale dashboards, and re-baselines cost on every model swap.
Follow-ups the interviewer will push
1You switch model families. What changes besides quality?
Tokenizer, so token counts for identical text shift (±10–30% on code and non-English is common), which moves cost, context headroom and any truncation logic. Re-run the token census before cutover.
2Now a regulated tenant in the UAE wants Arabic invoices. What do you check?
Arabic token ratio on real invoice text, output caps sized in tokens not characters, and that right-to-left and mixed-script numbers survive tokenisation and parsing round-trip.
Red flags interviewers listen for
- Equates tokens with words or characters.
- Uses one tokenizer to estimate cost for a different model family.
- Budgets a multilingual launch from English token counts.
How it's scored
| Dimension | What strong looks like |
|---|---|
| Mechanism | Explains subword merging and why rare strings split. |
| Measurement | Insists on the target tokenizer and a real sample per locale/payload type. |
| Cost | Works a per-locale cost example and names the gateway as the source of truth. |
Sources
- Neural Machine Translation of Rare Words with Subword Units — Sennrich et al., 2016paper