Models Don't Read Words. They Read Tokens.
When you send "Hello, how are you?" to Claude or ChatGPT, the model never sees letters or words. It sees a short list of numbers called tokens. Every price you pay, every context limit you hit, and some of the strangest mistakes models make come down to how text is split into tokens.
This guide explains tokenization simply, with examples, and why it matters for cost, speed and multilingual apps.
๐ฏ Quick Answer (30-Second Read)
- A token is a chunk of text: a word, part of a word, a punctuation mark or a space-plus-word.
- Most LLMs use subword tokenizers such as byte-pair encoding (BPE), which learn common chunks from training data.
- Rough rule for English: 1 token โ ยพ of a word, so 100 tokens โ 75 words. It varies by tokenizer.
- Non-English languages and code often need more tokens for the same meaning, so they cost more and fill context faster.
- Pricing and limits (per million tokens, context windows, max output) are all measured in tokens.
- Tokenization explains why models struggle to count letters or reverse words.
What Tokens Look Like
Here's an illustrative split (exact boundaries depend on the tokenizer):
| Text | Possible tokens |
|---|---|
Hello world |
Hello ยท world |
unbelievable |
un ยท believ ยท able |
tokenization |
token ยท ization |
2026-11-17 |
202 ยท 6 ยท - ยท 11 ยท - ยท 17 |
เคจเคฎเคธเฅเคคเฅ |
several smaller pieces |
Common words become single tokens. Rare words, numbers and non-Latin scripts get split into more pieces.
How BPE Learns Tokens
bytes / characters"] --> B["Count the most frequent
adjacent pair, e.g. 't'+'h'"] B --> C["Merge it into a new token 'th'"] C --> D{"Vocabulary
big enough?"} D -->|No| B D -->|Yes| E["Final vocabulary
(often 50kโ200k+ tokens)"]
Byte-pair encoding starts with individual bytes or characters and repeatedly merges the most frequent pair into a new token. After many merges, very common words ("the", " and") become single tokens, while rare words remain split into pieces. Because it can fall back to bytes, any text can be encoded, even emoji or unusual scripts.
From Text to Model and Back
The model only ever predicts the next token from a fixed vocabulary. Output streaming in chat apps is literally tokens arriving one by one.
Why Tokens Matter
1. Cost
API pricing is per million input and output tokens. More tokens = higher bill. (See Claude API Pricing.)
2. Context Limits
A context window (for example, 1M tokens on current Claude models) caps the total tokens of your instructions, documents, conversation and the model's reply.
3. Languages: The "Token Tax"
This is one reason Indian AI companies such as Sarvam build tokenizers designed for Indian languages. (See Why India Needs Bilingual Search Models.)
4. Strange Model Mistakes
"How many r's in strawberry?" The model sees tokens like str ยท aw ยท berry, not individual letters, so counting characters is harder than it looks. Similar issues affect reversing words, exact character limits and some arithmetic on long numbers.
Practical Tips
- Count tokens, don't guess: use the provider's token-counting endpoint or tokenizer library
- Trim prompts: remove repeated instructions and unnecessary context
- Cache long, repeated prefixes (Claude Prompt Caching Guide)
- Watch JSON and code: brackets, quotes and indentation all cost tokens
- Test non-English prompts separately for cost and latency
- For letter-level tasks (counting, exact lengths), let the model use a code tool
Frequently Asked Questions
How many words is 1,000 tokens?
For typical English text, roughly 750 words, using the rule of thumb that one token is about three-quarters of a word. The ratio varies by tokenizer, writing style and language; code and non-English text usually use more tokens per word.
What is BPE tokenization?
Byte-pair encoding is a method that builds a vocabulary by repeatedly merging the most frequent adjacent pairs of characters or bytes in training text. Common words become single tokens and rare words split into several, and any text can be represented using byte fallbacks.
Why do LLMs struggle to count letters in words?
Models process tokens, which are often chunks of several characters, rather than individual letters. They don't directly see a word's spelling, so tasks like counting letters or reversing strings are harder. Letting the model run code solves these reliably.
Conclusion
Tokens are the unit everything in an LLM is measured in: what the model reads, what it writes, what you pay and how much fits in context. Subword tokenizers like BPE split common words into single tokens and rare text into many, which is why Indian languages and code can cost more. Count your tokens, trim and cache prompts, and use tools for letter-level tasks.
Sources
- Sennrich, Haddow & Birch: Neural Machine Translation of Rare Words with Subword Units (BPE, 2016)
- Anthropic docs: Token counting
- Hugging Face course: Byte-Pair Encoding tokenization
Last verified: 2 November 2026.
Related reads: Claude API Pricing ยท Claude Prompt Caching Guide ยท Vector Databases Explained ยท Sarvam Bulbul TTS Guide
I write about AI and how real products work. Follow on LinkedIn or X, or invite me to speak.