Models Don't Read Words. They Read Tokens.

When you send "Hello, how are you?" to Claude or ChatGPT, the model never sees letters or words. It sees a short list of numbers called tokens. Every price you pay, every context limit you hit, and some of the strangest mistakes models make come down to how text is split into tokens.

This guide explains tokenization simply, with examples, and why it matters for cost, speed and multilingual apps.


๐ŸŽฏ Quick Answer (30-Second Read)

  • A token is a chunk of text: a word, part of a word, a punctuation mark or a space-plus-word.
  • Most LLMs use subword tokenizers such as byte-pair encoding (BPE), which learn common chunks from training data.
  • Rough rule for English: 1 token โ‰ˆ ยพ of a word, so 100 tokens โ‰ˆ 75 words. It varies by tokenizer.
  • Non-English languages and code often need more tokens for the same meaning, so they cost more and fill context faster.
  • Pricing and limits (per million tokens, context windows, max output) are all measured in tokens.
  • Tokenization explains why models struggle to count letters or reverse words.

What Tokens Look Like

Here's an illustrative split (exact boundaries depend on the tokenizer):

Text Possible tokens
Hello world Hello ยท world
unbelievable un ยท believ ยท able
tokenization token ยท ization
2026-11-17 202 ยท 6 ยท - ยท 11 ยท - ยท 17
เคจเคฎเคธเฅเคคเฅ‡ several smaller pieces

Common words become single tokens. Rare words, numbers and non-Latin scripts get split into more pieces.


How BPE Learns Tokens

flowchart LR A["Start: every text as
bytes / characters"] --> B["Count the most frequent
adjacent pair, e.g. 't'+'h'"] B --> C["Merge it into a new token 'th'"] C --> D{"Vocabulary
big enough?"} D -->|No| B D -->|Yes| E["Final vocabulary
(often 50kโ€“200k+ tokens)"]

Byte-pair encoding starts with individual bytes or characters and repeatedly merges the most frequent pair into a new token. After many merges, very common words ("the", " and") become single tokens, while rare words remain split into pieces. Because it can fall back to bytes, any text can be encoded, even emoji or unusual scripts.


From Text to Model and Back

sequenceDiagram participant U as Your text participant T as Tokenizer participant M as Model U->>T: "How does UPI work?" T->>M: [token IDs, e.g. 4438, 1587, 12...] M->>M: Predict next token, one at a time M->>T: [new token IDs] T->>U: "UPI routes payments..."

The model only ever predicts the next token from a fixed vocabulary. Output streaming in chat apps is literally tokens arriving one by one.


Why Tokens Matter

1. Cost

API pricing is per million input and output tokens. More tokens = higher bill. (See Claude API Pricing.)

2. Context Limits

A context window (for example, 1M tokens on current Claude models) caps the total tokens of your instructions, documents, conversation and the model's reply.

3. Languages: The "Token Tax"

Watch out for Indian languages: tokenizers trained mostly on English text often split Hindi, Tamil or Bengali into many more tokens than an English sentence of the same meaning. That means higher cost, slower responses and less room in the context window. Measure your real prompts with the provider's token counter.

This is one reason Indian AI companies such as Sarvam build tokenizers designed for Indian languages. (See Why India Needs Bilingual Search Models.)

4. Strange Model Mistakes

"How many r's in strawberry?" The model sees tokens like str ยท aw ยท berry, not individual letters, so counting characters is harder than it looks. Similar issues affect reversing words, exact character limits and some arithmetic on long numbers.


Practical Tips

  • Count tokens, don't guess: use the provider's token-counting endpoint or tokenizer library
  • Trim prompts: remove repeated instructions and unnecessary context
  • Cache long, repeated prefixes (Claude Prompt Caching Guide)
  • Watch JSON and code: brackets, quotes and indentation all cost tokens
  • Test non-English prompts separately for cost and latency
  • For letter-level tasks (counting, exact lengths), let the model use a code tool

Frequently Asked Questions

How many words is 1,000 tokens?

For typical English text, roughly 750 words, using the rule of thumb that one token is about three-quarters of a word. The ratio varies by tokenizer, writing style and language; code and non-English text usually use more tokens per word.

What is BPE tokenization?

Byte-pair encoding is a method that builds a vocabulary by repeatedly merging the most frequent adjacent pairs of characters or bytes in training text. Common words become single tokens and rare words split into several, and any text can be represented using byte fallbacks.

Why do LLMs struggle to count letters in words?

Models process tokens, which are often chunks of several characters, rather than individual letters. They don't directly see a word's spelling, so tasks like counting letters or reversing strings are harder. Letting the model run code solves these reliably.


Conclusion

Tokens are the unit everything in an LLM is measured in: what the model reads, what it writes, what you pay and how much fits in context. Subword tokenizers like BPE split common words into single tokens and rare text into many, which is why Indian languages and code can cost more. Count your tokens, trim and cache prompts, and use tools for letter-level tasks.


Sources

Last verified: 2 November 2026.


Related reads: Claude API Pricing ยท Claude Prompt Caching Guide ยท Vector Databases Explained ยท Sarvam Bulbul TTS Guide

I write about AI and how real products work. Follow on LinkedIn or X, or invite me to speak.