Inside a Tokenizer: How AI Actually Reads Your Words

Here's a fact that explains half the weird behavior you've ever seen from a language model: it has never read a word in its life.

Before your prompt reaches the model, a tokenizer chops it into tokens — chunks that might be a whole word, a piece of one, a space, or punctuation. The model sees only a sequence of token IDs: 'The cat sat' might become something like [464, 2415, 3332]. Everything the model does — reasoning, pricing, context limits — happens in token-space, not word-space.

If you work with LLMs and don't understand tokenization, you're debugging a system whose input format you've never looked at.

Why not just use words?

Two bad options bracket the design space. One token per word means a vocabulary of millions, and any typo, new name, or rare word becomes an unknown blank. One token per character keeps the vocabulary tiny but makes every text enormously long, and models pay quadratic attention cost over sequence length.

The answer in most modern models is byte-pair encoding, BPE — a compromise discovered by, essentially, compression logic. Start with individual bytes. Scan a mountain of training text. Find the most frequent adjacent pair, merge it into a new token. Repeat tens of thousands of times. Frequent strings ('the', 'ing', ' and') end up as single tokens; rare strings get spelled out from pieces. Common words are cheap, and nothing is ever unrepresentable, because in the worst case the tokenizer falls back to raw bytes.

The vocabulary that falls out is wonderfully unsentimental. It doesn't care about linguistics; it cares about frequency. 'Tokenization' might split into 'token' + 'ization'. A leading space is usually part of the token — ' cat' and 'cat' are different tokens with different IDs.

The strawberry problem, explained

Ask a model to count the letter 'r' in 'strawberry' and it may stumble. People cite this as proof models can't reason. Mostly, it's proof they can't see. If 'strawberry' arrives as two tokens — say 'str' + 'awberry' — the model receives two opaque IDs. There are no letters in its input to count. It knows about spelling only indirectly, from training text that happened to discuss it. It's like asking someone to count the letters in a word they've only ever heard spoken.

The same lens explains other classics:

• Arithmetic gets flaky partly because numbers tokenize inconsistently — '2024' might be one token while '2025' splits in two.
• Models handle English more cheaply than most other languages, because BPE vocabularies are trained on English-heavy corpora; the same sentence in another script can cost several times the tokens.
• Reversing a string, rhyming, and wordplay are all hard for the same reason: they're operations on characters, and the model doesn't get characters.

Why your bill and your context window care

Tokens are the billing unit and the budget unit. API pricing is per token, in and out. Context windows are token counts, not word counts. As a rough rule for English prose, one token is about three-quarters of a word — so a '128k context' is roughly 96,000 English words, less for code, much less for some languages.

This is where tokenization stops being trivia and starts being engineering. If you're building a [RAG](/glossary/retrieval-augmented-generation) system, your chunk sizes are token budgets. If your agent keeps blowing its context, the fix starts with knowing what's actually expensive in token-space. Verbose JSON keys, base64 blobs, and deeply indented code all cost more than they look.

Ten minutes that will change how you prompt

None of this sticks from reading. It sticks from seeing your own words come apart.

We built a free [tokenizer playground](/play/tokenizer) for exactly that. Paste anything — your last prompt, a poem, some JSON, another language — and watch it split, live, with per-token coloring and counts. Try 'strawberry'. Try the same sentence with and without a leading space. Try your name.

Then try the exercise we give every learner in the [Applied AI track](/applied-ai): take a prompt you use often and cut its token count by a third without losing meaning. You'll never look at a wall of boilerplate instructions the same way again.

The model reads tokens. Once you do too, you're both speaking the same language.