See your words the way the model sees them
Every prompt you have ever written was shredded into tokens before the model read it. Paste anything below and watch it happen.
What is this, why does it matter, and how do I use this lab?expand
What it is
Tokens are the units a language model actually reads and writes — not letters, not words, but pieces from a fixed vocabulary learned by byte-pair encoding (BPE). 'Deploying' might be one token; an emoji might be three; a Hindi word might be six. Every limit you hear about models — context windows, pricing, rate limits, max output — is denominated in tokens, and this lab makes them visible.
What this lab makes clear
- • Why token count ≠ word count, and why the ratio changes wildly between English, code, emoji and Indic scripts
- • Why the same prompt costs different amounts in different languages — the vocabulary was optimised on mostly-English text, so other scripts fragment into more tokens
- • What a context window really is: a hard budget of tokens shared by your system prompt, the conversation history, retrieved context AND the answer
- • Why API bills are quoted per million tokens, and how to estimate the cost of a feature before you build it
- • Why models sometimes butcher rare words, URLs or non-Latin names — they see them as awkward multi-token fragments, not familiar units
Where you meet it in the real world
- • Every LLM API bill (OpenAI, Anthropic, NVIDIA NIM) is metered in input + output tokens — cost engineering starts with token counting
- • RAG pipelines budget tokens explicitly: retrieved chunks are capped so the context + question + answer fit the window
- • Chat products truncate or summarise history when the token budget runs out — that's why long conversations 'forget' their start
- • Multilingual products pay a hidden tax: the same UX in Telugu can cost 3-5× the tokens of English, which changes model choice and pricing
Why it matters for you as an engineer
Token blindness is the most common source of surprise in LLM engineering: prompts that silently truncate, features that cost 10× the estimate, latency that scales with output length nobody measured. An engineer who can eyeball text and estimate its token weight designs better prompts, budgets context windows properly, and never gets ambushed by the invoice.
How to use this lab
- 1. Type a sentence and watch it split into coloured token pieces in real time — hover any piece for its token id
- 2. Press the English, Code, Emoji, Hindi and Telugu samples in turn and compare the chars-per-token ratio
- 3. Open the cost calculator, put in your real per-million-token prices, and see what 1,000 calls of your current text cost
- 4. Watch the context-window meter: paste a long document and see how much of an 8K vs 128K window it eats
- 5. Try a URL or an unusual name and see how many fragments it becomes — that's what the model 'sees'
Models don't read your words — they read tokens, and everything you pay for, fit in a window, or wait on is counted in them.
What the model sees
loading tokenizer…
What would this prompt cost?
API pricing is per million tokens — input (your prompt) and output (the reply) are billed at different rates.
Context-window budget
How much of a model's context window would this text alone consume?
Every message in a chat — plus system prompts, retrieved documents and the model's own replies — shares this one budget. That's why a long conversation with a few big pasted documents can suddenly hit the limit: the window fills up token by token, not message by message.
Tokens are the model's alphabet: byte-pair encoding (BPE) merges frequent byte sequences into single vocabulary entries, so common English words often cost just one token. Indic scripts like Devanagari and Telugu — and emoji — appear far less often in the training data behind the vocabulary, so they fragment into many byte-level tokens, inflating both cost and context usage for the same meaning. This lab uses OpenAI's cl100k_base vocabulary as a representative tokenizer; other models' tokenizers differ slightly but show the same pattern.
What is a token?
A token is the unit a language model actually reads — a frequent chunk of characters (often a word piece) mapped to a number. Models never see letters or words, only token IDs.
Why do LLMs miscount the letters in a word?
Because the model sees token IDs, not characters. If 'strawberry' arrives as two tokens, the model never looks at individual letters unless it reasons its way there.
Why do tokens matter for cost and context windows?
Model pricing and context limits are counted in tokens. The same paragraph can be 30% more tokens in one language or format than another — which is real money at scale.
Deeper dive: token, defined properly
Next: watch a document become retrievable
The Chunking Lab shows how RAG systems slice documents — the step that decides what an AI can and cannot find.