miatz™
Log inSit Chuun
Free concept playground · no account

Token Shredder

Type a word, watch AI shred it into tokens.

All 90 playgrounds Join the free cohort

Token Shredder

AvailableFree

Type or paste any text. It shreds into the tokens a model actually reads — not words, not letters. Long words shatter into pieces, digits and punctuation stand alone, and a leading space fuses into the next piece, just like a real tokenizer.

3tokens
1words
10characters
3tokens / word
3.33chars / token
What the model reads — 3 tokens
stra64669wber5163ry19588
What you read — 10 letters
strawberry
Why AI can't spell your wordtokens are opaque IDs, not letters

You typed 10 letters, but the model never sees them. It receives 3 opaque token IDs — stra · wber · ry. The letters are already gone by the time it “reads.”

That is the whole mechanism behind “how many R’s are in strawberry?” going wrong: the model is counting inside chunks like stra that hide the individual characters — it is guessing at a spelling it was never shown. Not magic, not stupidity: it simply operates one layer above letters.

Share on WhatsApp
Was this playground useful?

Token Shredder

Type any word and watch it shred into tokens — the units a language model actually reads. Tokens are not words and not letters: long words shatter into ~4-character pieces, digits and punctuation stand on their own, and a leading space fuses into the next piece exactly the way a production tokenizer encodes " word" as one token. You get a live token count, a words-to-tokens ratio, a chars-per-token readout, and a side-by-side letter view so the gap between what you read and what the model receives is visible, not asserted. Runs entirely in your browser (0 uploads, works offline).

How to use it

  1. Type or paste text into the box, or load a preset — strawberry, a long word, or a sentence.
  2. Read the stat row: tokens, words, characters, plus tokens / word and chars / token.
  3. Scan the token strip (what the model reads) against the letter view (what you read) — the two counts rarely match.
  4. Toggle Show token IDs to see each token as the opaque number the model actually operates on.
  5. Open the X-ray panel to see, on your own word, why the model can't reliably spell or count letters.

What this clears up (the fundamentals)

  • A token is not a word or a letter. "strawberry" is one word and ten letters, but three tokens. The unit the model reasons over is its own — and this tool shows you exactly where the cuts fall.
  • Long and rare words cost more tokens. Short common words are usually one token; long compounds shatter into pieces, so tokens / word climbs and your context (and bill) fills faster.
  • The leading space belongs to the token. Real tokenizers encode " the" as a single token, so spacing changes the split. This is why prompt formatting quietly shifts token counts.
  • This is why AI "can't spell." The model never sees your letters — only opaque token IDs. Counting the R's in a word it received as chunks like stra · wber · ry is guessing at a spelling it was never shown.

Where it's used

A top-of-funnel AI-literacy check and the fastest way to feel why a famous failure mode (miscounting letters) happens — a mechanism, not a meme. It's a miatz build-lab concept playable: play it here, then learn to build the heuristic tokenizer and the "why AI can't spell" reveal yourself in AIE-101's opening tokens-and-vocabulary lesson.

FAQ

What exactly is a token?

The atomic unit a language model reads and predicts. Modern models use subword tokens: common words are a single token, while rarer or longer words are split into pieces. The model never sees raw characters — it sees a sequence of token IDs.

Why is the token count different from my word and character counts?

Because tokens sit between words and characters. Whitespace-word count and character count are shown alongside so you can watch a single word become several tokens, and several characters collapse into one.

Why does this prove AI can't spell?

If the model receives "strawberry" as the tokens stra · wber · ry, the individual letters are already gone. Asking it to count R's forces it to reconstruct a spelling it was never given — which is exactly where models slip. The tool shows this on your word, not a screenshot of someone else's.

Is this a real GPT tokenizer?

No — it's a documented heuristic approximation so it can run instantly, fully offline, with no vocabulary download. The cut points won't match GPT-4, Llama, or Claude exactly, but the lesson (tokens != words != letters, long words fragment, spaces fuse) is the real behavior.

Is anything uploaded?

No. Your text is shredded entirely in your browser — nothing is transmitted, stored, or logged. Turn off your Wi-Fi and it still works.

Limits

This is a teaching heuristic, not a real BPE vocabulary. It splits long ASCII words into fixed ~4-character pieces rather than using a learned merge table, so exact token counts and IDs are illustrative — a real tokenizer would split differently and its IDs are model-specific. Non-Latin scripts (Hindi, Tamil, emoji) are treated as punctuation here; real tokenizers fragment them heavily, which is its own well-known "non-English token tax." Use it to build intuition, not to price a production prompt to the token.

Related

Part of the Demystify Playgrounds. Explore the rest from the Playgrounds home.

Bookmark this page (Ctrl+D, or ⌘D on Mac) — it works offline the next time you need it.

More playgrounds

Ninety playgrounds. Zero setup.

Every concept here is playable free, no account — and inside the program you learn to rebuild the machinery yourself.

Browse all playgrounds Join free