Every RAG answer starts with a knife
Before an AI can retrieve knowledge, someone sliced the documents. Slice them yourself — four splitters, live controls — and see why so many RAG systems miss.
What is this, why does it matter, and how do I use this lab?expand
What it is
Chunking is the step where a long document is cut into smaller pieces before being stored in a retrieval system. A language model can only usefully attend to a limited amount of text at once, and a retrieval system can only return whole pieces — so the shape of those pieces decides what any RAG system can ever find. A chunk is the atom of retrieval: if the answer to a question straddles two chunks, no amount of clever search puts it back together.
What this lab makes clear
- • Why 'just split every 500 characters' breaks sentences and destroys meaning at the boundaries
- • What overlap actually is — the same text deliberately stored in two neighbouring chunks so context survives the cut (the green regions in this lab)
- • Why recursive splitting (paragraphs first, then lines, then words) is the industry default: it respects the document's own structure
- • Why chunk size is a trade-off: small chunks retrieve precisely but lose context; big chunks carry context but dilute the match and waste the model's window
- • Why markdown and code need their own splitters — a function or a section heading is a natural boundary a character counter can't see
Where you meet it in the real world
- • Every document Q&A product (Notion AI, Copilot for Docs, support-bot knowledge bases) chunks its corpus exactly like this before embedding
- • This platform's own tutor does it: lessons and case studies are split with a recursive splitter (800 chars, 100 overlap) before being embedded into the index you query in the RAG Lab
- • Code-search tools chunk by function/class boundaries so a retrieved result is a runnable unit, not half a loop
- • Legal/medical AI systems tune chunking per document type — a contract clause and a lab report have very different natural atoms
Why it matters for you as an engineer
Most RAG failures people blame on 'the model' are actually chunking failures: the retriever returned a fragment whose sentence was cut in half, or a chunk so large the relevant line drowned in noise. Being able to look at a document and predict how it will chunk — and what questions will therefore be unanswerable — is a core skill when you build or debug anything with retrieval in it.
How to use this lab
- 1. Load the Knight Capital case study and keep the default recursive splitter — notice chunks end at paragraph boundaries
- 2. Switch to the Character splitter and find a sentence sliced mid-word; that fragment is what a retriever would return
- 3. Drag overlap from 0 upward and watch the green regions appear — that duplicated text is the safety margin across cuts
- 4. Turn on A/B compare and pit recursive-400/60 against character-400/0 on the same text; compare the histograms
- 5. Paste one of your own documents (a README, an essay) and find the chunk config that keeps its ideas whole
Retrieval can only ever return whole chunks — so the moment you chunk a document, you've already decided which questions it can answer.
Document
Splitter config
Why it matters: a RAG system can only retrieve whole chunks. If the answer to a question is split across a chunk boundary with no overlap, retrieval returns a fragment and the model guesses the rest. That's why production pipelines default to recursive splitting with 10–15% overlap — try reproducing that here.
What is chunking in RAG?
Chunking splits documents into retrievable pieces before they are embedded. Retrieval returns chunks, not documents — so chunk boundaries decide what an AI system can find.
Why do RAG systems give incomplete answers?
Often because the fact was split across two chunks, or a chunk lost its surrounding context. Bad chunking looks like a bad model but is really a data problem.
What chunk size should I use?
There is no universal number — it depends on the document structure and the questions asked. That is exactly what this lab lets you feel: drag the size and overlap and watch what breaks.
Deeper dive: RAG, defined properly
Next: dissect a real prompt
Prompt Anatomy walks a production-grade prompt apart with the COSTAR framework — click each piece to see what it does.