Trace Detective

Trace Detective

Read a real-format RAG agent trace — parse, embed, retrieve, rerank, generate, verify — where the agent answered a support question confidently and wrongly, and find the exact step that returned the bad data. The moat is what it teaches: the failure surfaces at the last step (a human flags the wrong answer), but the wrong data got in three steps earlier at retrieval, and every step after it ran faithfully on that poison. You accuse a step; the detective tells you whether you caught the culprit or a symptom, and counts the steps (and tokens) burned on the bad context. Runs entirely in your browser (0 uploads, works offline).

How to use it

1. Read the trace top to bottom — each row shows the tool, its input, and what it returned.
2. Decide where the wrong data first entered — not where the failure loudly surfaced — and click that step.
3. Press Accuse this step to lock in your guess.
4. Open the X-ray panel: it confirms the culprit, explains why the later steps only look wrong, and shows the "cost of this bug" — how many steps ran on the poisoned context before anyone noticed.

What this clears up (the fundamentals)

• The loudest step is rarely the cause — the answer looks wrong at generate, and the human catches it at verify, but the real fault is the retrieve call that fetched the wrong documents. Read traces backward from the symptom to the first bad decision.
• A grounded answer can still be wrong — the model faithfully answered from its context. Perfect generation over the wrong evidence produces a confident, wrong answer. Retrieval quality, not generation, is the bug.
• Bad data cascades — once retrieval hands over the wrong chunks, rerank politely orders them and generate faithfully cites them. One upstream error becomes a chain of plausible-looking downstream steps.
• Every wasted step has a cost — the steps that ran after the culprit burned real tokens and latency on poison. Naming the true root cause turns "the agent is flaky" into a specific, fixable retrieval bug.

Where it's used

A top-of-funnel AI-engineering reliability check — the fastest way to feel why "the agent hallucinated" is usually a retrieval bug, not a model bug. It's a miatz build-lab concept playable: play the trace here, then learn to build the seeded trace, the culprit-detection rubric, and the structural reveal yourself inside the agent-engineering module.

FAQ

How is this different from just reading logs?

Logs tell you what happened at each step; the detective makes you reason about where the wrong data entered versus where it merely surfaced. It's the same trace format engineers debug real agents with, minus the noise — one seeded fault, scored against the true root cause.

Why isn't the step that gave the wrong answer the culprit?

Because that step (generate) did its job correctly — it answered from the context it was given. The context was already poisoned by the earlier retrieve step. Blaming generation would have you tuning the prompt when the fix is in retrieval.

Is the trace random or scripted?

It's a fixed, seeded trace — deterministic, no model call, no randomness. Everyone debugs the same run, so the answer is checkable and the lesson is repeatable.

What's the "cost of this bug" counter?

It's the number of steps that executed after the culprit on the bad context — a stand-in for the tokens, latency, and money an agent burns before a wrong answer is caught. Catching the fault upstream saves all of it.

Is anything uploaded?

No. The trace and your accusation are evaluated entirely in your browser — nothing is transmitted, stored, or logged. Turn off your Wi-Fi and it still works.

Limits

A teaching trace with one seeded fault and a fixed root cause — it's a fast, opinionated lesson in reading agent failures, not a production observability tool. Real traces have noisier tools, multiple interacting faults, and no label telling you which step is wrong. The skill it trains (walk backward from the symptom to the first bad data) is exactly the real practice; the single-culprit simplification is not.

Related

Part of the Demystify Playgrounds. Explore the rest from the Playgrounds home.

Bookmark this page (Ctrl+D, or ⌘D on Mac) — it works offline the next time you need it.