Hallucination

A hallucination is output from an AI model that is fluent, confident, and false — fabricated facts, invented citations, nonexistent APIs, or made-up details presented as if they were true.

The unsettling part is not that models make errors; it is that hallucinated answers look exactly like correct ones. There is no stutter, no hedge, no formatting difference. A model will describe a court case that never happened with the same polished certainty it uses for one that did.

How it works

Hallucination is not a bug bolted onto language models — it falls out of how they work. An LLM is trained to predict the next token given the previous ones, which makes it an engine for producing plausible continuations, not verified statements. When the training data contains the answer strongly and consistently, the most plausible continuation is usually true. When the model is asked about something rare, recent, private, or precise — a niche library's function signature, a specific paper's authors — the most plausible-sounding continuation may simply be invented, assembled from statistical echoes of similar text.

Certain conditions reliably make it worse: questions at the edge of or beyond training data, prompts that pressure the model to answer rather than admit uncertainty, long chains of reasoning where one early error compounds, and requests for exact citations, numbers, quotes, or URLs.

Why it matters

Hallucination is the central reliability problem of applied AI. Lawyers have been sanctioned for filing briefs citing fabricated cases; developers waste hours on package names that do not exist — a failure mode common enough to spawn a supply-chain attack called slopsquatting, where attackers register the fake packages models tend to invent. Any system that puts raw LLM output in front of users without grounding or verification is shipping a confident liar.

The defenses are architectural, not rhetorical: retrieval-augmented generation grounds answers in real documents; prompts that explicitly permit "I don't know" reduce forced guessing; citations make claims checkable; and systematic evals measure the fabrication rate before and after every change instead of trusting vibes.

A worked example

Ask a model without retrieval: "What does the miatzOrm.batchUpsert() function return?" A model that has never seen this (fictional) library will often answer fluently: "It returns a Promise resolving to an array of upserted record IDs" — complete with a plausible code sample. Every detail is invented. Now wire the same model into a RAG pipeline over the actual docs, and the retrieval step comes back empty; a well-prompted system answers: "I can't find batchUpsert in the documentation — the closest match is bulkWrite." Same model, different architecture, opposite outcome.

How Miatz teaches it

Miatz treats hallucination as an engineering constraint to design around, not a curiosity. Learners induce hallucinations deliberately in the labs, then fix them with grounding: same question, with and without retrieval, measuring the difference. The institute practices what it teaches — Mysty, the AI shadow engineer, works propose-and-approve with an audit log precisely because unverified AI output is never trusted by default.