Hallucination & Groundedness
A hallucination is output that is fluent, plausible, and wrong. Groundedness is the property that makes it detectable: output is grounded when every claim in it is directly supported by source material supplied at call time.
The two are one topic because groundedness is the only mitigation that is measurable. You cannot test for truth in a generated paragraph, but you can test whether each claim traces back to a retrieved document.
Why It Happens
The model is optimising for a likely continuation, not a true one. Nothing in next-token prediction distinguishes a fact it has seen thousands of times from a pattern that merely looks like one, and there is no internal signal that fires when it moves from recall to invention. See Large Language Models for the mechanism.
This has two consequences that shape every mitigation below:
- Fluency is uncorrelated with accuracy. A wrong answer reads exactly as well as a right one, so human reviewers scanning for quality will not catch it.
- Stated confidence is not evidence. A model asked how sure it is produces another plausible continuation, not an introspective measurement.
Not One Failure, Four
Grouping these together is why mitigation efforts miss. Each type has a different cause and a different fix.
| Type | What it looks like | What actually fixes it |
|---|---|---|
| Fabrication | Invented citations, case numbers, clause references, API methods | Retrieval grounding with mandatory citation, and validating every citation against the source |
| Unsupported inference | A real source, but a conclusion the source does not state | Prompt constraint to quote before concluding, plus claim-level groundedness checking |
| Stale knowledge | Confidently correct as of the training cutoff, wrong now | Retrieval from a current system of record, not from model knowledge |
| Schema-forced invention | A plausible value produced to satisfy a required field | An explicit null path in the output schema. See Prompting |
The last one is self-inflicted and common. A required invoice_total with no way to say "not present on this document" guarantees a number gets invented.
The Mitigation Stack
Ordered by effect. Each layer reduces the rate; none of them reaches zero.
Layer 4 is the one most often skipped and the cheapest to add. A citation that does not resolve to a real chunk, a total that does not equal the sum of its lines, a date outside the contract term: all of these are deterministic checks, and they catch a large share of fabrication without another model call.
Layer 6 belongs where consequence is high, not everywhere. Routing all output to review is how a system produces rubber-stamping rather than oversight. Route on the confidence field and the value at stake.
Measuring Groundedness
Two numbers are worth tracking per release, and neither is a single quality score.
| Metric | Definition | How to get it |
|---|---|---|
| Groundedness | Share of claims in the answer that are supported by the retrieved context | Decompose the answer into claims, score each against the context with a judge model, spot-check against human labels |
| Retrieval recall | Share of questions where the answer-bearing chunk was actually retrieved | A labelled question set with known source chunks |
Measure retrieval recall first. If the correct chunk was never retrieved, the model was structurally unable to answer correctly and no amount of prompt work will fix it. Grounding failures caused by retrieval gaps and grounding failures caused by the model over-reaching need opposite responses, so separating them matters more than the absolute score.
Keep a labelled evaluation set of twenty to fifty real questions with known correct sources, and re-run it on every prompt, model, or chunking change. Judge models drift and agree with humans imperfectly, so calibrate the judge against a human-labelled subset rather than trusting its score outright.
What Does Not Work
- Asking the model to be accurate. "Do not make anything up" in a system prompt lowers the rate slightly and does not change the mechanism.
- Asking for self-reported confidence and trusting it. Useful only as a routing signal, and only after checking that its confidence correlates with correctness on your evaluation set.
- Assuming a larger model removes the problem. Larger models hallucinate less often and more convincingly, which raises the review cost per error.
- Eyeballing a few outputs before release. Fluent wrong answers pass casual review by construction.
Key Takeaways
- Hallucination is a property of next-token prediction, not a bug to be patched out. Design for a non-zero rate.
- Fluency and stated confidence carry no accuracy signal. Neither does a spot check.
- Treat the four types separately. Fabrication, unsupported inference, stale knowledge, and schema-forced invention need different fixes.
- Groundedness is the measurable proxy for truth. Require citations and validate that they resolve.
- Add deterministic validation before adding a judge model. Resolving citations and reconciling numbers is cheap and catches a lot.
- Measure retrieval recall separately from groundedness. A missing chunk is not a model failure.