AI Dojo Open on the typewriterThe typewriter

Explainer

Why models make things up

1 min read
Assumes: nothing.
First published 19 October 2026. Last edited 16 November 2026.

A model produces the most plausible continuation, and plausible and true are different properties.

A language model is trained to predict likely text. When it has seen a fact many times, the likely continuation is usually correct. When the fact is rare, missing, or newer than its training data, the model can still produce something with the right shape, such as a believable date, a real-sounding paper title, a quote nobody said or a statistic with a decimal point. Nothing in the basic process checks a claim against the world. Errors cluster around specifics such as names, numbers, citations and recent events, and around questions with a false premise. Giving the model the source text, letting it search, and asking it to flag uncertainty all reduce the problem without removing it.

Going deeper

Kalai et al. (2025) argue that evaluations which score "I don't know" the same as a wrong answer reward guessing, so models are partly trained towards confident errors (arxiv.org).