Explainer
How to judge an AI claim in the news
A short set of checks for headlines about what AI can now do.
Start with the source. A peer-reviewed paper, a preprint (a paper posted before peer review), a company announcement and a demo video carry different weight, and a company's claims about its own product deserve independent checking. Ask what was measured. "Passes a bar exam" or "beats doctors" usually means a particular test under particular conditions, which may not match the real job. Ask whether the test questions could have appeared in the training data. Look for how often the system failed, as well as its best example. Notice who is quoted, and whether anyone outside the company was asked. Then hold the claim loosely, because early results often change once others try to reproduce them.
Going deeper
Contamination, where test items leak into training data, is hard to rule out from outside. One 2023 paper notes that the extent of the problem is unknown because it is not straightforward to measure (Sainz et al., arxiv.org). That is one reason results on newly written or held-out tests carry more weight.