AI Dojo Open on the typewriterThe typewriter

Explainer

Regression tests for AI

1 min read
Assumes: nothing.
First published 17 November 2026. Last edited 17 November 2026.

Checks that a change did not break behavior that worked before.

You improve an assistant's answer to one question. Run earlier cases again to see whether that change introduced other failures. The previous version is the baseline for comparison. Keep the cases, grading rules and relevant conditions the same. Retain results per case so a better total cannot hide which behavior got worse. Google's agent evaluation documentation describes testing changes against expected behavior.

A newly discovered failure can become a future test case. Keep a separate set for later comparisons, because cases used to choose changes are development material. Repeatedly picking winners on those reserved cases can tune your choices to that set, so the gain may not carry over to other cases. Adaptive data analysis research explains that risk.

Going deeper

A changed result can come from the system, a different grader or a changed test environment. Record those versions and retain failed attempts. Repeat uncertain cases rather than treating one fluctuating answer as proof. For an agent, restore the starting data between independent attempts. The evaluation lesson works through the comparison.