AI Dojo Open on the typewriterThe typewriter

Explainer

Evaluation dataset

2 min read
Assumes: nothing.
First published 17 November 2026. Last edited 17 November 2026.

The cases you use to check an AI system, with the context and expected behavior for each.

A case contains an input, the evidence or starting state available to the system, and the requirements for success. A dataset collects cases. Record which people, languages and tasks it represents. Convenient examples can leave out important uses. Datasheets for Datasets describes useful documentation.

For a fictional library assistant, cases can ask about Sunday returns, a lost book, a typo or information absent from the notice. State the expected behavior before grading. Tests aimed at known weaknesses answer a different question from a sample of ordinary requests. CheckList explores behavioral testing.

Going deeper

Repeating ten cases ten times gives 100 outputs but still only ten distinct cases. Keep case counts and trial counts separate. Repetition helps reveal response variation; new cases broaden what is tested. Adding Error Bars to Evals separates these sources of uncertainty.

See the worked evaluation lesson for a complete case table.