Explainer
Evaluation dataset
The cases you use to check an AI system, with the context and expected behavior for each.
A case contains an input, the evidence or starting state available to the system, and the requirements for success. A dataset collects cases. Record which people, languages and tasks it represents. Convenient examples can leave out important uses. Datasheets for Datasets describes useful documentation.
For a fictional library assistant, cases can ask about Sunday returns, a lost book, a typo or information absent from the notice. State the expected behavior before grading. Tests aimed at known weaknesses answer a different question from a sample of ordinary requests. CheckList explores behavioral testing.
Going deeper
Repeating ten cases ten times gives 100 outputs but still only ten distinct cases. Keep case counts and trial counts separate. Repetition helps reveal response variation; new cases broaden what is tested. Adding Error Bars to Evals separates these sources of uncertainty.
See the worked evaluation lesson for a complete case table.
Read next: Evaluation rubric