Explainer
Evaluation rubric
Written rules for deciding whether an answer meets the task's requirements.
A rubric makes a judgment such as “good answer” specific enough to apply. It can require a correct fact, an answer to every part of the question and evidence for each claim. The grader is the person, program or model applying those rules. Human evaluation also needs defined criteria and attention to disagreement. van der Lee and colleagues examine those choices.
In our invented library exercise, an answer must explain Sunday returns and late fees, avoid unsupported claims and cite the right passages. All four are required. Passing three locates a mistake but still gives an overall fail. A different task can treat tone as a preference. Decide which requirements are necessary before combining scores.
Going deeper
If a rubric uses a scale, explain what each level means. A score of 4 out of 5 has little meaning without those anchors. Try known good, bad and borderline answers. If careful readers disagree, inspect the criteria and evidence before assuming one person is wrong. OpenAI's evaluation guidance discusses rubric design and human calibration.
Read the full worked rubric, then try applying it yourself.
Read next: Grade an AI answer