An exercise to try
Grade an AI answer
A short answer can be correct but incomplete. Use written rules to tell the difference.
- Time
- 10 minutes
- What you need
- This page and a place to write your grades. No chatbot or code is needed.
Illustration. The library notice, answers and grades here are invented teaching material, not outputs from a model.
The notice and question
D1. Returns. The outdoor book drop accepts returns every day. The library desk is closed on Sundays.
D2. Fees. Borrowed books have no late fees. Lost books can incur replacement charges.
The question is “Can I return a borrowed book on Sunday, and is there a late fee? Use these passages and cite the passage supporting each part.”
The rules
- Explain that Sunday returns are possible through the outdoor book drop.
- Say that borrowed books have no late fees.
- Make no material claim contradicted by or unsupported by D1 and D2.
- Cite a supporting passage for every claim the answer makes.
All four checks must pass. A partial score locates a mistake but does not give an overall pass.
Grade these answers
- A. “Yes, use the outdoor book drop on Sunday [D1]. Borrowed books have no late fees [D2].”
- B. “The library is closed on Sunday, so return the book on Monday to avoid a late fee [D1].”
- C. “Yes, use the outdoor book drop on Sunday [D1].”
- D. “Yes, use the outdoor book drop on Sunday [D1]. Borrowed books have no late fees [D1].”
Write pass or fail for each rule, then identify the passage or missing requirement behind your grade.
Check your grades
- A passes all four checks.
- B fails all four. It ignores the outdoor drop, invents a late fee and cites a passage that does not support its conclusion.
- C passes checks 1, 3 and 4, but fails check 2. Its one claim is supported; the fee question is unanswered.
- D passes checks 1, 2 and 3, but fails check 4. D2 supports the fee policy, while the answer cites D1.
Only A passes overall. C can win a comparison against B while still failing the task.
Try a different case
Hide D2. Write a suitable answer using only D1, and revise the expected behavior. The answer can explain Sunday returns but must acknowledge that the fee information is missing. A test cannot require evidence the system never received.
Continue with the evaluation lesson for case selection, model judges and uncertainty. The distinction between answer correctness and citation support also appears in ALCE.