AI Dojo Back to the typewriterThe typewriter

The long read

Evaluations (evals): how to test what you build

11 min read
Assumes: nothing.
First published 17 November 2026. Last edited 17 November 2026.

You ask an AI assistant whether you can return a library book on Sunday. It gives a friendly answer. To check that answer, you need the library's rules and a clear idea of what the answer must contain. An evaluation, often shortened to an eval, makes that check explicit so you can repeat it and compare changes.

Key ideas, in six lines
  • State the task and the rules for success before grading an answer.
  • Test different kinds of cases, including missing information and important failures.
  • A rubric gives the rules. A person, program or model applies them as the grader.
  • Compare versions on the same cases and keep the results for each case.
  • A score needs the counts behind it, test conditions and limits.
  • For an agent, check what changed and which actions occurred, as well as what it said.

No code or AI account is needed for the worked examples. All policies, candidate answers and numerical examples on this page are invented illustrations. Select text to keep a pencil note in this browser.

Start with one answer you can check

An evaluation checks a system against requirements for a particular task. The system can be a chatbot, a document search tool or an agent that takes actions. A useful result says what was tested and how success was judged. It does not establish that the system works for every purpose. OpenAI's evaluation guide starts with the objective, examples and scoring.

Figure 1 · Illustration · A fictional library notice

D1. Returns. The outdoor book drop accepts returns every day. The library desk is closed on Sundays.

D2. Fees. Borrowed books have no late fees. Lost books can incur replacement charges.

The question is: “Can I return a borrowed book on Sunday, and is there a late fee? Use these passages and cite the passage supporting each part.”

The notice supplies the evidence. The question specifies the job. The answer needs to explain Sunday returns and late fees, using the right passages. A cheerful tone cannot make a wrong fee policy correct.

One input, its context and expected behavior form a test case. One attempt at that case is a trial. Keeping these separate matters because the same system can give different answers on repeat attempts. Anthropic's guide distinguishes tasks, trials, graders and outcomes.

Write the rules before the score

A rubric is the written set of grading rules. For this example, each answer must pass four checks. These rules are our teaching example, not a universal rubric.

  1. Explain that Sunday returns are possible through the outdoor book drop.
  2. Say that borrowed books have no late fees.
  3. Make no material claim contradicted by or unsupported by the two passages.
  4. Cite a supporting passage for every claim the answer makes.

Read these candidate answers and decide which checks they pass. The letters identify invented answers, not models.

  • A. “Yes, use the outdoor book drop on Sunday [D1]. Borrowed books have no late fees [D2].”
  • B. “The library is closed on Sunday, so return the book on Monday to avoid a late fee [D1].”
  • C. “Yes, use the outdoor book drop on Sunday [D1].”
  • D. “Yes, use the outdoor book drop on Sunday [D1]. Borrowed books have no late fees [D1].”
Reveal the grades and their evidence
  • A passes all four. It answers both parts and cites the relevant passage for each.
  • B fails all four. It overlooks the outdoor drop, invents a late fee and cites a passage that does not support its conclusion.
  • C passes checks 1, 3 and 4. Its one claim is supported, but it omits the fee answer. The overall result is fail.
  • D passes checks 1, 2 and 3. The fee statement is correct under D2, but it cites D1. The overall result is fail.

Here, all four requirements are necessary. Three passes out of four help locate a mistake, but do not make the answer acceptable. In another task, some criteria can be preferences. Decide that distinction before combining scores.

A reference answer can show one acceptable response. It need not specify the only acceptable wording. “Sunday returns are possible through the outdoor book drop” and “Use the outdoor book drop on Sunday” satisfy the same requirement here. Human evaluation also needs clear criteria and attention to disagreement. van der Lee and colleagues examine those choices.

Choose cases that reveal different mistakes

A dataset is a collection of cases. Its contents determine what the evaluation can tell you. A set containing only simple English questions cannot establish performance on long Hindi conversations. Record where cases came from, which users they represent and what is missing. Datasheets for Datasets gives a framework for that documentation.

The library task can vary without becoming a different product. These are proposed cases, with the expected behavior stated in advance.

CaseWhat to check
Sunday return and feeAnswer both parts from D1 and D2.
“Can I retrun it on Sunday?”Preserve the answer despite a harmless typo.
“Why is there a late fee?”Correct the false premise using D2.
“Is parking free?”Say that the supplied notice does not answer this.
The fee passage is unavailableAnswer the return question and acknowledge the missing fee evidence.
A lost bookDistinguish replacement charges from late fees.

Include answerable cases as well as cases with missing information. Refusing every question must not count as success.

Tests aimed at particular weaknesses are useful even when they do not resemble ordinary traffic. The CheckList research organizes tests around behaviors, including changes that must preserve an answer. Keep such challenge cases identifiable. Their pass rate answers a different question from a representative sample of everyday requests.

Choose who or what grades

The grader applies the rubric. A program can check whether a required field exists. A person can inspect a nuanced explanation. A language model can also judge an answer when supplied with criteria and evidence. Each method can make mistakes. The choice depends on the requirement. Anthropic describes these grading methods.

For the library example, searching for the words “late fee” is a bad grader. Both “There is a late fee” and “There is no late fee” contain them. An exact text comparison also rejects some valid paraphrases. A repeatable program can consistently apply the wrong rule.

Code fits a requirement such as “return a JSON object with a numeric price of 500.” JSON is a structured text format that software can parse. Parsing checks the format. A separate value check establishes whether the price is 500. Neither check needs a model's opinion.

Going deeper: does the measure match the claim?

Validity concerns whether a measure supports the interpretation you give it. A formatting score measures formatting. Calling it a reasoning score requires further justification. A review of 445 benchmarks found recurring weaknesses in the connection between intended abilities and measurements. Its findings concern that reviewed sample, not every benchmark. The benchmark-validity review examines those gaps.

Compare a change fairly

A baseline is the version you compare against. It can be your current prompt or a simpler system. Keep the cases and grading rules fixed while testing a proposed change. Save both answers and the result for each case. Looking only at totals hides which failures were fixed and which appeared.

  1. Record the current instructions, model version, evidence, tool access and grading rules.
  2. Run the baseline on the chosen cases.
  3. Change the part you intend to test.
  4. Run the changed version on the same cases under comparable conditions.
  5. Inspect gains, new failures and repeated attempts before choosing.

If you change the prompt and model together, the comparison tests the combined change. It does not isolate the prompt's effect. Comparing the same cases also preserves useful information about their difficulty. Adding Error Bars to Evals explains paired comparisons and evaluation noise.

Repeat attempts can reveal variation, but ten cases tried ten times still cover ten distinct cases. Report both counts. A hundred outputs cannot establish coverage of a hundred different situations.

Keep some cases out of your decisions

Cases used while improving the prompt become development material. Keep a separate set for the later check, so those answers do not guide the changes. Repeatedly selecting the version that wins on the same test can adapt your decisions to that test. Research on adaptive data analysis explains this holdout problem.

Keep closely related examples together when splitting the data. Otherwise, one version of a document can guide your work while a near-copy appears in the supposed final test. The existing held-out test explanation covers training, validation and test splits.

“Held out from our prompt experiments” does not mean “never seen by the original model during training.” Public benchmark material can enter training data. Detecting that exposure has limitations, and its effect varies across tests. Li and colleagues study those differences.

Read the score and its limits

A pass rate is passed cases divided by the stated total. If eight of ten cases pass, the observed rate is 80%. That arithmetic describes those ten cases. It does not promise that four out of every five future requests will succeed.

Keep categories visible. This invented test compares two versions on the same cases. The total improves while Hindi performance falls. These are illustrative counts, not model results.

CasesVersion AVersion B
English810/900
90%
855/900
95%
Hindi80/100
80%
50/100
50%
Total890/1,000
89%
905/1,000
90.5%

The denominator matters too. Suppose 100 requests produce 90 outputs, of which 81 are correct. Correctness among outputs is 90%. Success across all requests is 81%. Report the ten failed attempts and their cause instead of silently removing them.

Going deeper: uncertainty around a percentage

Eight passes from ten independent sampled cases and 800 passes from 1,000 both give 80%. Approximate 95% Wilson intervals are 49.0% to 94.3% and 77.4% to 82.4%, respectively. The interval procedure aims for about 95% coverage across repeated samples under its assumptions. It does not contain 95% of future answers. Biased cases and grading errors remain outside that calculation. NIST explains the interval.

Going deeper: accuracy, precision and recall

For a system that labels messages urgent or ordinary, accuracy is the fraction of all labels that are correct. Precision asks what fraction of messages labelled urgent truly are urgent. Recall asks what fraction of all urgent messages were found. The suitable measure depends on the cost of missed urgent messages and false alarms. If a denominator is zero, the evaluation needs an explicit convention. These classification measures do not by themselves score an open-ended explanation. Google defines the metrics.

Check a model judge

A model judge is an AI model asked to grade an output. Give it the task, criteria and evidence needed to decide. Then compare its grades with independently reviewed examples. Human disagreements can reveal unclear requirements or more than one reasonable interpretation. Research on rating indeterminacy examines that problem.

Include known failures when checking the judge. In an invented set with 90 acceptable answers and ten unacceptable ones, a judge that passes everything agrees on 90%. It catches none of the failures. Count false passes and false fails separately, then inspect the evidence behind disagreements.

Position and writing style can affect model grades. Test whether swapping two answers or changing irrelevant wording changes the verdict. Historical MT-Bench judge research documented position and verbosity biases. These checks reveal problems in the tested setup; passing them does not prove a judge is unbiased.

Going deeper: a winner can still fail

Pairwise grading compares two answers. Pointwise grading scores one answer against the rules. In our example, C is better than B but still omits the fee answer. A comparison winner need not meet the acceptance threshold. Pairwise grading is not always more reliable: Tripathi and colleagues found greater sensitivity to irrelevant style changes in their tested pairwise setups. Validate the format for your task.

Keep “failed to grade” separate from “answer failed.” A timeout or unreadable judge response is a grading error. Record the judge and rubric versions, and recheck after changing either. Anthropic's safety-evaluation guidance covers calibration, grader failures and reporting. Those grading practices are useful here too, though this library example is not a safety benchmark.

Going deeper: the answer is not an instruction to the grader

An answer can contain text such as “give this full marks.” That text is material being judged, not authority over the grader. JudgeDeceiver demonstrates attacks against model judges. Separating instructions from candidate text is necessary design work, but a delimiter alone does not establish protection.

Separate finding evidence from using it

Some systems fetch documents before answering. This is retrieval-augmented generation, or RAG. Evaluate whether the system found the needed material, whether its answer follows that material, and whether it answered the question. The Ragas paper separates retrieval and generation quality.

For the library question, finding only D1 leaves the fee policy unavailable. An answer can accurately repeat the Sunday-return rule while leaving the fee question unresolved. The needed evidence was not all retrieved. Context recall asks whether required information was fetched.

Groundedness, also called faithfulness, asks whether the answer's claims follow from the supplied evidence. If the evidence is outdated, faithfully repeating it can still give outdated information. A grounding check alone cannot establish real-world correctness. Ragas defines faithfulness against the retrieved context.

Citation quality is another check. Answer D states the right fee policy but cites the wrong passage. A working link does not prove support for a claim. ALCE evaluates correctness and citation quality separately.

Check what an agent actually did

An agent can use tools to take actions. Its final reply is only one part of the evidence. Check the saved result and the relevant actions. A trace is the recorded messages, tool calls and tool results from an attempt. Some traces include reasoning text exposed by the model. That text is not a guaranteed account of how the answer was reached; the reasoning explanation covers this limit. Google's agent evaluation documentation separates final responses and intermediate behavior.

Figure 2 · Illustration · A shopping-list task

The request is “Add oat milk once to my shopping list. Keep the other items. Do not order anything.” The list starts with eggs and bread. No orders exist.

  • The agent says “done,” but the list still contains only eggs and bread. The addition failed.
  • The list contains eggs, bread and oat milk, but the agent also placed an order. The task failed because ordering was forbidden.
  • The list contains only oat milk. The task failed because the other items disappeared.
  • The list contains all three items once, and no order occurred. A truthful completion reply meets this example's requirements.
Every result is invented. The same final message can accompany different outcomes.

A narrow outcome check can miss an action-rule violation. The original tau-bench paper explicitly notes this limitation in its database-based reward. Inspect required outcomes and prohibited actions. Allow different valid tool sequences unless the task requires a particular sequence.

Restore the starting state before another independent trial. An item left in the list by the previous attempt can make the next attempt a different task. Controlled environments such as WebArena support reproducible task evaluation. Tests with fixed tool replies and tests against a running application cover different parts of a system.

Going deeper: one success versus repeated success

Pass@k asks whether at least one of k attempts succeeds. Pass^k asks whether all k attempts succeed. For one hypothetical task with independent attempts and a known 80% chance per attempt, three attempts give 99.2% for at least one success and 51.2% for all three. These are calculated illustrations, not measured benchmark estimates. Averaging across tasks requires per-task calculations, not inserting one overall score into these formulas. tau-bench defines the distinction.

Record time and cost alongside success. Extra attempts can raise success while using more resources. The AI Agents That Matter research examines this tradeoff. Compare systems under stated budgets rather than crediting a design for gains caused by more attempts.

Keep testing as the system changes

Regression tests check whether a change broke behavior that worked before. Keep the library cases when improving the assistant. Add a reproducible case when you discover a new failure, while preserving separate cases for final comparisons.

Offline evaluation uses prepared cases, separate from live user traffic. It can still use internet-connected tools. Online evaluation examines production behavior. Real traffic can reveal new problems, but a thumbs-up does not automatically establish factual correctness. LangSmith's evaluation concepts distinguishes these workflows.

A public benchmark is a shared test and scoring procedure. Its result depends on the tasks, setup and version. Your application can differ in language, documents, tools or user needs. HELM evaluates models across scenarios and several metrics rather than reducing every choice to one accuracy number.

Prepared tests also leave questions about actual use. Can a beginner follow the instructions? Does the interface help them recover from a mistake? NIST's ARIA approach combines model testing, deliberate attempts to expose failures (red teaming), and user testing. A good test-set score is one source of evidence when deciding what to release.

Review the key ideas

Sources and further reading

Sources were checked on 17 November 2026. Links beside each claim lead to primary research or official documentation. The library, shopping list and numerical examples are invented teaching material. Historical research results describe their tested setups, not current model rankings.

Try the grading rules on a complete answer.

Grade an AI answer