Explainer
Agent evaluation
Check the saved result and relevant actions, as well as the final reply.
An agent is AI that can use tools to take actions. In an invented task, it must add oat milk once to a list, preserve eggs and bread, and place no order. Saying “done” is insufficient. The list can remain unchanged, or the agent can add the item while also placing a forbidden order. Inspect all the requirements.
A trace records messages, tool calls and results from an attempt. It helps explain failures and find actions that a narrow final-state check misses. The original tau-bench paper notes that its score can miss a required-confirmation violation. Some traces include reasoning text exposed by the model. It is not a guaranteed account of how the answer was reached, as research on reasoning faithfulness shows.
Going deeper
Several action sequences can satisfy one task. Require exact steps only when the task requires them. Restore a known starting state between independent trials, and record resource limits. WebArena evaluates functional outcomes in controlled websites. Passing a tool-choice check alone does not establish that the whole job succeeded.
The shopping-list example shows the different failures side by side.