An agent that completes the task but wrecks the state hasn’t succeeded.
Success rate alone hides the failures that matter most in production. An agent can return the right answer while deleting a file, leaving a half-applied migration, or burning ten times the necessary tokens, and a binary pass/fail score will happily call that a win.
A useful evaluation scores the trajectory, not just the destination: did it respect side-effect boundaries, recover from a failed tool call, and stay within budget?
Each production failure the binary score hides maps to a trajectory check it should have run instead:
| Failure the success rate hides | Trajectory check |
|---|---|
| Deleting a file | Respected side-effect boundaries |
| Leaving a half-applied migration | Recovered from a failed tool call |
| Burning ten times the necessary tokens | Stayed within budget |
Logging each step makes these checks possible and turns a vague regression into a specific, fixable one.