An AI evaluation, usually shortened to eval, is a repeatable test of whether a model or agent does a defined job well enough. A useful eval combines representative tasks, a scoring method, and a threshold for acceptable performance. It can measure accuracy, instruction-following, tool use, safety, latency, cost, or the quality of a complete workflow.
An eval is narrower and more practical than a public benchmark. Benchmarks compare systems on a shared dataset. Teams build evals around their own work: extracting clauses from the contracts they actually see, answering questions from approved medical material, or editing code without breaking a particular test suite.
Agent evals are harder than checking a single answer. The final result may look right even though the agent used an unsafe route, exposed data, retried wastefully, or acted outside the test environment. The cyber-evaluation containment failure reported by Anthropic is an extreme reminder that the harness and test setup are themselves part of what must be evaluated.
Why it matters at your desk. A law firm should test whether an assistant finds the risky clauses its reviewers care about, not rely on a generic score. A research team comparing Claude Opus 5 or another model should include its own documents and failure cases alongside published benchmark results.
Start with a small set of real, anonymized examples and record the expected outcome before changing the prompt or model. Keep failed cases, rerun the suite after every material change, and review samples by hand. An eval is most valuable when it catches a regression before a user does.