Evals: how you know it works
A changed prompt feels better – until three weeks later something else is broken. An eval is a fixed set of cases against which you measure every change. How to build one in an hour.
Boaz Lichtenstein
Anyone working on an agent is constantly changing something: a sentence in the prompt, a tool, the model. Every change feels like an improvement because you check it against the case that just annoyed you. Whether three other cases broke in the process only shows in live operation.
An eval ends that guessing. It is a fixed set of cases that you run in full before and after every change.
The test set in an hour
1. Collect ten to thirty real cases. From the inbox, the ticket system, the history – not invented ones. Invented cases are too clean.
2. Build in the failures. At least a fifth of the cases should be ones the agent has already failed on or could fail on: incomplete details, contradictory sources, the request it should have declined.
3. Record the correct result for each case. Either the exact answer or the criteria a good answer must meet.
4. Freeze it. The set stays as it is. It only grows – see below.
Cases 02 and 03 are the valuable ones. An agent that can only do case 01 is not an agent but a template.
Three ways to score
Check automatically. Where the result is a number, a category or a field, a small script compares it with the target. That is the cheapest and most reliable scoring – use it wherever possible. It is often worth having the agent output the result in structured form just so that it can be checked this way.
Human with a criteria list. For texts: three to five yes/no questions per case. “Does the answer name the receipt number?” can be answered. “Is the answer good?” cannot.
Model as grader. A second model scores against the same criteria list. That scales – but the grader errs too. Compare its verdicts with your own on samples regularly, and give it the criteria, not just the question of overall impression.
What you measure
Not only “right or wrong”:
- Hit rate – how many cases meet their criteria?
- Boundary violations – did the agent do something it was not allowed to? A single such case weighs more than ten mediocre answers.
- Cost and duration per case – a change that brings two percentage points and triples the cost is rarely a gain.
And measure several times. Models do not answer the same way every time. Three runs per case show whether a result is stable or chance.
When you run it
- Before every change – that is the baseline.
- After every change – to prompt, tools, rules file, skills.
- On every model switch – even if the provider merely swaps the version behind the same name.
- At fixed intervals – because sources and connections change without you having done anything.
The set grows with every error
If the agent makes a mistake in live operation, it becomes a new case – before you fix the mistake. First you see it fail in the eval, then you change something, then you see it pass. That proves the fix works, and the same error cannot come back unnoticed.
Three traps
Optimising towards the set. Keep polishing the prompt until all thirty cases pass and you have an agent for exactly those thirty. Hold a few cases back that you run only occasionally.
Cases that are too easy. A hundred per cent hit rate usually means: the set tests nothing. A good set has cases that fail.
Scoring style rather than substance. Whether the answer sounds friendly is rarely the problem. Whether it names the right figure is.
With an eval in hand the next question can be answered without guessing: whether a reusable procedure – a skill – really improves the results.