Evals are the only roadmap that survives a model upgrade | Brain Quest
{{ post.deck }}
{{ sec.h }}
{{ para }}
- {{ li }}
{{ sec.quote }}
The market need
Every team building on a model provider has now lived through an upgrade that improved the benchmarks and degraded their product. Without a test set, the only detection mechanism is a customer complaint, and the only remedy is a week of prompt archaeology.
The teams that ship confidently are not the ones with the cleverest prompts. They are the ones who can answer “did that change make things better?” in ten minutes.
Start with thirty examples, not three hundred
The most common reason teams have no evals is that building them sounds like a quarter of work. It is an afternoon. Take thirty real inputs — actual user requests, including the ugly ones — write down what a good response looks like for each, and run them.
Thirty examples will catch the majority of regressions that matter. Three hundred catches slightly more and never gets built.
Draw examples from production traffic, not from your imagination.
Include the inputs that embarrassed you. Those are the regression tests.
Store the expected behaviour, not the expected string. Exact-match grading breaks on paraphrase.
Grade the property, not the prose
Most useful checks are boring and deterministic: is the JSON valid, is the citation present, is the refusal correct, is the number inside the allowed range. Reach for a model-as-judge only for the genuinely subjective remainder, and calibrate it against human ratings before trusting it.
A suite that is eighty percent deterministic runs in seconds, costs nothing, and can sit in CI. That is what makes people actually run it.
If running your evals costs more than a coffee, you will not run them, and they will not protect you.
Make the suite the unit of planning
Once the set exists, the roadmap writes itself: pick the failing cluster with the most user impact, fix it, watch the number move. Feature requests get expressed as new eval cases before they get expressed as code. Model upgrades become a five-minute decision rather than a leap of faith.