A student who memorized last year’s exam isn’t smart — they’re prepared for the wrong thing. We keep making the same mistake with AI. We celebrate a number on a benchmark, ship the model, and act surprised when it stumbles on the actual job. The model didn’t lie to us. Our test did.
Evaluation feels like the boring part — the spreadsheet after the science. But it’s the steering wheel. Every decision you make about a model — ship it, fine-tune it, pick it over a competitor — rides on a grade. And almost every way a grade goes wrong has a direct match in how a careless exam misleads a teacher. The good news: the fixes are the same ones schools worked out a century ago.
01Three ways the test lies to you
A bad grade usually fails in one of three ways. Each has a fix.
1. One question is luck, not skill
Score a model on a handful of prompts and you’re mostly measuring noise. Worse, the wording of the question moves the grade as much as the model’s ability does — reword the same task and the ranking can flip. A single number with no spread around it is a rumor, not a measurement.
› Go deeper: how much formatting alone can swing a score
Researchers held the task fixed and changed only superficial formatting — spacing, separators, how the choices were labeled. Accuracy swung by as much as 76 points on the same task for the same model (Sclar et al., “FormatSpread”). If a trivial reformat can move a model from near-zero to near-perfect, then any leaderboard that doesn’t report variance — across prompts, seeds, and runs — is hiding most of the story.
2. The model has already seen the answer key
Public benchmarks leak. Their questions end up in the web scrapes that models train on, so a high score can mean “memorized the test,” not “understands the material.” This is Goodhart’s law in one line: when a measure becomes a target, it stops measuring. The only honest test is one the model hasn’t seen.
› Go deeper: the contamination receipt
A team built GSM1k — a fresh set of grade-school math problems matched in difficulty to the wildly popular GSM8k benchmark — and re-tested everyone. Frontier models held their scores. But several others dropped by up to ~13 points on the equivalent-but-unseen questions, with the biggest fallers showing signs of having memorized the public set. Same skill on paper, very different skill in practice — the gap was the leak.
3. The grader has its own opinions
For open-ended work we increasingly let one model grade another. It’s fast and cheap — and quietly biased. LLM judges tend to prefer the first answer they’re shown, prefer longer answers, and prefer answers that sound like their own. A grader with a thumb on the scale produces a leaderboard that measures the grader’s taste as much as the model’s quality.
› Go deeper: how good (and how biased) an LLM judge is
In the MT-Bench study, a strong GPT-4 judge agreed with human raters about 80% of the time — roughly the rate humans agree with each other, which is genuinely useful. But the same work documented position bias (favoring whichever answer is shown first), verbosity bias (favoring longer answers), and self-enhancement bias (favoring its own style). The fix isn’t to abandon the judge — it’s to grade blind to a rubric, swap the answer order and average, and spot-check disagreements with a human.
02How a good exam is built
Here’s the reassuring part. Teachers have spent a century learning how to write a test that can’t be gamed — and almost every rule maps cleanly onto evaluating a model. Here’s the cheat sheet, with the receipt in the last column.
| Exam-design rule | Why it works | The eval move | Evidence it matters |
|---|---|---|---|
| Write fresh questions each term | Last year’s exam leaks; reused questions reward memory, not mastery | Use held-out, private, or rotating test sets — never grade on something that could be in training | On fresh GSM1k, overfit models fell up to ~13 pts |
| Ask many questions, not one | One question measures luck; a long exam measures ability | Use a large n and report the spread, not a single point estimate | Formatting alone swung scores up to 76 pts |
| Standardize the conditions | Same room, same time, same rules — or you’re comparing apples to noise | Fix the prompt template, decoding, and seeds; log them so it’s reproducible | Identical task, different format = different ranking |
| Grade blind, to a rubric | Knowing whose paper it is — or how long — sways the grader | Structured judge with a rubric; hide model identity; swap answer order and average | Judges show position, verbosity & self-preference bias |
| Two graders + appeals | One examiner has blind spots; disagreement flags the hard cases | Ensemble of judges + human spot-checks where they disagree | A strong LLM judge agrees with humans ~80%, not 100% |
| Test what the job needs | A spelling bee doesn’t tell you who can write | Construct validity: evaluate on production-like tasks, and score reliability@k, not a single pass | A leaderboard win ≠ a deployment that holds (τ-bench) |
| Don’t teach to the test | Drilling one exam inflates the score and hollows out the skill | Keep the eval out of training; rotate it before it saturates | Maxed benchmarks stop telling models apart |
| Proctor & audit the paper | Watch for shortcuts — and check the answer key is even right | Run shortcut / choices-only baselines; audit the questions themselves | ~57% of one popular MMLU subset had flawed items |
03I tested it with a simulation
To see how much grading method matters — separate from how good the models are — I ran a simple simulation. Take two models where one is genuinely better (72% vs 68% per question). Then ask: across grading regimes and test sizes, how often does the grade actually rank the better model first?
- Demo — an honest test, just too few questions.
- Leaderboard — lots of questions, but the weaker model has memorized a slice of the public set.
- Honest — fresh questions, each scored over several runs to measure consistency.
n=10 Demo 57% | Leaderboard 45% | Honest 66%
n=100 Demo 73% | Leaderboard 35% | Honest 91%
n=500 Demo 92% | Leaderboard 20% | Honest 100%
One result jumps out and it’s the whole point: on a contaminated test, more data makes things worse. The Leaderboard line doesn’t just stay flat — it sinks below a coin flip, dropping to 20% as you pile on questions, because every extra question drives the grade more confidently toward the wrong winner. Scale can’t fix a rigged test; it amplifies it. The Demo test is honest but slow — it needs hundreds of questions to be sure. Only the Honest regime — fresh questions, scored for consistency — climbs fast toward the truth. The method mattered more than the models.
› Go deeper: how the simulation works
It’s a Monte-Carlo model (20,000 trials per setting) of two models with true per-question accuracies of 72% and 68%. Demo scores each on n clean questions, once. Leaderboard uses the same n questions but lets the weaker model memorize 20% of them, inflating its observed score above the better model’s. Honest uses fresh questions and scores each over five independent runs, which sharpens the estimate. We then count how often the grade puts the genuinely-better model on top. The full script and plot are reproducible — the numbers above are the raw output.
04Where a good test still won’t save you
Better grading is powerful, but it isn’t magic. Three honest limits:
· A perfect test of the wrong thing is still useless. If your benchmark doesn’t resemble the real job, no amount of statistical rigor saves you — you’ll just measure the wrong skill very precisely. Construct validity comes before everything else.
· Private and rotating test sets cost real effort. Fresh questions don’t write themselves, and a held-out set you reuse too often quietly becomes a public one. Honest evaluation is an ongoing budget line, not a one-time setup.
· A leaderboard is a starting point, not a verdict. Public rankings can be shaped by who has more attempts, more data, or a friendlier format — so treat the top of the board as a shortlist, then run your own eval on your own tasks before you commit.
The takeaway
Stop chasing the high score. Interrogate the grade behind it. Fresh questions the model hasn’t seen. Enough of them, with the spread reported. Standardized conditions. A blind rubric and a second grader. And a score for consistency — reliability@k — not one lucky run.
The model that wins production won’t be the one with the best benchmark. It’ll be the one you graded honestly — and trusted anyway.