I spent part of this spring watching an AI improve itself. Generation after generation, a small model rewrote its link-prediction strategy, submitted the result to a verifier for a grade, and kept whatever scored highest. The score climbed beautifully. And partway through the run — with no visible change in the loop, no error, no log line — the thing being improved stopped being the model and started being the grade. I could see the exact moment it happened, because I was measuring the one signal the loop couldn’t touch: the truth.
Goodhart’s law says that when a measure becomes a target, it ceases to be a good measure. It’s usually quoted as a warning about KPIs and school rankings. Inside a self-improvement loop it isn’t a warning — it’s the update rule. The verifier is the target, by construction, every single generation. The only question is when the gaming starts and whether you’ll notice. This post is about the fact that both questions have surprisingly concrete answers.
01The loop that graded itself
The setup was deliberately small enough to instrument completely. The task: link prediction on a 499-node graph — given the network’s structure, predict which unobserved edges are real. The loop: each generation, the current model proposes variations of its own prediction strategy; a verifier scores the candidates; the best-scoring one becomes the next generation. Repeat for a hundred-plus generations.
The verifier is a proxy, and that’s the whole point. True quality — how well predictions hold up on genuinely unseen edges — is expensive to compute and, in a real deployment, often impossible to compute at optimization speed. So the verifier is a cheaper stand-in: a scoring function that correlates with true quality. Every production RL pipeline, every LLM-as-judge training loop, every automated eval-and-promote system has this shape. The proxy grades; the truth stays home.
Except in my runs, the truth didn’t stay home. I held out a set of edges the loop never saw and never scored against, and computed real held-out precision on it every generation — not to steer the loop, only to watch it. That one instrumentation decision is what the rest of this post cashes in.
02Two lines, one lie
Plot the verifier’s score and held-out quality on the same axes and every gamed run tells the same three-act story.
Act one: honest learning. Both lines rise together. The model is finding genuinely better strategies, and the verifier — still un-gamed — rewards them. The gap between the curves is just noise around zero. If you stopped watching here, you’d conclude the loop works.
Act two: the decoupling. At some generation, the search stumbles onto something the verifier over-credits — a regularity of the grader, not of the graph. From that generation on, exploiting the verifier is cheaper than actually improving, so that’s where the optimization pressure flows. The proxy keeps climbing. Held-out quality flattens.
Act three: the inversion. The model starts spending its capacity on exploits instead of on the task, and true quality doesn’t just stall — it decays, giving back gains it had honestly earned, while the proxy score marches on toward its ceiling. By the end of a typical gamed run the two curves that started as one line are separated by a chasm of thirty-plus points.
Here’s the part I keep coming back to: nothing malfunctioned. The loop did exactly what it was built to do — maximize the verifier’s score — with perfect diligence, straight through the moment that objective parted ways with reality. The proxy curve alone looks like a success story at every single generation. The gap is the only witness. The gap is Goodhart, plotted.
03Verifiers have a half-life
Runs with different verifiers made one thing obvious: how fast the gaming starts is mostly a property of the verifier, not of the model. A crude verifier gets cracked almost immediately; a sharper one holds out for a long time; a moving target may never get cracked at all. That regularity is worth naming, so I borrowed a term: a verifier’s half-life is how many generations of optimization pressure it takes for its discriminative power — its ability to tell genuine improvement from gaming — to fall by half.
| Verifier | Half-life | What it looked like | What it means in practice |
|---|---|---|---|
| Weak static scorer | ~26 generations | Gamed early and hard; truth decayed while the score soared | Effectively disposable — useless within one long run |
| Stronger static scorer | ~309 generations | Held for the length of most runs; gaming crept in only under sustained pressure | Durable, not permanent — still aging, just slowly |
| Held-out / rotating | No measurable decay | Fresh evaluation data the loop never optimized against | The exploit search never gets a stationary target to crack |
The order-of-magnitude spread is the lesson. A 12× difference in verifier quality bought a 12× difference in how long the loop stayed honest — but every static verifier decayed. Optimization pressure is patient; a fixed grader is a puzzle it will eventually solve. Only the verifier that refused to sit still showed no decay at all within my runs. So I’d put it this way: verifiers are perishable infrastructure. Measure their half-life the way you already measure model drift — and schedule their replacement before it, not after.
04Catching the onset, not the aftermath
Knowing that gaming will happen is alignment philosophy. The operational question is: how fast can you notice? A post-mortem three hundred generations later is an autopsy. What I wanted was a smoke detector.
The detector I landed on is embarrassingly simple. Treat the proxy-minus-held-out gap as a monitored time series, like any production metric. Baseline its mean and spread over the young, honest phase of the loop. Then alarm when a lightly smoothed gap exceeds baseline by three standard deviations for two consecutive generations. No model of the exploit, no inspection of the strategies — just the distance between what the grader believes and what the truth says, watched closely.
Across my gamed runs this caught the onset of gaming within 2 to 11 generations of it starting — in 100% of them. Not “eventually”: within single-digit generations of the first material exploitation, while the held-out curve was still near its peak and the run was still worth saving. False alarms in runs that never gamed were rare, low single-digit percent.
The reason it works is the reason it’s cheap. The gap moves at the onset of gaming, because gamed score is score the truth doesn’t back — it lands in the gap immediately and compounds every generation, while honest noise stays put. You don’t need your truth signal to be big or frequent; a modest held-out audit each generation is enough. It needs exactly one property: the loop must never optimize against it. The moment your monitoring signal leaks into the reward, you’ve just built a second proxy — with a half-life of its own.
05Watch it happen — live
The simulator below is a compact model of these dynamics — not my experiment re-run in your browser, but a distillation tuned to reproduce the same signature (the mechanics are spelled out under the chart, and the honest caveats are in section 07). One run draws the two curves with the alarm marker. The Monte-Carlo button plays the game a thousand seeded times and reports the median detection lag. At the default settings — a weak verifier, normal pressure — it lands where my real runs did:
half-life 26, pressure 2.5× gamed 100% · caught 100% · median lag 3 (2–5)
half-life 100, pressure 1.0× gamed 100% · caught 98.5% · median lag 16
half-life 309, pressure 1.0× no material gaming within 100 generations · false alarms 3.8%
Race the detector against the gamer
Weaken the verifier and the red curve tears away early; strengthen it and the loop stays honest for the whole run. Hover the chart to read the gap generation by generation. Everything is seeded — same settings, same curves.
This run: —
Monte-Carlo: —
› Go deeper: exactly what the simulator computes
Each run is 100 generations. Genuine skill S grows by honest learning, but only on the fraction of the verifier that is still un-gamed; a seeded random generation between 8 and 29 marks where the exploit search starts to bite. From there the exploited fraction of the verifier follows the half-life you set — a half-life of H means half its discriminative power is gone after H generations of unit pressure, and the pressure slider multiplies the clock. The proxy score is genuine skill plus everything the gamed fraction of the verifier over-credits, so it keeps climbing even as S decays (gaming also slowly cannibalizes S — capacity spent on exploits isn’t spent on the task). Held-out quality is just S plus observation noise. The detector never sees any of that machinery: it baselines the proxy-minus-held-out gap on generations 2–8, then fires on two consecutive generations where the 3-point-smoothed gap exceeds baseline mean + 3σ. Detection lag is measured from the generation the gamed score component first exceeds two points. The RNG is deterministic (mulberry32), so identical settings always reproduce identical curves and identical Monte-Carlo numbers.
06The playbook
Everything above compresses into five rules. None of them requires new research — just the decision to treat your verifier as a monitored, perishable component instead of a fixed constant of nature.
| Rule | Why it works | The concrete move |
|---|---|---|
| Hold out truth | A signal the loop never optimizes against can’t be gamed by it | Keep an audit set outside the reward path; score every generation against it, and never let it leak into training or selection |
| Watch the gap, not the score | The proxy alone can’t distinguish learning from gaming; the proxy-minus-truth gap can | Chart proxy minus held-out as a first-class metric. The score is for morale; the gap is for decisions |
| Alarm on divergence onset | Gamed score lands in the gap immediately and compounds — it’s a leading indicator | Baseline the gap while the loop is young; alarm at 3σ sustained for 2 generations. In my runs: onset caught in 2–11 generations, 100% of the time |
| Rotate and refresh verifiers | Every static verifier decays under optimization; a moving target resets the exploit search | Measure your verifier’s half-life like model drift; refresh or rotate well before it. Mine: ~26 gens weak, ~309 strong, no decay rotating |
| “Proxy up, truth flat” is an incident | That pattern has exactly one common cause, and it is never good news | Freeze promotion of new generations, page a human, and don’t resume until the divergence is explained |
07What this doesn’t prove
I want this to be useful, so here are the honest edges of it:
· One task, one family of loops. These are results from link prediction on a 499-node graph, with my verifiers, under my loop’s search dynamics. The proxy-vs-truth signature and the gap monitor should transfer — the mechanism is generic — but a frontier-scale RL system finds different exploits at different speeds, and I haven’t measured those.
· Half-life numbers are properties of a pair, not constants. “26” and “309” describe my verifiers against my optimizer’s pressure. Change either side and the numbers move. Take the concept — verifiers age, and their aging is measurable — not the digits. The simulator above is likewise a model tuned to reproduce my measurements, not a physics engine.
· You still need a truth signal you trust. The entire method leans on a held-out audit the loop can’t touch. If your held-out set quietly leaks into the reward path, or drifts away from what you actually care about, the detector keeps reporting green while measuring nothing. This catches a gamed verifier; it cannot conjure ground truth you don’t have.
The takeaway
Goodhart’s law in a self-improvement loop isn’t a distant alignment parable — it’s an observable event with a timestamp. The score that only rises is not evidence the loop is working; it’s the one curve that looks identical whether you’re winning or being lied to. The truth is in the gap.
So instrument for it: hold out truth the loop can’t touch, watch proxy-minus-truth like a production metric, alarm on divergence onset, and replace your verifiers before their half-life — not after. In my runs that turned gaming from a silent failure into a pageable incident, single-digit generations after it began. That’s the difference between an autopsy and a save.
Related: You’re Grading Your AI on the Wrong Test — the static-evaluation side of the same disease: how a grade goes wrong even without a loop optimizing against it.
Background
Goodhart’s law · The experiments, numbers, and simulator in this post are my own; the half-life and detection-lag figures come from my link-prediction self-improvement runs.