01Where the zero comes from
Group Relative Policy Optimization does away with the learned value model. Instead of asking a critic how good a response was, it samples K responses to the same prompt and scores each one against its own siblings. The advantage of rollout i is
That is an elegant trick — the group is its own baseline, and no second network has to be trained to guess what a good score looks like. It also has a consequence that follows immediately and gets discussed far less than it should.
Suppose the reward is binary, which it is for most verifiable tasks: the unit tests pass or they do not, the answer matches or it does not. If all K rollouts get a reward of 1, then
The bill arrives either way
That group still generated K full rollouts. If this is an agentic environment, each one was a complete episode — tool calls, retries, a sandbox, maybe minutes of wall clock. You paid for all of it and multiplied the result by zero. The run will not warn you: loss curves are computed over the groups that survived, so a run wasting most of its compute looks exactly like a healthy one, only slower.
02The arithmetic of a wasted group
Let p be the probability that one rollout on a given prompt succeeds. Take the K rollouts as independent draws. A group is wasted when it is unanimous, which happens two ways:
and so the share of rollouts that land in a group with any gradient at all is
Two things about that expression deserve more attention than they get.
First, it is symmetric in p and 1−p. Swap “the model always succeeds” for “the model always fails” and you get the identical number. We talk about tasks being too hard for a model as if that were the problem and being too easy as if that were merely wasteful, but the gradient does not distinguish them. Unanimity is the failure, in whichever direction.
Second, the exponent means it collapses fast. At K = 8:
| Pass rate p | Rollouts that do something | Rollouts wasted |
|---|---|---|
| 50% | 99.2% | 0.8% |
| 70% | 94.2% | 5.8% |
| 80% | 83.2% | 16.8% |
| 90% | 57.0% | 43.0% |
| 95% | 33.7% | 66.3% |
| 99% | 7.7% | 92.3% |
| 99.9% | 0.8% | 99.2% |
And by symmetry, the same row applies at 1%, 5%, 10% and so on.
A benchmark your model scores 99% on is not a slightly stale training signal. It is 7.7% of a training signal, paid for in full.
03Watching an environment expire
A real environment is not one pass rate. It is a spread of task difficulties, and the policy moves through that spread as it trains. The lab draws 600 prompts once from a fixed difficulty distribution and then lets you push the policy forward. Each cell is one prompt's group: green if the rollouts disagreed and produced a gradient, amber if they all passed, red if they all failed.
At the start — training progress 0, mean pass rate 39.1% — 89.5% of groups produce a gradient. The environment is well matched to the policy, which is what people mean when they say a task set is good.
Now drag training progress to the right and watch the green drain into amber. By the far end the mean pass rate is 88.7% and only 58.5% of groups still produce a gradient: 41.5% of every rollout you pay for now buys nothing. Nothing about the environment changed. No prompt went stale, no reward function broke. The policy got better, and getting better is what makes prompts stop teaching.
This is the part that is easy to misdiagnose
Late in a run, throughput looks the same, the loss looks fine, and improvement slows down. The natural reading is that the model is converging or the learning rate wants attention. An equally consistent reading is that most of your batch is now unanimous and you are training on a small and shrinking slice of it. Those two call for completely different responses, and the only way to tell them apart is to log the share of groups with non-zero advantage, which almost nobody does.
04Three ways out, each with a bill
Bigger groups. More samples, more chances to disagree. Tick K up in the lab and the amber recedes:
| Group size K | p = 0.90 | p = 0.95 | p = 0.99 | Cost per prompt |
|---|---|---|---|---|
| 4 | 34.4% | 18.5% | 3.9% | 4× |
| 8 | 57.0% | 33.7% | 7.7% | 8× |
| 16 | 81.5% | 56.0% | 14.9% | 16× |
| 32 | 96.6% | 80.6% | 27.5% | 32× |
| 64 | 99.9% | 96.2% | 47.4% | 64× |
It works, and it is expensive in exactly the wrong way. Going from K = 8 to K = 64 at p = 0.99 is eight times the compute for six times the signal — and even then more than half your rollouts are still unanimous. Group size can rescue a mildly stale environment. It cannot rescue a solved one.
Difficulty filtering. Keep only the prompts the current policy passes somewhere in the middle of the range. Tick the filter box in the lab at full training progress: the share of groups with a gradient jumps from 58.5% to 88.5%, which looks like a clean win until you read the other number. Only 8.7% of the prompt set survives the filter. You did not fix the environment; you found the small part of it that still works, and you will run out of that part.
Generating new tasks. Which is why the frontier of this work is not better algorithms but environments that produce fresh problems at the difficulty the current policy actually needs. That is a much harder engineering problem than filtering, and the arithmetic above is the reason people are taking it on.
Does a non-binary reward make this go away?
It softens it rather than removing it. With partial credit the spread is rarely exactly zero, so you get a tiny advantage instead of a zero one — but the normalisation by std(r) means a group that nearly agrees produces advantages that are mostly noise amplified to unit scale, which is arguably worse than an honest zero. The useful quantity stops being “did the group disagree” and becomes “did the group disagree meaningfully”, and the same shape returns: it collapses as the policy stops being uncertain. The binary case is the clean version of the problem, not a special case of it.
05What to measure
- Share of groups with non-zero advantage, per step. One number, nearly free to compute, and it is the single best early warning that your environment is expiring. If it is drifting down, extra steps are buying less than the step count suggests.
- The pass-rate histogram of your prompt set, re-measured as you train. Not once at the start. The distribution is what moves, and it is what tells you how much of your environment is still live.
- Useful rollouts, not rollouts. Report throughput as groups-with-gradient per hour. A run doing 10,000 rollouts an hour at 90% unanimity is doing less work than one doing 3,000 at 20%.
- Watch both tails. Because the formula is symmetric, a monitor that only flags “too easy” misses half the problem. Prompts nothing ever solves are just as inert, and they tend to sit in a corpus forever because nobody notices a task that never passes.
06Recap
- A unanimous group has zero advantage and costs the same K rollouts as a useful one.
- P(useful) = 1 − pK − (1−p)K, which at K = 8 and p = 0.99 is 7.7%.
- It is symmetric. Too hard and too easy are the same failure; only unanimity matters.
- Environments decay as the policy improves — 89.5% of groups useful down to 58.5% on a fixed prompt set, with nothing about the environment changing.
- Bigger groups buy signal sub-linearly; 8× the compute bought 6× the signal at p = 0.99.
- Filtering restores the rate but shrinks the pool to 8.7% of prompts, which is why generating tasks beats curating them.
Most of the attention in post-training goes to the objective — PPO versus GRPO versus the next acronym. But the objective can only use the disagreement the environment hands it, and that supply is set by task difficulty, measured against a policy that is changing underneath you the whole time. The algorithm decides what to do with the signal. The environment decides whether there is any.
Sources, and what is mine
The Luckiest Checkpoint, on the other place post-training compute goes quietly missing · Goodhart in the Act, on what happens when the reward is working and you still lose
Nothing here is quoted. The group-relative advantage is stated as the algorithm defines it; everything downstream is derived from that definition and from a binary reward, then checked against simulation before any prose was written. The closed form agrees with a Monte Carlo of 400,000 groups per setting to 9.4×10−4 across fifteen (p, K) pairs, and the three limiting cases — K = 1 can never disagree, p = 0 and p = 1 are always unanimous, and the exact p ↔ 1−p symmetry — hold to machine precision. The lab recomputes everything live from the same expressions, so the figures in the prose are the figures the page renders.
And it is a model. Three simplifications carry the whole thing, and each one makes the real picture somewhat kinder than the arithmetic. Rollouts within a group are treated as independent, when in practice they share a prompt and a policy and are positively correlated — which makes unanimity more likely, not less. The reward is binary, discussed in §04. And the policy is modelled as improving uniformly across tasks, which no real policy does; real improvement is lumpy, so real environments decay unevenly rather than smoothly.
The difficulty distribution in the lab is a Beta(2,3) I chose because it has a plausible shape — most tasks middling, a tail at each end — and the policy-improvement curve is a one-parameter convenience, not a model of learning dynamics. Neither is a measurement of a real training run. They are there to show the mechanism, which is why §05 asks you to log the share of non-zero groups on your own run rather than trusting a number from mine.