Here is a question worth sitting with before reading on. An agent completes individual steps correctly 99% of the time. That is a good agent — better than most things you have deployed. How long a task can it finish? Most people answer with something in the hundreds of steps, because 99% feels like “almost always.” The answer is sixty-nine. After sixty-nine steps, a 99% agent is a coin flip, and after three hundred it has a 5% chance of having gotten everything right.
This is not a subtle result or a contested one. It is just pn, and it is the single most load-bearing fact in long-horizon agent engineering — the reason a system that dazzles for ten minutes dissolves over six hours. Every technique in this series, memory and checkpointing and verification and decomposition, is ultimately an attempt to move one of the terms in that expression. This post is about what the expression actually says, which terms are movable, and how much each one is worth.
01The cliff
Assume for a moment that steps fail independently — a lie we will correct shortly, but a clarifying one. A task of n steps succeeds only if every step succeeds, so:
| Per-step reliability | 10 steps | 50 steps | 100 steps | 500 steps |
|---|---|---|---|---|
| 90% | 35% | 0.5% | 0.003% | ~0 |
| 95% | 60% | 7.7% | 0.6% | ~0 |
| 99% | 90% | 61% | 37% | 0.7% |
| 99.9% | 99% | 95% | 90% | 61% |
Read that table by row and it is unremarkable. Read it by column and something uncomfortable appears: at 500 steps, the difference between a 95% agent and a 99.9% agent is the difference between never and usually. On short tasks those two agents are hard to tell apart — 60% versus 99% on a ten-step task is one bad demo away from noise. The gap only becomes visible at exactly the length where you can no longer afford to discover it.
This is also why benchmark improvements feel disappointing in production. Moving a model from 95% to 97% per-step sounds like a 2-point gain and is described that way in release notes. On a 200-step task it takes you from 0.0035% to 0.23% — a 65× improvement that is still, in absolute terms, a system that does not work.
02The horizon is one number
Instead of asking “what is the success rate at length n,” invert the question: how long a task can this agent finish half the time? Set pn = 0.5 and solve:
That is the whole thing. One parameter in, one number out, and it behaves in a way worth memorizing:
| Per-step reliability | Error rate | 50% horizon | At 30s per step |
|---|---|---|---|
| 90% | 1 in 10 | 6.6 steps | 3 minutes |
| 99% | 1 in 100 | 69 steps | 35 minutes |
| 99.9% | 1 in 1,000 | 693 steps | 5.8 hours |
| 99.99% | 1 in 10,000 | 6,931 steps | 2.4 days |
The 10× rule
Every 10× increase in horizon requires a 10× reduction in per-step error rate. Not a better model in a vague sense — an order of magnitude fewer mistakes per action. That is the exchange rate, it is linear in the log, and nothing in the equation offers a discount.
The rule cuts both ways, which is the encouraging half. If your agent currently dies after twenty minutes and you need it to survive three hours, you do not need a fundamentally different system; you need to find and remove nine out of every ten errors. That is a concrete, boring, tractable engineering target, and it is a much better brief than “make the agent more reliable.”
03The calculator
Below is the model with its assumptions exposed. Two additions to the naive version make it honest. Attempts per step lets a failed step be retried. Deterministic failures is the fraction of failures that will fail again in exactly the same way no matter how many times you retry — a missing credential, a wrong assumption, a malformed schema, an API that is genuinely down. The remainder are transient: a flaky network call, a temperature-sampled slip, a race.
Start with the deterministic slider at zero, which is the assumption every retry-based argument quietly makes. Three attempts per step at 99% reliability produces an effective reliability of 99.9999% and a horizon of roughly 693,000 steps — a number so absurd it should immediately make you suspicious. Now drag deterministic failures to 25%.
A quarter of failures being stubborn costs you 99.96% of the benefit
Horizon falls from ~693,000 steps to 277. That is a 2,500× tax, paid entirely because of a modeling assumption most people never write down. And 25% is optimistic: in agent work, the majority of failures are the deterministic kind. The wrong file path is wrong every time. The misunderstood requirement stays misunderstood.
Which reframes what a retry loop is for. Retries are not a reliability mechanism; they are a transient-noise filter. They buy real but bounded value, and the moment a team starts treating “we retry three times” as the answer to reliability, the arithmetic has already left the building.
› Go deeper: the model, and where it is wrong
The calculator computes effective per-step reliability as peff = 1 − (1−p) × [ c + (1−c)(1−p)k−1 ], where k is attempts per step and c is the deterministic fraction; horizon is then ln(0.5)/ln(peff). Sanity checks: at k=1 it reduces to p; at c=0 it reduces to the familiar 1−(1−p)k; at c=1 retries do nothing at all.
Three ways this model is still too kind. It assumes steps are exchangeable — that every step carries the same risk. Real tasks have a few catastrophic steps (the migration, the delete, the deploy) and a long tail of trivial ones, and the interesting question is not average reliability but reliability on the steps that cannot be undone. It assumes failures are detected. A step that fails silently does not get retried; it corrupts everything downstream, which is strictly worse than stopping. And it assumes independence across steps, when in practice one early misunderstanding raises the failure probability of every step after it — the error is in the context now, and the agent keeps reading the context.
Each of those makes real horizons shorter than the calculator says. The model is a ceiling, not a forecast.
04Why every published horizon is a flattering number
The field has converged on measuring exactly this quantity. The best-known effort, METR’s time-horizon work, reports how long a task — measured in the time a skilled human would need — an agent can complete with 50% success, and tracks that number across model generations. It is the same construction as the formula above, with human-minutes as the unit instead of steps.
Two things about that metric are routinely misread, and the arithmetic settles both.
First: a horizon is not a runtime. “This model has a four-hour horizon” does not mean it works unattended for four hours. It means that on tasks a human would need four hours for, it succeeds about half the time — and therefore fails about half the time. An agent you would actually leave alone for four hours is one that almost never fails at four hours, which is a completely different and much shorter number.
Second, and more useful: the 50% threshold is doing enormous work, and exactly how much is derivable. The horizon at any success threshold s is ln(s)/ln(p), so the ratio between two thresholds is ln(s1)/ln(s2) — and p cancels. The relationship between a headline 50% horizon and the horizon you could actually ship is a fixed constant, identical for every agent that has ever existed or will:
| Success threshold | Share of the 50% horizon | Shorter by | At a 693-step 50% horizon |
|---|---|---|---|
| 50% — the published number | 100% | — | 693 steps |
| 80% — a generous demo | 32.2% | 3.1× | 223 steps |
| 95% — a tolerable internal tool | 7.4% | 13.5× | 51 steps |
| 99% — anything customer-facing | 1.4% | 69× | 10 steps |
Divide every headline horizon by thirteen
If you need 95% reliability — a modest bar for anything running without a human watching — your usable horizon is 7.4% of the published 50% figure. That is a 13.5× haircut, it is the same 13.5× for every model, and it does not improve as models improve. When a horizon number doubles, the 95% horizon doubles too — and stays 13.5× below the headline.
There is a useful diagnostic buried in that constant. Because 3.1× is what the constant-reliability model predicts for the gap between a 50% and an 80% horizon, it is the optimistic case — it assumes steps fail independently. If a measured 80% horizon comes in substantially worse than a third of the measured 50% horizon, the excess is evidence of exactly the correlation this post is about: failures clustering, one mistake making the next more likely. The gap between the predicted ratio and the observed one is a free estimate of how correlated your failures are.
› Go deeper: this result is older than LLM agents
Compounding error over a horizon is a known hazard in sequential decision-making, and the sharpest classical statement comes from imitation learning. Ross and Bagnell showed that a policy trained by behavior cloning with per-step error rate ε suffers regret that grows quadratically in the horizon — roughly εH² — because each mistake moves the agent into states unlike anything it trained on, where it is more likely to err again. Their DAgger algorithm brings this back to linear in H by training on the states the policy actually visits.
That is the same phenomenon as this post’s correlation term, arrived at from a different direction, and the fix rhymes: the danger is not the isolated mistake but the drift into a regime where mistakes breed. An LLM agent that misreads a requirement at step 5 is, by step 40, reasoning inside a context window that contains its own error — off the distribution where it is reliable, in precisely Ross and Bagnell’s sense.
A note on what this post does not claim
You will notice no specific frontier horizon figures here — no “model X reached N hours.” That is deliberate. Those numbers move monthly, the measurement suites they come from are near saturation at the top end, and published confidence intervals at the frontier are wide enough to span an order of magnitude. I could not verify any current figure against its primary source at the time of writing, so I have not printed one. Everything above is derived from the arithmetic and checkable in a Node REPL in about thirty seconds, which is the only kind of number this post is willing to assert.
05The three terms you can actually move
Once the horizon is written as ln(0.5)/ln(peff) over n steps, the engineering menu becomes finite and unusually clear. There are exactly three things to attack, and most agent infrastructure is one of them in disguise.
| Term | What moves it | What it buys |
|---|---|---|
Fewer steps (n) | Better tools that do more per call, batching, decomposition into sub-tasks with their own horizons, letting the agent write code instead of taking twenty actions by hand | Linear, and the cheapest win available. A tool that collapses ten steps into one is worth the same as a 10× reliability improvement on those steps. |
Higher per-step reliability (p) | Unambiguous tool contracts, constrained outputs, verification before commit, removing avoidable judgment calls from the loop | Logarithmic leverage — the 10× rule. The most valuable and the slowest to earn. |
Less correlation (c) | Making retries differ: new context, a different tool, an escalation path, a fresh sub-agent that has not read the poisoned reasoning | Enormous and almost entirely neglected. This is the term with the most headroom in most systems. |
That third row deserves more attention than it gets. A retry that replays the same prompt with the same context is not a second sample from the distribution — it is close to a deterministic repeat, because the thing that caused the failure is still sitting in the context window. The doom loop is exactly this: six identical attempts, six identical failures, c ≈ 1.
Whereas a retry that changes something — clears the context, swaps the tool, spawns a subagent with a clean window and just the task description — genuinely re-samples, and moves c toward zero. Most of the value people attribute to multi-agent architectures is, I think, this: not parallelism, but decorrelation. The subagent's real contribution is that it has not read the mistake.
The reframe I would push
“Make the agent more reliable” is not an engineering task. “Find the nine errors in ten we can remove, halve the number of steps, and make every retry differ from the attempt before it” is three engineering tasks, each with a number attached and a way to tell whether you are winning.
Takeaways
- The horizon is
ln(0.5)/ln(p). One number describes how long your agent can work. Compute it, publish it, and watch it move — it is a far better health metric than a benchmark score. - Ten times the horizon costs ten times fewer errors. Linear in the log, no discounts. Any roadmap promising a large horizon gain without an order-of-magnitude reliability gain is promising something the arithmetic does not allow.
- Retries filter noise; they do not create reliability. Their entire value lives in the fraction of failures that are transient, and that fraction is smaller than anyone's mental model of it.
- Divide headline horizons by thirteen. A 50% horizon is 13.5× longer than the 95% horizon you would actually deploy — a fixed ratio that never improves, because
pcancels out of it. - Decorrelation is the neglected lever. Make each retry differ from the last — new context, different tool, fresh subagent — and you move the one term in the equation with real headroom left.
- Shorter tasks beat smarter agents. Halving
nis usually easier than halving the error rate, and it is worth exactly as much.