Picture your on-call shift a year from now. There is no traffic graph, because there is barely any traffic — forty requests today, total. Instead there are forty processes, each several hours into its own long task, each holding a private pile of state, each quietly spending money. Three are stuck. One has been waiting ninety minutes for a human to answer a question. Two finished twenty minutes ago with confidently wrong output, and nothing anywhere will tell you which two.
Nothing in the standard operations toolkit was built for this. Latency percentiles, error rates, autoscaling, circuit breakers, blue-green deploys — every one of them assumes work that is short, stateless, and cheap to retry. Long-horizon agents are none of those things, and the gap between the tooling we have and the work we are now running is where the next year of agent engineering actually lives.
01The unit of operations changed
One substitution breaks everything downstream: the run replaces the request. It is worth being concrete about what that costs you.
| Assumption | A request | A long-horizon run |
|---|---|---|
| Duration | Milliseconds | Hours. Percentiles over a handful of six-hour runs describe nothing. |
| Retry cost | Free — just send it again | Hours of compute and an unknown amount of half-done side effects. |
| State | Stateless | A working directory, a context window, notes, checkpoints, and an opinion. |
| Failure | Returns a 500 you can count | May return success. That is the entire problem. |
| Deploying | Drain and swap | Someone is always four hours into something. You cannot drain what has no natural end. |
| Cost | Per call, negligible | Accrues every minute, whether or not the run is making progress. |
That last row deserves emphasis because it is the genuinely new operational hazard: a stuck agent and a working agent cost exactly the same per minute. In a request-shaped system, a hung process stops producing value and usually stops consuming resources too. An agent in a doom loop consumes tokens at full speed, forever, while producing nothing — and it looks busy the entire time.
02The fleet, drawn
Here is a whole shift on one screen. Each row is one agent working on its own task; each cell is ten minutes of its life; colour is what it was doing. Runs that need you are sorted to the top, which is the only sort order an operator ever wants.
Spend a moment just reading it. The texture tells you things no aggregate can: a run that is solid green finished cleanly; a run with an orange band in the middle got stuck and dug itself out; a run that goes orange and stays orange is the one you are about to get paged for. You can see the difference between a fleet that is healthy and a fleet that is busy, which is a distinction no success-rate number has ever conveyed.
Now the part worth doing slowly. Uncheck verify every step before accepting it — a change many teams make for latency or cost — and watch the two big numbers separate. Then press “what your dashboard shows.”
The dark red disappears, and nothing else changes
That toggle does exactly one thing: it recolours finished, but wrong as finished. That is not an artistic choice — it is a faithful rendering of what your monitoring can actually distinguish. The run completed. It exited zero. It wrote its output. Every signal it emitted was a success signal.
Which means the honest reading of any agent dashboard is: the green includes an unknown amount of dark red. You cannot measure your way out of this from telemetry alone, because the failure does not generate telemetry. It generates a wrong answer, delivered on time.
This is the operational face of the verification gap. In development it is an abstract worry. In production it is the reason your reported success rate is an upper bound and always will be.
03The four states worth alerting on
“Running” is not a state. It is four states wearing a trench coat, and they need different responses:
| State | What it looks like | What it costs while you ignore it | Response |
|---|---|---|---|
| Progressing | New steps, new artifacts, context advancing | Nothing — this is the good case | Leave it alone |
| Stuck | Busy, spending, but repeating itself — same tool, same error | Full token burn, zero progress | Kill or intervene. Never wait it out. |
| Blocked | Waiting on a human, a lock, an approval | Wall-clock only — the expensive resource is you | Route it to someone, and measure the queue |
| Silently wrong | Indistinguishable from progressing | Everything downstream builds on it | Cannot be detected live. Must be sampled. |
The distinction between stuck and progressing is the single highest-value piece of instrumentation you can build, and it is not hard: a progressing agent produces new artifacts — files touched, tests run, distinct tool calls, novel context. A stuck one repeats. A progress heartbeat that measures novelty rather than activity separates them, and it is the difference between paging someone at minute fifteen and discovering it at hour six.
The only defense against the fourth state is sampling
Silently-wrong runs cannot be found in telemetry, because from the telemetry’s point of view nothing happened. The only thing that works is what every other industry with invisible defects already does: pull a random sample of completed runs and have something competent grade them. A human, or a separate model that never saw the original reasoning.
Sample thirty completed runs a week and grade them properly. That number — not your success rate — is your actual quality metric, and it is the only one that can move without your dashboard noticing.
04What replaces p99 latency
Teams reach for latency because it is what they have. For work measured in hours it carries almost no information — a six-hour run and a nine-hour run are both “fine,” and the difference is dwarfed by task variance. Four metrics do carry information:
Completion rate, split by verified and unverified. One number is a lie for the reason above. Report “94% completed, of which 71% verified correct on sample” and the shape of the problem is immediately visible to everyone.
Intervention rate — human touches per completed run. This is the metric that decides whether the whole thing is economic. An agent that finishes 90% of tasks but needs a human twice per run has not automated anything; it has moved the work and added a queue. Watch it per run, not per day, and watch it trend.
Time-to-detect a stuck run. Not time to fix — time to notice. This is where the money leaks, because the burn is identical to healthy work. If your answer is “when someone checks,” your answer is hours.
Cost per correct completion. Not cost per run. Failed runs still cost full price, so the real figure is total spend divided by correct outputs. At 69% correctness that is 1.45× your nominal cost per run; at 50% it is double. This is the number to put in front of a finance team, because it is the one they would have computed themselves.
05Shipping when the feedback loop is a day long
Continuous deployment assumes you learn quickly whether a change was good. With six-hour runs, a canary started at 9am tells you nothing until mid-afternoon, and by then you have merged four more things.
Canary by run, not by traffic percentage. Route the next N runs to the new version rather than a share of requests; with forty runs a day, 5% of traffic is two runs and no statistical power whatsoever. Think in runs, and accept that meaningful comparisons take days.
Lean on leading indicators. You do not have to wait for terminal outcomes. Intervention rate in the first hour, stall rate in the first hour, tokens-per-step-of-progress — these are all measurable early and correlate with how the run ends. A version that is stalling twice as often at hour one is not a version you need six hours of patience to evaluate.
Pin a run to a version for its lifetime. A run that starts on v4 should finish on v4. Otherwise a deploy silently changes the rules mid-task, and you get failures that are unreproducible by construction, because the thing that ran no longer exists.
06The stop button is harder than it looks
Every serious agent deployment eventually needs to stop everything: a bad release, a runaway cost, an incident. This is meaningfully harder than draining a web tier, for reasons this series has already covered from the other side.
An agent killed at hour five without checkpoints loses five hours of work — and, worse, may leave the world half-changed. Files written, PRs opened, emails sent, rows updated. There is no undo button, so a kill switch is not an off switch; it is a transaction boundary problem.
What works is boring and needs to exist before the incident: checkpoint at every safe point so stopping costs minutes rather than hours; distinguish pause from kill, because pausing an agent that holds a lock is not the same as terminating it; make all external effects idempotent and logged, so a resumed run does not send the email twice; and drain first, kill second — stop admitting new runs before you start terminating live ones, which alone handles most incidents without losing any work at all.
07Five panels, buildable this week
If you take one thing from this post into a repository, take this list. None of it requires new infrastructure.
- The fleet strip. One row per active run, coloured by state, sorted with trouble on top. The chart above is a toy and still tells you more at a glance than any table of run IDs.
- Progress novelty, not activity. Per run: distinct tool calls and artifacts produced in the last fifteen minutes. Near zero with high token burn is the stuck detector.
- The intervention queue. Every run waiting on a human, with how long it has waited. This is a work queue, and it should look like one.
- Cost per correct completion. Spend divided by sampled-correct outputs. Expect it to be uncomfortable, and expect that to be useful.
- The sampled-quality number. Thirty completed runs a week, graded by something that did not produce them, tracked over time. This is the only line on the whole dashboard that can see the fourth state.
The reframe
Agent operations is not monitoring a service. It is closer to running a shift of junior staff who work extremely fast, never ask for help unless you build them a way to, and will confidently hand you finished work that is wrong. Every panel above exists to answer one of three questions a shift lead asks: who is stuck, who is waiting on me, and whose work should I check?
Takeaways
- The run is the unit now. Every ops primitive you inherited assumes the request, and quietly misleads you when work takes hours.
- Your success rate is an upper bound. Finished and finished-wrong emit identical signals. Sample and grade, or fly blind with a green dashboard.
- Stuck costs exactly what working costs. Detect it by novelty, not activity, and measure time-to-detect as a first-class metric.
- Intervention rate decides the economics. Completion without it is vanity; an agent that needs two human touches per run has relocated the work, not removed it.
- Build the stop button before you need it. Checkpoint, make effects idempotent, drain before you kill.
Related reading
Why long tasks fail: the arithmetic · What the context loses on the way · The verification gap · No undo button · Visualizing agent systems · How much autonomy to grant
The fleet view is a simulation with stated parameters, not telemetry from a live system. Its model — and every number quoted from it — was verified independently before publication and is reproducible in a few dozen lines of JavaScript.