Here is the most uncomfortable fact in agent-land right now: two agents can both “pass” a long-horizon task while one of them took 40 verified tool calls and the other took 400 flailing ones — retried the same broken command nine times, declared victory twice before it was true, and burned 15× the tokens. If your eval reports a leaderboard of final answers, those two agents are tied. Everything that distinguishes them — everything you actually care about in production — happened inside the trace your eval threw away.
Short tasks forgive this. A 3-step tool call has nowhere to hide. But as the horizon stretches to hundreds of steps and hours of wall-clock — and METR’s measurements say the tasks agents can attempt keep doubling — the gap between finished and finished well becomes the entire evaluation problem. This post is about closing that gap: a controlled six-agent experiment, a discrimination framework of trace metrics, a way to bundle agents into behavioral clusters, and a live lab where you can watch all of it move.
01Why outcome metrics stop discriminating
Three separate failures stack up as the horizon grows:
- Saturation and variance. pass@1 is a coin-flip statistic. On τ-bench, GPT-4o passed a task once ~61% of the time but passed all 8 identical retries only ~25% of the time. One number, drawn from a distribution that wide, cannot rank two agents that differ by a few points.
- Sparse terminal reward. A 200-step task graded at step 200 gives you one bit of signal for 200 decisions. All the information about where it went wrong — and whether it would go wrong again — is in the intermediate steps.
- Cost blindness. Outcome metrics are unit-free. An agent that succeeds by brute-force exploration looks identical to one that succeeds by planning — until you deploy it and the invoice arrives.
› Go deeper: pass@k, pass^k, and which one to report
pass@k (“at least one of k tries succeeds”) rewards lucky variance — it goes up as your agent gets flakier. pass^k (“all k tries succeed”, τ-bench’s formulation) measures the thing production actually needs: repeatability. For long-horizon work, report pass^k alongside pass@1, and treat a big gap between them as a reliability alarm, not a footnote. The “Beyond pass@1” line of work makes the same argument with self-consistency baked in.
02The experiment: six agents, one task, everything else pinned
Discrimination questions only make sense under control. So the setup is deliberately boring: one task list, one model, one tool set, one harness, one budget. The only thing that varies is the workflow scaffold — the “shenanigans” each agent is allowed to do between the same start and finish line:
| Agent | Workflow | Signature behavior |
|---|---|---|
| ReAct baseline | Think → act → observe, no plan, no memory | The “normal shenanigans” — honest, wandering, average at everything |
| Planner | Decompose first, batch tool calls, execute the plan | Fewest actions per subgoal — efficiency profile |
| Doom Looper | Retries a failing action unchanged | Same call, same error, nine times — redundancy profile |
| Eager Finisher | Declares done at the first plausible answer | Great pass@1 under a lazy judge, terrible under a verified one |
| Scattershot | Explores broadly, follows tangents | Real progress, enormous token bill — cost profile |
| Self-Corrector | Verify each step, record failures, never repeat them | Slow start, learns as it goes — the compounding profile |
Graded on final answers alone, four of the six land within a few points of each other — and the Eager Finisher can lead if the judge only checks that an answer was produced. Graded on traces, no two of them are confusable. That’s the entire thesis in one experiment, and you can run it yourself in Section 04.
03The discrimination framework: five metric families
“A is better than B” is not one question — it’s five. Each family below answers a different one, discriminates a different failure mode, and (Section 05) prescribes a different fix. The position family is the one most people skip: where in the trace progress happens is as diagnostic as whether it happens.
| Family | Metrics | The question it answers | Catches |
|---|---|---|---|
| Outcome | verified pass@1 · pass^k · reliability@k | Did it finish — repeatably, and did the work actually check out? | Eager Finisher |
| Efficiency | action efficiency (needed / taken) · tokens per verified subgoal · success per 1k calls | What did the result cost? | Scattershot |
| Position | recall@K (subgoals verified within first K actions) · position of first progress · progress-curve AUC | Where in the trace does progress happen — front-loaded or last-minute? | Planner vs. everyone |
| Redundancy | loop rate (repeated identical calls) · backtrack ratio · action n-gram entropy | Is it exploring, or circling? | Doom Looper |
| Verification | verified-step fraction · premature-done rate · unverified-claim count | Does it check its work before it believes itself? | Self-Corrector (positively) |
› Go deeper: recall@K and progress-AUC, precisely
Define the task as M verifiable subgoals. recall@K = (subgoals verified within the first K actions) / M — a direct import of the ranking metric, with the action index playing the role of rank position. It rewards agents that surface progress early, exactly like recall@K rewards retrieers that surface relevant documents early. Progress-AUC integrates the progress curve over the whole budget: two agents with the same endpoint but different curves (steady climber vs. last-minute sprinter) get very different AUCs — and the steady climber is the one you can checkpoint, resume, and trust mid-flight. Position-of-first-progress is the leading indicator: an agent that hasn’t verified a single subgoal by K actions almost never recovers, which makes it a cheap early-abort signal in CI.
› Go deeper: bundling agents — fingerprints, clusters, and kNN drift alarms
Once every run is a vector of these metrics, agents stop being names and become points in behavior space. Cluster the vectors (k-means is enough) and workflows bundle into families — efficient-verifiers, honest-wanderers, pathological-loopers — regardless of what their authors called them. The operational payoff is drift detection: embed every new version’s runs and ask its k-nearest neighbors which cluster they live in. A “prompt improvement” that quietly moves your agent’s fingerprint toward the looper cluster is a regression your outcome metrics will take weeks to notice — the fingerprint catches it in one eval run. The MAST failure taxonomy plays the same role for multi-agent systems: name the behavior, then measure your distance to it.
04The live lab: run the experiment yourself
Everything above, running now. Six agents attack the same task in your browser; every chart is computed from their actual traces this second — nothing is canned. Watch the two headline numbers: the answer-only spread (what a lazy leaderboard sees) sits at zero — a six-way tie — while the fingerprint spread grows as you stretch the horizon. Click a legend name to isolate an agent across all three views; re-run to feel the variance.
Progress vs. actions — mean of 24 runs (the position family, drawn)
Trace fingerprint — 6 metric axes, normalized
Behavior space — efficiency × verification, k-means bundles (size = tokens)
Synthetic agents, real arithmetic: traces are simulated per run from each workflow’s behavior profile; every metric, cluster, and curve is computed from those traces live. D3, no server, seeded so your first render matches mine.
Three things to notice while you play. First, the pass@1† column is a flat six-way tie at every horizon — every agent hands the judge something answer-shaped, and a judge that doesn’t verify literally cannot tell them apart. That’s not a contrived worst case; it’s what “the demo worked” means. Second, the verified column separates them — and the separation widens with horizon: stretch from short to long and watch every agent’s verified rate sag except the Self-Corrector’s, which holds at 100% because its per-step failure odds fall as its pitfall log grows — compounding that only trace metrics can see coming. Third, the Eager Finisher: answer-only says it’s tied for first, verified says it’s dead last by the medium horizon. If your judge doesn’t check work, you are actively selecting for this agent.
05Found a difference? The response framework
Discrimination without response is trivia. The point of finding which metric separates A from B is that each family maps to a specific intervention — this is the operational half of the framework:
| Symptom | Metric that catches it | Respond with |
|---|---|---|
| Circles on failures | loop rate ↑, n-gram entropy ↓ | Attempt memory (“tried, failed, why”) injected into context; hard cap on identical retries |
| Declares victory early | premature-done rate ↑, verified pass « answer-only pass | Verification gate before “done” is accepted; judge checks work, not answers |
| Succeeds expensively | action efficiency ↓, tokens per subgoal ↑ | Plan-then-act scaffold; batch tool calls; budget in the reward |
| Fades late in the task | progress-AUC front-loaded, late-third velocity ↓ | Context compaction + checkpointing (the marathon playbook) |
| Flaky across retries | pass@1 » pass^k | Report pass^k; find the variance source in the trace before shipping anything |
| New version “feels off” | fingerprint drifts toward a bad cluster (kNN) | Behavioral regression gate in CI — block merges that move the fingerprint |
The framing I’d push
Stop asking “which agent scored higher?” and start asking “which behavior do these traces exhibit, and is that the behavior I want to ship?” A trace is not eval exhaust — it’s the primary evaluation object. The final answer is one field in it.
06Where this framework doesn’t save you
· Trace metrics are gameable too. Optimize loop rate directly and agents learn to launder repetition through trivially-varied calls — Goodhart doesn’t spare process metrics (I’ve watched it happen). Keep a held-out verified-outcome anchor; use trace metrics to diagnose, not as the training objective.
· This lab is a model, not a benchmark. The six agents are behavior profiles with honest arithmetic on top — great for seeing why the framework discriminates, not evidence about any real product. On real systems the same pipeline applies, but subgoal verification is the hard engineering: you need checkable milestones, which most tasks don’t come with for free.
· Verification itself costs. Step-level judging of a 400-call trace is real money and real latency. Sample: verify every step in CI, spot-check in production, and let the cheap counters (loops, positions, budgets) run everywhere at full rate.
Takeaways for running long-horizon agents
- Grade the trace, not the finish line. Past ~50 steps, outcome-only evaluation measures luck and hides cost.
- Report pass^k next to pass@1. The gap between them is your reliability debt, quantified.
- Verify before you believe. An answer-only judge will actively select for premature finishers.
- Watch position, not just totals. recall@K and progress-AUC are your earliest early-warning signals — and the cheapest abort criterion you’ll ever ship.
- Normalize by cost. Success per 1k tool calls separates planners from scattershots that raw accuracy cannot.
- Fingerprint and cluster every version. Behavioral drift toward a bad bundle is a regression signal that arrives weeks before your outcome metrics move.
- Respond by family. Loop → memory. Premature → gates. Expensive → planning. Fade → checkpoints. The metric that caught it names the fix.
Guardrails and security get an agent allowed into production. Trace-level evaluation is what tells you it belongs there.