Every team deploying agents relearns the same two failure modes in the same order. First they ship the cautious agent — it confirms before renaming a variable, checks in before opening a draft PR, asks permission to read a file. Within a week the humans rubber-stamp every question without reading it, which quietly converts “human oversight” into “human latency.” So they crank the dial the other way — and some Tuesday the agent refunds the wrong customer $9,800, tells no one, and files the task as complete. The postmortem never says the model was weak. It says nobody designed the policy for when it should have asked.
That policy has exactly three inputs, and you already track all of them: the agent’s confidence it’s right (calibrated, not self-reported vibes), the stakes of the action (blast radius: dollars, users, data), and the reversibility — whether an undo exists, which is a property you can engineer, as the no-undo post argued. Multiply them and you get expected loss; compare that to the very real cost of interrupting a human — attention, flow, and the trust erosion of one-too-many pointless questions — and the whole space collapses into three regimes:
| Regime | When | Examples | The contract |
|---|---|---|---|
| Just do | Low stakes or cheap undo, decent confidence | Rename a variable, reformat a doc, read-only queries, draft anything | Act silently; the trace is the record |
| Tell after | Moderate expected loss, reversible, monitored | Merge with tests green, send routine reply, archive stale tickets | Act, then surface a one-line receipt a human could veto |
| Ask first | High stakes × irreversible, or low confidence on anything that matters | Wire payments, delete data, external comms, anything customer-visible & permanent | Stop, checkpoint state, escalate with full context — then respect the answer |
› Go deeper: the asymmetry that shapes everything
The two error costs are wildly asymmetric — but in both directions at once, which is why this is hard. A missed catastrophe is unbounded (one bad wire transfer outweighs a thousand saved questions), which argues for caution. But interruption cost compounds: every unnecessary question trains the human to stop reading, so over-asking doesn’t just waste time — it degrades the oversight you’ll need for the question that matters. The rubber-stamp is the quiet killer: an approval nobody read is worse than no approval, because it looks like oversight in the audit log. Your policy has to defend both frontiers simultaneously — which is exactly what the observatory below lets you feel.
01The observatory: drag the dial, watch the tradeoff
Below is a day in the life of an agent: 400 decisions streaming through confidence×stakes space. Circles are reversible actions, diamonds are irreversible ones; red rings mark decisions that were actually wrong. The shaded bands are your policy — drag the two thresholds and every view updates: the map recolors, the counters move, and the you-are-here dot slides along the Pareto curve that shows the whole frontier of policies you could have chosen. Hover any diamond to see the concrete decision behind it.
Decision space — confidence × stakes · ◆ irreversible · ● reversible · red ring = actually wrong
The policy frontier — interruptions vs. catastrophes
Push ask-first left: interruptions explode, the rubber-stamp begins. Push it right: watch the missed-catastrophe counter find the red diamonds. The dot on the frontier is the honest summary of your policy — most teams have never seen where theirs sits. One rule sits above both sliders: irreversible + extreme stakes always asks, no matter how confident the agent — expected value has no business averaging over an unbounded downside.
Three things the picture teaches that the prose can’t. First, the frontier is steep at both ends — near-zero missed catastrophes is cheap until the last one, and each further interruption you remove past the knee costs disproportionate risk; the knee is where deliberate policies live. Second, irreversibility does most of the work: the same threshold that quietly waves through fifty reversible mistakes (they get caught and undone downstream) must catch the three red diamonds — which is why engineering undo, per the rollback post, literally moves decisions out of the danger zone rather than just flagging them. Third, confidence alone is a terrible trigger: an agent 95% sure about a $50k wire is still a 1-in-20 disaster — the vertical band of high-confidence, high-stakes diamonds is exactly the region naive “ask when unsure” policies miss.
02The trust ratchet: autonomy is earned per task-class
The dial shouldn’t be global, and it shouldn’t be static. The right unit is the task-class (“dependency bumps”, “customer emails”, “payments”), and the right mechanism is a ratchet: every class starts supervised, graduates toward autonomy on a streak of verified successes — verified in the trace-metric sense, not “the agent said done” — and demotes instantly on a failure. Below, four task-classes earning (and losing) autonomy over sixty tasks:
Autonomy level per task-class · 0 = supervised · 1 = ask-first · 2 = tell-after · 3 = autonomous · ✕ = failure → demotion
Reversible classes climb fast and recover fast. Payments spend the whole run pinned at ask-first — correctly: no streak is long enough to buy autonomy over an unbounded downside.
The ratchet resolves the argument every team has (“should the agent be allowed to…?”) by making it empirical: it’s allowed when its track record on that class bought it the level, and the purchase reverses. It also gives you the safe rollout order for free: graduate the reversible classes first — they’re cheap to be wrong about, so streaks there are cheap to earn honestly.
03Mechanics that make the dial real
- Checkpoint before you ask. An escalation that loses the agent’s working state converts every question into a restart. Persist the progress file first (the memory post’s handoff discipline), then yield — the answer might arrive tomorrow.
- Escalate with a decision, not a situation. “Should I refund $9,800 to Acme — policy says ≤$500 without approval, their invoice history is attached, I recommend yes because…” A human can answer that in ten seconds. “What should I do about Acme?” is homework, and homework gets rubber-stamped.
- Budget the questions. N asks per task; exceeding the budget is itself a signal the task is misspecified — stop and escalate that. Batch the non-blocking ones.
- Default on timeout — along the reversible path. “If no answer in 30 minutes, take the undoable option and log it” keeps marathon tasks moving without converting silence into consent for anything permanent.
- Make the receipt greppable. Tell-after only works if the tells land somewhere a human actually scans — one line, one link to the trace, in the channel they already read.
The framing I’d push
Stop asking “how much do we trust the agent?” and start asking “what’s our interruption budget, and where does it buy the most safety?” Attention is the scarce resource. Spend it where expected loss is highest — which is a policy you can draw, measure, and tune, not a vibe.
04Where the dial doesn’t save you
· Models are miscalibrated. Self-reported confidence is not P(correct); it’s a vibe with a percent sign. Calibrate against your own eval traces per task-class, or use the track record itself as the confidence signal — the ratchet does this implicitly.
· Blast radius is a judgment call the agent makes. An agent that misjudges stakes routes itself around your policy. Anchor stakes to mechanical properties where possible — dollar amounts, row counts, externally-visible or not — and audit the classifier like any other component.
· Approval fatigue is measurable — so measure it. Track time-to-approve and approval-without-open rates. When median approval drops under a few seconds, your ask-first regime has become theater, and the honest fix is fewer, better questions — not more oversight.
Takeaways
- Design the policy, don’t inherit it. Ask/tell/do thresholds on expected loss — drawn, measured, tuned.
- Reversibility does the heavy lifting. Engineering undo moves decisions out of the danger zone; flagging them just decorates it.
- Never trigger on confidence alone. High-confidence, high-stakes, irreversible is the exact quadrant naive policies miss.
- Ratchet autonomy per task-class — earned on verified streaks, demoted on failure, reversible classes first.
- Checkpoint before every escalation, escalate decisions not situations, budget the questions, default reversible on timeout.
- Watch the humans too: sub-second approvals mean your oversight is already gone — the fix is better questions, not more of them.
- Know where you sit on the frontier. Every deployment has a point on that curve; most teams have simply never looked.
The 30-hour autonomous agent and the never-asks-anything agent are the same machine with different policies. The policy is the product decision — make it on purpose.