sugeerth.github.io / agentdiff

Open source · MIT · Python 3.10+, no dependencies

AgentDiff: git diff for AI agents

Two agents ran the same task. One succeeded and one did not. AgentDiff compares the two runs step by step and names the step where they parted, with the evidence for it. It also guards Claude Code while it works, and lets a harness evolve on your own repository's history. Every verdict is a rule over counts. A model may narrate, but it never changes a number.

Code on GitHub Download for macOS, Windows, Linux How self-evolving works How it works

Try it

pip install git+https://github.com/sugeerth/Agenttracing
agentdiff                          # the hub, on your own Claude Code sessions, live
agentdiff self-evolve --demo       # evolve your Claude Code's harness on six tasks
agentdiff self-evolve --from-git ~/your-repo   # ...or on your repository's own history
agentdiff demo --open              # eight shipped pairs, each compared step by step

What it does

Diff two runsIt aligns two agents' steps, finds where they diverged and attributes the failure to a step, or says it can't. On 2,200 pairs with an implanted cause it named the kind of cause 98.1% of the time.
Guard Claude Codeagentdiff guard --install works through Claude Code's hooks. It refuses a failing command about to run again unchanged, and sends an unchecked finish back to run the tests. Every session is traced.
Evolve a harnessAgents run under a harness: instructions, denied tools, a turn cap. Evals name a change and the paired runs decide it. The change is kept only if it wins more tasks than it loses and breaks none. A weak win is run again.
See it allA local hub shows traces read as loops, live runs, and every evolution run by run, as server-side SVG in light and dark.

The evolution, run by run

A recorded run of Claude Code (haiku) on six tasks whose specs state every edge case. Each task is graded by tests the agent never sees. Rows are tasks and columns are arms. Each square is one run: green passed, red failed. Click a square to open that run's trace.

the tasks the change moved, run again, both armsg0exploredBefore you finish, list every rule and edge case the task and the code's docstrings state, write a quick test for each one, run them, and fix what fails. — it won 1 task(s) and lost 0explore:edge_casesBefore you finish, list every rule and edge case the task and the code's docstrings state, write a quick test for each one, run them, and fix what fails. — it won 1 task(s) and lost 0kept · held on re-rung1every run passedas it wasas it wasv017 of 18 runs passed17/18with the changethe changev118 of 18 runs passed18/18run again, as it wasre-runv02 of 3 runs passed2/3run again, with the changere-runv13 of 3 runs passed3/3as it was (the runs of the kept change, carried)keptv118 of 18 runs passed18/18croncroncron · r1 · g0, as it was (v0) · passed, 37 scron · r2 · g0, as it was (v0) · passed, 37 scron · r3 · g0, as it was (v0) · passed, 31 scron · r1 · g0, with the change (v1) · passed, 73 scron · r2 · g0, with the change (v1) · passed, 65 scron · r3 · g0, with the change (v1) · passed, 57 scron · r1 · g1, as it was (v1) · passed, 73 scron · r2 · g1, as it was (v1) · passed, 65 scron · r3 · g1, as it was (v1) · passed, 57 sexprexprexpr · r1 · g0, as it was (v0) · passed, 25 sexpr · r2 · g0, as it was (v0) · passed, 31 sexpr · r3 · g0, as it was (v0) · passed, 25 sexpr · r1 · g0, with the change (v1) · passed, 271 sexpr · r2 · g0, with the change (v1) · passed, 290 sexpr · r3 · g0, with the change (v1) · passed, 52 sexpr · r1 · g1, as it was (v1) · passed, 271 sexpr · r2 · g1, as it was (v1) · passed, 290 sexpr · r3 · g1, as it was (v1) · passed, 52 sglobmatchglobmatchglobmatch · r1 · g0, as it was (v0) · passed, 30 sglobmatch · r2 · g0, as it was (v0) · passed, 23 sglobmatch · r3 · g0, as it was (v0) · passed, 29 sglobmatch · r1 · g0, with the change (v1) · passed, 49 sglobmatch · r2 · g0, with the change (v1) · passed, 25 sglobmatch · r3 · g0, with the change (v1) · passed, 36 sglobmatch · r1 · g1, as it was (v1) · passed, 49 sglobmatch · r2 · g1, as it was (v1) · passed, 25 sglobmatch · r3 · g1, as it was (v1) · passed, 36 sisodurisodurisodur · r1 · g0, as it was (v0) · passed, 22 sisodur · r2 · g0, as it was (v0) · passed, 30 sisodur · r3 · g0, as it was (v0) · passed, 23 sisodur · r1 · g0, with the change (v1) · passed, 42 sisodur · r2 · g0, with the change (v1) · passed, 30 sisodur · r3 · g0, with the change (v1) · passed, 36 sisodur · r1 · g1, as it was (v1) · passed, 42 sisodur · r2 · g1, as it was (v1) · passed, 30 sisodur · r3 · g1, as it was (v1) · passed, 36 sisoweekisoweekisoweek · r1 · g0, as it was (v0) · passed, 32 sisoweek · r2 · g0, as it was (v0) · passed, 277 sisoweek · r3 · g0, as it was (v0) · passed, 83 sisoweek · r1 · g0, with the change (v1) · passed, 39 sisoweek · r2 · g0, with the change (v1) · passed, 35 sisoweek · r3 · g0, with the change (v1) · passed, 46 sisoweek · r1 · g1, as it was (v1) · passed, 39 sisoweek · r2 · g1, as it was (v1) · passed, 35 sisoweek · r3 · g1, as it was (v1) · passed, 46 sthe change moved this taskrangesrangesranges · r1 · g0, as it was (v0) · passed, 80 sranges · r2 · g0, as it was (v0) · failed, 87 sranges · r3 · g0, as it was (v0) · passed, 70 sranges · r1 · g0, with the change (v1) · passed, 94 sranges · r2 · g0, with the change (v1) · passed, 82 sranges · r3 · g0, with the change (v1) · passed, 103 sranges: passed more often with the changeranges · r1 · g0, run again, as it was (v0) · passed, 75 sranges · r2 · g0, run again, as it was (v0) · passed, 87 sranges · r3 · g0, run again, as it was (v0) · failed, 114 sranges · r1 · g0, run again, with the change (v1) · passed, 107 sranges · r2 · g0, run again, with the change (v1) · passed, 84 sranges · r3 · g0, run again, with the change (v1) · passed, 121 sranges · r1 · g1, as it was (v1) · passed, 94 sranges · r2 · g1, as it was (v1) · passed, 82 sranges · r3 · g1, as it was (v1) · passed, 103 s
a run that passed its checkfailed▲the change won the task▼lost itbar: share of the arm's runs that passed▍a task the change movedshaded: the moved tasks, run againeach square opens its run

One ranges run failed and no eval named a change, so the harness explored a fixed, grader-blind instruction to test every stated edge case. It won one task, which a change that did nothing would also do about half the time. So ranges ran again, under both harnesses, and it held: 2/3 passed without the change, 3/3 with it. Then every run passed. That was 42 runs for $0.45.

On a real repository's history

--from-git turns a project's own commits into tasks. Each one starts from the parent with the commit's message, and the commit's tests, held out, grade the run. A commit is used only if its tests fail on the parent and pass on the commit. These are six commits from python-slugify. With only generic instructions, all three changes were reverted. With the project's own guide tried first:

the tasks the change moved, run again, both armsthe tasks the change moved, run again, both armsg0explored from the projectThis project keeps `AGENTS.md`. Read it before you change anything, and follow what it says about how changes are made, where they go and how they are tested. — it won 1 task(s) and lost 0explore:project_guideThis project keeps `AGENTS.md`. Read it before you change anything, and follow what it says about how changes are made, where they go and how they are tested. — it won 1 task(s) and lost 0kept · held on re-rung1explored from the projectBefore you finish, read the project's existing tests for the code you change (in `tests/`), add tests in the same style for exactly what the task asks, edge cases included, and run `python3 -m pytest -q`; fix what fails. — it won 1 task(s) and lost 1: not more wins than lossesexplore:project_testsBefore you finish, read the project's existing tests for the code you change (in `tests/`), add tests in the same style for exactly what the task asks, edge cases included, and run `python3 -m pytest -q`; fix what fails. — it won 1 task(s) and lost 1: not more wins than lossesrevertedg2exploredBefore you finish, list every rule and edge case the task and the code's docstrings state, write a quick test for each one, run them, and fix what fails. — it won 2 task(s) and lost 1explore:edge_casesBefore you finish, list every rule and edge case the task and the code's docstrings state, write a quick test for each one, run them, and fix what fails. — it won 2 task(s) and lost 1kept · held on re-runas it wasas it wasv06 of 12 runs passed6/12with the changethe changev17 of 12 runs passed7/12run again, as it wasre-runv00 of 2 runs passed0/2run again, with the changere-runv11 of 2 runs passed1/2as it was (the runs of the kept change, carried)keptv17 of 12 runs passed7/12with the changethe changev27 of 12 runs passed7/12as it wasas it wasv19 of 12 runs passed9/12with the changethe changev210 of 12 runs passed10/12run again, as it wasre-runv13 of 6 runs passed3/6run again, with the changere-runv26 of 6 runs passed6/6the change moved this task2f23599b-fix-add-uppercase-char-to-le2f23599b-fix-add-uppercase-c…2f23599b-fix-add-uppercase-char-to-le · r1 · g0, as it was (v0) · failed, 14 s2f23599b-fix-add-uppercase-char-to-le · r2 · g0, as it was (v0) · passed, 14 s2f23599b-fix-add-uppercase-char-to-le · r1 · g0, with the change (v1) · passed, 21 s2f23599b-fix-add-uppercase-char-to-le · r2 · g0, with the change (v1) · failed, 25 s2f23599b-fix-add-uppercase-char-to-le · r1 · g1, as it was (v1) · passed, 21 s2f23599b-fix-add-uppercase-char-to-le · r2 · g1, as it was (v1) · failed, 25 s2f23599b-fix-add-uppercase-char-to-le · r1 · g1, with the change (v2) · failed, 45 s2f23599b-fix-add-uppercase-char-to-le · r2 · g1, with the change (v2) · failed, 30 s2f23599b-fix-add-uppercase-char-to-le: passed less often with the change2f23599b-fix-add-uppercase-char-to-le · r1 · g2, as it was (v1) · failed, 25 s2f23599b-fix-add-uppercase-char-to-le · r2 · g2, as it was (v1) · passed, 39 s2f23599b-fix-add-uppercase-char-to-le · r1 · g2, with the change (v2) · passed, 61 s2f23599b-fix-add-uppercase-char-to-le · r2 · g2, with the change (v2) · passed, 53 s2f23599b-fix-add-uppercase-char-to-le: passed more often with the change2f23599b-fix-add-uppercase-char-to-le · r1 · g2, run again, as it was (v1) · passed, 21 s2f23599b-fix-add-uppercase-char-to-le · r2 · g2, run again, as it was (v1) · failed, 29 s2f23599b-fix-add-uppercase-char-to-le · r1 · g2, run again, with the change (v2) · passed, 68 s2f23599b-fix-add-uppercase-char-to-le · r2 · g2, run again, with the change (v2) · passed, 54 s548b14fa-preserve-fitting-post-replac548b14fa-preserve-fitting-po…548b14fa-preserve-fitting-post-replac · r1 · g0, as it was (v0) · passed, 26 s548b14fa-preserve-fitting-post-replac · r2 · g0, as it was (v0) · passed, 31 s548b14fa-preserve-fitting-post-replac · r1 · g0, with the change (v1) · passed, 46 s548b14fa-preserve-fitting-post-replac · r2 · g0, with the change (v1) · passed, 34 s548b14fa-preserve-fitting-post-replac · r1 · g1, as it was (v1) · passed, 46 s548b14fa-preserve-fitting-post-replac · r2 · g1, as it was (v1) · passed, 34 s548b14fa-preserve-fitting-post-replac · r1 · g1, with the change (v2) · passed, 72 s548b14fa-preserve-fitting-post-replac · r2 · g1, with the change (v2) · passed, 54 s548b14fa-preserve-fitting-post-replac · r1 · g2, as it was (v1) · passed, 40 s548b14fa-preserve-fitting-post-replac · r2 · g2, as it was (v1) · passed, 37 s548b14fa-preserve-fitting-post-replac · r1 · g2, with the change (v2) · passed, 115 s548b14fa-preserve-fitting-post-replac · r2 · g2, with the change (v2) · passed, 62 sthe change moved this task8f9a550a-avoid-trailing-post-replacem8f9a550a-avoid-trailing-post…8f9a550a-avoid-trailing-post-replacem · r1 · g0, as it was (v0) · failed, 57 s8f9a550a-avoid-trailing-post-replacem · r2 · g0, as it was (v0) · passed, 28 s8f9a550a-avoid-trailing-post-replacem · r1 · g0, with the change (v1) · passed, 55 s8f9a550a-avoid-trailing-post-replacem · r2 · g0, with the change (v1) · failed, 48 s8f9a550a-avoid-trailing-post-replacem · r1 · g1, as it was (v1) · passed, 55 s8f9a550a-avoid-trailing-post-replacem · r2 · g1, as it was (v1) · failed, 48 s8f9a550a-avoid-trailing-post-replacem · r1 · g1, with the change (v2) · passed, 95 s8f9a550a-avoid-trailing-post-replacem · r2 · g1, with the change (v2) · failed, 70 s8f9a550a-avoid-trailing-post-replacem · r1 · g2, as it was (v1) · passed, 57 s8f9a550a-avoid-trailing-post-replacem · r2 · g2, as it was (v1) · passed, 62 s8f9a550a-avoid-trailing-post-replacem · r1 · g2, with the change (v2) · failed, 92 s8f9a550a-avoid-trailing-post-replacem · r2 · g2, with the change (v2) · passed, 120 s8f9a550a-avoid-trailing-post-replacem: passed less often with the change8f9a550a-avoid-trailing-post-replacem · r1 · g2, run again, as it was (v1) · passed, 64 s8f9a550a-avoid-trailing-post-replacem · r2 · g2, run again, as it was (v1) · passed, 54 s8f9a550a-avoid-trailing-post-replacem · r1 · g2, run again, with the change (v2) · passed, 112 s8f9a550a-avoid-trailing-post-replacem · r2 · g2, run again, with the change (v2) · passed, 107 sthe change moved this taskc9e9698c-preserve-modern-word-boundarc9e9698c-preserve-modern-wor…c9e9698c-preserve-modern-word-boundar · r1 · g0, as it was (v0) · failed, 60 sc9e9698c-preserve-modern-word-boundar · r2 · g0, as it was (v0) · failed, 40 sc9e9698c-preserve-modern-word-boundar · r1 · g0, with the change (v1) · failed, 73 sc9e9698c-preserve-modern-word-boundar · r2 · g0, with the change (v1) · failed, 58 sc9e9698c-preserve-modern-word-boundar · r1 · g1, as it was (v1) · failed, 73 sc9e9698c-preserve-modern-word-boundar · r2 · g1, as it was (v1) · failed, 58 sc9e9698c-preserve-modern-word-boundar · r1 · g1, with the change (v2) · passed, 88 sc9e9698c-preserve-modern-word-boundar · r2 · g1, with the change (v2) · failed, 98 sc9e9698c-preserve-modern-word-boundar: passed more often with the changec9e9698c-preserve-modern-word-boundar · r1 · g2, as it was (v1) · failed, 60 sc9e9698c-preserve-modern-word-boundar · r2 · g2, as it was (v1) · passed, 82 sc9e9698c-preserve-modern-word-boundar · r1 · g2, with the change (v2) · passed, 118 sc9e9698c-preserve-modern-word-boundar · r2 · g2, with the change (v2) · passed, 142 sc9e9698c-preserve-modern-word-boundar: passed more often with the changec9e9698c-preserve-modern-word-boundar · r1 · g2, run again, as it was (v1) · failed, 77 sc9e9698c-preserve-modern-word-boundar · r2 · g2, run again, as it was (v1) · failed, 76 sc9e9698c-preserve-modern-word-boundar · r1 · g2, run again, with the change (v2) · passed, 118 sc9e9698c-preserve-modern-word-boundar · r2 · g2, run again, with the change (v2) · passed, 112 sthe change moved this taskcf6cd3da-skip-the-transliteration-baccf6cd3da-skip-the-transliter…cf6cd3da-skip-the-transliteration-bac · r1 · g0, as it was (v0) · failed, 36 scf6cd3da-skip-the-transliteration-bac · r2 · g0, as it was (v0) · failed, 37 scf6cd3da-skip-the-transliteration-bac · r1 · g0, with the change (v1) · passed, 62 scf6cd3da-skip-the-transliteration-bac · r2 · g0, with the change (v1) · failed, 84 scf6cd3da-skip-the-transliteration-bac: passed more often with the changecf6cd3da-skip-the-transliteration-bac · r1 · g0, run again, as it was (v0) · failed, 40 scf6cd3da-skip-the-transliteration-bac · r2 · g0, run again, as it was (v0) · failed, 38 scf6cd3da-skip-the-transliteration-bac · r1 · g0, run again, with the change (v1) · passed, 96 scf6cd3da-skip-the-transliteration-bac · r2 · g0, run again, with the change (v1) · failed, 81 scf6cd3da-skip-the-transliteration-bac · r1 · g1, as it was (v1) · passed, 62 scf6cd3da-skip-the-transliteration-bac · r2 · g1, as it was (v1) · failed, 84 scf6cd3da-skip-the-transliteration-bac · r1 · g1, with the change (v2) · failed, 75 scf6cd3da-skip-the-transliteration-bac · r2 · g1, with the change (v2) · passed, 91 scf6cd3da-skip-the-transliteration-bac · r1 · g2, as it was (v1) · passed, 80 scf6cd3da-skip-the-transliteration-bac · r2 · g2, as it was (v1) · failed, 83 scf6cd3da-skip-the-transliteration-bac · r1 · g2, with the change (v2) · passed, 430 scf6cd3da-skip-the-transliteration-bac · r2 · g2, with the change (v2) · failed, 87 sdf37f92c-support-uppercase-hexadecimadf37f92c-support-uppercase-h…df37f92c-support-uppercase-hexadecima · r1 · g0, as it was (v0) · passed, 18 sdf37f92c-support-uppercase-hexadecima · r2 · g0, as it was (v0) · passed, 22 sdf37f92c-support-uppercase-hexadecima · r1 · g0, with the change (v1) · passed, 28 sdf37f92c-support-uppercase-hexadecima · r2 · g0, with the change (v1) · passed, 33 sdf37f92c-support-uppercase-hexadecima · r1 · g1, as it was (v1) · passed, 28 sdf37f92c-support-uppercase-hexadecima · r2 · g1, as it was (v1) · passed, 33 sdf37f92c-support-uppercase-hexadecima · r1 · g1, with the change (v2) · passed, 55 sdf37f92c-support-uppercase-hexadecima · r2 · g1, with the change (v2) · passed, 41 sdf37f92c-support-uppercase-hexadecima · r1 · g2, as it was (v1) · passed, 30 sdf37f92c-support-uppercase-hexadecima · r2 · g2, as it was (v1) · passed, 35 sdf37f92c-support-uppercase-hexadecima · r1 · g2, with the change (v2) · passed, 73 sdf37f92c-support-uppercase-hexadecima · r2 · g2, with the change (v2) · passed, 87 s
a run that passed its checkfailed▲the change won the task▼lost itbar: share of the arm's runs that passed▍a task the change movedshaded: the moved tasks, run againits traces are not published here

Reading the project's AGENTS.md was kept (6/14 → 8/14) and held on the re-run. Adding tests in the project's style was reverted. Then the edge-case instruction, on top of the guide, was kept (12/18 → 16/18) and held again, 6/6 on the fresh runs. Both are weak wins, and the page they come from says so.

Measured, not claimed

Tasks (Claude Code, haiku)RunsCostWhat the harness did
Six demo tasks, graders held out12$0.5110 passed. Both semver runs failed on what the code does, and no eval could tell them apart from a pass, so it changed nothing.
The hard six, exploring and confirming42$0.45Explored the edge-case instruction, confirmed it on a re-run, then every run passed.
python-slugify history, generic instructions only100$1.29All three changes reverted: two didn't hold on the re-run, and one broke a task.
python-slugify history, the project's own guide first76$1.24Kept "read AGENTS.md", then the edge-case instruction. Both held on fresh runs.

The analysis engine contains no network code. Only the harness talks to a model, and none of the analysis needs one. Read the code, or start with the README.