LocalSWEAgent

Can a 3B Parameter LLM on a Laptop Solve Real GitHub Issues?

M3 Pro MacBook Pro · Ollama · Zero Cloud API Costs
6/6
Patches Apply
50%
Best Patch Similarity
~50s
Avg Time/Instance
3 strategies
Per File Attempted
$0.00
Total API Cost

Key Result

After aggressive optimization, the 3B model generates 100% apply-clean patches for 6 easiest SWE-bench Lite instances. One patch (django-10914) achieves 50% similarity to the gold fix - correctly outputting FILE_UPLOAD_PERMISSIONS = 0o644. For django-11179, the model finds the exact right file and function (deletion.py:277) but adds the wrong line. The bottleneck is no longer patch generation - it's code reasoning about root causes.

How It Works

1
Parse Issue
LLM analyzes the GitHub issue to understand the bug
→
2
Find Files
Grep + LLM-guided search for relevant source files
→
3
Read Code
Extract and contextualize the relevant code sections
→
4
Generate Fix
Search/Replace blocks converted to unified diff patches
→
5
Validate
git apply --check verifies patch syntax and applicability

SWE-bench Lite: Where We Stand

Resolve rate comparison. SWE-bench Lite = 300 curated instances from 11 Python repos. Our agent generates patches but hasn't resolved any instances yet (0% resolve rate).

Claude 3.5 + Agentless
33.0%
$0.34/inst
GPT-4o + SWE-agent
23.3%
$1.50/inst
GPT-4 + SWE-agent
18.0%
$2.00/inst
Claude 3 Haiku RAG
10.7%
$0.05/inst
RAG baseline
2.7%
$0.10/inst
LocalSWEAgent 3B (Ours)
0% (patches generated, wrong fix)
$0.00/inst

What the 3B Model Gets Right vs Wrong

Gets Right

Identifies the correct file to modify in 85%+ of cases. Generates syntactically valid Python code. Produces structurally sound patches with proper context lines. Understands the general area of the bug.

Gets Wrong

Misidentifies root cause - applies superficial fixes instead of addressing the real bug. Adds unnecessary code rather than modifying the right lines. Cannot reason about complex code interactions across modules.

Cost Analysis: Why This Matters

Cloud Agent (SWE-agent + GPT-4)

Per instance~$2.00
300 instances~$600
Data leaves machineYes
Internet requiredYes

LocalSWEAgent (Ours)

Per instance$0.00
300 instances$0.00
Data leaves machineNo
Internet requiredOnly for git clone

Key Techniques

Search/Replace Blocks

Instead of asking a 3B model to produce perfect unified diffs (which they can't), we ask for structured SEARCH/REPLACE blocks and programmatically build the patch. This compensates for small model limitations.

Smart File Discovery

Two-stage retrieval: regex extraction of file paths and identifiers from the issue, followed by grep-based search. LLM narrows down candidates only when needed.

Minimal Context Window

Unlike cloud agents that stuff 100k+ tokens, we work within 4k-8k context. Forces surgical precision and reduces noise for the small model.

Zero-Shot Prompting

No fine-tuning, no few-shot examples consuming context. The system prompt is under 200 tokens. All intelligence comes from the base model + good tooling.

Hardware

Apple M3 Pro
36GB Unified Memory
Ollama v0.11.4
Qwen 2.5 3B (1.9GB)
macOS Sequoia

Instance Results

Instance Status Patch Size Time Diff
Loading results...

Limitations & Future Work

Current Limitations

3B models have limited code understanding. Multi-file patches are rare. Complex logic changes beyond simple fixes are out of reach. No test execution feedback loop.

Roadmap

Add iterative refinement with test feedback. Try 7B-14B models (llava, deepseek-coder). Implement RAG over repo history. Add multi-turn agent loop with tool use.