Can a Small Local AI Make the Code Change? 360 Attempts Across Six Jobs
I gave two local models and Claude the same four small bugs and two small features, repeating each job 20 times per model. The 9B and Claude passed every attempt. The larger local Coder passed 119 of 120 directly. On the remaining attempt, the runner verified that it had undone the local edits, then Claude solved the job from a fresh copy.
Bart Porcincula · Published · Updated
Finding
On six fixed synthetic Python tasks repeated 20 times per setup, Qwen3.5-9B through Ollama and Claude Sonnet 5 each passed 120 of 120 attempts directly. Coder IQ1_M on Niko1221's Strata engine passed 119 of 120 directly and 120 of 120 with one measured Claude fallback. Throughput including fallback was 193.11, 185.72 and 104.38 accepted attempts per hour for the 9B, Claude and Coder respectively.
Six synthetic Python tasks repeated across three coding-agent setups (Nov 2026)
Four seeded bugs and two small features. Each of three setups repeated each task 20 times in fresh worktrees. The same allowed tool operations and independent full-suite acceptance check applied across setups. Elapsed time includes acceptance and measured fallback where used, and excludes repository setup and server startup.
| Measurement | Value | Note |
|---|---|---|
| Qwen3.5-9B, direct successes | 120/120 | 17.14 s median, 193.11 successes/h |
| Coder IQ1_M, direct / with fallback | 119/120 / 120/120 | 32.71 s median, 104.38 successes/h including fallback |
| Claude Sonnet 5, direct successes | 120/120 | 18.29 s median, 185.72 successes/h |
| One failed local run and fallback | 240.55 s + 25.47 s | 60-tool-call budget exhausted, rollback verified, Claude passed from a fresh seed |
The short version
I wanted to see whether a small local model could take a written task, edit the code and hand back a patch that passes the project's tests. I used four small Python bugs and two features, including a cross-file ranking bug and a request to limit API results. Each of the three setups started each job from a fresh copy, 20 times per job.
| Setup | Passed directly | Passed with fallback | Median time | Accepted per hour |
|---|---|---|---|---|
| Qwen3.5-9B through Ollama | 120/120 | 120/120 | 17.14 s | 193.11 |
| Coder IQ1_M on Niko1221's Strata engine | 119/120 | 120/120 | 32.71 s | 104.38 |
| Claude Sonnet 5, low effort | 120/120 | 120/120 | 18.29 s | 185.72 |
The model could edit only named source files through its tools. The runner accepted a patch after independently rerunning the full test suite and checking that protected files were unchanged. That matters when a model says it is done before the tests agree.
The one time the fallback ran
On one repeat of the ranking bug, the larger local Coder used all 60 tool calls in 240.55 seconds. The runner restored the starting files and verified the restoration. Claude then solved the job from the original fixture in 25.47 seconds. Both times count in the Coder setup's throughput. A typical attempt was much shorter, which is why I report throughput alongside the median.
What the repeats tell me
This was a reliability check on six fixed toy jobs. Repeats show how often the same job goes wrong under the same setup. A wider set of tasks would be needed to separate the two setups that passed every time. The arms ran one after another, prompt caches stayed warm between tasks, and Claude used its provider's sampling and context settings. The small timing difference between the 9B and Claude is specific to this run.
Does this show that local AI can maintain a business application?
It shows repeated success on six fixed synthetic Python jobs with independent tests and restricted edits. It does not establish success on an unfamiliar production application. The useful workflow demonstrated here is a bounded change, an independent acceptance check, and a verified restoration before fallback when the local attempt fails.
Cite this research
Bart Porcincula. "Can a Small Local AI Make the Code Change? 360 Attempts Across Six Jobs." Published 2026-11-07; updated 2026-11-08. https://porsync.com/research/local-coding-agent-six-tasks-360-attempts
Permanent article URL