ResearchPractical AIProjectsOpen SourceAbout
Research/The Same Coding Model Improved When Its Harness Changed
Coding AgentsLocal AIHarnessesEvaluationOriginal Research

The Same Coding Model Improved When Its Harness Changed

A separate 288-attempt follow-up put Aider ahead on three models and both Patchsmith versions near ceiling on Qwen3.6. Harness changes helped some models, without establishing a general winner.

Bart Porcincula · Published

Finding

In the separate n09-n16 cohort, Patchsmith v5b/v6/Aider passed 13/19/23 of 24 for gpt-oss, 14/15/17 for Qwen3-Coder, 24/24/23 for Qwen3.6 and 18/17/23 for Edge0. Edge0 v5b is not blind; cohorts are never pooled. Two simultaneous v6 changes and unequal budgets prevent a causal harness ranking.

Patchsmith and Aider with the same local models (November 2026)

Independent-test outcomes on ten tuning fixtures repeated five times, four historical fixes repeated three times, and eight later fixtures repeated three times per arm. Later fixtures were informed by both arms' failures. Patchsmith v5b and the gpt-oss recovery were changed after inspecting results. Cohorts remain separate; quants and serving flags differ across models. The separate n09-n16 follow-up adds 288 saved attempts; curated rows and offline analysis are public.

MeasurementValueNote
Separate n09-n16: v5b / v6 / Aider, each out of 24gpt-oss 13/19/23; Coder 14/15/17; Qwen3.6 24/24/23; Edge0 18/17/23288 attempts; Edge0 v5b not blind; cohorts never pooled
Edge0 tuning: v1 / v2 / v3 / Aider / v4 / v521 / 22 / 18 / 43 / 49 / 48 of 50tuning set, not generalisation; raw rows correct v3 from 19 to 18
Edge0 replay, same arm order6 / 5 / 4 / 4 / 5 / 8 of 12four commits, too small to rank arms
Edge0 later fixtures: v5b / Aider20/24 / 20/24v5b post-hoc, not blind
Qwen3-Coder: Patchsmith / Aider19/24 / 14/24Q3_K_M, three attempts per fixture
gpt-oss: Patchsmith pre / post fix; Aider14/24 / 15/24; 22/24MXFP4; post-hoc fix, Aider not rerun; 8 then 6 HTTP 500 failures
Qwen3.6: Patchsmith / Aider24/24 / 23/24UD-IQ3_XXS; near ceiling; one attempt is noise

A weak coding result can have more than one cause

When a local coding model fails to repair a small program, it is tempting to blame the model. But the software around it decides which files it sees, how it edits them, when it runs tests and when it must stop. That software is the harness.

Edge0 initially struggled in Patchsmith, my tool-call loop. Aider, an established coding harness, did much better on the same tuning tasks. After changes to Patchsmith, the gap narrowed. On eight later fixtures, Patchsmith v5b passed 20/24, matching Aider's 20/24. The Patchsmith result was not blind: I changed it after seeing its first failures on those fixtures.

Repeating the comparison on three other models made the conclusion more useful. Patchsmith was not consistently the weaker harness. It did encounter a substantial serving and output-format problem with gpt-oss.

What counted as a completed repair

Each attempt had to pass independent tests. A model saying it was finished did not count. Three workloads remain separate: ten synthetic maintenance tasks repeated five times, four historical fixes repeated three times, and eight later fixtures repeated three times. I do not pool these totals or combine them with frontier results.

Patchsmith asks the model to call tools to read, edit and test files. Aider v0.86.2 used whole-file edits, source supplied up front, tests marked read-only, automatic testing and no repository map. Patchsmith allowed 30 turns and 30,000 tokens of live context; Aider's fixture runner had a 600-second process timeout. These are whole harnesses with different prompts, stopping rules and editing interfaces, not a controlled test of format alone.

Patchsmith explicitly requested temperature zero and thinking off. Aider's wrapper described temperature zero but did not explicitly set it in the command, so sampling parity is not established.

Models and serving settings

The desktop had an RTX 5060 Ti 16 GB and about 32 GB of system RAM. Quantization compresses weights and can affect quality. Every comparison across models is confounded by these different formats and serving settings.

ModelWeight formatServing settings
Edge0-120Bq3kedge0-infer, context 32,768, port 8000
Qwen3-Coder-30B-A3B-Instructunsloth Q3_K_M, 14.71 GBVulkan llama-server, context 32,768, q8_0 K/V cache, 6 CPU MoE layers
gpt-oss-20bggml-org MXFP4, 12.11 GBVulkan llama-server, context 32,768; no extra KV/MoE launch flags; automatic fitting, four slots and 8,192 MiB prompt RAM cache in logs
Qwen3.6-35B-A3BUD-IQ3_XXS, 13.21 GBVulkan llama-server, context 32,768, q8_0 K/V cache, 8 CPU MoE layers; low-bit quality risk

The non-Edge0 launcher used port 8001, flash attention on, Jinja templates and 99 requested GPU layers. It was the native llama-server executable distributed by Docker. Effective gpt-oss KV dtype and the exact server build were not pinned in the saved launch record. This is not a ranking of the original models.

First, the harness learned the tuning tasks

HarnessTuning passes /50Historical-fix replay /12
Patchsmith v1216
Patchsmith v2225
Patchsmith v3184
Aider434
Patchsmith v4495
Patchsmith v5488

The v3 result is corrected here: the handoff said 19/50, but its saved independent-test rows and generated table give 18/50.

Early changes gave Patchsmith a file listing, more room before repeated-call stops, warnings about excessive reading and detection of edits that left a file unchanged. Version 4 supplied source, tests and initial failing test output in the prompt. Version 5 made that preload smaller and more selective, shortened old tool output as context filled, and added a last redirect before stopping a repeated call.

Those descriptions come from the code. They do not isolate which change caused improvement. The ten tasks were the tuning set. The 48/50 result is not evidence of generalisation. The four replay commits are too few to rank the arms.

The runs used uncommitted harness changes on top of the live-context-budget implementation. A source commit was created later during this write-up; it does not retrospectively pin every run-time snapshot.

Then came the later fixtures

The eight later tasks covered token buckets, semantic versions, word wrapping, decimal ledgers, business days, LRU caches, template expansion and topological sorting. They were written after looking at both arms' failures, so their design was not independent of tuning.

Patchsmith v5 first passed 17/24. After changes, v5b passed 20/24, not blind, alongside Aider's 20/24. Equal totals hid different weaknesses: both struggled with the ledger; Patchsmith also failed business-day and cache attempts, while Aider timed out twice on topological sorting.

ModelPatchsmith passes /24Aider passes /24
Edge020, v5b, not blind20
Qwen3-Coder1914
gpt-oss before fix1422
gpt-oss after post-hoc fix1522, original control not rerun
Qwen3.62423

Only three attempts were made per fixture. Differences of one attempt are noise, and Qwen3.6 is near the ceiling of this small set. Nothing here establishes that Patchsmith beats Aider.

The gpt-oss failures were different

Qwen3-Coder's failed Patchsmith runs spent 16-30 turns on semantic versions or topological sorting before blocking or hitting the turn limit. Server logs showed no parser exceptions, and saved traces described unsuccessful algorithms. Aider also failed those tasks. That is consistent with task-solving limits in this setup, rather than an observed transport fault. Patchsmith did solve all three template attempts where Aider solved none; that difference warrants investigation, not a general superiority claim.

gpt-oss had another problem. Before the fix, eight Patchsmith attempts ended in HTTP 500 errors from the output parser. Three involved a finish call sent on the wrong channel; five involved malformed JSON arguments. A narrow recovery change handled the finish-call case after those failures were inspected.

The new run still had six HTTP 500 failures: five code-writing arguments cut off inside a JSON string and one malformed channel/tool message. The result moved from 14/24 to 15/24, against Aider's original 22/24. That one-run improvement is not a reliable estimate of the fix's benefit. Other models were not rerun after the recovery change.

Aider's whole-file format avoids putting edited code inside a JSON argument. The pattern is consistent with a format cost on gpt-oss, but whole-harness differences prevent saying format alone explains the gap. A gpt-oss-only whole-file fallback is an untested follow-up; the data do not warrant a general fallback.

A separate follow-up changed the comparison

The n09-n16 follow-up used eight new synthetic tasks, three repeats, three arms and four models: 288 saved attempts, with no missing or duplicate keys. These tasks were held out from the observed earlier tuning, but the task author was not independent and this was not a preregistered evaluation. Edge0 v5b is not blind. Cohorts are never pooled with n01-n08, tuning, replay or frontier results.

ModelPatchsmith v5b /24Patchsmith v6 /24Aider /24
gpt-oss131923
Qwen3-Coder141517
Qwen3.6242423
Edge0181723

Version 6 raised the completion cap from 2,048 to 4,096 tokens and withdrew read-only tools after five reads until an edit or test. Because both changed together, these results cannot isolate their effects. Patchsmith retained a 30-turn limit, 30k live-context budget and 420-second request timeout; Aider had a 600-second process timeout. Sampling parity remains unpinned, and cross-model quantization and backend differences remain confounds.

gpt-oss gained six passes under v6, while remaining below Aider in this cohort. Qwen3-Coder's earlier Patchsmith advantage did not replicate. Edge0 showed no v6 gain; Qwen3.6 remained near ceiling. Three repeats on eight tasks are too narrow for a general harness ranking.

The successful Edge0 batch followed a reported reboot and retained context 32,768. Two failed 12:43 startup launches and an earlier morning failure are operational exclusions, not failed task attempts. Surviving logs do not establish that RAM caused those startup failures or independently verify the reboot.

What would settle the remaining question

The initial Edge0 deficit did not persist after harness tuning. Harness reliability depends on the model and serving setup, and changing the surrounding software can change what the same model completes.

The next comparison needs frozen source hashes, pinned serving and sampling settings, genuinely unseen tasks, counterbalanced arm order and more distinct repairs. A causal format claim requires an isolated format change. A broken OpenCode runner and smoke cohorts are excluded; they provide no extra comparison evidence.

This does not resolve the separate local routing study. Plugin token accounting, host and shadow integration, health calibration, natural recovery and held-out routing evidence remain open.

Evidence availability

The first-set report and separate n09-n16 report are public. The follow-up includes curated trial rows, configuration, portable offline analysis and original-source hashes. The release notes disclose scope and transformations.

Fixture source, reference solutions, private implementation, generated patches, prompts, transcripts and raw test output are withheld. Readers can recompute counts from the selected outcome rows, but cannot independently rescore generated code from this package.

Cite this research

Bart Porcincula. "The Same Coding Model Improved When Its Harness Changed." Published 2026-11-19; updated 2026-11-19. https://porsync.com/research/same-coding-model-different-harness

Permanent article URL