ResearchPractical AIProjectsOpen SourceAbout
Research/Seven Small Coding AIs, Four Benchmarks, One 16 GB Card: None Cleared the Bar
Local AICoding ModelsBenchmarksPre-RegisteredNegative ResultOriginal Research

Seven Small Coding AIs, Four Benchmarks, One 16 GB Card: None Cleared the Bar

My first coding test was too easy to rank anything, so I ran the same models up a ladder of public benchmarks, hardest last, with the pass marks and a time limit written down first. The best model solved 9 of 50 real open-source bug reports. Five of seven were stopped before the end, not for being wrong but for being too slow to wait for. The result is a floor, and the reasons each model stopped matter more than the scores.

Bart Porcincula · Published

Finding

On one 16 GB RTX 5060 Ti, among seven local coding models that all passed a tool-calling gate (valid-call rate 0.960 to 1.000 on 400 BFCL items), none cleared a pre-registered ladder: the 9B reference was the only model to finish all four rungs inside an 8-hour generation budget (LiveCodeBench 0.406; Aider Polyglot 27.6%), and it resolved 9 of 50 SWE-bench Lite instances through OpenCode (18.0%, below its own 22.1% gate; the 95% interval at n=50 is roughly 10% to 31%); the only other model to finish rung 3, Qwen3-Coder-30B, scored 18.2% there, under the gate; five models were stopped on the time rule and carry no accuracy score at the rung where they stopped; one run (Gemma 4 12B, time rule waived) resolved 0 of 50. Single runs, one quantisation per model, public benchmarks with unknown training exposure.

Benchmark ladder for small coding models on a 16 GB graphics card (Oct 2026)

Seven local coding models (the survivors of the earlier ten-task study) served with llama.cpp b11753 on an RTX 5060 Ti with 16 GB of memory, 32K context, run up four rungs of increasing difficulty: tool-call validity (BFCL simple, 400 items, a gate), LiveCodeBench (175 problems dated 2025-01-04 to 2025-04-06, single attempt), Aider Polyglot (225 Exercism exercises in six languages, two attempts, a gate for the next rung at 80% of the 9B reference) and SWE-bench Lite (a fixed sample of 50 of 300 instances, driven by OpenCode, 30 minutes per instance). Pass marks, the sample and an 8-hour generation budget per rung were fixed before the models ran, apart from one disclosed change (the budget was tightened from 24 to 8 hours after the first rung-2 runs had started). Single runs except Nemotron Bolt on rung 2 (three).

MeasurementValueNote
Rung 1: models passing the 95% valid-call gate (BFCL simple, 400)7 of 7range 0.960 (gpt-oss-20b) to 1.000 (Gemma 4 12B)
Rung 2: LiveCodeBench pass@1, best to worst finished0.583 / 0.549 / 0.406 / 0.337 / 0.312 / 0.286Gemma 4 12B / gpt-oss-20b / Qwen3.5-9B / Qwen3-Coder-30B / Nemotron Bolt (mean of 3) / Mellum 2; gaps under about 7 points are noise
Rung 2: Gemma 4 12B vs gpt-oss-20b0.583 vs 0.549within noise; 3.6 h vs 1.6 h of generation
Rung 2: Nemotron Bolt, three identical runs0.291 to 0.326one run has a standard error of about 3.5 points at 175 problems
Rung 3: Aider Polyglot, Qwen3.5-9B reference27.6%6.9 h of generation, 105 s per case; sets the next gate at 22.1%
Rung 3: Aider Polyglot, Qwen3-Coder-30B18.2% (41 of 225)5.6 h; under the 22.1% gate
Models stopped on the 8-hour rule5 of 7Devstral 24B (rung 2, about 12 h projected); at rung 3 Mellum 2 (40.5 h), Nemotron Bolt (14.4 h), gpt-oss-20b (10.5 h) and Gemma 4 12B (8.3 h): no accuracy score at the rung they stopped at
Rung 4: SWE-bench Lite, Qwen3.5-9B through OpenCode9 of 50 (18.0%)below the 22.1% gate; many instances produced no usable patch
Rung 4: Gemma 4 12B (time rule waived, gate not applied)0 of 5029 empty patches; 21 non-empty and none resolved; not a ladder pass
Rung 5 (code-review benchmark)not runno model cleared rung 4, so the rules ended the ladder; one 10-PR smoke test on one model scored F2 0.15 and is a harness check only

The short version

My first test of small coding models was too easy: six of eight models fixed all ten jobs, so it could not say which was better. The fix is a harder test that has an answer. So I took the seven models that had survived, and ran them up a ladder of public benchmarks, easiest rung first. The pass marks, the sample sizes and a time limit were written down before any model ran. A model that failed a rung stopped there.

  • Nobody cleared it. The best model, the 9B reference, solved 9 of 50 real bug reports from open-source projects (18%), below the bar I had set for it.
  • Most models never got to the end, and not because they were wrong. Five were stopped by a time rule: if the projected generation time for a rung was over 8 hours, the run was stopped. Those models have no score at that rung, which is different from a low one.
  • The easy rungs did separate them, a little. On a live coding-problems set, Gemma 4 12B (0.583) and gpt-oss-20b (0.549) led, with the 9B reference at 0.406. But the models that led on the single-shot rung did not get to show a result on the harder ones.

Takeaway: if you are picking a local coding model for a card like this one, the question is not just "how good is it" but "how long will I wait for it". On this setup, the practical answer today is a weak assistant that needs checking, and the honest summary of the harder benchmarks is a floor, not a ranking.

The ladder

RungBenchmarkWhat it asksHow it was used
1BFCL simple (400 items)Can the model write a tool call the assistant can run?Gate: at least 95% valid
2LiveCodeBench (175 problems)Write a solution to a dated programming problem, one attemptRank only
3Aider Polyglot (225 exercises, 6 languages)Edit code until the tests pass, two attemptsRank; gate for rung 4 at 80% of the 9B reference
4SWE-bench Lite (50 of 300, fixed sample)Fix a real bug report in a real open-source repo, through the OpenCode assistantRank
5Code-review benchmarkFind defects in a pull requestNever reached

The reference for every gate is Qwen3.5-9B on the same rung, because absolute pass marks on a hard rung can sit above everything. Every model ran on the same llama.cpp build, the same context size, with nothing from a hosted model or paid API.

What happened, rung by rung

Rung 1: everyone passed. All seven produced a valid tool call at least 96% of the time (0.960 to 1.000). The model that scored zero in the first study for writing its tool calls as plain text, Qwen2.5-Coder 14B, stayed cut and was not retried.

Rung 2: speed and score both vary.

ModelPass@1Generation timeNotes
Gemma 4 12B0.5833.6 h24 of 80 hard problems
gpt-oss-20b0.5491.6 hwithin noise of Gemma, in under half the time
Qwen3.5-9B (reference)0.4063.2 h
Qwen3-Coder-30B0.3371.5 h
Nemotron Bolt (3 runs)0.291 / 0.326 / 0.3201.5 to 1.8 h
Mellum 20.2866.4 habout 4 tokens a second
Devstral 24Bnot scoredstopped at 140 of 175about 12 h projected

I repeated one model three times to measure noise (Nemotron Bolt: 0.291 to 0.326). One run has a standard error of about 3.5 points at 175 problems, so gaps under about 7 points mean nothing, and Gemma vs gpt-oss is inside that.

Rung 3: only two models finished. The 9B reference scored 27.6% (6.9 hours, 105 seconds per case). That set the gate for rung 4 at 22.1%. Qwen3-Coder-30B finished inside the time limit at 18.2%, which is under the gate. gpt-oss-20b (10.5 h projected), Nemotron Bolt (14.4 h), Mellum 2 (40.5 h) and Gemma 4 12B (8.3 h, 0.3 hours over) were stopped on the time rule from their first ten cases. Devstral had already stopped at rung 2.

Rung 4: a weak assistant. The 9B reference, driving OpenCode on 50 fixed bug reports, resolved 9 (18.0%), under its own gate. Many instances ended with no usable patch, and one patch was an unrelated 188 KB rewrite, so the result mixes the model's limits with the assistant loop's. I also ran Gemma 4 12B on this rung as a labelled exception, because it missed the time limit by only 0.3 hours. It resolved 0 of 50, with 29 empty patches and 21 patches that fixed nothing, several touching only stray scripts. It is reported in its own row because it never faced the rung 3 gate and so did not pass the ladder.

What "stopped: usability" means

The time rule was added after the first rung-2 runs had started, because a developer will not wait a day for an answer. I tightened it from 24 hours to 8, after seeing Devstral's speed and before seeing Mellum 2's, and I say so. A model stopped on it carries a reason and its speed, and no accuracy score. It says nothing about how well that model edits code. The projection comes from only ten cases, so a model near the line, such as Gemma at 8.3 hours, could land on either side with a faster server setup.

What went wrong along the way

  • Two fake results, caught. After a hackathon session left Docker Desktop off, the Aider driver failed, exited with success, and wrote a summary for an old partial run, twice, so one model was silently skipped. Both junk results were deleted and the model re-run from scratch. The driver still does not check its own exit status. I left it alone because it is hash-logged, and the lesson is that a summary with a tiny generation time should never be trusted.
  • Windows needed three workarounds, all disclosed: SWE-bench's harness wrote patch files with Windows line endings so patches would not apply, the Aider container build failed on a blocked package source, and git checkouts converted line endings and broke script headers.
  • The shared GPU nearly spoiled a timing. An early timing was measured with Ollama still holding 9.6 GB, so every measured run starts from a check that the card is under 1 GB used.

Limits

  • Single runs. Only Bolt on rung 2 was repeated. Rung 3 is roughly +/-3 points at 225 exercises, and rung 4's interval at n=50 is roughly 10% to 31%.
  • One quantisation per model, one assistant (OpenCode), one edit format on Aider. A different setup could move any of these scores.
  • LiveCodeBench problems are later than most training cutoffs, not provably unseen. The Aider and SWE-bench sets are public.
  • My own generation script for rung 2 (prompt and sampling are mine), scored with the benchmark's own evaluator. The 8,192-token output cap truncated many answers, so the 9B's and gpt-oss's scores are lower bounds.
  • The BFCL checker is mine, calibrated against the upstream one (359 vs 363 of 400). It is a validity filter, not a published score.
  • The host had no project dependencies installed, so models on rung 4 could mostly not run a project's tests, which probably hurt every score there.

The finding, stated for citation

On one 16 GB graphics card, seven small local coding models that all passed a tool-calling gate did not clear a four-rung benchmark ladder: the 9B reference finished every rung inside an 8-hour generation budget and resolved 9 of 50 SWE-bench Lite instances through OpenCode (18.0%, below its own 22.1% gate), Qwen3-Coder-30B scored 18.2% on Aider Polyglot, under the same gate, and five models were stopped on the time rule without an accuracy score at the rung where they stopped. A wait-time limit ended more model runs than accuracy did.

Cite this research

Bart Porcincula. "Seven Small Coding AIs, Four Benchmarks, One 16 GB Card: None Cleared the Bar." Published 2026-10-28; updated 2026-10-28. https://porsync.com/research/no-small-coding-model-cleared-the-bar-benchmark-ladder-16gb

Permanent article URL