A Giant Coding Model on a 16 GB Graphics Card: Fast Enough to Chat With, Too Slow to Finish a Coding Benchmark
Niko1221's Strata engine is an open-source runtime that squeezes a very large mixture-of-experts model onto one consumer graphics card by keeping the hot parts in video memory and the rest in system memory and on disk. I ran its coding model on a 16 GB card with 32 GB of RAM. It decoded at 42-58 tokens per second, but my coding benchmark was stopped by a rule I wrote beforehand: at 12 finished cases it projected 11.9 hours against an 8-hour budget. There is no benchmark score, and I am not reporting one.
Bart Porcincula · Published
Finding
Niko1221's Strata engine (v0.2.8, Terminal 0.1.54) served its Coder model at the IQ1_M quantisation on an RTX 5060 Ti 16 GB with 31.6 GB RAM at 42 tokens per second cold and 56-58 warm (1.38 times), with 15.2 of 16.3 GB of video memory in use and the pagefile active. Aider Polyglot (225 cases, whole-file edits, one thread) was stopped by the pre-set usability cut after 12 finished cases: mean 171.6 s per case, projected 11.9 h against the 8 h budget. 4 of 12 passed at the last attempt and 2 of 12 on the first try; that sample is too small and too biased toward early finishers to count as a score, so the comparison with the 9B reference (27.6%, gate 22.1%) was not made.
Niko1221's Strata engine, Coder IQ1_M, on 16 GB VRAM / 32 GB RAM (Oct 2026)
Stage A speed and memory measurements and the stopped Stage B run. RTX 5060 Ti 16 GB, Ryzen 5 7500F, 31.6 GB RAM, NVMe SSD. Engine 0.1.54-windows-x86_64-cuda-cuda13, flags --spec 4 --mtp --resident-experts, 32K context, temperature 0, thinking off through a pass-through proxy. Aider Polyglot with whole-file edits, one thread, 225 cases planned. The run was killed by the pre-registered usability cut and has no pass rate.
| Measurement | Value | Note |
|---|---|---|
| Decode speed, cold | 42 tok/s | |
| Decode speed, warm | 56-58 tok/s | 1.38 times the cold figure |
| 8K-token prompt, first read vs cached | 12.7 s vs 4.1 s | |
| Video memory in use | 15.2 of 16.3 GB | free system RAM 0.4-2.2 GB, 1.9 GB of pagefile in use |
| Coding benchmark cases finished before the stop | 12 of 225 | |
| Mean time per case | 171.6 s | several cases took 300-400 s, mostly failures |
| Projected time for all 225 cases | 11.9 h | against an 8 h budget fixed in advance |
| Pass rate | not scored | 4 of 12 at the last attempt, 2 of 12 first try; not a comparable score |
The short version
I wanted to know whether a very large coding model, squeezed onto a consumer machine, could beat a small one that fits comfortably. The runtime is Niko1221's Strata engine, an open-source (MIT) engine that keeps the most-used parts of a huge mixture-of-experts model in the graphics card's memory and the rest in system RAM and on disk. My machine has a 16 GB RTX 5060 Ti and 31.6 GB of RAM, which is less RAM than the engine's own sizing assumes for its larger variants.
- It runs, and it is fast enough to talk to. The Coder model decoded at 42 tokens per second cold and 56-58 warm.
- It does not fit cleanly. Video memory sat at 15.2 of 16.3 GB, free RAM fell to 0.4 GB, and the pagefile was in use, so the "no paging" part of my hypothesis is refuted.
- The coding benchmark never finished. I had fixed in advance that the run stops if, after 10 finished cases, the projection exceeds 8 hours. At 12 cases it projected 11.9 hours.
- There is no score. Four of the 12 finished cases passed after retries, two on the first try. The cases that finish first are not a random sample, so I do not compare that to the small-model reference.
Takeaway: decode speed alone does not tell you whether a model is usable for a task. A coding benchmark is many rounds of whole-file edits and retries, and it punishes slow starts and long tails. A model that feels fine in a chat window can still be too slow for that job.
What ran
The engine was Terminal 0.1.54 (release v0.2.8), the Coder model at the IQ1_M quantisation, launched with speculative decoding on and the experts memory-mapped with a hot cache in video memory. Context was 32K, temperature 0. The model thinks by default, so a small pass-through proxy turned thinking off and changed nothing else, the same convention as the rest of this series. The benchmark was Aider Polyglot in its container, whole-file edit format, one thread, 225 cases. The reference was the earlier 9B result: 27.6% pass rate, with a pre-set gate at 22.1% (80% of it).
Speed and memory
Cold decode was 42 tokens per second and warm decode 56-58, a 1.38 times gain as the expert cache filled. An 8,000-token prompt took 12.7 seconds the first time and 4.1 seconds when cached. Those numbers clear the speed floor I set. The memory picture is tighter: the card was nearly full, system RAM almost exhausted, and the pagefile in use at only 8K of context.
Why the benchmark stopped
The mean was 171.6 seconds per finished case, with a long tail of cases at 300-400 seconds that mostly failed. A coding case is not one reply: it is a prompt carrying the whole file, a whole-file rewrite, a test run, and a retry if the tests fail. A slow tail multiplies through all of that. Projected over 225 cases it came to 11.9 hours, past the 8-hour limit, so the run was stopped by the rule rather than by me.
What I am not claiming
- I am not claiming this model is worse or better at code than the 9B. The comparison was never made.
- I did not rerun with a longer budget, because that would change the protocol after seeing the result.
- Two further hypotheses (a larger quantisation in mapped mode, and whether my own coding-agent loop works against this endpoint) were not run.
What it points to
The stop was informative in its own right. How long you are willing to wait is part of what you are measuring. A model that is too slow for interactive coding may still be useful for work nobody is waiting on, such as overnight batch jobs. That is the next study: the same fixed time window for both models, scored on correct items per hour and on accuracy.
Cite this research
Bart Porcincula. "A Giant Coding Model on a 16 GB Graphics Card: Fast Enough to Chat With, Too Slow to Finish a Coding Benchmark." Published 2026-10-30; updated 2026-10-30. https://porsync.com/research/giant-moe-on-16gb-gpu-32gb-ram-too-slow-for-coding-benchmark
Permanent article URL