Tuning llama.cpp Cut a Sixfold Speed Gap to 2.7 on My 16 GB Card
My first llama.cpp setup kept all of the model's experts on the CPU. Moving some onto the GPU more than doubled its output. Niko1221's Strata engine still got 2.71 times as many incidents right per hour as the tuned setup. I also turned off its multi-token prediction and tested a small draft model in stock llama.cpp to see how much each accounted for.
Bart Porcincula · Published
Finding
On 762 matched synthetic incidents with Coder IQ1_M, tuning stock llama.cpp increased correct throughput from 208 to 472 per hour (2.27×). Niko1221's Strata engine reached 1,017 with multi-token prediction off (2.15× tuned stock) and 1,278 with it on (2.71× tuned stock, 95% paired interval 2.66 to 2.76). Adding a 0.8B draft model to tuned stock reached 528 per hour (1.12×).
Expert placement and multi-token prediction on 16 GB (Nov 2026)
Coder IQ1_M, one request at a time, temperature zero, first 762 common synthetic incident-triage items. The tuned stock configuration was selected on a 40-item split before scoring. Paired intervals use 10,000 bootstrap draws across incidents.
| Measurement | Value | Note |
|---|---|---|
| Original stock, all experts on CPU | 208 correct/h | |
| Tuned stock | 472 correct/h | 2.27× original stock |
| Niko1221's Strata engine, prediction off / on | 1,017 / 1,278 correct/h | |
| Niko1221's Strata engine with prediction vs tuned stock | 2.71× | 95% paired interval 2.66 to 2.76 |
| Tuned stock with 0.8B draft | 528 correct/h | 1.12× tuned stock without a draft |
The short version
In my first incident-triage comparison, Niko1221's Strata engine got six times as many answers right per hour as stock llama.cpp. That was a comparison of two setups I could run. My stock configuration kept all the experts, the parts of the model that specialise in different inputs, on the CPU. I wanted to see what happened when I gave it a better shot and split those experts between CPU and GPU.
| Setup | Correct incidents per hour | Accuracy |
|---|---|---|
| Stock llama.cpp, all experts on CPU | 208 | 81.9% |
| Tuned stock llama.cpp | 472 | 81.8% |
| Tuned stock with a 0.8B draft model | 528 | 82.3% |
| Niko1221's Strata engine, multi-token prediction off | 1,017 | 82.2% |
| Niko1221's Strata engine, multi-token prediction on | 1,278 | 82.7% |
Where the time went
Tuning stock was worth 2.27 times as many correct answers per hour. Niko1221's Strata engine was still 2.15 times ahead with multi-token prediction off. That points to how the runtime handles experts across GPU memory, system memory and storage, though this comparison bundles other runtime differences with expert placement.
Turning on multi-token prediction, which proposes several tokens for the main model to check, lifted throughput another 26% in Niko1221's Strata engine. A separate 0.8B model drafting tokens for tuned stock lifted it by 12%. These are two different ways to draft ahead, each measured against its own setup.
I also replayed an offline rule that sent easier incident types to a 9B model and the rest to the Coder. It gained 6% in correct answers per hour over the fastest Coder run while losing 4.4 points of accuracy, before counting any cost of switching models.
Limits of the comparison
I ran each setup once, and some rows came from earlier runs. The intervals reflect variation across items, and the accuracy intervals still allow gaps of a point or more. Nearby stock settings ran slightly faster on the 40-item tuning split, so a better full-run stock baseline could shrink the 2.71 figure. The measurements are for one low-bit model file and one card. Credit to Niko1221 for the engine, the llama.cpp contributors, and the Qwen model authors.
Cite this research
Bart Porcincula. "Tuning llama.cpp Cut a Sixfold Speed Gap to 2.7 on My 16 GB Card." Published 2026-11-07; updated 2026-11-07. https://porsync.com/research/tuned-llama-cpp-vs-strata-16gb
Permanent article URL