ResearchPractical AIProjectsOpen SourceAbout
Research/Run Eight Requests at Once and a 9B Model Delivers 1.32 Times the Correct Answers per Hour, Not the 2 Times I Predicted. The Giant Coder Model Got Slower.
Local AIBatch ThroughputConcurrencyInference EnginesLatencyPre-RegisteredOriginal Research

Run Eight Requests at Once and a 9B Model Delivers 1.32 Times the Correct Answers per Hour, Not the 2 Times I Predicted. The Giant Coder Model Got Slower.

My earlier batch studies sent one request at a time. A batch job has nobody waiting, so it can send several at once. I tested 1, 2, 4 and 8 parallel slots for the 9B model and 1 and 2 for the giant Coder model on Niko1221's Strata engine, over the same queue of short business tasks. Batching helped the 9B by 1.32 times at eight slots, accuracy stayed within a third of a point, and the price was a 9-fold rise in the slowest 5% of response times. The Coder got slower at two slots.

Bart Porcincula · Published · Updated

Finding

On an RTX 5060 Ti 16 GB, Qwen3.5-9B Q4_K_M on stock llama.cpp b11753 delivered 2,584 correct items per hour at one slot and 3,408 at eight, on the first 8,180 items both runs finished: ratio 1.32 (paired bootstrap, 10,000 draws; the interval is very narrow because wall time is held fixed, so it covers item sampling only). Accuracy was 83.67% against 83.36%, a gap of -0.31 points (CI -0.51 to -0.10). The curve was 1.12 times at two slots, 1.26 times at four and 1.05 times more from four to eight. My pre-written prediction that four slots would reach at least 2 times was refuted. Median seconds per item rose from 1.36 at one slot to 4.31 at eight, and the slowest 5% from 2.27 to 20.95 seconds. The Coder IQ1_M on Niko1221's Strata engine (Terminal 0.1.54) delivered 0.75 times as many correct items per hour at two slots as at one on 2,612 matched items (accuracy -0.38 points, CI -0.88 to +0.08), with 410 MiB of free RAM at its lowest. At each model's best slot count the 9B delivered 3,328 correct items per hour against the Coder's 1,534, and at one slot the ratio was 1.68 times, as in the earlier study.

Parallel slots vs correct answers per hour, 9B on stock llama.cpp and Coder on Niko1221's Strata engine, 16 GB VRAM (Nov 2026)

One run per configuration, one machine, different days. RTX 5060 Ti 16 GB, 31.6 GB RAM. 9B Q4_K_M on llama.cpp b11753 with 1, 2, 4 and 8 slots (10,240 tokens of context per slot). Coder IQ1_M on Niko1221's Strata engine, Terminal 0.1.54, thinking off through a pass-through proxy, 1 slot (reused from the earlier 8 h run) and 2 slots (32K context). Same shuffled queue of synthetic items (seed 0) as the eight-hour batch study, temperature 0, 300 s cap, client requests in flight equal to the slot count. 2 h per new configuration. Compared on the first K items both runs finished. Hypotheses were written before the first scored run.

MeasurementValueNote
9B correct items per hour, 1 / 2 / 4 / 8 slots2,584 / 2,904 / 3,268 / 3,408K differs per comparison; headline K = 8,180 for 1 vs 8
9B 8 slots over 1 slot (headline)1.32interval covers item sampling only
9B 4 slots over 1 slot1.26prediction was at least 2; refuted
9B 8 slots over 4 slots1.05prediction was under 1.3; supported
9B accuracy, 4 slots minus 1 slot-0.23 pointsCI -0.46 to 0.00; within the 1-point bar
9B median / slowest-5% seconds per item, 1 vs 8 slots1.36 / 2.27 vs 4.31 / 20.95
Coder 2 slots over 1 slot0.752,612 matched items; accuracy -0.38 points
Coder lowest free RAM at 2 slots410 MiBpeak VRAM 15,488 MiB
9B / Coder correct per hour at each one's best slot count3,328 vs 1,534matched K = 2,612
Errors, empty replies, hit the 300 s cap0.0% in every configuration

The short version

The earlier batch studies sent one request at a time. A batch job has no one waiting for an answer, so a server can run several requests at once. This study asks how much that adds in correct answers per hour, whether accuracy survives it, and whether it changes which model comes out ahead.

  • The 9B gained, but about a quarter of what I predicted. Going from one slot to eight raised correct answers per hour from 2,584 to 3,408, a factor of 1.32. I had written down that four slots would deliver at least twice the one-slot figure. It delivered 1.26 times, so that prediction is refuted.
  • The curve flattens early. Two slots gave 1.12 times, four gave 1.26 times and eight added only 1.05 times more. Beyond four slots there was little left to gain on this card.
  • Accuracy mostly survived. At four slots the 9B was 0.23 points below one slot, inside my 1-point bar. At eight slots the gap was -0.31 points with an interval from -0.51 to -0.10, small but not zero. Extraction items carried most of the loss; log triage was 100% in every configuration.
  • The price is waiting time. The median time per item rose from 1.4 seconds to 4.3, and the slowest 5% from 2.3 seconds to 21. That is fine when nobody is waiting and wrong for anything interactive.
  • The giant Coder got slower with two slots. It delivered 0.75 times as many correct answers per hour as with one. It fit in memory, but only just, so I do not read this as a general statement about batching on that engine.
  • The ordering held. At each model's best setting the 9B delivered 3,328 correct answers per hour against the Coder's 1,534.

Takeaway: parallel slots are worth turning on for a small model in a batch queue, up to about four, and the accuracy cost was small here. They did not come close to the near-linear scaling I expected, and on a machine already at its memory edge they did not help the large model at all.

How it was run

Same card (RTX 5060 Ti, 16 GB), the same 30,000-item shuffled pool of short business tasks as the eight-hour batch study (intent labels, invoice extraction, log triage), temperature 0, thinking off, 300-second cap. The 9B ran on stock llama.cpp b11753 with one, two, four and eight slots, with the client keeping as many requests in flight as there were slots. The Coder ran on Niko1221's Strata engine with one slot (reused from the earlier eight-hour run) and two slots. Each new configuration ran for two hours. Correct answers per hour is correct items divided by wall-clock time, because per-item times overlap under batching. Configurations were compared on the first items that both finished, with a paired bootstrap over items (10,000 draws, seed 0).

What the hypotheses said

HypothesisVerdict
9B at 4 slots delivers at least 2 times the correct per hour of 1 slotRefuted (1.26)
9B accuracy at 4 slots is within 1 point of 1 slotSupported (-0.23)
8 slots add under 1.3 times over 4 slotsSupported (1.05)
Coder at 2 slots gains under 1.3 times over 1 slot, or does not fitSupported (0.75)
The 9B still delivers more correct per hour than the Coder at each one's best slot countSupported (3,328 vs 1,534)

Four of five held. The refuted one was the one that mattered most for the headline.

What I am not claiming

  • One run per configuration. The interval covers which items were sampled and holds time per item fixed, so it says nothing about run-to-run noise, and the third digit of each ratio is not settled.
  • The one-slot baselines are the earlier runs from different days. Versions are pinned, but drift is a confound.
  • I do not know why the Coder lost throughput at two slots. With 410 MiB of free RAM at its lowest, memory pressure is a candidate; I did not test it, and I did not try more than two slots on that engine.
  • The tasks are synthetic and the answers short. Longer outputs could batch differently.
  • The Coder is a heavily compressed (IQ1_M) quantisation, so this is about that file on that engine, not the model family.
  • Throughput is not responsiveness. The latency figures above are the cost of the throughput gain.
  • A load-test ramp to find each server's slot ceiling was planned and deliberately skipped.

Will running more AI jobs at once clear a backlog faster?

It helped the smaller model in this batch test, but slowed the larger one. The small model's gain also came with longer waits for individual responses. An overnight queue and an assistant serving waiting staff have different requirements. Measure completed correct work and response time on your setup before increasing simultaneous requests.

Cite this research

Bart Porcincula. "Run Eight Requests at Once and a 9B Model Delivers 1.32 Times the Correct Answers per Hour, Not the 2 Times I Predicted. The Giant Coder Model Got Slower.." Published 2026-11-03; updated 2026-11-08. https://porsync.com/research/batching-slots-small-model-1-32x-correct-answers-per-hour-giant-coder-model-slower-16gb

Permanent article URL