Expert Caching Made Tokens Faster. Coding Throughput Did Not Improve Clearly
A 6,000 MiB cache improved generation speed, but passed 8/50 repair attempts versus 9/50 without it. A larger timing-only cache was faster, with no agent result.
Bart Porcincula · Published · Updated
Finding
For Qwen3-30B-A3B Q4_K_M on RTX 5060 Ti 16 GB, a 6,000 MiB cache raised later-request decode speed from 30–33 to 41–50 tokens/s, while verified coding throughput was 12.35 versus 12.93 passes per task-hour; correctness equivalence is not established.
Expert Caching Made Tokens Faster. Coding Throughput Did Not Improve Clearly (November 2026)
Expert-cache comparison: same pinned llama.cpp build, 50 coding attempts per A/B/C arm; 90 timing requests across A/B/D; separate 80-attempt-per-arm A/B soak. Task-hour excludes server startup and orchestration overhead.
| Measurement | Value | Note |
|---|---|---|
| Cache-off / 6,000 MiB passes | 9/50 / 8/50 | ten known fixtures, five attempts each |
| Cache-off / 6,000 MiB passes per task-hour | 12.93 / 12.35 | failed task time included; startup and orchestration excluded |
| B/A throughput ratio, cluster-bootstrap interval | 0.96; 95% interval 0.00–3.53 | undefined zero-baseline resamples omitted; no equivalence claim |
| Later-request decode: A / B / D | 30–33 / 41–50 / 66–82 tokens/s | D: 9,000 MiB timing only, no coding trial |
| Sustained-session passes A / B | 15/80 / 10/80 | all 16 rounds completed, zero execution errors |
Faster text did not mean more completed fixes
A graphics-memory cache made this local model generate text faster. The coding assistant using it did not demonstrate higher throughput on the tested repairs.
With Qwen3-30B-A3B on an RTX 5060 Ti 16 GB, a 6,000 MiB expert cache raised warm generation speed from about 30–33 to 41–50 tokens per second. Across ten maintenance tasks repeated five times, cache-off passed 9/50, while cache-on passed 8/50.
Counting failed attempts' time, that was 12.93 versus 12.35 passing tasks per task-hour. The estimate is too uncertain to establish equal quality or a small performance difference. It does show why I would not use a token-speed chart alone to claim that a coding workflow got faster.
What was being cached
A mixture-of-experts model uses selected groups of weights, called experts, during inference. This experiment kept some of those weights in graphics memory while other expert weights remained on the CPU side. It tested llama.cpp's expert-cache setting with the same model and backend build in each arm.
This is different from remembering a prompt prefix. The main task runs retained the server's default prompt reuse. The separate timing test disabled prompt reuse so repeated input would not simply skip prompt processing.
The main comparison was cache-off against a 6,000 MiB cache. A 12,000 MiB exploratory configuration exceeded the available graphics-memory budget at the selected context capacity. A fourth configuration, 9,000 MiB, was added for timing only.
The generation gain was real in these requests
The timing test used document slices at roughly 1K, 8K and 16K input tokens, with a 256-token output limit. There were five slices per size, repeated across two sessions with reversed arm order: 90 requests altogether, with no recorded request errors. Every output reached the token limit; the summaries' correctness was not scored.
| Approximate input | Cache off, later-request tok/s | 6,000 MiB, later-request tok/s | 9,000 MiB, later-request tok/s |
|---|---|---|---|
| 1K | 33.1 | 49.6 | 82.1 |
| 8K | 32.2 | 47.8 | 76.4 |
| 16K | 30.2 | 41.0 | 65.6 |
Prompt processing improved as well. Median full-request times for cache-off versus 6,000 MiB were 11.1 versus 7.5 seconds at 1K, 26.9 versus 20.7 at 8K, and 48.3 versus 40.8 at 16K. The 9,000 MiB arm was faster still, but it never ran the repair tasks.
These are observations from one setup. The first request at each larger size followed smaller requests, so it was not a cold expert-cache measurement. Only the first 1K request followed a server restart, and restarting does not clear the operating system's file cache. The full report avoids treating those first-request figures as a controlled cold-versus-warm comparison.
The assistant still had to solve the task
The coding workload reused ten synthetic multi-file maintenance fixtures from the Edge0 coding-agent study. Patchsmith inspected and changed files, ran independent tests, and stopped on completion or its loop limits. Each arm got five attempts per fixture. A/B order alternated by round, with a fresh server for each arm-round. Sampling was unseeded.
| Configuration | Passing attempts | Median task time | Passes per task-hour |
|---|---|---|---|
| Cache off | 9/50 | 39.6 seconds | 12.93 |
| 6,000 MiB cache | 8/50 | 43.1 seconds | 12.35 |
| 12,000 MiB, exceeds memory budget | 10/50 | 136.4 seconds | 5.41 |
Task-hour means the sum of measured attempt durations, including failures. It excludes loading and between-round orchestration. The planned startup-inclusive throughput result was not delivered by this calculation.
The main throughput ratio was 0.96. Resampling whole fixtures gave a very wide 95% interval, 0.00–3.53; resamples with zero baseline throughput could not form a ratio and were omitted. Four paired attempts passed only with cache-off, and three only with cache-on. A full trace review covered one discordant fixture, not every disagreement.
The evidence does not show unchanged correctness. Nor does it establish exactly why generation speed failed to translate into task throughput. Tools, prompt processing and failed loops all consume time, but this run did not produce a complete breakdown of their contributions.
The 12,000 MiB arm is a separate configuration problem. Its launch projected 16,105 MiB of device memory with only 15,172 MiB free. Keeping those failed and slow attempts visible is useful; using them to conclude that a feasible larger cache hurts performance would be misleading.
Short sustained sessions stayed up
Two further sessions ran the arms in opposite orders. Across 80 attempts per arm, cache-off passed 15 and cache-on passed 10. The nominal gap is another reason to avoid claiming preserved quality; it was not a confirmed regression.
All 16 rounds completed with the server alive and zero execution errors. The local run handoff reports no out-of-memory or CUDA errors. Each observation window lasted about 30–35 minutes because the final round was allowed to finish. That is evidence about these sessions, not a guarantee of long-term stability. The cache-on arm also used substantially more graphics memory, approaching the available ceiling.
What this experiment settles
The 6,000 MiB configuration improved generation and request latency in the timing workload. It did not demonstrate better verified repair throughput in the agent workload. The faster 9,000 MiB configuration remains untested on those repairs.
This was a narrower execution than the original plan: fewer distinct timing prompts, no reported separate A/A analysis, no complete startup-inclusive throughput metric, and incomplete discordance review. Repetitive pilot prompts were excluded from the headline. Exact timing prompt hashes were not archived, so later edits to the source document corpus limit reconstruction.
This expert-cache comparison does not establish a new serving default. It also does not fix the unresolved Edge0 coding-agent reliability question: that study used a different model and inference stack. The next useful comparison would put a feasible larger cache through the same independently checked tasks, including failures and all elapsed costs.
Evidence
The Expert-cache comparison results record contains the setup, arm-by-arm tables, uncertainty and deviations. The frozen manifest pins the trial configuration, and the trial CSV retains all 150 attempts. Portable analysis scripts reproduce the trial and timing tables from the public files. Full agent transcripts and exact timing source prompts are not included; see the release notes.
Cite this research
Bart Porcincula. "Expert Caching Made Tokens Faster. Coding Throughput Did Not Improve Clearly." Published 2026-11-15; updated 2026-11-16. https://porsync.com/research/expert-cache-faster-tokens-no-coding-throughput-gain
Permanent article URL