ResearchPractical AIProjectsOpen SourceAbout
Research/Expert Caching Made Tokens Faster. Coding Throughput Did Not Improve Clearly
Local AICoding AgentsEvaluationOriginal Research

Expert Caching Made Tokens Faster. Coding Throughput Did Not Improve Clearly

A 6,000 MiB cache improved generation speed, but passed 8/50 repair attempts versus 9/50 without it. A larger timing-only cache was faster, with no agent result.

Bart Porcincula · Published · Updated

Finding

For Qwen3-30B-A3B Q4_K_M on RTX 5060 Ti 16 GB, a 6,000 MiB cache raised later-request decode speed from 30–33 to 41–50 tokens/s, while verified coding throughput was 12.35 versus 12.93 passes per task-hour; correctness equivalence is not established.

Expert Caching Made Tokens Faster. Coding Throughput Did Not Improve Clearly (November 2026)

Expert-cache comparison: same pinned llama.cpp build, 50 coding attempts per A/B/C arm; 90 timing requests across A/B/D; separate 80-attempt-per-arm A/B soak. Task-hour excludes server startup and orchestration overhead.

MeasurementValueNote
Cache-off / 6,000 MiB passes9/50 / 8/50ten known fixtures, five attempts each
Cache-off / 6,000 MiB passes per task-hour12.93 / 12.35failed task time included; startup and orchestration excluded
B/A throughput ratio, cluster-bootstrap interval0.96; 95% interval 0.00–3.53undefined zero-baseline resamples omitted; no equivalence claim
Later-request decode: A / B / D30–33 / 41–50 / 66–82 tokens/sD: 9,000 MiB timing only, no coding trial
Sustained-session passes A / B15/80 / 10/80all 16 rounds completed, zero execution errors

Faster text did not mean more completed fixes

A graphics-memory cache made this local model generate text faster. The coding assistant using it did not demonstrate higher throughput on the tested repairs.

With Qwen3-30B-A3B on an RTX 5060 Ti 16 GB, a 6,000 MiB expert cache raised warm generation speed from about 30–33 to 41–50 tokens per second. Across ten maintenance tasks repeated five times, cache-off passed 9/50, while cache-on passed 8/50.

Counting failed attempts' time, that was 12.93 versus 12.35 passing tasks per task-hour. The estimate is too uncertain to establish equal quality or a small performance difference. It does show why I would not use a token-speed chart alone to claim that a coding workflow got faster.

What was being cached

A mixture-of-experts model uses selected groups of weights, called experts, during inference. This experiment kept some of those weights in graphics memory while other expert weights remained on the CPU side. It tested llama.cpp's expert-cache setting with the same model and backend build in each arm.

This is different from remembering a prompt prefix. The main task runs retained the server's default prompt reuse. The separate timing test disabled prompt reuse so repeated input would not simply skip prompt processing.

The main comparison was cache-off against a 6,000 MiB cache. A 12,000 MiB exploratory configuration exceeded the available graphics-memory budget at the selected context capacity. A fourth configuration, 9,000 MiB, was added for timing only.

The generation gain was real in these requests

The timing test used document slices at roughly 1K, 8K and 16K input tokens, with a 256-token output limit. There were five slices per size, repeated across two sessions with reversed arm order: 90 requests altogether, with no recorded request errors. Every output reached the token limit; the summaries' correctness was not scored.

Approximate inputCache off, later-request tok/s6,000 MiB, later-request tok/s9,000 MiB, later-request tok/s
1K33.149.682.1
8K32.247.876.4
16K30.241.065.6

Prompt processing improved as well. Median full-request times for cache-off versus 6,000 MiB were 11.1 versus 7.5 seconds at 1K, 26.9 versus 20.7 at 8K, and 48.3 versus 40.8 at 16K. The 9,000 MiB arm was faster still, but it never ran the repair tasks.

These are observations from one setup. The first request at each larger size followed smaller requests, so it was not a cold expert-cache measurement. Only the first 1K request followed a server restart, and restarting does not clear the operating system's file cache. The full report avoids treating those first-request figures as a controlled cold-versus-warm comparison.

The assistant still had to solve the task

The coding workload reused ten synthetic multi-file maintenance fixtures from the Edge0 coding-agent study. Patchsmith inspected and changed files, ran independent tests, and stopped on completion or its loop limits. Each arm got five attempts per fixture. A/B order alternated by round, with a fresh server for each arm-round. Sampling was unseeded.

ConfigurationPassing attemptsMedian task timePasses per task-hour
Cache off9/5039.6 seconds12.93
6,000 MiB cache8/5043.1 seconds12.35
12,000 MiB, exceeds memory budget10/50136.4 seconds5.41

Task-hour means the sum of measured attempt durations, including failures. It excludes loading and between-round orchestration. The planned startup-inclusive throughput result was not delivered by this calculation.

The main throughput ratio was 0.96. Resampling whole fixtures gave a very wide 95% interval, 0.00–3.53; resamples with zero baseline throughput could not form a ratio and were omitted. Four paired attempts passed only with cache-off, and three only with cache-on. A full trace review covered one discordant fixture, not every disagreement.

The evidence does not show unchanged correctness. Nor does it establish exactly why generation speed failed to translate into task throughput. Tools, prompt processing and failed loops all consume time, but this run did not produce a complete breakdown of their contributions.

The 12,000 MiB arm is a separate configuration problem. Its launch projected 16,105 MiB of device memory with only 15,172 MiB free. Keeping those failed and slow attempts visible is useful; using them to conclude that a feasible larger cache hurts performance would be misleading.

Short sustained sessions stayed up

Two further sessions ran the arms in opposite orders. Across 80 attempts per arm, cache-off passed 15 and cache-on passed 10. The nominal gap is another reason to avoid claiming preserved quality; it was not a confirmed regression.

All 16 rounds completed with the server alive and zero execution errors. The local run handoff reports no out-of-memory or CUDA errors. Each observation window lasted about 30–35 minutes because the final round was allowed to finish. That is evidence about these sessions, not a guarantee of long-term stability. The cache-on arm also used substantially more graphics memory, approaching the available ceiling.

What this experiment settles

The 6,000 MiB configuration improved generation and request latency in the timing workload. It did not demonstrate better verified repair throughput in the agent workload. The faster 9,000 MiB configuration remains untested on those repairs.

This was a narrower execution than the original plan: fewer distinct timing prompts, no reported separate A/A analysis, no complete startup-inclusive throughput metric, and incomplete discordance review. Repetitive pilot prompts were excluded from the headline. Exact timing prompt hashes were not archived, so later edits to the source document corpus limit reconstruction.

This expert-cache comparison does not establish a new serving default. It also does not fix the unresolved Edge0 coding-agent reliability question: that study used a different model and inference stack. The next useful comparison would put a feasible larger cache through the same independently checked tasks, including failures and all elapsed costs.

Evidence

The Expert-cache comparison results record contains the setup, arm-by-arm tables, uncertainty and deviations. The frozen manifest pins the trial configuration, and the trial CSV retains all 150 attempts. Portable analysis scripts reproduce the trial and timing tables from the public files. Full agent transcripts and exact timing source prompts are not included; see the release notes.

Cite this research

Bart Porcincula. "Expert Caching Made Tokens Faster. Coding Throughput Did Not Improve Clearly." Published 2026-11-15; updated 2026-11-16. https://porsync.com/research/expert-cache-faster-tokens-no-coding-throughput-gain

Permanent article URL