ResearchPractical AIProjectsOpen SourceAbout
Research/Squeeze a Big AI Model Into Less Memory, or Run a Smaller One? A Pre-Registered Test on a 16 GB Card
Local AIQuantizationModel CascadesPre-RegisteredOriginal Research

Squeeze a Big AI Model Into Less Memory, or Run a Smaller One? A Pre-Registered Test on a 16 GB Card

When your graphics card is the limit, you can run a small model at full quality or a much bigger one compressed to fit. I tested eight setups on 102 real contracts, with the rules written down first. The compressed big models won clearly, and the most compressed of them (13.5 GB) did as well as the least compressed (16 GB). The best of them landed within noise of a frontier model with no escalation at all. Deciding which contracts to hand up to the frontier model mostly still did not work: of 28 setup-and-signal pairs, 1 passed, 7 were flaky and 20 failed.

Bart Porcincula · Published

Finding

On CUAD contract clause extraction (102 test contracts, 41 clause types), Qwen3.6-27B compressed to 2.7, 3.1 and 4.3 bits per weight scored 0.757, 0.769 and 0.762 macro-F1, statistically indistinguishable from each other and from the frontier model (Sonnet 5.5, 0.783; the paired differences from it were -0.026, -0.014 and -0.021, every 95% interval including zero), against 0.705 for a 4B small model. A 9B model at normal precision (0.703) and a 27B model squeezed to about 1 bit per weight (0.679) did not beat the 4B model. Across 28 pairs of setup and escalation signal, one passed the pre-registered routing test at every escalation level (contract length on the ternary 27B model), 7 passed at some levels and 20 at none; no confidence signal ever passed.

Local model size vs compression on a 16 GB graphics card, CUAD contracts (Oct 2026)

Seven new local setups (a 9B model, a 27B model at five compression levels including two purpose-built low-bit variants, and one reading whole contracts instead of 1,000-token chunks) answered all 41 clause types for each of 102 test contracts and 61 validation contracts, using the first study's request shape (temperature 0, constrained output). Accuracy is macro-F1 against the dataset's gold labels, compared with a paired bootstrap (10,000 draws, contract-level) against the 4B reference model and against random escalation to the frontier model's stored answers. Design, hypotheses and the verdict rule were frozen before the first run. Memory is the peak graphics-card reading during each setup's own run.

MeasurementValueNote
4B reference model, no escalation (macro-F1, test)0.705from the first study; the frontier model scores 0.783
27B at 2.7 bits per weight, 13.5 GB peak (Q27-IQ2)0.757+0.053 [+0.024, +0.083] over the 4B model
27B at 3.1 bits per weight, 15.0 GB peak (Q27-IQ3)0.769closes 83% of the gap to the frontier model [51%, 124%]
27B at 4.3 bits per weight, 16.0 GB peak (Q27-IQ4)0.762the largest file did not score higher than the smallest
Best 27B setup vs frontier model, difference-0.014 [-0.041, +0.015]interval includes zero; with 102 contracts this does not prove they are equal
9B model at normal precision (N9)0.703no better than the 4B model: -0.003 [-0.033, +0.027]
27B squeezed to about 1 bit per weight (BQ1)0.679slightly worse than the 4B model: -0.028 [-0.057, +0.001]
Reading the whole contract vs 1,000-token chunks, same weights+0.004 [-0.019, +0.025]no measurable difference
Routing pairs passing the pre-registered test at all three levels1 of 287 flaky, 20 no; contract length on the ternary 27B model passed, no confidence signal did

The short version

If you want to run an AI model on your own computer, the graphics card's memory (VRAM) decides what fits. A bigger model is usually more capable, but a bigger model also needs more memory. The workaround is quantization: store the model's numbers with fewer bits, so a 27-billion-parameter model that would need 54 GB fits in 8 to 16 GB. The worry is that compressing it hurts the answers.

I tested that on the same task as the first cascade study: pulling 41 kinds of clause out of 102 legal contracts. Eight local setups, one 16 GB graphics card, rules and hypotheses written down before the first run.

  • Compressed big beat uncompressed small. Three versions of a 27B model scored 0.757 to 0.769, against 0.705 for the 4B model from the first study. They landed within noise of the frontier model (0.783) without ever calling it.
  • More bits bought nothing. The most compressed of the three (13.5 GB) scored the same as the least compressed (16 GB), within noise.
  • Compression has a floor. At about 1 bit per weight the 27B model fell to 0.679, below the 4B model.
  • Model size alone did not help. A 9B model at normal precision scored 0.703, the same as the 4B model.
  • Routing still mostly failed. Picking which contracts to hand up to the frontier model beat random in 1 of 28 setup-and-signal pairs at every tested level, and the one that did is a weak result.

Takeaway: on a 16 GB card, if the choice is "small model at full quality" or "big model compressed to about 3 bits", the compressed big model won here, and extra bits past that were wasted memory. But the gain came from the model, not from a clever hand-off to a frontier model, and a 102-contract sample cannot tell a near-match from an exact one.

The setup

The reference was the 4B model from the first study, reading each contract in 1,000-token chunks. The frontier model was Claude Sonnet 5.5, whose answers on these contracts I already had, so this study spent nothing on frontier calls. Everything else was held fixed: the prompt, temperature 0, constrained JSON output, the scoring script, and the 61-contract validation / 102-contract test split. Every choice was made on validation. Test was scored once.

The setups, with the memory each actually peaked at:

SetupWhat it isBits per weightPeak memoryTest accuracy (macro-F1)
Reference (4B)Gemma 4B, from the first studyabout 4.8n/a0.705
N9Qwen3.5-9B, normal precisionabout 4.87.1 GB0.703
BQ1Bonsai-27B, trained for 1-bit weightsabout 1.17.1 GB0.679
BT2Ternary Bonsai-27B, trained for ternary weightsabout 1.710.2 GB0.742
Q27-IQ2Qwen3.6-27B compressed after trainingabout 2.713.5 GB0.757
Q27-IQ3same, less compressedabout 3.115.0 GB0.769
W27Q27-IQ3's weights, whole contract in one callabout 3.115.4 GB0.773
Q27-IQ4same base, least compressedabout 4.316.0 GB0.762
Frontier (Sonnet 5.5)stored answersn/an/a0.783

I wanted to answer "what is the best local model for an 8, 12 or 16 GB card", so I assigned each setup to a memory tier by its measured peak minus the idle desktop's usage, leaving 1 GB headroom. By that pre-registered rule N9 and BQ1 fit 8 GB, BT2 fits 12 GB, and IQ2, IQ3 and W27 fit 16 GB. Q27-IQ4 does not: its net peak was 15.2 GB, 0.2 GB over the 15 GB limit, so it is reported as fitting no tier, not rerun with a smaller context to squeeze it in.

Result: how good is each setup alone?

SetupGain over the 4B model [95% interval]Share of the gap to the frontier model closed [95% interval]
N9-0.003 [-0.033, +0.027]-6% [-54%, 30%]
BQ1-0.028 [-0.057, +0.001]-38% [-91%, 2%]
BT2+0.037 [+0.012, +0.062]47% [18%, 76%]
Q27-IQ2+0.053 [+0.024, +0.083]68% [35%, 106%]
Q27-IQ3+0.065 [+0.036, +0.095]83% [51%, 124%]
W27+0.068 [+0.033, +0.104]89% [44%, 147%]
Q27-IQ4+0.058 [+0.030, +0.086]74% [46%, 107%]

The four 27B arms that were compressed after training (IQ2, IQ3, IQ4, and W27 which shares IQ3's weights) are all clearly better than the 4B model and sit close together; their intervals overlap almost entirely. Against the frontier model itself the differences are -0.026 (IQ2), -0.014 (IQ3), -0.010 (W27) and -0.021 (IQ4), and every interval includes zero. That says no gap was detected. It does not say there is none: with 102 contracts, a real gap of two or three points would not show up.

The two purpose-built low-bit models split. Ternary (about 1.7 bits) gets partway (0.742, below the frontier model by a clear margin: -0.042 [-0.068, -0.014]). The 1-bit one does not get there at all.

Result: did the hypotheses hold?

I froze six hypotheses before the first run and report them as they came out.

HypothesisOutcome
H1. The 3-bit 27B closes at least half the gap to the frontier modelHeld. 83% [51%, 124%], interval excludes zero.
H2. Better-calibrated confidence with more bits (ranking quality for spotting wrong answers rises from 1 to 4 bits)Held on point estimates, as registered. Ranking quality (AUROC): 0.849 at 1 bit, 0.913, 0.914, 0.914 at 2.7, 3.1, 4.3 bits. It rises, then flattens from about 2.7 bits.
H3. Disagreement with the 4B model is a working contract-level router for at least one 27B armRefuted. None passed at all three levels; only Q27-IQ4 was flaky.
H4. Reading the whole contract beats chunking, on the same weightsRefuted. +0.004 [-0.019, +0.025] on test; on validation the sign was negative.
H5. A cheap cascade gets within 2 points of the frontier model at under 50% of its costNot evaluated as a cost question. At zero escalation, IQ3, W27 and IQ4 were already within 2 points of the frontier model, so the question moved. I did not price the cascade arms.
H6. At 8 GB, the 1-bit 27B beats the 9B at normal precision (exploratory)Refuted. 0.679 vs 0.703.

Result: routing

A router ranks contracts by some signal and hands the top share (20%, 30%, 40%) up to the frontier model. A signal only counted as "working" if its improvement over escalating at random had a 95% interval above zero at all three levels. 4 signals on 7 setups gives 28 pairs.

  • 1 worked: contract length on the ternary 27B model (+0.019, +0.027, +0.020 at the three levels). It is the weakest kind of result: 1 hit out of 28 tries, the signal picked after the fact. Treat it as a lead.
  • 7 were flaky (passed at one or two levels): contract length on N9, BQ1, Q27-IQ3 and W27; disagreement with the 4B model on N9 and Q27-IQ4; the near-tie count on Q27-IQ2.
  • 20 failed. The model's own confidence signals (average confidence and near-tie count) never passed anywhere, apart from the one flaky near-tie result.

The pattern from the first study repeated: the simplest router, escalate the long contracts, was the most frequent partial winner, and nothing built on the model's own confidence did better. What does work is finer-grained. When the question is "is this one answer wrong?", the models' low confidence separates the cases where the frontier model is right and the local one wrong very well (AUROC 0.85 to 0.92 across setups), but that does not make a contract-level router.

There is a second reason routing matters less here. Without any escalation, the best 27B setups are already within about 0.01 to 0.03 of the frontier model, so there is little left for a router to buy. Sending 20 to 40% of contracts up takes the strong setups to about 0.78 to 0.79, roughly what the frontier model scores alone.

Limits

  • One task, one frontier model, one sample of the frontier model's answers, 102 contracts, one graphics card. Differences under 2 to 3 points do not separate. Seven setups against one reference with paired intervals is not a ranking.
  • Memory tiers are emulated, not tested on 8 or 12 GB cards. For three setups (N9, BQ1 and Q27-IQ2) the graphics-card peak comes from one or two readings taken at the start, not a trace across the run, so those peaks are approximate. The other four have thousands of readings.
  • Not every difference is clean. The ternary setup ran on a different build of the inference software from the rest. The IQ2 and IQ3 files use mixed per-layer compression; the IQ4 file does not, so the bits axis is not perfectly controlled.
  • Thinking mode was off for parity with the first study. Vendor-published numbers for these models are mostly with thinking on, so they do not transfer.
  • The 27B models read 1,000-token chunks while the frontier model reads the whole contract, so escalating changes the input as well as the model. The whole-contract setup (W27) tested this on the local side and found no measurable difference.
  • Wall-clock times are not matched across setups (the largest file ran one request at a time and took about 33 hours; Q27-IQ3 ran four at once and took about 13), so I do not report speed as a finding.
  • Cost: local compute is not priced. Sending all 102 test contracts to the frontier model costs $13.94 at pay-per-use prices.
  • One signal from the first study (self-agreement across repeated samples) was not run here, because it triples the compute.

What went wrong along the way

  • The large model files did not all load first time. The ternary Bonsai file was rejected by the standard inference software and needed a separate vendor build; that is why it is a flagged confound, not folded in silently.
  • Some memory traces are thin. The sampler that records peak memory only captured one or two readings for three of the early setups. I found this when writing the results up, and it is why the limits above say so.

The finding, stated for citation

On contract clause extraction on a 16 GB graphics card, a 27B model compressed to 2.7 to 4.3 bits per weight scores about 0.76 to 0.77 macro-F1 against 0.705 for a 4B model and 0.783 for a frontier model, with no measurable difference between the three compression levels and none detected from the frontier model at n=102. A 9B model at normal precision and a 27B model at about 1 bit do not beat the 4B model. Choosing which contracts to escalate to the frontier model still mostly fails: 1 of 28 setup-and-signal pairs passed the registered test.

Cite this research

Bart Porcincula. "Squeeze a Big AI Model Into Less Memory, or Run a Smaller One? A Pre-Registered Test on a 16 GB Card." Published 2026-10-18; updated 2026-10-18. https://porsync.com/research/big-model-squeezed-into-less-memory-local-cascade

Permanent article URL