Can a Small AI Tell You When to Call the Big One? A Pre-Registered Test That Said No
A popular cost-saving idea: let a cheap small AI handle everything, and escalate to an expensive frontier model only when the small one seems unsure. I tested it on 102 real contracts, with the rules written down before any results existed. No confidence signal reliably beat escalating at random. Even where escalating individual answers did help, it saved no money, because the frontier model charges for reading the contract, not for answering.
Bart Porcincula · Published
Finding
On CUAD contract clause extraction (102 test contracts, 41 clause types), a 4B-parameter small model's own confidence did not reliably choose which contracts to escalate to a frontier model (Sonnet 5.5): no signal beat random escalation with a 95% confidence interval excluding zero at 20%, 30% and 40% escalation. Choosing individual answers instead of whole contracts did locate the frontier model's wins (+0.052 macro-F1 over random at 10% of answers), but saved no cost: the frontier bill is driven by reading the document (about $5 per million input tokens, roughly zero for output), so escalating 10% of answers touched 93 of 102 contracts and 95% of the always-frontier cost.
Small-to-frontier cascade routing study on CUAD (Oct 2026)
A small model (gemma-4-e4b, 4B parameters, chunked, constrained output, temperature 0) answered all 41 clause types for each of 102 test contracts. Five routing signals ranked contracts (and later individual answers) for escalation to the frontier model's existing answers, compared against random escalation with a paired bootstrap (10,000 draws). The design, thresholds and verdict rule were frozen before scoring. Every choice was fit on a separate 61-contract validation set.
| Measurement | Value | Note |
|---|---|---|
| Small model accuracy (macro-F1, test) | 0.705 | run-to-run noise is about 0.006 |
| Frontier model accuracy (macro-F1, test) | 0.783 | the gap a cascade would need to close is 0.078 |
| Cost of sending every contract to the frontier model | $13.94 | for the 102 test contracts, at pay-per-use prices |
| Best confidence signal vs random, 30% of contracts escalated | +0.019 [-0.009, +0.048] | interval includes zero: not distinguishable from random |
| Contract length vs random, 30% escalated | +0.034 [+0.009, +0.062] | clears zero at 30% and 40% but not 20%, so reported as flaky |
| Small-vs-tuned-model disagreement, 30% escalated | +0.032 [+0.004, +0.061] | also flaky; the only signal worth a follow-up |
| Cost to get within 2 points of the frontier model, best confidence signal | 66% of always-frontier | needs 53% of contracts escalated; the pre-registered goal was under 60% |
| Per-answer routing, disagreement signal, 10% of answers escalated | 0.766 vs 0.715 random | works, 95% interval excludes zero |
| Contracts touched when escalating 10% of answers | 93 of 102 | 95% of always-frontier cost: a read costs the same for 1 answer or 41 |
The short version
A common way to cut AI costs is a cascade: a cheap small model does all the work, and a few hard cases get passed up to an expensive frontier model. Everything hinges on one question: can the small model tell which cases it will get wrong?
I tested that on a real task, pulling 41 kinds of clause (governing law, termination rights, non-competes and so on) out of 102 legal contracts. The rules, thresholds and what would count as success were written down and frozen before any result existed.
- Choosing which contracts to escalate: nothing worked. The small model's confidence did no better than picking contracts at random, within the margin of error of a sample this size.
- Choosing which individual answers to escalate: this did work. The frontier model's improvements can be located.
- Money saved: none. The frontier model charges mostly for reading the contract, not for answering, so escalating even a few answers means paying for the whole read.
Takeaway: "use the cheap model and escalate when it's unsure" is a plausible-sounding plan that needs testing on the actual task before anyone builds it. Here it did not hold up. Anyone trying it should price the read, not just the answer.
The setup
The small model was a 4-billion-parameter open model running locally, which costs nothing per request. It answered every one of the 41 clause questions for each contract. A router then ranked contracts by some signal and escalated the top share (10%, 20%, 30%...) to the frontier model, Claude Sonnet 5.5, whose answers I already had. Final accuracy mixes the small model's answers for the rest with the frontier model's answers for the escalated set. Accuracy is macro-F1, a score from 0 to 1 that weights rare clause types as much as common ones.
| Validation (61 contracts) | Test (102 contracts) | |
|---|---|---|
| Small model | 0.698 | 0.705 |
| Frontier model | 0.760 | 0.783 |
| Gap to close | 0.062 | 0.078 |
| Cost of using the frontier model for everything | $8.32 | $13.94 |
Every setting was chosen on the validation set. The test set was scored once, so the numbers below are not tuned to it. Run-to-run noise in the small model is about 0.006, so differences smaller than that mean nothing.
What I used to decide when to escalate
Five signals, plus two reference points:
- Random. Escalate contracts at random. This is the floor every real signal has to beat.
- Average confidence from the small model's own token probabilities.
- Count of near-tie answers, where the model barely preferred one answer over the other.
- Self-agreement. Ask the small model several times and escalate where it contradicts itself.
- Contract length. No model needed: escalate the longest contracts.
- Disagreement between two small models, the original and a version fine-tuned on this task.
- Oracle (the ceiling, not a real router): escalate exactly where the frontier model beat the small one by the most.
Result: choosing contracts
A signal only counted as "working" if its 95% confidence interval for the improvement over random excluded zero at all three escalation levels tested (20%, 30% and 40%).
| Signal | Gain over random at 30% escalated | Share of the gap closed | Verdict |
|---|---|---|---|
| Average confidence | +0.019 [-0.009, +0.048] | 58% | no |
| Near-tie count | +0.002 [-0.024, +0.029] | 37% | no |
| Self-agreement | +0.021 [-0.003, +0.048] | 61% | no |
| Contract length | +0.034 [+0.009, +0.062] | 77% | flaky |
| Two-model disagreement | +0.032 [+0.004, +0.061] | 75% | flaky |
Every signal lands above random as a point estimate. That is what a small real effect looks like, and it is also what noise looks like with 102 contracts, which is why the intervals matter. Two signals (length and disagreement) cleared zero at 30% and 40% but not at 20%, so I report them as flaky, not as working.
Two further findings:
- Length is as good as confidence. The simplest router, "escalate the long contracts", matched or beat every confidence signal at 30% and 40% escalated (0.765 vs 0.751 for the confidence signal chosen on validation at 30%), and trailed the best of them by at most 0.004 at 10-20%. If a confidence signal cannot beat a rule that needs no model, it is not earning its complexity.
- The small model is often certain and wrong. Many of its decisions sit at near-100% confidence, so there is little uncertainty to rank on.
Result: choosing individual answers
Because the frontier model had already answered all 41 questions for every contract, I could test a finer router offline: swap in the frontier model's answer for single clause-and-contract cells instead of whole contracts. This was also pre-registered before scoring.
| Signal | Accuracy with 10% of answers escalated | Gain over random answers [95% interval] | Verdict |
|---|---|---|---|
| Random answers | 0.715 | n/a | n/a |
| Answer confidence | 0.744 | +0.029 [+0.008, +0.053] | works |
| Self-agreement | 0.742 | +0.027 [+0.007, +0.050] | flaky |
| Two-model disagreement | 0.766 | +0.052 [+0.025, +0.079] | works |
So the opposite of the contract-level result: a signal that cannot rank whole contracts can still rank individual answers. With the disagreement signal, escalating 20% of answers gets within 0.002 of using the frontier model for everything. One caveat: the two models disagree on only 11% of answers, so beyond that point the extra escalated answers are picked at random. The 5% and 10% figures are the cleanest reading.
Why it still saves no money
I measured what the frontier model actually charges by regressing its cost against tokens across the 102 contracts: about $5 per million input tokens and roughly nothing for output. Its bill is the read, not the answer.
- Escalating 10% of answers touches 93 of 102 contracts, so it costs 95% of the always-frontier price. At 20%, every contract is touched.
- Routing by clause type has the same economics: taking the frontier model on the 10 types where it helped most recovers 65% of the gap, at a realistic cost of 100%.
- At an equal number of contracts touched, sending whole contracts beats sending individual answers (0.786 vs 0.766), because once you have paid for the read you may as well take all 41 answers.
- Even the best-behaved contract-level confidence signal needs about 66% of the always-frontier price to get within 2 points of it. The pre-registered goal was under 60%.
Limits
- One task (contract clause extraction), one small model, one frontier model, one sample of the frontier model's answers, 102 contracts. Differences of 2-3 points are not separable at this size.
- "Frontier" here means the expensive model in the cascade. Sonnet 5.5 is a mid-tier model, not the strongest available, so it is not a ceiling on what escalation could buy.
- A follow-up swapped the 4B model for bigger and more compressed local models: see the second study. A 27B model at about 3 bits scored within noise of this frontier model on its own.
- The small model's 0.705 is a run reading 1,000-token chunks. The same model reading each whole contract at once scores 0.715, the figure in the fine-tuning study.
- The small model reads 1,000-token chunks, while the frontier model reads the whole contract. Escalating changes the input as well as the model, so some of any gain comes from context, not capability.
- The frontier cost is notional (the original run was covered by a subscription plan); the small model's local compute is not priced.
- A real per-answer prompt to the frontier model might answer differently from the all-41-questions call this test reuses.
- The signal data was captured by a second run of the small model that is not bit-identical to the first (98.9-99.0% of answers agree).
What went wrong along the way
I am including this because the credibility of a negative result depends on it.
- The first signal capture silently stored nothing. The probability values came back empty for every chunk (0 of 2,185). A text-matching rule swallowed a leading space in the model's output, and my unit test used a different token shape that hid the bug. I fixed it with a test on real tokenisation and re-ran the small model, about 5 hours of GPU time.
- My first analysis script picked the "best" signal using the test set. That violated my own pre-registration. I corrected it to choose on validation and re-ran; the numbers above are from the corrected run.
The finding, stated for citation
A small model's own confidence does not reliably decide which contracts to escalate to a frontier model on contract clause extraction at this sample size. It can decide which individual answers the frontier model improves, but that does not become a cost saving, because frontier pricing is dominated by reading the document. The untested route to a saving is amortising the read: batching contracts into one call, or a cached second pass at a fraction of the read price.
Cite this research
Bart Porcincula. "Can a Small AI Tell You When to Call the Big One? A Pre-Registered Test That Said No." Published 2026-10-12; updated 2026-10-12. https://porsync.com/research/small-model-confidence-cannot-route-to-frontier-cascade
Permanent article URL