Can the Apprentice Replace the Master? A Small AI Learning From a Big One
Big commercial AI models are good and cost money every time you use them. Small ones run on your own computer for the price of electricity. The tempting idea is to let the small one learn from the big one's answers and then retire the big one. I tested that on a real task, sorting customer banking questions into 77 categories, and the honest result is: the apprentice gets a lot better, but on this task it does not yet match the master. What it learns depends heavily on how good the master is.
Bart Porcincula · Published
Finding
On a 77-category banking-question task, a small 4-billion-parameter model trained on the strongest teacher's answers scored 0.838 on a 0-1 scale against the teacher's 0.936 (below it, outside the pre-set noise bar of 0.017), while training on a better teacher's answers instead of a weak one's lifted the same small model by 0.14 (0.697 to 0.838), and even when the small model answered only the 80% of questions it was surest of it stayed 0.034 behind the teacher on those same questions (0.926 vs 0.960); validation data only, the held-out test set was never opened.
Small model learning from larger models on banking-question sorting (Oct 2026, validation data)
Scores (macro-F1, 0-1 scale unless noted) on 965 validation questions across 77 categories. A small model is trained on the answers of progressively stronger teachers and compared with the teacher itself. The held-out test set was never opened.
| Measurement | Value | Note |
|---|---|---|
| Best small model, trained on the strongest teacher | 0.838 | 4-billion-parameter model, compressed to 4-bit |
| The strongest teacher itself | 0.936 | a larger commercial model |
| Gap to the teacher | about 0.10 | outside the pre-set noise bar of 0.017, so the small model is meaningfully behind |
| Same small model, trained on the weakest teacher | 0.697 | the lift from a better teacher is +0.14 |
| Does a better teacher help? (3 model sizes tested) | yes, at all three | +0.12, +0.14 and +0.14 for the 0.6B, 1.7B and 4B models |
| Does the apprentice match the master? | no, at every size tested | on this task, with these settings, on validation data |
| Small model on the 80% of questions it is surest of | 0.926 accuracy | against 0.851 on all questions |
| Teacher on those same questions | 0.960 accuracy | gap 0.034, about twice the 0.017 noise bar, so still behind |
| Held-out test set | never opened | the plan stopped before it, see the companion piece on the safety check |
The short version
Using a top-tier AI costs a little every time you ask it something. Run it across thousands of customer messages and the bill adds up. A small model that lives on your own computer costs only electricity. So the obvious idea: let the small one learn from the big one, then let the big one go. An apprentice watching a master.
I tested that on a real task: reading a customer's banking question and sorting it into one of 77 categories ("my card was declined", "how do I top up", and so on). Everything below is on the practice-exam half of the data. The final exam stayed sealed, for a reason I explain in the companion piece on the safety check.
What happened
1. The apprentice improves a lot, and depends on who it learns from. Taught by a weak master, the small model scored about 0.70 out of 1. Taught by the strongest master, the same small model reached 0.84. That is a jump of 0.14 purely from picking a better teacher, and it held for all three apprentice sizes I tried.
2. It still doesn't catch up. The strongest master scored 0.94. The best apprentice scored 0.84. That gap of about 0.10 is far bigger than the noise bar (0.017), so this isn't a rounding error. On this task, with these settings, the apprentice did not replace the master.
What this means if you're deciding
- "Train a small copy of the expensive model" is not free money. You can get a lot of the way. Whether the remaining gap is acceptable depends on what a wrong answer costs you.
- Spend on the teacher before the student. The biggest gain here came from the quality of the answers the small model learned from, not from making the small model bigger.
- A 5-point gap is not the same as a 10-point gap. A tool sorting support tickets can live with some errors; one approving payments can't. The question to ask is how many mistakes you can afford, and whether a human can catch them.
- Handing over the hard cases helps, but it doesn't close the gap. I also tried the obvious fix: let the small model answer only the 80% of questions it was surest of, and pass the rest up. On those questions it scored 0.926, up from 0.851 overall. The master scored 0.960 on the very same questions. A gap of 0.034 is about twice the noise bar, so the pre-set test for "as good as the master there" failed too.
- A small model's own confidence is a weak guide. I asked a local model to rate how likely its own answer was to be right, and compared that with a large model stating its confidence in plain words. The large model's stated numbers did better on the main score I'd set (0.10 against 0.18, lower is better), although it was more overconfident on a second measure. The local model's rating was no better than simply reading its confidence off directly.
What this does not show
One task, one dataset, one set of settings, and practice-exam numbers only. A different task, such as reading contracts, could come out differently. I'd rather tell you the apprentice fell short on this task than imply a general rule.
Cite this research
Bart Porcincula. "Can the Apprentice Replace the Master? A Small AI Learning From a Big One." Published 2026-10-22; updated 2026-10-22. https://porsync.com/research/can-the-small-ai-replace-the-big-one-apprentice
Permanent article URL