Copying a Bigger AI's Judgement Into a Smaller One: The Teacher Sets the Ceiling
An experiment replacing a mid-size AI classifier with a small trained model. The small model's size, the amount of training data and the output format made no measurable difference. Changing who wrote the training answers made the largest one. And without real outcomes to check against, every score turned out to measure agreement with a judge, not correctness.
Bart Porcincula · Published · Updated
Finding
When distilling an LLM classifier (training a small model on a larger model's labels), the label source sets the accuracy ceiling: a small student model matches its teacher regardless of size, data volume or output constraints, while retraining the identical recipe on a stronger teacher's labels raised balanced accuracy on the accept/reject decision by 10 points over the original student and 7.6 points over the teacher it replaces (both significant on 200 held-out items). After a later rubric fix the gain over the teacher shrank to 2.7 points and was no longer significant. Without outcome data, every gain is agreement with a better-specified judge, not proof of better decisions.
Small-model distillation experiment: findings (Sep 2026)
A small trained model (the student) versus the larger general-purpose model whose answers it was trained on (the teacher), plus variations and a swap to a stronger teacher, on a three-way scoring task with a yes/no accept-or-reject decision. Local hardware, one consumer graphics card.
| Measurement | Value | Note |
|---|---|---|
| Small model vs teacher, accuracy | Tie | within the teacher's own run-to-run variation |
| Speed | ~12× faster | small model vs the larger general-purpose model |
| Consistency between identical runs | Deterministic | the small model's decisions never changed across repeats; the teacher's did |
| Smaller base model, less training data, no output constraint | No measurable effect | each variation was indistinguishable from the reference |
| Swapping in a stronger teacher (same recipe) | 0.634 vs 0.534 / 0.558 balanced accuracy | vs the original student / the teacher, accept/reject decision; McNemar p=0.0001 / p=0.015 |
| Final retrained model after the rubric fix, vs the teacher | 0.656 vs 0.629 | not significant (p=0.54), on a re-labelled held-out set |
| Balanced accuracy as the headline metric | Chosen after first results | a deviation from the pre-registered metric, disclosed in the text |
| Real-world outcomes available to check against | None | every score is agreement with some labeller |
The short version
A classifier in one of my automation pipelines runs on a mid-size general-purpose AI. It works, but it is slow, and its answers drift between identical runs. So I asked the obvious question: could a much smaller model, trained on that bigger model's answers, replace it?
- It could. The small model tied the bigger one on accuracy, ran about 12 times faster, and gave the same answer every time.
- Almost nothing about the small model mattered. A smaller version, a fraction of the training data, a looser output format: none made a measurable difference.
- What mattered was who wrote the training answers. Retraining the same model on answers from a stronger AI lifted accuracy on the accept-or-reject decision by up to ten points.
- The honest limit: I had no real-world outcomes to check against. Every score measures agreement with some judge, not whether a decision was actually right.
Corrected 2026-10-12: the first version did not say that the headline metric was chosen after the first results, overstated the gain against the larger AI, and understated how much of the answer key was random. The details are in the findings below.
Takeaway: when shrinking an AI system to make it faster or cheaper, the effort belongs in the quality of the examples it learns from, not in tuning the small model. And without real outcomes, choosing a model is really choosing a judge.
What I tested
The setup is "teacher and student". The teacher is the larger, general-purpose AI that already does the job. The student is a small model (1.7 billion parameters) trained to imitate the teacher's answers, using a lightweight method called LoRA. The job was a three-way scoring task with an accept-or-reject decision at the end.
The pass/fail thresholds were written down before any training ran, and every comparison used a fixed held-out set of 200 examples, labelled against a written rubric. One deviation: the gate on the reject decision was originally measured by how many true rejects the model caught. After the first results showed that number swinging on a handful of items, I switched it to balanced accuracy. That was a judgement made after seeing data, and the balanced-accuracy results below should be read with that in mind.
Finding 1: the student matches the teacher, and nothing about the student matters
Trained on the teacher's own answers, the small model tied it on the held-out set, ran roughly 12× faster, and was deterministic: its decisions never moved across repeated runs, while the teacher's did.
Then I varied one thing at a time: a smaller base model, a quarter of the training data, a few percent of the training data, and the strict output format switched off. None of them made a measurable difference. A student trained on a small fraction of the examples scored the same as one trained on all of them.
That pattern has one explanation. A student can only learn its teacher's judgement. When size, data volume and output format all stop mattering, the limit is not the student. It is the answers it learns from.
Finding 2: change the teacher, and the ceiling moves
So I kept the recipe and changed the teacher. A stronger, more capable AI re-labelled the same examples against the same written rubric, and I retrained the identical small model on those answers. (One trap worth naming: the new split had quietly pulled some held-out examples into training. Removing them first is the difference between a result and a leak.)
On the accept-or-reject decision, balanced accuracy rose from 0.534 to 0.634, ten points over the original small model and 7.6 points over the larger AI it was meant to replace (0.558). Balanced accuracy is a fair score when one outcome is much rarer than the other. Both gains were statistically significant (McNemar p=0.0001 and p=0.015). Same model, same settings; only the source of the answers changed.
Finding 3: a bad score was really a bad answer key
The three-way result actually got worse, and the loss came from one category the new model never predicted. Tracing it back, I found two answer keys that disagreed about what that category meant. One of them was largely noise: 179 of the 200 held-out answers had been assigned by a category rule, with a random pick inside a score range. For some categories, the top-versus-middle call was literally a coin flip. No model can predict a coin flip.
I made the rule explicit and applied it to both the training answers and the held-out set. After retraining, the rare category went from never predicted to the model's most precise call. Two caveats. That precision is partly by construction, because the training labels and the held-out labels now follow the same rule. And the accept-or-reject gain shrank: the final model scored 0.656 against the larger AI's 0.629, which is no longer statistically distinguishable (p=0.54).
Finding 4: without ground truth, you are choosing a judge
Last, the retrained model ran in shadow alongside production on live traffic, writing to a separate table so production was never touched. It disagreed with the existing system far more than the original small model did, which is exactly what retraining on a different teacher should produce.
This is where the study has to stop claiming things. There was no ground truth. The held-out set is one rubric. The new teacher is another AI's reading of that rubric. The shadow run measures agreement with the existing system. Every number measures agreement with some judge, and none measures whether a decision was right.
What I would take from it
- Copying judgement does not improve it. If the small model ties the teacher regardless of its size or training data, stop tuning the small model and look at the answers it learns from.
- Check the answer key before blaming the model. The worst result in this experiment came from the test set, not the model.
- A small specialist is an operational win before it is an accuracy win. It is faster, consistent, and fits a fixed memory budget, while matching the incumbent.
- Without outcomes, picking a model is picking a judge. The real test is to send a sample of the cases where two models disagree through your normal process and record what actually happens.
Nothing from this experiment is deployed. Its job was to produce the evidence for that decision, not to make it.
Cite this research
Bart Porcincula. "Copying a Bigger AI's Judgement Into a Smaller One: The Teacher Sets the Ceiling." Published 2026-09-28; updated 2026-10-07. https://porsync.com/research/the-teacher-sets-the-ceiling-slm-distillation
Permanent article URL