ResearchPractical AIProjectsOpen SourceAbout
Research/Copying a Bigger AI's Judgement Into a Smaller One: The Teacher Sets the Ceiling
Small Language ModelsDistillationLoRAEvaluationMethodologyOriginal Research

Copying a Bigger AI's Judgement Into a Smaller One: The Teacher Sets the Ceiling

An experiment replacing a mid-size AI classifier with a small trained model. The small model's size, the amount of training data and the output format made no measurable difference. Changing who wrote the training answers made the largest one. And without real outcomes to check against, every score turned out to measure agreement with a judge, not correctness.

Bart Porcincula · Published · Updated

Finding

When distilling an LLM classifier (training a small model on a larger model's labels), the label source sets the accuracy ceiling: a small student model matches its teacher regardless of size, data volume or output constraints, while retraining the identical recipe on a stronger teacher's labels raised balanced accuracy on the accept/reject decision by 10 points over the original student and 7.6 points over the teacher it replaces (both significant on 200 held-out items). After a later rubric fix the gain over the teacher shrank to 2.7 points and was no longer significant. Without outcome data, every gain is agreement with a better-specified judge, not proof of better decisions.

Small-model distillation experiment: findings (Sep 2026)

A small trained model (the student) versus the larger general-purpose model whose answers it was trained on (the teacher), plus variations and a swap to a stronger teacher, on a three-way scoring task with a yes/no accept-or-reject decision. Local hardware, one consumer graphics card.

MeasurementValueNote
Small model vs teacher, accuracyTiewithin the teacher's own run-to-run variation
Speed~12× fastersmall model vs the larger general-purpose model
Consistency between identical runsDeterministicthe small model's decisions never changed across repeats; the teacher's did
Smaller base model, less training data, no output constraintNo measurable effecteach variation was indistinguishable from the reference
Swapping in a stronger teacher (same recipe)0.634 vs 0.534 / 0.558 balanced accuracyvs the original student / the teacher, accept/reject decision; McNemar p=0.0001 / p=0.015
Final retrained model after the rubric fix, vs the teacher0.656 vs 0.629not significant (p=0.54), on a re-labelled held-out set
Balanced accuracy as the headline metricChosen after first resultsa deviation from the pre-registered metric, disclosed in the text
Real-world outcomes available to check againstNoneevery score is agreement with some labeller

The short version

A classifier in one of my automation pipelines runs on a mid-size general-purpose AI. It works, but it is slow, and its answers drift between identical runs. So I asked the obvious question: could a much smaller model, trained on that bigger model's answers, replace it?

  • It could. The small model tied the bigger one on accuracy, ran about 12 times faster, and gave the same answer every time.
  • Almost nothing about the small model mattered. A smaller version, a fraction of the training data, a looser output format: none made a measurable difference.
  • What mattered was who wrote the training answers. Retraining the same model on answers from a stronger AI lifted accuracy on the accept-or-reject decision by up to ten points.
  • The honest limit: I had no real-world outcomes to check against. Every score measures agreement with some judge, not whether a decision was actually right.

Corrected 2026-10-12: the first version did not say that the headline metric was chosen after the first results, overstated the gain against the larger AI, and understated how much of the answer key was random. The details are in the findings below.

Takeaway: when shrinking an AI system to make it faster or cheaper, the effort belongs in the quality of the examples it learns from, not in tuning the small model. And without real outcomes, choosing a model is really choosing a judge.

What I tested

The setup is "teacher and student". The teacher is the larger, general-purpose AI that already does the job. The student is a small model (1.7 billion parameters) trained to imitate the teacher's answers, using a lightweight method called LoRA. The job was a three-way scoring task with an accept-or-reject decision at the end.

The pass/fail thresholds were written down before any training ran, and every comparison used a fixed held-out set of 200 examples, labelled against a written rubric. One deviation: the gate on the reject decision was originally measured by how many true rejects the model caught. After the first results showed that number swinging on a handful of items, I switched it to balanced accuracy. That was a judgement made after seeing data, and the balanced-accuracy results below should be read with that in mind.

Finding 1: the student matches the teacher, and nothing about the student matters

Trained on the teacher's own answers, the small model tied it on the held-out set, ran roughly 12× faster, and was deterministic: its decisions never moved across repeated runs, while the teacher's did.

Then I varied one thing at a time: a smaller base model, a quarter of the training data, a few percent of the training data, and the strict output format switched off. None of them made a measurable difference. A student trained on a small fraction of the examples scored the same as one trained on all of them.

That pattern has one explanation. A student can only learn its teacher's judgement. When size, data volume and output format all stop mattering, the limit is not the student. It is the answers it learns from.

Finding 2: change the teacher, and the ceiling moves

So I kept the recipe and changed the teacher. A stronger, more capable AI re-labelled the same examples against the same written rubric, and I retrained the identical small model on those answers. (One trap worth naming: the new split had quietly pulled some held-out examples into training. Removing them first is the difference between a result and a leak.)

On the accept-or-reject decision, balanced accuracy rose from 0.534 to 0.634, ten points over the original small model and 7.6 points over the larger AI it was meant to replace (0.558). Balanced accuracy is a fair score when one outcome is much rarer than the other. Both gains were statistically significant (McNemar p=0.0001 and p=0.015). Same model, same settings; only the source of the answers changed.

Finding 3: a bad score was really a bad answer key

The three-way result actually got worse, and the loss came from one category the new model never predicted. Tracing it back, I found two answer keys that disagreed about what that category meant. One of them was largely noise: 179 of the 200 held-out answers had been assigned by a category rule, with a random pick inside a score range. For some categories, the top-versus-middle call was literally a coin flip. No model can predict a coin flip.

I made the rule explicit and applied it to both the training answers and the held-out set. After retraining, the rare category went from never predicted to the model's most precise call. Two caveats. That precision is partly by construction, because the training labels and the held-out labels now follow the same rule. And the accept-or-reject gain shrank: the final model scored 0.656 against the larger AI's 0.629, which is no longer statistically distinguishable (p=0.54).

Finding 4: without ground truth, you are choosing a judge

Last, the retrained model ran in shadow alongside production on live traffic, writing to a separate table so production was never touched. It disagreed with the existing system far more than the original small model did, which is exactly what retraining on a different teacher should produce.

This is where the study has to stop claiming things. There was no ground truth. The held-out set is one rubric. The new teacher is another AI's reading of that rubric. The shadow run measures agreement with the existing system. Every number measures agreement with some judge, and none measures whether a decision was right.

What I would take from it

  1. Copying judgement does not improve it. If the small model ties the teacher regardless of its size or training data, stop tuning the small model and look at the answers it learns from.
  2. Check the answer key before blaming the model. The worst result in this experiment came from the test set, not the model.
  3. A small specialist is an operational win before it is an accuracy win. It is faster, consistent, and fits a fixed memory budget, while matching the incumbent.
  4. Without outcomes, picking a model is picking a judge. The real test is to send a sample of the cases where two models disagree through your normal process and record what actually happens.

Nothing from this experiment is deployed. Its job was to produce the evidence for that decision, not to make it.

Cite this research

Bart Porcincula. "Copying a Bigger AI's Judgement Into a Smaller One: The Teacher Sets the Ceiling." Published 2026-09-28; updated 2026-10-07. https://porsync.com/research/the-teacher-sets-the-ceiling-slm-distillation

Permanent article URL