How a small model learns to do one job as well as a large one, what it costs, and when it is the wrong idea.

Conceptual and procedural guide. Last reviewed August 2026.

https://open.spotify.com/episode/7wii0EMLIvHNZsVdhnp2MB?si=mA_c2hj7SzyRDEzbvgctww

What this covers

Distillation is the technique behind most small models that punch above their weight, and behind a large share of the cost reduction teams achieve after their first year in production with a frontier model. The word covers several different procedures, which is why two engineers can agree that they are “distilling a model” and mean incompatible things.

This article explains the mechanism, the variants, the published evidence, and the decision itself. It is written to be read at three depths:

• If you decide budgets: read “Why teams distil”, “What the numbers say”, “Deciding whether to distil” and “Licences, terms and risk”.
• If you build with models: add “The vocabulary problem”, “Choosing an approach” and “Running a distillation project”.
• If you train models: read all of it, including “Inside the mechanics”.

Prerequisites: familiarity with what a token is, what fine-tuning does, and what an API call to a hosted model costs. No training experience is assumed until the mechanics section, which uses probability notation.

What distillation is

Model distillation trains a small model, the student, to reproduce the behaviour of a large model, the teacher, on a defined task. The teacher supplies the training signal. Instead of learning from human-written labels, the student learns from what the teacher produces: its generated text, its output probabilities, or its judgement of the student’s own attempts.

The result is a model that is cheaper and faster to serve, and that is usually good at one narrow thing rather than at everything. The teacher stays available for the requests the student cannot handle.

Geoffrey Hinton, Oriol Vinyals and Jeff Dean introduced the modern formulation in 2015 for image classifiers. Language models inherited it and added variants that suit sequences of tokens rather than single labels.

Figure 1. The seven stages of a distillation project. The loop matters more than any single stage: without step 5 you cannot tell whether the student is ready.

Not the same thing: distillation trains a new model. Quantisation and pruning compress an existing one without changing what it learned. Retrieval augmentation changes what the model sees at inference time, not what it knows. These are complementary and often stacked, but they solve different problems.

Why teams distil

Four motivations come up repeatedly, and they are not equally common.

Motivation	What it looks like in practice
Unit cost	A classification, extraction or routing task runs millions of times a month on a frontier model. The same task on a 3B or 8B student costs a fraction of that.
Latency	The workload sits in a user-facing path where the frontier model’s time to first token is too slow, or where a long reasoning trace is not worth waiting for.
Locality and control	The model must run inside a controlled perimeter, on your own hardware, without sending data to a third party. That rules out the frontier model and leaves you with a small one that needs to be taught.
Behaviour recovery	A model was fine-tuned on internal documents and lost the instruction-following behaviour it used to have. Distillation from the pre-fine-tuning version restores it.

The fourth is the least known and the most useful to enterprise teams. It is treated in more detail under “What the numbers say”.

The economics only work when the task is narrow. A student trained to match a teacher on invoice extraction will not match it on anything else, and every task you add multiplies the training and evaluation work.

The vocabulary problem

Three distinct procedures travel under the same word. Reading a vendor page or a paper without knowing which one is meant leads to wrong expectations about cost and quality.

Name	What the student sees	Requires
Logit distillation (white-box)	The teacher’s full probability distribution over the vocabulary at every position	Teacher weights, or an API that returns full logits. Same or mapped tokenizer.
Sequence-level distillation (black-box)	Only the text the teacher generated. Standard supervised fine-tuning on that text.	API access. This is what most managed services do.
On-policy distillation	The teacher’s per-token scoring of text the student itself generated	Ability to query teacher log-probabilities on arbitrary token sequences.

A fourth usage is looser: teams say “we distilled GPT into our model” when they mean they fine-tuned on a few thousand teacher answers. That is sequence-level distillation, and calling it by its name makes its limits easier to predict.

Inside the mechanics

This section assumes you want to know why the methods differ, not only that they do.

Soft targets and temperature

A one-hot training label says one token was right and all others were equally wrong. The teacher’s distribution says more than that. It ranks the alternatives, and the ranking carries information about the structure of the task.

Figure 2. The same prediction under three supervision regimes. The one-hot label discards the information that “position” was a close second and “door” was not.

Hinton’s paper calls these runner-up probabilities dark knowledge. To make them usable you raise the softmax temperature, which flattens the distribution and pulls the informative tail up out of the numerical floor. Both teacher and student are softened by the same temperature during training. The student is served at temperature 1.

The gradients produced by a softened distribution scale roughly as 1/T squared, so the distillation term is multiplied by T squared to keep its weight stable when you change the temperature.

import torch, torch.nn.functional as Fn

def kd_loss(student_logits, teacher_logits, labels, T=2.0, alpha=0.7):
    """Classic offline KD: soft teacher targets + the hard label."""
    soft_t = Fn.softmax(teacher_logits / T, dim=-1)
    soft_s = Fn.log_softmax(student_logits / T, dim=-1)
    # forward KL(teacher || student), scaled to keep gradients comparable across T
    kd = Fn.kl_div(soft_s, soft_t, reduction="batchmean") * (T * T)
    ce = Fn.cross_entropy(student_logits, labels)
    return alpha * kd + (1.0 - alpha) * ce

Two properties are worth remembering. The loss is zero when the two distributions are identical, and it stays comparable in magnitude as you vary T, which is the point of the T squared factor. In a language model, this is applied per position over the sequence.

Which divergence, and why it matters

The choice of divergence decides what the student does when it cannot match the teacher everywhere, and small models frequently cannot.

Divergence	Behaviour	Consequence
Forward KL, teacher relative to student	Mode covering. The student spreads probability mass over everything the teacher might say.	A student with too little capacity hedges, producing bland or averaged output.
Reverse KL, student relative to teacher	Mode seeking. The student commits to one of the teacher’s behaviours.	Sharper, more decisive output. This is what on-policy methods optimise.

Reverse KL has a practical advantage as a reward signal: it cannot be gamed. A low value always means the teacher would have been comfortable with what the student said, which is not true of a learned reward model.

Who writes the training text

The second axis is more consequential than the loss function. Off-policy means the student trains on text the teacher wrote. On-policy means it trains on text it wrote itself, with the teacher scoring it.

Figure 3. The two axes that separate the methods. Reinforcement learning shares the on-policy column with on-policy distillation, but gives one number per rollout instead of one per token.

Off-policy training has a failure mode that gets worse as outputs get longer. The student learns to continue from states the teacher visits. At inference it will occasionally produce a token the teacher never would, and from that point it is in territory no training example covered. The error compounds.

Figure 4. Left: the compounding error problem, sometimes called exposure bias. Right: on-policy grading, where every token the student produced receives a score from the teacher.

There is a second, subtler problem with imitating text alone. Research on imitating proprietary models found that students learn the teacher’s style and confidence faster than its factual accuracy, which flatters human evaluations while leaving benchmark performance behind.

On-policy distillation addresses both. The student samples its own trajectories, the frozen teacher computes log-probabilities over those exact tokens, and the per-token reverse KL becomes the training signal. Credit lands on the token that started the wrong reasoning branch rather than on the final answer, which is often perfectly predictable given everything that came before it.

# On-policy distillation, adapted from the Tinker cookbook recipe
teacher = service.create_sampling_client(base_model=TEACHER)

for step in range(n_steps):
    # 1. the STUDENT generates; this is what makes it on-policy
    traj = do_group_rollout(student_client, prompts)
    student_lp = traj.loss_fn_inputs["logprobs"]

    # 2. the frozen teacher scores those same tokens (one forward pass)
    teacher_lp = teacher.compute_logprobs(traj)

    # 3. per-token reverse KL becomes a per-token (negative) advantage
    traj["advantages"] = -(student_lp - teacher_lp)

    # 4. ordinary policy-gradient update
    training_client.forward_backward(traj, loss_fn="importance_sampling")

Three things follow from this shape. You never need a separate reward model or a human labeller. You never need to wait for a rollout to finish, since the score exists for partial sequences, so training can run at shorter context than evaluation. And on an existing reinforcement learning stack that already regularises against a reference model, the change is close to swapping which model plays that role.

Note: the per-token reverse KL of a single sampled token can be negative. It is the expectation over the student’s own distribution that is non-negative, and that is what the update optimises.

What the numbers say

The clearest published comparison comes from the Qwen3 technical report, where the same initialisation was pushed further by two different methods.

Figure 5. Accuracy and cost for the final post-training stage. The distillation stage reached a higher score than the reinforcement learning stage for about a tenth of the GPU hours.

Thinking Machines Lab reproduced the pattern independently. Starting from a student fine-tuned on 400,000 teacher-generated prompts, which scored 60 percent on AIME 2024, on-policy distillation reached 70 percent in roughly 150 steps. Reaching the same score by continuing supervised fine-tuning was estimated to need about 2 million prompts. Counted in FLOPs, the reduction was 9 to 30 times depending on whether you already own the fine-tuning dataset.

A separate self-distillation experiment isolated the effect of feedback density. A model was trained with reinforcement learning, then that trained model was distilled back into the original base model. The distillation run recovered the same score in 7 to 10 times fewer gradient steps.

The case most enterprises will recognise

A more relevant experiment for internal deployments: an 8B assistant was mid-trained on internal company documents to teach it domain knowledge. Its knowledge score rose from 18 to 43 percent. Its instruction-following score collapsed from 85 to 45 percent. Mixing 30 percent general chat data into the training set limited the damage but never removed it.

On-policy distillation from the model’s own earlier version, using only general chat prompts and nothing from the document set, restored instruction following to 83 percent while knowledge stayed at 41 percent. The document training and the behaviour repair are separable, which makes a practical loop possible: teach new knowledge, then repair behaviour, then repeat as the documents change.

Stage	Internal knowledge	Instruction following
Original 8B model	18%	85%
After training on internal documents only	43%	45%
With 30% general chat data mixed in	36%	79%
Plus on-policy distillation from the original	41%	83%

Vendor figures

Managed services publish their own numbers, which describe sequence-level distillation and should be read as vendor claims rather than independent results. AWS states that distilled models on Bedrock run up to 500 percent faster and cost 75 percent less than the original, with under 2 percent accuracy loss on retrieval-augmented use cases. Treat those as the shape of the opportunity and measure your own task.

On the research side, black-box methods keep improving. Microsoft Research reported that a 14B student trained with a generative-adversarial variant of on-policy distillation, using only teacher text, became comparable to its much larger teacher on an automatic chat evaluation.

Choosing an approach

Work down this table until a row matches your access and your constraints.

Situation	Method	Practical note
Teacher is an open-weight model you can host	Logit or on-policy distillation	Best quality per unit of compute. Tokenizer must match, or be mapped.
Teacher is a commercial API, task is narrow and stable	Sequence-level, via a managed distillation service	Lowest effort. Check the terms before you start.
Student must recover behaviour lost to fine-tuning	On-policy distillation from an earlier checkpoint of the same model	The teacher here is your own past model, which removes all licence questions.
You need reasoning, not just format compliance	Off-policy warm start, then on-policy	The warm start puts the teacher’s behaviour inside the student’s reach; the on-policy phase makes it reliable.
You have prompts but no correct answers	Rejection sampling, then distillation	Generate many teacher answers, keep those a checker accepts, train on those.

Deciding whether to distil

Most workloads that look like distillation candidates are not. Run the four checks before writing any training code.

Figure 6. Four checks in order. Failing any of them has a cheaper answer than training a model.

The eval is the check teams skip, and it is the one that decides whether the project can finish. Without a scored evaluation set you cannot answer the only question that matters at the end: is the student close enough to the teacher to take over?

Running a distillation project

Before starting, you need: a defined task with a stable output format, at least several hundred representative prompts and preferably a few thousand, a scored evaluation set that is disjoint from the training prompts, a chosen student model, and written confirmation that the teacher’s terms allow what you are about to do.

1. Freeze the task. Write down the input format, the output format, and the acceptance criteria. Anything still in flux will invalidate the training run.
2. Build the evaluation set first. Take 200 to 500 real cases, get the teacher’s answers, have a human confirm them, and keep this set out of training.
3. Measure the teacher on that set. This is your ceiling and your budget baseline in cost per thousand calls and p95 latency.
4. Collect prompts. Production logs are better than synthetic prompts because they carry the real distribution, including the malformed inputs.
5. Generate the training data. Sample teacher answers at a low temperature for deterministic tasks, higher for tasks where diversity helps. Keep the teacher version string with the data.
6. Train the student. Start with supervised fine-tuning on the teacher outputs. LoRA is usually enough for narrow tasks and cheaper to iterate on.
7. Evaluate against the frozen set. Compare with the teacher, not with an abstract target. Report the gap per category, not only the average.
8. If the gap is too large, add supervision rather than data. More prompts follow a log-linear curve with diminishing returns; per-token supervision on the student’s own outputs is the step change.
9. Deploy behind a router. Send the workload to the student, keep the teacher for low-confidence or out-of-distribution requests, and log both.
10. Re-evaluate on a schedule. Input distributions drift, and a student trained on last quarter’s traffic degrades quietly.

Managed paths

If you do not want to own the training loop, three services cover the sequence-level case.

• Amazon Bedrock Model Distillation. You supply prompts, it generates teacher responses, synthesises additional data, fine-tunes the student and hosts the result.
• OpenAI stored completions. Set store true on production calls, filter the stored pairs in the dashboard, then start a fine-tuning job from the selection.
• Azure AI Foundry stored completions. The same pattern inside Azure. It accepts as few as ten stored completions, though hundreds to thousands are recommended.

The trade is control. You cannot export the training file in some of these workflows, and you cannot use the teacher to distil into a model the provider does not host.

When the student underperforms

Symptom	Likely cause	What to do
Good on the eval set, poor in production	Training prompts came from a different distribution than real traffic	Rebuild the prompt set from production logs, including malformed inputs
Fluent and confident but factually worse	The student imitated style faster than substance, a known effect of text-only imitation	Move to per-token supervision, or add a verifiable checker to the training loop
Degrades on long outputs only	Compounding error from off-policy training	Switch the final phase to on-policy distillation
Lost general ability after training on internal data	Catastrophic forgetting during mid-training	Distil from the pre-training checkpoint on general prompts to restore behaviour
Output is bland or hedged	Forward KL with insufficient student capacity, so the model covers every mode	Use reverse KL, or accept that the student is too small for the task
Quality drops after a teacher upgrade	The student was frozen against an older teacher	Pin the teacher version, re-run evaluation on every teacher change

Licences, terms and risk

The technique is standard machine learning. The exposure comes from where you point it.

Copyright is a weak theory in most jurisdictions, because model outputs typically lack the human authorship that copyright requires. The live questions are contractual and, increasingly, trade secret. Most frontier providers prohibit using their outputs to train competing models, and some prohibit training any model on their outputs without permission.

Two disputes made this concrete. OpenAI accused DeepSeek of using its models to train a competitor in violation of its terms of use. Anthropic later said it had identified large-scale campaigns by DeepSeek, Moonshot and MiniMax to extract Claude’s capabilities through thousands of fraudulent accounts, while acknowledging that distillation itself is a legitimate training method. Neither case turned on whether distillation is allowed as a technique.

Practical positions, from safest to riskiest:

• Your own model as teacher. No third-party terms involved. This covers the behaviour-recovery case entirely.
• An open-weight teacher. Read the licence: several community licences permit distillation but require you to name the origin model in the derived model’s name and attribution.
• A commercial API through the provider’s own distillation product. Permitted by construction, within that provider’s ecosystem.
• A commercial API used to generate training data for an internal, non-competing model. This is the grey zone. Get the provider’s terms reviewed before you start, not after.
• A commercial API used to train something that competes with the provider. Prohibited by essentially every set of terms.

For a regulated environment, add documentation to the deliverable: teacher model and version, prompt provenance, generation parameters, evaluation results, and the licence review. A distilled model is a derived asset and its lineage belongs in your model inventory.

When not to distil

• The task is broad. General assistants need general capability, and a small student will lose the long tail you did not train on.
• The requirements change monthly. Every change means a new training run and a new evaluation cycle.
• The problem is missing knowledge rather than missing behaviour. Retrieval is cheaper and easier to keep current.
• Volume is low. Under a few hundred thousand calls a month, the engineering time will usually cost more than the inference it saves.
• You have no evaluation set and no appetite to build one. You will not be able to tell whether the student is good enough.

Next steps

If distillation fits your workload, the fastest useful experiment is small: pick one high-volume task, build the 200-case evaluation set, and run a sequence-level distillation through a managed service. That answers the economic question in days. The on-policy work is worth the extra effort only once you know the size of the prize.

Further reading

Hinton, Vinyals and Dean, Distilling the Knowledge in a Neural Network (2015). The origin of soft targets and temperature scaling.

Kim and Rush, Sequence-Level Knowledge Distillation (2016). The adaptation to sequence models.

Agarwal et al., On-Policy Distillation of Language Models (2023), and Gu et al., MiniLLM (2023). The on-policy and reverse-KL foundations.

Gudibande et al., The False Promise of Imitating Proprietary LLMs (2023). Why style transfers faster than accuracy.

Thinking Machines Lab, On-Policy Distillation (2025), with runnable recipes in the Tinker cookbook.

Qwen Team, Qwen3 Technical Report (2025). Source of the accuracy and GPU-hour comparison.

AWS, Customize a model with distillation in Amazon Bedrock, and OpenAI, Model Distillation in the API. The managed workflows.

Fenwick, DeepSeek, Model Distillation, and the Future of AI IP Protection. A legal overview of the contractual and trade secret arguments.