Distilling means using a large model to generate examples of how your task is solved, and using those examples to train a small model that does it just as well. The small one does not know everything, it knows your thing, and that is enough when the task is always the same.
It is the slowest lever to apply and the one that cuts the unit cost the most. That is why it only pays off when volume is high and stable.
At a glance
| What it does | Trains a small model on the large one’s answers |
| Typical saving | Very high per request, once it is running |
| Entry cost | Generating data, training, evaluating and maintaining |
| When it pays off | A single, repeated task with stable volume |
When it makes sense
When one task concentrates a good share of your spend and has not changed shape in months. Classifying tickets, extracting fields from one document type, moderating content: narrow tasks, heavily repeated, with stable criteria.
It makes no sense if your traffic is a hundred different tasks, if the definition of the task changes every month, or if the volume is so low that the large model costs you less than the work of setting up the distillation.
What it really costs
The training bill is usually the smallest part. The expensive part is the rest: generating and cleaning the examples, having an evaluation that says whether the small model matches the large one, and above all the upkeep. A distilled model is an asset of your own, with its own life cycle: when the task changes, you distill again.
Before committing, check that the cheap lever is not enough. Often picking the model per task with a small off-the-shelf model already delivers 80% of the saving without training anything.
Distillation versus the alternatives
Against classic fine-tuning, the difference is where the data comes from: in distillation the large model generates it, so you do not depend on having a labeled history. Against putting examples in the prompt, you win because those examples stop travelling on every request: the knowledge sits in the weights, not in the context. It is the same axis discussed in RAG, fine-tuning or context.
Put a number on it before you start
Work out three figures: what you spend today per month on that task, what you would spend with the small model, and the hours of work to build it and maintain it for a year. If the annual saving does not cover that effort comfortably, the answer is no. You can do the first part with the token cost calculator.
Frequently asked questions
Can I distill any model? Not always: some providers’ terms restrict using their outputs to train competing models. Worth reading them first.
How much quality is lost? On narrow tasks, little or none: the small model can match the large one inside its domain. Outside it, it collapses, which is exactly why you need your own evaluation.
What if the task changes? You distill again. That is the hidden cost almost nobody mentions and the reason it only pays off on stable tasks.
At e-ficient we measure the cost per task of companies that already run AI in production and tell you how much you would save with each lever at your real volume. The first audit is free if you start a plan with a three-month commitment and returns a diagnosis within 72 hours.