Model Distillation Explained

Model distillation trains a small "student" model to mimic a large "teacher," compressing most of the big model's capability into something far faster and cheaper to run. With model distillation explained properly, you'll see how a 671-billion-parameter reasoning model's ability can end up in a 7B model you can self-host—and when that's the right move. This guide covers how distillation works, the crucial difference between soft-label and hard-label methods, real-world examples, when to use it, and the tradeoffs that trip people up.

What model distillation is

Distillation, formally knowledge distillation, was introduced by Geoffrey Hinton and colleagues on a simple premise: a small model can learn more from watching a large model's behavior than from raw training data alone. You start with a capable but expensive teacher model and train a smaller student to reproduce its outputs, ending up with a model that keeps most of the teacher's quality at a fraction of the size, latency, and cost.

The process has two stages. First, the teacher is trained (or you simply use an existing strong model) and then frozen. Second, the student is trained with a special loss that pushes it to match the teacher's predictions. The payoff is what makes distillation so widely used: a student that runs on cheaper hardware, responds faster, and costs less per query—exactly what you need for edge deployment, high-volume inference, or latency-sensitive products—while retaining most of the teacher's skill.

The key insight is that a teacher's output carries more information than a plain label. When a teacher assigns probabilities across many possible answers, those relative confidences—sometimes called "dark knowledge"—tell the student not just the right answer but how the teacher thinks about the problem. Capturing that is what separates distillation from ordinary training.

How it works: soft labels vs hard labels

The single most important distinction in distillation is what the student learns from the teacher, because it determines what access you need and what's even possible.

Hard-label (data) distillation

In hard-label distillation, the teacher acts as a high-quality annotator: you prompt it to generate outputs, then train the student on those outputs with ordinary supervised fine-tuning. The student only sees the final answers, not the teacher's internal confidence. This is the most accessible form because it works through a black-box API—you never need the teacher's internals—and it's effectively the same as generating synthetic training data from a strong model and then preparing that into a dataset. DeepSeek used exactly this approach to transfer reasoning into smaller models.

Soft-label (logit) distillation

In soft-label distillation, the student learns from the teacher's full probability distribution over next tokens—the richer signal. It's implemented by minimizing the KL divergence between the teacher's and student's softened output distributions (using a temperature parameter to sharpen or soften them). This transfers more nuance, but it has hard requirements: you need white-box access to the teacher's logits, and in practice the teacher and student usually must share the same tokenizer and model family. Feature-based variants go further, aligning the student's intermediate layer activations with the teacher's. Soft-label distillation extracts more, but only when you control both models.

White-box vs black-box, and real-world examples

That access question—can you see the teacher's internals or only its outputs?—is the practical fork. A white-box teacher (an open model whose logits you can read) unlocks soft-label and feature-based distillation. A black-box teacher (a commercial model behind an API) limits you to hard-label distillation on its outputs.

The landmark recent example is DeepSeek-R1: researchers took the 671-billion-parameter reasoning model and distilled its chain-of-thought ability into smaller open models—1.5B, 7B, 8B, 14B, 32B, and 70B variants built on Qwen and Llama bases—by fine-tuning them on R1's reasoning traces. The distilled models reason in a similar style at a tiny fraction of the inference cost, and this single result reinvigorated the whole field. The classic earlier example is DistilBERT, which is roughly 40% smaller and 60% faster than BERT while retaining about 97% of its language understanding. Other well-known cases include Alpaca and Orca, both distilled from larger models' outputs, and major platforms now offer distillation as a managed feature, including Amazon Bedrock and OpenAI.

When to use distillation

Reach for distillation when your goal is a smaller, faster, cheaper model rather than a more capable one: deploying to edge or mobile devices, cutting latency, slashing per-query cost at scale, or shrinking a general-purpose model into a specialist that's competitive on your task.

It helps to see how distillation relates to its neighbors:

• Distillation is a form of fine-tuning. Hard-label distillation in particular is just fine-tuning where the training targets come from a teacher model rather than human labels—and you can train that student efficiently with adapters, which is where the LoRA versus full fine-tuning choice comes back in (the DeepSeek-R1 distillations themselves used efficient adaptation techniques).
• Distillation is not quantization. Both shrink a model, but differently: distillation trains a new, smaller model to mimic a big one, while quantization reduces the numerical precision of an existing model's weights. They're complementary—you can distill a model and then quantize the student for even more savings.

If you only need a model to know more facts rather than run smaller, distillation is the wrong tool—that's a retrieval problem, so settle whether to fine-tune or use RAG first.

Tradeoffs and common mistakes

The teacher is the quality ceiling. A student rarely exceeds its teacher on the distilled capability—it's learning to imitate, so it inherits the teacher's limits along with its strengths. Distillation transfers capability; it doesn't create new capability beyond the teacher.

The student inherits the teacher's flaws. Quirks, biases, formatting habits, and outright errors transfer too. Distilled reasoning models, for instance, have been observed to "overthink," producing long, meandering answers inherited from the teacher's style. Evaluate the student on real tasks, not just training loss.

Soft-label distillation has strict prerequisites. It needs white-box logit access and usually a matching tokenizer and model family. If you only have an API teacher, you're doing hard-label distillation whether you meant to or not.

The legal terms matter. Distilling a commercial frontier model's outputs to build a competing model may violate that provider's terms of service—a genuine risk that has drawn real scrutiny. Using open teacher models avoids the question.

Expecting distillation to add knowledge. It compresses a teacher's behavior into a smaller model; it doesn't inject facts the teacher didn't have. For changing knowledge, use retrieval.

Frequently asked questions

What is model distillation in simple terms? It's training a small "student" model to copy a large "teacher" model's behavior, so you get most of the big model's capability in a model that's far smaller, faster, and cheaper to run. The teacher generates the training signal, and the student learns to reproduce it.

What's the difference between soft-label and hard-label distillation? Hard-label distillation trains the student on the teacher's final outputs using ordinary supervised fine-tuning, and works with any black-box teacher through its API. Soft-label distillation trains the student on the teacher's full probability distribution—richer information—but requires white-box access to the teacher's logits and usually a matching model family.

How did DeepSeek use distillation? DeepSeek distilled its large 671B R1 reasoning model into smaller open models (from 1.5B up to 70B, on Qwen and Llama bases) by fine-tuning them on R1's reasoning traces. The smaller models inherited much of the teacher's chain-of-thought reasoning at a fraction of the inference cost.

Is distillation the same as quantization? No. Distillation trains a new, smaller model to imitate a larger one, while quantization reduces the numerical precision of an existing model's weights to shrink it. They solve the size-and-cost problem differently and are often combined—distill first, then quantize the student.

Can a distilled model be better than its teacher? Generally no, at least on the distilled capability—the student learns to mimic the teacher, so the teacher's performance is the practical ceiling, and the student inherits its biases and errors. Distillation is about efficiency, transferring most of the teacher's ability into a much smaller, cheaper model.

The takeaway

With model distillation explained, the core idea is transfer: a small student learns to reproduce a large teacher's behavior, giving you most of the capability at a fraction of the cost. Choose hard-label distillation when working from an API teacher (it's just fine-tuning on the teacher's outputs) and soft-label distillation when you control an open teacher and want the richer signal. Your next step is to decide whether your problem is really about size and cost—if so, pick a strong teacher, generate or extract its outputs, and fine-tune a smaller student, remembering that the student will be only as good, and as flawed, as the teacher it learns from.