Winning Kaggle the Reproducible Way

Chapter 40

Knowledge Distillation

Soft teacher targets are one candidate supervision signal: compare them with a tuned hard-label student.

A forest teacher's cross-fitted probabilities train a small neural student. One demonstration compares students fed hard labels, a half-and-half mix and pure soft targets, with and without early stopping on development rows, against the teacher, on genuine test labels, as the training budget grows.

Every example is constructed for teaching. Its numbers come from running the chapter activity's code, not from a competition result.

Demonstration 1 of 1

Hard, mixed and pure soft-target students, by training budget

Do soft teacher targets beat hard labels for a small neural student, or does that depend on how well the hard-label baseline is tuned?

Soft targets are smoother than 0 and 1, so a student trained on them overfits more slowly, which makes it look much better than a hard-label student that is left to train to the end. Early stopping on development rows gives the hard-label student the same protection for free, and the gap disappears. Pure distillation copies cross-fitted teacher predictions, each from a forest that saw only 80% of the rows and their noisy labels, so the student inherits that teacher's errors and adds its own approximation error.

Equation: The student target q i equals alpha times the teacher probability t i plus one minus alpha times the hard label y i, and the loss is the average soft cross-entropy between q i and the student probability p i

Scroll sideways for the whole equation

t is the teacher's cross-fitted probability for a training row, y its (noisy) hard label, alpha the weight on the soft target, q the mixed target the student is trained on and p the student's predicted probability.

Predict first. After 30 epochs with no early stopping, a half-soft student (alpha 0.5) has a log loss of 0.644 against 0.991 for the hard-label student. If both are early-stopped on development rows instead, how do they compare?

Your prediction

Choose an example

Scroll sideways for the whole figure

Figure: Hard, mixed and pure soft-target students, by training budget. Left: test log loss for students trained on hard labels, a half-soft mix and pure soft targets after 30 epochs and when early-stopped on development rows, with the forest teacher at 0.594. Right: the same comparison by AUC.
Training epochs allowed: 30
Constructed example: seeded synthetic data, a random forest teacher and a small neural student, measured by the chapter activity.

Calculated values

Hard labels, last epoch
0.991
Mixed, last epoch
0.644
Hard labels, early-stopped
0.598
Mixed, early-stopped
0.602
Pure soft, early-stopped
0.627
Forest teacher
0.594

With 30 epochs allowed and no early stopping, the hard-label student scores 0.991 and the half-soft student 0.644, a gain of 0.991 - 0.644 = +0.347. Early-stopped on development rows they score 0.598 and 0.602, a gain of 0.598 - 0.602 = -0.004. Pure soft targets score 0.627 early-stopped, and the forest teacher 0.594. The student has 833 parameters against about 4,109 tree nodes. The mixed student beat the hard-label student in 40% of 5 worlds once both were early-stopped.

Worked steps

  1. Apparent gain from soft targets, no early stopping: 0.991 - 0.644 = +0.347.
  2. Gain once both are early-stopped: 0.598 - 0.602 = -0.004.
  3. Pure soft against hard labels, early-stopped: 0.627 - 0.598 = +0.029.
  4. Tuned hard-label student against the teacher: 0.598 - 0.594 = +0.004.

Use the idea

Keep alpha = 0 and early stopping in the comparison, choose alpha and the stopping epoch on development rows, and report the student against the teacher and a tuned hard-label model on genuine labels, with artifact size.

Where the conclusion applies

Constructed data with 20% flipped labels, a random forest standing in for the chapter's GBM teacher and a small scikit-learn network as the student, trained with soft cross-entropy. This is the chapter's soft-target variant, not matched-temperature distillation. Early stopping uses development labels that are as noisy as the training labels, so the stopping epoch is itself noisy. Other noise levels, student sizes and teachers can change the ordering.

Check your understanding: Pure distillation (alpha = 1) stays worse than hard labels after early stopping and behind the teacher. Why, and what does that say about when to distill?
Alpha = 1 trains the student on the teacher's cross-fitted predictions, which were fitted to the noisy labels on 80% of the rows, so the student inherits those errors: 0.627 against 0.598 for hard labels and 0.594 for the teacher. Here the hard-label student (833 parameters) is already within 0.005 of the teacher (about 4,100 tree nodes), so distillation was not needed to compress it. Distill when a tuned hard-label student falls short of the teacher, not by default.

Chapter 40 source: section "GBM-to-NN Soft Label Distillation".