Demonstration 1 of 1
Pseudo-label student against a supervised model, with and without teacher leakage, by confidence threshold
Does a stricter confidence threshold make pseudo-labeling pay, and does it remove the leak from a teacher that saw the validation labels?
Raising the threshold makes the kept pseudo-labels more accurate and far fewer, so the student has less new information to learn from. Here the gain is largest at the loosest threshold, where the labels are least accurate, and gone at the strictest. Separately, a teacher trained on the validation fold's labels passes that knowledge to the student through its pseudo-labels, so cross-validation overstates the score at every threshold that keeps a meaningful share of the pool, however accurate the kept labels are.
keep pool row i iff max_k hat p_(ik) ≥ τ
Scroll sideways for the whole equation
The threshold tau keeps a pool row when the teacher's highest class probability is at least tau. Coverage is the share of the pool kept, pseudo-label accuracy the share of kept rows whose teacher label is correct, and the gain is the student's accuracy on fresh rows minus the supervised model's.
Predict first. At threshold 0.9, the kept pseudo-labels are about 98% correct. How much does the student improve on the supervised model on fresh rows?
Choose an example
Scroll sideways for the whole figure
Constructed example: seeded synthetic clusters and extremely randomized trees teacher and student, measured by the chapter activity.
Calculated values
- Pseudo-labels kept
- 35.3%
- Pseudo-label accuracy
- 98.1%
- Student gain (standard error)
- +0.006 (0.003)
- Leaky CV error
- +0.033
- Nested CV error
- +0.003
- Supervised accuracy, fresh
- 0.866
At threshold 0.9 the teacher's labels are kept for 35.3% of the pool and are 98.1% correct. The student scores 0.872 on fresh rows against 0.866 for the supervised model: 0.872 - 0.866 = 0.006 (standard error 0.003). The gain is small, between two and three standard errors, and may not survive another seed. Cross-validation with a teacher that saw all labeled rows reports 0.897 against a fresh score of 0.864, an error of +0.033; rebuilding the teacher inside each fold gives an error of +0.003. The teacher used alone scores 0.875 on fresh rows, so the student is -0.003 against the better of the two supervised models.
Worked steps
- Student gain on fresh rows: 0.872 - 0.866 = +0.006.
- Leaky estimate error: 0.897 - 0.864 = +0.033 (standard error 0.009).
- Nested estimate error: 0.856 - 0.853 = +0.003 (standard error 0.008).
- Pseudo-labels kept: 35.3% of 1000 pool rows, 98.1% correct.
Use the idea
Rebuild the teacher from outer-training labels inside every fold, choose the threshold by the student's score on genuine held-out labels rather than by pseudo-label accuracy, and keep the best supervised-only model, including the teacher itself, as the comparator.
Where the conclusion applies
Constructed clusters, one teacher family and a small labeled set; the pool has the same distribution as the labeled rows, which is the favourable case (no confirmation of a shifted region). The leak matters here because the teacher memorises; a smoother teacher leaks less. The nested pipeline is unbiased by construction (its teacher never sees the scored labels); its remaining error, up to about 0.015 here, is noise from scoring on 100 labeled rows per world, shared across thresholds because the same rows are reused. Gains of 0.01 or less are comparable to their standard errors. The supervised comparator has the student's settings; the memorising teacher on its own scores 0.875 on fresh rows, so against it the student gains only 0.010 at threshold 0.6 and nothing at 0.9.
Check your understanding: At threshold 0.9 the labels are 98% correct, yet the leaky cross-validation still overstates the fresh score. Why, and what fixes it?
Chapter 48 source: section "What Can Go Wrong".