Winning Kaggle the Reproducible Way

Chapter 1

What Actually Wins on Kaggle

Match the representation to the task, then test the complete pipeline.

Familiar-subject rows and new-subject rows share columns but need different assessment. One demonstration measures what a model upgrade does to a random-fold estimate, a subject-held-out estimate and the score on subjects the model never saw.

Every example is constructed for teaching. Its numbers come from running the chapter activity's code, not from a competition result.

Demonstration 1 of 1

A model upgrade, scored three ways

When the model gets more capacity on the same columns, which assessment rewards the upgrade, and which one tracks the score on subjects the model never saw?

Because rows from the same subject sit on both sides of a random split, a flexible model can use the fingerprint channels to look up each subject's bias, and the random-fold score rewards that. New subjects have fingerprints and biases it has never seen, so the same lookup is noise there. Subject-held-out folds remove the lookup from the estimate. Move the control to see where the random-fold ranking and the new-subject ranking of the models disagree.

Equation: error equals the AUC from a fold estimate minus the AUC on new subjects

Scroll sideways for the whole equation

AUC_random, AUC_subject and AUC_new are the pooled AUCs from random folds, subject-held-out folds and new subjects; the estimate error is a fold estimate minus the new-subject score, so a positive value is optimism.

Predict first. Upgrade from trees with 2 leaves to trees with 31 leaves on the same columns. What happens to the random-fold estimate and to the score on new subjects?

Your prediction

Choose an example

Scroll sideways for the whole figure

Figure: A model upgrade, scored three ways. Bars of AUC from random folds, subject-held-out folds and new subjects for trees with 31 leaves, with a dot for each of 8 studies and a dashed line at the new-subject score of 0.585; random folds are +0.149 from it.
Model capacity (leaves per tree): 31: deep trees
Constructed example: eight seeded synthetic subject studies and boosted trees, measured by the chapter activity.

Calculated values

Random-fold AUC
0.735
Subject-fold AUC
0.576
New-subject AUC
0.585
Random-fold optimism
+0.149
Upgrade effect, new subjects
-0.040

With 31 leaves per tree, random folds report 0.735 and the score on new subjects is 0.585, so 0.735 - 0.585 = 0.149 of optimism. Subject-held-out folds report 0.576, -0.009 from the new-subject score. Against the 2-leaf model the upgrade moves random folds by +0.027, subject folds by -0.008 and new subjects by -0.040. The upgrade looks like progress in random folds and is a loss on new subjects.

Worked steps

  1. Random folds: 0.735 - 0.585 = +0.149 (estimate minus new subjects).
  2. Subject-held-out folds: 0.576 - 0.585 = -0.009.
  3. Upgrade effect on random folds: 0.735 - 0.707 = +0.027.
  4. Upgrade effect on new subjects: 0.585 - 0.625 = -0.040.

Use the idea

Before choosing between models, write down what a new row will be (a new subject, a later date or a familiar entity) and split development data to match. Judge each upgrade by the matching estimate, not by the one that is cheapest to run.

Where the conclusion applies

Constructed data and a boosted-tree model from scikit-learn standing in for a gradient-boosting library. The fingerprint channels are built to be useless for new subjects, which is the chapter's familiar-subject case. In this generator the simplest model scores highest on new subjects; with different data a larger model can help there too, and the lesson is that only a matching split can tell. The subject-fold estimate is noisy and sits below the new-subject score (by 0.041 at 2 leaves): each fold model trains on fewer subjects, and the pooled predictions mix five fold models whose overall level differs.

Check your understanding: Which of the three assessments would have told you not to adopt the bigger model, and what does that say about where to spend effort first?
The subject-held-out folds did not reward it: they changed by -0.008 at 31 leaves, within the noise of eight studies, while random folds moved by +0.027 and the new-subject score by -0.040. Random folds sit 0.149 above the new-subject score and subject folds 0.009 below it. The model change could not resolve a split that does not match the test; fixing the split came first.

Chapter 1 source: section "Start with the Prediction Contract".