Winning Kaggle the Reproducible Way

Chapter 2

The Process Mindset

Preserve the comparison, and treat repeatedly inspected feedback as development evidence.

Every logged version that is chosen by the public score uses that score for development. One demonstration measures how much of the public gain of the winning version survives on private rows and on fresh rows, as the log grows.

Every example is constructed for teaching. Its numbers come from running the chapter activity's code, not from a competition result.

Demonstration 1 of 1

Picking the best of N logged versions by the public sample

How much of the winning version's public gain is still there on rows the choice never used, as the number of logged versions grows?

Each extra version is another noisy comparison against the baseline on the same 400 public rows. The best of many such comparisons is high even when no version helps, so the public gain keeps rising with N. The private rows played no part in the choice, so they show what is left, and the fresh rows show the truth the private rows estimate. Move the control to find where the public and private gains separate.

Equation: gain g equals the baseline loss minus the winner's loss, where the winner is the version with the lowest public loss among N

Scroll sideways for the whole equation

g_public, g_private and g_fresh are the baseline's log loss minus the winning version's log loss on public rows, on the remaining private rows and on 20,000 fresh rows; the winner is chosen on the public rows only.

Predict first. Thirty versions are logged and the best one on the 400 public rows is kept. How does its private gain compare with its public gain?

Your prediction

Choose an example

Scroll sideways for the whole figure

Figure: Picking the best of N logged versions by the public sample. Lines of public, private and fresh-row gain against the number of logged versions, with the 30-version point marked: public 0.0045, private 0.0008, fresh 0.0010.
Versions logged (including the baseline): 30
Constructed example: sixteen seeded synthetic competitions and logistic regressions, measured by the chapter activity.

Calculated values

Public gain
0.0045
Private gain
0.0008
Fresh-row gain
0.0010
Public minus private
+0.0037
Picks truly better
34%

With 30 versions logged, the winner's public gain is 0.0045 and its private gain is 0.0008, so 0.0045 - 0.0008 = 0.0037 of the public gain is not there on private rows. On 20,000 fresh rows the winner gains +0.0010. Most of the public gain is the luck of picking the best of many comparisons. The winner truly beats the baseline in 34% of the picks.

Worked steps

  1. Overstatement: 0.0045 - 0.0008 = +0.0037 (public gain minus private gain).
  2. Private gain against the truth: 0.0008 - 0.0010 = -0.0002.
  3. Public gain against the truth: 0.0045 - 0.0010 = +0.0035.
  4. Picks that truly beat the baseline: 34% of 160 public-split picks.

Use the idea

Log every version, but decide with data the choice has not touched. When the log is long, discount the public gain of the best version, and keep a private comparison or a reserved assessment that did not take part in picking it.

Where the conclusion applies

Constructed competitions with a logistic regression, one added column per version, and 2 truly useful candidates among 100. With every candidate useless, the private gain is negative at every N (checked by the build with the two useful candidates removed). The gain sizes are properties of this generator.

Check your understanding: At 10 logged versions the winner improved the public loss by 0.0030 yet gained 0.0001 on private rows. What did the public gain measure?
Mostly the best of ten noisy comparisons: the winner's public gain rose from 0.0007 at 3 versions to 0.0083 at 100, while the private gain stayed near zero until the log was long enough to contain a truly useful column (0.0032 at 100 against 0.0036 on fresh rows). The public gain overstates by 0.0051 at 100 versions.

Chapter 2 source: section "Preserve the Comparison".