Winning Kaggle the Reproducible Way

Chapter 17

Stacking and Hill-Climbing

Hill-climbing's development score is a search result; judge the candidate library on rows the search never used.

Greedy ensemble search keeps whatever lowers its development loss. One demonstration runs the chapter's hill-climbing function as the library grows from four distinct models to many near-duplicates, and measures what the extra candidates add on development and assessment rows.

Every example is constructed for teaching. Its numbers come from running the chapter activity's code, not from a competition result.

Demonstration 1 of 1

What extra candidates add to a hill-climbing ensemble, on development and assessment rows

As the candidate library fills with near-duplicate models, how much of the hill-climbing improvement shows up on assessment rows?

Every extra near-duplicate gives the search another small, mostly noise-driven way to lower its development loss, so the development improvement keeps growing. The assessment improvement grows far more slowly, because the extras carry almost no new information. The gap between the two is the search's optimism.

Equation: Brier loss equals the mean over rows of the squared difference between predicted probability p i and label y i

Scroll sideways for the whole equation

B_dev and B_assess are the Brier loss on development and assessment rows. The added gain of a library is the Brier loss of the search on the four original models minus the loss of the search on the whole library; positive is better.

Predict first. Growing the library from 4 to 100 near-duplicates lowers the search's development Brier loss by 0.007. How much of that appears on assessment rows?

Your prediction

Choose an example

Scroll sideways for the whole figure

Figure: What extra candidates add to a hill-climbing ensemble, on development and assessment rows. Left: Brier loss on development and assessment rows for the best single candidate, the equal average and hill-climbing with a library of 100. Right: loss gained from the 96 extra candidates, 0.0069 on development rows and 0.0019 on assessment rows.
Candidates in the library: 100
Constructed example: seeded synthetic data, logistic regressions and the chapter's hill-climbing function, measured by the chapter activity.

Calculated values

Candidates
100
Hill-climbing, development
0.196
Hill-climbing, assessment
0.205
Added gain, development
0.0069
Added gain, assessment
0.0019

With 100 candidates the search kept 4.4 on average. Its development Brier loss is 0.196; on assessment rows it is 0.205, so 0.205 - 0.196 = 0.009 of optimism (the equal average, which chooses nothing on the development rows, differs by 0.001). The 96 extra candidates lowered the development loss by 0.0069 and the assessment loss by 0.0019, 28% of it. The best single candidate scores 0.215 on assessment rows and the equal average 0.219.

Worked steps

  1. Optimism of the search: 0.205 - 0.196 = 0.009 Brier (assessment minus development).
  2. Development gain from 96 extra candidates: 0.203 - 0.196 = 0.007.
  3. Assessment gain from the same candidates: 0.207 - 0.205 = 0.002.
  4. Hill-climbing against the best single candidate on assessment rows: 0.215 - 0.205 = 0.010.

Use the idea

Freeze the candidate library and the search recipe, score the frozen ensemble on rows the search never used, and compare it with the best single candidate, equal averaging and the same search on a smaller library.

Where the conclusion applies

Constructed data, logistic regressions standing in for gradient boosting and other model families, and extras that are bootstrap refits of four originals, which is deliberately the near-duplicate case. A library of diverse candidates would show a larger real gain. Assessment loss did not get worse as the library grew here, so no claim is made that adding models must hurt, only that the development score overstates what they add.

Check your understanding: The assessment loss barely improves with 100 candidates. Why does the development loss keep falling, and what should stop the search?
The 96 extra candidates are bootstrap refits of the same four models, so they differ only by noise. Each one gives hill-climbing another chance to match noise in the 200 development rows, and the search kept 4.4 candidates on average to do it. The search should stop when the assessment gain, not the development gain, stops moving: here the 4-model search already earns most of what the 100-model library does.

Chapter 17 source: section "Hill-Climbing Ensemble".