Winning Kaggle the Reproducible Way

Chapter 37

Stacking Deep

A stack's score on its own OOF rows is meta-training, not assessment of the whole stack.

A meta-model fitted on out-of-fold predictions is a fitted model with its own degrees of freedom. One demonstration grows the library of base models and compares the meta-model's score on its own rows, a meta-CV on one global OOF matrix and a fully nested assessment, against the equal average, the strongest single model and a regularized meta-model.

Every example is constructed for teaching. Its numbers come from running the chapter activity's code, not from a competition result.

Demonstration 1 of 1

Scoring a stack on its own rows, by library size

How far does a meta-model's score on its own OOF rows drift from an honest assessment as the library of base models grows, and when does a regularized meta-model overtake the unregularized one?

An ordinary least squares meta-model gets one free weight per base model. With 50 correlated columns and 200 fitting rows it can fit noise, so its own-row score keeps falling while the nested score stops improving and then worsens. Meta-CV on one global OOF matrix lands near the nested and fresh-row scores in this generator on average, though the chapter warns it can depend on assessment labels indirectly; the own-row score is far off. A ridge meta-model spends its freedom more slowly and keeps improving with the library.

Equation: Optimism equals the nested mean squared error minus the mean squared error on the meta-model's own rows

Scroll sideways for the whole equation

MSE_own is the mean squared error of a meta-model on the OOF rows it was fitted to, MSE_nested the mean squared error of the same procedure on outer-assessment rows it never saw, and optimism their difference.

Predict first. With 50 base models and an ordinary least squares meta-model, the stack scores 1.27 (mean squared error) on the OOF rows it was fitted to. What does nested assessment give?

Your prediction

Choose an example

Scroll sideways for the whole figure

Figure: Scoring a stack on its own rows, by library size. Left: mean squared error of the least squares stack scored on its own rows (1.66), by meta-CV on one global OOF matrix (2.14) and nested (2.18), with 30 base models; fresh rows give 2.23. Right: own-row and nested errors for the strongest model, equal average, least squares stack and ridge stack.
Number of base models in the library: 30
Constructed example: seeded synthetic regression data and scikit-learn base models, measured by the chapter activity.

Calculated values

Own OOF rows
1.66
Global-matrix meta-CV
2.14
Nested
2.18
Fresh rows
2.23
Equal average, nested
2.74
Ridge stack, nested
2.13

With 30 base models the least squares stack scores 1.66 on its own OOF rows but 2.18 under nested assessment (2.18 - 1.66 = 0.52 of optimism); the global OOF matrix gives 2.14 and fresh rows 2.23. Against the strongest single model the stack gains 1.08 on its own rows but +0.54 nested (2.72 - 2.18 = 0.54). The equal average scores 2.74 and the ridge stack 2.13, which is -0.05 relative to least squares (a negative sign means ridge is better; it was better in 40% of 5 worlds).

Worked steps

  1. Optimism of the least squares stack: 2.18 - 1.66 = 0.52.
  2. Honest gain over the strongest single model: 2.72 - 2.18 = 0.54.
  3. Gain the own-row scores would claim: 2.74 - 1.66 = 1.08.
  4. Nested ridge against nested least squares: 2.13 - 2.18 = -0.05.

Use the idea

Compare any stack with the strongest model and the equal average under the same nested assessment, and prefer a regularized meta-model once the library is large relative to the rows it is fitted on.

Where the conclusion applies

Constructed data with simple scikit-learn base models, not GBM libraries; one fixed ridge strength (alpha 50) and one data size. The library size where ridge overtakes least squares depends on the rows and on how correlated the base models are. The nested estimate is slightly pessimistic because its meta-model trains on 80% of the rows.

Check your understanding: From 30 to 50 base models the least squares stack's own-row score improves while its nested score worsens. What does the regularized meta-model do, and which score would have told you?
Least squares moves from 1.663 to 1.272 on its own rows but from 2.184 to 2.380 nested, because the extra columns are mostly noise to fit. Ridge improves from 2.133 to 2.006 nested over the same range. Only the nested score shows the reversal; the own-row score would have recommended the largest library with the freest meta-model.

Chapter 37 source: section "Distinguish Meta Fitting from Outer Assessment".