Winning Kaggle the Reproducible Way

Chapter 10

Interactions, Groups, and Text

A stepwise interaction search produces a development score, not an assessed gain.

The chapter's stepwise search keeps any product that raises its own cross-validation score. One demonstration measures what that score claims, and what the selected recipe earns on fresh rows, as more pairs are screened.

Every example is constructed for teaching. Its numbers come from running the chapter activity's code, not from a competition result.

Demonstration 1 of 1

A stepwise interaction search, scored on its own folds and on fresh rows

As more candidate pairs are screened, does the search score keep describing what the selected features earn on new rows?

The true pair is easy to find and earns a real gain. Every other pair is noise, but some noise products raise the development AUC by chance, and the greedy rule keeps each of them. The more pairs are screened, the more of these chance gains are kept, so the search score keeps climbing while the fitted noise columns dilute the model on rows that were not used for the search.

keep c = x_a x_b iff S(F cup c) - S(F) > ε

Scroll sideways for the whole equation

S(F) is the development AUC (mean over the search folds) of a model using feature set F, c = x_a * x_b is a candidate product, and epsilon is the minimum gain (0.00001). A pair is kept when S(F plus c) - S(F) exceeds epsilon.

Predict first. When all 66 pairs of 12 features are screened, how does the gain the search reports compare with the gain on fresh rows?

Your prediction

Choose an example

Scroll sideways for the whole figure

Figure: A stepwise interaction search, scored on its own folds and on fresh rows. Left: development AUC and fresh-row AUC after each pair the search accepted on the first dataset, with 66 pairs screened. Right: mean AUC gain by search score (0.071), fresh rows (0.044) and the true pair alone (0.059).
Top features whose pairwise products are screened: 12 features (66 pairs)
Constructed example: seeded synthetic data, one built-in interaction and a logistic regression, measured by the chapter activity.

Calculated values

Pairs screened
66
Search score gain
+0.071
Gain on fresh rows
+0.044
True pair only
+0.059
Noise pairs kept
14.3

Screening 66 pairs, the search reports a gain of 0.071 AUC and the same recipe earns 0.044 on fresh rows, so 0.071 - 0.044 = 0.027 of the reported gain is selection bias (standard error 0.008 over 6 datasets). It kept 14.3 pairs other than the true one on average and found the true pair in 100% of datasets. Adding only the true pair earns 0.059, so the noise pairs cost 0.015 on fresh rows.

Worked steps

  1. Search gain minus fresh gain: 0.071 - 0.044 = 0.027 AUC.
  2. Fresh gain minus true-pair-only gain: 0.044 - 0.059 = -0.015 AUC.
  3. Pairs screened: 66; pairs other than the true one kept: 14.3 on average.

Use the idea

Treat the search score as a development result. Reserve a partition before the search, or rerun the whole search (including the top-feature screen and the stopping rule) inside each outer training fold, and report the gain of the frozen recipe there.

Where the conclusion applies

Constructed data, one real interaction that the search finds every time, and a logistic regression in place of the chapter's LightGBM. With a weak or absent true pair the picture is worse, because nothing real offsets the noise pairs. A fresh 20,000-row sample from the generator stands in for a reserved assessment partition, which in a competition would be far smaller and noisier.

Check your understanding: The true interaction is found at every screen size. Why does screening more pairs still make the selected recipe worse on fresh rows?
The extra pairs are noise, but the greedy rule keeps any product that raises its own folds by more than 0.00001. At 66 pairs the search kept 14.3 pairs other than the true one on average against 1.7 at 6 pairs, each one a fitted column that describes chance in the 1,200 search rows. The search score cannot show that cost, because it is computed on the rows that chose them.

Chapter 10 source: section "Interaction Features: The Stepwise Approach".