Winning Kaggle the Reproducible Way

Chapter 29

Advanced Feature Selection

Compare each importance with its own label-permutation null before screening columns out.

Raw importance favours columns with many distinct values, and 'drop the bottom N' is arbitrary. One demonstration compares a raw importance ranking with a null-importance screen on a table where the truth is known, across null percentiles, and scores the selected columns on new rows.

Every example is constructed for teaching. Its numbers come from running the chapter activity's code, not from a competition result.

Demonstration 1 of 1

Raw importance against null importance, by null percentile

Which columns does each screen keep, and at what null percentile does a strict screen start discarding real signal?

A column with many distinct values offers a tree many places to split, so it earns importance even when it carries nothing: here the noise and ID-like columns outrank the weak binary features that really matter. Shuffling the labels measures what each column earns by chance, so comparing a column with its own null removes that advantage. Raising the percentile removes more chance winners but starts to cut weak real columns.

keep j iff I_j > N_j^((q))

Scroll sideways for the whole equation

I_j is column j's importance on the real labels, N_j^(q) the q-th percentile of its importance over the label permutations, and a column is kept when I_j exceeds N_j^(q). The percentile rule is a screening heuristic, not a probability that the column is real.

Predict first. A raw-importance ranking and a 75th-percentile null screen each keep the same number of columns. Which table scores higher on new rows?

Your prediction

Choose an example

Scroll sideways for the whole figure

Figure: Raw importance against null importance, by null percentile. Left: real impurity importance of 25 columns in one dataset with a null-threshold tick for each, filled when kept at the 75th percentile. Right: AUC on new rows for all columns (0.767), a raw top-k ranking (0.759) and the null screen (0.786).
Null percentile a column's importance must exceed: 75
Constructed example: six seeded synthetic datasets, a random forest and twelve label permutations, measured by the chapter activity.

Calculated values

Columns kept
5.8
Weak real kept (of 4)
2.7
Noise kept (of 18)
0.2
Null screen AUC
0.786
Raw top-k AUC
0.759
All columns AUC
0.767

At the 75th null percentile the screen keeps 5.8 columns on average: 3.0 of 3 strong, 2.7 of 4 weak, 0.2 of 18 noise. The same number chosen by raw importance rank holds 3.0 of 3 strong, 0.0 of 4 weak, 2.8 of 18 noise. Scored on new rows the null screen reaches 0.786, the raw top columns 0.759 and the full table 0.767, so 0.786 - 0.767 = 0.019 is the change from screening with the null (negative means it cost AUC).

Worked steps

  1. Null screen: 0.786 - 0.767 = 0.019 AUC against keeping every column.
  2. Raw ranking of the same size: 0.759 - 0.767 = -0.008.
  3. Kept by the null screen: 3.0 of 3 strong, 2.7 of 4 weak, 0.2 of 18 noise.
  4. Kept by the raw ranking: 3.0 of 3 strong, 0.0 of 4 weak, 2.8 of 18 noise.

Use the idea

Fit the screen on development rows only, compare each column with its own permutation null, choose the percentile by held-out loss rather than by habit, and assess the selected table separately. Keep a raw-rank selection of the same size as the comparison.

Where the conclusion applies

Constructed data and one forest as the importance model; the chapter's BorutaShap and LightGBM are not installed. With only 12 permutations the 99th percentile is close to the largest null value. In this generator the real labels also shrink the importance of the noise columns below their null, so the null screen is conservative; differences in AUC between percentiles are small and are properties of this generator.

Check your understanding: The 99th percentile keeps fewer noise columns than the median. Why does it score lower on new rows?
At the 50th percentile 0.7 noise columns survive on average along with 3.3 of the 4 weak real columns, and the screen scores 0.792; at the 99th the weak real columns fall to 2.0 kept and the score drops to 0.780. A stricter rule removes chance winners and real but weak columns together, and here the weak columns were worth more than the noise cost, so the percentile is a choice to validate, not a constant.

Chapter 29 source: section "Null Importance: Distinguishing Real Signal from Chance Correlation".