Winning Kaggle the Reproducible Way

Chapter 14

What Doesn't Work

A failed addition is a scoped observation: measure it against the unchanged baseline and say what it did not show.

K-means is a common feature idea that the chapter lists among additions that did not help. One demonstration adds a K-means column to boosted trees and measures it against the unchanged baseline and against the true grouping supplied directly.

Every example is constructed for teaching. Its numbers come from running the chapter activity's code, not from a competition result.

Demonstration 1 of 1

A K-means column on boosted trees, against the true grouping supplied directly

When the hidden grouping matters to the label, how much of that value does a K-means column recover for a boosted-tree model?

When the groups overlap, they matter a great deal to the label, but K-means cannot find them, so the column adds nothing and the clusters are unstable. When the groups are well separated, K-means finds them, but boosted trees already separate them from the raw features, so there is little left to add. Between the two, the column recovers part of the grouping and adds a small gain.

Equation: gain equals AUC with the added column minus AUC of the unchanged baseline

Scroll sideways for the whole equation

Gain is the AUC on new rows of the model with the added column minus the AUC of the unchanged baseline; ARI is the adjusted Rand index, 1 for identical groupings and 0 for chance-level agreement.

Predict first. With group centres spread 0.8 apart, supplying the true groups would add about 0.07 AUC. How much does a K-means column add?

Your prediction

Choose an example

Scroll sideways for the whole figure

Figure: A K-means column on boosted trees, against the true grouping supplied directly. Left: AUC gain over the baseline at centre spread 0.8, +0.004 for the K-means column and +0.069 for the true group id, with a dot per dataset. Right: adjusted Rand index for clustering stability 0.62 and recovery of the true groups 0.49.
Spread of the group centres (standard deviations of the within-group noise): 0.8
Constructed example: seeded Gaussian groups, K-means and scikit-learn histogram gradient boosting, measured by the chapter activity.

Calculated values

Baseline AUC
0.768
K-means column gain
+0.004
True group id gain
+0.069
Stability (ARI)
0.62
Recovery (ARI)
0.49

With centre spread 0.8, the true group id adds +0.069 AUC to the baseline of 0.768. The K-means column adds +0.004 (standard error 0.001), which is 7% of it, so 0.069 - 0.004 = 0.065 of the value stays out of reach. Two halves of the training rows agree on the clusters at ARI 0.62, and the clusters match the true groups at 0.49.

Worked steps

  1. Value of the true grouping: 0.837 - 0.768 = 0.069 AUC.
  2. Value of the K-means column: 0.773 - 0.768 = 0.005 AUC (standard error 0.001).
  3. Left out of reach: 0.069 - 0.004 = 0.065 AUC.

Use the idea

Fit scaling and clusters inside training, add the column to the unchanged baseline, and score both on the same assessment. Check that two training subsets give the same clusters before trusting the column, and supply a known grouping directly instead of re-deriving it.

Where the conclusion applies

Constructed Gaussian groups with equal sizes, six clusters chosen to match the true number, one scaling choice (standardization) and a gradient boosting stand-in for LightGBM. Real data rarely has a true number of clusters, and a different base model (for example a linear one) would gain more from cluster columns. The K-means gains are only a few standard errors from zero, so the activity claims only that they stay small.

Check your understanding: Why does the K-means column help least where the grouping matters most, and why does it still add little where K-means finds the groups?
When the centres are close (0.4), the true groups add +0.143 AUC, but the groups overlap so much that K-means recovers almost none of them (adjusted Rand index 0.10) and the column adds -0.001. When the centres are far apart (1.8), K-means recovers them (0.96) but the boosted trees already separate them from the raw features, so even the true group id adds only +0.008. In between, the column recovers part of the grouping and adds at most about 0.01 AUC, well below the true grouping.

Chapter 14 source: section "K-Means Clustering as a Feature".