Winning Kaggle the Reproducible Way

Chapter 5

Adversarial Validation

Locate visible separation between populations, then test the target model before acting.

A domain classifier asks whether inputs can tell train from test; it does not ask whether the label rule changed. One demonstration trains the same target model against four test populations and measures domain AUC beside what the target model actually scores, with and without the chapter's density-ratio weights.

Every example is constructed for teaching. Its numbers come from running the chapter activity's code, not from a competition result.

Demonstration 1 of 1

Domain AUC beside the target model's score, for four kinds of shift

When train and test differ, does the domain AUC tell you whether the target model's score has changed?

Domain AUC reads only the inputs. A shifted input distribution and a later timestamp both raise it, yet the label rule can be stable in both; a reversed label rule leaves it at chance. The target model's score on test rows is the only column that answers the question that matters. Weighting by q/(1-q) can help when the shift is overlap-preserving, and can wreck the fit when one feature separates the populations perfectly.

Equation: w i is proportional to the smaller of q i over one minus q i times n train over n test, and c

Scroll sideways for the whole equation

q_i is the out-of-fold probability that training row i looks like a test row, n_train and n_test the row counts, c the weight cap (10) and w_i the weight handed to the target model. Domain AUC is the out-of-fold AUC of the domain classifier; 0.5 means it found no separation.

Predict first. Training and test rows have the same input distribution (domain AUC about 0.50), and cross-validation on training gives AUC 0.771. What does the model score on the test rows when the effect of x2 on the label has reversed?

Your prediction

Choose an example

Scroll sideways for the whole figure

Figure: Domain AUC beside the target model's score, for four kinds of shift. Left: bar of domain AUC 0.499 with a dashed line at the shuffled-label value. Right: bars of the target model's cross-validation AUC 0.771, test AUC 0.576 and weighted test AUC 0.575, for the x2 reverses state.
How the test population differs from training: 3: x2's effect on the label reverses
Constructed example: seeded synthetic rows, a gradient-boosting domain classifier and a logistic target model, measured by the chapter activity.

Calculated values

Domain AUC
0.499
Cross-validation AUC
0.771
Test AUC, unweighted
0.576
Test AUC, weighted
0.575
Weight effective sample
93.3%

Domain AUC is 0.499 (shuffled labels give 0.497). The target model scores 0.576 on test rows against 0.771 in cross-validation: 0.771 - 0.576 = 0.195 below cross-validation. The domain classifier sees nothing, yet the target model's score changed: the label rule moved, not the inputs. Weighting changes the test AUC by less than 0.01.

Worked steps

  1. Domain AUC: 0.499 against 0.497 with shuffled labels.
  2. Test minus cross-validation: 0.576 - 0.771 = -0.195.
  3. Weighting gain on test: 0.575 - 0.576 = -0.001.
  4. Effective sample size of the weights: 93.3% of the 2,000 training rows.

Use the idea

Use domain AUC to find what separates the populations, name the separating feature, and then compare the target model on a held-out set that mimics the test conditions before deciding whether to drop, keep or reweight.

Where the conclusion applies

Constructed data. The logistic target model is deliberately a little misspecified (the true log odds contain an x1 squared term it cannot represent), which is the situation where covariate-shift weighting has something to fix; a flexible model would gain less. The sign-flip state reverses one of two effects, so the test AUC falls to about 0.57, not below 0.5. The weights use the chapter's cap and are not calibrated further.

Check your understanding: A timestamp alone gives domain AUC 1.000. Why does the target model still score about the same on test, and why does weighting hurt?
The target model never uses the timestamp, and the label rule did not change, so it scores 0.776 on test against 0.771 in cross-validation. The domain classifier separates the groups perfectly on a field the target does not need; weighting by that separation gives almost every training row a near-zero weight (effective sample 0.5% of rows), and the weighted model scores 0.703. Reweighting cannot recover behaviour where training has no support.

Chapter 5 source: section "Interpreting the AUC".