Winning Kaggle the Reproducible Way

Chapter 33

Data Augmentation for Tabular Data

Resample inside the training fold, and compare with weighting and with no resampling.

SMOTE builds synthetic minority rows between real ones. Applied before the split, those rows carry information about validation rows into training. One demonstration measures how far each resampling policy's cross-validation estimate sits from the score on new rows, and what resampling adds at all, by minority prevalence.

Every example is constructed for teaching. Its numbers come from running the chapter activity's code, not from a competition result.

Demonstration 1 of 1

SMOTE before the split against inside the fold, by minority prevalence

How much does resampling before the split inflate cross-validation, and does resampling inside the fold beat weighting or doing nothing?

A synthetic minority row lies on a segment between two real minority rows. If the table is resampled before the folds are drawn, the segments that end at a validation row sit in the training folds, so the model has effectively seen the answer. The rarer the minority, the more synthetic rows are built from each real one and the larger the artefact; once the classes are close to balanced there is little to synthesize and the artefact disappears. Resampling inside the fold removes the leak, and then the estimate describes new rows but resampling itself adds no AP.

Equation: a synthetic row x new equals the minority row x a plus u times the difference between the neighbour x b and x a, with u uniform between zero and one

Scroll sideways for the whole equation

AP is average precision, the area under the precision-recall curve; a random ranking scores the prevalence. A synthetic row is x_new = x_a + u (x_b - x_a), where x_a is a minority row, x_b one of its nearest minority neighbours and u is uniform on [0, 1].

Predict first. With 5% positives, SMOTE before the split reports what cross-validation AP, compared with the AP the final model scores on new rows?

Your prediction

Choose an example

Scroll sideways for the whole figure

Figure: SMOTE before the split against inside the fold, by minority prevalence. Paired bars of cross-validation AP and new-row AP for four resampling policies at 5% positives, with a dashed line at the random-ranking level 0.05; SMOTE before the split reports 0.076 above its new-row score.
Minority class share of the rows: 0.05
Constructed example: eight seeded synthetic imbalanced datasets and a histogram gradient boosting model, measured by the chapter activity.

Calculated values

Pre-split CV AP
0.302
Inside-fold CV AP
0.202
SMOTE, new rows
0.226
No resampling, new rows
0.234
Class weights, new rows
0.246

With 5% positives, SMOTE before the split reports AP 0.302 but the model scores 0.226 on new rows, 0.302 - 0.226 = 0.076 of optimism (scoring the synthetic rows too reports 0.879). Inside the fold the estimate is 0.202 against 0.226 on new rows. Against 0.234 for no resampling, SMOTE inside the fold changes AP by -0.008 and class weights by +0.012.

Worked steps

  1. SMOTE before the split: 0.302 - 0.226 = 0.076 (estimate minus new rows).
  2. SMOTE inside the fold: 0.202 - 0.226 = -0.024.
  3. Value of SMOTE on new rows: 0.226 - 0.234 = -0.008.
  4. Value of class weights on new rows: 0.246 - 0.234 = +0.012.

Use the idea

Put resampling inside the training fold (a pipeline that fits it on training rows only), score untouched validation rows, and always keep the no-resampling baseline and a class-weight arm. If SMOTE does not beat them on the official metric, drop it.

Where the conclusion applies

Constructed data and one gradient boosting model, scored by average precision; the chapter's weighting options are LightGBM and XGBoost parameters, represented here by the class weights of a histogram gradient boosting model. Scoring the synthetic rows too (what a careless pipeline reports) is higher still and is reported beside the main estimate. Three-fold models train on two thirds of the rows, so honest estimates read a little low (0.202 against 0.226 for SMOTE inside the fold at 5% positives). In this generator SMOTE inside the fold did not improve on no resampling; with other data and metrics the result can differ, and the chapter's advice is to test it against weighting.

Check your understanding: Does SMOTE inside the fold improve the score on new rows over no resampling, and when does the pre-split artefact vanish?
No: at 5% positives it scores 0.226 against 0.234 with no resampling, and class weights score 0.246. The pre-split artefact is 0.076 at 5% and 0.021 at 15%, and it is gone at 30% (0.685 against 0.686), where the classes are nearly balanced.

Chapter 33 source: section "SMOTE and Oversampling: Handle With Care".