Winning Kaggle the Reproducible Way

Chapter 39

Imbalanced Data

Downsampling negatives is cheap, but it changes the probabilities; fix them and check the fix.

Keeping every positive and a fraction of the negatives cuts training cost and leaves ranking almost intact, but the raw probabilities describe the sampled data. One demonstration measures the damage to log loss and compares the chapter's analytic correction with a calibrator fitted on development rows, for a gradient-boosted model and a logistic model.

Every example is constructed for teaching. Its numbers come from running the chapter activity's code, not from a competition result.

Demonstration 1 of 1

Downsampling negatives: raw, corrected and calibrated probabilities

How much log loss does negative downsampling cost, and at what keep rate does the analytic correction stop being as good as calibrating on development rows?

The correction follows from odds: keeping a fraction r of negatives multiplies the sampled odds by 1/r, so the natural odds are the sampled odds times r. It assumes the model is calibrated on the sampled data. A logistic regression stays calibrated there almost regardless of the sample size; a boosted model fitted on a few hundred rows does not, and the correction then inherits its overconfidence. A calibrator fitted on natural-prevalence development rows does not depend on that assumption.

Equation: p true equals p times r divided by p times r plus one minus p

Scroll sideways for the whole equation

p is the probability from the model trained on the sample, r the fraction of negatives kept (all positives are kept) and p_true the corrected probability for the natural population.

Predict first. Every positive and 2% of the negatives are kept for training. Scored on natural-prevalence rows, how does the raw gradient-boosted model's log loss compare with the same model trained on all rows (0.056)?

Your prediction

Choose an example

Scroll sideways for the whole figure

Figure: Downsampling negatives: raw, corrected and calibrated probabilities. Left: test log loss of the boosted model's raw, corrected and development-calibrated probabilities when 2% of negatives are kept, on a log scale; raw is 0.338 and the all-rows reference is 0.056. Right: log loss above the all-rows reference for corrected and calibrated boosted output and corrected logistic output.
Share of negatives kept for training: 2%
Constructed example: seeded synthetic data, a boosted model and a logistic regression, measured by the chapter activity.

Calculated values

Training rows
1,302
Raw log loss
0.338
Corrected log loss
0.056
Calibrated log loss
0.055
Mean corrected prediction
0.013 vs prevalence 0.013
AUC
0.853

Keeping 2% of negatives leaves 1,302 training rows. Raw boosted-model probabilities average 0.230 against a true prevalence of 0.013, so log loss is 0.338 against 0.056 when trained on all rows (0.338 / 0.056 = 6.0 times). The analytic correction brings it to 0.0556 (0.0556 - 0.0563 = -0.0007 against the all-rows model, mean prediction 0.013); calibrating on development rows gives 0.0554. The logistic model's corrected log loss is +0.0001 from its own all-rows reference. AUC moves +0.001 from 0.851.

Worked steps

  1. Raw against all rows: 0.338 / 0.056 = 6.0 (ratio of log losses).
  2. Correction, boosted: 0.0556 - 0.0563 = -0.0007.
  3. Calibration, boosted: 0.0554 - 0.0563 = -0.0009.
  4. Correction, logistic: 0.0580 - 0.0579 = +0.0001.

Use the idea

Downsample negatives to cut cost, then restore probabilities with the formula only when the model is calibrated on the sample; otherwise calibrate on development rows with the natural prevalence, and assess on a separate partition.

Where the conclusion applies

Constructed data; HistGradientBoostingClassifier with 60 trees stands in for LightGBM. The correction's own assumptions hold by construction (negatives kept independently with equal probability, features unchanged). The keep rate where the correction falls behind depends on the number of positives and the model.

Check your understanding: At a 0.2% keep rate the analytic correction is clearly worse than development calibration for the boosted model but not for the logistic regression. Why?
The formula assumes the model is calibrated on the sample. With 563 training rows the boosted model is not, so the corrected probabilities average 0.050 against a true prevalence of 0.013 and log loss is 0.089 against 0.059 after calibration. The logistic regression has only nine parameters and stays calibrated on the sample, so its corrected log loss (0.059) is within 0.002 of the model trained on every row (0.058).

Chapter 39 source: section "Negative Downsampling at Scale".