Winning Kaggle the Reproducible Way

Chapter 21

Financial Competitions: Time, Regime, and Leakage

Choose validation from the forecast contract, and keep overlapping labels out of the assessment.

When labels span several future days, neighbouring rows share most of their answer. One demonstration measures how far shuffled, walk-forward and purged validation each drift from the score a model actually earns on the future.

Every example is constructed for teaching. Its numbers come from running the chapter activity's code, not from a competition result.

Demonstration 1 of 1

Three validation schemes against the future, as the label horizon grows

How far does each validation scheme's estimate drift from the model's score on the future as the label horizon grows?

With h = 1 each label is its own day, so every scheme estimates the future about equally well. As h grows, a shuffled split trains on rows whose labels contain most of each validation row's answer, and the trees memorize them. Chronological schemes keep training rows in the past, so only the first few rows of each block can share a label with training data. They come out a little low instead, because the early origins train on less history than the final model.

Equation: y t equals the sum over k from 1 to h of r t plus k

Equation: keep training row t only if t plus h is less than v

Scroll sideways for the whole equation

y_t is the label for day t, r_t the return on day t, h the label horizon in days and v the first day of a validation block. IC is the Pearson correlation between predictions and realized labels on the scored rows; higher is better and 0 means no skill.

Predict first. With a 20-day label horizon, how does shuffled 5-fold cross-validation compare with the score on the future?

Your prediction

Choose an example

Scroll sideways for the whole figure

Figure: Three validation schemes against the future, as the label horizon grows. Bars of three validation estimates of IC at a 20-day label horizon, dots for each of eight series, and a dashed line at the future IC of 0.256; the shuffled estimate is +0.090 from it.
Label horizon (days of future return in each label): 20: four-week return
Constructed example: eight seeded synthetic daily series and an extremely randomized trees model, measured by the chapter activity.

Calculated values

Shuffled 5-fold IC
0.346
Walk-forward IC
0.191
Purged walk-forward IC
0.187
IC on the future
0.256
Shuffled minus future
+0.090

With a 20-day label horizon the model's IC on the future is 0.256. Shuffled cross-validation reports 0.346, and 0.346 - 0.256 = 0.090 is its error. Walk-forward reports 0.191 and purged walk-forward 0.187, -0.065 and -0.069 from the future. The shuffled estimate is the one that misleads.

Worked steps

  1. Shuffled: 0.346 - 0.256 = +0.090 (estimate minus future).
  2. Walk-forward: 0.191 - 0.256 = -0.065.
  3. Purged walk-forward: 0.187 - 0.256 = -0.069.
  4. Effect of the purge gap: 0.187 - 0.191 = -0.004.

Use the idea

Validate forward-return targets chronologically, purge at least the label horizon before every validation block, and judge model choices by the chronological estimate, never by a shuffled one.

Where the conclusion applies

Constructed series with a stable relationship (no regime change), one model family and eight replicates; per-series estimates vary, so the chart shows each series as a dot. In this generator the purge gap moved the walk-forward estimate by less than 0.02 in either direction: the boundary leak touches only the first h rows of each block. Shuffling, not the gap, is the large error here.

Check your understanding: Why do the walk-forward and purged walk-forward estimates come out almost the same here, when the chapter insists on purging?
Without a gap, only the training rows just before each block have labels that reach into it, so the leak touches the first h of 150 validation rows and was within the noise of this generator. The purge costs only h training rows and removes that leak by construction, which is why the chapter makes it a rule; the shuffled split is the error that moved the estimate by 0.090 at h = 20.

Chapter 21 source: section "Failure Mode 2: Regime Change".