Winning Kaggle the Reproducible Way

Chapter 30

External Data and Leakage

Join each row to the version of an external source that existed at that row's cutoff, in validation and in training.

An external indicator that is revised after its first release can be joined by event date or as of the prediction time. One demonstration measures how far each join's validation score sits from the score the model earns once only the versions available at the time exist, as the revisions grow.

Every example is constructed for teaching. Its numbers come from running the chapter activity's code, not from a competition result.

Demonstration 1 of 1

Event-date join against point-in-time join, by revision size

How far does a validation score built on revised data drift from deployment, and when does the training join start to cost accuracy too?

The event-date join hands every row the revised value, including rows whose cutoff came before even the first release. Its validation score is high because the indicator is nearly the truth. At deployment only the first release (or the previous month, before it is out) exists, so the same model scores far lower, and it was fitted to trust a column that is cleaner than the one it receives. The as-of join reproduces deployment conditions in training and in validation, so its estimate lands near the deployment score.

Equation: v star of c equals the version v with the latest publication time p v among versions published at or before the cutoff c

Scroll sideways for the whole equation

c is a row's prediction cutoff, v ranges over the published versions of an indicator, p_v is the time version v became public, and v*(c) is the version the row may use: the latest one published at or before c.

Predict first. With a first release of noise 0.6, how does the validation score of the event-date join compare with its score at deployment?

Your prediction

Choose an example

Scroll sideways for the whole figure

Figure: Event-date join against point-in-time join, by revision size. Paired bars of validation R-squared and deployment R-squared for the event-date and as-of joins at first-release noise 0.6, with error bars; the event-date join validates 0.241 above its deployment score.
Noise in the first release (standard deviation of its error): 0.6
Constructed example: one hundred seeded synthetic series with a declared release and revision schedule and a ridge regression, measured by the chapter activity.

Calculated values

Event-date validation
0.691
Event-date at deployment
0.450
As-of validation
0.468
As-of at deployment
0.494
Validation inflation
0.241

With first-release noise 0.6, the event-date join validates at 0.691 but scores 0.450 at deployment, 0.691 - 0.450 = 0.241 of inflation. The as-of join validates at 0.468 and scores 0.494, a difference of -0.026. Training on the event-date join also costs 0.044 of deployment R-squared (0.494 for the as-of model against 0.450).

Worked steps

  1. Event-date join: 0.691 - 0.450 = 0.241 (validation minus deployment).
  2. As-of join: 0.468 - 0.494 = -0.026.
  3. Cost of the event-date training join at deployment: 0.494 - 0.450 = +0.044.
  4. Spread of the as-of validation estimate over 100 series: one standard deviation is 0.074.

Use the idea

Store event period and publication timestamp for every external source, join each row to the latest version published at or before its cutoff in training, validation and the final pipeline, and count rows with nothing eligible. Compare against the model without the source.

Where the conclusion applies

Constructed series; the indicator, the lags and the revision process are declared in the method and change the sizes below. Even when the first release is exact, rows whose cutoff precedes it have only the previous month, so the event-date estimate is still too high. Single validation windows are noisy (the standard deviation over series is shown), which is why 100 series are averaged. A 40-month validation window also reads a little below the 160-month deployment window for any model, because a slow indicator spans less of its range in a short window; that is why the as-of estimate is close to, not exactly at, its deployment score.

Check your understanding: When the first release has no noise at all, the event-date join still misleads. Why, and does it cost accuracy at deployment?
Rows whose cutoff falls before the first release have no value for their own period, so the event-date join gives them information they could not have had: validation 0.691 against 0.553 at deployment. The model fitted on it still deploys within 0.010 of the as-of model, so at zero noise the join misstates the estimate more than it harms the model; the harm to the model grows with the revision size (0.146 at noise 1.0).

Chapter 30 source: section "A Version-Aware Join".