Winning Kaggle the Reproducible Way

Chapter 41

Online Learning

Replay the timeline: a label can only update the model after it is released.

An adaptive model is only as good as the labels it is allowed to see. One demonstration replays a drifting stream with delayed labels and compares a frozen model, a rolling refit and an incremental update, all using released labels only, against a replay that ignores the delay.

Every example is constructed for teaching. Its numbers come from running the chapter activity's code, not from a competition result.

Demonstration 1 of 1

What label delay does to an adaptive model

How much of an adaptive model's advantage over a frozen model survives as labels arrive later, and how wrong is a replay that ignores the delay?

With no delay the newest rows describe the current coefficients and a short window tracks the drift. With a delay every released row is already d steps old, so the model lags the drift, and the best window is longer to average out noise. A frozen model still holds the stable part of the signal, so at long delays the adaptive gain nearly vanishes. The invalid replay does not lag, so it reports the zero-delay result at every delay: an estimate that stops describing what a live system would score.

Equation: The prediction at origin t is x t transposed times w hat t, where w hat t is fitted only on rows i with i plus d less than t

Scroll sideways for the whole equation

t is the prediction origin, i a training row, d the label delay, w-hat_t the coefficients fitted for origin t and x_t its features; row i is eligible at origin t only if i + d < t.

Predict first. With labels released instantly, a rolling refit beats a frozen model by 1.05 in squared error. If labels arrive 40 steps late (window re-selected), how much of that gain remains?

Your prediction

Choose an example

Scroll sideways for the whole figure

Figure: What label delay does to an adaptive model. Left: running mean squared error over the scored origins for the frozen, rolling, incremental and invalid-replay policies with labels released 40 steps late. Right: final errors, frozen 2.623, rolling 2.320, incremental 2.174, invalid replay 1.518.
Label delay (steps between a row and its label's release): 40
Constructed example: a seeded synthetic drifting stream and ridge models, measured by the chapter activity.

Calculated values

Frozen
2.623
Rolling refit
2.320
Incremental
2.174
Invalid replay
1.518
Selected window
126 rows
Rolling gain over frozen
0.303

With labels released 40 steps after their rows, the frozen model scores 2.623, the rolling refit 2.320 (window 126 rows on average) and the incremental policy 2.174. The rolling gain over frozen is 2.623 - 2.320 = 0.303. The invalid replay scores 1.518, suggesting a gain of 1.105 over frozen; its optimism against the eligible rolling refit is 2.320 - 1.518 = 0.802. The noise floor is 1.000.

Worked steps

  1. Rolling gain over frozen: 2.623 - 2.320 = 0.303.
  2. Incremental gain over frozen: 2.623 - 2.174 = 0.449.
  3. Optimism of the invalid replay: 2.320 - 1.518 = 0.802.
  4. Rolling error above the noise floor: 2.320 - 1.000 = 1.320.

Use the idea

Store release time beside event time, fit each origin only on rows with release time before it, choose window and update schedule on earlier origins, and compare against a frozen model under the same replay.

Where the conclusion applies

Constructed linear stream with stable coefficients plus mean-reverting drift, ridge models and fixed settings grids. The frozen model is fitted once on the first 300 steps' released labels. Delays are in steps; the shape of the curves depends on the drift speed relative to the delay.

Check your understanding: A replay that fits on the latest rows by event time scores 1.52 at every delay. Why is that invalid, and what does the selected window length tell you?
It uses labels that a live system would not have received yet, so it never lags the drift and reports the no-delay score (1.52) even at a delay of 100, where the eligible rolling policy scores 2.48 and a frozen model 2.70. The selected window grows from 30 rows with no delay to about 213 at delay 100, because stale rows are only useful averaged over more of them.

Chapter 41 source: section "A Delayed-Label Contrast".