Winning Kaggle the Reproducible Way

Chapter 6

Running Case: March Mania 2026

Treat the Elo offseason decay as a hypothesis, and choose it on past seasons.

The chapter's Elo ratings regress toward the mean each offseason by a fixed fraction. One demonstration builds leagues in which the true carryover is known, picks the decay on past seasons only, and judges the choice on later seasons.

Every example is constructed for teaching. Its numbers come from running the chapter activity's code, not from a competition result.

Demonstration 1 of 1

Choosing the Elo offseason decay on past seasons, against how much strength truly persists

Does an Elo decay chosen on past seasons find how much team strength truly carries over, and does it beat the chapter's 0.65?

With persistence p, a team's best forecast for next season keeps a fraction p of its deviation from average, so the decay that minimizes Brier should sit near p. A decay that is too low throws away real information and too high keeps stale ratings. Choosing on seasons 2 to 6 finds a value near p; whether that beats a fixed example value depends on how far p is from it.

Equation: r becomes 1500 plus d times r minus 1500; e A equals one over one plus ten to the power r B minus r A over 400

Scroll sideways for the whole equation

r is a team's Elo rating, d the offseason decay (the fraction of its deviation from 1500 that carries into the next season), r_A and r_B the two teams' pre-game ratings and e_A the probability the chapter's Elo assigns to team A winning.

Predict first. True persistence is 0.9 and the chapter's example decay is 0.65. If each league picks its decay on seasons 2 to 6, what decay does it pick on average?

Your prediction

Choose an example

Scroll sideways for the whole figure

Figure: Choosing the Elo offseason decay on past seasons, against how much strength truly persists. Left: Brier against offseason decay for the seasons that choose and the seasons that judge, with the true persistence 0.9 marked and each league's chosen decay as a triangle. Right: bars of Brier above the true-probability floor for five decay policies; tuned 0.0165, chapter value 0.0192.
True persistence of team strength from one season to the next: 0.9: strength mostly persists
Constructed example: seeded synthetic leagues with known persistence and the chapter's Elo update, measured by the chapter activity.

Calculated values

Mean decay chosen
0.92
Tuned Brier
0.2077
Chapter 0.65 Brier
0.2104
Best in hindsight
0.2069
Leagues where tuning wins
85%

True persistence 0.9: leagues choosing on seasons 2 to 6 pick a decay of 0.92 on average. On seasons 7 to 10 the tuned decay scores 0.2077 Brier and the chapter's 0.65 scores 0.2104: 0.2104 - 0.2077 = 0.0027 in the tuned decay's favour. Tuning beats 0.65 in 85% of the 20 leagues. Tuning on past seasons recovers a real gain over the example value.

Worked steps

  1. Mean decay chosen on seasons 2 to 6: 0.92 (true persistence 0.9).
  2. Chapter value minus tuned, seasons 7 to 10: 0.2104 - 0.2077 = +0.0027.
  3. Tuned minus best in hindsight: 0.2077 - 0.2069 = +0.0008.
  4. Tuned decay beats 0.65 in 85% of leagues; spread of the gain across leagues 0.0030.

Use the idea

Treat decay, K and any similar constant as hypotheses. Tune them on earlier seasons with past-only folds, judge the choice on later seasons, and keep the example value when the gain over it is smaller than the noise of the choice.

Where the conclusion applies

Constructed leagues: random pairings, neutral sites, no home advantage, strength following an AR(1) process, and ratings that all start at 1500. Real tournament teams, schedules and transfer behaviour differ, so 0.65 or any other value is not a recommendation. The Brier floor is the score of the true win probabilities, which no rating can reach. The first third of each season is scored because the carried rating matters most there; the chapter's matchup feature is read at tournament start, after a full season of updates. On the last third of each season the same tuned decay gained at most 0.0012 Brier over 0.65, under half of its first-third gain at persistence 0.3 and 0.9.

Check your understanding: When true persistence is 0.65, the chapter's own example value, does tuning on past seasons still help?
No. The chosen decay averages 0.67, but the tuned and default scores differ by only 0.0001 Brier and tuning wins in only half of the leagues (50% here): five seasons of games cannot separate nearby decays, so the choice adds noise about as large as the gain. At persistence 0.3 or 0.9 the same procedure beats 0.65 in most leagues (85% here) by about 0.0027 Brier.

Chapter 6 source: section "The Feature Set: Domain Knowledge Encoded as Numbers".