Winning Kaggle the Reproducible Way

Chapter 56

Reinforcement Learning for Game Competitions

Shape a reward toward winning, then check the policy against the real win rate.

An intermediate reward gives the learner feedback, and a learner optimizes the reward it is given. One demonstration trains Q-learners on a corridor game with a farmable resource bonus and compares the shaped return and the real win rate of the policies they learn.

Every example is constructed for teaching. Its numbers come from running the chapter activity's code, not from a competition result.

Demonstration 1 of 1

Shaped return against real win rate, as the resource bonus grows

At what size of intermediate reward does a learner stop trying to win, and does the shaped return reveal it?

A small bonus does no harm: the policy still heads for the goal and picks the resource up on the way. Once the bonus is large enough, circling the resource pays more than the discounted value of winning, the learner settles on circling, and the win rate falls to zero while the shaped return rises. The best policy for the bonus reward does the same, so this is a property of the reward and not a failure to explore. In between, at 0.05, the learner already loses most games although farming lowers even the shaped return per game: it maximizes a discounted reward with no clock, which matches neither score, so at that weight the shaped return still flags the loss and at 0.1 it no longer does. A potential-based reward of the same weight leaves the best policy unchanged.

Equation: the shaped return equals the win indicator plus w times the number of arrivals at the resource cell; a potential-based shaping reward F of a move from s to s prime is gamma times phi of s prime minus phi of s

Scroll sideways for the whole equation

w is the bonus per resource arrival, a win pays 1, and the shaped return of a game is the win indicator plus w times the number of fresh arrivals at the resource cell. The win rate is the share of games that reach the goal within 30 steps. gamma is the learner's discount, 0.97.

Predict first. With a resource bonus of 0.1 per arrival (a win pays 1), what win rate does the learner trained with the bonus reach?

Your prediction

Choose an example

Scroll sideways for the whole figure

Figure: Shaped return against real win rate, as the resource bonus grows. Left: real win rate at a bonus of 0.10: true reward only 100%, with the bonus 0%, potential-based 100%, best policy 0%. Right: shaped return 1.13 for the true-reward policy and 1.23 with the bonus.
Reward per fresh arrival at the resource cell (a win pays 1): 0.1
Constructed example: a seeded corridor game, tabular Q-learners and value iteration, measured by the chapter activity.

Calculated values

Win rate, true reward only
100%
Win rate, resource bonus
0%
Win rate, potential-based
100%
Shaped return, true reward only
1.13
Shaped return, resource bonus
1.23

With a bonus of 0.10 per arrival, the learner trained on the true reward wins 100% of games and the one trained with the bonus 0%, so 1.00 - 0.00 = 1.00 of the wins are lost to the bonus. The potential-based reward of the same weight wins 100% and the best policy for the bonus reward 0%. In shaped return the bonus-trained policy scores 1.23 and the true-reward policy 1.13, a difference of 0.10 in favour of the policy that stopped winning.

Worked steps

  1. Wins lost to the bonus: 1.00 - 0.00 = 1.00.
  2. Shaped return, bonus-trained minus true-reward policy: 1.23 - 1.13 = +0.10.
  3. Potential-based learner win rate: 100%.
  4. Best policy for the bonus reward: wins 0% with shaped return 1.24.

Use the idea

Pick checkpoints and reward weights by the real win rate against the declared opponents, not by the shaped return the learner was trained on, and make intermediate rewards pay once and point toward winning.

Where the conclusion applies

A constructed one-dimensional game, tabular Q-learning instead of PPO, and a bonus that can be farmed by design; real games have richer shaped rewards, and whether a given one can be farmed depends on the game. Optimistic initial values make the learner explore enough to find the goal: without them, even a tiny bonus stops this learner from ever finding it, which is a different failure (no exploration) from the one measured here. On this short corridor the sparse reward alone is learned quickly, so the benefit of shaping for very long horizons is not shown.

Check your understanding: At a bonus of 0.1 the shaped return of the farming policy is only 0.10 above that of the winning policy. Why does the learner prefer farming, and what does the shaped return hide?
The learner discounts: a bonus collected now is worth more than a win 13 steps away, and a farming loop pays again every two steps. The shaped return per game of the farming policy is already higher at a bonus of 0.1 (1.23 against 1.13), so ranking policies by it picks one that never wins. Only the win rate shows the loss.

Chapter 56 source: section "The Fundamental Challenge: Sparse Rewards, Long Horizons".