Winning Kaggle the Reproducible Way

Chapter 65

Recommendation Systems

Measure candidate coverage and the metric-specific oracle before tuning the reranker.

Retrieval fixes the ceiling of the whole pipeline and every candidate costs a scoring row. One demonstration builds a co-visitation retriever and a gradient-boosted reranker on generated sessions and measures the oracle score and the achieved score as the candidate cap grows.

Every example is constructed for teaching. Its numbers come from running the chapter activity's code, not from a competition result.

Demonstration 1 of 1

Oracle score and achieved score as the candidate cap grows

As the candidate cap grows, how much of the rising oracle score does a trained reranker actually capture?

Micro recall is the share of all relevant items that stage one keeps. The oracle is a different number: it takes the retained relevant items as perfectly ordered and applies the actual metric per query. Both rise with the cap because more relevant items are inside it. The trained reranker can only reorder what is there, and the items that enter at ranks 50 to 100 are the ones its features separate least from the rest, so its score levels off while the ceiling and the cost keep rising.

Equation: The oracle average precision at 10 for query q at cap K equals the smaller of the retained relevant count and 10, divided by the smaller of 10 and the number of relevant items

Scroll sideways for the whole equation

K is the candidate cap, R_q the number of relevant items of query q, r_q(K) how many of them stage one keeps in its top K, AP@10 the average precision with denominator min(10, R_q), and MAP@10 its mean over queries.

Predict first. Raising the cap from 20 to 100 candidates costs five times as many scoring rows and lifts the oracle MAP@10 from 0.289 to 0.491. How much does the trained reranker's MAP@10 rise?

Your prediction

Choose an example

Scroll sideways for the whole figure

Figure: Oracle score and achieved score as the candidate cap grows. Left: bars for micro recall 0.374, oracle MAP@10 0.402, stage-one order 0.096, trained reranker 0.145 and a random-negative reranker 0.142 at a cap of 50. Right: oracle MAP@10 and micro recall rising with the cap, with the reranker's score marked at the current cap.
Candidate cap (items kept from stage one, scored per query): 50
Constructed example: seeded synthetic sessions, a co-visitation retriever and a gradient-boosted reranker, measured by the chapter activity.

Calculated values

Micro recall
0.374
Oracle MAP@10
0.402
Trained reranker MAP@10
0.145
Stage-one order MAP@10
0.096
Rows scored per 1,000 queries
50,000

With a cap of 50, stage one keeps 0.374 of all relevant items, the oracle MAP@10 is 0.402 and the trained reranker reaches 0.145: 0.402 - 0.145 = 0.257 of the ceiling is not captured. Keeping stage one's own order scores 0.096, so the reranker adds 0.049. A reranker trained on random negatives scores 0.142. Scoring 50 candidates per query costs 50,000 rows per 1,000 queries.

Worked steps

  1. Uncaptured ceiling: 0.402 - 0.145 = 0.257.
  2. Reranker gain over stage one's order: 0.145 - 0.096 = 0.049.
  3. Oracle against micro recall: 0.402 - 0.374 = 0.028.
  4. Random-negative reranker against the retrieved-candidate reranker: 0.142 - 0.145 = -0.003.

Use the idea

Plot candidate recall and the oracle score by cap, train the reranker on the same capped candidates it will see at inference, and set the cap where the achieved score stops improving, not where the ceiling does.

Where the conclusion applies

Constructed sessions and a histogram-based gradient-boosted classifier standing in for the chapter's LightGBM ranker (pointwise scoring, not lambdarank). One feature set, one retrieval rule and three catalogs. The reranker beats stage one's own order here because the generator rewards the latest item, which stage one's sum over the whole history blurs; a different generator could shrink that gap. A reranker trained on positives plus random negatives scored within about 0.005 of the one trained on retrieved candidates here, so no training-mismatch penalty is claimed.

Check your understanding: At a cap of 100, micro recall is 0.468 but the oracle MAP@10 is 0.491. Why are they different, and which one is the ceiling?
Micro recall divides all retained relevant items by all relevant items, so queries with many relevant items count more. The oracle averages a per-query ratio, min(retained, 10) divided by min(10, relevant), so each query counts once. They are close here and the oracle is slightly higher, but only the oracle applies the metric the host scores, so only it is the ceiling for MAP@10.

Chapter 65 source: section "The Defining Shape: Retrieve, Then Rank".