Winning Kaggle the Reproducible Way

Chapter 7

Domain Features as Testable Hypotheses

An entity history is a hypothesis about identity, and it needs an availability check and a held-out comparison.

A purchase is unusual only relative to its customer, so a past-only entity mean can add a lot. One demonstration measures how much it adds as the pseudo-ID merges more customers into one key, and what a mean that sees later rows appears to add in development.

Every example is constructed for teaching. Its numbers come from running the chapter activity's code, not from a competition result.

Demonstration 1 of 1

Entity history against pseudo-ID collisions, scored on later transactions

How much does an entity-history feature add over the raw amount when the key merges several customers, and how much of its apparent value comes from rows the prediction could not see?

With a perfect key the past-only ratio removes each customer's spend level and the fraud shows clearly. As the key merges customers, its mean describes a blend of spend levels, so the ratio stops describing the row's own customer and the lift over the raw amount fades. A whole-period mean also looks better in development, because early rows borrow later rows of the same key. At deployment those later rows do not exist, and the extra development score does not carry over.

Equation: r i equals log x i minus log h i, where h i is the mean amount of the earlier transactions with the same key

Scroll sideways for the whole equation

x_i is the amount of transaction i, h_i the mean amount of earlier transactions with the same key, and r_i the log ratio fed to the model. Rows with no history get r_i = 0 and a missing-history flag. A whole-period mean replaces h_i by the mean of every development row of the key.

Predict first. With 20 customers sharing each key, what does the past-only ratio add over the raw amount on the forward holdout?

Your prediction

Choose an example

Scroll sideways for the whole figure

Figure: Entity history against pseudo-ID collisions, scored on later transactions. Paired bars of development cross-validation AUC and forward-holdout AUC for three feature sets with 1 customers per key; the past-only ratio lifts the holdout AUC by 0.202 over the raw amount.
Customers sharing one pseudo-ID key: 1: the key is the customer
Constructed example: seeded synthetic transactions with a built-in spend level per customer, measured by the chapter activity.

Calculated values

Raw amount, holdout
0.706
Past-only ratio, holdout
0.908
Past-only lift
+0.202
Whole-period, cross-validation
0.907
Whole-period, holdout
0.906

With 1 customer per key, the raw amount scores 0.706 on the forward holdout and the past-only ratio 0.908: 0.908 - 0.706 = 0.202 of lift, so the past-only history is worth building at this key quality. In cross-validation the whole-period mean scores 0.043 above the past-only mean (0.907 against 0.864); on the holdout, where later rows do not exist, the difference is -0.002.

Worked steps

  1. Lift of the past-only ratio on the holdout: 0.908 - 0.706 = +0.202.
  2. Development advantage of the whole-period mean: 0.907 - 0.864 = +0.043.
  3. Holdout advantage of the whole-period mean: 0.906 - 0.908 = -0.002.
  4. Past-only ratio, cross-validation minus holdout: 0.864 - 0.908 = -0.044 (holdout rows have longer histories).

Use the idea

Name the entity, check how many customers each key holds, build every history from rows before the prediction time, and score the history feature against the raw baseline on a later period.

Where the conclusion applies

Constructed data and one model family. The collision here merges customers at random, and the lift stays positive while a key holds a few customers; real collisions may be milder or worse. Holdout rows come later and so have longer histories than development rows, which is why history scores there exceed the cross-validation scores. The whole-period arm is scored on the holdout with the mean of rows up to and including the present, the most generous deployable version.

Check your understanding: With an exact ID, why does the whole-period mean score higher than the past-only mean in development but not on the holdout?
In development the whole-period mean gives each row a history that includes later rows, so even early rows get a well-estimated customer level (0.907 against 0.864 for the past-only mean). On the holdout only history to date exists, and the two arms score alike (0.906 against 0.908). The development gap measures access to the future, not a better feature.

Chapter 7 source: section "The Pattern: Find the Entity the Problem Is Really About".