Winning Kaggle the Reproducible Way

Chapter 8

The Five-Stage Feature Engineering Process

A target transform changes the quantity a model optimizes, so the official metric decides how to undo it.

Squared error on log(y) optimizes a different quantity from raw-scale RMSE. One demonstration measures a raw-target fit, a log fit exponentiated back, and a smearing-corrected log fit under both metrics as the multiplicative noise grows.

Every example is constructed for teaching. Its numbers come from running the chapter activity's code, not from a competition result.

Demonstration 1 of 1

Three target policies under raw-scale RMSE and log-scale RMSLE

When the target is multiplicative and skewed, does fitting log1p(y) help, and does the answer depend on the metric and on how the prediction is brought back to the raw scale?

Squared error on log1p(y) targets a typical value, close to the median, while raw-scale RMSE targets the mean. With log-normal noise the mean exceeds the median by about exp(s squared over 2), so the plain back-transform predicts low on average, and more so as s grows; the fitted model falls a little further short than that factor alone. The smearing correction brings the average prediction most of the way back (above 80% of the target mean here) and wins on RMSE, but it moves the prediction away from the log-scale median and loses on RMSLE. Fitting the raw target needs no inverse, but it chases the extreme values.

Equation: The conditional mean of y is e to the mu of x plus s squared over 2. The smeared prediction is e to the m of x times the average of e to the e i, minus 1

Scroll sideways for the whole equation

y is the target, m(x) the model's predicted mean of log1p(y), e_i the cross-fitted residuals on the training rows and s the log-scale noise spread. RMSE is computed on y, RMSLE on log1p(y).

Predict first. At noise spread 0.8, which target policy has the lowest RMSLE, and which has the lowest raw-scale RMSE?

Your prediction

Choose an example

Scroll sideways for the whole figure

Figure: Three target policies under raw-scale RMSE and log-scale RMSLE. Two bar charts at noise spread 0.8 comparing a raw-target fit, a plain log fit, a smeared log fit and a constant: raw-scale RMSE on the left with a known-mean reference line, RMSLE on the right with a known-median reference line.
Multiplicative noise spread s (standard deviation of the log-scale noise): 0.8
Constructed example: seeded synthetic log-normal targets and a boosted regressor, measured by the chapter activity.

Calculated values

RMSLE, plain log fit
0.749
RMSLE, raw-target fit
1.034
RMSE, plain log fit
213.1
RMSE, smeared log fit
203.4
RMSE, raw-target fit
214.3
Mean prediction, plain log fit
39.2

At noise spread 0.8, the plain log fit scores RMSLE 0.749 (raw-target fit 1.034) but its predictions average 39.2 against a target mean of 58.6, so 39.2 / 58.6 = 0.669 of the mean is recovered. The smearing correction lowers RMSE from 213.1 to 203.4 and raises RMSLE from 0.749 to 0.804. The raw-target fit has RMSE 214.3 and the constant 256.7.

Worked steps

  1. Mean recovered by the plain back-transform: 39.2 / 58.6 = 0.669.
  2. Gain of smearing on RMSE: 213.1 - 203.4 = +9.7 (positive means smearing is better).
  3. Cost of smearing on RMSLE: 0.804 - 0.749 = +0.055 (positive means smearing is worse).
  4. Raw-target fit against the plain log fit on RMSLE: 1.034 - 0.749 = +0.286.

Use the idea

Read the official metric, fit the target on a scale you can justify, choose the inverse policy that suits that metric, and compare against a raw-target fit and a simple predictor on the same assessment rows.

Where the conclusion applies

Constructed data with log-normal noise, one boosting model and 6,000 training rows. Raw-scale RMSE here is dominated by noise no model can remove (the known-mean line), so the raw fit and the plain log fit are within about 3% of each other for s of 0.8 and above, and their order is not stable; no claim is made about which of those two wins on RMSE. A histogram gradient boosting regressor stands in for LightGBM.

Check your understanding: Why does the plain log fit win on RMSLE yet predict an average well below the average target?
It targets the log-scale typical value, which is below the raw-scale mean when noise is multiplicative. At s = 0.8 its predictions average 39.2 against a target mean of 58.6, and the raw-target fit's RMSLE is 1.034, against 0.749 for the log fit. RMSLE rewards the typical value; raw-scale RMSE rewards the mean, which is why smearing helps there.

Chapter 8 source: section "Stage 4: Target Transforms".