Winning Kaggle the Reproducible Way

Chapter 32

Denoising Autoencoders for Tabular Pretraining

Judge a pretrained representation by a matched downstream comparison, not by its reconstruction loss.

A denoising autoencoder is trained to rebuild clean rows from corrupted ones. One demonstration trains one with swap noise at several strengths and measures both how well it reconstructs and what its features add to a classifier fitted on a few labeled rows, against an untrained network of the same shape.

Every example is constructed for teaching. Its numbers come from running the chapter activity's code, not from a competition result.

Demonstration 1 of 1

Swap-noise pretraining: reconstruction error against downstream value

Does a lower reconstruction error mean the pretrained features help a small labeled model more than an untrained encoder would?

With no corruption a wider-than-input network can copy its input, so reconstruction is nearly perfect and the hidden layer learns little: its features score a little below those of an untrained network. Swap noise stops the copy and forces the network to use the dependencies between features, so reconstruction error rises while downstream AUC rises too, reaching about 0.01 above the untrained encoder at the highest rate. Most of the gain over raw features is the nonlinear expansion any hidden layer provides, which the untrained control captures as well.

Equation: the pretraining loss is the average over rows of the squared distance between the clean row x i and the decoder g applied to the encoder f applied to the corrupted row

Scroll sideways for the whole equation

x is a clean row, x-tilde its swap-corrupted copy, f the encoder (the hidden layer) and g the decoder. The pretraining loss is the mean squared difference between x and g(f(x-tilde)); the downstream score is the AUC of a classifier on the labeled rows.

Predict first. At the usual swap rate of 0.1, how do the pretrained features compare with those of an untrained network of the same shape?

Your prediction

Choose an example

Scroll sideways for the whole figure

Figure: Swap-noise pretraining: reconstruction error against downstream value. Left: reconstruction error of an untrained network (1.405) and a swap-noise autoencoder (0.037). Right: classifier AUC with raw features (0.676), plus an untrained encoder (0.765) and plus the trained encoder (0.768), with a dot per dataset.
Swap probability per cell: 0.1: the chapter's usual rate
Constructed example: fourteen seeded synthetic datasets, a small scikit-learn autoencoder with swap noise and a logistic regression, measured by the chapter activity.

Calculated values

Reconstruction, trained
0.037
Reconstruction, untrained
1.405
AUC raw features
0.676
AUC + untrained encoder
0.765
AUC + trained encoder
0.768

At swap probability 0.1 the trained autoencoder reconstructs new rows with error 0.037 against 1.405 for the untrained network. Its features give a downstream AUC of 0.768; the raw features alone give 0.676 and an untrained encoder of the same shape 0.765. So 0.768 - 0.765 = 0.003 is what pretraining adds beyond a random encoder (one standard deviation over datasets: 0.011), while 0.768 - 0.676 = 0.092 is the gain over raw features.

Worked steps

  1. Reconstruction: 0.037 trained against 1.405 untrained (mean squared error, standardized units).
  2. Downstream AUC: trained 0.768, untrained encoder 0.765, raw features 0.676.
  3. Pretraining beyond a random encoder: 0.768 - 0.765 = +0.003.
  4. Gain over raw features: 0.768 - 0.676 = +0.092.

Use the idea

Run the untrained control (same shape, random weights) beside every pretrained representation, and compare all of them with the raw-feature model on matched labeled folds. Keep a pretrained encoder only if it beats the control on the downstream metric, and record the result when it does not.

Where the conclusion applies

Constructed data and a scikit-learn network with one hidden layer, trained for 30 epochs, standing in for the chapter's deeper PyTorch autoencoder; the downstream model is a logistic regression with a fixed regularization. A deeper autoencoder, a different head or a different target could favour pretraining; this setup measures that pretraining added little beyond an untrained encoder here, which is a result about this generator, not a verdict on DAEs.

Check your understanding: Without any corruption the autoencoder reconstructs almost perfectly. What does that tell you about its downstream value?
Nothing good: its reconstruction error is 0.003, the best of all four settings, yet its features score 0.751, below the untrained encoder's 0.765 and below every swap-noise setting. The network learned to copy its input, and reconstruction loss ranks the corruption levels in the opposite order from downstream AUC.

Chapter 32 source: section "Setting Realistic Expectations".