Winning Kaggle the Reproducible Way

Chapter 43

Image Classification Tricks

Crops of one image are not independent examples: split by source image.

Cutting several overlapping crops from each image is a common way to get more training data, and a random split then puts siblings on both sides of the validation boundary. One demonstration measures how far a random-split score, a split by source image and a truly held-out score drift apart as the number of crops per image grows.

Every example is constructed for teaching. Its numbers come from running the chapter activity's code, not from a competition result.

Demonstration 1 of 1

Random and grouped validation for crops of the same image

How much does a random split of crops overstate accuracy compared with splitting by source image, and how does that grow with crops per image?

Overlapping crops of one image are nearly the same picture. A random split leaves a crop's siblings in the training folds, and a nearest-neighbor model simply copies a sibling's label. Splitting by source image removes the siblings, so the score matches new images. With one crop per image there are no siblings and the two splits agree.

Equation: Overstatement equals the cross-validated accuracy minus the accuracy on new source images

Scroll sideways for the whole equation

Overstatement is the cross-validated accuracy minus the accuracy on new source images; it is positive when validation describes the training images better than new ones.

Predict first. Nine overlapping crops are cut from each of 300 source images and split at random into five folds. How does the cross-validated accuracy compare with the accuracy on crops of new source images?

Your prediction

Choose an example

Scroll sideways for the whole figure

Figure: Random and grouped validation for crops of the same image. Left: accuracy under a random split of crops (0.995), a split by source image (0.957) and on new source images (0.962) with 9 crops per image. Right: the same scores as error rates.
Crops cut from each source image: 9
Real data: scikit-learn's bundled digits, enlarged to 16 x 16; the crop construction and the classifier are constructed and measured by the chapter activity.

Calculated values

Random split
0.995
Split by source image
0.957
New source images
0.962
Random overstatement
+0.033
Grouped difference
-0.005

With 9 crops per source image (2,700 training crops), a random split reports 0.995 and a split by source image 0.957, while new source images score 0.962. The random split overstates accuracy by 0.995 - 0.962 = 0.033; the grouped split is 0.957 - 0.962 = -0.005 away. The random split claims an error rate 7.3 times smaller than the real one, and overstated accuracy in 100% of 8 draws.

Worked steps

  1. Random-split overstatement: 0.995 - 0.962 = +0.033.
  2. Grouped-split difference: 0.957 - 0.962 = -0.005.
  3. Error rate on new images against the random-split claim: 0.038 / 0.005 = 7.3.
  4. Training crops: 300 sources * 9 = 2,700.

Use the idea

Carry the source image ID through every crop and split by it for validation and early stopping; keep a separate assessment of whole images, and never tune crop policy on a random split of crops.

Where the conclusion applies

Real 16 x 16 digit images and a one-nearest-neighbor classifier, which memorizes nearby examples and so shows the leak clearly; a linear or heavily regularized model would leak less. Crops overlap heavily by construction. Grouped validation trains on 80% of the sources and sits slightly below the held-out score.

Check your understanding: At one crop per image the random split is honest, but at nine it is not. What changed, and what does the grouped split report at nine?
With one crop per image no sibling can sit in a training fold, so random and grouped splits agree (0.877 against 0.885). At nine, every crop has near-duplicate siblings in the training folds and a nearest-neighbor model copies their label, so random validation reaches 0.995. Grouping by source image gives 0.957, within 0.005 of the 0.962 on new images.

Chapter 43 source: section "Separate Development From Assessment".