Winning Kaggle the Reproducible Way

Chapter 64

Audio and Signal Competitions

Keep chunks of one recording together, because the independent unit is the recording.

Chunks cut from one recording share its background, so a split by chunk lets a model recognise recordings instead of calls. One demonstration generates audio and measures chunk-level and recording-level cross-validation against recordings the model never saw.

Every example is constructed for teaching. Its numbers come from running the chapter activity's code, not from a competition result.

Demonstration 1 of 1

Chunk-level and recording-level cross-validation against new recordings

How far does chunk-level cross-validation overstate accuracy on new recordings, and does it distort the comparison between models?

Every chunk of a recording carries the same background. A flexible model can learn that background as a fingerprint of the species, and a chunk-level split puts siblings of every held-out chunk in training, so the fingerprint scores well. A recording-level split removes the siblings, so the model is judged on the call itself. More chunks per recording gives the fingerprint more to memorize and the optimism grows.

Equation: Optimism equals the cross-validation AUC minus the AUC on new recordings

Scroll sideways for the whole equation

AUC is the area under the ROC curve for the species label, pooled over all held-out chunks. Optimism is a cross-validation AUC minus the AUC on new recordings.

Predict first. With 8 chunks per recording, how does the flexible model's chunk-level cross-validation AUC compare with its AUC on recordings it never saw?

Your prediction

Choose an example

Scroll sideways for the whole figure

Figure: Chunk-level and recording-level cross-validation against new recordings. Left: AUC of a flexible and a simple model under chunk-level cross-validation, recording-level cross-validation and on new recordings with 8 chunks per recording; the flexible model scores 0.895, 0.628 and 0.684. Right: the flexible model's lead over the simple one under each scheme: +0.154, -0.011 and +0.009.
Chunks cut from each recording: 8
Constructed example: seeded synthetic audio, FFT features and two classifiers, measured by the chapter activity.

Calculated values

Flexible, chunk-level CV
0.895
Flexible, recording-level CV
0.628
Flexible, new recordings
0.684
Simple, new recordings
0.675
Chunk-level optimism (flexible)
+0.211

With 8 chunks per recording, the flexible model scores AUC 0.684 on new recordings. Chunk-level cross-validation reports 0.895, an optimism of 0.895 - 0.684 = 0.211; recording-level cross-validation reports 0.628. The flexible model's AUC minus the simple model's is +0.154 under chunk-level validation, -0.011 under recording-level validation and +0.009 on new recordings.

Worked steps

  1. Chunk-level optimism, flexible model: 0.895 - 0.684 = 0.211.
  2. Recording-level error, flexible model: 0.628 - 0.684 = -0.056.
  3. Chunk-level lead of the flexible model: 0.895 - 0.741 = 0.154.
  4. Lead on new recordings: 0.684 - 0.675 = 0.009.

Use the idea

Group chunks by recording (or site, or subject) in every split, choose models on the grouped score, and check the support in each fold when classes are rare.

Where the conclusion applies

Constructed audio with a recording-level label, a single kind of background shift and tree and logistic models in place of the chapter's CNN on mel spectrograms. Only 60 recordings, so the recording-level estimate itself varies by a few hundredths between replicates, and the flexible-versus-simple gap on new recordings is small here (under 0.06 across the seeds tried).

Check your understanding: At 8 chunks per recording, chunk-level validation says the flexible model beats the simple one by 0.154. What do the other two scores say?
Recording-level cross-validation puts the flexible model 0.011 below the simple one, and on new recordings it is 0.009 above, a lead of at most a few hundredths either way. Chunk-level validation would pick the flexible model for its memorized fingerprints, a gain that is mostly absent on new recordings.

Chapter 64 source: section "Record-Level Cross-Validation Is Not Optional".