Winning Kaggle the Reproducible Way

Chapter 23

Sensors and HAR: The Cross-Subject Problem

Match the validation boundary to the test boundary: new subjects, not new windows.

Random stratified folds put the same person's windows on both sides of the split. One demonstration measures how far that estimate sits above leave-one-subject-out and above the score on genuinely new people, as people differ more from one another.

Every example is constructed for teaching. Its numbers come from running the chapter activity's code, not from a competition result.

Demonstration 1 of 1

Random folds against leave-one-subject-out, scored on new people

How far above the score on new people does a random stratified split sit, and does leave-one-subject-out close the gap?

With identical people a model that has seen a window's neighbours cannot cheat, because nothing about the person helps. As people differ, the random split lets the model learn each person's own cadence, amplitude and orientation and recognize their other windows, so it scores far above what it can do on someone new. Leave-one-subject-out withholds the whole person and lands near the new-people score.

Equation: macro F1 is the mean over C classes of the per-class F1 score

Scroll sideways for the whole equation

F1_c is the F1 score of activity class c, C the number of classes, y the true activity and the prediction is the model's class for a window. Macro-F1 averages F1_c over classes.

Predict first. With moderately distinct people (1.0), how does the macro-F1 of random stratified 5-fold compare with the score on new people?

Your prediction

Choose an example

Scroll sideways for the whole figure

Figure: Random folds against leave-one-subject-out, scored on new people. Two panels of bars for random stratified and leave-one-subject-out macro-F1 at distinctiveness 1.0, with a dashed line at the score on new people, 0.604; left is all six activities with dots per cohort, right is sitting and standing only.
How much each person differs from the average (0 means identical people): 1.0
Constructed example: four seeded synthetic cohorts of simulated wearable recordings and extremely randomized trees, measured by the chapter activity.

Calculated values

Random stratified macro-F1
0.799
Leave-one-subject-out macro-F1
0.588
Macro-F1 on new people
0.604
Random minus new people
+0.195
Sitting vs standing, new people
0.528

At distinctiveness 1.0, random stratified folds report macro-F1 0.799 and leave-one-subject-out 0.588, while the model scores 0.604 on 12 new people. The random estimate is off by 0.799 - 0.604 = 0.195; leave-one-subject-out is off by 0.017. For sitting against standing alone the three scores are 0.724, 0.473 and 0.528. The random split overstates the transfer.

Worked steps

  1. Random stratified: 0.799 - 0.604 = +0.195 (estimate minus new people).
  2. Leave-one-subject-out: 0.588 - 0.604 = -0.017.
  3. Sitting against standing, random minus new people: 0.724 - 0.528 = +0.196.
  4. Cohorts averaged: 4, each with 8 development subjects and 12 new people.

Use the idea

Group the validation splits by subject (and by recording or session where those also repeat), report the subject-held-out estimate, and check class-pair confusions such as sitting against standing separately.

Where the conclusion applies

Constructed signals with a simple generator, six activities, and extremely randomized trees standing in for the chapter's gradient boosting. Subject differences are a built-in mechanism of cadence, amplitude and orientation; real subjects differ in other ways. With identical people the overlapping windows cost nothing measurable here because the activities are separable; with weaker class signal they might, and this activity does not test that.

Check your understanding: With identical people (0), why do all three estimates agree even though the random folds mix overlapping windows?
Windows from one person look alike only because the person is distinctive, and with identical people there is no person-specific pattern to memorize. The estimates then agree (0.955, 0.953 and 0.961), so in this generator the overlap alone costs almost nothing. The error appears as soon as people differ and grows with how much they do.

Chapter 23 source: section "The Fix: Leave-One-Subject-Out CV".