Winning Kaggle the Reproducible Way

Chapter 22

Vision and Multimodal: When You Need a VLM

A pretrained embedding and its reduction are separate choices; explained variance does not say what task information is kept.

PCA keeps variance, not necessarily the directions that carry the label. One demonstration places the label signal at different places in an embedding's variance ranking and measures what a variance rule and a cross-validated choice each keep.

Every example is constructed for teaching. Its numbers come from running the chapter activity's code, not from a competition result.

Demonstration 1 of 1

PCA on an embedding before a second stage: the 90% variance rule against cross-validation

If the label signal sits low in the embedding's variance ranking, what does a PCA that keeps 90% of the variance leave for the second stage?

Each principal component is ranked by the variance it explains, which is a property of the embedding alone and never sees the label. When the signal sits among the top directions a few components keep it. When it sits lower, the components that explain 90% of the variance leave it out and the second stage falls back to the soft label. Keeping every component avoids the loss, and cross-validating the choice finds that without needing to know where the signal is.

Equation: EV of k equals the sum of the k largest variances lambda j divided by the sum of all D variances

Scroll sideways for the whole equation

lambda_j is the variance of embedding direction j (largest first), k the number of principal components kept and EV(k) the fraction of total variance they explain.

Predict first. With the signal starting at variance rank 20, how well does a second stage on a PCA that keeps 90% of the variance do, compared with the soft label alone?

Your prediction

Choose an example

Scroll sideways for the whole figure

Figure: PCA on an embedding before a second stage: the 90% variance rule against cross-validation. Left, accuracy of a second stage using k principal components with the variance explained under each k, with dashed lines for the soft label alone and the raw embedding; right, bars for four choices with the signal starting at rank 20.
Variance rank where the label signal starts (0 is the largest-variance direction): 20
Constructed example: twenty-four seeded synthetic embeddings with a planted signal, PCA and logistic regression, measured by the chapter activity.

Calculated values

Soft label alone
0.707
Raw embedding
0.823
90% variance rule
0.695
Cross-validated choice
0.826
Components in the 90% rule
16

With the signal starting at variance rank 20, the 90% rule keeps about 16 components (91% of the variance) and scores 0.695, against 0.707 for the soft label alone and 0.823 for the raw embedding: 0.695 - 0.707 = -0.012, so the reduced embedding has lost the signal. Choosing by cross-validation scores 0.826, and 0.826 - 0.695 = 0.131 is its margin over the rule.

Worked steps

  1. Soft label alone: 0.707. Raw embedding plus soft label: 0.823.
  2. 90% variance rule: 0.695 with about 16 components.
  3. Variance rule against soft label alone: 0.695 - 0.707 = -0.012.
  4. Cross-validated choice: 0.826; it picked the soft label alone in 0.00 of datasets.

Use the idea

Treat the reduction as a candidate: compare no reduction, the soft outputs alone and a few component counts inside the development folds, fit the projection on training rows only, and assess the frozen choice on untouched rows.

Where the conclusion applies

Constructed data with a linear, equal-strength signal in five directions, a logistic regression stand-in for the chapter's LightGBM second stage, and an independent soft label. Real embeddings do not place their signal on convenient axes, and this activity does not show that this mechanism caused the PCA loss in the Autopilot comparison, which the chapter says was not isolated.

Check your understanding: The 90% rule keeps the same fraction of variance wherever the signal sits. Why does its accuracy change so much?
Explained variance counts how much the embedding varies along a direction, not whether the label varies with it. The signal here is equally strong at every rank, but the 90% rule keeps about 15 or 16 components, which contain all five signal directions only when the signal starts at rank 10 or earlier. Cross-validation measures what each candidate delivers on held-out rows, so it keeps the signal wherever it sits.

Chapter 22 source: section "When VLMs Beat Traditional CV Pipelines".