Winning Kaggle the Reproducible Way

Chapter 50

LLM Fine-Tuning for Kaggle

Compare the adapted model with the original checkpoint and a full update under matched data, and record rank and trainable parameters.

A low-rank update trains far fewer parameters than a full one. One demonstration adapts a frozen linear layer with a LoRA-style update of growing rank on 100 target rows and measures held-out error against the original layer and a full update.

Every example is constructed for teaching. Its numbers come from running the chapter activity's code, not from a competition result.

Demonstration 1 of 1

Low-rank updates of growing rank against the original layer and a full update

How does the rank of a LoRA-style update change held-out error against the original checkpoint and a full update when target data is small?

A low-rank update can only express changes of rank r, so it needs r at least as large as the true change, and every extra rank adds parameters that fit noise in 100 rows. The error therefore falls until r matches the hidden rank and then rises, approaching the full update (800 parameters) as r grows. The original layer shows what adaptation has to beat.

Equation: The adapted weight W equals the frozen weight W zero plus alpha over r times the product B A, with B of size d in by r and A of size r by d out

Scroll sideways for the whole equation

W0 is the frozen layer, B A the low-rank update with B of size 40 by r and A of size r by 20, r the rank, and the held-out error is the mean squared difference between the adapted layer's output and the noiseless target.

Predict first. The true change has rank 2. Which LoRA rank gives the lowest held-out error, and how does rank 16 compare with a full update?

Your prediction

Choose an example

Scroll sideways for the whole figure

Figure: Low-rank updates of growing rank against the original layer and a full update. Two panels. Left, held-out error against LoRA rank, with a falling then rising curve whose lowest value 0.095 is at rank 2, dashed lines for the original layer (1.015) and the full update (0.682), and a marker at rank 4 (0.245). Right, bars of held-out error for the original layer, the selected LoRA rank and the full update, with a dot per task.
LoRA rank r (trainable parameters: 60 r): 4: 240 parameters
Constructed example: seeded synthetic linear layers and numpy gradient training, measured by the chapter activity.

Calculated values

Original layer error
1.015
Full update error
0.682
LoRA error, rank 4
0.245
Gain over full (SE)
+0.438 (0.010)
Trainable parameters
240
Best rank in the sweep
2

At rank 4 (240 trainable parameters against 800 for a full update) the adapted layer has held-out error 0.245. The original layer scores 1.015 and the full update 0.682, so the LoRA update is 0.682 - 0.245 = 0.437 better than the full update (paired standard error 0.010) and 1.015 - 0.245 = 0.770 better than doing nothing. Across the ranks tried the lowest error is 0.095 at rank 2; the hidden change has rank 2.

Worked steps

  1. Gain over the original layer: 1.015 - 0.245 = +0.770.
  2. Gain over the full update: 0.682 - 0.245 = +0.437 (standard error 0.010).
  3. Trainable parameters: 240 of 800 (30.0%).
  4. Best rank in the sweep: 2 with held-out error 0.095.

Use the idea

Report rank, scaling and trainable parameters, compare every adapter with the original checkpoint and with a full update at matched data, and choose the rank by a sweep on development rows, not by habit.

Where the conclusion applies

A single linear layer trained on squared error stands in for a transformer, and the target change has an exact low rank by construction, which real tasks do not guarantee: the sweep, not rank 2, is the transferable part. Scaling is fixed at alpha / r = 1; other scalings or learning rates move the curve. The full update has no weight decay or early stopping; in a side check, stopping it early helped only a little. With much more target data the full update catches up, and with a smaller task shift it can lose to the original layer.

Check your understanding: Rank 2 is only best because the true rank is known here. What do you do in a competition, and what does the original-layer line add?
Run the sweep on development rows and pick the smallest rank after which the error stops falling; the curve here has a clear minimum, and rank 4 already costs 0.245 against 0.095 at rank 2. The original-layer line (1.015) shows the gain from adapting at all, and the full update (0.682) shows that touching every weight gives most of that gain back as noise.

Chapter 50 source: section "Why LoRA Changes the Game".