Winning Kaggle the Reproducible Way

Chapter 15

When the Problem Isn't Tabular

Evaluation discipline carries over: compare whole policies, including the errors a second model introduces.

A second-model reviewer can fix a base answer or break it. One demonstration measures three policies on held-out questions as the reviewer's quality changes: no review, review everything, and review only the low-confidence answers.

Every example is constructed for teaching. Its numbers come from running the chapter activity's code, not from a competition result.

Demonstration 1 of 1

Review everything, or route only the doubtful answers

How good must a reviewer be before reviewing every answer beats the base model, and does routing change that?

A reviewer that is independent of the base model is right about as often on a doubtful question as on any other, while the base model is much less accurate on the questions it is unsure about. Reviewing everything therefore overturns many correct base answers, and the net can be a loss. Routing only the doubtful questions to the reviewer keeps a large share of the fixes while overturning fewer correct answers, with fewer calls. The threshold chosen on development questions routes few questions to a weak reviewer and nearly all of them to a strong one.

Equation: change in accuracy equals the share of answers fixed minus the share of answers introduced as errors

Scroll sideways for the whole equation

Fixed is the share of questions the base answered wrongly and the policy answers correctly, introduced the share the base answered correctly and the policy gets wrong; accuracy changes by fixed minus introduced.

Predict first. The reviewer alone is slightly less accurate than the base model (0.545 against 0.574). What does reviewing every question do to accuracy?

Your prediction

Choose an example

Scroll sideways for the whole figure

Figure: Review everything, or route only the doubtful answers. Left: accuracy of base only (0.574), review all (0.545) and routed review (0.636) with reviewer noise 1.9. Right: wrong answers fixed and correct answers overturned per 100 questions for review all and routed review.
Reviewer's view noise (larger means a weaker reviewer): 1.9 (about 0.55)
Constructed example: seeded synthetic questions and two logistic regressions, measured by the chapter activity.

Calculated values

Base only
0.574
Review all
0.545
Routed
0.636
Reviewer alone
0.545
Routed calls
51%

With reviewer noise 1.9 the reviewer alone scores 0.545 against 0.574 for the base model. Reviewing everything scores 0.545: it fixes 23.4 and overturns 26.2 correct answers per 100 questions, so 0.545 - 0.574 = -0.029. Routing the 51% least confident questions scores 0.636, +0.062 against the base, and beat the base in 100% of the 30 question sets.

Worked steps

  1. Review all minus base: 0.545 - 0.574 = -0.029 accuracy.
  2. Routed minus base: 0.636 - 0.574 = 0.062 accuracy.
  3. Routed, per 100 questions: 16.0 fixed - 9.7 overturned = 6.3.
  4. On the routed questions the base is right 0.426 of the time and the reviewer 0.547.

Use the idea

Compare base only, review everything and routed review on the same untouched labelled questions, with the number of reviewer calls and the count of correct answers the reviewer overturned. Choose the confidence threshold on a development set, not on the test questions.

Where the conclusion applies

Constructed data with two logistic regressions standing in for a multimodal base model and a reviewer. The reviewer's errors are independent of the base model's, and it answers from its own view rather than judging the base answer; a real reviewer may anchor on the answer it is shown, or return text that is not a valid answer. Confidence here is a real model probability, and the schema-validity check the chapter describes is not modelled.

Check your understanding: Routed review helps even though the reviewer is weaker overall. Why?
Among the questions it routes (51% of all questions), the base model is right only 0.426 of the time while the reviewer is right 0.547 of the time, because the base model's low confidence marks the questions it is worst at. Elsewhere the base model is better, so routing leaves those answers alone.

Chapter 15 source: section "Visual Question Answering: Autopilot 2026".