Training history

My entry to Overmind's "Small Model, Big Maths" two-day model training hackathon in San Francisco (submissions closed 2026-10-07 17:00 PDT): a light LoRA fine-tune of Qwen3.5-4B that scored 87.2278 on GSM8K (1,212/1,319 correct, 91.89%, minus 4.66 B parameters), rank 6 on the final leaderboard.

P1: Pick the base model (Oct 5 evening)

2B, 4B and 9B Qwen3.5 with standard LoRA SFT. The 4B won; heavy SFT made every model worse than the untouched 4B. Trainer: old (packed rows kept separate).

P2: Light touch wins (Oct 6 morning)

LR 1e-5, one epoch, 48 packed steps: +1 pt on validation, then 1,212/1,319 on test = the submission (87.2278). Trainer: old.

P3: Re-rolls after the trainer change (Oct 6 afternoon-evening)

The same recipe re-run collapsed into short `<<calc>>` answers: 1,074-1,170 on test. Trainer: packed rows attend to each other (Overmind commit 1be6037, 10:12 PDT).

P4: Overnight probes (Oct 6 22:00 - Oct 7 09:00)

18 automated runs on 960-row subsets (batch, LR, attention-only, Full-FT, 2 epochs; a 19th never finished): none more than 2 items above the base on 200 val items. Trainer: new; 960-row subsets train unpacked.

P5: Other bases and full fine-tuning (Oct 7 09:30 - 13:00)

Qwen3-4B (127/200), packed light LoRA = base on test (1,198 / 1,197), unpacked LoRA and Full-FT below base. Trainer: packed until ~11:30, then everything unpacked (PR #167).

P6: Around the submitted recipe (Oct 7 13:00 - 17:00)

Exact copy 1,058; 48 steps at LR 1e-5 collapsed; 48 steps at LR 5e-6 batch 16 was the best since the change: 1,198-1,208. Trainer: new, unpacked.

All of it in one file: retrospective.md · llms.txt · data/history.json