Mistakes and lessons

My entry to Overmind's "Small Model, Big Maths" two-day model training hackathon in San Francisco (submissions closed 2026-10-07 17:00 PDT): a light LoRA fine-tune of Qwen3.5-4B that scored 87.2278 on GSM8K (1,212/1,319 correct, 91.89%, minus 4.66 B parameters), rank 6 on the final leaderboard.

Mistakes

1. One draw of the winning recipe. The light LoRA was trained once before Overmind changed its trainer; afterwards the same settings scored 1,112/1,319 instead of 1,212, so the result could never be repeated.
2. Clearing jobs to free slots. The 2-job limit pushed me to clear out jobs, and the best validation run (r6) went before it was ever tested.
3. A night of automation, no answers. The overnight controller ran 18 small probes, none clearly above the base, and produced no test evidence.
4. Many tests, no gain. 16 official tests after the submission; none beat 1,212 (best 1,208).

What to do next time

1. Re-draw a winner at once, several times, while the platform is unchanged.
2. Never drop an untested promising model; archive adapters first.
3. Re-run a known reference after any platform change before spending more runs.
4. Gate on cheap signals: the share of <<calc>> answers flagged every collapse (submission 0.26, collapsed models 0.87+).
5. Choose on validation, test once, and settle the parameter count on day 1.

All of it in one file: retrospective.md · llms.txt · data/history.json