All training runs

Every run of the hack: recipe, validation and official test. Val = correct out of 200 or 500 held-out train rows, with the gap to the untouched base on the same items in brackets (40-item rows are style peeks). Test = correct out of 1,319. <<calc>> and Chars describe the answers: test where tested, else validation (marked v).

Runs as plain text (for screen readers, LLMs and agents)

My entry to Overmind's "Small Model, Big Maths" two-day model training hackathon in San Francisco (submissions closed 2026-10-07 17:00 PDT): a light LoRA fine-tune of Qwen3.5-4B that scored 87.2278 on GSM8K (1,212/1,319 correct, 91.89%, minus 4.66 B parameters), rank 6 on the final leaderboard.

Run	Phase	Base	Method	Data	LR	Validation	Official test	Outcome
r1-q35-2b-zs	P1	Qwen3.5-2B	LoRA r32	6,973 rows	2e-4	328/500 (base 457)		too weak
r3-q35-2b-zs-3ep	P1	Qwen3.5-2B	LoRA r32	6,973 rows	2e-4	330/500 (base 457)		too weak
r1-q35-4b-zs-bs8	P1	Qwen3.5-4B	LoRA r32	6,973 rows	2e-4	419/500 (base 457)		beats 78.39 board at the time
r3-q35-4b-zs-1ep-lr1e4	P1	Qwen3.5-4B	LoRA r32	6,973 rows	1e-4	416/500 (base 457)		same as 2 epochs
r3-q35-4b-zs-r64-lr1e4	P1	Qwen3.5-4B	LoRA r64	6,973 rows	1e-4	420/500 (base 457)		no gain from rank
r3-q35-4b-full-1ep-lr1e5	P1	Qwen3.5-4B	Full fine-tune	6,973 rows	1e-5	408/500 (base 457)		worst 4B
r5-q35-9b-zs-1ep-lr1e4	P1	Qwen3.5-9B	LoRA r32	6,973 rows	1e-4	434/500 (base 457)		score 77 (9.65 B)
r1-q3-4b-zs	P1	Qwen3-4B	LoRA r32	6,973 rows	2e-4			failed x3 before step 1
r0-q35-4b-untouched	P1	Qwen3.5-4B	LoRA r32 at LR 1e-8 (= untouched)	6,973 rows	1e-8	457/500 (base 457)		the reference: beats every SFT run
r6-q35-4b-zs-1ep-lr1e6	P2	Qwen3.5-4B	LoRA r32	6,973 rows	1e-6	180/200 (base 180)		= untouched
r6-q35-4b-zs-1ep-lr1e5	P2	Qwen3.5-4B	LoRA r32, 48 packed steps	6,973 rows	1e-5	462/500 (base 457)		best clean val (462/500); never tested
final-q35-4b-full-lr1e5	P2	Qwen3.5-4B	LoRA r32/a64 all-linear, 48 packed steps	7,473 rows (all)	1e-5		1,212/1,319, score 87.23	SUBMITTED: 87.2278 (board 87.2, 6th)
final-q35-4b-full-lr1e8	P2	Qwen3.5-4B	LoRA r32 at LR 1e-8 (= untouched)	7,473 rows	1e-8		1,198/1,319, score 86.17	control: 86.17
s1-q35-4b-full-lr125e-6	P3	Qwen3.5-4B	LoRA r32, packed (rows attend to each other)	7,473 rows	1.25e-5		1,074/1,319, score 76.77	collapsed
s2-q35-4b-full-lr8e-6	P3	Qwen3.5-4B	LoRA r32, packed	7,473 rows	8e-6		1,134/1,319, score 81.31	collapsed
s3-q35-4b-full-r16-lr1e5	P3	Qwen3.5-4B	LoRA r16, packed	7,473 rows	1e-5		1,170/1,319, score 84.04	partly collapsed
s4-q35-4b-full-lr1e5-replica	P3	Qwen3.5-4B	exact submitted config, packed	7,473 rows	1e-5		1,112/1,319, score 79.65	collapsed: 7.6 pts below the submission
on-full-lr2.5e-6	P4	Qwen3.5-4B	LoRA r32, packed	7,473 rows	2.5e-6	179/200 (base 180)
on-fullms-lr1e-5-ms16	P4	Qwen3.5-4B	LoRA r32, packed, 16 steps	7,473 rows	1e-5	175/200 (base 180)
on-targ960-lr1.6e-6	P4	Qwen3.5-4B	LoRA r32, unpacked	960 targeted rows	1.6e-6	180/200 (base 180)
on-targ960-lr2.08e-6	P4	Qwen3.5-4B	LoRA r32, unpacked	960 targeted rows	2.08e-6	175/200 (base 180)
on-targ960-lr3.2e-6	P4	Qwen3.5-4B	LoRA r32, unpacked	960 targeted rows	3.2e-6	173/200 (base 180)
on-targ960-lr4e-6	P4	Qwen3.5-4B	LoRA r32, unpacked	960 targeted rows	4e-6	169/200 (base 180)
on-sub960-lr4e-6	P4	Qwen3.5-4B	LoRA r32, unpacked	960 stratified rows	4e-6	168/200 (base 180)
on-sub960h40-lr4e-6	P4	Qwen3.5-4B	LoRA r32, unpacked	960 rows, 40 % hard	4e-6	168/200 (base 180)
on-sub960h40-lr2.4e-6	P4	Qwen3.5-4B	LoRA r32, unpacked	960 rows, 40 % hard	2.4e-6	175/200 (base 180)
on-sub960h40-lr1.68e-6	P4	Qwen3.5-4B	LoRA r32, unpacked	960 rows, 40 % hard	1.68e-6	458/500 (base 457)
on-targ960ep2-lr1.6e-6-ep2	P4	Qwen3.5-4B	LoRA r32, unpacked, 2 epochs	960 targeted rows	1.6e-6	173/200 (base 180)
on-targ960attn-lr4e-6-attn	P4	Qwen3.5-4B	LoRA attention-only	960 targeted rows	4e-6	178/200 (base 180)
on-targ960attn-lr5.2e-6-attn	P4	Qwen3.5-4B	LoRA attention-only	960 targeted rows	5.2e-6	181/200 (base 180)
on-targ960attn-lr8e-6-attn	P4	Qwen3.5-4B	LoRA attention-only	960 targeted rows	8e-6	176/200 (base 180)
on-targ960b4-lr8e-7-b4	P4	Qwen3.5-4B	LoRA r32, batch 4	960 targeted rows	8e-7	182/200 (base 180)
on-targ960b2-lr4e-7-b2	P4	Qwen3.5-4B	LoRA r32, batch 2	960 targeted rows	4e-7	180/200 (base 180)
on-fullft960-lr2e-7-fullft	P4	Qwen3.5-4B	Full fine-tune	960 targeted rows	2e-7	177/200 (base 180)
on-long960-lr2e-6	P4	Qwen3.5-4B	LoRA r32, unpacked	960 longest answers	2e-6	176/200 (base 180)
p1-q3-4b-untouched-targ960-b2	P5	Qwen3-4B	LoRA at LR 1e-8 (= untouched)	960 rows	1e-8	127/200 (base 180)		dead end
b1-q35-4b-r6data-lr2e-6	P5	Qwen3.5-4B	LoRA r32, packed	6,973 rows	2e-6	177/200 (base 180)		below gate
f1-full-lr2.5e-6-r32	P5	Qwen3.5-4B	LoRA r32, packed	7,473 rows	2.5e-6	180/200 (base 180)	1,198/1,319, score 86.17	= base on test
f3-full-lr5e-6-r16	P5	Qwen3.5-4B	LoRA r16, packed	7,473 rows	5e-6	179/200 (base 180)	1,197/1,319, score 86.09	= base on test
g1-full-b16-lr1e-6-r32	P5	Qwen3.5-4B	LoRA r32, unpacked, 468 steps	7,473 rows	1e-6	169/200 (base 180)		collapsed
g2-full-b16-lr5e-7-r32	P5	Qwen3.5-4B	LoRA r32, unpacked, 468 steps	7,473 rows	5e-7	176/200 (base 180)		below base
h2-fullft-b16-lr5e-8	P5	Qwen3.5-4B	Full fine-tune, unpacked	7,473 rows	5e-8	176/200 (base 180)		below base
h1-fullft-b16-lr1e-7	P5	Qwen3.5-4B	Full fine-tune, unpacked	7,473 rows	1e-7	174/200 (base 180)		below base
x1-exact-submitted-recipe	P6	Qwen3.5-4B	exact submitted config (now 935 unpacked steps)	7,473 rows	1e-5	27/40 (base 34)	1,058/1,319, score 75.55	collapsed
x2-submitted-ms48-b16	P6	Qwen3.5-4B	submitted config, max_steps 48, batch 16	7,473 rows	1e-5	32/40 (base 34)	1,135/1,319, score 81.39	collapsed
x3-submitted-ms48-b16-fs5	P6	Qwen3.5-4B	x2 on 5-shot-prompt rows	6,973 rows (eval format)	1e-5	29/40 (base 34)	1,058/1,319, score 75.55	collapsed
x4-submitted-ms48-b16-fs5-lr2.5e-6	P6	Qwen3.5-4B	x3 at a quarter LR	6,973 rows (eval format)	2.5e-6	33/40 (base 34)	1,197/1,319, score 86.09	= base
x5-submitted-ms48-b16-fs5-lr5e-6	P6	Qwen3.5-4B	x3 at half LR	6,973 rows (eval format)	5e-6		1,124/1,319, score 80.56	collapsed
x6-submitted-ms48-b16-lr5e-6	P6	Qwen3.5-4B	submitted config, max_steps 48, batch 16	7,473 rows	5e-6		1,205/1,319, score 86.70	best since the trainer change until x7
x7-submitted-ms48-b16-lr5e-6-r6data	P6	Qwen3.5-4B	x6 setting	6,973 rows	5e-6		1,208/1,319, score 86.92	best since the trainer change: 4 items short
x8-ms48-b16-lr5e-6-sub999	P6	Qwen3.5-4B	x6 setting	999 stratified rows	5e-6		1,202/1,319, score 86.47	below
x9-ms48-b16-lr5e-6-sub960	P6	Qwen3.5-4B	x6 setting	960 stratified rows	5e-6		1,198/1,319, score 86.17	= base

All of it in one file: retrospective.md · llms.txt · data/history.json