The single biggest constraint on model quality in this project is not the model size or the step count — it is that most of the training corpus does not fit in the context a free-tier GPU can hold.
These are R1-style reasoning traces. Measured across the whole of solutions_py_decontaminated:
- median: 13,770 estimated tokens
- p90: 23,374
- p99: 27,315
Yield by length bound:
max_tokens |
rows kept |
| 5,120 |
1,348 (17%) |
| 8,192 |
2,412 (30%) |
| 16,384 |
4,753 (58%) |
| 24,576 |
7,635 (94%) |
The current config uses 8,192, which is about the most a 3B in 4-bit holds on a 16GB T4 or P100 with room for activations. So 70% of the corpus is unreachable, and it is not a random 70% — long traces are long because the problems are hard, so what gets dropped is disproportionately the difficult end, which is exactly the part worth learning from.
Traces past the bound are dropped rather than truncated, deliberately: a truncated trace teaches the model to stop mid-thought. That is the right call given the choice, but it makes the bound expensive.
Options, roughly in order of how much they'd help per unit of effort:
- Shorter-trace source. Some traces are long because the model rambled, not because the problem is hard. Selecting the shortest passing trace per problem (rather than the first) would raise yield at the same bound with no hardware change. Cheap, and testable offline.
- Sequence packing with correct attention masking. Currently
packing=False in train/sft.py, on purpose, so two problems never blend into one window. Packing short traces together would improve throughput but not reach — it does not help a single 20k trace fit.
- Gradient checkpointing is already on. Beyond it, the levers are a smaller base model (more room for context) or offloading, both with real costs.
- Rented GPU. An A100 40GB holds a 3B at 24k+ context comfortably and would put ~94% of the corpus in range. This is the only option that actually solves it, and it costs money.
Worth measuring before choosing: does training on the short 30% actually underperform, and by how much? That needs the baseline run to exist first. Filed now so the constraint is recorded rather than rediscovered.
The single biggest constraint on model quality in this project is not the model size or the step count — it is that most of the training corpus does not fit in the context a free-tier GPU can hold.
These are R1-style reasoning traces. Measured across the whole of
solutions_py_decontaminated:Yield by length bound:
max_tokensThe current config uses 8,192, which is about the most a 3B in 4-bit holds on a 16GB T4 or P100 with room for activations. So 70% of the corpus is unreachable, and it is not a random 70% — long traces are long because the problems are hard, so what gets dropped is disproportionately the difficult end, which is exactly the part worth learning from.
Traces past the bound are dropped rather than truncated, deliberately: a truncated trace teaches the model to stop mid-thought. That is the right call given the choice, but it makes the bound expensive.
Options, roughly in order of how much they'd help per unit of effort:
packing=Falseintrain/sft.py, on purpose, so two problems never blend into one window. Packing short traces together would improve throughput but not reach — it does not help a single 20k trace fit.Worth measuring before choosing: does training on the short 30% actually underperform, and by how much? That needs the baseline run to exist first. Filed now so the constraint is recorded rather than rediscovered.