From ce1e079e3bbd04aeb9660328aa2fa91ad843629a Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 1 Oct 2026 05:38:05 +0000 Subject: [PATCH 1/2] docs: add how-to guide for fine-tuning on your own dataset Covers the JSONL dataset format, a local fine-tune from the pinned published checkpoint with laya.vlm_train.train, finetune_long on Modal (volume upload, --init-from the Hub checkpoint, mixing with The Cauldron), and loading the result. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011MHDFrSKQo5r47X9SuTrpT --- mkdocs.yml | 1 + site-docs/how-to/fine-tune.md | 191 ++++++++++++++++++++++++++++++++++ site-docs/how-to/overview.md | 2 + 3 files changed, 194 insertions(+) create mode 100644 site-docs/how-to/fine-tune.md diff --git a/mkdocs.yml b/mkdocs.yml index 5440cdc..a4cb441 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -27,6 +27,7 @@ nav: - How-to: - Overview: how-to/overview.md + - Fine-tune on your own dataset: how-to/fine-tune.md - Calibrate on your own data: how-to/calibrate.md - Run jobs on Modal: how-to/run-on-modal.md - Evaluate a checkpoint: how-to/evaluate.md diff --git a/site-docs/how-to/fine-tune.md b/site-docs/how-to/fine-tune.md new file mode 100644 index 0000000..ad58297 --- /dev/null +++ b/site-docs/how-to/fine-tune.md @@ -0,0 +1,191 @@ +# Fine-tune on your own dataset + +The published checkpoint answers general questions about photos, charts and documents. If your images or questions +look different (a product catalogue, inspection photos, your own rubric), fine-tune it on a few hundred to a few +thousand labelled questions of your own. This guide writes the data, trains locally in Python or on a GPU on Modal, +and loads the result with `load_vlm`. + +If the model already gets your answers right and only its probabilities are off, you do not need to fine-tune: +[Calibrate on your own data](calibrate.md) instead. + +## 1. Write your data as JSONL + +A dataset is a directory with one JSONL file per split and the images next to it: + +```text +my_dataset/ + train.jsonl + val.jsonl + images/ + 0001.jpg + ... +``` + +Each line is one question about one image, written with the same question types as +[`predict`](../reference/predict.md#question-types), plus the index of the right answer: + +```json +{"id": "0001-color", "image": "images/0001.jpg", "state_text": "listing photo, seller note: barely used", + "question": {"type": "choice", "instructions": "What colour is the item?", "criteria": ["red", "blue", "green"]}, + "label": 1} +{"id": "0001-damaged", "image": "images/0001.jpg", + "question": {"type": "noul", "instructions": "Is the item visibly damaged?"}, + "label": 0} +{"id": "0001-condition", "image": "images/0001.jpg", + "question": {"type": "score", "instructions": "What condition is the item in?", + "criteria": ["poor: broken or missing parts", "fair: heavy wear", "good: light wear", "like new"]}, + "label": 2} +``` + +| Field | Meaning | +|---|---| +| `image` | Path relative to the dataset directory. `images` (a list of paths) instead, for a question about several images. Leave both out for a text-only question. | +| `state_text` | Optional text the model reads next to the image, like the extra keys of a `predict` state. | +| `question` | `type`, `instructions` and `criteria`, exactly as you will pass them to `predict`. | +| `label` | The index of the right option: the position in `criteria` for `choice`, the level for `score`, `0` (false) or `1` (true) for `noul`. | +| `target` | Optional: a probability per option, in the same order, replacing the one-hot target from `label`. Use it when your labels are votes, e.g. 3 of 5 raters said "good" (`[0, 0.2, 0.6, 0.2]`). `label` is still what accuracy is scored against. | +| `id` | Optional, carried through to the evaluation records. | + +Records with a `label` out of range, or a `choice` with duplicate option names, are skipped when loading. + +Some things that make the fine-tuned model better: + +- **Phrase questions the way you will ask them.** The model learns your instructions and option wording; ask the + same at inference. Several phrasings of one question in training make it less brittle. +- **Put every image's questions in the same split.** A question about an image the model has trained on is not a + fair validation question. +- **Keep a few hundred validation questions** per question you care about; accuracy on 30 questions moves by + ±15 points by chance. + +Check that the files load before you train: + +```python +from laya.vlm_train import load_jsonl_examples + +train_ex = load_jsonl_examples(".", "my_dataset", "train") # //.jsonl +print(len(train_ex), train_ex[0]["q"], train_ex[0]["target"]) +``` + +## 2a. Fine-tune locally + +For a small dataset, or to try the data before paying for a GPU, train in Python. This starts from the published +checkpoint, pinned to a Hub commit, so you keep what it already knows: + +```python +import laya +from laya.vlm_train import (collect_logits, fit_temperatures_from, format_metrics, load_jsonl_examples, + metrics_from, train) + +agent = laya.load_vlm("thaitea/laya-vision", revision="8b318c99d7ad3ce19c24369263463882eada9d1e") + +train_ex = load_jsonl_examples(".", "my_dataset", "train") +val_ex = load_jsonl_examples(".", "my_dataset", "val") +calib_ex, train_ex = train_ex[-300:], train_ex[:-300] # held out of training, to fit temperatures + +print("before:", format_metrics(metrics_from(collect_logits(agent.model, agent.processor, val_ex)))) + +train(agent.model, agent.processor, train_ex, steps=500, batch_size=8, freeze="head", + lr_head=1e-4, lr_backbone=2e-5, warmup=25, device=str(agent.device), log_every=50) + +agent.temperature = fit_temperatures_from(collect_logits(agent.model, agent.processor, calib_ex)) +val = collect_logits(agent.model, agent.processor, val_ex) +print("after:", format_metrics(metrics_from(val, agent.temperature))) + +agent.save("my-laya") +``` + +`freeze` picks what trains: + +| `freeze` | Trains | Use it when | +|---|---|---| +| `"head"` | The answer head only; the backbone is frozen. | You have a few hundred questions, or no GPU. Fast, and cannot forget what the backbone knows. | +| `"last_n"` | The head and the last `n_last` language-model layers. | A few thousand questions on images the head alone does not separate. | +| `"full"` | Everything except the vision tower. | Many thousands of questions, on a GPU. This is how the published checkpoints were trained. | + +- `steps × batch_size` is the number of questions seen; aim for 2 to 3 passes over your training set and watch the + validation accuracy, not the loss. `max_minutes=` caps wall-clock time instead. +- Options are shuffled for every training question, so the model cannot learn that the answer is usually first. +- Fitting the temperatures on held-out questions is what keeps `confidence` meaningful after training. With fewer + than about 300 training questions, hold out fewer (and expect noisier temperatures), or keep the checkpoint's by + skipping that line. +- Save the whole model (`agent.save(path)`, the default): `include_backbone=False` writes only the head and + reloads the backbone from the original pretrained weights, which is wrong for a checkpoint whose backbone was + itself fine-tuned, as the published one's was. + +On a CPU, each step at batch 4 takes a few seconds with `freeze="head"`; anything beyond the head wants a GPU. + +## 2b. Fine-tune on Modal + +For a larger dataset, `finetune_long` in [`modal_app.py`](https://github.com/r33drichards/laya-vision/blob/main/modal_app.py) +trains on an A100 with per-epoch evaluation, keeps the best checkpoint and fits the temperatures for you. The +volumes and setup it expects are in [Run jobs on Modal](run-on-modal.md). + +Upload the dataset to the `laya-datasets` volume under `vqa/`, with an empty `_READY` file that marks it +complete (jobs skip a dataset without one). Use a name that is not on the volume yet: prepared datasets are never +overwritten. + +```bash +touch my_dataset/_READY +modal volume put laya-datasets my_dataset /vqa/my_dataset +``` + +To start from the published checkpoint rather than from the bare backbone, put it on the `laya-checkpoints` volume +once, then pass it to `--init-from`: + +```bash +hf download thaitea/laya-vision --revision 8b318c99d7ad3ce19c24369263463882eada9d1e --local-dir laya-vision-8b318c9 +modal volume put laya-checkpoints laya-vision-8b318c9 /smolvlm/hub/laya-vision-8b318c9 + +modal run --detach modal_app.py::finetune_long --init-from hub/laya-vision-8b318c9 \ + --datasets my_dataset --val-datasets my_dataset,vqa --run-name my-dataset-3ep --epochs 3 --n-calib 300 +``` + +- `--run-name` must be new: the run saves to `/ckpt/smolvlm//`, with `best/`, `last/` and a + `metrics.json` of every evaluation. +- `--n-calib` questions from the end of each training file are held out to fit the temperatures; keep the + training file well above that, or lower it. +- `--val-datasets my_dataset,vqa` also scores A-OKVQA, ScienceQA and VQAv2 yes/no at every evaluation (when they + are prepared on your volume), so you see whether the model is forgetting its general skills. The best checkpoint is chosen by the mean accuracy over all + validation sets. +- To keep those skills, train on your data mixed with The Cauldron (prepared with `prepare_cauldron`): `--datasets my_dataset,cauldron` with + `--mix my_dataset=5` to draw your set five times as often as each Cauldron subset, and `--max-passes 4` to stop + a small set from repeating too often. +- Without `--init-from`, the run starts from the pretrained backbone (`--backbone`) with an untrained head; that + needs far more data than continuing from the checkpoint. + +Follow the run with `modal app logs laya-smolvlm`, then download the result: + +```bash +modal volume get laya-checkpoints /smolvlm/my-dataset-3ep/best my-laya +``` + +To score it on every prepared set, run `modal run modal_app.py::evaluate --run-name my-dataset-3ep/best --datasets +my_dataset,vqa`; the full suite is in [Evaluate a checkpoint](evaluate.md). + +## 3. Use the fine-tuned model + +A saved directory loads like a Hub checkpoint: + +```python +import laya +from PIL import Image + +agent = laya.load_vlm("my-laya") +result = agent.predict( + {"image": Image.open("photo.jpg"), "note": "listing photo, seller note: barely used"}, + {"condition": {"type": "score", "instructions": "What condition is the item in?", + "criteria": ["poor: broken or missing parts", "fair: heavy wear", "good: light wear", "like new"]}}, +) +``` + +The checkpoint records the Hub commit it started from (`loaded_from` in `vlm_agent_config.json`). To share it, +upload the directory to the Hub (`hf upload user/name my-laya`), and load it pinned to the commit that upload made, +as for the published checkpoint. + +## See also + +- [Training data](../concepts/data.md): the datasets the published checkpoints were trained on, in the same format. +- [How it works](../concepts/how-it-works.md): the answer head and the training objective. +- [Calibration](../concepts/calibration.md): what the fitted temperatures change. +- [`laya/vlm_train.py`](https://github.com/r33drichards/laya-vision/blob/main/laya/vlm_train.py): `train`, + `load_jsonl_examples` and every training option. diff --git a/site-docs/how-to/overview.md b/site-docs/how-to/overview.md index a29992c..ddb0493 100644 --- a/site-docs/how-to/overview.md +++ b/site-docs/how-to/overview.md @@ -6,6 +6,8 @@ These are **task-oriented** recipes for getting specific things done. They assum ## Guides +- [Fine-tune on your own dataset](fine-tune.md): write your labelled questions as JSONL and train the checkpoint + on them, locally or on Modal. - [Calibrate on your own data](calibrate.md): fit per-type temperatures on your labelled questions and check that they help. - [Run jobs on Modal](run-on-modal.md): the volumes and secret `modal_app.py` expects, and the commands for data From 36a6451e9a6cdd9317984fab7036cc0d089371e1 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 1 Oct 2026 05:38:15 +0000 Subject: [PATCH 2/2] docs: tidy fine-tune guide wording Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_011MHDFrSKQo5r47X9SuTrpT --- site-docs/how-to/fine-tune.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/site-docs/how-to/fine-tune.md b/site-docs/how-to/fine-tune.md index ad58297..39fc152 100644 --- a/site-docs/how-to/fine-tune.md +++ b/site-docs/how-to/fine-tune.md @@ -147,8 +147,8 @@ modal run --detach modal_app.py::finetune_long --init-from hub/laya-vision-8b318 - `--val-datasets my_dataset,vqa` also scores A-OKVQA, ScienceQA and VQAv2 yes/no at every evaluation (when they are prepared on your volume), so you see whether the model is forgetting its general skills. The best checkpoint is chosen by the mean accuracy over all validation sets. -- To keep those skills, train on your data mixed with The Cauldron (prepared with `prepare_cauldron`): `--datasets my_dataset,cauldron` with - `--mix my_dataset=5` to draw your set five times as often as each Cauldron subset, and `--max-passes 4` to stop +- To keep those skills, train on your data mixed with The Cauldron (prepared with `prepare_cauldron`): + `--datasets my_dataset,cauldron` with `--mix my_dataset=5` to draw your set five times as often as each Cauldron subset, and `--max-passes 4` to stop a small set from repeating too often. - Without `--init-from`, the run starts from the pretrained backbone (`--backbone`) with an untrained head; that needs far more data than continuing from the checkpoint. @@ -159,7 +159,7 @@ Follow the run with `modal app logs laya-smolvlm`, then download the result: modal volume get laya-checkpoints /smolvlm/my-dataset-3ep/best my-laya ``` -To score it on every prepared set, run `modal run modal_app.py::evaluate --run-name my-dataset-3ep/best --datasets +To score it on your validation set and the VQA sets, run `modal run modal_app.py::evaluate --run-name my-dataset-3ep/best --datasets my_dataset,vqa`; the full suite is in [Evaluate a checkpoint](evaluate.md). ## 3. Use the fine-tuned model