Repository navigation
Add JSPaint drawing environment and circle verifier for mouse-only tasks - #36
Open
r33drichards wants to merge 24 commits into
Open
r33drichards wants to merge 24 commits into
r33drichards wants to merge 24 commits into
Conversation
JSPaintEnv drives an unmodified JSPaint checkout in headless Chromium through Playwright. The observation is a canvas screenshot with the cursor drawn on; the actions are eight fixed-length moves plus pen down, pen up and done, sent as real pointer events. The same moves are exposed as a mouse-only tool API (TOOLS / call_tool) for tool-calling agents. score_circle fits a least-squares circle to the canvas ink and scores fit, angular coverage, cleanliness, closure and size. Rings and the octagon the eight moves can draw pass; lines, arcs, squares, ellipses, discs, dots and scribbles fail. examples/jspaint_circle.py plays the model, a scripted circle expert and a random policy on the same seeds and writes trajectories, GIFs and a summary. Zero-shot thaitea/laya-vision scores 0 (near-uniform actions, early DONE); the expert scores 0.76 with a 100% pass rate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01He3SxZcrbzN2YgsDSzN1m4
…ifier Eight 24 px moves could only draw an octagon. The action set is now a move toward each of 16 compass points (every 22.5 degrees) of step_px (default 8, max_steps 200); directions=8 and step_px stay configurable, and paint_question builds the options from them. The verifier now scores roundness as the spread of per-sector mean radii, which cancels stroke width and measures the shape itself: a ring scores 0.99, a 16-gon 0.88, an octagon 0.56 (now failing), with the pass bar at 0.7. The scripted expert takes the forward move that stays closest to the radius and scores 0.91 (was 0.76 as an octagon). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01He3SxZcrbzN2YgsDSzN1m4
The action set now defaults to a 6 px move toward each of 32 compass points (every 11.25 degrees, max_steps 260); 8 and 16 directions stay available. Each option's text states its bearing in degrees. The scripted expert walks straight to the nearest point on the circle and presses only within half a step of the radius, removing the nub where the stroke started off-circle. Expert score rises from 0.91 to 0.95. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01He3SxZcrbzN2YgsDSzN1m4
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01He3SxZcrbzN2YgsDSzN1m4
The released checkpoints read each image as one 512x512 view and the
processor resizes to it without keeping the aspect ratio, so the 683x384
JSPaint canvas reached the model squashed: a drawn circle arrived as a tall
ellipse and the model called a perfect ring "blank". The environment now
sets JSPaint's canvas to canvas_size x canvas_size (default 512) through
localStorage before the page loads (JSPaint is unmodified), uses a 900x720
viewport so it fits, and checks both at reset. The runner sizes the canvas
from the loaded model's image_size.
On the 512 canvas laya-vision recognizes the scripted expert's circles
uncropped ("circle" at 0.67-0.94) and still calls a square a square; the
expert scores 0.97.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01He3SxZcrbzN2YgsDSzN1m4
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01He3SxZcrbzN2YgsDSzN1m4
…uncut Each step the model now sees the canvas frame_gap (8) steps ago and now, plus its pen state, step count and last 12 actions as state text (JSPaintEnv.state). The 35-option drawing question with full descriptions is ~1,050 tokens; the checkpoint's head_max_len of 256 had silently cut every option and 48 of the 82 instruction tokens. The runner now loads the model with PAINT_HEAD_MAX_LEN=1536 / PAINT_MAX_LEN=2560, ModelPolicy calls predict with strict=True so nothing can be cut silently, and a test checks the budget. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01He3SxZcrbzN2YgsDSzN1m4
paint_judgements adds progress (score 0-4), on_track (named choice) and drawn (choice over shapes, incl. oval and arc), asked about the same state in the same predict call as the action. ModelPolicy logs them with the act gate, can end the episode on its own completion judgement (judge_stop), and the runner labels each episode's clean final canvas (judged_final) for hindsight relabelling. examples/jspaint_judge_check.py compares the judgements with true progress along expert and random play. Zero-shot, drawn works (3/3 circles) but progress and on_track are nearly flat (Spearman 0.40 over a narrow range; random play rated slightly higher), so they need training; the environment can supply their labels. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01He3SxZcrbzN2YgsDSzN1m4
…aint, paint_eval
laya/paintdata.py labels any JSPaint state from the canvas as displayed: the
action (finish the target ring from wherever the cursor and ink are, DAgger
style, soft over neighbouring compass points) and the judgement questions
(progress, on track, drawn). JSPaintEnv.visible_pixels reads the canvas
including the stroke in progress (JSPaint commits strokes only on pointerup)
and hides the brush preview under the pointer. As a policy the labeller draws
a passing circle and stops itself (6/6, verifier 0.95-0.97).
env.state() is now {images, context}, the same shape as a training record's
images + state_text, so play and training build identical inputs.
The training loader (make_item) now takes head_max_len from the checkpoint
(processor.laya_head_max_len) instead of a fixed 256, and finetune_long takes
max_len / head_max_len overrides that also apply with --init-from and are
saved with the new checkpoint, so the ~1,050-token drawing question trains
and plays uncut.
modal_app.py: prepare_paint (episodes over parallel paint_shard containers,
JSPaint pinned at 53be67a, Chromium via playwright) writes
/data/vqa/paint_circle; paint_eval plays a checkpoint in JSPaint on an L4.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01He3SxZcrbzN2YgsDSzN1m4
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01He3SxZcrbzN2YgsDSzN1m4
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01He3SxZcrbzN2YgsDSzN1m4
A model trained on the first labeller walked past the start and never knew when to press: the labeller always drew one fixed circle centred on the canvas, so where to press was invisible. The labeller now draws the way a person does: press where the cursor is (if a circle fits, else move away from the visible edges), draw clockwise from the circle's top, and read the rest of the target off the drawing (implied_circle: fixed radius 0.18 of the side, top from a known-curvature parabola fit to the topmost ink). Actions and judgement labels are measured against that implied circle. As a policy it draws a passing circle and stops itself (6/6, verifier 0.92-0.94). Data: a third of random-start episodes are messy (the random steps may draw), so off-track states exist; the final canvas of each episode is judged too. prepare_paint takes a dataset --name and refuses to replace an existing one (prepared datasets are create-only). paint_eval adds the labeller as a reference policy. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01He3SxZcrbzN2YgsDSzN1m4
play_labelled_episode takes a driver (a trained ModelPolicy) that takes the step with probability 1 - beta instead of the labeller; every state is still labelled with the labeller's action. prepare_paint --model runs the shards on L4 containers (paint_shard_dagger). paint-circle-v2 draws the first half of the circle correctly, then drifts off at the bottom into states the labeller's data never visits; this collects exactly those states. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01He3SxZcrbzN2YgsDSzN1m4
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01He3SxZcrbzN2YgsDSzN1m4
paint-circle-v3 (one DAgger round on top of the visible-target labeller) draws a closed circle in one stroke and ends the episode itself in 5/5 unseen episodes, mean verifier 0.61, 2/5 passing on roundness; v1 and v2 scored 0. Metrics per episode in docs/paint-circle-metrics.json. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01He3SxZcrbzN2YgsDSzN1m4
The model and labeller were only ever trained on circles; this lets a checkpoint be asked for a square to see whether it reads the task at all. There is no square verifier yet; the episode score is still the circle verifier's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01He3SxZcrbzN2YgsDSzN1m4
laya/quickdraw.py: fetch recognised doodles per category (range requests on the simplified ndjson, CC BY 4.0; data stays on the Modal volume), place a doodle on the canvas starting at the cursor, and DoodleLabeller, which follows it from any state (walk to each stroke's start, press, trace, lift, DONE; lift and return when the pen strays) with judgement labels from the requested doodle. It reproduces human doodles in JSPaint and stops itself. An independent scorer: a small CNN on rasterised doodles for 24 training and 6 held-out categories (DoodleScorer; JSPaintEnv(scorer=...)), trained on doodles disjoint from the episodes'. A local 800-per-category version scores 84% on held-back doodles and 20/24 on the labeller's JSPaint canvases. Task text works for any category (paint_goal, task_text); episodes take any labeller; ModelPolicy caches questions per task. modal_app.py: prepare_paint --categories train (doodle episodes), train_quickdraw_classifier, doodle_eval (one L4 per category, summarised for training vs held-out categories, labeller as ceiling and random as floor). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01He3SxZcrbzN2YgsDSzN1m4
A lost client connection while the parent waited on 48 doodle shards cancelled all of them. Shards are now spawned individually and each result is awaited with retries on connection errors; --resume keeps the .tmp directory and skips shards whose file is already written. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01He3SxZcrbzN2YgsDSzN1m4
It had been inserted between @app.function and def prepare_paint, so the decorator wrapped the helper and prepare_paint was no longer a Modal function. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01He3SxZcrbzN2YgsDSzN1m4
A doodle model toggled PEN_DOWN / PEN_UP for whole episodes: human doodles start in many directions, so the probability of moving spread over the 32 compass options and PEN_UP (0.44) beat every single direction although moving was likelier overall. grouped_choice sums the moves before comparing them with the pen actions (on by default; grouped=False restores the plain argmax). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01He3SxZcrbzN2YgsDSzN1m4
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01He3SxZcrbzN2YgsDSzN1m4
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01He3SxZcrbzN2YgsDSzN1m4
grouped choice made the circle checkpoint worse (0.61 -> 0.51 on the same 5 seeds) and did not fix the doodle one, so the default is back to argmax. sample draws move-vs-pen, then the direction, from the model's probabilities: with many plausible doodles per category, sampling commits to one plan where the argmax averages them. doodle_eval takes --decide. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01He3SxZcrbzN2YgsDSzN1m4
Asks the action question under each task prompt for the same val states and reports the mean total-variation distance between the answers, the labelled action's probability under its own prompt vs the others, and how often the top action differs by prompt. A doodle checkpoint drew the same thing for most categories; this measures that directly. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01He3SxZcrbzN2YgsDSzN1m4
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds a complete reinforcement learning environment for drawing tasks in JSPaint, an MS Paint clone running in headless Chromium. The model can only use the mouse to draw, receiving screenshots as observations and selecting from discrete compass-direction moves plus pen control actions. A circle verifier scores the final canvas pixels.
Key Changes
Core Environment (
laya/paintenv.py)state()returns two-frame observations for temporal contextcircle_expert(scripted),random_policy, andmodel_policyfor benchmarkingcompass_moves(),compass_bearing()for action space,TOOLSschema for tool-calling agentsCircle Verifier (
laya/circle_verifier.py)Data Labeling (
laya/paintdata.py)play_labelled_episode()runs episodes with behaviour policy (labeller + epsilon random) but labels with expert actions, spreading probability to neighboring compass pointsIntegration
paint_question()inlaya/games.pygenerates the full choice question with 32+ direction options and pen actionsmodal_app.pyaddshead_max_lenparameter to support larger token budgets (1536/2560) for the full paint question (~1050 tokens)laya/vlm.pytrackshead_max_lenseparately frommax_lenfor task-specific budgetsExamples and Tests
examples/jspaint_circle.py: Command-line runner for playing episodes with any policy, saving trajectories and GIFsexamples/jspaint_judge_check.py: Validates model judgements against true canvas statetests/test_paintenv.py: End-to-end environment tests (skipped without Playwright/Chromium)tests/test_circle_verifier.py: Verifier unit tests on synthetic shapestests/test_paintdata.py: Labeller tests on synthetic canvasesdocs/paint-circle-metrics.json: Benchmark results (expert ~0.97, labeller ~0.94, model v3 0.61 with DAgger)Documentation
site-docs/how-to/jspaint-env.md: Comprehensive guide covering observation/action design, canvas sizing rationale, move granularity trade-offs, and memory/question designNotable Implementation Details
https://claude.ai/code/session_01He3SxZcrbzN2YgsDSzN1m4