Skip to content

Add JSPaint drawing environment and circle verifier for mouse-only tasks - #36

Open
r33drichards wants to merge 24 commits into
mainfrom
claude/bold-babbage-czsvha
Open

r33drichards wants to merge 24 commits into
mainfrom
claude/bold-babbage-czsvha

Conversation

@r33drichards

Copy link
Copy Markdown
Owner

Summary

This PR adds a complete reinforcement learning environment for drawing tasks in JSPaint, an MS Paint clone running in headless Chromium. The model can only use the mouse to draw, receiving screenshots as observations and selecting from discrete compass-direction moves plus pen control actions. A circle verifier scores the final canvas pixels.

Key Changes

Core Environment (laya/paintenv.py)

  • JSPaintEnv: Full RL environment wrapping JSPaint in headless Chromium via Playwright
    • Configurable canvas size (default 512×512 to match vision tower input), step size (6px), and directions (8/16/32 compass points)
    • Observations: two screenshots (current and 8 steps ago) with cursor overlay, plus pen state and action history as text
    • Actions: relative moves in compass directions, pen down/up, and done
    • Reward: verifier score at episode end or shaped per-step
    • Memory: state() returns two-frame observations for temporal context
  • JSPaintServer: Serves JSPaint checkout as static files from a background thread
  • Reference policies: circle_expert (scripted), random_policy, and model_policy for benchmarking
  • Utilities: compass_moves(), compass_bearing() for action space, TOOLS schema for tool-calling agents

Circle Verifier (laya/circle_verifier.py)

  • Pure numpy circle scoring without browser dependency
  • Fits algebraic circle to ink pixels (Kasa method)
  • Scores on: roundness (radius consistency), coverage (angular sectors), cleanliness (ink on ring), closure (gap width), and size constraints
  • Passes fine-grained walks (16+ sides) and fails octagons, lines, arcs, squares, and scribbles

Data Labeling (laya/paintdata.py)

  • LabellerPolicy: State-based labeller that reads canvas pixels to infer the target circle and label actions
    • Infers circle from topmost ink (drawn clockwise from top)
    • Labels actions: move to start if needed, walk the ring, press/release pen, finish when complete
    • Handles stray ink gracefully (finishes anyway)
  • Judgement labels: progress (0-4 coverage), on_track (vs stray ink), drawn (nothing/arc/circle)
  • DAgger-style training: play_labelled_episode() runs episodes with behaviour policy (labeller + epsilon random) but labels with expert actions, spreading probability to neighboring compass points

Integration

  • Games: paint_question() in laya/games.py generates the full choice question with 32+ direction options and pen actions
  • Training: modal_app.py adds head_max_len parameter to support larger token budgets (1536/2560) for the full paint question (~1050 tokens)
  • VLM: laya/vlm.py tracks head_max_len separately from max_len for task-specific budgets

Examples and Tests

  • examples/jspaint_circle.py: Command-line runner for playing episodes with any policy, saving trajectories and GIFs
  • examples/jspaint_judge_check.py: Validates model judgements against true canvas state
  • tests/test_paintenv.py: End-to-end environment tests (skipped without Playwright/Chromium)
  • tests/test_circle_verifier.py: Verifier unit tests on synthetic shapes
  • tests/test_paintdata.py: Labeller tests on synthetic canvases
  • docs/paint-circle-metrics.json: Benchmark results (expert ~0.97, labeller ~0.94, model v3 0.61 with DAgger)

Documentation

  • site-docs/how-to/jspaint-env.md: Comprehensive guide covering observation/action design, canvas sizing rationale, move granularity trade-offs, and memory/question design

Notable Implementation Details

  • Canvas sizing: 512×512 (not JSPaint's default

https://claude.ai/code/session_01He3SxZcrbzN2YgsDSzN1m4

JSPaintEnv drives an unmodified JSPaint checkout in headless Chromium through
Playwright. The observation is a canvas screenshot with the cursor drawn on;
the actions are eight fixed-length moves plus pen down, pen up and done, sent
as real pointer events. The same moves are exposed as a mouse-only tool API
(TOOLS / call_tool) for tool-calling agents.

score_circle fits a least-squares circle to the canvas ink and scores fit,
angular coverage, cleanliness, closure and size. Rings and the octagon the
eight moves can draw pass; lines, arcs, squares, ellipses, discs, dots and
scribbles fail.

examples/jspaint_circle.py plays the model, a scripted circle expert and a
random policy on the same seeds and writes trajectories, GIFs and a summary.
Zero-shot thaitea/laya-vision scores 0 (near-uniform actions, early DONE);
the expert scores 0.76 with a 100% pass rate.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01He3SxZcrbzN2YgsDSzN1m4
…ifier

Eight 24 px moves could only draw an octagon. The action set is now a move
toward each of 16 compass points (every 22.5 degrees) of step_px (default 8,
max_steps 200); directions=8 and step_px stay configurable, and
paint_question builds the options from them.

The verifier now scores roundness as the spread of per-sector mean radii,
which cancels stroke width and measures the shape itself: a ring scores
0.99, a 16-gon 0.88, an octagon 0.56 (now failing), with the pass bar at
0.7. The scripted expert takes the forward move that stays closest to the
radius and scores 0.91 (was 0.76 as an octagon).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01He3SxZcrbzN2YgsDSzN1m4
The action set now defaults to a 6 px move toward each of 32 compass points
(every 11.25 degrees, max_steps 260); 8 and 16 directions stay available.
Each option's text states its bearing in degrees. The scripted expert walks
straight to the nearest point on the circle and presses only within half a
step of the radius, removing the nub where the stroke started off-circle.
Expert score rises from 0.91 to 0.95.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01He3SxZcrbzN2YgsDSzN1m4
The released checkpoints read each image as one 512x512 view and the
processor resizes to it without keeping the aspect ratio, so the 683x384
JSPaint canvas reached the model squashed: a drawn circle arrived as a tall
ellipse and the model called a perfect ring "blank". The environment now
sets JSPaint's canvas to canvas_size x canvas_size (default 512) through
localStorage before the page loads (JSPaint is unmodified), uses a 900x720
viewport so it fits, and checks both at reset. The runner sizes the canvas
from the loaded model's image_size.

On the 512 canvas laya-vision recognizes the scripted expert's circles
uncropped ("circle" at 0.67-0.94) and still calls a square a square; the
expert scores 0.97.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01He3SxZcrbzN2YgsDSzN1m4
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01He3SxZcrbzN2YgsDSzN1m4
…uncut

Each step the model now sees the canvas frame_gap (8) steps ago and now, plus
its pen state, step count and last 12 actions as state text (JSPaintEnv.state).

The 35-option drawing question with full descriptions is ~1,050 tokens; the
checkpoint's head_max_len of 256 had silently cut every option and 48 of the
82 instruction tokens. The runner now loads the model with
PAINT_HEAD_MAX_LEN=1536 / PAINT_MAX_LEN=2560, ModelPolicy calls predict with
strict=True so nothing can be cut silently, and a test checks the budget.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01He3SxZcrbzN2YgsDSzN1m4
paint_judgements adds progress (score 0-4), on_track (named choice) and
drawn (choice over shapes, incl. oval and arc), asked about the same state in
the same predict call as the action. ModelPolicy logs them with the act gate,
can end the episode on its own completion judgement (judge_stop), and the
runner labels each episode's clean final canvas (judged_final) for hindsight
relabelling.

examples/jspaint_judge_check.py compares the judgements with true progress
along expert and random play. Zero-shot, drawn works (3/3 circles) but
progress and on_track are nearly flat (Spearman 0.40 over a narrow range;
random play rated slightly higher), so they need training; the environment
can supply their labels.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01He3SxZcrbzN2YgsDSzN1m4
…aint, paint_eval

laya/paintdata.py labels any JSPaint state from the canvas as displayed: the
action (finish the target ring from wherever the cursor and ink are, DAgger
style, soft over neighbouring compass points) and the judgement questions
(progress, on track, drawn). JSPaintEnv.visible_pixels reads the canvas
including the stroke in progress (JSPaint commits strokes only on pointerup)
and hides the brush preview under the pointer. As a policy the labeller draws
a passing circle and stops itself (6/6, verifier 0.95-0.97).

env.state() is now {images, context}, the same shape as a training record's
images + state_text, so play and training build identical inputs.

The training loader (make_item) now takes head_max_len from the checkpoint
(processor.laya_head_max_len) instead of a fixed 256, and finetune_long takes
max_len / head_max_len overrides that also apply with --init-from and are
saved with the new checkpoint, so the ~1,050-token drawing question trains
and plays uncut.

modal_app.py: prepare_paint (episodes over parallel paint_shard containers,
JSPaint pinned at 53be67a, Chromium via playwright) writes
/data/vqa/paint_circle; paint_eval plays a checkpoint in JSPaint on an L4.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01He3SxZcrbzN2YgsDSzN1m4
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01He3SxZcrbzN2YgsDSzN1m4
A model trained on the first labeller walked past the start and never knew
when to press: the labeller always drew one fixed circle centred on the
canvas, so where to press was invisible. The labeller now draws the way a
person does: press where the cursor is (if a circle fits, else move away
from the visible edges), draw clockwise from the circle's top, and read the
rest of the target off the drawing (implied_circle: fixed radius 0.18 of the
side, top from a known-curvature parabola fit to the topmost ink). Actions and
judgement labels are measured against that implied circle. As a policy it
draws a passing circle and stops itself (6/6, verifier 0.92-0.94).

Data: a third of random-start episodes are messy (the random steps may draw),
so off-track states exist; the final canvas of each episode is judged too.
prepare_paint takes a dataset --name and refuses to replace an existing one
(prepared datasets are create-only). paint_eval adds the labeller as a
reference policy.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01He3SxZcrbzN2YgsDSzN1m4
play_labelled_episode takes a driver (a trained ModelPolicy) that takes the
step with probability 1 - beta instead of the labeller; every state is still
labelled with the labeller's action. prepare_paint --model runs the shards on
L4 containers (paint_shard_dagger). paint-circle-v2 draws the first half of
the circle correctly, then drifts off at the bottom into states the
labeller's data never visits; this collects exactly those states.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01He3SxZcrbzN2YgsDSzN1m4
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01He3SxZcrbzN2YgsDSzN1m4
paint-circle-v3 (one DAgger round on top of the visible-target labeller) draws
a closed circle in one stroke and ends the episode itself in 5/5 unseen
episodes, mean verifier 0.61, 2/5 passing on roundness; v1 and v2 scored 0.
Metrics per episode in docs/paint-circle-metrics.json.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01He3SxZcrbzN2YgsDSzN1m4
The model and labeller were only ever trained on circles; this lets a
checkpoint be asked for a square to see whether it reads the task at all.
There is no square verifier yet; the episode score is still the circle
verifier's.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01He3SxZcrbzN2YgsDSzN1m4
laya/quickdraw.py: fetch recognised doodles per category (range requests on
the simplified ndjson, CC BY 4.0; data stays on the Modal volume), place a
doodle on the canvas starting at the cursor, and DoodleLabeller, which
follows it from any state (walk to each stroke's start, press, trace, lift,
DONE; lift and return when the pen strays) with judgement labels from the
requested doodle. It reproduces human doodles in JSPaint and stops itself.

An independent scorer: a small CNN on rasterised doodles for 24 training and
6 held-out categories (DoodleScorer; JSPaintEnv(scorer=...)), trained on
doodles disjoint from the episodes'. A local 800-per-category version scores
84% on held-back doodles and 20/24 on the labeller's JSPaint canvases.

Task text works for any category (paint_goal, task_text); episodes take any
labeller; ModelPolicy caches questions per task.

modal_app.py: prepare_paint --categories train (doodle episodes),
train_quickdraw_classifier, doodle_eval (one L4 per category, summarised for
training vs held-out categories, labeller as ceiling and random as floor).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01He3SxZcrbzN2YgsDSzN1m4
A lost client connection while the parent waited on 48 doodle shards
cancelled all of them. Shards are now spawned individually and each result is
awaited with retries on connection errors; --resume keeps the .tmp directory
and skips shards whose file is already written.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01He3SxZcrbzN2YgsDSzN1m4
It had been inserted between @app.function and def prepare_paint, so the
decorator wrapped the helper and prepare_paint was no longer a Modal function.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01He3SxZcrbzN2YgsDSzN1m4
A doodle model toggled PEN_DOWN / PEN_UP for whole episodes: human doodles
start in many directions, so the probability of moving spread over the 32
compass options and PEN_UP (0.44) beat every single direction although moving
was likelier overall. grouped_choice sums the moves before comparing them with
the pen actions (on by default; grouped=False restores the plain argmax).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01He3SxZcrbzN2YgsDSzN1m4
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01He3SxZcrbzN2YgsDSzN1m4
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01He3SxZcrbzN2YgsDSzN1m4
grouped choice made the circle checkpoint worse (0.61 -> 0.51 on the same 5
seeds) and did not fix the doodle one, so the default is back to argmax.
sample draws move-vs-pen, then the direction, from the model's probabilities:
with many plausible doodles per category, sampling commits to one plan where
the argmax averages them. doodle_eval takes --decide.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01He3SxZcrbzN2YgsDSzN1m4
Asks the action question under each task prompt for the same val states and
reports the mean total-variation distance between the answers, the labelled
action's probability under its own prompt vs the others, and how often the
top action differs by prompt. A doodle checkpoint drew the same thing for
most categories; this measures that directly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01He3SxZcrbzN2YgsDSzN1m4

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants