feat(protocol): MMCP 1.2 — motion and video references as prompts (0.9.0)

CHANGELOG.md

@@ -0,0 +1,112 @@
+# Changelog
+
+## 0.9.0
+
+MMCP protocol 1.2: motion and video references as prompts.
+
+### Added
+
+- `video_reference` segment (`VideoReferenceSegment`): a prompt given as a
+  video instead of text. `video` is exactly one of `{"url": "https://..."}`
+  (the server fetches it) or `{"data": <base64>, "media_type": "video/mp4" |
+  "video/quicktime" | "video/webm"}`; optional `start_s` / `end_s` trim
+  (`0 <= start_s < end_s`) and `fps` (a hint); the server follows the most
+  prominent person in the video. Otherwise it **is a text segment**,
+  validated with the text rules exactly as `motion_reference` is. The SDK
+  checks the payload (one source; an https URL with a host and no user
+  info, whitespace or bad port; standard padded base64, its size worked
+  out from its length; finite numbers; the trim; 422) and never decodes
+  the video: a backbone gets it parsed (`VideoSource.decoded()`,
+  `.num_bytes`).
+- `limits.max_video_bytes` (decoded inline data over it is
+  `413 payload_too_large`) and `limits.max_video_seconds` (a trimmed video
+  over it is `400 invalid_options`), both optional, `details.segment`
+  naming the segment, left out of `/capabilities` when unset. A URL's size
+  and an untrimmed video's length are the server's to enforce.
+- Models opt in by listing `"video_reference"` in `supported_segments`;
+  the existing gate answers `unsupported_segment` for models that don't.
+  No server implements it yet.
+- `GenerateRequest.video_reference` / `.video_references`, `VideoSource`.
+- `motionmcp.client.video_source(path | url=, max_bytes=)`: a video file
+  (base64, media type from the extension, size checked before reading) or
+  an https URL (the server's URL rule) as a `video` object. Standard
+  library only.
+- `motionmcp.fetch_video_url(url, max_bytes=, timeout=)`: fetch a
+  `video.url` without SSRF -- https only, every resolved address and every
+  redirect hop must be public (IPv6 forms that embed an IPv4 address --
+  mapped, compatible, NAT64, 6to4, Teredo -- judged by it), the connection
+  pinned to the checked address, a streamed byte cap (default 100 MiB) and
+  one hard deadline over connect, TLS, headers and body. Synchronous,
+  standard library only.
+- `motion_reference` segment (`MotionReferenceSegment`): a prompt given as a
+  motion instead of text. Send a clip (`joint_names`, per-frame local
+  `rotations` T×J×4, `root_positions` T×3, `fps`) where a text segment
+  carries its prompt; the model reads it as it reads a caption and makes new
+  motion of that kind. Otherwise the segment **is a text segment** and the
+  SDK validates it with the text rules: `duration_frames` (required),
+  per-segment `seed`, any number of them mixed in any order with `text` /
+  `unconditioned` / `pose` segments, `options.loop` (on a `supports_loop`
+  model, one segment), constraints over the whole take, `num_samples`,
+  guidance, batch bodies. The response is a text response's (no extra
+  metadata).
+- Its payload's own checks: T ≥ 2 and consistent lengths (422); joints on
+  the request skeleton, or on the segment's own `skeleton` when it carries
+  one (`unknown_joint`); a non-canonical segment `skeleton` needs
+  `supports_retargeting` (`retargeting_unsupported`).
+- `MotionReferenceSegment.skeleton` (optional, the request `Skeleton`
+  schema): the clip's own rig, so a reference can come from any rig.
+- `limits.max_reference_frames` (optional): when set, a clip longer than it
+  is `400 invalid_options` (`details.segment` names it). It counts frames
+  as sent: a server MUST accept a clip within it at any `fps` (resampling
+  or trimming to its own window). It caps the clip, not the output (that is
+  `max_duration_seconds`, as for text). Left out of `/capabilities` when
+  unset.
+- A motion reference must include its skeleton's root joint
+  (`400 invalid_request`), and its numbers must be finite (422).
+- Models opt in by listing `"motion_reference"` in `supported_segments`; the
+  existing gate still answers `unsupported_segment` for models that don't.
+  `supports_motion_reference_mixed` is advertised `true` exactly when a
+  model lists `"motion_reference"` (a reference always mixes like text).
+- `GenerateRequest.motion_reference` / `.motion_references`,
+  `MotionReferenceSegment.reference_frames`.
+- Error code `invalid_request` (400), for backbones.
+- Client: `poll_job(server_url, location, ...)` resumes polling an
+  accepted job, e.g. after refreshing a token on a 401 mid-poll, instead of
+  starting a second job. (#8)
+
+### Changed
+
+- `PROTOCOL_VERSION` is `"1.2"` (was `"1.0"`; 1.1 was never stamped here).
+  The SDK still sets no `options.guidance` default: a backbone applies its
+  text default, to a reference as to text.
+- Trajectory: the root moves along its shape (still / line / arc / Bezier,
+  chosen by fit) with the body's centre of mass, not at a constant speed, so
+  a stride's speeding up and slowing down no longer shows as the body
+  floating forward and back when played in place. Models are just
+  still/line/arc/bezier; params gain `speed_range` and drop the distance
+  keys. (#7)
+- Client: every `MmcpError` from `generate()` carries `details["phase"]`
+  (`"generate"` or `"poll"`); poll errors also carry `details["location"]`.
+  Codes and messages are unchanged. (#8)
+
+- *Clarified:* clients SHOULD send identity `rest_rotation` on every
+  joint (1.0 meanings unchanged: `rest_translation` / `rest_rotation` are
+  parent-local, `fill_mode: "rest"` pins to `rest_rotation`); a server that
+  can't honour a non-identity one MUST refuse it with `400 invalid_skeleton`.
+- The SDK server refuses a body over `limits.max_request_bytes` before
+  parsing it (`413 payload_too_large`: from `Content-Length`, or while
+  streaming one sent without), and each model's own cap after.
+- A segment type the SDK knows but the model doesn't list is
+  `unsupported_segment`, checked (after the protocol major) before the
+  segment's payload is parsed; a type the SDK doesn't know stays 422.
+- Batch items are held to their own model's `max_request_bytes`.
+
+### Fixed
+
+- A request failing one of the schema's own validators (e.g. `root_path`
+  lengths) is a `422 schema_validation` envelope with the message, not a
+  500 from an unserialisable error context; the envelope no longer echoes
+  the offending input (which could be megabytes of `video.data`). A body
+  nested too deeply to parse is a 422 envelope too, not a 500.
+- A skeleton with a bad topology (the request's or a segment's) is
+  `400 invalid_skeleton`, as documented, not `422 schema_validation`.

docs/docs/concepts/constraints.mdx

@@ -102,7 +102,8 @@ A pose at a single frame.
 Fill mode semantics:
 
 - **`rest`** — unspecified joints are pinned to their `rest_rotation` at
-  this frame.
+  this frame (identity, i.e. the rest pose, when identity rest rotations
+  are sent, as clients SHOULD; see [Rest rotations](./skeleton#rest-rotations)).
 - **`generate`** — unspecified joints are unconstrained; the backbone
   generates them as part of the motion. The constraint pins only the
   listed joints.

docs/docs/concepts/segments.mdx

@@ -1,7 +1,7 @@
 ---
 id: segments
 title: Segments
-description: How to compose a motion request from text, unconditioned, and pose spans.
+description: How to compose a motion request from text, unconditioned, pose, motion-reference and video-reference spans.
 sidebar_position: 2
 ---
 
@@ -11,9 +11,12 @@ A `Segment` describes one stretch of motion in time. Sequential segments
 compose into a clip — the request's total length is
 `sum(segments[].duration_frames)`. A single [`pose`](#posesegment--type-pose)
 segment is the one exception: it produces exactly one frame and can't be
-composed with others.
+composed with others. A [`motion_reference`](#motionreferencesegment--type-motion_reference)
+segment (1.2) is a prompt given as a clip instead of words, and a
+[`video_reference`](#videoreferencesegment--type-video_reference) segment
+(1.2) one given as a video; both compose exactly like a `text` segment.
 
-Three segment types are defined in v1:
+Five segment types are defined in v1:
 
 ```json
 // Multi-segment clip
@@ -27,13 +30,34 @@ Three segment types are defined in v1:
 [
   { "type": "pose", "prompt": "person waves hello with their right hand" }
 ]
+
+// Motion like a reference clip (1.2)
+[
+  { "type": "motion_reference", "duration_frames": 90, "joint_names": ["Hips", "Spine1"],
+    "rotations": [[[0, 0, 0, 1], [0, 0, 0, 1]], [[0, 0, 0, 1], [0, 0.01, 0, 1]]],
+    "root_positions": [[0, 0.95, 0], [0, 0.95, 0.03]], "fps": 30 }
+]
+
+// A walk like one clip, then a sit like another (1.2)
+[
+  { "type": "motion_reference", "duration_frames": 90,  "joint_names": [/* … */],
+    "rotations": [/* walk clip */], "root_positions": [/* … */], "fps": 30 },
+  { "type": "motion_reference", "duration_frames": 60,  "joint_names": [/* … */],
+    "rotations": [/* sit clip */],  "root_positions": [/* … */], "fps": 30 }
+]
+
+// A dance like the one in a video, seconds 2–6 of it (1.2)
+[
+  { "type": "video_reference", "duration_frames": 120,
+    "video": { "url": "https://cdn.example.com/dance.mp4" }, "start_s": 2, "end_s": 6 }
+]
 ```
 
 `Segment` is a **discriminated union** with `type` as the discriminator.
 Servers advertise which types they accept via
 `ModelSpec.supported_segments` in `/capabilities` — clients SHOULD
 check before sending a non-default type. Future minor versions MAY add
-types (e.g. `"audio"`, `"motion_reference"`) under the same key.
+types (e.g. `"audio"`) under the same key.
 
 ## TextSegment — `type: "text"`
 
@@ -96,6 +120,165 @@ rejects pose segments sent to a model that doesn't claim support, with
 the `unsupported_segment` error code. See
 **[Capabilities reference →](../reference/capabilities)**.
 
+## MotionReferenceSegment — `type: "motion_reference"`
+
+*Since 1.2.* A **prompt given as a motion instead of words**. The client
+sends a clip sampled from its own rig (a take, hand keys, mocap) where a
+text segment would carry its prompt; the model reads the clip the way it
+reads a caption — a backbone turns it into the same kind of conditioning
+vector its text encoder makes for a prompt — and generates new motion *of
+that kind*, not a copy of the clip. Apart from that, a reference segment
+**is a text segment**: it covers `duration_frames` of the take, takes a
+`seed`, and mixes with `text`, `unconditioned` and other reference segments
+in any order, stitched with the same transitions; constraints,
+`options.loop`, `num_samples`, guidance and batching apply exactly as they
+do to text, and the SDK validates it with the text segment's rules. The
+clip can come from **any rig**: send that rig as the segment's own
+`skeleton` and the server retargets the clip from it; the output is always
+on the request `skeleton`.
+
+```json
+{
+  "type": "motion_reference",
+  "duration_frames": 150,
+  "skeleton": { "joints": [ /* the clip's rig, optional */ ] },
+  "joint_names": ["Hips", "Spine1", "..."],
+  "rotations": [[[0, 0, 0, 1], "... one per joint"], "... one per frame"],
+  "root_positions": [[0, 0.95, 0], "... one per frame"],
+  "fps": 30
+}
+```
+
+| Field | Type | Required | Notes |
+|---|---|---|---|
+| `type` | `"motion_reference"` | yes | Discriminator |
+| `duration_frames` | int (≥ 1) | yes | Output length, in frames at the request's effective fps, as for `text`. Independent of the clip's length |
+| `skeleton` | [`Skeleton`](./skeleton), same schema as the request `skeleton` | no | The reference clip's own rig. When present, `joint_names`, `rotations` and `root_positions` refer to it and the server retargets the clip from it; when absent they refer to the request `skeleton` |
+| `joint_names` | string[] (unique, ≥ 1) | yes | Joints of the segment's `skeleton` if given, else of the request `skeleton`; named as in `pose_keyframe.joint_rotations` |
+| `rotations` | `[x, y, z, w][][]`, T × J | yes | Local-to-parent quaternions per frame, one per `joint_names` entry, in the same convention and order as `pose_keyframe.joint_rotations`. T ≥ 2, at most `limits.max_reference_frames` |
+| `root_positions` | `[x, y, z][]`, T | yes | The root joint's world position per frame, Y-up metres, as `pose_keyframe.root_position` |
+| `fps` | number (> 0) | yes | Frame rate of the clip's samples (may differ from the request's) |
+| `seed` | int | no | Per-segment seed, as [`TextSegment.seed`](#textsegment--type-text) |
+
+To build one from a rig (the conventions, sampling a clip in numpy or
+Blender, BVH and FBX, the size budget), see
+**[Building reference segments →](../guides/reference-segments)**.
+
+The payload's own checks:
+
+- `len(rotations) == len(root_positions) == T`, every frame has one
+  quaternion per `joint_names` entry, and T ≥ 2 → otherwise
+  `422 schema_validation`. Every number must be finite (no `NaN` /
+  `Infinity`). Quaternions are taken as sent (as for `pose_keyframe`): not
+  required to be exactly unit length.
+- Every `joint_names` entry must be a joint of the segment's `skeleton`
+  when it has one, else of the request `skeleton` → `unknown_joint`. The
+  clip must include that skeleton's root joint → `400 invalid_request`.
+- A segment `skeleton` with a bad topology (no root, several roots, a
+  parent listed after its child) → `400 invalid_skeleton`, as for the
+  request skeleton.
+- A segment `skeleton` other than the model's canonical skeleton needs a
+  model with `supports_retargeting` → otherwise `retargeting_unsupported`.
+- A clip longer than `limits.max_reference_frames` (when set) →
+  `400 invalid_options`. `limits.max_reference_frames` counts the clip's frames **as sent** (`len(rotations)`), whatever its `fps`: a server MUST accept a clip within it, and MUST NOT refuse one within it because of how it resamples the clip internally (it resamples or trims to fit its own window instead).
+
+Everything else is the `text` segment's rule, unchanged. The response is
+the usual glTF, exactly as for a text request.
+
+`MotionReferenceSegment` is an **optional** capability: a model lists
+`"motion_reference"` in `supported_segments` and SHOULD set
+`limits.max_reference_frames`. The SDK rejects the segment for a model that
+doesn't (`unsupported_segment`), before reading the segment's payload.
+`supports_motion_reference_mixed` is `true` exactly when a model lists
+`"motion_reference"` (a reference mixes like text).
+
+## VideoReferenceSegment — `type: "video_reference"`
+
+*Since 1.2.* A **prompt given as a video instead of words**: footage of a
+person moving — a phone clip, a reference from the web, a performance
+capture pass. The model reads the person's motion in it as it reads a
+caption and generates new motion *of that kind* on the request `skeleton`,
+not a copy of the footage. It is to video what
+[`motion_reference`](#motionreferencesegment--type-motion_reference) is to
+a clip, and in every other respect it **is a text segment**: it covers
+`duration_frames` of the take, takes a `seed`, and mixes with `text`,
+`unconditioned`, `motion_reference` and other video segments in any order;
+constraints, `options.loop`, `num_samples`, guidance and batching apply
+exactly as they do to text, and the SDK validates it with the text
+segment's rules.
+
+```json
+// The server fetches the video
+{
+  "type": "video_reference",
+  "duration_frames": 120,
+  "video": { "url": "https://cdn.example.com/dance.mp4" },
+  "start_s": 2.0,
+  "end_s": 6.0
+}
+
+// The video inline
+{
+  "type": "video_reference",
+  "duration_frames": 90,
+  "video": { "data": "AAAAGGZ0eXBtcDQy…", "media_type": "video/mp4" },
+  "fps": 30
+}
+```
+
+| Field | Type | Required | Notes |
+|---|---|---|---|
+| `type` | `"video_reference"` | yes | Discriminator |
+| `duration_frames` | int (≥ 1) | yes | Output length, in frames at the request's effective fps, as for `text`. Independent of the video's length |
+| `video` | object, exactly one of the two forms below | yes | The video |
+| `video.url` | string, an `https` URL with a host | one of | The server fetches it. No other fields go with it |
+| `video.data` | string, standard base64 (RFC 4648, padded) | one of | The video file inline, non-empty |
+| `video.media_type` | `"video/mp4"` \| `"video/quicktime"` \| `"video/webm"` | with `data` | The container of `data` |
+| `start_s` | number (≥ 0) | no | Where to start reading, in seconds into the video. Default: the start |
+| `end_s` | number (> `start_s`) | no | Where to stop reading, in seconds into the video. Default: the end |
+| `fps` | number (> 0) | no | A hint for the video's frame rate, for containers that report it badly |
+| `seed` | int | no | Per-segment seed, as [`TextSegment.seed`](#textsegment--type-text) |
+
+Unknown fields are rejected (`422`), in the segment and in `video`.
+
+**Which person.** The server follows the **most prominent person** in the
+video — the largest, most visible one over the stretch it reads. There is
+no field to pick another in 1.2; trim or crop the video so the person you
+want is the clear subject. (Choosing among several people may come in a
+later version; see the [roadmap](../protocol/roadmap).)
+
+To build one (inline or by URL, the size math, trimming),
+see **[Building reference segments →](../guides/reference-segments)**.
+
+The payload's own checks:
+
+- `video` has exactly one of `url` / `data`; a `url` is `https` with a host
+  and has no `media_type`; `data` is valid, non-empty base64 with one of
+  the three `media_type`s; `0 ≤ start_s < end_s`; `fps > 0`
+  → otherwise `422 schema_validation`.
+- Inline `data` whose decoded size is over `limits.max_video_bytes` (when
+  set) → `413 payload_too_large`, `details.segment` naming it. The whole
+  body must still fit `limits.max_request_bytes`; base64 is a third bigger
+  than the file.
+- `end_s − (start_s or 0)` over `limits.max_video_seconds` (when set) →
+  `400 invalid_options`. It caps the video read, not the output (that is
+  `max_duration_seconds`, as for text).
+- What the SDK cannot see, the server checks once it has the video: a
+  `url`'s size against `max_video_bytes`, an untrimmed video's length
+  against `max_video_seconds`, and whether the video can be fetched and
+  decoded and has a person in it (`400 invalid_request`).
+
+Everything else is the `text` segment's rule, unchanged. The response is
+the usual glTF, exactly as for a text request. The SDK never decodes video:
+a backbone receives the segment parsed (`VideoReferenceSegment`, with
+`video.decoded()` for inline data) and reads the video itself.
+
+`VideoReferenceSegment` is an **optional** capability: a model lists
+`"video_reference"` in `supported_segments` and SHOULD set
+`limits.max_video_bytes` and `limits.max_video_seconds`. The SDK rejects the
+segment for a model that doesn't list it (`unsupported_segment`). No server
+implements it yet.
+
 ## Composing
 
 ```json