Skip to content

Support ModelScope-hosted checkpoints - #1225

Open
1195343015 wants to merge 1 commit into
vllm-project:mainfrom
1195343015:modelscope-checkpoint-links
Open

1195343015 wants to merge 1 commit into
vllm-project:mainfrom
1195343015:modelscope-checkpoint-links

Conversation

@1195343015

@1195343015 1195343015 commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Recipe authors can set hub: modelscope on a variant. Hugging Face remains the default. The selected hub supplies the checkpoint link, download environment, Docker cache mount, and benchmark environment. The builder, Markdown, and JSON expose the ModelScope client install step when the selected variant and hardware support pip.

ModelScope selection applies to the whole vLLM process, including external draft models and tokenizers. The builder and Markdown surface that constraint. The build flags recipes that set the environment variable manually or reference an external model with a ModelScope variant.

All prefill and decode node commands carry the hub environment and Docker cache. ModelScope variants fall back from Dynamo and SMG to vLLM Router until their tokenizer paths are verified. The JSON selection reports the requested router, the one used, and the reason. The Ascend proxy forwards HTTP requests to workers without loading a tokenizer, so supported Ascend presets retain their proxy.

Existing Eco-Tech variants can adopt the new field in their source recipes and overlays after this merges.

Testing

  • node scripts/build-recipes-api.mjs built the JSON API for 231 models.
  • node --test scripts/test-checkpoint-link.mjs scripts/test-model-hubs.mjs passed all 7 tests, including Docker shell and argv checks on all eight workers in a generated Ascend PD deployment. Ascend verification covered command generation only.
  • node --test scripts/test-*.mjs passed 174 of 175 tests. The preset-render parity test reported the same seven mismatches on upstream main using the same recipe content (six MiniMax points and one Mistral preset).

@vercel

vercel Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
vllm-recipes Ready Ready Preview Nov 16, 2026 2:01am UTC

Request Review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces support for custom checkpoint links (model_url) for model variants, allowing users to override the default Hugging Face repository link with alternative providers like ModelScope. It includes schema updates, a new ModelScope SVG logo, a helper utility to parse and validate HTTPS links, UI updates to display the correct provider icon, and comprehensive unit tests. The review feedback suggests making the Hugging Face provider detection more robust by also recognizing hf.co and hf-mirror.com domains, along with adding corresponding test cases.

Comment thread src/lib/checkpoint-link.js Outdated
Comment on lines +16 to +18
const provider = host === "modelscope.cn" || host.endsWith(".modelscope.cn")
? "ModelScope"
: host === "huggingface.co" ? "Hugging Face" : "Checkpoint";

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

To make the Hugging Face provider detection more robust, consider supporting other official domains like hf.co (and its subdomains) as well as the widely used hf-mirror.com mirror. This ensures that links using these domains are correctly identified as Hugging Face instead of falling back to the generic 'Checkpoint' provider.

  const isHuggingFace = host === "huggingface.co" || host.endsWith(".huggingface.co") ||
    host === "hf.co" || host.endsWith(".hf.co") ||
    host === "hf-mirror.com" || host.endsWith(".hf-mirror.com");
  const provider = host === "modelscope.cn" || host.endsWith(".modelscope.cn")
    ? "ModelScope"
    : isHuggingFace ? "Hugging Face" : "Checkpoint";

Comment thread scripts/test-checkpoint-link.mjs Outdated
Comment on lines +19 to +20
assert.equal(checkpointLink(recipe, { model_url: "https://www.modelscope.cn/models/org/base" }).provider, "ModelScope");
assert.equal(checkpointLink(recipe, { model_url: "https://example.org/model" }).provider, "Checkpoint");

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Add test cases for the alternative Hugging Face domains (hf.co and hf-mirror.com) to verify they are correctly identified as Hugging Face providers.

  assert.equal(checkpointLink(recipe, { model_url: "https://www.modelscope.cn/models/org/base" }).provider, "ModelScope");
  assert.equal(checkpointLink(recipe, { model_url: "https://hf.co/org/base" }).provider, "Hugging Face");
  assert.equal(checkpointLink(recipe, { model_url: "https://hf-mirror.com/org/base" }).provider, "Hugging Face");
  assert.equal(checkpointLink(recipe, { model_url: "https://example.org/model" }).provider, "Checkpoint");

@esmeetu esmeetu left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks — the problem is real: eight variants in today's recipes point at checkpoints that exist only on ModelScope (Eco-Tech/*; HF API returns 401, ModelScope 200), so their Hugging Face link is dead. I'd like to change the shape of the fix, though, because this PR changes the link but not where the command downloads from.

The problem with model_url

"This checkpoint lives on ModelScope" is one fact, but with this PR it has to be written twice: model_url for the link and a hand-written VLLM_USE_MODELSCOPE for the download. The two already disagree in today's recipes:

Variant VLLM_USE_MODELSCOPE set
Qwen3-30B-A3B ascend_w8a8, Qwen3.5-397B-A17B ×2, Qwen3.6-27B ×2, DeepSeek-V4-Flash ascend_w8a8 ✅
Qwen3.8-27B ascend_w8a8 ❌ (mentioned only in the guide)
Kimi-K3 ascend_w4a8 ❌

The last two render a vllm serve Eco-Tech/... that fetches from Hugging Face and gets a 401. With model_url added, their pages would show a ModelScope link above a command that still fails. A free-form URL also isn't needed, since ModelScope uses the same <org>/<repo> path (https://modelscope.cn/models/<model_id>). It would let any HTTPS link through, rendered as a generic "Checkpoint".

Suggested design: declare the hub, derive the rest

The same pattern as taxonomy.frontends ({ label, env }, enum generated into the schema, env merged by synthesis):

# taxonomy.yaml
model_hubs:
  huggingface:
    label: "Hugging Face"
    url: "https://huggingface.co/{model_id}"
    cache: "~/.cache/huggingface:/root/.cache/huggingface"
  modelscope:
    label: "ModelScope"
    url: "https://modelscope.cn/models/{model_id}"
    env: { VLLM_USE_MODELSCOPE: "True" }
    cache: "~/.cache/modelscope:/root/.cache/modelscope"
    install: "uv pip install modelscope"
# recipe
variants:
  ascend_w8a8:
    model_id: "Eco-Tech/Qwen3.8-27B-w8a8"
    hub: modelscope        # default: huggingface

The recipe states one fact; everything else follows from it:

Surface Today With hub
Link + icon hard-coded HF (CommandBuilder.jsx) url template
Download hand-written env, missing on 2 variants hub env merged at the base layer like the frontend env: every role, PD, Docker -e
Docker cache mount hard-coded ~/.cache/huggingface (buildDockerRun / buildDockerArgv) hub cache, so containers don't re-download on every start
pip install — modelscope isn't in vLLM's requirements, so the pip path needs uv pip install modelscope; both the upstream image (docker/Dockerfile) and the vllm-ascend image already install it
JSON API / .md — hub + checkpoint_url on each rendering

Two places outside vllm serve that load the tokenizer by model id, which I haven't verified. Could you check them?

  • Bench tab: vllm bench serve probably needs the same env prefix.
  • PD routers: SMG's gRPC mode loads the tokenizer in the router (that path also mounts the HF cache), and so may the Dynamo frontend. If they don't honor ModelScope, either pass the env through or disable those routers when hub isn't Hugging Face.

Known limit: VLLM_USE_MODELSCOPE is process-wide, so a --speculative-config draft model or a --tokenizer that lives only on Hugging Face would also be fetched from ModelScope. None of the eight variants do this today. A build warning for hub: modelscope plus an external model reference would catch it.

Scope

  • Variants only. Reject model.hub: site URLs, hf_id, HF dates and provider logos all assume the base model is on Hugging Face.
  • Not hardware-keyed. Hosting is a property of the checkpoint.
  • Not inferred from model_id or from the env.
  • User preference later. Downloading a both-hubs checkpoint from ModelScope (e.g. for users in China) is a separate axis. It can be added later as hubs: [...] plus a builder pill and ?hub= (in SELECTION_PARAMS) without migrating the single-value form.

Validation and migration

  • Schema: hub enum generated from taxonomy.model_hubs. Once migrated, the build rejects a hand-written VLLM_USE_MODELSCOPE (warn in the meantime).
  • Optional: fetch-hf-dates.mjs already queries the HF API per model; it could warn when a variant's model_id is 401/404 on Hugging Face and declares no hub. That would have caught the two broken variants.
  • Order: this PR (taxonomy + schema + synthesis + link), then the recipes in vllm and the vllm-ascend overlays switch the eight variants to hub: modelscope and drop the env, which also fixes Qwen3.8-27B and Kimi-K3.

The ModelScope logo, checkpointLink and the test layout carry over as they are; only the input changes from model_url to hub + the taxonomy entry. I'd keep the env, cache mount and pip dependency in this same PR, so a ModelScope link never sits above a command that downloads from Hugging Face.

@1195343015 1195343015 changed the title Support ModelScope checkpoint links Support ModelScope-hosted checkpoints Nov 9, 2026
@1195343015

Copy link
Copy Markdown
Contributor Author

Thanks for the detailed review. I changed this to variant-level hub metadata and derived the link, Engine environment, Docker cache mount, and pip dependency from it. I also added the process-wide draft/tokenizer warning, build warnings, and conservative fallbacks for Dynamo and SMG pending tokenizer verification. The existing variants can be migrated in their source repositories separately.

For a ModelScope main model with an independent Hugging Face draft, vllm-project/vllm#64197 proposes explicit per-model modelscope:// and hf:// prefixes. It is not merged yet and initially supports single-node execution, so this PR does not rely on or emit those prefixes.

@1195343015
1195343015 requested a review from esmeetu November 9, 2026 06:06
MAyang38 added a commit to MAyang38/vllm-ascend that referenced this pull request Nov 10, 2026
Declare hub: modelscope for A3 W4A8 and A5 MXFP4 so the renderer derives checkpoint links, download environment, and Docker cache mounts. Remove redundant hand-written ModelScope environment settings.

Requires the hub-aware renderer from vllm-project/recipes#1225.

Signed-off-by: mafajun <mafajun@huawei.com>

@esmeetu esmeetu left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for reworking this around model_hubs + variant hub. I re-reviewed 7561bc4 by building the JSON API for the base (cdaa860) and the PR from the same assembled content, and by rendering a test recipe with one HF variant and one hub: modelscope variant (single-node, multi-node, PD + Dynamo).

What I verified works

  • Existing output is unchanged: across all renderings the only difference is the new hub and checkpoint_url fields; no existing value changes, so no current command changes.
  • A ModelScope variant gets VLLM_USE_MODELSCOPE=True on single-node, the multi-node head and every worker, and both PD roles, including Docker -e. The cache mount switches to ~/.cache/modelscope in both the shell and argv Docker forms. The checkpoint link is https://modelscope.cn/models/<id>, and the page and .md show the modelscope pip step and the process-wide note.
  • Dynamo/SMG fall back to the vLLM router with router_unavailable_reason.
  • The build warns for the 4 recipes that still set VLLM_USE_MODELSCOPE by hand (Qwen3-30B-A3B, Qwen3.5-397B-A17B, Qwen3.6-27B, DeepSeek-V4-Flash).

Needed before merge: rebase

The PR conflicts with main. #1141, #1206 and #1226 changed the same code since cdaa860:

  1. The .md pip step for the hub checks recipe.model?.install?.pip !== false. After #1141, install is variant-aware (resolveInstall / isPipInstallAvailable); please use those, so a variant or GPU without a pip path doesn't show the ModelScope pip step.
  2. After #1226, PD renders per-node node_commands[]. Please make sure the hub env and cache mount reach every node command and its Docker form, and decide whether the new ascend router also needs the ModelScope fallback, or whether it works with ModelScope.

Suggested (non-blocking)

  1. JSON is missing the hub install step. The UI and .md show uv pip install 'modelscope>=1.18.1', but the JSON rendering's install is null, so an agent reading JSON doesn't get it. Please carry it in the rendering (for example with the other install/dependency steps).
  2. Avoid hard-coding hub === "modelscope". It appears in isDynamoSupportedOnHardware, smgUnsupportedReason, the router-unavailable reason, the UI tooltip, and the Bench prefix, which hard-codes the string VLLM_USE_MODELSCOPE=True instead of reading model_hubs.<hub>.env. The UI also compares the label "Hugging Face" instead of the hub id. Putting this on the taxonomy entry (for example a router allowlist on the hub, and the Bench prefix built from the hub's env) keeps the next hub data-only. It would also leave isDynamoSupportedOnHardware about hardware only, rather than taking a variant.
  3. .md shows the hub id. It prints Checkpoint: [modelscope](…) and adds Checkpoint: [huggingface](…) to every recipe's .md; it should use the label. You could also print the line only for a non-default hub.
  4. selection.adjusted doesn't report the router fallback. ?pd_router=dynamo on a ModelScope variant renders the vLLM router, but selection.adjusted is empty (resolveSelection only checks the id is dynamo/smg). The existing SMG fallback has the same gap; it would be good to report both.

Follow-up after merge

The Eco-Tech variants in the vllm recipes and vllm-ascend overlays should move to hub: modelscope (8 variants across 6 recipes). Qwen3.8-27B and Kimi-K3 don't set the env by hand, so the build won't warn about them, but their commands currently download from HF and get 401. Those two are the priority.

Signed-off-by: Jiayi Yan <1195343015@qq.com>

This branch was successfully deployed

1 active deployment
Preview — a60d28d9 Deployed Nov 16, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants