Repository navigation
Support ModelScope-hosted checkpoints - #1225
1195343015 wants to merge 1 commit into
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
There was a problem hiding this comment.
Code Review
This pull request introduces support for custom checkpoint links (model_url) for model variants, allowing users to override the default Hugging Face repository link with alternative providers like ModelScope. It includes schema updates, a new ModelScope SVG logo, a helper utility to parse and validate HTTPS links, UI updates to display the correct provider icon, and comprehensive unit tests. The review feedback suggests making the Hugging Face provider detection more robust by also recognizing hf.co and hf-mirror.com domains, along with adding corresponding test cases.
| const provider = host === "modelscope.cn" || host.endsWith(".modelscope.cn") | ||
| ? "ModelScope" | ||
| : host === "huggingface.co" ? "Hugging Face" : "Checkpoint"; |
There was a problem hiding this comment.
To make the Hugging Face provider detection more robust, consider supporting other official domains like hf.co (and its subdomains) as well as the widely used hf-mirror.com mirror. This ensures that links using these domains are correctly identified as Hugging Face instead of falling back to the generic 'Checkpoint' provider.
const isHuggingFace = host === "huggingface.co" || host.endsWith(".huggingface.co") ||
host === "hf.co" || host.endsWith(".hf.co") ||
host === "hf-mirror.com" || host.endsWith(".hf-mirror.com");
const provider = host === "modelscope.cn" || host.endsWith(".modelscope.cn")
? "ModelScope"
: isHuggingFace ? "Hugging Face" : "Checkpoint";| assert.equal(checkpointLink(recipe, { model_url: "https://www.modelscope.cn/models/org/base" }).provider, "ModelScope"); | ||
| assert.equal(checkpointLink(recipe, { model_url: "https://example.org/model" }).provider, "Checkpoint"); |
There was a problem hiding this comment.
Add test cases for the alternative Hugging Face domains (hf.co and hf-mirror.com) to verify they are correctly identified as Hugging Face providers.
assert.equal(checkpointLink(recipe, { model_url: "https://www.modelscope.cn/models/org/base" }).provider, "ModelScope");
assert.equal(checkpointLink(recipe, { model_url: "https://hf.co/org/base" }).provider, "Hugging Face");
assert.equal(checkpointLink(recipe, { model_url: "https://hf-mirror.com/org/base" }).provider, "Hugging Face");
assert.equal(checkpointLink(recipe, { model_url: "https://example.org/model" }).provider, "Checkpoint");
esmeetu
left a comment
There was a problem hiding this comment.
Thanks — the problem is real: eight variants in today's recipes point at checkpoints that exist only on ModelScope (Eco-Tech/*; HF API returns 401, ModelScope 200), so their Hugging Face link is dead. I'd like to change the shape of the fix, though, because this PR changes the link but not where the command downloads from.
The problem with model_url
"This checkpoint lives on ModelScope" is one fact, but with this PR it has to be written twice: model_url for the link and a hand-written VLLM_USE_MODELSCOPE for the download. The two already disagree in today's recipes:
| Variant | VLLM_USE_MODELSCOPE set |
|---|---|
Qwen3-30B-A3B ascend_w8a8, Qwen3.5-397B-A17B ×2, Qwen3.6-27B ×2, DeepSeek-V4-Flash ascend_w8a8 |
✅ |
Qwen3.8-27B ascend_w8a8 |
❌ (mentioned only in the guide) |
Kimi-K3 ascend_w4a8 |
❌ |
The last two render a vllm serve Eco-Tech/... that fetches from Hugging Face and gets a 401. With model_url added, their pages would show a ModelScope link above a command that still fails. A free-form URL also isn't needed, since ModelScope uses the same <org>/<repo> path (https://modelscope.cn/models/<model_id>). It would let any HTTPS link through, rendered as a generic "Checkpoint".
Suggested design: declare the hub, derive the rest
The same pattern as taxonomy.frontends ({ label, env }, enum generated into the schema, env merged by synthesis):
# taxonomy.yaml
model_hubs:
huggingface:
label: "Hugging Face"
url: "https://huggingface.co/{model_id}"
cache: "~/.cache/huggingface:/root/.cache/huggingface"
modelscope:
label: "ModelScope"
url: "https://modelscope.cn/models/{model_id}"
env: { VLLM_USE_MODELSCOPE: "True" }
cache: "~/.cache/modelscope:/root/.cache/modelscope"
install: "uv pip install modelscope"# recipe
variants:
ascend_w8a8:
model_id: "Eco-Tech/Qwen3.8-27B-w8a8"
hub: modelscope # default: huggingfaceThe recipe states one fact; everything else follows from it:
| Surface | Today | With hub |
|---|---|---|
| Link + icon | hard-coded HF (CommandBuilder.jsx) |
url template |
| Download | hand-written env, missing on 2 variants | hub env merged at the base layer like the frontend env: every role, PD, Docker -e |
| Docker cache mount | hard-coded ~/.cache/huggingface (buildDockerRun / buildDockerArgv) |
hub cache, so containers don't re-download on every start |
| pip install | — | modelscope isn't in vLLM's requirements, so the pip path needs uv pip install modelscope; both the upstream image (docker/Dockerfile) and the vllm-ascend image already install it |
JSON API / .md |
— | hub + checkpoint_url on each rendering |
Two places outside vllm serve that load the tokenizer by model id, which I haven't verified. Could you check them?
- Bench tab:
vllm bench serveprobably needs the same env prefix. - PD routers: SMG's gRPC mode loads the tokenizer in the router (that path also mounts the HF cache), and so may the Dynamo frontend. If they don't honor ModelScope, either pass the env through or disable those routers when
hubisn't Hugging Face.
Known limit: VLLM_USE_MODELSCOPE is process-wide, so a --speculative-config draft model or a --tokenizer that lives only on Hugging Face would also be fetched from ModelScope. None of the eight variants do this today. A build warning for hub: modelscope plus an external model reference would catch it.
Scope
- Variants only. Reject
model.hub: site URLs,hf_id, HF dates and provider logos all assume the base model is on Hugging Face. - Not hardware-keyed. Hosting is a property of the checkpoint.
- Not inferred from
model_idor from the env. - User preference later. Downloading a both-hubs checkpoint from ModelScope (e.g. for users in China) is a separate axis. It can be added later as
hubs: [...]plus a builder pill and?hub=(inSELECTION_PARAMS) without migrating the single-value form.
Validation and migration
- Schema:
hubenum generated fromtaxonomy.model_hubs. Once migrated, the build rejects a hand-writtenVLLM_USE_MODELSCOPE(warn in the meantime). - Optional:
fetch-hf-dates.mjsalready queries the HF API per model; it could warn when a variant'smodel_idis 401/404 on Hugging Face and declares nohub. That would have caught the two broken variants. - Order: this PR (taxonomy + schema + synthesis + link), then the recipes in vllm and the vllm-ascend overlays switch the eight variants to
hub: modelscopeand drop the env, which also fixes Qwen3.8-27B and Kimi-K3.
The ModelScope logo, checkpointLink and the test layout carry over as they are; only the input changes from model_url to hub + the taxonomy entry. I'd keep the env, cache mount and pip dependency in this same PR, so a ModelScope link never sits above a command that downloads from Hugging Face.
|
Thanks for the detailed review. I changed this to variant-level For a ModelScope main model with an independent Hugging Face draft, vllm-project/vllm#64197 proposes explicit per-model |
Declare hub: modelscope for A3 W4A8 and A5 MXFP4 so the renderer derives checkpoint links, download environment, and Docker cache mounts. Remove redundant hand-written ModelScope environment settings. Requires the hub-aware renderer from vllm-project/recipes#1225. Signed-off-by: mafajun <mafajun@huawei.com>
esmeetu
left a comment
There was a problem hiding this comment.
Thanks for reworking this around model_hubs + variant hub. I re-reviewed 7561bc4 by building the JSON API for the base (cdaa860) and the PR from the same assembled content, and by rendering a test recipe with one HF variant and one hub: modelscope variant (single-node, multi-node, PD + Dynamo).
What I verified works
- Existing output is unchanged: across all renderings the only difference is the new
hubandcheckpoint_urlfields; no existing value changes, so no current command changes. - A ModelScope variant gets
VLLM_USE_MODELSCOPE=Trueon single-node, the multi-node head and every worker, and both PD roles, including Docker-e. The cache mount switches to~/.cache/modelscopein both the shell and argv Docker forms. The checkpoint link ishttps://modelscope.cn/models/<id>, and the page and.mdshow themodelscopepip step and the process-wide note. - Dynamo/SMG fall back to the vLLM router with
router_unavailable_reason. - The build warns for the 4 recipes that still set
VLLM_USE_MODELSCOPEby hand (Qwen3-30B-A3B, Qwen3.5-397B-A17B, Qwen3.6-27B, DeepSeek-V4-Flash).
Needed before merge: rebase
The PR conflicts with main. #1141, #1206 and #1226 changed the same code since cdaa860:
- The
.mdpip step for the hub checksrecipe.model?.install?.pip !== false. After #1141, install is variant-aware (resolveInstall/isPipInstallAvailable); please use those, so a variant or GPU without a pip path doesn't show the ModelScope pip step. - After #1226, PD renders per-node
node_commands[]. Please make sure the hub env and cache mount reach every node command and its Docker form, and decide whether the newascendrouter also needs the ModelScope fallback, or whether it works with ModelScope.
Suggested (non-blocking)
- JSON is missing the hub install step. The UI and
.mdshowuv pip install 'modelscope>=1.18.1', but the JSON rendering'sinstallisnull, so an agent reading JSON doesn't get it. Please carry it in the rendering (for example with the other install/dependency steps). - Avoid hard-coding
hub === "modelscope". It appears inisDynamoSupportedOnHardware,smgUnsupportedReason, the router-unavailable reason, the UI tooltip, and the Bench prefix, which hard-codes the stringVLLM_USE_MODELSCOPE=Trueinstead of readingmodel_hubs.<hub>.env. The UI also compares the label"Hugging Face"instead of the hub id. Putting this on the taxonomy entry (for example a router allowlist on the hub, and the Bench prefix built from the hub'senv) keeps the next hub data-only. It would also leaveisDynamoSupportedOnHardwareabout hardware only, rather than taking a variant. .mdshows the hub id. It printsCheckpoint: [modelscope](…)and addsCheckpoint: [huggingface](…)to every recipe's.md; it should use thelabel. You could also print the line only for a non-default hub.selection.adjusteddoesn't report the router fallback.?pd_router=dynamoon a ModelScope variant renders the vLLM router, butselection.adjustedis empty (resolveSelectiononly checks the id isdynamo/smg). The existing SMG fallback has the same gap; it would be good to report both.
Follow-up after merge
The Eco-Tech variants in the vllm recipes and vllm-ascend overlays should move to hub: modelscope (8 variants across 6 recipes). Qwen3.8-27B and Kimi-K3 don't set the env by hand, so the build won't warn about them, but their commands currently download from HF and get 401. Those two are the priority.
Signed-off-by: Jiayi Yan <1195343015@qq.com>
7561bc4 to
a60d28d
Compare
Summary
Recipe authors can set
hub: modelscopeon a variant. Hugging Face remains the default. The selected hub supplies the checkpoint link, download environment, Docker cache mount, and benchmark environment. The builder, Markdown, and JSON expose the ModelScope client install step when the selected variant and hardware support pip.ModelScope selection applies to the whole vLLM process, including external draft models and tokenizers. The builder and Markdown surface that constraint. The build flags recipes that set the environment variable manually or reference an external model with a ModelScope variant.
All prefill and decode node commands carry the hub environment and Docker cache. ModelScope variants fall back from Dynamo and SMG to vLLM Router until their tokenizer paths are verified. The JSON selection reports the requested router, the one used, and the reason. The Ascend proxy forwards HTTP requests to workers without loading a tokenizer, so supported Ascend presets retain their proxy.
Existing Eco-Tech variants can adopt the new field in their source recipes and overlays after this merges.
Testing
node scripts/build-recipes-api.mjsbuilt the JSON API for 231 models.node --test scripts/test-checkpoint-link.mjs scripts/test-model-hubs.mjspassed all 7 tests, including Docker shell and argv checks on all eight workers in a generated Ascend PD deployment. Ascend verification covered command generation only.node --test scripts/test-*.mjspassed 174 of 175 tests. The preset-render parity test reported the same seven mismatches on upstream main using the same recipe content (six MiniMax points and one Mistral preset).