Skip to content

feat(examples): add remote custom encoder orchestrator - #14859

Draft
furionw wants to merge 1 commit into
qiwa/vllm-workflow-componentsfrom
qiwa/external-encoder-orchestrator-example
Draft

furionw wants to merge 1 commit into
qiwa/vllm-workflow-componentsfrom
qiwa/external-encoder-orchestrator-example

Conversation

@furionw

@furionw furionw commented Sep 15, 2026

Copy link
Copy Markdown
Contributor

Depends on #13293.

Why

Show the application-owned path enabled by #13293 without introducing a workflow product: a bespoke Dynamo worker runs a custom encoder inline, attaches its output to GenerateRequest.encoder_result, and invokes a remote stock aggregated dynamo.vllm worker.

What Change

  • Add a replaceable inline VisionEncoderBackend runner inside a bespoke orchestrator worker.
  • Package ordered CPU embeddings with encode_request_plane_tensor and the versioned ExternalEncoderResult contract.
  • Call the remote Generate endpoint through LLMUnaryClient and serve the completed response with serve_unary_endpoint.
  • Add a launch script, usage documentation, and focused orchestration tests.
flowchart LR
    Client --> Frontend
    Frontend --> Orchestrator[bespoke orchestrator worker]
    Orchestrator --> Encoder[inline custom encoder]
    Encoder -->|encoder_result| Orchestrator
    Orchestrator -->|GenerateRequest| VLLM[remote stock dynamo.vllm]
    VLLM -->|token stream| Orchestrator
    Orchestrator -->|unary completion| Frontend
Loading

Test Plan

  • Applicable pre-commit hooks, Python compilation, and shell syntax checks pass.
  • On GB200 job 2196083 with nvcr.io/nvidia/ai-dynamo/vllm-runtime-nightly:latest (ai-dynamo 1.5.0.dev20260914, vLLM 0.28.0), the three focused test files pass: 24 passed in 4.66s.
  • Direct stock-decoder baseline: HTTP 200 and ready.
  • Frontend → orchestrator/inline encoder → remote stock vLLM: HTTP 200 and The answer ... is 42.
  • Debug evidence for the same feature request records handlers._assemble_external_encoder_prompt: prepared external encoder prompt; no fatal service errors were found.

@github-actions github-actions Bot added documentation Improvements or additions to documentation backend::vllm Relates to the vllm backend feat labels Sep 15, 2026
@furionw
furionw force-pushed the qiwa/external-encoder-orchestrator-example branch from 6dae5af to c339852 Compare September 15, 2026 16:52
Signed-off-by: furionw <qiwa@nvidia.com>
@furionw
furionw force-pushed the qiwa/external-encoder-orchestrator-example branch from c339852 to b1b98a4 Compare September 15, 2026 17:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend::vllm Relates to the vllm backend documentation Improvements or additions to documentation feat size/XL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant