Skip to content

fix(sglang): inherit reasoning_parser and tool_call_parser in LoRA registration - #14863

Closed
julianStreibel wants to merge 1 commit into
ai-dynamo:release/1.4.2from
julianStreibel:fix/sglang-lora-runtime-config
Closed

julianStreibel wants to merge 1 commit into
ai-dynamo:release/1.4.2from
julianStreibel:fix/sglang-lora-runtime-config

Conversation

@julianStreibel

Copy link
Copy Markdown

Summary

Backport of the LoRA runtime_config fix (already on main via commit 8957ec3 / PR #12092) to the release/1.4.2 branch.

Problem

The SGLang backend's load_lora handler publishes adapter ModelDeploymentCards without runtime_config. The adapter card has no reasoning_parser or tool_call_parser, so the Rust frontend postprocessor does not:

  • Split <think>...</think> blocks into reasoning_content
  • Parse <tool_call> blocks into structured tool_calls

for adapter-name requests. Base model requests work correctly because the base worker's card carries the parsers.

Fix

Call _get_runtime_config() before register_llm() in the load_lora method to construct a ModelRuntimeConfig with the worker's --dyn-reasoning-parser and --dyn-tool-call-parser settings. This mirrors:

Changes

  • components/src/dynamo/sglang/request_handlers/handler_base.py:
    • Lazy-import _get_runtime_config from dynamo.sglang.register
    • Build runtime_config from the engine and dynamo args
    • Pass runtime_config=runtime_config to register_llm()

Fixes #14862

…gistration

The SGLang backend's load_lora handler publishes adapter ModelDeploymentCards
without runtime_config, so the frontend does not apply reasoning or tool-call
parsing for adapter-name requests. Apply the same fix as PR ai-dynamo#7559 (vLLM backend):
call _get_runtime_config() to build a ModelRuntimeConfig with the worker's
--dyn-reasoning-parser and --dyn-tool-call-parser and pass it to register_llm().

This is a backport of the fix already on main (commit 8957ec3, PR ai-dynamo#12092)
to the release/1.4.2 branch.

Fixes ai-dynamo#14862

Signed-off-by: Julian Streibel <julian@construct-labs.com>
@julianStreibel
julianStreibel requested a review from a team as a code owner September 15, 2026 17:11
@copy-pr-bot

copy-pr-bot Bot commented Sep 15, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@julianStreibel
julianStreibel deployed to external_collaborator September 15, 2026 17:11 — with GitHub Actions Active
@julianStreibel
julianStreibel deployed to external_collaborator September 15, 2026 17:11 — with GitHub Actions Active
@github-actions github-actions Bot added the fix label Sep 15, 2026
@github-actions

Copy link
Copy Markdown
Contributor

👋 Hi julianStreibel! Thank you for contributing to ai-dynamo/dynamo.

Just a reminder: The NVIDIA Test Github Validation CI runs an essential subset of the testing framework to quickly catch errors.Your PR reviewers may elect to test the changes comprehensively before approving your changes.

🚀

@github-actions github-actions Bot added external-contribution Pull request is from an external contributor backend::sglang Relates to the sglang backend labels Sep 15, 2026
@rmccorm4

Copy link
Copy Markdown
Contributor

Hi @julianStreibel, thanks for raising this. The 1.4.2 release is already shipped, so we won't backport new changes to it. However, the v1.5.0 release should be coming out very soon, which contains this fix - the recommended path forward is to upgrade to v1.5.0 to pickup this fix, or install a nightly wheel or build from source for custom builds as needed.

@rmccorm4 rmccorm4 closed this Sep 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend::sglang Relates to the sglang backend external-contribution Pull request is from an external contributor fix size/S

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants