Describe the Bug
ThunderAgent registers its proxy endpoint as an aggregated model. This registration creates a second ModelDeploymentCard for the same backend model.
The proxy card contains model-file metadata and fields that ThunderAgent copies manually. It does not contain the backend's complete runtime contract.
For example, SGLang publishes runtime_config.token_budget from its live engine configuration. The value includes these settings:
- The effective context limit.
- Reserved speculative tokens.
- Prompt-overflow rejection.
- Combined input-and-output rejection.
- Automatic truncation behavior.
ThunderAgent creates a new ModelRuntimeConfig. It adds parser settings and optional capability flags, but it does not inherit the backend token budget.
The frontend uses the ThunderAgent card because ThunderAgent advertises the public model endpoint. As a result, the frontend cannot apply the backend's admission policy.
Current Topology
Client -> frontend -> ThunderAgent model endpoint -> KV router -> backend
ThunderAgent acts as a proxy engine in this topology. It does not own the model runtime, but it publishes the frontend-facing model card.
Steps to Reproduce
- Start an SGLang worker with a 131,072-token context limit.
- Hide the worker's OpenAI surface behind ThunderAgent.
- Register ThunderAgent with the same model path and a public model name.
- Send a request with a 135,010-token prompt.
The model files advertise an architectural limit of 262,144 tokens. SGLang publishes its lower runtime limit in runtime_config.token_budget.
The ThunderAgent card omits that token budget. The frontend admits the request, and SGLang rejects it after routing.
Expected Behavior
The backend model card must remain authoritative for preprocessing and admission. A routing policy must not replace or reconstruct the backend runtime contract.
The request path must have this order:
Client -> frontend preprocessing and admission -> pluggable routing or flow control -> backend
ThunderAgent must operate as a routing or flow-control policy. It must not register a second model or publish a synthetic model card.
Actual Behavior
ThunderAgent calls register_model() for thunderagent_router.generate. The generic registration path loads static model metadata from model_path.
Only the backend knows the effective runtime policy. ThunderAgent cannot derive that policy from the model files.
This design can lose or stale these model-card fields:
- Token-budget and truncation policy.
- Effective context length.
- Backend capabilities.
- Parser configuration.
- Future engine-specific runtime metadata.
Copying selected fields into the proxy card is not a complete fix. Each new backend field creates another synchronization requirement.
Proposed Direction
Remove public model registration from ThunderAgent. Keep model discovery, model-card normalization, preprocessing, and admission in the standard frontend path.
Invoke ThunderAgent after preprocessing through the existing pluggable router or flow-control contract. Pass the admitted PreprocessedRequest to that policy.
The policy can apply stricter admission rules. It must inherit the frontend and backend rules first.
Environment
- Dynamo:
0027a8ef0bc108152b1382bfe9584ffa90ec4c7c (origin/main during reproduction).
- Backend: SGLang.
- Request-payload tracing: Enabled.
Related Work
Describe the Bug
ThunderAgent registers its proxy endpoint as an aggregated model. This registration creates a second
ModelDeploymentCardfor the same backend model.The proxy card contains model-file metadata and fields that ThunderAgent copies manually. It does not contain the backend's complete runtime contract.
For example, SGLang publishes
runtime_config.token_budgetfrom its live engine configuration. The value includes these settings:ThunderAgent creates a new
ModelRuntimeConfig. It adds parser settings and optional capability flags, but it does not inherit the backend token budget.The frontend uses the ThunderAgent card because ThunderAgent advertises the public model endpoint. As a result, the frontend cannot apply the backend's admission policy.
Current Topology
ThunderAgent acts as a proxy engine in this topology. It does not own the model runtime, but it publishes the frontend-facing model card.
Steps to Reproduce
The model files advertise an architectural limit of 262,144 tokens. SGLang publishes its lower runtime limit in
runtime_config.token_budget.The ThunderAgent card omits that token budget. The frontend admits the request, and SGLang rejects it after routing.
Expected Behavior
The backend model card must remain authoritative for preprocessing and admission. A routing policy must not replace or reconstruct the backend runtime contract.
The request path must have this order:
ThunderAgent must operate as a routing or flow-control policy. It must not register a second model or publish a synthetic model card.
Actual Behavior
ThunderAgent calls
register_model()forthunderagent_router.generate. The generic registration path loads static model metadata frommodel_path.Only the backend knows the effective runtime policy. ThunderAgent cannot derive that policy from the model files.
This design can lose or stale these model-card fields:
Copying selected fields into the proxy card is not a complete fix. Each new backend field creates another synchronization requirement.
Proposed Direction
Remove public model registration from ThunderAgent. Keep model discovery, model-card normalization, preprocessing, and admission in the standard frontend path.
Invoke ThunderAgent after preprocessing through the existing pluggable router or flow-control contract. Pass the admitted
PreprocessedRequestto that policy.The policy can apply stricter admission rules. It must inherit the frontend and backend rules first.
Environment
0027a8ef0bc108152b1382bfe9584ffa90ec4c7c(origin/mainduring reproduction).Related Work
choices, but not this model-card ownership defect.