perf(nvidia): reduce refactored inference overhead

[voltjia]

Summary

• Remove repeated host-side work from runtime stream lookup, InfiniOps default implementation selection, paged decode metadata construction, paged-cache setup, and greedy sampling.
• Extend graph-backed execution for reviewed NVIDIA decode/prefill shapes, static attention metadata, and single-token speculative verification while retaining eager fallbacks for unsupported inputs.
• Add focused runtime, Python, graph-safety, and speculative-runner contract coverage.

Motivation

The modern Infini stack refactor introduced material inference latency regressions in several NVIDIA workloads. This PR collects the focused optimizations developed from per-case profiling, while keeping the fast paths narrowly gated by device, model configuration, tensor shape, layout, and dtype.

This PR targets refactor/adopt-modern-infini-stack; no issue is linked.

Type of Change

• feat - new feature / new model
• fix - bug fix
• perf - performance improvement (no behavioral change)
• refactor - code restructuring without behavior change
• test - adding or fixing tests only
• docs - documentation only
• build / ci - build system or CI configuration
• chore - tooling, formatting, or other non-code changes
• Breaking change

Test Results of Involved Models on Supported Platforms (Please attach screenshots)

Platform	Scope	Result
NVIDIA A100-SXM4-80GB	Focused P03, P09, P11, P13, P14, and N02 stage comparisons	Completed; output identity and per-stage validation are summarized below
Local static contracts	python -m unittest discover -s test/static -p "test_*.py" -v	103 passed
Local stack-builder tests	python -m unittest test/scripts/test_build_infini_stack.py -v	23 passed
NVIDIA A100-SXM4-80GB	Fresh native Release build of aggregate head 326cd456	Passed; _infinilm built in 47.857 s using the recorded modern-stack prefix
NVIDIA A100-SXM4-80GB	P12 ChatGLM short-decode candidate	Static contracts passed; native graph stress and performance comparison not run yet
Other supported platforms	Cross-platform regression testing	Not run because this iteration is scoped to the NVIDIA regression

Screenshots are not attached yet. The focused runs retained structured logs, output hashes, telemetry, and result archives outside this source branch.

Benchmark / Performance Impact

All measurements used NVIDIA A100-SXM4-80GB GPUs. Negative deltas mean the later candidate was faster. These are separate causal stage comparisons and must not be added together. Except for the P03 cache result, they are two-pair order-balanced screening estimates rather than formal confidence intervals.

Workload / change	Baseline median	Candidate median	Paired effect	Remaining relation to old main
Default InfiniOps implementation lookup microbenchmark	28.043 ns/call	2.108 ns/call	-92.471%, 95% CI [-92.953%, -91.225%]	N/A
P03 total time for the same cache change	25,312.140 ms	25,169.840 ms	-0.587%, 95% CI [-1.577%, +0.644%]	Not used as a cumulative estimate
P09 paged-decode metadata reuse	10,122.50 ms	9,478.22 ms	-6.36%	+9.23% direct total-time regression remains
P11 safe equal-head decode graph	173.605 ms	157.095 ms	-9.441%	-4.142% composed total-time estimate
P13 short decode graph	2,210.62 ms	2,094.01 ms	-5.275%	+9.360% normalized composed estimate
N02 Eagle draft graph reuse	6,366.230 ms	6,111.015 ms	-4.0024%	+9.4672% chained estimate
P14 safe decode graph	571.130 ms	550.065 ms	-3.6886%	+8.7428% composed estimate
P12 current refactor baseline	993.260 ms old main	1,167.575 ms current	+17.5492% regression	Candidate in this PR is not benchmarked yet

The focused measurements were made while developing individual stages. The branch is rebased onto 7c393761 and ends at 326cd456; a fresh A100 Release build passes. Focused inference confirmation remains pending.

Notes for Reviewers

• 574ee4ed adds the P12 ChatGLM graph eligibility as an intentionally isolated candidate commit. Its predicates are exact, but it is not yet claimed as a measured improvement.
• P11/P14 device-graph safety depends on the matching persistent InfiniOps implementation-17 path from the current modern stack.
• Unsupported shapes and layouts continue through the existing eager or host-segment fallback paths.
• Raw benchmark directories and transfer archives were deliberately excluded from the branch.
• Review the graph input lifetime, snapshot policy, persistent provider ownership, and exact workload predicates before broadening any eligibility checks.

CI / ChatOps

PR-triggered Check Format and ruff checks pass at 326cd456. The full ci job is intentionally limited to workflow_dispatch; focused P12 inference validation and an explicitly requested full CI run remain pending.

---

Checklist

Title, Branch, and Commits

• PR title follows Conventional Commits.
• Branch name follows <type>/xxx-yyyy-zzzz and matches the perf PR type.
• Each commit message follows Conventional Commits.
• This is a large PR whose twelve commits are meaningful and well-formed.
• N/A: The requested base is refactor/adopt-modern-infini-stack, not main; the branch is rebased on its current 7c393761 tip with no merge commits.
• No fixup! / squash! / wip commits remain.
• N/A: This does not use the legacy issue branch/commit format.

Scope and Design

• Changes are limited to the measured inference-performance work and its tests.
• No temporary benchmark artifacts or transfer archives are included.
• No unrelated formatting churn obscures the diff.
• N/A: No user-facing public API change is intended.

General Code Hygiene (applies to all languages)

• Comments are limited to non-obvious invariants and safety boundaries.
• Exact scripts/format.py validation passes in a clean LF checkout with the CI formatter versions.
• git diff --check origin/refactor/adopt-modern-infini-stack..HEAD passes.
• Comment and error-message style needs confirmation with the repository formatting/style toolchain.

C++ Specific (if C++ files changed)

• clang-format 21.1.8 validation passes for the changed C++ files.
• LLVM error/warning wording has not been independently audited.
• Constructor initializer order has not been independently audited.
• Raw allocation usage has not been independently audited.
• scripts/format.py and the PR Check Format job pass with the CI tool versions.
• No changes reference csrc/models/llama_legacy/.

Python Specific (if Python files changed)

• ruff 0.15.20 check and ruff format --check pass for the changed Python files.
• Python comment style validation is pending.
• N/A: No public docstrings were added or changed.
• scripts/format.py and the PR ruff job pass with the CI tool versions.
• No changes reference python/infinilm/auto_config.py.

Testing

• Untested platform and aggregate-head gaps are explicitly listed above.
• Focused single-request inference runs passed for the measured model cases.
• Focused offline performance runs passed for the measured benchmark cases.
• Full test/bench/test_benchmark.py sanity coverage was not rerun in this focused iteration.
• Service performance testing was not rerun in this focused iteration.

Build, CI, and Tooling

• Aggregate head 326cd456 passed a fresh-directory NVIDIA A100 Release _infinilm build.
• PR formatting and Ruff CI pass; the full ci job requires an explicit workflow_dispatch run.

Documentation

• N/A: No user-visible behavior, build flag, or developer workflow is changed.
• N/A: No breaking change is introduced.

Security and Safety

• No secrets, tokens, internal URLs, customer data, or hardware identifiers are committed.
• N/A: No third-party code is added.
• Native graph lifetime and bounds safety still require final aggregate-head NVIDIA validation.