Repository navigation
Hopper RMSNorm fast paths: warp-per-row dynamic FP8 and register-resident few-row kernel - #2
Merged
Merged
Conversation
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
Comment |
Rows that the plain launch would give a 64-thread block (fp16/bf16 N 128-1024, fp32 N 128-512, M >= 256) now run on a single-pass, register-resident kernel: one warp per row, four rows per 128-thread block, one read of the row, one normalize-and-round, and shuffle-only reductions. Each lane reproduces threads l and l+32 of the old block (two partial-sum chains and the same butterfly order), so rstd and every output byte are identical to the plain launch. Fused-add rows use the path only above one wave of the block launch (a conservative bound). An explicit row-size guard backs the 64-thread dispatch assumption.
RMSNorm forward calls with fewer than 256 rows on sm_90 (rows of 128-16384 elements for fp16/bf16, up to 8192 for fp32 and 4096 for fp64, 16-byte-aligned tensors) now use a new kernel: one block per row with up to 512 threads, 16-byte loads with the row kept in registers, a single global read, one-barrier all-reduces for the sum of squares and amax, and one packed store per vector for FP8 outputs. Plain, fused-add, static and dynamic FP8 calls of a shape share the kernel, so FP8 bytes still equal the quantized plain output. A host-side guard keeps the row within the register budget.
JonSnow1807
force-pushed
the
h200-rmsnorm-fp8-fast-paths
branch
from
October 9, 2026 04:05
6412929 to
396af24
Compare
Owner
Author
|
A100 (sm_80) check of this branch at 396af24, built with PyTorch 2.13.0+cu129 and nvcc 12.9 on an A100-SXM4-40GB (SM clock locked at 1410 MHz):
So change 1 also runs and helps on A100; only change 2 is gated to sm_90. |
The forward launchers chose the kernel family from every pointer's alignment, while the composite plain_norm(new_residual) only sees new_residual's. A contiguous but misaligned input or residual (a base[1:] view) therefore moved the fused call to the scalar kernel while the composite took the vectorised kernel (from 256 rows) or, on sm_90, the register-resident few-row kernel, and out differed from the composite in the last bits. When the two choices differ, the RMSNorm, fused-add LayerNorm and LayerNorm fp8 launchers now write round(input + residual) into residual_out and run the plain launch on it (training forward included), so out == plain_norm(new_residual) holds for every alignment. Launches whose kernel choice already matched the composite are unchanged, and no kernel changes (identical SASS). Tests: misaligned input or residual, out-of-place and in-place, for plain, autograd-forward and fp8 fused-add at 192 and 512 rows. The contract fuzzer now cycles input/residual alignment and also checks the fp8 fused-add composite.
…cy rule precisely, bump to 0.6.0 - benchmarks/results/2026-10-29_h100-nvl_v060_fastpaths/: bench_norms.py JSON/md for the branch (5268275) and origin/main (93d5882), a derived per-row ratio table, and a README with environment, commands and caveats (H100 NVL in a Vast.ai container, torch 2.13.0+cu130 / nvcc 13.4, SM clock 1785 MHz at every run start but not lockable, git_dirty = true from in-tree output, runs not interleaved). - README: new H100 NVL section (few-row kernel per-op medians fp16 0.62-0.66, bf16 0.64-0.70, fp32 0.65-0.85 of v0.5.0 kernel time; warp-per-row fp8 path 0.67-0.95 at hidden size 1024 only; 70 control rows at 1.00 median, 0.96-1.05), ledger and limitations updated. - CHANGELOG 0.6.0: the two fast paths, the fused-add alignment fix, the H100 NVL A/B, the H200 numbers from PR #2 (not committed) and the fp32 note with the additive rule verified over 16,800 cases (new_max_abs <= old_max_abs + 2^-21, 0 failures; worse 2435 / better 2489) in place of the earlier "1 of 34" statement. - __version__ 0.5.0 -> 0.6.0.
JonSnow1807
marked this pull request as ready for review
November 2, 2026 16:45
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two forward-path changes for RMSNorm, measured on one NVIDIA H200. Change 2 is sm_90-only; change 1 is not gated by architecture and also runs on A100 (see the A100 check in the comments). Both keep the library's numerical contract (fp32 statistics,
yrounded to the input dtype before amax and quantization, per-token dynamic FP8 scale withscale_uband the 1e-12 floor, saturating e4m3), and existing kernels outside the targeted shapes are unchanged in SASS.890d73e): rows that the plain launch would give a 64-thread block (fp16/bf16 hidden size 128-1024, fp32 128-512, at least 256 rows) run on a single-pass, register-resident kernel with one warp per row and shuffle-only reductions. Each lane reproduces two threads of the old block in the same order, so outputs are bitwise identical to v0.5.0.6412929): RMSNorm forward calls with fewer than 256 rows on sm_90 (rows of 128-16384 elements for fp16/bf16, up to 8192 for fp32 and 4096 for fp64, 16-byte-aligned tensors) use one block per row with up to 512 threads, a single global read with the row in registers, one-barrier all-reduces and packed FP8 stores. Plain, fused-add, static and dynamic FP8 calls of a shape share the kernel, so FP8 bytes still equal the quantized plain output. In-target output bits can change (summation order), within the contract.Results (H200, SM clock locked at 1830 MHz, torch 2.13.0+cu129, nvcc 12.9, kernel time; each result independently re-measured)
Tests
fuzz_contractspasses in each mode.scale_ubbelow the floor, saturation, aliasing, misaligned inputs).Fused-add alignment (fixed in 5268275)
With a contiguous but not 16-byte-aligned input or residual (a
base[1:]view), the fused-add call took a different kernel than the compositeplain_norm(new_residual), which only seesnew_residual's alignment. So the README's bitwiseout == plain_norm(new_residual)could fail. This predates this PR from 256 rows (the vectorised kernel), and change 2 extended it below 256 rows on sm_90.round(input + residual)intoresidual_outand run the plain launch on it, training forward included. Launches whose kernel choice already matched are unchanged, and no kernel code changed: all 250 kernels have identical SASS.new_residualitself was always correct.fuzz_contractsnow cycles input/residual alignment and also checks the fp8 fused-add composite: it reports 83 failures at 396af24 and passes with the fix in each kernel mode.Notes
benchmarks/results/2026-10-29_h100-nvl_v060_fastpaths, branch 5268275 against v0.5.0 at 93d5882, clocks not lockable) show the few-row kernel at 0.62 to 0.70× of the v0.5.0 kernel time in fp16/bf16 and 0.65 to 0.85× in fp32, and the warp-per-row fp8 path at hidden 1024 only (0.67 to 0.95×); N ≤ 512 and fp32 for that path are still covered only by the H200 numbers.