Repository navigation
Fix/pin bump - #11
Merged
Merged
Fix/pin bump#11
Conversation
) physicsnemo main @ fae6bb2b contains NVIDIA/physicsnemo#2140: Strata's DepthwiseConv defaults to chunk_size="auto" instead of fixed 2-channel vmap chunks. Every config that leaves use_chunked_depthwise_conv at its default, including checkpoints that predate the field, gets the fast path with no config or checkpoint change. On H100 (t128 checkpoint, BF16 forward) the old pin is 1.07-1.72x slower than auto and launches ~8x more kernels; auto is within 1% of grouped nn.Conv2d and, unlike it, runs above 2**31 conv elements. Outputs are unchanged (FP32 rel RMS 1.7e-6). Every physicsnemo commit containing #2140 requires torch>=2.13, so: - docker: NGC PyTorch 26.01 -> 26.09 (torch 2.14). 26.09 ships an empty pip constraint file, so write one that pins the torch/CUDA/numpy stack (plus packaging<=26.2 for DALI) and keep it active for every step. - NATTEN 0.21.5 -> 0.21.7, built from the PyPI sdist with csrc patched to C++20 (torch 2.14 headers require it; NATTEN hard-codes C++17). - physicsnemo now imports treelib and termcolor; tensordict<0.16 and timm>=1.0.30 per its pyproject. earth2studio needs nest-asyncio and pandas<3 (26.09 ships pandas 3); marimo needs the pip Pygments. - earth2grid 3415382a -> v2025.11.1 (6e39b069): the old commit returns NaN for lat/lon targets west of the first source longitude, failing test_latlon_to_point_grid_regridder_keeps_seam_and_out_of_bounds_lat_targets_finite. - pyproject (torch>=2.13, natten>=0.21.7, physicsnemo>=2.3.0a0, uv source), Makefile, README base image, license inventory and the smoke test follow. Adds test_factory_default_depthwise_conv_uses_auto_chunking: fails on the old pin (chunk_size == 2), passes on fae6bb2b. Signed-off-by: Zeyuan Hu <zeyuanh@nvidia.com>
docker: default CUDA_ARCH to "8.9;9.0" (was 9.0). A 9.0-only NATTEN build aborts the whole process on an Ada GPU with "no kernel image is available for execution on the device" -- no Python exception, so pytest exits with no summary (test_dit_3d, test_dit_3d_pixel and the characterization tests on an RTX 6000 Ada). Also install GitPython and hydra-core with the other physicsnemo runtime imports, so the README no longer has to tell users to add them before launching. tests: test_characterization_inference disables TF32 for the forward and restores it afterwards. NGC images enable TF32, which moves these outputs by ~2e-4 (above ATOL), so goldens generated in one TF32 mode failed in the other. With goldens generated from main (physicsnemo 07cdcc8b), all three production configs reproduce on this branch (physicsnemo fae6bb2b). Signed-off-by: Zeyuan Hu <zeyuanh@nvidia.com>
Production checkpoints with coszr forcing compute it from time + lead_time; the test passed an empty coords["time"], so the forward raised "coords['time'] must contain at least one timestamp to compute 'coszr'" before reaching the model. Set one timestamp, as the wrapper tests do. Signed-off-by: Zeyuan Hu <zeyuanh@nvidia.com>
Contributor
|
Thanks for doing this. Merging |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix the slow pixel-stage depthwise conv (#6): pin physicsnemo fae6bb2b
Fixes #6.
What changes
physicsnemo main @
fae6bb2bcontains NVIDIA/physicsnemo#2140: Strata'sDepthwiseConvnow defaults tochunk_size="auto"instead of fixed2-channel
vmapchunks. STRATA only passesuse_chunked_depthwise_conv(never a chunk size), so every config that leaves it at the default,
including checkpoints that predate the field, gets the fast path. No
config, model code or checkpoint change is needed.
Every physicsnemo commit that contains #2140 requires torch >= 2.13, so the
environment moves too:
07cdcc8b(2.2.0a0)fae6bb2b(2.3.0a0)3415382a(2025.7.1)6e39b069(v2025.11.1)test_latlon_to_point_grid_regridder_keeps_seam_and_out_of_bounds_lat_targets_finitefails on main)Dockerfile details for NGC 26.09:
/etc/pip/constraint.txt. The Dockerfile writes/etc/pip/strata-constraint.txt(the NGC torch/CUDA/numpy stack pluspackaging<=26.2for DALI) and keeps it active for every step. 26.01'szarr<3constraint is gone, so it no longer needs clearing.treelibandtermcolor(viaphysicsnemo.distributed);tensordict>=0.15.3,<0.16andtimm>=1.0.30follow its pyproject.
nest_asyncioand requirespandas<3(26.09ships pandas 3).
chunk_size_grouped_convdefaulting to
"auto".pyproject.toml(torch, natten, physicsnemo floors and the uv source),Makefile, the README base image, the license inventory and a code commentpinned to
07cdcc8bare updated to match.Test added
tests/test_dit_3d_pixel.py::test_factory_default_depthwise_conv_uses_auto_chunkingbuilds the model from the default
PixelDiTConfig(on the meta device) andchecks that every pixel-block conv is a
DepthwiseConvwithchunk_size == "auto". It fails on main's pin (chunk_size == 2) andpasses on this branch.
Performance and numerics (H100 80GB HBM3)
Full network forward, t128 production checkpoint, BF16 autocast. Time is
the median ms per forward; kernels is the number of GPU kernel launches in
one forward:
nn.Conv2dThe kernel ratio (8-15x) is much larger than the time ratio because the
extra kernels are small: each adds about 4-14 us, while attention and MLPs,
which are the same in all three columns, dominate the forward. At 128 x 1
the ~12,300 extra kernels add 170 ms to a 235 ms forward (1.72x); at larger
tiles the rest of the network grows faster than the overhead (1.07-1.09x).
On faster GPUs the same overhead is a larger share (GB300: up to 4.5x).
(auto) vs 45 ms (grouped).
every size; auto and chunk 2 give identical outputs.
1.6-4.5x slower.
Keeping
use_chunked_depthwise_conv=Trueas the default is deliberate:grouped
nn.Conv2dfails at or above 2**31 conv elements (tile 160 xbatch 4, tile 320 x batch 1).
Also in this PR
CUDA_ARCHis now"8.9;9.0"(was9.0). With a 9.0-onlyNATTEN build, the first NATTEN call on an Ada GPU aborts the whole
process with "no kernel image is available for execution on the device".
There is no Python exception, so pytest exits with no summary. On an RTX
6000 Ada this kills
test_dit_3d,test_dit_3d_pixeland thecharacterization tests in main's image.
runtime imports. The README no longer tells users to add them.
test_characterization_inferencedisables TF32 for its forward pass. NGCimages enable TF32, which moves these outputs by about 2e-4, above the
1e-4 tolerance, so goldens generated in one TF32 mode failed in the other.
Verification
make lintpasses.docker build -f docker/Dockerfile .builds with the new default; NATTENhas
sm_89andsm_90akernels, and the smoke test passes.(S3), the aux data and
SCREAMCAST_CHECKPOINTset to the t128 productioncheckpoint,
--ignore=tests/distributed: 229 passed, 2 skipped. The2 skips are the multi-GPU tests in
test_distributed.py.tests/distributed(8-GPUtorchrun) was not run.test_from_checkpointneeded a fix to run at all: it passed an emptycoords["time"], which coszr checkpoints reject. It now sets onetimestamp, as the wrapper tests do.
physicsnemo
07cdcc8b, in the same image with TF32 off. All threeproduction configs (sweep1 DiT3D and both pixeldit configs) reproduce
within the test's 1e-4 tolerance with physicsnemo
fae6bb2b. So the pinbump does not change production inference.
scripts/debug/bench_depthwise_conv.pyandfrom_checkpointtests on theequivalent screamcast image, not from this repo's test suite.