Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion ci/cscs-ci-dace-determinism.yml
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,6 @@ build_cscs_amd_rocm:
USE_MPI: 0 # TODO(havogt): to workaround the libfabric hook injecting incompatible libraries
SLURM_JOB_NUM_NODES: 1
SLURM_TIMELIMIT: 60
allow_failure: true
artifacts:
when: always
paths:
Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -98,7 +98,7 @@ dependencies = [
'click>=8.0.0',
'cmake>=3.22',
'cytoolz>=1.0.1',
'dace>=2.0.0a5',
'dace>=2.0.0a6',
'deepdiff>=8.1.0',
'devtools>=0.6',
'factory-boy>=3.3.3',
Expand Down
2 changes: 0 additions & 2 deletions src/gt4py/cartesian/backend/dace_backend.py

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Root Cause Analysis
The bug is in DaCeComputationCodegen._postprocess_dace_code at src/gt4py/cartesian/backend/dace_backend.py:599-600.
What happens
The GT4Py DaCe GPU backend merges DaCe's separately-generated Frame (.cpp) and CUDA (.cu) code objects into a single computation.hpp. The keep_line filter (dace_backend.py:595-603) strips all DACE_EXPORTED … ); forward declarations, on the assumption that every declaration's definition also lives in the merged file and appears before any use:

if line.startswith("DACE_EXPORTED") and line.endswith(");"):
    return False   # strips EVERY forward declaration

That assumption is violated by __dace_gpu_drain_error. DaCe's CUDA code generator (dace/codegen/targets/cuda.py) emits, in order:

  1. Forward declaration DACE_EXPORTED void __dace_gpu_drain_error(...); (cuda.py:421)
  2. __dace_init_cuda definition, which calls __dace_gpu_drain_error(__state); before the GPU context exists (cuda.py:454)
  3. __dace_gpu_drain_error definition (cuda.py:503)

Original file line number Diff line number Diff line change
Expand Up @@ -596,8 +596,6 @@ def keep_line(line: str) -> bool:
line = line.strip()
if line == '#include "../../include/hash.h"':
return False
if line.startswith("DACE_EXPORTED") and line.endswith(");"):

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@romanc @FlorianDeconinck If you prefer, I can just keep the export line for __dace_gpu_drain_error().

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for tracking this down for us. For me, this looks good. we might get a bunch of warnings for unused things, but it's better to have warnings than to crash. We can clean this up in follow-up PRs.

return False
if line == "#include <cuda_runtime.h>":
return False
return True
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -83,14 +83,10 @@ def set_dace_config(
# creating any further sub-folder to compile the SDFG.
dace.Config.set("cache", value="single")

# Workaround to disable detection of the CUDA architecture in DaCe, and instead use the one provided by GT4Py.
# TODO(edopao): revisit this workaround once it is possible to disable GPU detection in DaCe.
# (see https://github.com/spcl/dace/pull/2424)
# Disable detection of the CUDA architecture in DaCe, and instead use the one provided by GT4Py.
if device_arch := gtx_compilation_common.get_device_arch():
dace.Config.set(
"compiler.extra_cmake_args",
value=f"-DLOCAL_CUDA_ARCHITECTURES={device_arch}",
)
dace.Config.set("compiler.cuda.cuda_arch", value=device_arch)
dace.Config.set("compiler.cuda.hip_arch", value=device_arch)

# Prevents the implicit change of Memlets to Maps. Instead they should be handled by
# `gt4py.next.program_processors.runners.dace.transfromations.gpu_utils.gt_gpu_transform_non_standard_memlet()`.
Expand Down
8 changes: 5 additions & 3 deletions uv.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.