Repository navigation
mpl: make MPL_gpu_attr_is_dev mean device memory; add MPL_gpu_attr_is_host - #8049
Conversation
MPL_gpu_attr_is_host returns true when a buffer can be handled as ordinary host memory. On CUDA and HIP that is unregistered host, registered host and managed memory; on ZE it is only unregistered host memory, since host-registered memory is a device object in the driver. MPIR_GPU_query_pointer_is_host wraps it for a pointer. Previously callers used the non-strict MPL_gpu_attr_is_dev, whose meaning differs on ZE, to express this. model: Opus 5.5
Callers that need to know whether a buffer can be handled as host memory now check MPL_gpu_attr_is_host or MPIR_GPU_query_pointer_is_host. This has the same result on every backend. Previously they used the non-strict MPL_gpu_attr_is_dev, which on ZE also returned true for host-registered and shared memory. model: Opus 5.5
MPL_gpu_attr_is_dev now returns true only for device memory on every backend, and MPL_gpu_attr_is_strict_dev and MPIR_GPU_query_pointer_is_strict_dev are removed. Callers that need host memory check MPL_gpu_attr_is_host; a buffer can be neither, e.g. ZE host-registered or shared memory. Previously MPL_gpu_attr_is_dev on ZE returned true for any ZE allocation, and the device-only check was a separate is_strict_dev. model: Opus 5.5
Checks for device memory in ccl, the GPU IPC local mmap and MPIR_Localcopy_gpu now call MPL_gpu_attr_is_dev. This has the same result. Previously they compared attr.type with MPL_GPU_POINTER_DEV directly. model: Opus 5.5
|
test:mpich/ch4/gpu/ofi |
|
|
||
| /* FIXME: do we allocate buffer for non-strict dev buffer? */ | ||
| if (!MPL_gpu_attr_is_strict_dev(&attr)) { | ||
| /* FIXME: do we allocate buffer for memory that is neither host nor device? */ |
There was a problem hiding this comment.
The question is whether non-strict dev buffer can operated like a host buf, e.g. memcpy, a+=b. Run a test to check
|
|
||
| MPIR_GPU_query_pointer_attr(buf, &queue->attr); | ||
| queue->need_pack = (!dt_contig || MPL_gpu_attr_is_dev(&queue->attr)); | ||
| queue->need_pack = (!dt_contig || !MPL_gpu_attr_is_host(&queue->attr)); |
There was a problem hiding this comment.
Depend on the same question as prev.
| MPL_pointer_attr_t * outattr) | ||
| { | ||
| if (!MPL_gpu_attr_is_dev(inattr) && !MPL_gpu_attr_is_dev(outattr)) { | ||
| if (MPL_gpu_attr_is_host(inattr) && MPL_gpu_attr_is_host(outattr)) { |
hzhou
left a comment
hzhou
left a comment
There was a problem hiding this comment.
This PR poses no behavior changes. We'll merge it as is. There is policy decision that whether in the collective and typerep path, should we perform staging if the buffer is non-strict buffer (registered host or managed). I am leaning toward not-to staging. Registered host should directly perform memcpy and load/store without loss of performance. Managed buffer depends on first touch but should work without error. Using managed buffer is a user choice. If they worry about performance, they should avoid managed buffer in the first place. We'll leave that to a separate PR
Commit 3dddca4 converted getfence1, putfence1 and lock_dt into the fence (-get/-put) and lock_x_dt test functions but left the old names in testlist.gpu, so the GPU tests failed to build. model: Opus 5.5
Commit 3dddca4 set flush_type instead of flush_local_type for the -flushlocal and -flushlocalall options. The enum values coincide, so those tests ran MPI_Win_flush and MPI_Win_flush_all instead of the flush_local variants. model: Opus 5.5
|
test:mpich/ch4/gpu |
|
CI failure analysis for gpu-ofi: 2 tests fail in all 3 cells (am-only, debug, default):
Both segfault inside These failures are on main. They appear in every gpu-ofi build since #455 (Oct 13), including builds of other PRs, and the last clean builds were #450 and #454. The cause is 8d2e84d ("coll/binding: collective bindings call MPIR_Xxx_impl", part of #8016). The bindings used to call the generated gpu-ucx, other failures (also not from this PR):
|
Pull Request Description
We had both
MPL_gpu_attr_is_devandMPL_gpu_attr_is_strict_dev. They are identical on CUDA and HIP, but on ZEis_devreturned true for any ZE allocation (host-registered, shared, and mmap'd device memory included), because host-registered memory is a device object in the ZE driver and cannot be used with XPMEM. That madeis_devmean "not plain host memory" on ZE, and the real device check had to beis_strict_dev. The pair is confusing.This PR makes
MPL_gpu_attr_is_devmean device memory on every backend and addsMPL_gpu_attr_is_hostfor "can be handled as ordinary host memory" (ZE: unregistered host only; CUDA/HIP: unregistered host, registered host, managed). A buffer may be neither. Every former non-strictis_devcaller becomes!is_host, and everyis_strict_devcaller becomesis_dev, so there is no behavior change on any backend.MPIR_GPU_query_pointer_is_host; removeMPL_gpu_attr_is_strict_devand the unusedMPIR_GPU_query_pointer_is_strict_dev.attr.type == MPL_GPU_POINTER_DEVcomparisons in ccl, GPU IPC local mmap andMPIR_Localcopy_gpuwithMPL_gpu_attr_is_dev.test/mpi/rma/testlist.gpu, which still listedgetfence1,putfence1andlock_dtafter they becamefence -get/-putandlock_x_dt, and fix-flushlocal/-flushlocalallinlock_x_dtsetting the wrong flush variable.[warnings:normal]
Author Checklist
🤖 Generated with Claude Code