Updates needed for PyPI wheel support (part 1 of 2) - #571
Open
xylar wants to merge 4 commits into
Open
Conversation
Prepare the ESMPy loader for self-contained PyPI wheels that bundle libesmf_fullylinked and esmf.mk inside the installed package. - Add a package-relative guess (esmpy/_esmf/lib/esmf.mk) to esmf.mk discovery, tried before the conda sys.prefix locations. - When esmf.mk is found inside the installed esmpy package (a wheel), resolve ESMF_LIBSDIR relative to esmf.mk instead of the absolute path baked in at build time, which is invalid once pip relocates the package. - Add a Windows (nt) branch that loads libesmf_fullylinked.dll and puts its bundled dependencies on the DLL search path, plus a private _ESMF_OS_WINDOWS constant. Forward-prep for the Windows wheel phase. Discovery and bundled-detection logic are factored into testable helpers (_find_esmf_mk, _esmf_mk_is_bundled) in loadESMF_helpers.py with unit tests. Behavior is unchanged for conda/source/HPC installs, where ESMFMKFILE is set or esmf.mk lives outside the package. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A real-MPI ESMPy wheel (ESMF_COMM != mpiuni) links libesmf_fullylinked against the C MPI runtime (libmpi). To keep the wheel from double-shipping an MPI, the runtime is provided by a separate wheel (e.g. `mpich`) that installs libmpi into <prefix>/lib via the wheel "data" scheme -- a directory not on the default dynamic-loader search path. Preload it with RTLD_GLOBAL before dlopening libesmf, discovering it via sysconfig's data path. Best-effort and gated on a real-MPI build, so serial (mpiuni) wheels and conda/HPC/system-MPI installs (libmpi already on the loader path) are unaffected. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
esmpy/__init__.py fetched its own version/metadata via a hard-coded
importlib.metadata.metadata("esmpy"). That breaks when ESMPy is installed under
a variant distribution name that ships the same `esmpy` import package -- e.g.
`esmpy-mpich`, the MPI-enabled wheel -- raising PackageNotFoundError at import
(and the version it sets feeds loadESMF's ESMF-version check).
Fall back to resolving whichever distribution provides the `esmpy` package
(packages_distributions on 3.10+, a distributions() top_level.txt scan on
3.8/3.9) when the direct "esmpy" lookup misses. The serial `esmpy` distribution
still resolves via the direct lookup, unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This was referenced Jul 18, 2026
Extend the loadESMF.py libmpi preload candidate list to include Open MPI's C libmpi soname (libmpi.so.40 / libmpi.40.dylib) alongside MPICH's libmpi.so.12, so the esmpy-openmpi wheel finds and preloads the libmpi shipped by the PyPI `openmpi` runtime wheel. The list stays comm-agnostic: the first present candidate is loaded; a serial or system-MPI install skips it. Serial and esmpy-mpich behavior is unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR makes the ESMPy loader "wheel-aware" so that ESMPy can be packaged as a self-contained,
pip install-able wheel that bundles the compiled ESMF library, without changing behavior for existing conda, source, and HPC installations. It is the ESMF-side prerequisite for distributing ESMPy via PyPI, as requested in #256 and conda-forge/esmpy-feedstock#72.The actual wheel-building pipeline (building ESMF + HDF5 + NetCDF from source, grafting
libesmf_fullylinkedandesmf.mkinto the pure-Python ESMPy package, and vendoring native dependencies with auditwheel/delocate) is prototyped in a separate repository: https://github.com/xylar/esmpy-wheels. This PR contains only the changes to ESMF itself that that pipeline depends on.Motivation
Today ESMPy locates its native ESMF library through the
ESMFMKFILEenvironment variable or a small set of hard-coded conda paths undersys.prefix, and reads the absoluteESMF_LIBSDIRbaked intoesmf.mkat build time. Neither assumption holds for a relocatable pip wheel: the library ships inside the installed package,pipcan place that package anywhere, and the baked-in absolute path is meaningless after installation. This PR teaches the loader to discover and load a bundled ESMF while leaving every existing installation path untouched.Changes
esmf.mkdiscovery. The lookup is factored into_find_esmf_mkand a new_esmf_mk_is_bundledhelper (src/addon/esmpy/src/esmpy/interface/loadESMF_helpers.py). The resolution order is unchanged for existing users —ESMFMKFILEfirst, then the conda layouts undersys.prefix— with a new step in between that finds anesmf.mkbundled inside the installed package (esmpy/_esmf/lib/esmf.mk). When, and only when,esmf.mklives inside the package,ESMF_LIBSDIRis resolved relative to it rather than trusting the absolute build-time path.ESMF_COMM != mpiuni),libmpiis supplied by a separate runtime wheel (e.g.mpich) that installs it under the wheel "data" scheme — a directory not on the default dynamic-loader search path. The loader now best-effort preloadslibmpiwithRTLD_GLOBALbeforelibesmfis loaded, so its MPI symbols bind correctly. This is skipped for serial (mpiuni) builds and whenever MPI is already on the loader path (conda/HPC/system MPI), so it is a no-op there.src/addon/esmpy/src/esmpy/__init__.py). An MPI build may be published under a variant distribution name (e.g.esmpy-mpich) that still ships theesmpyimport package. The metadata lookup no longer assumes the distribution is namedesmpy; it resolves whichever installed distribution provides theesmpypackage, with a fallback for Python 3.8/3.9._ESMF_OS_WINDOWSconstant and a Windows.dllload branch (usingos.add_dll_directory) are included so the same loader can drive a Windows wheel. The larger Windows ESMF/dependency build changes will follow in a separate PR; this PR only lays the Python-side groundwork.Compatibility
All changes are no-ops for existing conda, source-build, and HPC installations:
ESMFMKFILEand the condasys.prefixpaths are tried exactly as before, the baked-inESMF_LIBSDIRis used unlessesmf.mkis detected inside the package, and the MPI preload is skipped wheneverlibmpiis already reachable.Tests
Extended
src/addon/esmpy/test/test_interface/test_loadESMF_helpers.pyto cover_find_esmf_mk(environment variable, bundled, conda, and not-found cases) and_esmf_mk_is_bundled.