[jswhit]

Add extra_mode_flag kwarg to Dataset for user-defined formats (closes #1490)

netcdf-c (>= 4.10.0) supports user-defined formats (UDFs), which let
external dispatch layers read non-netCDF formats through the netCDF API.
The NetCDF Expansion Pack (NEP)
uses this to add readers for GRIB2, FITS, NASA CDF, PDS4, DICOM, PDB,
mmCIF and GeoTIFF.

netcdf-c normally picks the dispatch layer from the file's magic number,
so Dataset() already works for these formats with no changes. Some files
have no magic number, though (mmCIF, for example). For those, netcdf-c
needs a UDF mode flag passed to nc_open(), such as
nc_open(path, NC_NOWRITE|NC_UDF8, &ncid). Until now there was no way to
pass one from Python.

Almost all NEP supported formats contain the "magic number" and can be read by netcdf-c 4.10.0 without the extra flag. AFAICT, only DICOM files require it.

Changes

• New Dataset kwarg extra_mode_flag (default None). An integer
  that is ORed into the mode passed to nc_open():

  ds = netCDF4.Dataset("structure.cif", extra_mode_flag=netCDF4.NEP_UDF_MMCIF)

• New module constants NEP_UDF_*: the names match the macros in NEP's
  nep.h, and each maps a format to its NC_UDFn slot:
  NEP_UDF_GEOTIFF_BIGTIFF (UDF0), NEP_UDF_GEOTIFF_STANDARD (UDF1),
  NEP_UDF_GRIB2 (UDF2), NEP_UDF_FITS (UDF3), NEP_UDF_CDF (UDF4),
  NEP_UDF_PDS4 (UDF5), NEP_UDF_DICOM (UDF6), NEP_UDF_PDB (UDF7),
  NEP_UDF_MMCIF (UDF8), NEP_UDF_NEXTCDF4 (UDF9). They are hard-coded,
  like NC_DISKLESS/NC_PERSIST, so the module still compiles against
  older netcdf-c.
• Validation: the flag is only used for a plain read-only nc_open()
  (cmode == NC_NOWRITE). These cases raise an error:
  • mode other than 'r' (including 'rs'), or diskless, memory or
    parallel ‚Üí ValueError. NEP readers are read-only, and parallel or
    in-memory opens aren't expected to work with UDFs.
  • a value that isn't one of the NEP_UDF_* constants ‚Üí ValueError.
    This stops arbitrary mode bits (such as NC_WRITE) from being passed
    through.
  • netcdf-c < 4.10.0 ‚Üí NetCDF4MissingFeatureException.
• Docstring and type stubs (__init__.pyi, _netCDF4.pyi) updated.

The UDF itself still has to be registered with netcdf-c. For NEP that's
done through its .ncrc file, for example with NCRCENV_RC pointing at it.

Tests

The NEP-dependent tests are skipped unless the NEP readers are registered
with netcdf-c, so CI without NEP is unaffected.

• test/test_extra_mode_flag.py
  • Argument checks (runs on any netcdf-c >= 4.10.0).
  • Writes a small mmCIF file with its data_ magic number removed, checks
    that it fails to open by default, and checks that it opens and reads
    correctly with extra_mode_flag=NEP_UDF_MMCIF.
• test/test_nep_grib2.py with test/gfs_sample.grb2 (27 KB, the first 3
  messages of pygrib's sampledata/gfs.grb): reads HGT, TMP and RH and
  checks them against reference values from pygrib. It runs twice, once with
  the format detected from the magic number and once with
  extra_mode_flag=NEP_UDF_GRIB2. MANIFEST.in now includes test/*grb2.

Tested on macOS arm64 with netcdf-c 4.10.1, HDF5 2.2.0 and NEP v4.1.0 built
from source. The full test suite passes with and without NEP, and so do
stubtest and mypy test.

Notes

• NEP currently declares GRIB2 dimensions in the wrong order. For this
  2.5¬∞ grid it gives (y=144, x=73), but the data are stored as
  (lat=73, lon=144). The GRIB2 test reshapes before comparing, so it will
  keep passing once that's fixed in NEP.
• nep isn't on conda-forge yet
  (Add the NetCDF Expansion Pack (NEP) - allowing netCDF to read additional data formats as if they were netCDF conda-forge/staged-recipes#34309 is still open), and that package
  (v3.0.0) doesn't include the format readers, so testing currently needs
  a source build of NEP.

🤖 Generated with Claude Code

[jswhit]

This PR currently has no test coverage under Github Actions - the tests are skipped because NEP is not installed in any of the environments the tests run in. This can be remedied once the conda-forge package exists (conda-forge/staged-recipes#34309).

EDIT: Actually, that conda-forge recipe only builds compression plugins, not the extra file formats. @captainkirk99 are you planning to create a conda-package for the file formats capabilities in NEP?

[captainkirk99]

@jswhit thanks for this effort. Unfortunately, as my AI informs me, it will not work because I am also changing how UDF mode flags are assigned and that changes the values of existing UDF flags. See Unidata/netcdf-c#3442.

Here's what my AI has to say:

PR #1518: UDF mode flags are hard-coded and break with the new netcdf-c encoding

PR: #1518 (add extra_mode_flag Dataset kwarg for user-defined formats)
Issue: #1490

1. What is wrong

1.1 The flag values are hard-coded

The PR defines the NEP mode flags as numbers in _netCDF4.pyx and in __init__.pyi:

NEP_UDF_GEOTIFF_BIGTIFF = 0x0040     # NC_UDF0
NEP_UDF_GEOTIFF_STANDARD = 0x0080    # NC_UDF1
NEP_UDF_GRIB2 = 0x10000              # NC_UDF2
NEP_UDF_FITS = 0x80000               # NC_UDF3
NEP_UDF_CDF = 0x100000               # NC_UDF4
NEP_UDF_PDS4 = 0x200000              # NC_UDF5
NEP_UDF_DICOM = 0x400000             # NC_UDF6
NEP_UDF_PDB = 0x800000               # NC_UDF7
NEP_UDF_MMCIF = 0x1000000            # NC_UDF8
NEP_UDF_NEXTCDF4 = 0x2000000         # NC_UDF9

These are the values of NC_UDF0-NC_UDF9 in netcdf-c 4.10.x, which uses one
mode bit per slot and has 10 slots.

1.2 netcdf-c is changing the encoding

netcdf-c commit dcce177bb ("increase number of UDF slots and free some mode
bits for other uses", branch ejh_udf_082126) replaces the one-bit-per-slot
scheme with a flag plus a slot-number field:

#define NC_UDF_FLAG      0x0040 /* Mode selects a user-defined format (bit 6). */
#define NC_UDF_NUM_SHIFT 19     /* Bit position of the UDF slot number field. */
#define NC_UDF_NUM_MASK  0x3F   /* 6-bit UDF slot number field: slots 0-63. */
#define NC_UDF(n)        (NC_UDF_FLAG | ((n) << NC_UDF_NUM_SHIFT))
#define NC_UDF0          NC_UDF(0)
...
#define NC_UDF9          NC_UDF(9)
#define NC_MAX_UDF_FORMATS 64

From its RELEASE_NOTES entry:

The macros NC_UDF0 through NC_UDF9 continue to work, but the numeric values of
NC_UDF1 through NC_UDF9 changed; code that uses the macros needs only
recompilation, but code that hard-codes the old numeric values, or bit-tests
modes against them, must be updated. NC_UDF0 keeps its old value (0x0040).

NEP is fine with this change: nep.h defines NEP_UDF_MMCIF as NC_UDF8 (and
so on for each format), so recompiling NEP picks up the new values. PR #1518
copies the numbers, not the macros, so it breaks.

slot	NEP name	PR value (4.10.x)	new netcdf-c value	What new netcdf-c does with the PR value
0	GEOTIFF_BIGTIFF	0x0040	0x0040	correct
1	GEOTIFF_STANDARD	0x0080	0x000C0040	no NC_UDF_FLAG, so no UDF is selected
2	GRIB2	0x10000	0x00140040	no NC_UDF_FLAG, so no UDF is selected
3	FITS	0x80000	0x001C0040	slot-field bits without NC_UDF_FLAG, so no UDF is selected
...	...	...	...	...
8	MMCIF	0x1000000	0x00440040	no UDF is selected

NC_infermodel() (libdispatch/dinfermodel.c) only treats a mode as a UDF
selection when NC_UDF_FLAG is set. With any PR constant except
NEP_UDF_GEOTIFF_BIGTIFF, extra_mode_flag therefore does nothing: netcdf-c
falls back to magic-number detection. A file with no magic number, which is the
only case the kwarg exists for, fails to open.

1.3 The PR's tests don't catch this

• test_nep_grib2.py passes either way. It opens gfs_sample.grb2 with
  extra_mode_flag=NEP_UDF_GRIB2, but GRIB2 files have a magic number (GRIB),
  so netcdf-c opens the file even when the flag is ignored.
• Only the mmCIF no-magic-number test would fail, and it only runs when NEP
  is installed and registered. No CI job does that, as jswhit notes on the PR.
• ExtraModeFlagArgsTestCase checks Python-side validation only. It never
  checks that the flag reaches nc_open() in a form netcdf-c understands.

1.4 Smaller issues

• _nep_udf_flags rejects valid modes. The allow-list contains only the
  ten NEP values, so other UDF plugins can't be used, and neither can slots
  10-63 on the new encoding.
• has_nep_mmcif / has_nep_grib2 detect NEP by trying to open a file.
  Any OSError, including one caused by the bug above, turns into a silent
  skip. netcdf-c already answers "is slot N registered?" via
  nc_inq_user_format().

2. How to fix it

Work out the mode flag for a slot at runtime, from the libnetcdf that is
actually loaded. Don't use hard-coded numbers, and don't use values taken from
headers at compile time. Wheels bundle their own libnetcdf, and conda
environments can swap the library under an existing build, so the headers used
at build time may not match the library loaded at run time.

2.1 Detecting the encoding

nc_inq_user_format() is public API in every netcdf-c with UDF support. It
rejects any mode that does not map to exactly one slot:

• 4.10.x (udf_mode_to_index() on main): counts the NC_UDF0-NC_UDF9
  bits that are set. 0x0080 is NC_UDF1, so it returns NC_NOERR.
• new encoding (udf_mode_to_index() on ejh_udf_082126): returns -1 if
  NC_UDF_FLAG is not set. For 0x0080 it returns NC_EINVAL.

So nc_inq_user_format(0x0080, NULL, NULL) == NC_NOERR means the library uses
the one-bit-per-slot encoding; otherwise it uses NC_UDF(n). Both
implementations accept NULL for both output pointers. The check does not need
any UDF to be registered.

2.2 Checking whether a slot is registered

UDF plugins listed in .ncrc are loaded in NC4_initialize(), which runs from
nc_initialize(). nc_inq_user_format() does not initialize the library
itself, so call nc_initialize() first; it returns at once if the library is
already initialized. A non-NULL dispatch table means the slot is registered.

3. Replacement code

The code below replaces the corresponding parts of PR #1518. Everything not
mentioned here stays as in the PR: the Dataset.__init__ signature, the
read-only/diskless/memory/parallel restrictions, ORing the flag into cmode
before nc_open(), MANIFEST.in, gfs_sample.grb2, test_nep_grib2.py
apart from its skip check, and the Changelog/version bump.

3.1 include/netCDF4.pxi

Add after the existing cdef extern from "netcdf.h": block:

# user-defined formats (UDFs)
cdef extern from "netcdf.h":
    ctypedef struct NC_Dispatch:
        pass
    int nc_initialize() nogil
    int nc_inq_user_format(int mode_flag, NC_Dispatch **dispatch_table, char *magic_number) nogil

3.2 src/netCDF4/_netCDF4.pyx: module-level constants

Replace the PR's NEP_UDF_* / _nep_udf_flags block (after the NC_PERSIST
definition) with:

# User-defined format (UDF) mode flags, used with the extra_mode_flag
# Dataset kwarg. netcdf-c has used two encodings for these:
#  - netcdf-c 4.10.x: one mode bit per slot, slots 0-9 ("bitmask").
#  - later netcdf-c: NC_UDF(n) = NC_UDF_FLAG | (n << NC_UDF_NUM_SHIFT),
#    slots 0-63 ("slotfield").
# Only slot 0 (0x0040) has the same value in both, so the encoding is
# detected from the loaded library instead of being hard-coded.
_UDF_FLAG = 0x0040
_UDF_NUM_SHIFT = 19
_UDF_BITMASK_FLAGS = (0x0040, 0x0080, 0x10000, 0x80000, 0x100000,
                      0x200000, 0x400000, 0x800000, 0x1000000, 0x2000000)

cdef _detect_udf_encoding():
    if _nclibversion < Version("4.10.0"):
        return None, 0
    # 0x0080 is NC_UDF1 in the bitmask encoding. In the slotfield encoding
    # it lacks NC_UDF_FLAG, so nc_inq_user_format rejects it.
    if nc_inq_user_format(0x0080, NULL, NULL) == NC_NOERR:
        return "bitmask", len(_UDF_BITMASK_FLAGS)
    return "slotfield", 64

_udf_encoding, _udf_max_slots = _detect_udf_encoding()


def udf_mode(slot):
    """
**`udf_mode(slot)`**

Return the mode flag that selects user-defined format (UDF) slot `slot`
in the netcdf-c library in use, for the `extra_mode_flag` kwarg of
`Dataset`. The numeric value depends on the netcdf-c version, so use this
function (or the `NEP_UDF_*` constants) instead of hard-coding integers.
Valid slots are 0-9 for netcdf-c 4.10.x and 0-63 for later versions.
Requires netcdf-c >= 4.10.0."""
    if _udf_encoding is None:
        raise NetCDF4MissingFeatureException("user-defined formats", "4.10.0")
    if not isinstance(slot, int) or isinstance(slot, bool):
        raise TypeError(f"UDF slot must be an int, got {slot!r}")
    if not 0 <= slot < _udf_max_slots:
        raise ValueError(f"UDF slot must be in range 0-{_udf_max_slots - 1}, got {slot}")
    if _udf_encoding == "bitmask":
        return _UDF_BITMASK_FLAGS[slot]
    return _UDF_FLAG | (slot << _UDF_NUM_SHIFT)


def _udf_registered(slot):
    """True if a dispatch layer is registered with netcdf-c for UDF slot `slot`."""
    cdef NC_Dispatch *dispatch = NULL
    cdef int mode = udf_mode(slot)
    cdef int ierr
    with nogil:
        ierr = nc_initialize()  # loads UDF plugins listed in .ncrc; no-op if already done
    _ensure_nc_success(ierr)
    with nogil:
        ierr = nc_inq_user_format(mode, &dispatch, NULL)
    _ensure_nc_success(ierr)
    return dispatch != NULL


# All valid UDF modes for the loaded library (empty if UDFs unsupported).
_udf_modes = frozenset(udf_mode(n) for n in range(_udf_max_slots))

# Mode flags for the user-defined formats provided by the NetCDF Expansion
# Pack (NEP). Names and slot numbers match those in nep.h. If the library has
# no UDF support the 4.10.x values are used, so the names always exist;
# passing them to Dataset then raises NetCDF4MissingFeatureException.
def _nep_udf(slot):
    return udf_mode(slot) if _udf_encoding is not None else _UDF_BITMASK_FLAGS[slot]

NEP_UDF_GEOTIFF_BIGTIFF = _nep_udf(0)
NEP_UDF_GEOTIFF_STANDARD = _nep_udf(1)
NEP_UDF_GRIB2 = _nep_udf(2)
NEP_UDF_FITS = _nep_udf(3)
NEP_UDF_CDF = _nep_udf(4)
NEP_UDF_PDS4 = _nep_udf(5)
NEP_UDF_DICOM = _nep_udf(6)
NEP_UDF_PDB = _nep_udf(7)
NEP_UDF_MMCIF = _nep_udf(8)
NEP_UDF_NEXTCDF4 = _nep_udf(9)
del _nep_udf

3.3 src/netCDF4/_netCDF4.pyx: Dataset.__init__ validation

Replace the PR's validation block with:

        # extra mode flag for user-defined formats, only used for plain
        # read-only nc_open (cmode == NC_NOWRITE). Need to check read_mode
        # too, since NC_CLOBBER (mode='w') is also zero.
        if extra_mode_flag is not None:
            if _udf_encoding is None:
                raise NetCDF4MissingFeatureException("extra_mode_flag", "4.10.0")
            if not read_mode or cmode != NC_NOWRITE or memory is not None or parallel:
                raise ValueError("extra_mode_flag only allowed with mode='r', "
                                 "and not with diskless, memory or parallel")
            if extra_mode_flag not in _udf_modes:
                raise ValueError("extra_mode_flag must be one of the NEP_UDF_* "
                                 "constants or a value returned by udf_mode(), "
                                 f"got {extra_mode_flag!r}")

The cmode |= extra_mode_flag before nc_open() is unchanged.

3.4 src/netCDF4/_netCDF4.pyx: Dataset.__init__ docstring

Replace the PR's extra_mode_flag paragraph with:

        **`extra_mode_flag`**: integer mode flag selecting a user-defined format
        (UDF) dispatch layer, such as those provided by the NetCDF Expansion Pack
        (NEP). Only needed for files that cannot be identified by their magic
        number. Use one of the `NEP_UDF_*` module constants (for example
        `NEP_UDF_MMCIF`) or `udf_mode(slot)`; do not hard-code integers, since the
        encoding depends on the netcdf-c version. The UDF must be registered with
        netcdf-c (for NEP, via the .ncrc file). Only allowed with `mode='r'`, and
        not with `diskless`, `memory` or `parallel`. Requires netcdf-c >= 4.10.0.
        Default `None`.

3.5 src/netCDF4/__init__.py

Add 'udf_mode' to __all__:

__all__ = [
    'Dataset', 'Variable', 'Dimension', 'Group', 'MFDataset', 'MFTime', 'CompoundType',
    'VLType', 'date2num', 'num2date', 'date2index', 'stringtochar', 'chartostring',
    'stringtoarr', 'getlibversion', 'EnumType', 'get_chunk_cache', 'set_chunk_cache',
    'set_alignment', 'get_alignment', 'rc_get', 'rc_set', 'udf_mode',
]

3.6 Stubs

src/netCDF4/__init__.pyi: the values are no longer literals.

NEP_UDF_GEOTIFF_BIGTIFF: int
NEP_UDF_GEOTIFF_STANDARD: int
NEP_UDF_GRIB2: int
NEP_UDF_FITS: int
NEP_UDF_CDF: int
NEP_UDF_PDS4: int
NEP_UDF_DICOM: int
NEP_UDF_PDB: int
NEP_UDF_MMCIF: int
NEP_UDF_NEXTCDF4: int

def udf_mode(slot: int) -> int: ...

src/netCDF4/_netCDF4.pyi: add udf_mode to the from . import (...) list next
to the NEP_UDF_* names.

3.7 Changelog

 * Allow mode flag (`extra_mode_flag` Dataset kwarg) to be passed to nc_open() to support
   additional dispatch layers available via the NetCDF Expansion Pack (NEP) (issue #1490).
   Mode flags are obtained from the `NEP_UDF_*` constants or the new `udf_mode(slot)`
   function, which work with both the netcdf-c 4.10.x and the newer 64-slot UDF encodings.

3.8 test/test_extra_mode_flag.py (replaces the PR's version)

The new UDFModeFlagReachesNcOpenTestCase needs neither NEP nor any registered
UDF, so it runs in normal CI on netcdf-c >= 4.10.0. It fails on PR #1518 as
written when built against the new encoding. If a UDF slot is selected but has
no dispatch table, netcdf-c returns NC_ENOTBUILT (libdispatch/dfile.c,
NC_open, in both encodings). If the flag is not recognised as a UDF
selection, the file opens as plain netCDF instead.

import unittest, os, tempfile
from numpy.testing import assert_array_almost_equal
import netCDF4
from netCDF4._netCDF4 import _udf_encoding, _udf_max_slots, _udf_registered

# test extra_mode_flag Dataset kwarg (issue #1490), used to select a
# user-defined format (UDF) dispatch layer for files with no magic number.
# The mmCIF read test only runs if the NetCDF Expansion Pack (NEP) mmCIF
# reader is registered with netcdf-c (e.g. via NCRCENV_RC pointing to NEP's .ncrc).

NC_ENOTBUILT = -128

MMCIF = """\
data_TINY
#
_entry.id TINY
#
loop_
_atom_site.group_PDB
_atom_site.id
_atom_site.type_symbol
_atom_site.label_atom_id
_atom_site.label_alt_id
_atom_site.label_comp_id
_atom_site.label_asym_id
_atom_site.label_entity_id
_atom_site.label_seq_id
_atom_site.pdbx_PDB_ins_code
_atom_site.Cartn_x
_atom_site.Cartn_y
_atom_site.Cartn_z
_atom_site.occupancy
_atom_site.B_iso_or_equiv
_atom_site.auth_seq_id
_atom_site.auth_asym_id
_atom_site.pdbx_PDB_model_num
ATOM 1 N N . VAL A 1 1 ? 6.204 16.869 4.854 1.00 49.05 1 A 1
ATOM 2 C CA . VAL A 1 1 ? 6.913 17.759 4.607 1.00 43.14 1 A 1
ATOM 3 C C . VAL A 1 1 ? 8.504 17.378 4.797 1.00 24.80 1 A 1
#
"""
CARTN_X = [6.204, 6.913, 8.504]

# slot numbers from NEP's nep.h
NEP_SLOTS = dict(GEOTIFF_BIGTIFF=0, GEOTIFF_STANDARD=1, GRIB2=2, FITS=3, CDF=4,
                 PDS4=5, DICOM=6, PDB=7, MMCIF=8, NEXTCDF4=9)
BITMASK = (0x0040, 0x0080, 0x10000, 0x80000, 0x100000,
           0x200000, 0x400000, 0x800000, 0x1000000, 0x2000000)

has_udf = _udf_encoding is not None
has_nep_mmcif = has_udf and _udf_registered(NEP_SLOTS['MMCIF'])


def _write(text, suffix):
    f = tempfile.NamedTemporaryFile(suffix=suffix, delete=False, mode='w')
    f.write(text)
    f.close()
    return f.name


@unittest.skipIf(has_udf, "only for netcdf-c < 4.10.0")
class NoUDFSupportTestCase(unittest.TestCase):

    def runTest(self):
        """extra_mode_flag and udf_mode need netcdf-c >= 4.10.0"""
        with self.assertRaises(netCDF4.NetCDF4MissingFeatureException):
            netCDF4.udf_mode(0)
        with self.assertRaises(netCDF4.NetCDF4MissingFeatureException):
            netCDF4.Dataset('unused.nc', extra_mode_flag=netCDF4.NEP_UDF_MMCIF)


@unittest.skipUnless(has_udf, "extra_mode_flag requires netcdf-c >= 4.10.0")
class UDFModeTestCase(unittest.TestCase):

    def runTest(self):
        """udf_mode values match the encoding used by the loaded netcdf-c"""
        self.assertIn(_udf_encoding, ("bitmask", "slotfield"))
        self.assertEqual(_udf_max_slots, 10 if _udf_encoding == "bitmask" else 64)
        for slot in range(_udf_max_slots):
            mode = netCDF4.udf_mode(slot)
            if _udf_encoding == "bitmask":
                self.assertEqual(mode, BITMASK[slot])
            else:
                self.assertEqual(mode, 0x0040 | (slot << 19))
        # NC_UDF0 is the same in both encodings
        self.assertEqual(netCDF4.udf_mode(0), 0x0040)
        modes = [netCDF4.udf_mode(s) for s in range(_udf_max_slots)]
        self.assertEqual(len(set(modes)), len(modes))
        # NEP constants map to the nep.h slot numbers
        for name, slot in NEP_SLOTS.items():
            self.assertEqual(getattr(netCDF4, 'NEP_UDF_' + name), netCDF4.udf_mode(slot))
        for bad in (-1, _udf_max_slots):
            with self.assertRaises(ValueError):
                netCDF4.udf_mode(bad)
        for bad in (1.0, '1', True):
            with self.assertRaises(TypeError):
                netCDF4.udf_mode(bad)


@unittest.skipUnless(has_udf, "extra_mode_flag requires netcdf-c >= 4.10.0")
class ExtraModeFlagArgsTestCase(unittest.TestCase):

    def setUp(self):
        self.file = tempfile.NamedTemporaryFile(suffix='.nc', delete=False).name
        netCDF4.Dataset(self.file, 'w').close()

    def tearDown(self):
        os.remove(self.file)

    def runTest(self):
        """only allowed for plain read-only opens with a valid UDF mode"""
        flag = netCDF4.NEP_UDF_MMCIF
        for kwargs in (dict(mode='w'), dict(mode='a'), dict(mode='r+'),
                       dict(mode='rs'), dict(mode='r', diskless=True)):
            with self.assertRaises(ValueError, msg=str(kwargs)):
                netCDF4.Dataset(self.file, extra_mode_flag=flag, **kwargs)
        with open(self.file, 'rb') as f:
            buf = f.read()
        with self.assertRaises(ValueError):
            netCDF4.Dataset('inmemory.nc', memory=buf, extra_mode_flag=flag)
        # not a UDF mode (NC_WRITE)
        with self.assertRaises(ValueError):
            netCDF4.Dataset(self.file, extra_mode_flag=0x0001)
        # a UDF mode from the *other* encoding is rejected
        other = (0x0040 | (1 << 19)) if _udf_encoding == "bitmask" else BITMASK[1]
        with self.assertRaises(ValueError):
            netCDF4.Dataset(self.file, extra_mode_flag=other)
        # extra_mode_flag=None is the default
        netCDF4.Dataset(self.file, extra_mode_flag=None).close()


@unittest.skipUnless(has_udf, "extra_mode_flag requires netcdf-c >= 4.10.0")
class UDFModeFlagReachesNcOpenTestCase(unittest.TestCase):

    def setUp(self):
        self.file = tempfile.NamedTemporaryFile(suffix='.nc', delete=False).name
        netCDF4.Dataset(self.file, 'w').close()
        self.unregistered = [s for s in range(_udf_max_slots) if not _udf_registered(s)]

    def tearDown(self):
        os.remove(self.file)

    def runTest(self):
        """netcdf-c sees the flag as a UDF selection (fails with NC_ENOTBUILT
        for an unregistered slot instead of opening the file as netCDF)"""
        if not self.unregistered:
            self.skipTest("all UDF slots are registered")
        for slot in self.unregistered:
            with self.assertRaises(OSError, msg=f"slot {slot}") as cm:
                netCDF4.Dataset(self.file, extra_mode_flag=netCDF4.udf_mode(slot))
            self.assertEqual(cm.exception.errno, NC_ENOTBUILT, msg=f"slot {slot}")


@unittest.skipUnless(has_nep_mmcif, "NEP mmCIF reader not registered")
class ExtraModeFlagNEPTestCase(unittest.TestCase):

    def setUp(self):
        # without the leading data_ magic number netcdf-c can't detect the format
        self.file = _write("# no magic number\n" + MMCIF, '.cif')

    def tearDown(self):
        os.remove(self.file)

    def runTest(self):
        """open mmCIF file with no magic number using NEP_UDF_MMCIF"""
        with self.assertRaises(OSError):
            netCDF4.Dataset(self.file)
        with netCDF4.Dataset(self.file, extra_mode_flag=netCDF4.NEP_UDF_MMCIF) as ds:
            assert ds.entry_id == 'TINY'
            assert len(ds.dimensions['atom']) == 3
            assert_array_almost_equal(ds['atom_site_Cartn_x'][0], CARTN_X)


if __name__ == '__main__':
    unittest.main()

3.9 test/test_nep_grib2.py

Replace the open-and-catch-OSError detection with the registration check, so
that a broken flag shows up as a failure instead of a skip:

from netCDF4._netCDF4 import _udf_encoding, _udf_registered

has_nep_grib2 = _udf_encoding is not None and _udf_registered(2)  # NEP_UDF_GRIB2 slot

The rest of the file is unchanged. GRIB2 has a magic number, so this test can't
show whether extra_mode_flag works. UDFModeFlagReachesNcOpenTestCase and
the mmCIF test cover that.

4. Verification

netcdf-c	NEP	expected
< 4.10.0	-	NoUDFSupportTestCase passes, rest skipped
4.10.x (main)	none	all non-NEP tests pass, NEP tests skipped
ejh_udf_082126	none	all non-NEP tests pass, NEP tests skipped
4.10.x (main)	registered via NCRCENV_RC	all pass
ejh_udf_082126	rebuilt against it, registered	all pass
PR #1518 unchanged + ejh_udf_082126	none	an equivalent of UDFModeFlagReachesNcOpenTestCase using the PR constants fails (the file opens as netCDF)

Also run stubtest netCDF4 and mypy test, as the PR does.

5. Notes

• The probe depends on current netcdf-c behaviour. It assumes the new
  encoding keeps rejecting 0x0080 in nc_inq_user_format(). A netcdf-c test
  asserting that would lock it in.
• A cleaner long-term option is for netcdf-c to report the encoding
  directly, for example an NC_HAS_UDF_SLOTFIELD entry in netcdf_meta.h
  plus a runtime query. The probe is still needed for 4.10.x libraries already
  in the field.
• Possible API change: a udf_slot= kwarg taking a slot number would hide
  the encoding from users entirely. extra_mode_flag plus udf_mode() keeps
  the PR's API, and the two would differ only in which value the user passes.

[captainkirk99]

In terms of testing, this can be tested if you also expose the nc_def_user_format() function. Then a test can construct it's own user format, and ensure that a file is opened with the correct user format.

WRT compression and the NEP: I have just separated out the compression filters into separate package nep-compress. nep-compress currently has pending PRs for both spack and conda.

The next release of NEP will have no compression, and the conda package will build formats. I hope that will happen soon.