Skip to content

Carry the AxisArray's static metadata across the shmem boundary - #8

Merged
cboulay merged 3 commits into
devfrom
feat/shmem-axis-metadata
Aug 4, 2026
Merged

Carry the AxisArray's static metadata across the shmem boundary#8
cboulay merged 3 commits into
devfrom
feat/shmem-axis-metadata

Conversation

@cboulay

@cboulay cboulay commented Aug 4, 2026

Copy link
Copy Markdown
Member

Teaches the shared-memory bridge to transport the part of an AxisArray it was dropping, so a consumer on the far side can say what the data is — not just plot it.

Why

ShMemCircBuff transports raw samples plus whatever fits in a fixed ctypes header: dtype, shape, srate, key. Everything else is lost at the boundary — the ch coordinate axis holding per-channel bank/elec/label, the axis units, the message attrs. A viewer can therefore draw the signal but can only label the traces ch0….

Consumers have been working around this by bolting a second, parallel transport onto their pipelines — an extra unit tapping the signal and republishing the ch axis on a side topic purely so the GUI can read it. That is a lot of pipeline for something the bridge should carry itself.

What

EZShmMirror gains .axes, .attrs, .dims, and register_metadata_callback():

mirror = EZShmMirror("my_stream")
mirror.auto_view()
labels = [str(v) for v in mirror.axes["ch"]["data"]["label"]]
unit = mirror.attrs.get("unit")

Plus ezmsg.tools.chmeta.channel_names(), which turns a structured ch axis into display strings. Which field to use is a per-system question — LSL and NWB populate label, Blackrock users read bank/elec off the front panel — so it is a parameter, defaulting to label as the field most sources fill in.

Design

A third segment, at <shmem_name>/meta<meta_generation>, following the same generation-named pattern already proven for the data buffer. That makes a resize free and means a reader never reads a segment being rewritten underneath it. The segment is written in full before the header names it, so a generation is either invisible or completely readable.

Plain dicts on the wire, never ezmsg classes. Axes decode to {"kind": "linear", "unit", "gain", "offset"} or {"kind": "coord", "unit", "dims", "data"}. The two halves of a link are separate processes and may be separate environments with different ezmsg versions; pinning the format to ezmsg's dataclass layout would make an upgrade on one side a silent decode failure on the other. attrs values that are not plain types are dropped with a warning rather than pickled — an arbitrary object would force the reading process to import the class that defines it.

Change detection is the whole ballgame, since the check runs on every message. Three tiers, each only reached when the one before is inconclusive:

  1. identity/value comparison of the axes held by reference — a producer passing its axes through untouched costs a handful of pointer comparisons;
  2. encode and compare bytes — a producer that rebuilds equal metadata every message costs an encode but never wakes a reader;
  3. republish under a new generation.

For a typical stream that means one publish per session.

The buffered axis is deliberately truncated to its static descriptors. Its offset advances every message and a coordinate time axis's data is wholly new each message, so including either would make the metadata change continuously and defeat the point of a low-rate side channel.

Breaking: writer and reader must now be the same version

magic and struct_version lead the header, and EZShmMirror raises ShmemVersionError on connect if either does not match. There is no compatibility shim, by choice — the layout is private between two processes we deploy together, and carrying forward every past field shape would cost more than it is worth. What a mismatch must not be is silent, since the failure being prevented is a reader interpreting a differently-shaped struct as its own and plotting the result. The magic distinguishes "written by a build too old to stamp one, or a name collision with an unrelated segment" from "our format, wrong version", and each message says which.

Also: the bridge is N-D, and now has a test saying so

ShMemCircBuff has always handled arbitrary trailing dimensions — frame_shape is every dim but the buffered one, shape is a uint32[64], and the ring write is shape-agnostic. This adds an end-to-end test pushing a 3-D (time, ch, metric) envelope through and checking shape, orientation, and that the metric axis's labels arrive over the metadata channel. Worth pinning: it is what lets a display path send a min/max envelope without flattening it into an awkward 2-D representation.

Note on _axis_equal

It compares axes field by field rather than with ==, which is not merely defensive. As of ezmsg 3.9, CoordinateAxis.__eq__ resolves through the MRO to the dataclass-generated AxisBase.__eq__ and compares unit and nothing else — two coordinate axes with different data, or even different lengths, compare equal. Building the change check on that would mean a channel relabelling silently never reaching the far side, which is the one thing this feature exists to deliver. Filed upstream as ezmsg-org/ezmsg#255.

Tests

17 new. 20 pass across test_shmem_aux_meta and test_shmem_sink; lint clean.

Not run: tests/test_shmem_mirror.py hangs when run on its own. That reproduces identically on the unmodified base commit — pre-existing and order-dependent, not from this branch.

cboulay added 3 commits August 3, 2026 20:46
The shmem link transports samples plus whatever fits in a fixed ctypes
header: dtype, shape, srate, key. Everything that says what the data *is*
-- the ch coordinate axis holding per-channel bank/elec/label, the axis
units, the message attrs -- was dropped at the boundary, so a viewer on
the far side could plot a signal it could not name. Consumers have been
working around this by bolting a second, parallel transport onto their
pipelines to ship the ch axis out of band; this puts it where it belongs.

A third segment, at "<shmem_name>/meta<meta_generation>", carries a
serialized snapshot. It follows the same generation-named pattern already
proven for the data buffer, which makes a resize free and means a reader
never reads a segment being rewritten underneath it. The segment is
written in full before the header names it, so a generation is either
invisible or completely readable.

ShmemArrMeta gains three fields, appended rather than inserted so that a
writer from an older build -- whose segment is page-rounded well past its
own shorter struct -- leaves struct_version reading 0. A reader seeing 0
knows the fields past write_index are meaningless rather than stale, and
degrades to the old behaviour instead of decoding zeros as metadata.

The wire format is a pickled dict of plain Python and numpy types, never
ezmsg classes. The two halves of a link are separate processes and may be
separate environments with different ezmsg versions; pinning the format
to ezmsg's dataclass layout would turn an upgrade on one side into a
silent decode failure on the other. attrs values that are not plain are
dropped, with a warning, rather than pickled -- an arbitrary object would
force the reading process to import the class that defines it.

Detecting change cheaply is the whole ballgame here, since the check runs
on every message. It is three tiers: identity/value comparison of the
axes held by reference, then an encode-and-compare-bytes, then finally a
republish. A producer that passes its axes through untouched -- the
normal case -- costs a handful of pointer comparisons. A producer that
rebuilds equal metadata every message costs an encode but never wakes a
reader. Only a real change reaches tier three.

The buffered axis is deliberately truncated to its static descriptors.
Its offset advances every message, and a coordinate time axis's data is
wholly new each message, so including either would make the metadata
change continuously and defeat the point of a low-rate side channel.

aux_meta._axis_equal compares axes field by field rather than with ==,
which is not merely defensive: as of ezmsg 3.6 CoordinateAxis.__eq__
resolves through the MRO to the dataclass-generated AxisBase.__eq__,
comparing unit and nothing else. ArrayWithNamedDims.__eq__, written to
compare dims and data, is shadowed and never runs, so two coordinate axes
with different data -- or different lengths -- compare equal. Building the
change check on that would mean a channel relabelling silently never
reaching the far side, which is the one thing this feature exists to do.

Also adds ezmsg.tools.chmeta, which turns a structured ch axis into
display names. Which field to use is a per-system question -- LSL and NWB
populate label, Blackrock users read bank/elec off the front panel -- so
it is a parameter, defaulting to label as the field most sources fill in.
The metadata header started out trying to let an older writer talk to a
newer reader, by appending fields and treating a zero struct_version as
"legacy, degrade quietly". That is compatibility machinery we did not ask
for and would have to keep paying for on every future field change, to
serve a case -- mismatched builds on the two ends of a link we deploy
together -- that is a deployment error rather than a scenario.

So: one version, checked strictly. What we do owe is that a mismatch is
obvious instead of silent, since the failure being prevented is a reader
interpreting a differently-shaped struct as though it were its own and
plotting the result.

magic and struct_version now lead the header, so a reader validates the
layout before trusting any field that follows, and EZShmMirror raises
ShmemVersionError on connect rather than returning a status: a caller has
nothing useful to do with it, and it will not fix itself on the next poll.
The magic separates "written by a build too old to stamp one, or a name
collision with an unrelated segment" from "our format, wrong version", and
each message says which.

Also pins that the bridge is N-D, with an end-to-end test pushing a
(time, ch, metric) envelope -- the shape BinnedAggregate produces with a
tuple operation -- through shmem and checking shape, orientation, and that
the metric axis's labels arrive over the metadata channel. ShMemCircBuff
has always handled this (frame_shape is every dim but the buffered one,
shape is a uint32[64], the ring write is shape-agnostic), so a display
envelope never needed flattening into 2-D. Worth a test so it stays true.
@cboulay
cboulay merged commit 9ba84f9 into dev Aug 4, 2026
11 checks passed
@cboulay
cboulay deleted the feat/shmem-axis-metadata branch August 4, 2026 05:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant