Carry the AxisArray's static metadata across the shmem boundary - #8
Merged
Conversation
The shmem link transports samples plus whatever fits in a fixed ctypes header: dtype, shape, srate, key. Everything that says what the data *is* -- the ch coordinate axis holding per-channel bank/elec/label, the axis units, the message attrs -- was dropped at the boundary, so a viewer on the far side could plot a signal it could not name. Consumers have been working around this by bolting a second, parallel transport onto their pipelines to ship the ch axis out of band; this puts it where it belongs. A third segment, at "<shmem_name>/meta<meta_generation>", carries a serialized snapshot. It follows the same generation-named pattern already proven for the data buffer, which makes a resize free and means a reader never reads a segment being rewritten underneath it. The segment is written in full before the header names it, so a generation is either invisible or completely readable. ShmemArrMeta gains three fields, appended rather than inserted so that a writer from an older build -- whose segment is page-rounded well past its own shorter struct -- leaves struct_version reading 0. A reader seeing 0 knows the fields past write_index are meaningless rather than stale, and degrades to the old behaviour instead of decoding zeros as metadata. The wire format is a pickled dict of plain Python and numpy types, never ezmsg classes. The two halves of a link are separate processes and may be separate environments with different ezmsg versions; pinning the format to ezmsg's dataclass layout would turn an upgrade on one side into a silent decode failure on the other. attrs values that are not plain are dropped, with a warning, rather than pickled -- an arbitrary object would force the reading process to import the class that defines it. Detecting change cheaply is the whole ballgame here, since the check runs on every message. It is three tiers: identity/value comparison of the axes held by reference, then an encode-and-compare-bytes, then finally a republish. A producer that passes its axes through untouched -- the normal case -- costs a handful of pointer comparisons. A producer that rebuilds equal metadata every message costs an encode but never wakes a reader. Only a real change reaches tier three. The buffered axis is deliberately truncated to its static descriptors. Its offset advances every message, and a coordinate time axis's data is wholly new each message, so including either would make the metadata change continuously and defeat the point of a low-rate side channel. aux_meta._axis_equal compares axes field by field rather than with ==, which is not merely defensive: as of ezmsg 3.6 CoordinateAxis.__eq__ resolves through the MRO to the dataclass-generated AxisBase.__eq__, comparing unit and nothing else. ArrayWithNamedDims.__eq__, written to compare dims and data, is shadowed and never runs, so two coordinate axes with different data -- or different lengths -- compare equal. Building the change check on that would mean a channel relabelling silently never reaching the far side, which is the one thing this feature exists to do. Also adds ezmsg.tools.chmeta, which turns a structured ch axis into display names. Which field to use is a per-system question -- LSL and NWB populate label, Blackrock users read bank/elec off the front panel -- so it is a parameter, defaulting to label as the field most sources fill in.
The metadata header started out trying to let an older writer talk to a newer reader, by appending fields and treating a zero struct_version as "legacy, degrade quietly". That is compatibility machinery we did not ask for and would have to keep paying for on every future field change, to serve a case -- mismatched builds on the two ends of a link we deploy together -- that is a deployment error rather than a scenario. So: one version, checked strictly. What we do owe is that a mismatch is obvious instead of silent, since the failure being prevented is a reader interpreting a differently-shaped struct as though it were its own and plotting the result. magic and struct_version now lead the header, so a reader validates the layout before trusting any field that follows, and EZShmMirror raises ShmemVersionError on connect rather than returning a status: a caller has nothing useful to do with it, and it will not fix itself on the next poll. The magic separates "written by a build too old to stamp one, or a name collision with an unrelated segment" from "our format, wrong version", and each message says which. Also pins that the bridge is N-D, with an end-to-end test pushing a (time, ch, metric) envelope -- the shape BinnedAggregate produces with a tuple operation -- through shmem and checking shape, orientation, and that the metric axis's labels arrive over the metadata channel. ShMemCircBuff has always handled this (frame_shape is every dim but the buffered one, shape is a uint32[64], the ring write is shape-agnostic), so a display envelope never needed flattening into 2-D. Worth a test so it stays true.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Teaches the shared-memory bridge to transport the part of an
AxisArrayit was dropping, so a consumer on the far side can say what the data is — not just plot it.Why
ShMemCircBufftransports raw samples plus whatever fits in a fixed ctypes header: dtype, shape, srate,key. Everything else is lost at the boundary — thechcoordinate axis holding per-channelbank/elec/label, the axis units, the messageattrs. A viewer can therefore draw the signal but can only label the tracesch0….Consumers have been working around this by bolting a second, parallel transport onto their pipelines — an extra unit tapping the signal and republishing the
chaxis on a side topic purely so the GUI can read it. That is a lot of pipeline for something the bridge should carry itself.What
EZShmMirrorgains.axes,.attrs,.dims, andregister_metadata_callback():Plus
ezmsg.tools.chmeta.channel_names(), which turns a structuredchaxis into display strings. Which field to use is a per-system question — LSL and NWB populatelabel, Blackrock users readbank/elecoff the front panel — so it is a parameter, defaulting tolabelas the field most sources fill in.Design
A third segment, at
<shmem_name>/meta<meta_generation>, following the same generation-named pattern already proven for the data buffer. That makes a resize free and means a reader never reads a segment being rewritten underneath it. The segment is written in full before the header names it, so a generation is either invisible or completely readable.Plain dicts on the wire, never ezmsg classes. Axes decode to
{"kind": "linear", "unit", "gain", "offset"}or{"kind": "coord", "unit", "dims", "data"}. The two halves of a link are separate processes and may be separate environments with different ezmsg versions; pinning the format to ezmsg's dataclass layout would make an upgrade on one side a silent decode failure on the other.attrsvalues that are not plain types are dropped with a warning rather than pickled — an arbitrary object would force the reading process to import the class that defines it.Change detection is the whole ballgame, since the check runs on every message. Three tiers, each only reached when the one before is inconclusive:
For a typical stream that means one publish per session.
The buffered axis is deliberately truncated to its static descriptors. Its
offsetadvances every message and a coordinate time axis'sdatais wholly new each message, so including either would make the metadata change continuously and defeat the point of a low-rate side channel.Breaking: writer and reader must now be the same version
magicandstruct_versionlead the header, andEZShmMirrorraisesShmemVersionErroron connect if either does not match. There is no compatibility shim, by choice — the layout is private between two processes we deploy together, and carrying forward every past field shape would cost more than it is worth. What a mismatch must not be is silent, since the failure being prevented is a reader interpreting a differently-shaped struct as its own and plotting the result. The magic distinguishes "written by a build too old to stamp one, or a name collision with an unrelated segment" from "our format, wrong version", and each message says which.Also: the bridge is N-D, and now has a test saying so
ShMemCircBuffhas always handled arbitrary trailing dimensions —frame_shapeis every dim but the buffered one,shapeis auint32[64], and the ring write is shape-agnostic. This adds an end-to-end test pushing a 3-D(time, ch, metric)envelope through and checking shape, orientation, and that the metric axis's labels arrive over the metadata channel. Worth pinning: it is what lets a display path send a min/max envelope without flattening it into an awkward 2-D representation.Note on
_axis_equalIt compares axes field by field rather than with
==, which is not merely defensive. As of ezmsg 3.9,CoordinateAxis.__eq__resolves through the MRO to the dataclass-generatedAxisBase.__eq__and comparesunitand nothing else — two coordinate axes with different data, or even different lengths, compare equal. Building the change check on that would mean a channel relabelling silently never reaching the far side, which is the one thing this feature exists to deliver. Filed upstream as ezmsg-org/ezmsg#255.Tests
17 new. 20 pass across
test_shmem_aux_metaandtest_shmem_sink; lint clean.Not run:
tests/test_shmem_mirror.pyhangs when run on its own. That reproduces identically on the unmodified base commit — pre-existing and order-dependent, not from this branch.