DRAFT: transport: rpmsg (OpenAMP) transport for MPU+MCU systems

[beriberikix]

Warning

Draft / do not merge. Stacked on #232 (serial transport core) — base is transport/serial, not main. Rebase onto main once #232 lands. Tracks the RFC in #274.

Also depends on #317 (two serial-core fixes against #232's branch). Without those the device crashes on the first ACK and the uplink stalls every session, so this branch does not run on hardware until they land.

What this is

An rpmsg (OpenAMP) transport for Pouch on heterogeneous MPU+MCU SoCs: the MCU runs Zephyr + the Pouch device stack and syncs to a Linux gateway over on-die OpenAMP/rpmsg, keeping its own cloud identity and end-to-end encryption while the Linux side forwards opaque pouches.

Per #274, this is an adapter on the serial transport core (#232), not a new transport — each rpmsg message carries one Pouch Serial frame, reusing the serial channel model, no SAR.

Status

• rpmsg device adapter, ipc_service (port/zephyr/transport/serial/rpmsg_device.c): ipc_service endpoint ↔ serial device core + POUCH_RPMSG_DEVICE Kconfig.
• rpmsg device adapter, resource table (port/zephyr/transport/serial/rpmsg_rsc_device.c): the raw-OpenAMP twin, required to talk to Linux. See "Why two adapters" below.
• UART device adapter (port/zephyr/transport/serial/uart_device.c): interrupt-driven native-tty adapter for native_sim, sharing the framing module.
• Framing module (uart_framing.[ch]): pure, testable length-delimited framing (encoder + streaming decoder).
• SAEAD example (examples/zephyr/rpmsg_device): builds in nsim CI; compiles the UART adapter + serial device path with SAEAD. Now also builds for imx95_evk/mimx9596/m7.
• MCU-mediated OTA (src/transport/endpoints/device/fw.[ch], two new serial channels): a core with no flash of its own downloads its own image through Pouch OTA and relays the plaintext to the host, which verifies the SHA-256 and applies it via remoteproc.
• Tests (tests/pouch/rpmsg/exchange, native_sim, passing in twister CI): framing round-trip / resync / rejection, plus an end-to-end broker↔device serial exchange routed through the framing.
• Renode Tier-2 (mechanism validated) — see doc/renode-tier2/. Real Linux remoteproc + virtio-rpmsg in Renode brings up the RFC's exact interface (/dev/rpmsg_ctrl0), loads/starts the Zephyr R5 firmware, and the full R5 firmware bring-up is worked out and documented. The final channel announcement is blocked by an emulator-model IPI stall after remoteproc start.
• Hardware validated — the round-trip Renode deferred is done. FRDM-IMX95 (Cortex-M7 under Linux remoteproc) against production Golioth, with connect-agent as the broker: device online, telemetry flowing, and the gateway seeing only ciphertext. Three complete OTA runs (install, self-hosted upgrade, downgrade), each installed image byte-identical to the cloud artifact by SHA-256, plus a negative test in which a corrupted image was rejected on hash with nothing installed and the running firmware untouched.

Why two adapters

Zephyr's DT-instantiable ipc_service backends use a static vring layout that the Linux kernel's virtio_rpmsg_bus does not understand. Against Linux the vdev has to come from the resource table, exactly as in the openamp_rsc_table sample — which is what rpmsg_rsc_device.c does. The ipc_service adapter remains the right choice for Zephyr-to-Zephyr and for RTOS peers that speak the same static layout.

Naming the endpoint rpmsg-raw also gets a /dev/rpmsgN for free on the Linux side, with no RPMSG_CREATE_EPT_IOCTL dance.

MCU-mediated OTA

On these parts the MCU is loaded from the host filesystem at every boot, so it has no flash to write an update into — but it is still the authenticated Pouch endpoint. It downloads its own image through Pouch OTA and streams the plaintext out over a firmware channel; the host verifies the SHA-256 from the manifest, installs alongside the previous image (<name>.prev for rollback) and restarts the core.

The image is never resident on the MCU. Golioth OTA reports only a state upstream, never a received offset, so the cloud streams a whole component within one session and expects the device to consume it. A flashless MCU with a small relay buffer cannot do that if the firmware channel is only collected after the downlink verb finishes — so the broker collects the firmware channel concurrently with the downlink that feeds it, and the channel holds its transfer open when the relay is momentarily empty rather than reporting "nothing to collect".

Notes for reviewers

• This branch now does touch the serial core: it adds two channels to include/pouch/transport/serial/common.h and registers them in src/transport/serial/device.c. That is the OTA feature, not the transport. The two actual serial-core fixes are split out into fix(serial): guard optional endpoint->end and re-arm a stalled channel #317 against Add serial transport #232's branch.
• The west-ncs twister job fails because this branch's base is behind main; resolves on rebase after Add serial transport #232. The nsim example job is red only from pre-existing backend-dependent coap/http pytests; sample.pouch.rpmsg_device builds clean.
• Renode Tier-2 is local-only (no OpenAMP/ZynqMP in pouch CI).
• The example's device credentials are placeholders; hardware validation used real ones kept out of the tree.
• Known gap: a rejected image leaves the OTA component marked "updating", so the device needs a restart before it will retry — the manifest is delivered once per connection and never comes round again. A fix is drafted but is the one piece not yet hardware-verified, so it is deliberately not in this branch.

🤖 Generated with Claude Code

[beriberikix]

Hardware validation: findings

Pushed 4a14a0e, which closes out the item this PR deferred — the end-to-end round-trip now runs on real MPU+MCU hardware. Setup: FRDM-IMX95, Cortex-M7 under Linux remoteproc, connect-agent as the broker, production Golioth. Three complete OTA runs plus a negative test.

Most of the work turned out to be in things the diff does not make obvious, so they are worth writing down.

1. ipc_service cannot talk to Linux

This is the biggest one, and it is why there are now two adapters. Zephyr's DT-instantiable ipc_service backends implement their own static-vring layout. The Linux kernel's virtio_rpmsg_bus does not speak it. Against Linux the vdev must come from the resource table.

The adapter this PR originally shipped could never have worked against a Linux host as written. rpmsg_rsc_device.c is the resource-table twin; the ipc_service one is still correct for Zephyr-to-Zephyr.

2. A host/device deadlock that is entirely on the Linux side

The link would freeze mid-transfer with the gateway blocked in write() on /dev/rpmsg0 — for minutes. Goroutine stacks showed it parked in Go's netpoller waiting for EPOLLOUT.

virtio_rpmsg_poll() reports the endpoint writable only when a TX buffer is already free. It never enables the tx-complete callback. Only the blocking write(2) path calls rpmsg_upref_sleepers(), which is what enables it. So a non-blocking writer that drains the ring — routine when streaming a firmware image — waits for a wakeup that structurally cannot arrive.

Worth flagging for anyone writing a gateway in a runtime that makes fds non-blocking by default (Go, Node, async Rust). The fix was to keep the fd out of the poller and use blocking writes.

3. Thread priority is load-bearing, not a tuning knob

The rpmsg management thread ran at the same priority as the Pouch work queue. Same priority + preemptive means the management thread drains the entire RX vring without ever yielding — we measured 137 KB arriving in 6 ms — while the decrypt queue never runs. pouch_downlink_push() heap-allocates one buffer per encrypted block, so it allocated until malloc failed.

The failure mode is nasty: the allocation failure puts the downlink channel into a permanent error state, and every subsequent session fails with "peer rejected transfer". The device never recovers. A transient allocation failure arguably should not be terminal for the channel.

The management thread now runs below the work queue, and the RX path applies real backpressure by stalling while the relay is backed up, so the pressure reaches the broker instead of pooling on the device.

4. OTA has no resume, which constrains the design

golioth_ota_receive_component() reports only a state upstream (idle / downloading / updating), never a received byte offset. The cloud therefore streams an entire component within one session and expects the device to consume all of it.

A flash-backed device writes as it goes. A flashless MCU cannot: there is no way to buffer a 253 KB image in 256 KB of RAM. And it cannot drain to the host either, because the broker only collected the firmware channel after the downlink verb completed — the buffer fills partway through the very verb that has to finish before it can empty.

So the broker has to service the firmware channel concurrently with the downlink. The Pouch Serial frame format already supports this (every frame carries its channel); it was only the broker's sequential verb loop that did not. If OTA ever grows an offset in its upstream status, this gets much simpler.

5. Known gap: a rejected image needs a reboot to retry

Deliberately not in this branch, because it is the one piece I could not verify on hardware before losing the board.

When the host rejects an image, the device has already called golioth_ota_mark_updating(), which stops the cloud offering the component again. It never retries. The obvious place to undo that — the OTA manifest handler — does not work: the manifest is delivered once per connection, so a device that has already marked itself updating never sees another one. I confirmed this: 18 sessions after a rejection, zero retries.

The fix is to hang it off the apply verdict itself rather than the manifest. That is drafted, but the M7 took a bus fault on boot with it in and I could not isolate whether the cause was the change or this board's known load-order-dependent TCM ECC behaviour. Left out rather than shipped unverified.

Two serial-core bugs → #317

Both block any device talking to a real broker, and both are unreachable from the current tests: every mock in stub_endpoints.c defines .end, and they are all synchronous so none ever returns MORE_DATA with zero bytes. Split out into #317 against #232's branch.

Platform notes for anyone reproducing this

• The i.MX95 System Manager config name must be mx95evkrpmsg — U-Boot's power_on_m7() compares it and silently skips otherwise, and that function is what zero-initialises both TCMs. The M7's ITCM is ECC-protected with a 64-bit granule, and arm64's memcpy_toio() writes a segment's unaligned tail as single-byte stores, so a partial write into a never-written granule leaves invalid ECC and the first read of it is a precise data bus fault. Skipping the zeroing makes booting depend on what ran before.
• Relatedly, soc/nxp/imx/imx9/imx95/m7/linker.ld gives .resource_table only 4-byte alignment with no explicit address, so the tail alignment of the last PT_LOAD depends on the parity of every preceding ROMABLE section. That is a latent trap for any OpenAMP app on this SoC.
• Zephyr's drivers/ipm/ipm_mbox.c dispatches its callback with no NULL check. The host can kick the MU before the application registers one — the MU's interrupt-enable bits survive a remoteproc restart — and the core faults during early boot. Both Zephyr fixes to be sent upstream separately.

🤖 Generated with Claude Code