feat(rtsp/socket): Improve network comms and make RTSP component more stream agnostic, to support audio as well and not just MJPEG

[finger563]

Description

• feat(rtsp): overhaul the RTSP stack to support multiple tracks and additional stream types (MJPEG, H264, and audio)
• fix(socket): improve socket close / cleanup handling and rewrite the socket example as a scenario-driven teardown/self-test
• fix(python): improve the RTSP Python client/server wrappers, multitrack validation, and audio handling

This PR refactors the RTSP component so it is no longer MJPEG-only and can support codec-specific packetizers/depacketizers and multiple tracks per session.

It also includes a broader set of fixes and follow-on work that were needed to make the refactor usable in practice:

• refactor components/rtsp around multi-track RTP packetizers/depacketizers for MJPEG, H264, and generic audio payloads
• preserve backward-compatible MJPEG behavior while enabling stream-agnostic send_frame(track_id, ...)
• improve RTSP client/server/session behavior:
  • better SDP parsing and track metadata handling
  • correct handling for clients that SETUP only a subset of tracks
  • configurable RTSP task stack sizes
  • reduced per-frame heap churn in the server path
  • improved disconnect / backpressure / cooldown behavior
• improve socket close / cleanup behavior and update the socket example into a scenario-driven self-test that exercises teardown, timeout, reconnect, multicast, and failure paths
• add and improve Python host-side tooling:
  • multitrack RTSP client and server wrappers
  • audio playback / validation support
  • live microphone capture support
  • end-to-end multitrack RTSP test coverage

Motivation and Context

Closes #524

Enables multi-track client/server (which enables audio+video simultaneously) while also enabling other video formats besides MJPEG.

It also fixes some issues:

• some teardown / cleanup paths in the socket layer could hang
• the server path still had avoidable allocation churn and backpressure issues
• downstream validation and host-side tooling were not robust enough to exercise various cases

This PR completes that work so the RTSP stack can reliably support MJPEG video, H264 video, and PCM audio tracks, while keeping the older MJPEG-oriented workflows working.

How has this been tested?

Host / Python:

• rebuilt the host library and bindings with:
  • cd lib && ./build.sh
• exercised the Python RTSP flows:
  • python rtsp_server.py
  • python rtsp_server.py --use-display
  • python rtsp_client.py
  • python rtsp_server_multitrack.py
  • python rtsp_client_multitrack.py
• ran multitrack end-to-end validation for MJPEG + audio and H264 + audio
• verified the Python multitrack client correctly follows SDP metadata for audio frame sizing / sample rate

ESP-IDF example builds:

• cd components/rtsp/example && . ~/esp/esp-idf/export.sh && idf.py build
• cd components/socket/example && . ~/esp/esp-idf/export.sh && idf.py build

Hardware / manual validation:

• verified the XIAO ESP32S3 Sense example streams camera video and microphone audio successfully
• validated audio reception from the XIAO example using the Python multitrack client

Screenshots (if appropriate, e.g. schematic, board, console logs, lab pictures):

N/A

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation Update
• Hardware (schematic, board, system design) change
• Software change

Checklist:

• My change requires a change to the documentation.
• I have added / updated the documentation related to this change via either README or WIKI

Software

• I have added tests to cover my changes.
• I have updated the .github/workflows/build.yml file to add my new test to the automated cloud build github action.
• All new and existing tests passed.
• My code follows the code style of this project.

[github-actions]

✅Static analysis result - no issues found! ✅

[Copilot]

Pull request overview

This PR modernizes the ESPP RTSP stack into a multi-track, codec-agnostic RTP framework (video + audio) and improves underlying socket reliability, with accompanying host-side Python tooling/tests and documentation updates.

Changes:

• Refactors RTSP server/client to support multiple tracks using codec-specific packetizers/depacketizers (MJPEG, H264, generic audio).
• Improves socket teardown and transient-error handling (UDP send retries, safer cleanup/close semantics).
• Updates Python wrappers, adds multitrack client/server scripts plus API/E2E tests, and refreshes RTSP/socket docs and bindings.

Reviewed changes

Copilot reviewed 49 out of 49 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
python/rtsp_server.py	Wrapper to run multitrack server with MJPEG defaults
python/rtsp_server_multitrack.py	New multitrack RTSP server (video + audio)
python/rtsp_multitrack_test.py	New subprocess E2E tests for multitrack RTSP
python/rtsp_client.py	Wrapper to run multitrack client with MJPEG defaults
python/rtsp_client_multitrack.py	New multitrack RTSP client with audio validation/playback
python/rtsp_api_test.py	Offline host-side API tests for RTP/RTSP helpers
python/requirements.txt	Adds sounddevice dependency
python/README.md	Documents new RTSP multitrack scripts and audio behavior
lib/include/espp.hpp	Exposes new RTP/packetizer headers to host bindings
lib/espp.cmake	Adds new RTSP source files to host library build
lib/autogenerate_bindings.py	Exports new RTSP/RTP headers to bindings generator
doc/en/rtsp.rst	Updates RTSP docs for multicodec/multitrack support
doc/Doxyfile	Adds new RTSP/RTP headers to Doxygen inputs
components/socket/src/udp_socket.cpp	Adds send retry + stop_receiving teardown path
components/socket/src/tcp_socket.cpp	Makes close() reset connected state and cleanup
components/socket/src/socket.cpp	Adjusts select() + makes cleanup() more robust
components/socket/README.md	Updates example description (scenario-driven self-test)
components/socket/include/udp_socket.hpp	Adds stop_receiving() API
components/socket/example/README.md	Updates example README to new scenario-test behavior
components/rtsp/src/rtsp_session.cpp	Adds per-track RTP/RTCP sending + SDP generator hook
components/rtsp/src/rtsp_server.cpp	Adds multitrack sending, SDP generation, backpressure handling
components/rtsp/src/rtsp_client.cpp	Adds SDP parsing, depacketizers, monitor/disconnect handling
components/rtsp/src/mjpeg_packetizer.cpp	New MJPEG RTP payload chunker
components/rtsp/src/mjpeg_depacketizer.cpp	New MJPEG depacketizer (frame reconstruction)
components/rtsp/src/h264_packetizer.cpp	New H264 packetizer (Annex B + FU-A)
components/rtsp/src/h264_depacketizer.cpp	New H264 depacketizer (STAP-A/FU-A)
components/rtsp/src/generic_packetizer.cpp	New generic packetizer for audio/other payloads
components/rtsp/src/generic_depacketizer.cpp	New generic depacketizer (marker/timestamp framing)
components/rtsp/README.md	Updates component README for multicodec architecture
components/rtsp/include/rtsp_session.hpp	Adds Track abstraction + SDP generator config
components/rtsp/include/rtsp_server.hpp	Adds TrackConfig + multitrack send_frame APIs
components/rtsp/include/rtsp_client.hpp	Adds generic on_frame callback + tracks() + depacketizers
components/rtsp/include/rtp_types.hpp	Introduces MediaType and RtpPayloadChunk
components/rtsp/include/rtp_packetizer.hpp	Introduces packetizer base class contract
components/rtsp/include/rtp_depacketizer.hpp	Introduces depacketizer base class contract
components/rtsp/include/rtp_jpeg_packet.hpp	Header include tweaks for new usage patterns
components/rtsp/include/mjpeg_packetizer.hpp	Declares MJPEG packetizer
components/rtsp/include/mjpeg_depacketizer.hpp	Declares MJPEG depacketizer
components/rtsp/include/jpeg_header.hpp	Header include tweaks for formatting helpers
components/rtsp/idf_component.yml	Adds RTSP tags for RTP/audio/H264
components/rtsp/example/main/rtsp_example.cpp	Greatly expands example to cover new APIs/modes
components/rtsp/example/main/Kconfig.projbuild	Adds example mode selection + API-test toggle

[Copilot]

Pull request overview

Copilot reviewed 49 out of 49 changed files in this pull request and generated 4 comments.

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

[Copilot]

Pull request overview

Copilot reviewed 53 out of 53 changed files in this pull request and generated 5 comments.