Add realtime speech transcription transport

[sajjad-mazaheri]

This PR adds the realtime speech transport needed for automatic spoken responses in RSVP reading.

This is the foundation phase only.

What changed

Microphone audio capture

• Extends the existing microphone session with continuous audio-frame subscriptions.
• Uses AudioWorklet when available, with a ScriptProcessorNode fallback.
• Adds an explicit start/stop gate so the task controller can later begin capture at the precise stimulus onset without reopening the microphone.
• Converts browser audio to mono 16 kHz PCM16 chunks for realtime STT.
• Preserves resampling continuity across audio frames.

Provider-independent transcription

• Adds a common StreamingTranscriber interface with typed states, events, and errors.
• Adds realtime adapters for:
  • ElevenLabs Scribe v2 Realtime
  • Deepgram Nova-3
• Supports caller-supplied target keyterms, which can later be populated from the targets selected for each RSVP trial.
• Supports interim transcripts, provider commits, explicit finalization, cancellation, and connection cleanup.

Preflight robustness

• Allows a temporarily muted microphone track to recover during preflight.
• Pauses the preflight measurement deadline while the track is temporarily muted.
• Still rejects a track that remains muted beyond the recovery grace period.

Tests

• npm run check:ts
• Focused speech test suite:
  • 9 test suites passed
  • 86 tests passed

The tests cover:

• microphone lifecycle and cleanup
• audio gating and PCM conversion
• streaming resampling
• preflight and transient mute recovery
• ElevenLabs and Deepgram URL/configuration generation
• keyterm behavior
• privacy-related provider flags
• WebSocket lifecycle and provider error mapping
• stale utterance isolation
• transcript accumulation and finalization
• credential endpoint validation and trusted-origin checks

[Copilot]

Pull request overview

Adds the realtime speech transcription “transport” layer for RSVP spoken responses: microphone capture → PCM16@16kHz chunking → provider-independent streaming transcription session → provider adapters (ElevenLabs Scribe v2 Realtime, Deepgram Nova-3), plus preflight robustness and a focused test suite.

Changes:

• Introduces a provider-neutral StreamingTranscriber contract and a SpeechSession that gates audio and manages one-at-a-time utterances.
• Adds microphone continuous audio-frame subscription (AudioWorklet w/ ScriptProcessor fallback) and a PCM16 16kHz capture/resampling pipeline.
• Adds token/credential endpoint handling and expands preflight logic to tolerate transient muted tracks (with timeouts paused during mute).

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tests/speechToken.test.ts	Tests trusted-origin validation and prevents leaking session tokens to untrusted endpoints.
tests/speechSession.test.ts	Tests utterance gating, commit/finalization flows, error mapping, cancellation, and idempotent close.
tests/speechPreflight.test.ts	Adds coverage for transient mute recovery and rejection beyond the grace period.
tests/elevenLabsRealtimeTranscriber.test.ts	Validates URL/config generation, keyterms policy, token provider behavior, and WS lifecycle/error mapping for ElevenLabs.
tests/deepgramRealtimeTranscriber.test.ts	Validates Deepgram URL/config, auth protocols, audio send/finalize behavior, and close-during-credential retrieval.
tests/audioCapture.test.ts	Tests PCM conversion, streaming resampler continuity, and gating behavior in microphone capture.
components/speech/transcriber.ts	Defines shared transcriber types, events, and error model for realtime streaming.
components/speech/speechToken.ts	Implements credential/token retrieval with timeout, request body, and trusted-origin checks for default endpoint selection.
components/speech/speechSession.ts	Implements session state machine, utterance lifecycle, audio forwarding gate, and finalization/cancellation logic.
components/speech/speechPreflight.ts	Enhances preflight to tolerate temporary microphone mute and pauses deadlines during mute windows.
components/speech/microphoneCaptureWorkletSource.ts	Provides an AudioWorklet processor source string for continuous frame capture.
components/speech/microphone.ts	Extends MicrophoneSession with audio-frame subscriptions and implements worklet/script-processor capture sources.
components/speech/elevenLabsRealtimeTranscriber.ts	Implements ElevenLabs realtime WS adapter, URL construction, keyterm normalization, and event/error mapping.
components/speech/deepgramRealtimeTranscriber.ts	Implements Deepgram realtime WS adapter, URL construction, auth protocol usage, and event/error mapping.
components/speech/audioCapture.ts	Converts streamed microphone frames into gated PCM16@16kHz chunks with streaming resampling.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.