flowcat-core: opt-in full-duplex (barge-in) support for the cascaded pipeline

[rolandknight]

Closes #60.

Adds an opt-in full-duplex path for the cascaded pipeline with working barge-in, validated live end-to-end (str0m WebRTC + aiortc client, whisper.cpp STT, OpenRouter LLM with tool calls, Kokoro TTS). The stock half-duplex builder and its TurnMute behavior are untouched; nothing changes unless you call the new builder.

The problem (details in #60)

1. The runtime intercepts Frame::Interruption (drain + forward) and never delivers it to process_frame — the existing Interruption arms in the transport sinks are unreachable, so frame-level barge-in cannot work on the cascaded path.
2. Nothing on the cascaded path emits BotStartedSpeaking/BotStoppedSpeaking, so VadProcessor's barge-in gate never arms.
3. Even delivered, the frame path stalls behind any mid-await hop — we measured detection→sink delivery of 14 ms to 2.1 s depending on TTS/LLM activity — and an in-flight LLM stream cannot be cancelled, so the interrupted reply is spoken afterwards anyway.

The changes

Piece	What it does
FrameProcessor::on_interruption() (new, default no-op)	Runtime calls it in the Interruption arm after draining — the real delivery path for barge-in reactions
VadProcessor::{with_interrupt_flag, with_interrupt_notify}	Generation counter + Notify bumped synchronously at detection (before the broadcast)
LlmProcessor::with_interrupt_flag	Cooperative cancel between streamed chunks; closes response framing on cancel
AssistantContextAggregator::on_interruption	Keeps the partial reply in context, drops the open span (late LlmResponseEnd can no longer speak the reply)
SpeechGate (new)	VAD-edged segmentation: 300 ms pre-roll at the rising edge, all-zero flush marker (SPEECH_GATE_FLUSH_SAMPLES) at the falling edge — without the turn lock, fixed-window batch STT hallucinates turns on silence and splits utterances
BotSpeakingNotifier (new) + sink wiring	Emits bot-speaking edges into the pipeline head from playout tracking; arms the VAD gate
Out-of-band interrupt reactor + stale-audio latch	Notify-woken task flushes the carrier immediately (~110 µs from detection in our runs); the latch drops TTS audio that outran its interruption
build_cascaded_call_duplex (new, exported)	Assembles all of the above; generic over VadAnalyzer so it stays feature-agnostic (caller passes e.g. SileroVad)

Validation

• Live barge-in test (long spoken reply interrupted mid-playback over WebRTC): bot audio stops within the harness' 1 s budget consistently; interrupted reply does not resume; the follow-up turn (a tool call) executes normally. Repeated 3× + a 7-test regression suite, all green.
• cargo test -p flowcat-core --lib: 302 passed. cargo clippy -p flowcat-core --lib -- -D warnings: clean. cargo fmt applied.

Notes for review

• The reactor + latch exist because of measured need (the 2.1 s frame-path stall); if you'd rather solve preemption differently (e.g. cancellable process_frame), the hook + flag still stand alone.
• One default worth a look while you're here: VAD_MIN_VOLUME = 0.6 gated out moderate-volume speech entirely in our runs (only the loudest tail of utterances passed); we had to run with 0.2. Left untouched here since it's pipecat parity.
• Happy to split this into smaller PRs (hook/runtime fix first) if preferred.

🤖 Generated with Claude Code

[sathish-mg]

Reviewed this against the runtime. Both of the first two findings reproduce exactly as described, and the third is the right diagnosis of why the frame path alone can't carry barge-in:

1. run_processor's Frame::Interruption arm drains the interruptible backlog and forwards, but never calls process_frame. So every Frame::Interruption arm anyone has written is dead code. Confirmed.
2. Nothing on the cascaded chain emits the bot-speaking edges — s2s.rs is the only emitter and it only emits BotStoppedSpeaking. VadProcessor::bot_speaking can therefore never become true and the broadcast never fires. Confirmed.
3. An Interruption can't preempt a process_frame already inside an .await. That is inherent to the design, not a bug in it, and the out-of-band flag/notify is the right escape hatch as long as the frame path stays the correctness path — which it does here.

The shape of the fix is right. Interruption is a lifecycle frame, so a hook alongside start/stop is the correct delivery path rather than trying to route it into process_frame. Keeping the duplex builder opt-in with the stock half-duplex path and its TurnMute untouched is also the right call — no need to split the PR.

Merged with a follow-up commit on top (a0f4139). What changed and why:

The hook was only half-applied. Three dead Frame::Interruption arms were left behind, so the codebase now had two mechanisms with inconsistent use. Migrated all of them:

• TransportOutput in s2s.rs — worth calling out, because this is a real bug on the realtime path too, not only the cascaded one. The model stops generating service-side on barge-in, but send_clear() was just as unreachable there, so whatever was already handed to the carrier kept playing over the interrupting caller. Now covered by barge_in_flushes_the_carrier_playback_at_the_sink.
• Both text_filter.rs processors (TextAggregatorProcessor, TextFilterProcessor) — their resets never ran, so a half-built sentence leaked into the next turn.
• Dropped the arm you left in the cascaded sink "for clarity". Dead code carrying a comment that says it's dead is worse than no code; the hook is directly above it.

Replaced SPEECH_GATE_FLUSH_SAMPLES with a real seam. The all-zero 333-sample marker doesn't do anything for the in-tree provider — whisper_local buffers it like any other audio and 333 samples never crosses the segment threshold, so it only works against an STT that's in on the convention. Since the diagnosis is correct (batch STT has no endpointing), the fix belongs at the trait boundary: SttService::flush(), defaulted to a no-op, called by SttProcessor on UserStoppedSpeaking, implemented for whisper_local to drain its partial buffer. Additive — every streaming provider keeps the default and is unaffected — and it takes a magic constant back out of the public pipeline API. SpeechGate now just gates and forwards the edge.

Fixed a latch race that could mute the bot for the rest of the call. The reactor and the sink's hook are woken independently and either can be scheduled first. In the common ordering the reactor wins, but if the sink's on_interruption cleared the bool before the reactor set it, nothing ever cleared it again and every subsequent OutputAudio was silently dropped. StaleAudioLatch now stamps the barge-in generation at both ends, so it's correct in either order. Test: stale_audio_latch_is_order_independent.

notify_one instead of notify_waiters for the reactor wakeup. notify_waiters only wakes tasks currently parked, so a barge-in raised while the reactor is inside send_clear() was dropped and that one fell back to the slow frame path. notify_one stores the permit.

Tests. This was the main gap — 513 lines with none. Added 16 covering hook delivery and error handling in the runtime, the realtime sink flush, the STT flush seam, the cooperative LLM cancel both with and without the flag wired, the speech gate (pre-roll replay, pre-roll bound, no synthetic flush audio), the bot-speaking edges including the superseded-watchdog case, the latch ordering, and the aggregator/filter resets. cargo test 318 passing in flowcat-core, cargo clippy --workspace --all-targets -D warnings clean, cargo fmt --all --check clean.

Also tightened two comments that overstated their mechanism: AssistantContextAggregator::on_interruption doesn't drop the span via in_response (which is never read) — it's taking the buffer that makes a late LlmResponseEnd inert. And SpeechGate's pre-roll clear is a no-op on the VAD's own barge-in, since UserStartedSpeaking leads the broadcast and has already drained the ring.

Docs updated in PROCESSOR-DESIGN §2.1/§2.2/§2.5 and CONTRIBUTING. Worth noting CONTRIBUTING claimed the task loop cancels "an in-flight interruptible process_frame" — it doesn't, and that sentence is plausibly what made the barge-in arms look wired to begin with. §2.5 now states plainly what the frame path can and can't do.

Two things left open, both yours to pick up if you want:

• Recording on the duplex path. SpeechGate sits upstream of the RecorderProcessor inbound tap, so the caller leg records gated speech only and the replayed pre-roll lands ~300 ms late within its utterance. Wall-clock stamping keeps the legs from drifting apart, so it's cosmetic, and I've documented it as the trade rather than restructuring the chain. Fixing it properly means splitting the recorder's inbound tap from its metrics fold.
• VAD_MIN_VOLUME = 0.6. Your 0.2 finding matches what I'd expect; pipecat parity is a weak reason to keep a default that gates out normal speech. Please open a separate issue with what you measured so it can be changed on its own merits rather than inside a barge-in PR.

On the two extras in #60 — factory::tts dropping options.base_url for kokoro, require_key rejecting keyless local providers, and the FLOWCAT_WEBRTC_BIND_IP loopback default breaking same-host ICE. All three sound real and none are related to this change. Please file them separately; the bind-IP one in particular is worth its own issue since it makes the local WebRTC demo unusable out of the box.

Thanks — good bug report, and the live measurements made the case far easier to evaluate than a description would have.