conversational-voice-agent

Autonomous voice agent running on Lenovo Go hardware (NVIDIA DGX Spark). Multi-turn conversational sessions with local STT, LLM agent brain, and TTS — no cloud speech services.

Architecture

Press the Teams button to start a session. The agent greets with "Ready." Speak naturally back and forth across multiple turns. Press the button again to end the session.

• LED solid = services running
• LED blinks = session ended

_{Longer write-up: docs/architecture.md · speech path stays local; LLM may be xAI Grok or local Ollama}

Turn path: Teams button → named pipe → voice_loop → capture + energy VAD → Parakeet STT (NeMo) → spark-agent-brain (thelab-langchain + Supermemory) → sentence-chunked Piper TTS → speaker. Telemetry is local JSONL plus optional OTLP; a Docker Compose stack (Postgres · Loki · Grafana) lives in observability/.

Two repos work together:

Repo	Role
conversational-voice-agent (this one)	Voice I/O, hardware control, telemetry
spark-agent-brain	LangGraph agent brain with Supermemory (get_agent())

The brain is optional: make smoke runs LED + Piper with no brain and no API keys.

Components

STT — NVIDIA Parakeet TDT 0.6B v3 via NeMo. Direct CPU inference (no Riva server). Pre-loaded at startup. Energy-based VAD with SILENCE_SECONDS=0.5.

TTS — Piper TTS (en_US-amy-medium, 61MB ONNX). 30x real-time on CPU. Fallback chain: espeak-ng, then diagnostic tone.

Agent brain — spark-agent-brain LangGraph agent with Supermemory (installed as an editable dependency). Default LLM: Grok via xAI (grok-3-mini). Alternative: local Ollama (qwen2.5:32b via make ollama).

Button listener — Teams button toggles sessions. Volume dial adjusts ALSA PCM (amixer +/-4%) with pitch feedback ticks (400-800Hz). Half-duplex USB audio means ticks only play between sessions.

Speech cancellation — Button press during speech kills aplay immediately via stop_event.

Telemetry — Local JSONL plus optional OTLP histograms. Events: session start/end, recording, ASR, agent request/response (including tool calls), TTS, turn timing, volume changes, button presses, errors. Transcripts never leave the host on the OTEL path.

Observability stack — Postgres 16 + Loki 3.0 + Grafana 11.0 + FastAPI ingest service. Docker Compose config in observability/. Optional; runs on a separate LAN host if you want it.

Hardware

Component	Detail
Speaker	Lenovo Go Wired Speaker on plughw:1,0
Button	Teams button (BTN_0, only controllable button)
Volume	Dial controls ALSA PCM on hw:1
LED	One programmable LED (HID report 0x05)
Mute	Handled on-device
Platform	DGX Spark — AArch64, CUDA 13.0, NeMo 2.7.3, PyTorch 2.12

Note: torch.cuda.is_available() may return False even with CUDA installed.

Quick Start

Prerequisites

• Lenovo Go Wired Speaker (USB audio + HID). ALSA card is discovered by name (Lenovo / Speaker).
• udev rule for VID 17ef / PID a03f so hidraw/input are writable without daily sudo (example: /etc/udev/rules.d/99-lenovo-go.rules). LED discovery walks /dev/hidraw*.
• User in input and audio groups
• Sibling clone of spark-agent-brain at ../spark-agent-brain (or ../thelab if that is still the folder name)
• Project-local .env with XAI_API_KEY (and SUPERMEMORY_API_KEY for memory)

Plug / unplug

local-tts-buttons.service stays active if the Go is unplugged (boot before plug, yank, port move). It waits, re-binds Teams button + volume + ALSA card index, and resumes without a manual restart. Watch with: journalctl --user -u local-tts-buttons.service -f ([HOTPLUG] Waiting… / Bound…).

Smoke test (no brain, no keys)

make smoke

Run manually (foreground)

# 0. Brain package (once):
#    git clone https://github.com/derekclair/spark-agent-brain ../spark-agent-brain
#    (cd ../spark-agent-brain && make)

# 1. Keys
make keys          # copies .env.example → .env; then edit it

# Terminal 1: voice agent
make

# Terminal 2: button listener
make button

make handles the venv, installs spark-agent-brain as an editable dep, and starts the voice loop.

Install as systemd services (auto-start on boot)

make install    # installs + enables local-tts.service and local-tts-buttons.service
make uninstall  # removes them

Use local Ollama instead of Grok

make ollama

Makefile Targets

Target	Description
make / make local	Run voice agent in foreground (Grok)
make ollama	Run with local Ollama (qwen2.5:32b)
make button	Run button listener in foreground
make install	Install as systemd services (auto-start on boot)
make uninstall	Remove systemd services
make demo	Test agent + TTS without mic
make smoke	Quick LED + TTS sanity check (no brain, no keys)
make deps	Set up venv + install dependencies
make keys	Create project-local .env from .env.example
make stop	Graceful stop of listeners
make kill	Force kill rogue processes
make reset	Stop + clean up pipes
make audio-reset	Kill stale arecord/aplay processes
make pids	List running voice-related processes
make service-status	Check systemd service status
make service-logs	Tail systemd service logs

Key Files

docs/architecture.svg          System architecture diagram (README)
docs/architecture.md           Architecture write-up (embeds the SVG)
local_tts/voice_loop.py        Main voice loop with multi-turn sessions
local_tts/text_chunk.py        Sentence split for chunked TTS (spec 005)
local_tts/button_listener.py   Teams button + volume control
local_tts/telemetry.py         Structured event emitter
local_tts/otel_export.py       Content-free OTEL histograms (opt-in)
local_tts/led_control.py       HID LED control
local_tts/play_tone.py         Diagnostic tone fallback
Makefile                       All build/run/install targets
run_voice_loop.sh              Venv wrapper for voice loop
systemd/                       local-tts.service, local-tts-buttons.service
observability/                 Docker Compose + ingest API + Grafana dashboards
specs/                         Design specs (001–005)
history/                       Folded Lenovo Go SDD pack (WZY framing dropped)
tests/                         Hardware-free unit tests

Latency Tuning

Parameter	Default	Env override
SILENCE_SECONDS (EOU)	0.5	VOICE_SILENCE_SECONDS
SILENCE_THRESHOLD (RMS)	300	VOICE_SILENCE_THRESHOLD
SETTLE_TIME	0.3	(code constant)
Memory injection	Skips LLM summarization	—
Supermemory recall	limit=3	—

TTS is sentence-chunked (Piper/espeak) so first audio starts after the first sentence synthesizes. Session end abandons in-flight agent replies (best-effort; does not kill CUDA mid-forward).

Turn metrics (turn_complete + OTEL)

asr_ms, agent_ms, tts_ms, total_ms, eou_ms, tts_ttfa_ms

OTEL export is opt-in. Set VOICE_OTEL_ENDPOINT=http://<collector-host>:4318 (base URL; /v1/metrics is appended). With no endpoint, nothing is sent.

Generate a latency table from local JSONL:

python bench/latency_report.py --jsonl /tmp/local-tts-telemetry/events-*.jsonl

License

See LICENSE.