Lucasmind · frostybittn · May 22, 2026 · May 22, 2026 · May 22, 2026 · May 22, 2026
diff --git a/.env.example b/.env.example
@@ -7,7 +7,19 @@ STT_PORT=8080
 TTS_PORT=8880
 LLM_PORT=8081
 
-# LLM settings
+# LLM backend (Plan #11): "ollama" (default — the upstream workshop path) or "lmstudio".
+#   ollama   → host-installed Ollama; setup scripts pull qwen3:4b-instruct + smollm2.
+#   lmstudio → orchestrator talks LM Studio's native /api/v0 API + auto-discovers
+#              your whole LM Studio model library for the dashboard picker.
+LLM_BACKEND=ollama
+# (Used only when LLM_BACKEND=lmstudio.) LM Studio server URL as seen from inside the
+# orchestrator container. LM Studio must be "Served on Local Network" (binds 0.0.0.0)
+# so host.docker.internal resolves.
+LMSTUDIO_URL=http://host.docker.internal:1234
+# Model the LM Studio "auto" brain falls back to when nothing is loaded (must exist in LM Studio).
+LMSTUDIO_DEFAULT_MODEL=qwen/qwen3-4b-2507
+
+# LLM settings (Ollama path)
 LLM_MODEL_FILE=Qwen_Qwen3-4B-Instruct-2507-Q4_K_M.gguf
 LLM_CTX_SIZE=4096
 

diff --git a/CLAUDE.md b/CLAUDE.md
@@ -11,11 +11,13 @@ Browser (localhost:3000)
   └── nginx ──┬── /api/stt/ ──► whisper.cpp (port 8080)        [Vulkan]
               ├── /api/tts/ ──► Kokoro-FastAPI (port 8880)      [CUDA/ROCm]
               └── /api/llm/ ──► Orchestrator (port 8082)
-                                 └─► Ollama on host (port 11434) [GPU]
+                                 ├─► Ollama on host (:11434)                    [default]
+                                 └─► LM Studio on host (:1234, native /api/v0)  [LLM_BACKEND=lmstudio, Plan #11]
 ```
 
 - **Frontend**: Vite + Three.js + TalkingHead + VAD-web (browser-based orchestrator)
-- **LLM**: Qwen3-4B via Ollama on host (thinking model with `<think>` tags)
+- **LLM**: Qwen3-4B via Ollama on host (default), OR LM Studio (native `/api/v0` API; whole
+  library auto-discovered) — selected by `LLM_BACKEND` (see `docs/lmstudio-runbook.md`)
 - **TTS**: Kokoro-82M via Kokoro-FastAPI (returns PCM + word timestamps)
 - **STT**: Whisper base.en via whisper.cpp (Vulkan)
 - **Orchestrator**: OpenAI-compatible proxy + agentic tool loop + command center backend
@@ -182,3 +184,25 @@ Docker Desktop on macOS runs a Linux VM — no GPU passthrough to Metal/MPS. Doc
 - Tool toggles (web_search / wiki) now live in `state.tools` rather than browser localStorage; frontend's ControlPanel POSTs to /v1/swap on change
 - Interactive teaching scripts in `scripts/demos/` (test-llm, test-tts, test-stt, test-pipeline, test-orchestrator, list-models) back slides 13-24 of the workshop deck
 - Setup: `bash scripts/setup-linux.sh` (Linux/WSL2) or `bash scripts/setup-mac.sh` (macOS) installs Ollama and pulls default models
+
+## Plan #11 — LM Studio backend (native API + dynamic discovery)
+
+Full guide: **`docs/lmstudio-runbook.md`**. Selected by `LLM_BACKEND` env (`ollama` default | `lmstudio` opt-in).
+
+- **New brain kind `lmstudio`** → `LMStudioProvider` (`providers/lmstudio.py`), a 3-line subclass of
+  `OllamaProvider` that targets LM Studio's NATIVE endpoint `POST /api/v0/chat/completions`. The wire
+  format is identical OpenAI-shaped SSE — streaming + `tool_calls` both verified — so the parser is reused.
+  The native response also carries `stats`/`model_info`/`runtime` (TTFT, tok/s).
+- **Discovery + residency** in `system/lmstudio.py` (`LMStudioBackend`): `GET /api/v0/models` →
+  `list_models()` (filters to `llm`/`vlm`, flags `loaded` + `thinks`) and `query()` (residency snapshot in
+  the same shape as `OllamaResidency`). Never raises.
+- **Dynamic catalog**: `/v1/catalog` calls `Catalog.sync_dynamic_brains(...)` to merge every LM Studio model
+  as an `lmstudio:<id>` brain (loaded-first). The dashboard Brain dropdown auto-lists the whole library.
+- **`lmstudio-auto` brain** (`model: auto`) resolves at request time to the loaded model, else
+  `LMSTUDIO_DEFAULT_MODEL`. `main.py` promotes it to the default brain when `LLM_BACKEND=lmstudio` and
+  resets a stale non-LM-Studio active brain on boot.
+- **Backend-aware routes**: `/v1/models`, `/v1/state`, `/v1/swap`, `/v1/catalog`, `/v1/chat/completions`
+  all follow `llm_backend`. `nginx.conf` + `vite.config.js` add `/api/lmstudio/` (raw passthrough, Lab 1).
+- **TTS unchanged**: LM Studio has no audio API (tested — see runbook §5). Kokoro remains the TTS engine.
+- Tests: `tests/test_lmstudio.py` (config, provider, discovery/residency, dispatcher, dynamic merge, and
+  route-level catalog + chat-auto). Whole suite: 171 passing.
diff --git a/README.md b/README.md
@@ -38,9 +38,9 @@ NodeAva ships with a default avatar (sourced from the [TalkingHead](https://gith
 
 ## Prerequisites
 
-**Ollama (required on all platforms):**
-- Install on the host (not in Docker): `curl -fsSL https://ollama.com/install.sh | sh` (Linux/WSL2) or `brew install ollama` (macOS)
-- The workshop setup scripts download default models: `bash scripts/setup-linux.sh` (Linux/WSL2) or `bash scripts/setup-mac.sh` (macOS)
+**LLM backend — Ollama (default) _or_ LM Studio (opt-in):**
+- **Ollama** (default, host install): `curl -fsSL https://ollama.com/install.sh | sh` (Linux/WSL2) or `brew install ollama` (macOS). Setup scripts pull defaults: `bash scripts/setup-linux.sh` / `bash scripts/setup-mac.sh`.
+- **LM Studio** (opt-in, set `LLM_BACKEND=lmstudio` in `.env`): run LM Studio with "Serve on Local Network" enabled. The orchestrator talks its native `/api/v0` API and **auto-discovers your whole model library** into the dashboard's Brain picker. See **`docs/lmstudio-runbook.md`**. _(LM Studio serves the LLM only; TTS stays Kokoro.)_
 
 **Docker + Compose (Linux/Windows only):**
 - Docker Engine with Compose V2

diff --git a/configs/catalog.yml b/configs/catalog.yml
@@ -2,6 +2,8 @@ brains:
   # `thinks: true` flags models that emit <think>...</think> reasoning blocks.
   # The frontend gates token-streaming detection on this flag — thinking models
   # buffer until </think> appears, non-thinking models stream immediately.
+  #
+  # ── Ollama brains (default — used when LLM_BACKEND=ollama) ───────────────
   - id: qwen3-4b-instruct
     label: "Qwen3 4B Instruct (default — fast, no reasoning)"
     kind: ollama
@@ -23,6 +25,25 @@ brains:
     kind: ollama
     model: deepseek-r1:8b
     thinks: true
+  # ── LM Studio brains (Plan #11, opt-in via LLM_BACKEND=lmstudio) ─────────
+  # kind: lmstudio talks LM Studio's NATIVE REST API at <LMSTUDIO_URL>/api/v0.
+  # When LLM_BACKEND=lmstudio, the orchestrator ALSO auto-discovers every model
+  # in your LM Studio library at runtime and adds each as a selectable
+  # `lmstudio:<model-id>` brain — so this static list is just the stable
+  # defaults. `lmstudio-auto` follows whatever model you have loaded in LM
+  # Studio; it is promoted to the default brain at startup only when LM Studio
+  # is the active backend.
+  - id: lmstudio-auto
+    label: "LM Studio (auto — uses your loaded model)"
+    kind: lmstudio
+    model: auto
+    thinks: false
+  - id: lmstudio-qwen3-4b
+    label: "LM Studio · Qwen3 4B 2507 (small + fast)"
+    kind: lmstudio
+    model: qwen/qwen3-4b-2507
+    thinks: false
+  # ── Cloud brains (require API keys) ──────────────────────────────────────
   - id: claude-sonnet
     label: "Claude Sonnet 4.6 (cloud)"
     kind: cloud-litellm

diff --git a/docker-compose.yml b/docker-compose.yml
@@ -35,6 +35,15 @@ services:
       # Plan #7: catalog.yml must be visible inside the container.
       - ./configs:/app/configs:ro
     environment:
+      # Plan #11: local LLM backend. Defaults to "ollama" (the upstream
+      # workshop path). Set LLM_BACKEND=lmstudio in .env to opt in to LM Studio
+      # — the orchestrator then talks LM Studio's native /api/v0 API and the
+      # dashboard auto-discovers your whole LM Studio model library.
+      # LM Studio must be served on the local network (Developer ▸ Settings ▸
+      # "Serve on Local Network") so the container reaches host.docker.internal:1234.
+      - LLM_BACKEND=${LLM_BACKEND:-ollama}
+      - LMSTUDIO_URL=${LMSTUDIO_URL:-http://host.docker.internal:1234}
+      - LMSTUDIO_DEFAULT_MODEL=${LMSTUDIO_DEFAULT_MODEL:-qwen/qwen3-4b-2507}
       - OLLAMA_URL=http://host.docker.internal:11434
       - REQUEST_TIMEOUT=300
       - BIND_HOST=0.0.0.0

diff --git a/docs/lmstudio-runbook.md b/docs/lmstudio-runbook.md
@@ -0,0 +1,141 @@
+# LM Studio Backend — Runbook (Plan #11)
+
+**Opt-in alternative** to the default Ollama backend. Switch NodeAva's brain from Ollama to
+**LM Studio** by setting `LLM_BACKEND=lmstudio`, with a live, auto-discovered picker for
+*every* model in your LM Studio library. The avatar's LLM then runs through LM Studio's
+**native** REST API (`/api/v0`); TTS (Kokoro) and STT (Whisper) are unchanged.
+
+> **Verified end-to-end** against a live LM Studio (115-model library): backend switch,
+> 112-model auto-discovery, residency, per-model swap, and streaming + non-streaming chat
+> all working (`[happy] Welcome to NodeAva!` in ~155 ms on `qwen/qwen3-4b-2507`).
+
+---
+
+## 0. Prerequisites (one-time)
+
+1. **LM Studio is running** with the local server started (Developer tab → **Start Server**,
+   or `lms server start`). Default port **1234**.
+2. **"Serve on Local Network" is ON** (Developer ▸ Server Settings). This binds the server to
+   `0.0.0.0` so the orchestrator *container* can reach it via `host.docker.internal:1234`.
+   Without it, LM Studio only listens on `127.0.0.1` and Docker can't see it.
+3. At least one chat model downloaded (e.g. `qwen/qwen3-4b-2507` — small & fast, good default).
+   Models JIT-load on first use; you don't need to pre-load.
+
+Quick reachability check from the host:
+
+```bash
+curl http://localhost:1234/api/v0/models | head      # should list your models
+```
+
+---
+
+## 1. Activate
+
+Set `LLM_BACKEND=lmstudio` in your `.env` (the docker-compose default is `ollama`).
+Then rebuild + restart so the new orchestrator code, env, and dashboard take effect:
+
+```bash
+docker compose up -d --build orchestrator frontend
+# or rebuild the whole stack:  docker compose up -d --build
+```
+
+- **orchestrator** rebuild → new provider/discovery code + `LLM_BACKEND=lmstudio` env.
+- **frontend** rebuild → dashboard brain-picker chips + the `/api/lmstudio/` nginx route.
+- `configs/catalog.yml` is bind-mounted, so brain edits there only need a restart, not a rebuild.
+
+---
+
+## 2. Verify
+
+```bash
+# Active brain + LM Studio residency (which model is loaded right now)
+curl -s http://localhost:8082/v1/state | python3 -m json.tool
+
+# The full picker: your LM Studio library merged as lmstudio:<model> brains
+curl -s http://localhost:8082/v1/catalog | python3 -c \
+ "import sys,json;b=json.load(sys.stdin)['brains'];d=[x for x in b if x.get('dynamic')];\
+print('dynamic LM Studio brains:',len(d));print('loaded first:',[x['id'] for x in d[:3]])"
+
+# A real chat through LM Studio's native API
+curl -s -X POST http://localhost:8082/v1/chat/completions -H 'Content-Type: application/json' \
+ -d '{"messages":[{"role":"user","content":"Hi in five words"}],"stream":false}'
+```
+
+Then open the dashboard (**http://localhost:3000**, drawer ▸ Controls). The **Brain**
+dropdown now lists every LM Studio model — loaded ones first, with a green **loaded** chip.
+
+---
+
+## 3. Using it
+
+- **Pick any model live** from the Brain dropdown. Selecting one and chatting JIT-loads it in
+  LM Studio. (Big 27B–35B models take a few seconds + VRAM to load the first time.)
+- **`LM Studio (auto)`** brain = "use whatever I have loaded in LM Studio right now". It resolves
+  at request time to your loaded model, else falls back to `LMSTUDIO_DEFAULT_MODEL`.
+- **Thinking models** (names containing `thinking`/`reasoning`/`r1`/`magistral`/`qwq`) are auto-flagged
+  `thinks: true` so the frontend buffers the `<think>` block instead of speaking it.
+- **Native API bonus**: chat runs on `/api/v0/chat/completions`, which also returns
+  `stats` (tokens/sec, TTFT), `model_info`, and `runtime` — available for a future telemetry hook.
+
+### Raw model playground (Lab 1)
+**Lab 1 ("The Brain")** now talks directly to LM Studio's native API: pick any model from
+your library (loaded ones first) and measure raw TTFT / tokens-per-sec — it even logs LM
+Studio's own *server-measured* stats. Powered by the `/api/lmstudio/` passthrough, available
+in both Docker (nginx) and `npm run dev` (Vite proxy). Labs 4 & 6 already exercise LM Studio
+via the orchestrator.
+
+---
+
+## 4. Switch back to Ollama
+
+Ollama is the docker-compose default, so just remove the override (or set it explicitly):
+
+```bash
+sed -i '/^LLM_BACKEND=/d' .env          # or: echo "LLM_BACKEND=ollama" >> .env
+docker compose up -d --build orchestrator frontend
+```
+
+The host-Ollama path takes over. Both backends coexist in the catalog; only the *active*
+backend and the discovery source change.
+
+Per-deploy overrides (in `.env`):
+
+| Var | Default | Meaning |
+|---|---|---|
+| `LLM_BACKEND` | `ollama` | `ollama` (default) or `lmstudio` (opt-in) |
+| `LMSTUDIO_URL` | `http://host.docker.internal:1234` | LM Studio server as seen from the container |
+| `LMSTUDIO_DEFAULT_MODEL` | `qwen/qwen3-4b-2507` | fallback for the `auto` brain |
+
+---
+
+## 5. TTS reality check ⚠️
+
+**LM Studio cannot do text-to-speech — through any API.** This was tested, not assumed:
+
+| Probe | Result |
+|---|---|
+| `POST /v1/audio/speech` (OpenAI-compat) | `{"error":"Unexpected endpoint or method"}` |
+| `POST /v1/audio/transcriptions`, `/v1/audio/generations`, `/api/v0/audio/speech` | none exist |
+| Official docs (`lmstudio.ai/docs/developer/rest`) | "no audio endpoints" |
+| `voxtral-4b-tts-2603` prompted directly | `arch: llama`, `type: llm`; returned **empty** — it's a text model, not a vocoder |
+
+The same is true of **Ollama** — both are *LLM inference engines* (text / embeddings / vision-input).
+Text→speech is a different model class needing a dedicated server, which is why this kit uses
+**Kokoro-FastAPI**. So TTS stays Kokoro:
+
+- **Change voice** in the dashboard (Voice dropdown: Bella / Nova / Fenrir / Emma / George), or
+- **Swap the TTS engine** entirely (Piper / XTTS / Orpheus) by replacing the `tts` service in
+  `docker-compose.yml`. (A future *experimental* path: an Orpheus-style model that emits SNAC audio
+  tokens via LM Studio chat + an external SNAC decoder — that's a separate service, not wired here.)
+
+---
+
+## 6. Troubleshooting
+
+| Symptom | Fix |
+|---|---|
+| Brain dropdown shows no LM Studio models | LM Studio not reachable from the container. Confirm **Serve on Local Network** is ON and `curl http://localhost:1234/api/v0/models` works on the host. |
+| `LM Studio unreachable` in `/v1/catalog` | Server stopped, or port ≠ 1234. Set `LMSTUDIO_URL` in `.env`. |
+| Chat errors / long pause on first message | Model JIT-loading in LM Studio (esp. 27B+). Pick a smaller model or pre-load it in LM Studio. |
+| Avatar speaks the `<think>` reasoning | You picked a thinking model not caught by the name heuristic. Use an `-instruct` model, or mark it `thinks: true`. |
+| Want it to "just use whatever I loaded" | Select the **LM Studio (auto)** brain. |
diff --git a/frontend/nginx.conf b/frontend/nginx.conf
@@ -81,6 +81,19 @@ server {
         chunked_transfer_encoding on;
     }
 
+    # Plan #11: direct passthrough to LM Studio's native API on the host
+    # (bypasses the orchestrator) for Lab 1's raw model-latency playground.
+    # Same rationale as /api/ollama/ above. LM Studio must be "Served on Local
+    # Network"; requires host.docker.internal:host-gateway on this service.
+    location /api/lmstudio/ {
+        proxy_pass http://host.docker.internal:1234/;
+        proxy_set_header Host $host;
+        proxy_set_header X-Real-IP $remote_addr;
+        proxy_read_timeout 120s;
+        proxy_buffering off;
+        chunked_transfer_encoding on;
+    }
+
     # Health check
     location /health {
         return 200 'ok';