Multi-GPU model serving and opencode integration

[tobocop2]

Problem

lilbee ran its models on a single GPU, and nothing outside lilbee could use them. A coding agent like opencode had no way to reach lilbee's local models, and lilbee couldn't call tools at all. Indexing a large library was strictly one card's worth of work at a time.

What this adds

Models run across all your GPUs

lilbee now spreads its models across every GPU on the machine instead of running on a single card. A model too big for one card tensor-splits across several; smaller roles pack onto the cards that have room. The whole box goes to work, so bigger models and heavier workloads have somewhere to run, and the fleet serves several requests at once instead of one at a time.

Bulk ingest fanned across the whole box

For indexing a large scanned library, lilbee can run several copies of the OCR and embedding models, one per spare GPU, and spread the work across all of them. A single lilbee sync then saturates every card instead of funneling through one, which is the difference between days and hours on a big corpus. Set vision_replicas / embed_replicas and lilbee places one server per GPU, load-balances pages and embedding batches to the least-busy one, and scales the ingest pipeline's file concurrency to match so the extra cards never sit idle.

opencode can use lilbee's models

lilbee speaks the OpenAI API. Point opencode, or any OpenAI-compatible client, at lilbee and it uses your local models directly: chat, tool calls, and lilbee_search over the same connection.

Tool calling

lilbee's models can call tools now. That is what makes the opencode integration useful: the model searches your library and acts on the results instead of only chatting.

Feedback while models load

lilbee launch opencode (and the other launchers) now show real progress while a cold model loads: a byte bar as the weights are read, then an engine-load spinner, then a clear handoff to the client. Before this it sat silent for what could be minutes on a large model. Once a server is up, repeat launches reuse it and start fast.

Better answers and longer conversations

The RAG path gained grounded refusal (it says when the library has nothing relevant instead of guessing), a context budget so retrieved chunks and history fit the model's window, and a cited-subset so a grounded answer is distinguishable from an off-corpus one. Long agentic conversations no longer overflow a native model's context: history is windowed to the served window instead of failing the turn. Retrieval quality improved too, with asymmetric query embedding for instruction-tuned embedders and LLM-based rerankers served alongside cross-encoders.

Indexes images, not just PDFs

Scanned images now route through the same vision OCR path as PDFs, so a folder of page scans indexes the same way a document does.

Architecture

Each request routes to a hosted model or to the local engine, which runs your models as a fleet of llama-server processes spread across the GPUs (one per role: chat, embedding, reranking, vision; plus extra replicas of a role when configured). A llama-swap supervisor owns the process lifecycle, and gguf-parser sizes each load so the fleet packs onto the available memory.

flowchart TD
    CLI[CLI] --> SVC
    TUI[TUI] --> SVC
    MCP[MCP server] --> SVC
    API[HTTP API] --> SVC

    SVC["App services · RAG pipeline<br/>(embed → rerank → chat)"] --> FAC{{"create_provider<br/>config.llm_provider"}}

    FAC -->|REMOTE| SDK["SdkLLMProvider"]
    FAC -->|AUTO| ROUTE["RoutingProvider<br/>(route by model-ref prefix)"]
    ROUTE -->|"hosted ref"| SDK
    ROUTE -->|"local GGUF ref"| FLEET["FleetProvider"]
    SDK --> CLOUD[("Hosted<br/>OpenAI-compatible API")]

    subgraph ENGINE["Local engine · managed llama-server fleet (one process per role, plus replicas)"]
        direction TB
        CHAT["chat · --jinja<br/>chat + native tool calls"]
        EMB["embed · --embeddings<br/>(×N replicas)"]
        RR["rerank · --pooling rank"]
        VIS["vision / OCR · --mmproj<br/>(×N replicas)"]
    end

    FLEET -->|"spawn · /health · restart<br/>+ localhost OpenAI HTTP"| ENGINE

Loading

On one GPU it is a fleet-of-one; on a multi-GPU host the supervisor packs the servers across cards by memory and, with replicas set, fans a role across the spare cards. The engine installs with lilbee, so there is nothing extra to set up.

Performance

Profiling a 595-page scanned PDF on a single GPU showed where OCR time actually went, and the fixes roughly halved it (this run used one card, not the replica fan-out):

Metric	Before	After
595-page scan, indexed end to end on one GPU	36 min	18 min
OCR pages in flight at once	1 (strictly sequential)	up to vision_ocr_concurrency (4) per server, batched on the vision server's slots
Worst page (a repetition runaway)	29,779 chars over 51 s, uncapped	capped at vision_ocr_max_tokens (4096)

Rasterization turned out to be negligible (1.9% of wall time); the real costs were a single-sequence decode that left the GPU about half idle and the occasional page that looped into a repetition, which the two changes above address. Embedding also regained a token/sequence-aware sub-batching step, so a large library no longer overruns the embed server's batch limit while indexing. Those numbers are a single GPU with one vision server; with vision_replicas the same OCR work fans across every card on the box on top of this per-GPU gain, so a multi-GPU host scales it further.

Supported models

These families were verified end to end through the local engine, so opencode can chat and call tools against them:

Family	Verified ref
Qwen3	Qwen3-4B, Qwen3-Coder-30B-A3B
Llama 3.1	Meta-Llama-3.1-8B-Instruct
Gemma 4	gemma-4-E2B-it
GLM	GLM-4.5-Air
MiniMax	MiniMax-M2
gpt-oss	gpt-oss-120b

Other families were tried but don't tool-call reliably through the engine yet: some ship no tool-call template (Hermes 3, GLM-4-9B), some never dispatch a tool in practice (functionary, the small DeepSeek-R1 distills, Granite, Phi-4-mini), and Mistral Nemo still errors on the fleet path for some message sequences. Very small or reasoning-only models may also decline to call tools.