Hey Look, It's an LLM

Last updated: 2026-09-23

Local multimodal LLM API server with an Anthropic Messages-conformant inference endpoint, a vanilla-JS web UI, and on-the-fly model swapping.

Built on Apple MLX for text and vision, with GGUF models served through a managed llama.cpp llama-server subprocess -- one API, one UI, per-model engine choice.

Documentation: start with the wiki -- how the system works end to end, from the API and the web UI down to how llama-server is built and spawned and where caching happens. The full doc index is docs/README.md. What the project is for and the principles it holds to: VISION.md.

Features

• One inference API: Anthropic Messages-conformant /v1/messages with typed content blocks (text, image, thinking) plus documented heylook extensions; /v1/models keeps the OpenAI-shaped list for discovery clients. The OpenAI-compatible /v1/chat/completions route was removed in v1.79.66: no page of the web UI had used it since v1.74.0 and the owner's other project speaks /v1/messages, so one generation now has one grammar.
• Two providers: MLX text + vision (mlx-vlm's engine, with a streaming detokenizer vendored from mlx-lm), and GGUF via a managed llama-server subprocess (vision mmproj sidecars, audio input, speculative decoding/MTP, prefix caching)
• Thinking blocks: format-aware reasoning parsing driven by the model's own chat template (Qwen <think> styles, gemma channel format), with round-trip editing, streaming, and a per-request toggle
• Vision and audio input: images on both APIs and in the chat UI; audio clips (WAV/MP3/FLAC) on GGUF models -- MLX models reject audio with a clear 400
• Model management: watch folders, configure, load/unload from the web UI or API
• Local-only logging: opt-in JSONL metrics/events under logs/

Things to know (heylook-specific)

• Defaults: port 8000 (--port), bound to 127.0.0.1 (--host 0.0.0.0 to expose it on your network). HTTPS is not built in; put your own reverse proxy in front if you need it.
• Opt-in admin auth, off by default: HEYLOOK_ADMIN_TOKEN gates admin and destructive endpoints (X-Heylook-Admin-Token header). Inference has no key: the server is meant for a trusted home LAN.
• heylook.toml is override-only. Anything under a [scan].folders watch folder is served automatically with defaults derived from the model's own files (modalities, chat template, sampling, KV-cache sizing) -- a new download needs no edit. A [[models]] entry always wins as an explicit override; the Models page's config editor writes one when you change a setting.
• One model resident by default (max_loaded_models = 1): LRU eviction, optional idle unload (idle_unload_seconds). Loading a model on request is disclosed in the UI, never confirmed away.
• GGUF needs a llama-server binary, built by uv run scripts/build_llama.py (uv sync cannot build C++). One subprocess per loaded model. The chat template is the operator override chat_template.heylook.jinja beside the .gguf (what the models page's template editor writes) if there is one, else a chat_template.jinja beside it, else the template embedded in the GGUF by the quant publisher; chat_template_path overrides all of them, and use_sidecar_chat_template = false keeps the embedded one over the chat_template.jinja without deleting the file. Every spawn logs which one it used. An omitted max_tokens gets a 16384 default (an explicit value always wins); llama-server's own "unlimited" default is never passed through.
• Telemetry is off by default (observability_level, settable via PUT /v1/admin/config); raising it writes JSONL under logs/, local files only.
• Server-side storage (conversations, notebooks, presets, settings) is a single DuckDB file; HEYLOOK_DB_PATH relocates it, HEYLOOK_LOGS_DIR the log dir.

Web UI

Vanilla JS frontend at / -- no bundler, no node_modules, no build step; served directly by the backend. Conversations, notebooks, and presets are stored server-side in DuckDB (messages as content blocks; images round-trip).

Pages: Chat (streaming with thinking blocks, capability-gated image/audio attach, per-conversation system prompt + presets, honest mid-conversation model switching), Notebook (base-model continuation), Models (watch folders, load/unload, schema-driven per-model config editor), and Performance.

Build contract: docs/frontend_v3_spec.md (§4 = the backend API contract).

Quick Start

git clone https://github.com/fblissjr/heylookitsanllm
cd heylookitsanllm
uv sync    # one step -- full runtime + dev tooling, no extras to remember

# point it at your models: add a watch folder to heylook.toml ([scan].folders)
# or on the Models page; everything under it is served with no entry

heylookllm --log-level INFO             # serves API + UI on :8000

Then open http://localhost:8000/.

Run it as a background service with heylookllm service install (status|start|stop|restart|uninstall; --host 0.0.0.0 for LAN).

Adding models

Put the model under a watch folder ([scan].folders in heylook.toml, the Models page, or PUT /v1/admin/models/scan-config). Everything under one is served with no entry, with its settings derived from the model's own files at load time. The scan understands MLX/safetensors dirs and GGUF dirs (mmproj projectors and mtp-* drafter sidecars auto-paired). A heylook.toml entry exists only to override something. Formats: heylook.example.toml (the server config, including allowed_hosts for reaching the server by a LAN or VPN name) and model.heylook.example.toml (a model's own settings, in its folder).

After hand-editing heylook.toml on a running server: curl -X POST http://localhost:8000/v1/admin/reload.

API

Interactive docs at http://localhost:8000/docs; live schema at /openapi.json. Key endpoints: /v1/messages (inference), /v1/conversations/{id}/generate (server-owned chat), /v1/models.

A first call, streaming off:

curl -s http://localhost:8000/v1/messages \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "<an id from GET /v1/models>",
    "max_tokens": 256,
    "messages": [{"role": "user", "content": "Say hello."}]
  }'

Images ride the same route as content blocks with a nested source ({"type": "image", "source": {"type": "base64", "media_type": "image/jpeg", "data": "..."}}); downscale before sending, the server does not resize.

Wiring an external app to this server: docs/api_integration.md covers which endpoint to pick, capability discovery, the image block shape, the SSE grammar and the error taxonomy -- the parts a schema cannot express.

Related apps

• apps/batch-labeler/ -- standalone CLI for labeling image directories with VLMs. Needs a port: it still posts to /v1/chat/completions, removed in v1.79.66, so it 404s against a current server.
• apps/optloop-lib/ -- library-level benchmark harness for mlx-lm/mlx-vlm fork experiments (guide)

Monitoring

With observability_level raised, tail -f logs/metrics.jsonl (per-request metrics) or logs/events.jsonl (errors + model lifecycle), or point DuckDB at the files for aggregates: duckdb -c "SELECT model, quantile_cont(generation_tps,0.95) FROM read_json_auto('logs/metrics.jsonl') GROUP BY model".

Troubleshooting

heylookllm --log-level DEBUG

License

MIT License -- see LICENSE