TensorSharp is a .NET 10 inference engine for local GGUF models. Run it on Windows, macOS or Linux through the CLI, browser chat, or Ollama/OpenAI-compatible APIs, or embed it in your own .NET application. Use managed C# CPU kernels or native CUDA, Metal and Vulkan backends, with support that varies by model.
Current source covers text and reasoning, multimodal input, embeddings, image generation/editing, video with audio, personal knowledge collections, and Agent Skills with code tools. TensorSharp also powers TensorAgent, the local app for iPhone, iPad, Mac and Windows. Start below, or check the project status for capabilities and validation limits.
- Local models, several interfaces. One engine for CLI use, browser chat and Ollama/OpenAI-compatible APIs, across managed CPU and native accelerator backends.
- Text and multimodal models. Dense and MoE GGUF models, reasoning, image/audio input and document questions. See Supported models for each family's capabilities.
- Embeddings and media generation. EmbeddingGemma 2 provides text/code embeddings and optional image, audio and video inputs. Generate and edit images with Qwen-Image-2.1 and Turbo, or generate video with stereo audio using LTX 2.5 Distilled (48 kHz) and MiniMax-H3 (32 kHz).
- Editable video projects. Video Studio in WebUX Chat and TensorAgent shares a persistent multitrack editor, native generation jobs, reviewed agent proposals, undo/redo and local FFmpeg export. Long videos are made as chains of takes, each starting on the previous segment's last frame; reference photos keep a character in new shots. See the validation matrix for tested models and platform limits.
- Efficient serving. Continuous batching and a paged, Radix prefix-shared KV cache are on by default. Speculative decoding and multi-GPU placement are available for supported models. See Features.
- Agent Skills and code tools.
TensorSharp.AgentHostadds file, shell and document workflows. The 17 bundled skills include research, documents, TDD, design interviews and handoff;find-skillssearches, inspects and installs additional skills when network access is enabled. Server and TensorAgent chats also support bounded sub-agent delegation with private workspaces and read-only defaults. - Personal knowledge collections. Upload files or nested folders, and import saved chats with their attachments and artifacts. Select collections per chat, retrieve cited excerpts with page/row/time locations, preview originals, or let code agents open full datasets. Local BM25 works by default; optional GGUF embeddings add semantic search, and compatible media models index images, audio and sampled video. Extraction status and warnings show coverage limits.
- Steer an active response. Submit a new instruction while the CLI, server Web UI or TensorAgent is reasoning or answering. The next response uses the conversation, partial output and your guidance; Stop remains available.
- Latest recorded comparisons. DeepSeek V4.1 Flash 1017 on five A40 GPUs and Muse Spark on ten have workload-specific comparisons against llama.cpp. In the matched ~1.8K DeepSeek prompt, exact TP5 prefill was 3.23× the reference rate; default approximate YOCO was 5.50× with a measured quality trade-off. See Benchmarks for the exact settings, slower cells and limitations.
- TensorAgent apps. Local chat, attachments, skills and saved work on phones and desktops. See the Mac/Windows installation guide and app coverage notes.
| Qwen inference and agentic runtimes | Gemma 4 and multimodal inference |
|---|---|
 |
 |
| Building LLM Inference Engines and Agentic Runtimes from Scratch: Qwen Dense and MoE Models with TensorSharp and TensorAgent | From Tensors to Tokens: Building a Multimodal LLM Inference Engine from Scratch with TensorSharp and Gemma 4 E4B |
| Build Qwen dense/MoE inference and controlled agent workflows in C#. Follow tensors, tokenization, attention, expert routing, quantization, and caching through GPU acceleration, multimodal execution, tools, skills, sandboxed code, and desktop/mobile deployment with TensorSharp and TensorAgent. | Build a multimodal inference engine in C#/.NET with Gemma 4 E4B, from tensors, GGUF model loading, quantization, and tokenization to text, image, video, and audio execution. Connect correctness checks and serving optimizations to the running TensorSharp code. |
| Buy on Amazon | Buy on Amazon |
Explore both books and their repository reading paths
TensorSharp/TensorAgent is free. If you like it, a coffee keeps the work on it going.
The Releases page provides self-contained CLI and Server archives for Windows x64 (CPU/CUDA), Linux x64 (CPU/CUDA), and macOS arm64.
To build from source you need the full .NET 10 SDK (how to install it), git, curl, CMake 3.20+, and the toolchain for your GPU. Then run the verified Gemma 4 E4B model (7.48 GiB). On Windows with an NVIDIA GPU (PowerShell):
git clone https://github.com/zhongkaifu/TensorSharp.git; Set-Location TensorSharp
New-Item -ItemType Directory -Force models | Out-Null
curl.exe -L --fail "https://huggingface.co/ggml-org/gemma-4-E4B-it-GGUF/resolve/main/gemma-4-E4B-it-Q8_0.gguf?download=true" -o models\gemma-4-E4B-it-Q8_0.gguf
'Answer in one short sentence: what is TensorSharp?' | Set-Content prompt.txt
$env:TENSORSHARP_GGML_NATIVE_ENABLE_CUDA = 'ON'
dotnet run --project TensorSharp.Cli -c Release -p:TensorSharpSkipMlxNative=true -- --model models\gemma-4-E4B-it-Q8_0.gguf --input prompt.txt --max-tokens 128 --backend ggml_cudaOn other machines, change the backend (see Pick a backend):
- macOS (Apple Silicon): drop the CUDA environment variable and use
--backend ggml_metal. - Linux + NVIDIA: prefix the
dotnet runwithTENSORSHARP_GGML_NATIVE_ENABLE_CUDA=ONand use--backend ggml_cuda. - AMD / Intel / NVIDIA Vulkan: set
TENSORSHARP_GGML_NATIVE_ENABLE_VULKAN=ONand use--backend ggml_vulkan.
Host the same model as a server: a browser chat at http://localhost:5000 plus Ollama- and OpenAI-compatible APIs.
dotnet run --project TensorSharp.Server.Host -c Release -p:TensorSharpSkipMlxNative=true -- --model models/gemma-4-E4B-it-Q8_0.gguf --backend ggml_cuda --max-tokens 512The server listens on
0.0.0.0:5000with no built-in authentication or TLS; keep it behind a firewall or an authenticated HTTPS reverse proxy.
| Your hardware | Backend |
|---|---|
| Apple Silicon (Mac) | --backend ggml_metal |
| Windows / Linux + NVIDIA GPU | --backend ggml_cuda |
| Windows / Linux + AMD / Intel / NVIDIA GPU | --backend ggml_vulkan |
| No GPU | --backend ggml_cpu (native kernels), or --backend cpu (pure C#, no native dependencies) |
The Getting started guide has the rest: installing the SDK on each platform, multi-GPU and multi-node runs, NVIDIA DGX Spark, multimodal input, embeddings, and making it fast. Every option is in the CLI and Server references, and both programs print them with --help.
dotnet build TensorSharp.slnx also builds TensorAgent's available desktop heads and the iOS simulator head on Apple Silicon when the selected SDK has the required MAUI workloads and staged native/Python files. Missing prerequisites skip the affected app head with a warning; see TensorAgent build instructions.
Download the latest release and expand Assets. Choose a file beginning with tensoragent-desktop-:
| Platform | Download and install |
|---|---|
| macOS 14+ on Apple Silicon | tensoragent-desktop-<version>-osx-arm64.dmg: open and drag TensorAgent to Applications. A PKG installer and ZIP are also available. |
| Windows x64 | tensoragent-desktop-<version>-win-x64-cpu.msi, or win-x64-cuda.msi for a compatible NVIDIA GPU/driver: install and open TensorAgent from Start. ZIPs are also available. |
The app includes its .NET runtime and native engine. Windows needs WebView2 Evergreen Runtime if missing; Python/Node are optional tools for skills. The current built-in catalog needs at least 12 GB system RAM, and model weights download separately. Open ☰ → Models → Download → Use, then type a message. The Desktop user guide covers package verification, unsigned-app prompts, first-run setup, attachments, skills, updates and troubleshooting. Historical releases may have no Desktop assets until a release runs the updated workflow; iPhone/iPad remain source builds.
- Text, reasoning, and multimodal LLMs: DeepSeek V4 Flash / V4.1 Flash, GLM 5.x, Gemma 4, Qwen 3.5 / 3.6 / 3.8 27B, Qwen 3.8 Flash Next, Bonsai2 (Qwen family), GPT OSS, Nemotron-H, Mistral 3, Hunyuan Dense, Muse-Glimmer, Muse Spark (816B MoE, multi-GPU), and MiniMax-M3.1-Flash (428B MoE, multi-GPU).
- Text diffusion: DiffusionGemma, including Jev typed decision inference at
/v1/systemone, over text, images, uploaded documents, sampled video frames and audio transcripts (configured ASR companion). - Image generation/editing and video generation: Qwen-Image-2.1, LTX 2.5 Distilled (video + 48 kHz stereo audio), MiniMax-H3 (video + stereo audio), and Wan 2.1 / 2.2.
- Embeddings: EmbeddingGemma 2, including text/code and optional image, audio, and video inputs; BERT / XLM-R encoders including Snowflake Arctic Embed L v2.0 and all-MiniLM-L6-v2.
Backend, modality, feature support, and validation coverage vary by model. See the supported models tables, the model cards, and the embedding guide for details.
TensorAgent's current catalog has 38 entries, grouped into searchable model families with memory and download filters, including LTX 2.5 Distilled at an experimental 48 GB memory tier, Qwen-Image-2.1 Turbo at 4-bit and 8-bit quantizations, fourteen image LoRA plug-ins and two MiniMax-H3 Turbo speed plug-ins. The expanded catalog includes experimental entries whose memory tiers and app/device behavior have not all been measured. Qwen3.8 Flash Next uses SSD-backed weights; its Q2 package was measured on a 48 GB Mac, and IQ1_M has separate Windows catalog-host validation. Multi-GPU --layer-split and --tp are separate controls; support and performance depend on the architecture and quantization. See TensorAgent's coverage notes and the model cards. Desktop installers appear in releases built with the updated Release Binaries workflow, while older releases may lack them.
One engine, four ways to use it, each an unedited capture of a real run.
 TensorSharp.Cli Models in your terminal |
 TensorSharp.Server.Host Web UI chat and Ollama/OpenAI-compatible APIs |
 TensorAgent · iPhone simulator CPU inference and in-app Python; this is a simulator capture |
 TensorAgent on Mac A saved Qwen-Image 2.1 edit in the Mac app |
What each run shows, step by step: Screenshots.
Latest recorded comparisons, audited on 2026-10-23. DeepSeek V4.1 Flash 1017 was measured on five A40 46 GB GPUs on October 22–23; Muse Spark's ten-A40 report was refreshed October 22 (its exact measurement date is not recorded). These are different models and workloads, not a combined engine speedup.
| Model / placement / workload | TensorSharp tok/s | llama.cpp tok/s | Ratio |
|---|---|---|---|
| DeepSeek 1017, TP5, ~1.8K prefill, exact | 563.9 | 174.6 | 3.23× |
| DeepSeek 1017, TP5, ~1.8K prefill, approximate YOCO default | 960.5 | 174.6 | 5.50× |
| DeepSeek 1017, TP5, ~7.3K prefill, exact | 549.7 | 225.7 | 2.44× |
| DeepSeek 1017, TP5, ~7.3K prefill, approximate YOCO default | 1,000.7 | 225.7 | 4.43× |
| Muse Spark, layer split 10, 4,075-token prefill | 999.7 | 483.2 | 2.07× |
| Muse Spark, layer split 10, 15,767-token prefill | 1,346.8 | 826.1 | 1.63× |
| Muse Spark, TP10, one-request decode | 22.63 | 16.19 | 1.40× |
| Muse Spark, TP10, four-request aggregate output | 45.73 | 24.49 | 1.87× |
DeepSeek: UD-IQ2_XXS, 32,768-token context, one request, 256 output tokens, greedy/thinking off; prefill uses internal server timers, not HTTP TTFT. TensorSharp TP5 is compared with the reference's supported layer split. Default YOCO is approximate: wikitext-2 perplexity was 2.4978, versus 2.4811 with exact prefill and 2.4697 for llama.cpp. 5.50× applies only to that approximate ~1.8K prefill cell, not decode or general engine performance. Model, method and provenance.
Muse Spark: UD-IQ3_XXS, 32,768-token context, F16 KV, greedy/thinking off; matching placement modes, 32 output tokens for long-prompt prefill and 256 for short decode / concurrent requests. Prefill is uncached prompt tokens divided by streamed TTFT. Layer-split decode was slightly slower: 16.69 vs 16.79 tok/s (0.99×). On this host P2P was broken and llama.cpp tensor split ran without NCCL, while TensorSharp used NCCL; that limits the TP comparison. Full report.
Methods, repetitions, quality trade-offs and build limitations: Benchmarks. No new benchmark runs are claimed; these results do not qualify every backend or the published release binaries.
MiniMax-M3.1-Flash on eight A40 GPUs (measured 2026-10-24, separately from the table above): its comparison with llama.cpp.
New here? The sections above are all you need to get running. Everything else is detailed reference:
| Doc | What's inside |
|---|---|
| TensorAgent Desktop user guide | Mac/Windows downloads, DMG/PKG/MSI/ZIP installation, model setup, first chat, attachments, skills, updates and troubleshooting |
| TensorSharp and TensorAgent book guide | Building LLM Inference Engines and Agentic Runtimes from Scratch, plus From Tensors to Tokens: introductions, Amazon links, and repository reading paths |
| Getting started | The full first-run guide: the .NET SDK on each platform, every backend, multi-GPU and multi-node runs, NVIDIA DGX Spark, embeddings, choosing a backend, and making it fast |
| Supported models | Implemented model families and their validation scope: example GGUFs, modalities, thinking, tools, and speculative decoding |
| Benchmarks | Latest scoped DeepSeek 1017 and Muse Spark comparisons, including approximation, quality and build limits |
| Screenshots | The CLI, the Web UI, and TensorAgent on iPhone and Mac at work, with what each run did |
| Model Downloads | Per-model huggingface-cli download + run quick reference (quant tiers, projectors, companions) |
| Usage | Full CLI reference (options, interactive REPL, JSONL batch), server hosting, logging, HTTP API examples, backends, and the env-var matrix |
| Multi-node inference | Whole-layer pipelines and tensor parallelism across nodes, supported stage executors, shared transport and validation |
| Features | Deep dives on continuous batching, speculative decoding, tool calling, thinking mode, multimodal, MoE, KV codecs, and more |
| Configuration files | Put options in a reusable JSON file with ${variables} and auto-downloading models |
| Development | Prerequisites, building the native GGML/MLX libraries, repository layout, package boundaries, internal architecture, and the test harness |
| Per-model architecture cards | End-to-end docs of each architecture (forward graph, components, parameters, prefill/decode optimizations) |
| Paged attention & continuous batching | The vLLM-style paged KV cache, prefix sharing, and iteration-level scheduler |
| Agent Skills & agentic work | The SKILL.md format, progressive disclosure and its budget, the in-process tool loop, sandboxed code execution, workspaces and artifacts, the path/ZIP/exec security model, and the HTTP + C# surfaces |
| Multiple agents | Automatic task delegation, private child workspaces, dependency scheduling, permission limits, server controls, and reproducible evaluation |
| Personal knowledge | Collections, folder and saved-chat imports, document/PDF/media extraction, local retrieval, cited previews, dataset tools, APIs and validation limits |
| Browser automation skill (Playwright) | Running the bundled playwright skill, which drives a browser through @playwright/cli via skills_run: the flags it needs, the macOS Chromium-sandbox config, account handoff, and TensorAgent desktop hosting (not iOS) |
| Speculative decoding | The three-layer design (model adapter / algorithm / speculator weights), the shipped auto / draft-head / block / ngram algorithms, and what to write to add a new one |
| Environment variable feature matrix | Which high-impact runtime flags affect which models, backends, and prompt types |
| Muse Spark vs llama.cpp | The 816B MoE on ten A40 GPUs, in layer split and tensor parallelism: setup, per-scenario results, teacher-forced quality, and the limits of the comparison |
| Test/benchmark matrix runner | Sweep model × backend × feature × env-var cells and generate regression reports |
| Server API examples | Complete curl and Python examples for the server surface |
Actively developed, and the source tree runs ahead of the published packages.
| Area | Where it stands |
|---|---|
| Models | A dozen autoregressive families plus text diffusion, image generation and editing, and video with audio. See Supported models. |
| Inference hosts | CLI, Web UI, Ollama- and OpenAI-compatible APIs, and TensorAgent for iPhone, iPad, Mac and Windows. The release workflow packages Mac/Windows Desktop; historical releases may lack those assets. iPhone/iPad use source builds. |
| Backends | Pure C# CPU, direct CUDA/cuBLAS, MLX Metal, and GGML CPU/Metal/CUDA/Vulkan, with per-architecture exceptions. |
| Serving features | Continuous batching with a shared prefix cache, speculative decoding, tensor parallelism, structured output, and tool calling. |
| Agentic work | Agent Skills, sandboxed file and shell tools, and bounded sub-agents. See Agent Skills and Multiple agents. |
| Knowledge | Local document, dataset and media collections; folder/saved-chat import in TensorAgent, per-chat scope, lexical/optional semantic retrieval, source previews and complete-dataset agent tools. Media extraction and OCR have explicit coverage limits. See Personal knowledge. |
| TensorAgent | 38 catalog entries with family search and filters, saved chats and artifacts, personal knowledge, response steering, masked image edits, 8-step Turbo, fourteen image LoRAs and two MiniMax-H3 Turbo plug-ins, eight interface languages, and persisted text-turn statistics. Expanded catalog entries remain experimental; iOS media generation and Windows image/audio/video generation remain unverified. |
Per-area detail (which architecture runs on which backend, which features each family supports, and the known limits) is in the status matrix.
Zhongkai Fu
See LICENSE for details.