feat: mcp

docs/cli/mcp.md

@@ -0,0 +1,63 @@
+---
+title: stardust mcp
+description: Serve a built site's docs to AI clients over the Model Context Protocol (stdio).
+---
+
+# stardust mcp
+
+Serve a built Stardust site to AI clients (Claude Desktop, Cursor, and other [MCP](https://modelcontextprotocol.io/) clients) over the stdio transport. See [MCP Server](/features/mcp) for the full guide.
+
+## Usage
+
+```bash
+stardust mcp [dir] [options]
+```
+
+`[dir]` is the built site directory. When omitted, it resolves from `build.outDir` in `stardust.yaml`, falling back to `dist`.
+
+The site must be built first — the server reads the `llms.json` manifest and per-page `.md` files that `stardust build` emits (with `build.llms` enabled, the default).
+
+## Examples
+
+```bash
+# Build, then serve dist/ over MCP
+stardust build
+stardust mcp dist
+
+# Inside a project, resolve the output dir from stardust.yaml
+stardust mcp
+```
+
+## What it exposes
+
+- **`list_pages`** tool — the table of contents (path, title, description), with an optional path-prefix filter.
+- **`search_docs`** tool — full-text search across the documentation.
+- **`read_page`** tool — the full markdown of a page by its path.
+- **Resources** — one `text/markdown` resource per page.
+
+## Serve over HTTP
+
+By default the server speaks the **stdio** transport (clients launch it as a subprocess). Pass `--http` to serve a live **Streamable HTTP** `/mcp` endpoint instead — for self-hosters who run the binary as a service:
+
+```bash
+stardust mcp dist --http --port 8080
+# → Endpoint: http://localhost:8080/mcp
+```
+
+It binds to `localhost` and validates the `Origin` header (DNS-rebinding guard); loopback origins and non-browser clients are always allowed, other browser origins need `--allow-origin`. See [MCP Server](/features/mcp) for details.
+
+## Options
+
+| Option | Default | Description |
+|--------|---------|-------------|
+| `[dir]` | `build.outDir` (else `dist`) | Built site directory to serve |
+| `-c`, `--config` | `stardust.yaml` | Config file, used only to resolve the output dir when `[dir]` is omitted |
+| `--http` | off | Serve Streamable HTTP (a live `/mcp` endpoint) instead of stdio |
+| `-p`, `--port` | `8080` | Port for `--http` |
+| `--host` | `localhost` | Bind host for `--http` |
+| `--allow-origin` | — | Allowed browser `Origin` for `--http` (repeatable; `*` for any) |
+
+## Notes
+
+- In stdio mode, communication is JSON-RPC on **stdin/stdout** and all logs go to **stderr**, keeping stdout a clean protocol channel. In `--http` mode, requests come over `POST /mcp`.
+- The server is **read-only** and holds no state between runs.

docs/features/llm-output.md

@@ -70,6 +70,34 @@ https://example.com/guide.md   → its markdown source
 Readers get the same thing through the **Copy page as Markdown** button at the
 top of each page — one click to paste a page into an AI chat.
 
+## Machine-Readable Manifest (llms.json)
+
+Alongside `llms.txt`, Stardust writes `llms.json` — a structured index of every
+page with its `path`, `title`, `description`, absolute `url`, and the `md` path
+to its raw markdown:
+
+```json
+{
+  "name": "My Project",
+  "description": "…",
+  "url": "https://example.com",
+  "generator": "stardust",
+  "pages": [
+    { "path": "/guide", "title": "Guide", "description": "…",
+      "url": "https://example.com/guide", "md": "/guide.md" }
+  ]
+}
+```
+
+Where `llms.txt` is written for a model to read, `llms.json` is written for a
+program to parse: a remote agent can fetch it, enumerate pages, and pull exactly
+the `.md` files it needs — consuming the whole site with no server to run. It is
+also what powers the [MCP Server](/features/mcp): `stardust mcp` serves a built
+site to Claude, Cursor, and other AI clients straight from this manifest.
+
+Pages with `llm: false` are excluded here too. The manifest is written whenever
+`build.llms` is enabled (the default).
+
 ## Use Cases
 
 ### AI Chatbots

docs/features/mcp.md

@@ -0,0 +1,89 @@
+---
+title: MCP Server
+description: Serve any built site's docs to Claude, Cursor, and other AI clients over the Model Context Protocol — locally, air-gapped, no service to run.
+---
+
+# MCP Server
+
+Every hosted docs platform now auto-generates an [MCP](https://modelcontextprotocol.io/) server so AI assistants can search and read the docs. Stardust brings that to **static, self-hosted sites**: `stardust mcp` turns the binary you already have into an MCP server over any built site — no service to run, works air-gapped.
+
+A Claude or Cursor user points their client at `stardust mcp <dir>` and gets:
+
+- **`list_pages`** — the table of contents (path, title, description), optionally filtered by a path prefix.
+- **`search_docs`** — full-text search across your documentation, returning matching pages with snippets.
+- **`read_page`** — the full markdown of any page by its path.
+- **Resources** — every page exposed as a `text/markdown` resource, so clients can browse and attach pages directly.
+
+The server reads a **built** site — its `llms.json` manifest and the per-page `.md` files that `stardust build` already emits. It never re-parses your markdown and needs no `stardust.yaml` at serve time, so it can serve any built site, even one you didn't build.
+
+## Usage
+
+Build your site, then serve it:
+
+```bash
+stardust build          # emits dist/, including llms.json + per-page .md
+stardust mcp dist       # serves dist/ over MCP on stdio
+```
+
+With no directory argument, `stardust mcp` resolves the output directory from `stardust.yaml` (falling back to `dist`), so inside a project you can just run:
+
+```bash
+stardust mcp
+```
+
+The server speaks the [stdio transport](https://modelcontextprotocol.io/specification/2025-06-18/basic/transports): it reads JSON-RPC requests on stdin and writes responses on stdout. All logging goes to stderr, so stdout stays a clean protocol channel.
+
+## Connect a client
+
+MCP clients launch the server as a subprocess. Point yours at the `stardust` binary with `mcp` and your built directory.
+
+**Claude Desktop / Cursor** (`claude_desktop_config.json` or the client's MCP settings):
+
+```json
+{
+  "mcpServers": {
+    "my-docs": {
+      "command": "stardust",
+      "args": ["mcp", "/absolute/path/to/dist"]
+    }
+  }
+}
+```
+
+Restart the client and your docs appear as a connected server — search and read your guides without leaving the chat.
+
+## Serve over HTTP (self-hosting)
+
+Hosted docs platforms expose a live `https://yoursite/mcp` endpoint because they run a server for you. If **you** run a server — a VPS, on-prem box, or air-gapped host — `--http` gives you the same thing:
+
+```bash
+stardust mcp dist --http --port 8080
+# → Endpoint: http://localhost:8080/mcp
+```
+
+This is the MCP [Streamable HTTP transport](https://modelcontextprotocol.io/specification/2025-06-18/basic/transports): a single `/mcp` endpoint answering JSON-RPC over `POST`. Point any MCP client that accepts a URL at `http://your-host:8080/mcp`.
+
+<Warning>
+A **pure static host** (GitHub Pages, Netlify, a plain CDN) can't serve `--http` — a live endpoint needs a running process. Those deployments use the static [`llms.json` + `.md` files](/features/llm-output) instead (see below). `--http` is for when you actually run the binary as a service.
+</Warning>
+
+**Security.** The server binds to `localhost` by default and validates the `Origin` header to block [DNS-rebinding](https://modelcontextprotocol.io/specification/2025-06-18/basic/transports#security-warning) attacks: loopback origins and non-browser clients (which send no `Origin`) are always allowed; other browser origins must be listed with `--allow-origin`. To expose it beyond localhost, set `--host 0.0.0.0` and add each browser origin (`--allow-origin https://app.example.com`, or `--allow-origin '*'` to accept any), ideally behind your own TLS/auth proxy. The server is read-only and stateless.
+
+## Options
+
+| Option | Default | Description |
+|--------|---------|-------------|
+| `[dir]` | `build.outDir` (else `dist`) | The built site directory to serve |
+| `-c`, `--config` | `stardust.yaml` | Config file, used only to resolve the output dir when `[dir]` is omitted |
+| `--http` | off | Serve Streamable HTTP (a live `/mcp` endpoint) instead of stdio |
+| `-p`, `--port` | `8080` | Port for `--http` |
+| `--host` | `localhost` | Bind host for `--http` |
+| `--allow-origin` | — | Allowed browser `Origin` for `--http` (repeatable; `*` for any) |
+
+## Consume without a server
+
+The same build also produces a static, machine-readable manifest — `llms.json` — plus per-page `.md` files and `llms.txt`. Remote agents that can't launch a subprocess can fetch these directly over HTTP, no MCP server required. See [LLM-Friendly Output](/features/llm-output).
+
+<Note>
+The MCP server is **read-only** and requires a built site. Run `stardust build` (with `build.llms` enabled, the default) first — the manifest and per-page markdown it emits are what the server serves.
+</Note>

lib/src/cli/cli_runner.dart

@@ -10,6 +10,7 @@ import 'commands/clean_command.dart';
 import 'commands/dartdoc_command.dart';
 import 'commands/dev_command.dart';
 import 'commands/init_command.dart';
+import 'commands/mcp_command.dart';
 import 'commands/new_command.dart';
 import 'commands/openapi_command.dart';
 import 'commands/serve_command.dart';
@@ -30,6 +31,7 @@ class StardustCliRunner extends CommandRunner<int> {
     addCommand(CleanCommand());
     addCommand(OpenApiCommand());
     addCommand(DartdocCommand());
+    addCommand(McpCommand());
 
     argParser.addFlag(
       'version',

lib/src/cli/commands/mcp_command.dart

@@ -0,0 +1,116 @@
+import 'dart:async';
+import 'dart:convert';
+import 'dart:io';
+
+import 'package:args/command_runner.dart';
+import 'package:path/path.dart' as p;
+import 'package:shelf/shelf.dart' as shelf;
+import 'package:shelf/shelf_io.dart' as shelf_io;
+
+import '../../config/config_loader.dart';
+import '../../core/file_system.dart';
+import '../../mcp/docs_source.dart';
+import '../../mcp/mcp_http_server.dart';
+import '../../mcp/mcp_server.dart';
+import '../../utils/exceptions.dart';
+import '../../utils/logger.dart';
+import '../../version.dart';
+
+/// Serve a built Stardust site's docs to AI clients (Claude Desktop, Cursor)
+/// over MCP — stdio by default, or Streamable HTTP with `--http`.
+class McpCommand extends Command<int> {
+  final FileSystem fileSystem;
+
+  @override
+  final name = 'mcp';
+
+  @override
+  final description = 'Serve a built site to AI clients over MCP (stdio, or --http)';
+
+  @override
+  String get invocation => 'stardust mcp [dir]';
+
+  McpCommand({FileSystem? fileSystem}) : fileSystem = fileSystem ?? const LocalFileSystem() {
+    argParser
+      ..addOption('config',
+          abbr: 'c', help: 'Path to stardust.yaml (used only to resolve the output dir)', defaultsTo: 'stardust.yaml')
+      ..addFlag('http', help: 'Serve over Streamable HTTP (a live /mcp endpoint) instead of stdio', negatable: false)
+      ..addOption('port', abbr: 'p', help: 'Port for --http', defaultsTo: '8080')
+      ..addOption('host', help: 'Host to bind for --http', defaultsTo: 'localhost')
+      ..addMultiOption('allow-origin',
+          help: 'Allowed browser Origin for --http (repeatable; "*" for any). '
+              'Loopback origins and non-browser clients are always allowed.')
+      ..addFlag('verbose', abbr: 'v', help: 'Verbose logging (to stderr)', negatable: false);
+  }
+
+  @override
+  Future<int> run() async {
+    final args = argResults;
+    if (args == null) return 1;
+
+    final http = args['http'] == true;
+    // In stdio mode, stdout is the protocol channel, so logs go to stderr only.
+    // In HTTP mode, stdout is free — log there like `stardust serve`.
+    final logger = http
+        ? Logger(onLog: stdout.writeln, onError: stderr.writeln)
+        : Logger(onLog: stderr.writeln, onError: stderr.writeln);
+    final configPath = args['config'] as String;
+
+    final dir = switch (args.rest) {
+      [final dir, ...] => dir,
+      [] when await fileSystem.fileExists(configPath) =>
+        p.normalize((await ConfigLoader.load(configPath, logger: logger)).build.outDir),
+      [] => 'dist',
+    };
+
+    try {
+      final source = await DocsSource.load(dir, fileSystem: fileSystem, logger: logger);
+      final server = McpServer(source, name: source.siteName, version: version, logger: logger);
+
+      if (http) {
+        final port = int.tryParse(args['port'] as String);
+        if (port == null || port < 0 || port > 65535) {
+          stderr.writeln('❌ Invalid port: ${args['port']}');
+          return 1;
+        }
+        return _serveHttp(source, server, logger,
+            host: args['host'] as String, port: port, allowedOrigins: (args['allow-origin'] as List<String>).toSet());
+      }
+
+      logger.log('📡 stardust mcp — serving ${source.pages.length} pages from $dir/ over stdio');
+      final lines = stdin.transform(utf8.decoder).transform(const LineSplitter());
+      await server.serve(lines, stdout.writeln);
+      return 0;
+    } on ContentException catch (e) {
+      stderr.writeln('❌ ${e.message}');
+      return 1;
+    }
+  }
+
+  Future<int> _serveHttp(
+    DocsSource source,
+    McpServer server,
+    Logger logger, {
+    required String host,
+    required int port,
+    required Set<String> allowedOrigins,
+  }) async {
+    final mcp = McpHttpServer(server, allowedOrigins: allowedOrigins, logger: logger);
+    final handler = const shelf.Pipeline().addMiddleware(shelf.logRequests()).addHandler(mcp.handle);
+    final httpServer = await shelf_io.serve(handler, host, port);
+
+    logger.log('');
+    logger.log('📡 stardust mcp — serving ${source.pages.length} pages over MCP (Streamable HTTP)');
+    logger.log('  ➜ Endpoint: http://$host:$port/${McpHttpServer.endpoint}');
+    logger.log('');
+
+    ProcessSignal.sigint.watch().listen((_) async {
+      logger.log('\n👋 Shutting down...');
+      await httpServer.close();
+      exit(0);
+    });
+
+    await Completer<void>().future;
+    return 0;
+  }
+}

lib/src/generator/site_generator.dart

@@ -90,6 +90,7 @@ class SiteGenerator {
     if (config.build.llms.enabled) {
       await _generateLlms(pagesWithNav);
       await _generateLlmsFull(pagesWithNav);
+      await _generateManifest(pagesWithNav);
     }
 
     if (config.seo.ogImage == null && !config.devMode) {
@@ -511,6 +512,33 @@ class SiteGenerator {
     logger.log('🤖 Generated llms-full.txt');
   }
 
+  /// A machine-readable page index (`llms.json`) — the static manifest remote
+  /// agents can consume without a server, and the input `stardust mcp` reads to
+  /// serve a built site. Each `md` points at the verbatim source `_writePageMarkdown`
+  /// emits, so tools can fetch clean markdown per page.
+  Future<void> _generateManifest(List<Page> pages) async {
+    final urls = UrlResolver(config);
+    final manifest = <String, Object?>{
+      'name': config.name,
+      if (config.description case final desc?) 'description': desc,
+      if (config.url case final url?) 'url': url,
+      'generator': 'stardust',
+      'pages': [
+        for (final page in pages.where((page) => page.frontmatter['llm'] != false))
+          {
+            'path': page.path,
+            'title': page.title,
+            if (page.description case final desc?) 'description': desc,
+            if (urls.absolute(page.path) case final abs?) 'url': abs,
+            'md': page.path == '/' ? '/index.md' : '${page.path}.md',
+          },
+      ],
+    };
+    await fileSystem.writeFile(
+        p.join(outputDir, 'llms.json'), '${const JsonEncoder.withIndent('  ').convert(manifest)}\n');
+    logger.log('🤖 Generated llms.json');
+  }
+
   Future<void> _generateRedirects(List<Page> pages) async {
     final hasConfigRedirects = config.build.redirects.isNotEmpty;
     final hasFrontmatterRedirects = pages.any((p) => p.redirectFrom.isNotEmpty);