Skip to content

Add markdown-split for context loading#9

Open
coolai86 wants to merge 6 commits into
therootcompany:mainfrom
coolai86:add-markdown-split4
Open

Add markdown-split for context loading#9
coolai86 wants to merge 6 commits into
therootcompany:mainfrom
coolai86:add-markdown-split4

Conversation

@coolai86

@coolai86 coolai86 commented Apr 5, 2026

Copy link
Copy Markdown

Adds markdown-split.cjs to the download-markdown skill for splitting large markdown files into smaller reference files.

Problem: Large markdown docs exceed context windows. Reading 100KB of documentation burns tokens.

Solution: Progressive disclosure — split by headers, read only what you need.

Features:

  • --toc — View TOC with line numbers before splitting
  • --index — Create INDEX.md listing all splits
  • --depth N — Header depth to split on (default: 2)
  • --max-lines / --max-bytes — Size limits per file
  • --flatten — Flat numbered filenames
  • Skips code blocks and YAML frontmatter when parsing

Workflow:

# View structure
./scripts/markdown-split.cjs --toc large-doc.md

# Split for skill context loading
./scripts/markdown-split.cjs --index large-doc.md ./references/

Creates files suitable for progressive disclosure: use grep -n '^#' file.md to find headers, then read specific sections.

root added 6 commits April 5, 2026 00:07
Single-binary Go web application patterns:
- Project structure (cmd/, internal/, web/dist/)
- Single embed file pattern
- SPA routing with fallback handler
- Development mode patterns
- Anti-patterns to avoid

Framework-agnostic: no vite/esbuild references.
- Document sql.Scanner/driver.Valuer pattern for custom types
- Add JSONB override configuration example
- Add nullable types guidance (sql.Null[T] and legacy types)
- Update PostgreSQL type mappings table to reference custom types

This helps projects use sqlc effectively for JSONB columns,
domain-specific types, and custom nullable types.
- download-markdown.cjs: Download URL, detect Markdown via extensions, Accept headers, and HTML link parsing
- html-to-markdown-recursive.cjs: Convert HTML files to Markdown recursively
- website-download-recursive.sh: wget-based recursive website download
- Markdown detection: 8 strategies including URL variants, Accept headers, and HTML link parsing (edit/source buttons, .md links, raw repo URLs)
- Dependencies: @mozilla/readability, turndown, jsdom
- Supports content-type headers, content sniffing, and frontmatter generation
- New script to split large markdown files for context loading
- View TOC with line numbers (--toc)
- Split by headers with size limits
- Create index file for navigation
- Skip code blocks and YAML frontmatter when parsing
- Handles oversized sections with warnings

Enables progressive disclosure: grep for headers, read only needed sections.
Modern Node.js supports 'import' on CJS modules natively.
- Renamed .cjs → .mjs
- Converted require() to import statements
- Updated SKILL.md examples
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant