Skip to content

Latest commit

 

History

History
197 lines (150 loc) · 9.78 KB

PLAN.md

File metadata and controls

197 lines (150 loc) · 9.78 KB

Gemma documentation roadmap

This repository ports and re-imagines the Gemma documentation with Zensical. The immediate goal is a polished proof of concept that is safe to merge and easy to continue maintaining.

This file is a living roadmap, not a changelog. Completed implementation detail belongs in Git history; durable authoring instructions belong in README.md, docs/contributing/, or skills/gemma-docs/SKILL.md.

Current state

  • The Zensical design system, navigation, page-context header, global Gemma 4 selector, and MathJax formula rendering work.
  • Agent-facing raw Markdown routes and llms.txt are generated by the build (docs/llms.txt is committed and verified by lint_docs.py).
  • Twenty-three Markdown-first tutorials export 66 Jupyter notebooks (docs/notebooks/*.ipynb) with Apache 2.0 license headers, including model-specific variants for .model-swap pages and dynamic Colab link switching in the browser.
  • Three interactive demos (FunctionGemma physics playground HF Space, AIventure web game, and Embedding Draw Challenge), a task-first Learn hub, visual guides (tutorials/visual-guides/), and the 6-part Gemma Explained series (tutorials/gemma-explained/) demonstrate richer documentation experiences.
  • Google Analytics 4 (G-CWMC2KTTGQ) and cookie-consent management are configured in zensical.toml (with GA_MEASUREMENT_ID override support in scripts/build_docs.py) and verified by tests/test_analytics.py.
  • All 65 imported pages in scripts/importers/sources.toml are frozen (frozen = true) and hand-maintained across 97 total documentation pages.
  • The lint baseline (.lintbaseline.json) is empty (0 new, 0 known across all 15 checks).
  • Repository governance (CONTRIBUTING.md, CODEOWNERS, Apache 2.0 LICENSE), contributor guides (docs/contributing/), and the authoring skill (skills/gemma-docs/SKILL.md) are complete.

Deferred demos and interactive tools are not POC merge blockers.

Production cutover & remaining tasks

Keep this list short. These tasks materially affect production cutover or repository maintenance; broader product ideas belong in the deferred roadmap.

Generated notebooks (optional CI publishing)

Currently, docs/notebooks/*.ipynb are committed to main and strictly validated by scripts/lint_docs.py --check generated so Colab links (blob/main/docs/notebooks/...) work directly across all model sizes. If removing committed .ipynb files from main is desired later:

  • Publish notebooks from CI to a dedicated generated-notebooks branch.
  • Point every Colab URL (and model-switcher.js URL rewriter) at that branch.
  • Remove committed .ipynb files from main only after the published URLs work.
  • Give contents: write only to the post-build publishing job on successful main builds, using SHA-pinned actions and remaining zizmor-clean.

Legacy upstream pages

Five legacy upstream pages remain unported (/meme-generator is delinked in link_map.toml):

  • /core/gemma_library
  • /core/pytorch_gemma
  • /core/deploy_to_cloud_run_from_ai_studio
  • /gemma_chat
  • /meme-generator

Decide post-merge whether to permanently retire or port any of these legacy guides.

Durable authoring rules

Import once, then hand-maintain

Imported content is validated upstream. A port changes formatting, not prose or code.

  • Do not reword, reorder or silently correct imported prose during conversion.
  • Do not alter imported code without an explicit technical reason and review.
  • Every imported page records source_url in frontmatter.
  • Every existing manifest entry is frozen. Never reconvert frozen pages in bulk.
  • A new page is added to scripts/importers/sources.toml unfrozen, imported and reviewed once, then marked frozen = true before documentation-specific editing begins.
  • Converter improvements affect future imports only; repair existing pages deliberately by hand.

The official-doc import pipeline is:

scripts/importers/port_docs.py
  -> scripts/importers/vendor_images.py  # only if the source needs local assets
  -> scripts/importers/rewrite_links.py
  -> scripts/lint_docs.py --baseline

port_docs.py runs the Markdown converter and fidelity check before writing. Visual guides use scripts/importers/import_newsletter.py instead. Temporary demo/vendor images are not worth reworking because they will be replaced with original PNGs.

Markdown-first notebooks

  • Add .notebook-cell only to fences that should execute in Jupyter.
  • Ordinary prose becomes Markdown cells.
  • .output fences are omitted and recreated when the notebook runs.
  • Site-only regions use the notebook skip markers documented in docs/contributing/components.md.
  • Tutorials with .model-swap code blocks export separate model-specific notebooks (<stem>-gemma-4-<size>.ipynb) for all candidate models matching the page's engines and requires: modality constraint. Non-swappable tutorials export a single <stem>.ipynb.
  • Frontmatter colab: specifies the default model notebook (-gemma-4-e2b.ipynb), and docs/javascripts/model-switcher.js dynamically rewrites the Colab link when a reader chooses another model.
  • Exported notebooks automatically include Google LLC Apache 2.0 license header cells.
  • Do not maintain Markdown and notebook JSON in parallel.

Data and rendering

  • data/models.yml, data/frameworks.yml, data/links.yml, and data/glossary.yml are the single sources of truth for model metadata, engine facts, shared links, and glossary tooltips. Do not hand-maintain copies in pages or JavaScript.
  • Referenced Zensical icons are vendored into overrides/.icons/ via scripts/vendor_icons.py so standalone CI lint jobs run without installing Zensical.
  • Frontmatter keys can shadow template data. Template macros use modeldata.* for this reason.
  • Measure rendered layout in a browser. Source inspection alone is not sufficient for visual work.
  • Keep Zensical pinned. A package update must not silently restyle the documentation.
  • New workflow changes must use least privilege, SHA-pinned actions and pass zizmor.

Maintained tooling

The directory layout separates production infrastructure, optional import utilities and tests.

Active build and validation: scripts/

File Responsibility
build_docs.py Production entry point: generators, GA4 config injection, Zensical build and raw Markdown publication.
agent_docs.py Builds llms.txt and raw .md routes.
lint_docs.py CI gate for documentation and repository invariants across 15 checks.
frontmatter_schema.py Shared frontmatter schema and controlled vocabularies.
generate_model_names.py Compiles model/framework data for templates and the browser selector.
generate_snippets.py Renders data/links.yml and data/glossary.yml into snippets/links.md and snippets/glossary.md.
notebooks.py Exports registered Markdown tutorials to single or multi-model .ipynb files.
vendor_icons.py Vendors referenced Zensical SVG icons into overrides/.icons/.
pinned_links.py Scans and reports external URLs pinned to a specific model generation (data-model).
check_contrast.py Validates WCAG contrast ratios for the CSS brand color ramp (--gemma-blue-seed).

Reusable ingestion: scripts/importers/

These remain supported because more official Markdown and notebook content is expected.

File Responsibility
convert_notebook.py Convert official .md.txt/notebook exports into Zensical Markdown.
port_docs.py Manifest-driven, freeze-aware import orchestration.
check_fidelity.py Detect prose lost during a new conversion.
rewrite_links.py, link_map.toml Convert upstream links after import.
vendor_images.py Optional remote-asset step for a new import.
import_newsletter.py Import additional first-party visual guides; remove if that roadmap is dropped.
sources.toml Import provenance and frozen-page safety boundary.

Tests: tests/

Regression tests and converter fixtures belong in tests/, not beside executable maintenance commands. CI discovers test_*.py from this directory (test_agent_docs.py, test_analytics.py, test_convert_notebook.py, test_notebooks.py).

Removed from the maintained surface:

  • backfill_frontmatter.py: completed one-time migration.
  • PLAN-ARCHIVE.md: duplicated Git history and made the supported workflow harder to find.

Verification

Run before handoff or merge:

uv run --frozen python -m unittest discover -s tests -p "test_*.py"
uv run --frozen python scripts/lint_docs.py --baseline
uv run --frozen python scripts/build_docs.py

For a new official-doc import, additionally run the relevant importer in --dry-run mode and review its fidelity report before writing files.

Deferred roadmap

Content and Learn

  • Continue the Markdown-first notebook ingestion/export workflow, but keep it one-way after import.
  • Add visual-guide assets only when original, production-quality PNGs are ready.
  • Expand the simple task-first Learn page only when new content justifies another route.

Demos and embeds

  • Keep the FunctionGemma HF Space and AIventure demo as the initial interactive examples.
  • Defer additional spaces, media embeds and interactive demos until the core content is settled.

Interactive tools

  • VRAM and context calculator driven by data/models.yml, with model, quantization, context, batch size, KV-cache dtype and GPU presets; encode state in the URL.
  • In-browser E2B q4 playground via WebGPU/transformers.js with an explicit large-download opt-in.
  • "Ask Gemma about Gemma" documentation assistant.
  • Interactive Gemma 4 architecture and MTP explainers.
  • DiffusionGemma denoising animation.
  • Community throughput leaderboard, only if its maintenance cost is justified.
  • Self-hosting cost calculator.