Gate-Zero Growth

Code and results for "Gate-Zero Growth: A Geometric Framework for Function-Preserving Continual Learning" (Dante Lok, Votee AI / Beever AI).

Project Overview

Gate-zero growth adds new residual blocks through a zero-initialised gate. Under a transversality condition it induces rank separation in the functional Jacobian: old directions are unchanged, new-weight directions are exactly flat at the growth point, and new gate directions are the only first-order source of new functional variation. As gates open during continual learning, function drift is O(‖α‖²) and Jacobian leakage O(‖α‖∞).

On a 300M → 857M Transformer adapted from WikiText-103 to BookCorpus, gate-zero growth reaches near-zero old-domain forgetting (Δ_A < 0.1) under both Isolation and Freeze-Nothing, while a non-FP control (G_stack) forgets an order of magnitude more under the same recipe.

Repository Structure

gate-zero-growth/
├── src/gatezero/                     # The library (installed via `pip install -e .`)
│   ├── growth.py                     # Growth operators: gate-zero FP, G_stack, zero-init stack
│   ├── trainer.py                    # Base-train → grow → CL pipeline (seeded, deterministic)
│   ├── cl_methods.py                 # CL recipes: replay, distillation, isolation, hybrid
│   ├── config.py                     # Experiment configs — the run matrices live here
│   ├── data.py                       # WikiText-103 / BookCorpus loading + replay buffer
│   ├── evaluation.py                 # Perplexity + CL evaluation
│   ├── lora.py                       # LoRA-CL baseline
│   └── models/                       # Gated Transformer + MoE variants
├── scripts/                          # Experiment entry points (run from repo root)
│   ├── run_exp1_cl_matrix.py         # Table 2: growth × CL-recipe matrix (300M → 857M)
│   ├── run_exp2_moe.py               # Table 3: MoE cross-architecture
│   ├── run_ablations.py              # Table 5: growth factor / freezing / gate init / timing
│   ├── run_no_growth_cl.py           # Table 6 baselines: no-growth CL
│   ├── run_zero_init_stack.py        #   ": zero-init residual stacking
│   ├── run_lora_cl.py                #   ": LoRA-CL (rank 64)
│   ├── run_exp3_manifold.py          # Appendix F: manifold geometry
│   ├── diagnose_transversality.py    # Appendix B: direct transversality diagnostic
│   ├── diagnose_spectral_leakage.py  # Appendix G: spectral leakage estimator
│   └── plot_*.py                     # Figures (Pareto frontier, MoE diagnostic, …)
├── results/                          # One results.json per run — the paper's numbers (43 runs)
├── figures/                          # Generated figures
├── preliminary/                      # Appendix E: Phase 1–4 cross-architecture validation
│   ├── code/                         # Standalone MLP / ResNet / SE-ResNet / Transformer studies
│   ├── reproduce_table_e2.py         # One-command Table E.2 reproduction (seed 42 + determinism)
│   └── REPRODUCIBILITY.md            # Expected values, determinism notes, caveats
├── docs/
│   └── RUN_ORDER.md                  # Full ~45-run reproduction sequence (7 phases)
├── pyproject.toml
└── requirements.txt                  # Pinned environment

preliminary/ predates the gatezero package and is deliberately standalone — those scripts do not import gatezero. See preliminary/README.md.

Quick Start

Install

python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
pip install -e .

The editable install is what lets scripts/ do from gatezero.trainer import Trainer. Datasets (WikiText-103, BookCorpus) download on first run via Hugging Face datasets.

Reading the code

Start at src/gatezero/growth.py — grow_transformer is the paper's growth operator, and verify_function_preservation is the FP check reported in Table 1. From there:

File	Role
src/gatezero/growth.py	Growth operators: gate-zero FP, G_stack, zero-init stacking
src/gatezero/models/transformer.py	Gated Transformer blocks — where the gate lives
src/gatezero/models/moe.py	MoE variant (Section 5.3)
src/gatezero/cl_methods.py	The CL recipes: replay, distillation, isolation, hybrid
src/gatezero/trainer.py	Base-train → grow → CL pipeline
src/gatezero/config.py	Every experiment config; the run matrices live here

The derivations behind the implementation live in the paper's theory sections; the code comments in growth.py and models/transformer.py mark where each result is used.

Reproduce

All commands run from the repository root — results/ and checkpoints/ resolve relative to the working directory.

# Shared 300M base — every later phase reuses this checkpoint
python3 scripts/run_exp1_cl_matrix.py --base-only

# One cell of the CL matrix (Table 2)
python3 scripts/run_exp1_cl_matrix.py --run-index 0 \
    --base-checkpoint checkpoints/exp1_shared_base/base_final.pt

docs/RUN_ORDER.md is the full sequence: ~45 runs across 7 phases, with the dependency order between them. Runs range from 30 minutes to ~80 GPU-hours.

The results are already here. results/<run_name>/results.json contains the numbers behind every table in the paper — 43 runs — so you can inspect or re-plot them without spending the compute:

python3 scripts/plot_pareto.py          # Figure 2 (Pareto frontier)
python3 scripts/plot_moe_diagnostic.py  # MoE diagnostic figure

Checkpoints are not distributed (size); checkpoints/ is created by training.

Citation

@article{lok2026gatezero,
  title   = {Gate-Zero Growth: A Geometric Framework for
             Function-Preserving Continual Learning},
  author  = {Lok, Dante},
  journal = {arXiv preprint arXiv:2607.14571},
  year    = {2026}
}

License

Apache License 2.0 — see LICENSE. Copyright 2026 Votee AI and Beever AI.