Add opt-in C kernel for spectrum-native holographic trace memory

[atimics]

Summary

• add a C99 holographic vector-symbolic kernel with scalar and optional Accelerate/vDSP backends
• add cached trace memory, trace benchmarks, C tests, and a root Makefile for build/test/experiment targets
• add an opt-in Python bridge so HOLOSTUFF_USE_C=1 can replace bind, bind_fixed, weighted_sum, unbind, and HolographicMemory while default NumPy behavior remains unchanged
• add an adaptive C bind_fixed path for small fixed-vector row stacks, while leaving wider stacks on NumPy's batched real FFT by default
• make the C trace kernel spectrum-native: learn() now accumulates FFT(state) * FFT(action) directly, while .trace/save/load materialize the real trace lazily for compatibility
• add a c-kernel-evidence CI job that compiles/tests the scalar C path, gates C-vs-NumPy trace speedups, and uploads raw JSONL plus a Markdown evidence report
• add C-backed raw weighted accumulation plus vectorized FPE batch encoding, so field-valued FPE bundles preserve amplitudes while building about 2.1x faster locally

Update

Rebased onto upstream main at de13c31 on 2026-06-28. The newer upstream work adds sparse/FPE field faculties, mesh bridge/QEM updates, unified integration, tour updates, and fresh bind_fixed usage in integration coverage. This PR still keeps bind_batch on NumPy and accelerates only the fixed-vector row-stack case where the C backend measured well. The default cutoff is 8 rows and can be tuned with HOLOSTUFF_C_BIND_FIXED_MAX_ROWS.

Latest Update

Added 4f5b013 after checking the latest upstream field work: the C backend now exports an unnormalized weighted_sum primitive, holographic_ai.bundle() layers normalization on top of that shared accumulator, and VectorFunctionEncoder.bundle() uses vectorized encode_many() plus chunked weighted accumulation for FPE function/field builds. This keeps symbolic bundles normalized while preserving raw amplitudes for function-valued FPE fields.

Representative local timing on the 12^3 / dim-2048 FPE field build:

• NumPy mode: old row loop 0.3454s, new bundle 0.1631s -> 2.12x speedup, max abs drift 3.553e-15
• Strict C mode: old row loop 0.3313s, new bundle 0.1602s -> 2.07x speedup, max abs drift 4.441e-15

Latest Performance Comparison

Fresh local run on 2026-06-29 from codex/c-kernel-plan using /Users/ratimics/develop/.venvs/holostuff/bin/python.

Accelerate C trace vs Python/NumPy

Command: make c-bench PYTHON=/Users/ratimics/develop/.venvs/holostuff/bin/python

This compares the Python/NumPy HolographicMemory trace path against the C trace/action-index benchmark built with the macOS Accelerate backend. Workload: pairs=8, actions=8, queries=1024, repeats=5. Accuracy was 1.0 for both Python/NumPy and C on every row.

dim	Python store/s	C store/s	store speedup	Python query/s	C query/s	query speedup
128	101,964	2,666,646	26.15x	87,815	1,882,353	21.44x
256	93,249	1,142,858	12.26x	79,798	950,789	11.91x
512	72,562	499,999	6.89x	64,760	431,521	6.66x
1024	57,641	249,999	4.34x	50,447	200,039	3.97x

Scalar C trace gate vs Python/NumPy

Command: make c-ci-evidence PYTHON=/Users/ratimics/develop/.venvs/holostuff/bin/python

This is the same portable scalar path the CI job gates. Workload: pairs=8, actions=8, queries=2048, repeats=7. Accuracy was 1.0 for both Python/NumPy and C on every row.

dim	scalar C store speedup	scalar C query speedup
128	6.83x	6.77x
256	3.54x	3.16x
512	1.98x	1.75x

Geomean across gated scalar dimensions: 3.63x store, 3.34x query. The gate passed: every measured dimension exceeded the 1.05x store/query thresholds with 1.0 C accuracy.

bind_fixed support rows

bind_fixed is intentionally narrower than the trace path. Accelerate C wins on the small row stacks where the bridge routes to C, while row-32 uses the NumPy fallback by default because batched NumPy real FFT is generally better for wider stacks.

dim	rows	C path	speedup vs Python/NumPy	max abs error
128	1	yes	1.44x	1.11e-16
128	8	yes	1.30x	1.67e-16
128	32	fallback	0.96x	0.00e+00
256	1	yes	1.49x	7.98e-17
256	8	yes	1.30x	8.67e-17
256	32	fallback	0.94x	0.00e+00
512	1	yes	1.45x	5.55e-17
512	8	yes	1.17x	7.98e-17
512	32	fallback	1.04x	0.00e+00
1024	1	yes	1.42x	4.86e-17
1024	8	yes	1.09x	5.55e-17
1024	32	fallback	1.00x	0.00e+00

The important distinction is that the trace win is not merely "C instead of Python." The baseline Python path is already NumPy, so much of it runs inside compiled FFT/vector kernels. The C path wins where this PR changes ownership of the hot loop and representation: spectrum-native trace storage, C-owned action-index lookup, and fewer Python/NumPy intermediate transitions.

Why

The architectural hot path for this project is holographic bind/unbind trace memory plus repeated binding against a fixed trace, role, or key. This keeps the Python research surface intact while adding an optional hardware-aware substrate for repeated store/recall and small fixed-role batch workloads.

The latest kernel change stores trace memory in the representation the algebra actually consumes: frequency-space bound pairs. That removes an inverse FFT from every trace store and avoids transforming the real trace back into frequency space before recall.

CI Evidence

The new c-kernel-evidence job runs on Ubuntu with the portable scalar C backend. It:

• builds and runs c unit tests with HOLO_USE_ACCELERATE=0
• runs repeated C-vs-NumPy trace memory benchmarks for dims 128, 256, and 512
• fails CI unless every gated dimension reaches at least 1.05x store speedup, 1.05x query speedup, and 0.99 C accuracy
• records bind_fixed rows as supporting evidence without gating on them, because scalar Linux performance there is platform-sensitive
• appends c/build/ci-evidence/report.md to the GitHub Actions job summary
• uploads raw.jsonl, summary.jsonl, and report.md as the c-kernel-evidence artifact

Current upstream PR Actions run is waiting for maintainer approval because this is a forked PR: https://github.com/AnOversizedMooseWithSocks/holostuff/actions/runs/28319664732. Once approved, the CI job will compile and publish the evidence in the run summary/artifact.

Latest local dry run of the same evidence target passed:

• make c-ci-evidence PYTHON=/Users/ratimics/develop/.venvs/holostuff/bin/python
• scalar trace geomean speedup: 3.630x store, 3.341x query
• dim 128: 6.833x store, 6.771x query
• dim 256: 3.540x store, 3.155x query
• dim 512: 1.977x store, 1.746x query

Before the spectrum-native trace rewrite, the same local evidence target reported about 2.598x store / 3.397x query geomean. The spectrum-native change specifically improves the trace-store side while keeping query throughput and accuracy intact.

Validation

Latest validation on 2026-06-29:

• git diff --check de13c3187a5f8b10cee097d032948bb2867768d1...HEAD -> clean
• make c-test PYTHON=/Users/ratimics/develop/.venvs/holostuff/bin/python -> test_core ok, test_trace ok
• make c-bench PYTHON=/Users/ratimics/develop/.venvs/holostuff/bin/python -> PASS; Accelerate trace speedups listed above
• make c-ci-evidence PYTHON=/Users/ratimics/develop/.venvs/holostuff/bin/python -> PASS; scalar gate speedups listed above
• HOLOSTUFF_USE_C=1 HOLOSTUFF_C_STRICT=1 /Users/ratimics/develop/.venvs/holostuff/bin/python -m pytest -q test_holographic_c_backend.py test_holographic_fpe.py -> 16 passed

Earlier validation after 4f5b013:

• git diff --check
• make c PYTHON=/Users/ratimics/develop/.venvs/holostuff/bin/python
• make c-test PYTHON=/Users/ratimics/develop/.venvs/holostuff/bin/python -> test_core ok, test_trace ok
• make -C c test HOLO_USE_ACCELERATE=0 PYTHON=/Users/ratimics/develop/.venvs/holostuff/bin/python -> test_core ok, test_trace ok
• HOLOSTUFF_USE_C=1 HOLOSTUFF_C_STRICT=1 /Users/ratimics/develop/.venvs/holostuff/bin/python -m pytest test_holographic_c_backend.py test_holographic_fpe.py test_holographic_fpefield.py -q -> 22 passed
• HOLOSTUFF_USE_C=1 HOLOSTUFF_C_STRICT=1 /Users/ratimics/develop/.venvs/holostuff/bin/python -m pytest test_holographic_fpe.py test_holographic_fpefield.py test_holographic_sparsefield.py test_holographic_meshbridge.py test_holographic_meshqem.py test_holographic_mesh.py test_holographic_lod.py -q -> 84 passed
• HOLOSTUFF_USE_C=1 HOLOSTUFF_C_STRICT=1 /Users/ratimics/develop/.venvs/holostuff/bin/python -m pytest -q <latest FS/mesh integration slice> -> 10 passed
• Full NumPy-mode suite: 1609 passed, 17 skipped, 5 failed; the same 5 targeted tests fail from a clean worktree at pre-edit 344d4cb, so they appear to be baseline/environment failures rather than regressions from this update.

Post-rebase validation on de13c31 base:

• git diff --check
• make c-test -> test_core ok, test_trace ok
• make -C c test HOLO_USE_ACCELERATE=0 PYTHON=/Users/ratimics/develop/.venvs/holostuff/bin/python -> test_core ok, test_trace ok
• /Users/ratimics/develop/.venvs/holostuff/bin/python -m pytest test_holographic_c_backend.py -q -> 5 passed
• HOLOSTUFF_USE_C=1 HOLOSTUFF_C_STRICT=1 /Users/ratimics/develop/.venvs/holostuff/bin/python -m pytest test_holographic_fpe.py test_holographic_fpefield.py test_holographic_sparsefield.py test_holographic_meshbridge.py test_holographic_meshqem.py test_holographic_mesh.py test_holographic_lod.py -q -> 83 passed
• HOLOSTUFF_USE_C=1 HOLOSTUFF_C_STRICT=1 /Users/ratimics/develop/.venvs/holostuff/bin/python -m pytest -q <latest FS/mesh integration slice> -> 10 passed

Current validation before the latest base refresh:

• git diff --check
• make -C c clean && make -C c test HOLO_USE_ACCELERATE=0 PYTHON=/tmp/holostuff-pr2-venv/bin/python -> test_core ok, test_trace ok
• make c-test PYTHON=/tmp/holostuff-pr2-venv/bin/python -> test_core ok, test_trace ok
• make c PYTHON=/tmp/holostuff-pr2-venv/bin/python
• HOLOSTUFF_USE_C=1 HOLOSTUFF_C_STRICT=1 /tmp/holostuff-pr2-venv/bin/python -m pytest -q test_holographic_c_backend.py -> 5 passed
• HOLOSTUFF_USE_C=1 HOLOSTUFF_C_STRICT=1 /tmp/holostuff-pr2-venv/bin/python -m pytest -q test_integration.py::test_exact_matmul_is_exact_where_superposition_is_lossy_and_range_federates -> 1 passed
• make c-ci-evidence PYTHON=/tmp/holostuff-pr2-venv/bin/python -> PASS

Earlier benchmark/check coverage before the upstream rebase:

• HOLOSTUFF_USE_C=1 HOLOSTUFF_C_STRICT=1 python -m pytest test_holographic.py -q -> 43 passed
• HOLOSTUFF_USE_C=1 HOLOSTUFF_C_STRICT=1 python -m pytest test_holographic_c_backend.py test_algebra_properties.py::test_bind_batch_and_fixed_match_scalar_bind test_isa_conformance.py::test_bind_batch_is_value_conformant test_holographic_compute.py test_holographic_superposed.py test_holographic_rns.py test_holographic_array.py test_holographic_creature_batch.py -q -> 18 passed
• make c-bench

Benchmark Notes

• The baseline being compared here is Python-facing NumPy, not a pure Python loop. That matters: NumPy already dispatches FFT and vector math into compiled code. The C path wins where this PR owns the representation and loop across operations, not merely because the implementation language changed.
• Spectrum-native scalar trace benchmark, latest local CI evidence path: store/query about 6.833x/6.771x at dim 128, 3.540x/3.155x at dim 256, and 1.977x/1.746x at dim 512.
• Trace benchmark, latest Accelerate backend, median speedup over Python/NumPy: store/query about 26.15x/21.44x at dim 128, 12.26x/11.91x at dim 256, 6.89x/6.66x at dim 512, and 4.34x/3.97x at dim 1024.
• bind_fixed benchmark, Accelerate C path enabled for rows <= 8: row-1 speedup ranged about 1.42x-1.49x; row-8 speedup ranged about 1.09x-1.30x; max absolute error stayed below 2e-16.
• Row-32 cases use the NumPy fallback by default because NumPy's batched real FFT is better for wider stacks.

Notes

• The C backend is default-off and activated with HOLOSTUFF_USE_C=1 or holographic_c.install(...).
• Full benchmark targets preflight optional analysis dependencies from requirements-experiments.txt.
• A full HOLOSTUFF_USE_C=1 pytest run previously had 5 failures, and the same 5 targeted tests failed without the C backend; they appear to be upstream/baseline suite issues rather than regressions from this PR.

[atimics]

C VSA program-layer update

Added 9495a04 with a C runner for the core HoloMachine stored-program subset:

• C API: holo_program_run_basic(...)
• Python bridge: holographic_c.program_run_basic(...)
• VM method: HoloMachine.run_c_basic(...)
• Covered instructions: LOAD, BIND, BUNDLE, PERMUTE, IFMATCH, HALT
• Host-bound features still run in the Python VM: CALL, APPLY, ITERATE, REPEAT, registers, and stack operations

This moves the VSA program decode/execute loop into C for straight-line/core programs: materialize the program spectrum once, unbind each instruction position, clean opcode/operand against C-side codebooks, and update ACC inside one C loop.

Validation

• make c-test PYTHON=/Users/ratimics/develop/.venvs/holostuff/bin/python -> test_core ok, test_trace ok, test_program ok
• HOLOSTUFF_USE_C=1 HOLOSTUFF_C_STRICT=1 /Users/ratimics/develop/.venvs/holostuff/bin/python -m pytest -q test_holographic_c_backend.py test_holographic_c_program.py test_holographic_machine.py -> 23 passed
• make c-ci-evidence PYTHON=/Users/ratimics/develop/.venvs/holostuff/bin/python -> PASS, including scalar test_program ok
• git diff --check -> clean

Python VM vs C VSA Program Runner

Command: make c-bench PYTHON=/Users/ratimics/develop/.venvs/holostuff/bin/python

Benchmark compares Python HoloMachine.run() against C HoloMachine.run_c_basic() on the same encoded program vectors. The accumulator cosine between Python and C was ~1.0 for every case.

dim	executed instructions	Python runs/s	C runs/s	C speedup
1024	3	1,751	10,276	5.87x
1024	7	824	5,587	6.78x
1024	11	541	3,883	7.18x
1024	15	416	2,976	7.15x
2048	3	1,347	5,405	4.01x
2048	7	640	2,746	4.29x
2048	11	418	1,821	4.35x
2048	15	313	1,370	4.38x
4096	3	870	2,561	2.94x
4096	7	407	1,225	3.01x
4096	11	268	811	3.03x
4096	15	199	606	3.05x

Program-layer geomean across all 12 cases: 4.42x faster than the Python VM, with exact trace parity and accumulator parity.

The important boundary is still the same: this does not make all VSA programs C-native. It gives the core algebraic VM path a C executor; richer host-bound instructions remain Python-side because they call Python handlers, function libraries, or exact host state.

[atimics]

Updated PR #2 with b21e93e (accelerate batched FPE field reads).

What changed:

• Added frequency-space batched FPE reads via VectorFunctionEncoder.query_many(...).
• Changed FPE function bundling to accumulate spectra and inverse FFT once, instead of materializing every real encoded row first.
• Routed the new batched path through noise/fBm, holographic fields, material sampling, attribute baking, and terrain mesh/SDF generation.
• Added parity coverage for query_many and field-value batched read semantics.

Validation:

• NumPy focused slice: 96 passed
• Strict C-backed focused slice: HOLOSTUFF_USE_C=1 HOLOSTUFF_C_STRICT=1 -> 96 passed
• git diff --check -> clean

Local microbenchmarks on the DCC/FPE path:

• FPE 4096-point read loop -> query_many: ~2.97x faster
• FPE 512-point bundle old materialized-row path -> spectrum accumulation: ~1.22x faster
• 2048-point fBm read loop -> batched fBm: ~2.29x faster
• 48x48 terrain mesh generation old scalar height loop -> batched terrain: ~2.35x faster

This keeps the DCC/procedural layer in Python/NumPy, but accelerates the shared FPE field kernel it now leans on.

[atimics]

Performance update after rebasing PR2 on origin/main:

• 8396710 accelerates FPE readout by storing/using the real-FFT half spectrum and Parseval weights instead of full complex spectra.
• 250feec parallelizes FPE field batches: VectorFunctionEncoder.bundle/query_many now split chunk work across worker threads for large batches, and FractalNoise.query_many fans out independent octave reads. workers=1 keeps the serial path; HOLOSTUFF_FPE_THREADS can cap/force the auto worker count.

Validation:

• pytest -q test_holographic_fpe.py test_holographic_noise.py test_holographic_fpefield.py -p no:cacheprovider -> 26 passed
• Procedural NumPy slice -> 98 passed
• Strict C backend procedural slice with HOLOSTUFF_USE_C=1 HOLOSTUFF_C_STRICT=1 -> 98 passed

Local microbench on this 14-CPU machine, serial workers=1 vs default auto workers:

• FPE query_many 8192x2048: 0.2276s -> 0.0318s, 7.15x
• FPE bundle 8192x2048: 0.2264s -> 0.0298s, 7.59x
• fBm query_many 4096 points, 4 octaves: 0.4440s -> 0.1274s, 3.49x
• terrain 64x64 height grid, 4 octaves: 0.4124s -> 0.1129s, 3.65x

This should help the new DCC/procedural layer from main directly: terrain, noise, materials, attributes, and FPE-backed fields all route through these batch reads.

[atimics]

Follow-up to the parallel-core pass: cca2524 now treats the fBm worker count as a budget across both octaves and each octave FPE chunk. That lets terrain/noise use most of the machine instead of stopping at one worker per octave.

Updated local microbench on the same 14-CPU machine, serial workers=1 vs default auto workers:

• FPE query_many 8192x2048: 0.2274s -> 0.0306s, 7.44x
• FPE bundle 8192x2048: 0.2263s -> 0.0295s, 7.68x
• fBm query_many 4096 points, 4 octaves: 0.4434s -> 0.0517s, 8.58x
• terrain 64x64 height grid, 4 octaves: 0.4135s -> 0.0484s, 8.55x

Validation after the scheduler refinement remains green:

• Focused FPE/noise/FPE-field slice -> 26 passed
• Procedural NumPy slice -> 98 passed
• Strict C backend procedural slice -> 98 passed

[atimics]

Another speed pass: dc27cbb factors regular 2D FPE grid reads.

What changed:

• Added VectorFunctionEncoder.query_grid(...) for Cartesian grids. In the 2D case it computes x-axis spectra and y-axis spectra once, then uses one matrix multiply per field read instead of materializing every (x, y) point spectrum.
• Routed FractalNoise.sample_grid() through that path.
• Reused terrain heightmaps in terrain_to_mesh() and terrain_to_sdf(); the SDF path no longer recomputes the same (x, y) height for every z slice.

Validation:

• Focused FPE/noise/terrain/FPE-field slice -> 33 passed
• Procedural NumPy slice -> 100 passed
• Strict C backend procedural slice -> 100 passed

Local microbench on the same 14-CPU machine:

• 64x64 fBm heightmap, direct point stack vs factored grid: 0.0533s -> 0.0026s, 20.14x
• Repeated SDF (x, y) height reads vs reused grid heightmap: 0.0809s -> 0.0011s, 74.13x for that substep
• Full terrain_to_mesh(64): 0.0656s -> 0.0076s, 8.61x
• Full terrain_to_sdf(res=18): 0.0851s -> 0.0172s, 4.94x

This should make the new main-branch DCC terrain/heightfield workflows feel much less like they are dragging a full point cloud through every grid sample.

[atimics]

Real-time renderer pass: a44cd8c adds a standalone pure-C Sokol asteroid-belt renderer.

What landed:

• Vendored the required Sokol headers in vendor/sokol (sokol_app.h, sokol_gfx.h, sokol_glue.h, sokol_log.h).
• Added c/sokol/holo_sokol_asteroids.c: deterministic procedural low-poly asteroid-belt generation in C, one GPU vertex buffer, depth-tested GLCore Sokol pipeline, GLSL shader, and orbiting real-time camera.
• Added make sokol-asteroids and make sokol-run top-level targets, delegated through c/Makefile.
• Added c/sokol/README.md with build/run notes. On macOS, Sokol app internals require Objective-C compilation for Cocoa, but the renderer code and geometry generation are plain C.

Validation:

• make -C /Users/ratimics/develop/holostuff sokol-asteroids builds cleanly with -Wall -Wextra -Wpedantic.
• Launch smoke test opened the realtime app and uploaded 453600 vertices (15.6 MB) before I interrupted it cleanly.

This turns the earlier offline asteroid-belt preview into an actual realtime renderer foothold inside the repo.