Skip to content

refactor[next]: give the build caches one home too - #2792

Draft
havogt wants to merge 1 commit into
havogt/next-cache-managerfrom
havogt/next-build-cache-dir
Draft

refactor[next]: give the build caches one home too#2792
havogt wants to merge 1 commit into
havogt/next-cache-managerfrom
havogt/next-build-cache-dir

Conversation

@havogt

@havogt havogt commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Top of the stack: #2769 (layout names) → #2768 (cache manager CLI) → this.

Why

#2769 gave the two translation caches one home. The build caches were left
loose in the cache base — one folder per compiled program variant, sitting
directly next to translation_cache/:

.gt4py_cache/
├── translation_cache/{dace,gtfn}/        <- a fixed directory
├── lap_program_pyext_<fp>_<version>/     <- ... next to an unbounded set of
├── laplap_program_pyext_<fp>_<version>/     program folders
└── compile_commands_cache_..._pyext_.../

So the base mixed the two, and nothing in the listing said which entry was
which kind of cache. That is also what invites "is the DaCe build directory
inside translation_cache?"
, the question that started #2769.

What

.gt4py_cache/
├── translation_cache/{dace,gtfn}/
└── build_cache/<program>_pyext_<fingerprint>_<version>/

The base now holds exactly the two kinds of cache gt4py.next keeps, each named
for what it is.

No <backend> level here, deliberately. Which backend produced a build
folder is not knowable where the folder is named: get_cache_folder is reached
from otf/compilation/ and its build systems, which are generic over the source
language rather than aware of backends, so a backend level would mean either
threading a name through that layer or mapping language→backend. The fingerprint
already keeps variants apart, so the flat set is correct — only its location
changes.

dace_determinism.py scans that directory and so far learned only the
folder-name pattern from the gt4py under test. Its probe now returns the
directory name too, rather than gaining a second hardcoded literal, so a
gt4py-side move of the build cache surfaces as a fetch error instead of as "no
programs observed". The stub interpreters in its tests answer with a different
directory name than gt4py's own, which is what proves nothing is hardcoded.

Migration

Not migrated — and these are the large caches. They are already discarded on any
change of BUILD_CACHE_VERSION_ID (every gt4py version bump, and every commit
for an editable install), so orphaned build folders are the normal state of this
cache; the stale directories can simply be removed.

Verification

  • Compiled a program with both backends into a fresh cache and confirmed the
    resulting tree, then read it back through gt4py-next-cache path and
    list --by-program.
  • Ran dace_determinism check against real caches in the new layout: it reports
    3 in run1, 3 in run2, i.e. it locates the program folders one level down.
    Before this change that scan returns 0 and raises NoProgramsObservedError.
  • scripts/test python: 46 passed. tests/next_tests/unit_tests/test_gt_cache_manager.py
    and otf_tests: 137 passed. Hooks clean.
  • The delete guard in the CLI was tightened with the move: a folder with a valid
    build-folder name but the wrong parent is still refused.

The translation caches now share a parent, but the build caches were still
loose in the cache base, one folder per compiled program variant sitting next
to `translation_cache/` — so the base mixed a fixed directory with an unbounded
set of program folders, and neither said which kind of cache it was.

Put them under `build_cache/`, so the base holds exactly the two kinds of cache
gt4py.next keeps and each says what it is:

    .gt4py_cache/
    ├── translation_cache/{dace,gtfn}/
    └── build_cache/<program>_pyext_<fingerprint>_<version>/

No backend level here: which backend produced a build folder is not knowable
where the folder is named, since `get_cache_folder` is reached from the
build-system layer, which is generic over the source language rather than aware
of backends. The fingerprint already keeps variants apart, so the flat set is
correct — only its location changes.

`dace_determinism` scans that directory and so far learned only the folder-name
pattern from the gt4py under test. Extend that probe to the directory name as
well, rather than hardcoding a second literal: a gt4py-side move of the build
cache now surfaces as a fetch error rather than as "no programs observed".

Entries under the previous layout are not migrated, and build caches are the
large ones. They are already discarded whenever the build-cache version
changes, so the stale directories can simply be removed.
@havogt
havogt force-pushed the havogt/next-build-cache-dir branch from 52a904d to 62795cc Compare August 20, 2026 11:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant