Skip to content

feat[next]: cache manager CLI for the translation caches - #2766

Closed
havogt wants to merge 6 commits into
GridTools:mainfrom
havogt:next-cache-manager
Closed

feat[next]: cache manager CLI for the translation caches#2766
havogt wants to merge 6 commits into
GridTools:mainfrom
havogt:next-cache-manager

Conversation

@havogt

@havogt havogt commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

Why

gt4py.next keeps two persistent caches side by side under the cache base:

Cache Where Holds
Build cache <program>_pyext_<hash>_<version>/ build artifacts, the compiled library
Translation cache translation_cache/ (dace), gtfn_cache/ (gtfn) the already translated program: optimized SDFG / generated sources

The translation cache key covers the program and gt4py.__version__, not the
gt4py sources. Editing a transformation or an optimization pass therefore leaves
the key intact (an editable install only changes its version on a commit — the
dirty marker is a constant .dirty suffix — and a non-editable install never
does). The next run replays the cached translation and skips the pass, while the
build step still recompiles from it and refreshes the library's mtime.

That combination is easy to misread: clearing only the *_pyext_* folder and
observing a rebuilt library looks like proof that a changed pass ran, when it
never executed. It cost four multi-node benchmark jobs and about a day of
investigation once, with two further tells misread along the way (successive SDFG
dumps differing only in guid fields, and debug logging inside the pass
producing no output).

What

gt4py-next-cache (also python -m gt4py.next.gt_cache_manager), mirroring the
existing gt4py.cartesian.gt_cache_manager: stdlib argparse, library functions
plus a __main__ block, no new dependency. It covers both translation caches,
with --backend to narrow and --cache-dir to point at another cache base.

Three sub-commands:

  • path — the cache directories this environment resolves, honouring
    GT4PY_BUILD_CACHE_DIR / GT4PY_BUILD_CACHE_LIFETIME. Answers the first
    question everyone has, since the default session lifetime puts the cache in a
    temporary directory that is gone when the process exits.
  • list — the entries: backend, key, program, size, mtime; --filter,
    --sort, --json. With --by-program, one row per program plus its build
    folders, which is where the two-cache trap becomes visible. --fail-if-cached
    exits non-zero if anything matched, to gate a job on an empty cache.
  • delete — by --key, --program <glob> or --all. Lists what matched,
    then asks [y/N]; per clig.dev it only prompts when stdin
    is a terminal and otherwise requires --yes, so a job file never blocks on a
    prompt (nor crashes on EOF the way pip uninstall does). -n/--dry-run
    previews. Takes the same locking.lock as the runtime (these caches are shared
    between MPI ranks) and leaves build folders alone unless --include-build-dirs.
$ gt4py-next-cache list --by-program
PROGRAM            ENTRIES         BUILDS
lap_program        3 dace, 2 gtfn  0 (+15 stale)
laplap_program     9 dace, 6 gtfn  0 (+15 stale)

WARNING: 2 program(s) have a cached translation but no usable build folder. Their
next run rebuilds the library while the cached translation is replayed, so a fresh
library there would NOT mean a changed pass ran.

What it deliberately does not do: predict. A cache hit is decided by a
fingerprint taken at run time over the lowered program and its arguments
(CachedStep.cache_key), which no tool outside a run can compute. The caches
record only a program name, so everything here is a count of what is on disk.
Read in the one direction that holds: no entries for a program means it will be
re-translated; entries present is a reason to delete, never evidence that a run
replayed. Build folders are the exception in one respect — their name records the
build-cache version, so folders no run here can hit are counted as stale.

Entries are read directly rather than through FileCache, whose __getitem__
deletes what it cannot unpickle — an inspection tool must not do that. A corrupt
or version-skewed entry degrades to <unreadable> instead of crashing.

Cache layout is read from where it is defined instead of being duplicated:
TRANSLATION_CACHE_DIR_NAMES and BINDINGS_NAME_SUFFIX now live next to
get_cache_folder; the compiledb prototype name prefix became a constant so the
shared compiledb folder — named like a program build folder — is neither reported
as a program nor deleted along with one; and the DaCe compile-completion marker
moved to build_data, next to the BuildData status it complements, so "did this
build finish" is answered in one place.

[project.scripts] gains the distribution's first console script, so the tool
is on PATH wherever gt4py is installed — including an icon4py environment on a
cluster, which is where this failure mode actually bites. Named after the
subpackage rather than gt4py-cache because gt4py.cartesian has a separate
cache with its own manager. Happy to add an ADR for the new packaging surface if
reviewers want one.

Also adds an agent skill (.agents/skills/translation-cache/) that fires before
benchmarking a codegen change, states the anti-pattern explicitly (a fresh
library mtime is not evidence a pass ran), and gives the delete/verify recipe plus
the GT4PY_BUILD_CACHE_VERSION_ID alternative.

Verification

  • 43 unit tests building real ProgramSource payloads through FileCache:
    path resolution under both lifetimes, empty/missing/populated/corrupt caches,
    grouping and staleness, the confirmation flow (terminal yes/no, EOF, and that a
    non-interactive run never prompts), delete dry-run vs --yes, refusal to touch
    anything outside the cache, and that build folders survive a delete without
    --include-build-dirs.
  • Full tests/next_tests/unit_tests: 2211 passed. mypy src/ and pre-commit
    clean.
  • Exercised against a real 224-entry icon4py cache written by a different gt4py
    version, and end to end with actual compiles on both backends.

The behavioural claims were measured, not assumed, and two of them corrected the
tool: a re-run with both caches warm rebuilds nothing (26 identical .so mtimes,
184s → 26s), and editing a program's source re-translates it although its entries
still sit there under the unchanged program name.

gt4py.next keeps a translation cache next to the build cache: the pickled
output of the translation step, i.e. the optimized SDFG for dace and the
generated sources for gtfn. Its key covers the program and the gt4py version,
not the gt4py sources, so editing a transformation or an optimization pass
without committing leaves the entry valid. The next run then replays the cached
translation while still recompiling, which refreshes the library's mtime and
makes the change look applied when it never ran.

Add 'python -m gt4py.next.gt_cache_manager', mirroring the existing cartesian
cache manager, with 'path', 'list', 'show', 'delete' and 'status' subcommands
over both translation caches. 'status' reports per program whether the next run
re-translates or replays and warns about a cached translation without a build
folder; '--fail-if-cached' turns it into a pre-flight gate. 'delete' defaults to
a dry run, takes the same lock as the runtime, and leaves build folders alone
unless '--include-build-dirs' is given.

The cache layout is read from where it is defined rather than duplicated: the
translation cache directory names and the bindings name suffix move next to
'get_cache_folder', and the compiledb prototype name prefix becomes a constant
so its shared folder is not mistaken for a program's build folder.
@havogt

havogt commented Aug 10, 2026

Copy link
Copy Markdown
Contributor Author

Note it's not in the script directory in purpose, because you want to run it in the venv where gt4py lives that created the cache dir.

havogt added 5 commits August 10, 2026 12:05
Declare the first console script of the distribution so the tool is available
wherever gt4py is installed, without having to spell out the module path. The
module form keeps working; the program name in the help text follows the
invocation.

Named after the subpackage rather than `gt4py-cache`, since `gt4py.cartesian`
has a separate cache with its own manager.
Two corrections to `gt_cache_manager`, both about not claiming more than the
data supports.

`status` fused translation and build into one verdict and claimed a cached
translation always came with a recompile. The two caches hit and miss
independently: with both warm nothing is redone at all. Measured on the
laplacian tests, a second run rebuilt no library (identical mtimes, 184s ->
26s), while `status` claimed it would recompile. Report a verdict per cache
instead: TRANSLATION replays or re-translates, BUILD reuses or recompiles, and
the warning now fires on the combination that is actually misleading -- a
rebuilt library next to a replayed translation.

A build folder only counts towards `reuse` if it can still be hit. It is stale
when the build never finished, or when it was built under another build-cache
version, which salts the folder name -- visible as soon as an editable install
goes dirty, where every folder gains a `.dirty` suffix and the previous ones
become unreachable. Translation entries record no version and cannot be told
apart this way; `status --help` says so rather than implying more precision
than the data allows. The completion marker moves from the DaCe compile step to
`build_data`, next to the `BuildData` status it complements, so both ways a
folder records a finished build are asked in one place.

`delete` needed `--yes` to do anything, which reads as a safety net but makes
the ordinary interactive case awkward: you always run the command twice. Follow
the convention instead (clig.dev, `docker system prune`, `apt`): list what the
selector matched, then ask `[y/N]` -- defaulting to no -- and remove on
confirmation. Prompt only when stdin is a terminal; elsewhere require `--yes`
and refuse otherwise, so a script or job file never blocks on a prompt and
never crashes on EOF the way `pip uninstall` does. `-n/--dry-run` keeps the
preview available on its own, and `-y` replaces `-f` as the short form: this
skips a confirmation, it does not force past an obstacle.
Build folders carry the build-cache version in their name, translation entries
do not, so `status` can rule out an unreachable build but not an unreachable
entry. After the version changes, entries written by earlier runs are still
reported as REPLAY although no key can reach them: observed on the laplacian
tests, where a run following a version change re-translated every program
(10 -> 15 entries) while `status` claimed REPLAY.

The evidence is visible even though the entries are not: stale build folders in
the same cache show the version moved. Report that, so a REPLAY that is really
a leftover is recognizable instead of silently overstating the risk.
A hit is decided by a fingerprint taken at run time over the lowered program and
its arguments; the caches record only a program name, so `status` cannot predict
one. It nevertheless read as a prediction, and got it wrong in the most ordinary
case there is: edit a program's source, and the entries under its unchanged name
are reported as a replay although that edit already invalidated them. Observed
on a two-line probe program -- after the edit, `status` claimed replay and reuse,
while the run translated and compiled it afresh.

Only one direction is knowable, so say which: `will re-translate` and `will
recompile` are guarantees, since nothing on disk can serve the program, while
`may replay` and `may reuse` say something on disk could. This keeps the gate
useful -- after a delete, the guarantee is what it reports -- and downgrades the
alarm to what it is, a reason to delete rather than evidence of a replay.
`status` was `list` grouped by program: the same entries, the same filter under a
second flag name, plus the build folders and the warning. Make that a mode of the
lister instead -- `list --by-program` -- so there is one command over one data
set, and `--fail-if-cached` moves with it as a plain "anything matched" gate.

Grouped output now reports counts only. Predicting what a run will do needs a
fingerprint over the lowered program and its arguments, which the caches do not
record, so the verdict columns were dressing an entry count up as a forecast --
one that is wrong whenever a program's source was edited. The warning stays,
since a cached translation without a usable build folder is the trap worth
naming, and it is now in the output rather than behind a command few would think
to run.

`show` goes with the payload introspection it needed: reporting SDFG name, state
and map counts for a single entry answers a question nobody has had yet, and it
was the only reason to reach inside a cached payload at all.
@havogt

havogt commented Aug 10, 2026

Copy link
Copy Markdown
Contributor Author

Superseded by #2768 (same content, now on a branch in this repo so it can carry a stack) and #2769 (the cache directory rename on top). Closing to avoid a duplicate; no review had started here.

@havogt havogt closed this Aug 10, 2026
@havogt
havogt deleted the next-cache-manager branch August 10, 2026 13:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant