feat[next]: cache manager CLI for the translation caches - #2766
Closed
havogt wants to merge 6 commits into
Closed
Conversation
gt4py.next keeps a translation cache next to the build cache: the pickled output of the translation step, i.e. the optimized SDFG for dace and the generated sources for gtfn. Its key covers the program and the gt4py version, not the gt4py sources, so editing a transformation or an optimization pass without committing leaves the entry valid. The next run then replays the cached translation while still recompiling, which refreshes the library's mtime and makes the change look applied when it never ran. Add 'python -m gt4py.next.gt_cache_manager', mirroring the existing cartesian cache manager, with 'path', 'list', 'show', 'delete' and 'status' subcommands over both translation caches. 'status' reports per program whether the next run re-translates or replays and warns about a cached translation without a build folder; '--fail-if-cached' turns it into a pre-flight gate. 'delete' defaults to a dry run, takes the same lock as the runtime, and leaves build folders alone unless '--include-build-dirs' is given. The cache layout is read from where it is defined rather than duplicated: the translation cache directory names and the bindings name suffix move next to 'get_cache_folder', and the compiledb prototype name prefix becomes a constant so its shared folder is not mistaken for a program's build folder.
Contributor
Author
|
Note it's not in the script directory in purpose, because you want to run it in the venv where gt4py lives that created the cache dir. |
Declare the first console script of the distribution so the tool is available wherever gt4py is installed, without having to spell out the module path. The module form keeps working; the program name in the help text follows the invocation. Named after the subpackage rather than `gt4py-cache`, since `gt4py.cartesian` has a separate cache with its own manager.
Two corrections to `gt_cache_manager`, both about not claiming more than the data supports. `status` fused translation and build into one verdict and claimed a cached translation always came with a recompile. The two caches hit and miss independently: with both warm nothing is redone at all. Measured on the laplacian tests, a second run rebuilt no library (identical mtimes, 184s -> 26s), while `status` claimed it would recompile. Report a verdict per cache instead: TRANSLATION replays or re-translates, BUILD reuses or recompiles, and the warning now fires on the combination that is actually misleading -- a rebuilt library next to a replayed translation. A build folder only counts towards `reuse` if it can still be hit. It is stale when the build never finished, or when it was built under another build-cache version, which salts the folder name -- visible as soon as an editable install goes dirty, where every folder gains a `.dirty` suffix and the previous ones become unreachable. Translation entries record no version and cannot be told apart this way; `status --help` says so rather than implying more precision than the data allows. The completion marker moves from the DaCe compile step to `build_data`, next to the `BuildData` status it complements, so both ways a folder records a finished build are asked in one place. `delete` needed `--yes` to do anything, which reads as a safety net but makes the ordinary interactive case awkward: you always run the command twice. Follow the convention instead (clig.dev, `docker system prune`, `apt`): list what the selector matched, then ask `[y/N]` -- defaulting to no -- and remove on confirmation. Prompt only when stdin is a terminal; elsewhere require `--yes` and refuse otherwise, so a script or job file never blocks on a prompt and never crashes on EOF the way `pip uninstall` does. `-n/--dry-run` keeps the preview available on its own, and `-y` replaces `-f` as the short form: this skips a confirmation, it does not force past an obstacle.
Build folders carry the build-cache version in their name, translation entries do not, so `status` can rule out an unreachable build but not an unreachable entry. After the version changes, entries written by earlier runs are still reported as REPLAY although no key can reach them: observed on the laplacian tests, where a run following a version change re-translated every program (10 -> 15 entries) while `status` claimed REPLAY. The evidence is visible even though the entries are not: stale build folders in the same cache show the version moved. Report that, so a REPLAY that is really a leftover is recognizable instead of silently overstating the risk.
A hit is decided by a fingerprint taken at run time over the lowered program and its arguments; the caches record only a program name, so `status` cannot predict one. It nevertheless read as a prediction, and got it wrong in the most ordinary case there is: edit a program's source, and the entries under its unchanged name are reported as a replay although that edit already invalidated them. Observed on a two-line probe program -- after the edit, `status` claimed replay and reuse, while the run translated and compiled it afresh. Only one direction is knowable, so say which: `will re-translate` and `will recompile` are guarantees, since nothing on disk can serve the program, while `may replay` and `may reuse` say something on disk could. This keeps the gate useful -- after a delete, the guarantee is what it reports -- and downgrades the alarm to what it is, a reason to delete rather than evidence of a replay.
`status` was `list` grouped by program: the same entries, the same filter under a second flag name, plus the build folders and the warning. Make that a mode of the lister instead -- `list --by-program` -- so there is one command over one data set, and `--fail-if-cached` moves with it as a plain "anything matched" gate. Grouped output now reports counts only. Predicting what a run will do needs a fingerprint over the lowered program and its arguments, which the caches do not record, so the verdict columns were dressing an entry count up as a forecast -- one that is wrong whenever a program's source was edited. The warning stays, since a cached translation without a usable build folder is the trap worth naming, and it is now in the output rather than behind a command few would think to run. `show` goes with the payload introspection it needed: reporting SDFG name, state and map counts for a single entry answers a question nobody has had yet, and it was the only reason to reach inside a cached payload at all.
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
gt4py.nextkeeps two persistent caches side by side under the cache base:<program>_pyext_<hash>_<version>/translation_cache/(dace),gtfn_cache/(gtfn)The translation cache key covers the program and
gt4py.__version__, not thegt4py sources. Editing a transformation or an optimization pass therefore leaves
the key intact (an editable install only changes its version on a commit — the
dirty marker is a constant
.dirtysuffix — and a non-editable install neverdoes). The next run replays the cached translation and skips the pass, while the
build step still recompiles from it and refreshes the library's mtime.
That combination is easy to misread: clearing only the
*_pyext_*folder andobserving a rebuilt library looks like proof that a changed pass ran, when it
never executed. It cost four multi-node benchmark jobs and about a day of
investigation once, with two further tells misread along the way (successive SDFG
dumps differing only in
guidfields, and debug logging inside the passproducing no output).
What
gt4py-next-cache(alsopython -m gt4py.next.gt_cache_manager), mirroring theexisting
gt4py.cartesian.gt_cache_manager: stdlibargparse, library functionsplus a
__main__block, no new dependency. It covers both translation caches,with
--backendto narrow and--cache-dirto point at another cache base.Three sub-commands:
path— the cache directories this environment resolves, honouringGT4PY_BUILD_CACHE_DIR/GT4PY_BUILD_CACHE_LIFETIME. Answers the firstquestion everyone has, since the default session lifetime puts the cache in a
temporary directory that is gone when the process exits.
list— the entries: backend, key, program, size, mtime;--filter,--sort,--json. With--by-program, one row per program plus its buildfolders, which is where the two-cache trap becomes visible.
--fail-if-cachedexits non-zero if anything matched, to gate a job on an empty cache.
delete— by--key,--program <glob>or--all. Lists what matched,then asks
[y/N]; per clig.dev it only prompts when stdinis a terminal and otherwise requires
--yes, so a job file never blocks on aprompt (nor crashes on EOF the way
pip uninstalldoes).-n/--dry-runpreviews. Takes the same
locking.lockas the runtime (these caches are sharedbetween MPI ranks) and leaves build folders alone unless
--include-build-dirs.What it deliberately does not do: predict. A cache hit is decided by a
fingerprint taken at run time over the lowered program and its arguments
(
CachedStep.cache_key), which no tool outside a run can compute. The cachesrecord only a program name, so everything here is a count of what is on disk.
Read in the one direction that holds: no entries for a program means it will be
re-translated; entries present is a reason to delete, never evidence that a run
replayed. Build folders are the exception in one respect — their name records the
build-cache version, so folders no run here can hit are counted as stale.
Entries are read directly rather than through
FileCache, whose__getitem__deletes what it cannot unpickle — an inspection tool must not do that. A corrupt
or version-skewed entry degrades to
<unreadable>instead of crashing.Cache layout is read from where it is defined instead of being duplicated:
TRANSLATION_CACHE_DIR_NAMESandBINDINGS_NAME_SUFFIXnow live next toget_cache_folder; the compiledb prototype name prefix became a constant so theshared compiledb folder — named like a program build folder — is neither reported
as a program nor deleted along with one; and the DaCe compile-completion marker
moved to
build_data, next to theBuildDatastatus it complements, so "did thisbuild finish" is answered in one place.
[project.scripts]gains the distribution's first console script, so the toolis on
PATHwherever gt4py is installed — including an icon4py environment on acluster, which is where this failure mode actually bites. Named after the
subpackage rather than
gt4py-cachebecausegt4py.cartesianhas a separatecache with its own manager. Happy to add an ADR for the new packaging surface if
reviewers want one.
Also adds an agent skill (
.agents/skills/translation-cache/) that fires beforebenchmarking a codegen change, states the anti-pattern explicitly (a fresh
library mtime is not evidence a pass ran), and gives the delete/verify recipe plus
the
GT4PY_BUILD_CACHE_VERSION_IDalternative.Verification
ProgramSourcepayloads throughFileCache:path resolution under both lifetimes, empty/missing/populated/corrupt caches,
grouping and staleness, the confirmation flow (terminal yes/no, EOF, and that a
non-interactive run never prompts), delete dry-run vs
--yes, refusal to touchanything outside the cache, and that build folders survive a delete without
--include-build-dirs.tests/next_tests/unit_tests: 2211 passed.mypy src/andpre-commitclean.
version, and end to end with actual compiles on both backends.
The behavioural claims were measured, not assumed, and two of them corrected the
tool: a re-run with both caches warm rebuilds nothing (26 identical
.somtimes,184s → 26s), and editing a program's source re-translates it although its entries
still sit there under the unchanged program name.