docs: skill for working with a gt4py.next cache from a shell - #2794
docs: skill for working with a gt4py.next cache from a shell#2794havogt wants to merge 1 commit into
Conversation
afceb2e to
11621d4
Compare
| ## Find the cache | ||
|
|
||
| ```bash | ||
| find . -maxdepth 3 -name '.gt4py_cache' -type d |
There was a problem hiding this comment.
Probably instead of trying to find it, we should just say something like "it's in .gt4py_cache, next to the
working directory of the run (or under GT4PY_BUILD_CACHE_DIR)."
There was a problem hiding this comment.
Done — dropped the find invocation; the section now just states that it is .gt4py_cache next to the working directory of the run, or under GT4PY_BUILD_CACHE_DIR.
| @@ -0,0 +1,113 @@ | |||
| --- | |||
| name: translation-cache-shell | |||
| description: Inspect and prune a gt4py.next cache with shell tools only — find, grep, rm — when no gt4py-capable interpreter is at hand. TRIGGER when clearing or examining a `.gt4py_cache` on a machine where gt4py cannot be imported (a compute node, a container, a copied cache, a gt4py too old to ship `gt4py-next-cache`), or when a benchmark must be cleared of cached translations before it runs. SKIP when `gt4py-next-cache` is available — it takes the same lock as the runtime, which these recipes cannot. | |||
There was a problem hiding this comment.
Let's assume a cache manager cli does not exist.
There was a problem hiding this comment.
Done — every mention of a cache-manager CLI is gone, from the description and from the body.
| Under the default *session* lifetime there is nothing to find: the cache lives in | ||
| a temporary directory that is deleted when the process exits. Only | ||
| `GT4PY_BUILD_CACHE_LIFETIME=persistent` puts it in `.gt4py_cache`, next to the | ||
| working directory of the run (or under `GT4PY_BUILD_CACHE_DIR`). |
There was a problem hiding this comment.
Instead of mentioning the default explicitly, say it's only there if GT4PY_BUILD_CACHE_LIFETIME=persistent is set and link to the config module for defaults.
There was a problem hiding this comment.
Done — no longer names a default; it says the cache is only written there when GT4PY_BUILD_CACHE_LIFETIME=persistent, and links to config.py for the defaults.
| This **over-matches**: a name that contains another name returns both, so | ||
| `lap_program` also lists the entries of `laplap_program`. That is harmless — a | ||
| wrongly deleted entry is simply re-translated — and it never misses, because the | ||
| name is always literally present. Rely on it in that direction only: *entries | ||
| found* is a reason to delete, *nothing found* is trustworthy, but "found" does | ||
| not prove those entries would have been replayed. |
There was a problem hiding this comment.
| This **over-matches**: a name that contains another name returns both, so | |
| `lap_program` also lists the entries of `laplap_program`. That is harmless — a | |
| wrongly deleted entry is simply re-translated — and it never misses, because the | |
| name is always literally present. Rely on it in that direction only: *entries | |
| found* is a reason to delete, *nothing found* is trustworthy, but "found" does | |
| not prove those entries would have been replayed. | |
| This **over-matches**: a name that contains another name returns both. |
| Deleting every translation is `find <cache> -name '*.pkl' -delete`. Build folders | ||
| are untouched either way, which is what you want: the point is to force | ||
| re-translation, and the rebuild follows from it. |
There was a problem hiding this comment.
Not needed, we can just delete the whole cache folder.
There was a problem hiding this comment.
Done — replaced with "To clear everything, delete the cache folder."
| `gt4py-next-cache` — shipped with newer gt4py — takes `gt4py._core.locking.lock` | ||
| around each unlink, the same lock the runtime holds while writing. These recipes | ||
| do not, so an `rm` here can race a concurrent writer. On a shared filesystem with | ||
| MPI ranks compiling, clear the cache between jobs rather than during one. |
There was a problem hiding this comment.
We should not mention this tool here.
There was a problem hiding this comment.
Done — the section now states the constraint on its own: the runtime locks each cache file while writing it (gt4py._core.locking) and these recipes do not, so do not run them while ranks are compiling.
The translation cache is what makes a benchmark measure the old compiler, and clearing it means reaching into `.gt4py_cache` by hand wherever gt4py cannot be imported -- a compute node, a container, a cache copied from elsewhere. The reflex is `grep`. That works, but only in one form, and the two obvious refinements fail by reporting nothing -- which reads as nothing cached, the wrong way to be wrong here. Write down the form that holds and why the others do not: `-a` because the entries are binary and `ugrep` silently skips them otherwise, `-F` because a word-boundary pattern is defeated by pickle's length prefix, which sits immediately before the name and is itself alphanumeric for names of 48-57, 65-90 or 97-122 characters. On a real 224-entry cache that silently lost 3 of 22 programs. The delete pipeline gets `--null`/`-0` so a path with a space does not split, and `-r` so an empty match set does not invoke `rm` bare. The recipes scope by `*.pkl` rather than by directory, since only translation entries are pickles, so they hold across cache layouts. Each was run verbatim against two real caches before being written down.
11621d4 to
a698d26
Compare
|
Probably safer to provide a script in some form to delete entries (for deterministic results). |
1 participant
Independent of the cache-manager stack (#2769 → #2768 → #2792) — this adds one
skill file and nothing else, and its recipes are written to hold whatever cache
layout is in place.
Why
Clearing the translation cache normally needs a gt4py-capable interpreter: to
resolve the cache directory, to read an entry, to take the lock. On a compute
node, in a container, against a copied cache, or with a gt4py too old to ship
gt4py-next-cache, none of that is available — and the reflex isgrep.grepdoes work. But of the three forms one would naturally try, two fail byreporting nothing, which reads as nothing cached — the wrong direction to be
wrong in when the next step is a multi-node benchmark.
Measured against a real 224-entry icon4py cache and a smaller one containing
lap_program,laplap_programandskewedlap_program:grep -rlaF <name>grep -rlaP '(?<!\w)<name>(?!\w)'grep -rlaE '[^A-Za-z0-9_]<name>…'-augrepThe boundary failure has an exact cause: pickle stores a short string as
0x8c <length-byte> <bytes>, so the byte before the name is its length. Thethree that vanished are the three whose names are 48, 48 and 49 characters — the
length byte is then
'0','0','1', alphanumeric, so(?<!\w)rejects thematch. Any name of 48–57, 65–90 or 97–122 characters disappears the same way.
Plain
-Fis sound precisely because it errs the other way: it can never miss(the name is literally in the file) and only over-matches when one program name
contains another, which is harmless since a wrongly deleted entry is simply
re-translated.
What
.agents/skills/translation-cache-shell/— find the cache, inventory it, listthe entries of a program, clear them, and name a single entry; plus the two
anti-recommendations above and the one thing these recipes cannot do: take
gt4py._core.locking.lock, so they must not run while ranks are compiling.Recipes scope by
*.pklrather than by directory name, because only translationentries are pickles — verified 34/34 and 331/331 on two caches — so they survive
the layout changes in #2769 and #2792 regardless of merge order.
Kept separate from the
translation-cacheskill added in #2768, which drives theCLI: the two are selected in different situations, and this one exists to say
what it cannot do.
Verification
Every command in the file was run verbatim against two real caches, including
the entry-naming one-liner (exact on all 331 entries, names 14–58 chars; the
documented caveat is that a name of 65+ characters would leave a spurious leading
letter).
grephere resolves tougrep 7.5.0, which is how the missing--afailure was found — the documented form was then checked under both
ugrepandGNU grep 3.12.