Feat: incremental glacier directory creation

[gampnico]

It's finally here!

For an easier-to-read demonstration of the new system, download and try the notebook:

Features

• Backwards compatible
• 30% less data
• Append your own files
• Manifests for each glacier directory, tracking OGGM version, user-added files, dataset tags, unique hashed IDs, timestamps and more...
• Manifest peeking so you can check what's in a gdir before loading it into memory
• Wrappers to batch convert existing glacier directories in utils/compat.py, with support for multiple base URLs and checks to make sure datasets are compatible. You can convert any region, number or combination of RGI IDs.
• Comes with zarr npz.

Overview

This changes the default glacier directory system from cumulative levels to incremental ones. Most levels on the cluster are now deltas, which contain only the files that are different or added relative to the previous levels. Clients layer levels they need into a single glacier directory, which now contain manifests to track changes and identify datasets.

L3 and L5 datasets are standalone. L3 because there are differences in certain files between L1 and L3 e.g. inversion_flowlines, and L5 because it's a "minimal" preset. I'm calling a combination of levels a "materialisation".

Every prepro level comes with a manifest, and creating a glacier directory now works by layering the levels they need into one glacier directory. Legacy datasets keep working via a fallback that treats them as cumulative materialisations.

Points for Discussion

• This is a really big change that includes changes from feat: read and write npz methods to replace read_pickle #1908.
• This is incremental in the sense that users pick from preset deltas available on the cluster, rather than "get this one file". I've decided to do this for multiple reasons:
  • This issue is solved with streaming, because we can use fsspec and xarray to pick and mix the data we want without downloading it first!
  • Zarr increases the number of files stored in each glacier directory. This means if a user downloads data for the globe, this is potentially millions of small files, which is going to strain the server and makes [Bug]: OGGM can't detect if cached downloads are truncated #1947 impossible to solve.
  • Compatibility checks between datasets become significantly harder when checking multiple single files.
  • Most OGGM workflows require a standard set of files anyway. If we want to give users access to extra files, we could introduce this with a higher preprocessing level (since these are now deltas the storage impact is minimal).

TODO:

• Incorporate @pat-schmitt's changes after dev is merged to master
• Replace the test base URL with an actual URL once data is available.
• Decide on terminology for documentation. So far I'm calling merged deltas or standalone prepros in a single gdir "materialisations" (e.g. L3, L5). but I think this is a bit awkward.
• Maybe add manifest schema version to dataset_id hash to prevent future manifests clashing?

Closes #1900
Refs: #1896

• Tests added/passed
• Fully documented
• Entry in whats-new.rst

[gampnico]

Review notes from today's meeting @fmaussion, @pat-schmitt.

• It should be possible to build different deltas from the same parent, i.e. deltas should be "parent-agnostic".
• Add a new kind="materialisation" to schema for clarity, especially since materialisations are going to be a very niche case.
• Deltas should use chained downloads: e.g. read L2 manifest -> download L1 -> read L1 manifest -> download L0.
• Need to review nomenclature both generally and in schema e.g. "artifact", "dataset" etc. This must be documented.
• Conversion scripts in compat are more useful for development or a small handful of users.
• More discussion needed on dataset_tag vs base_url: some compatibility checks can be relaxed since deltas should be parent-agnostic. My personal concern is having some kind of identifier for reproducibility, and it's a faster way of checking compatibility (when needed). With v0 this is handled with base_url and log.txt, but in practice a user can have two directories with the same base URL but different files.

@fmaussion your suggestions for expanding scope:

• Maybe have a watcher or some similar mechanism to see what exactly has changed between two glacier directories (or rather log.txt)?
• Maybe add a check to see if a settings.yml has changed? (Note this will already be marked in the manifest under files.updated, so maybe which specific parameters, e.g. via a diff).

[fmaussion]

thanks for the meeting @gampnico ! Quick notes:

It should be possible to build different deltas from the same parent, i.e. deltas should be "parent-agnostic".

Mh, more the other way around? Artifacts should be "child-agnostic". Some deltas need a very specific parent, while some deltas (briefly mentioned this morning) should be parent-fixed and others might be "parent-flexible".

Add a new kind="materialisation" to schema for clarity, especially since materialisations are going to be a very niche case.

To me, materialisations aren't niche but the main case from the user perspective: all of the gdirs they will build locally are materialisations. A materialisations hence needs to know all the parents it was built from, which parameter settings they used, which tasks were run, etc. not only incremental anymore.

Deltas should use chained downloads: e.g. read L2 manifest -> download L1 -> read L1 manifest -> download L0.

yes

Need to review nomenclature both generally and in schema e.g. "artifact", "dataset" etc. This must be documented.

I grew quite accustomed to "artifact" today!

More discussion needed on dataset_tag vs base_url: some compatibility checks can be relaxed since deltas should be parent-agnostic. My personal concern is having some kind of identifier for reproducibility, and it's a faster way of checking compatibility (when needed). With v0 this is handled with base_url and log.txt, but in practice a user can have two directories with the same base URL but different files.

Agreed that some id would be good. parent_base_url is a "must" I think, together with perhaps some hash version of params.yaml + base_url or something for the id. If the id check fails, the params can be checked. TBD