Document the obs4MIPs reference data that must come from ESGF, and test the fetch script against the requirements - #864
Conversation
The download tutorial covered the obs4REF registry and the CMIP6 input data, but never said which reference datasets neither of those provides. Five obs4MIPs datasets are only available from ESGF: 20CR-V2 psl, C3S-GTO-ECV-9-0 toz, CERES-EBAF-4-2-1 rlut/rlutcs/rsut/rsutcs, ERA-5 psl/ta/tas/ua, and NOAA-NCEI-LAI-AVHRR-5-0 lai. Missing any of them costs you the diagnostics that need it, and because an unmet reference requirement plans no executions rather than failing, the loss is silent. Document them with the diagnostics that require each, and give the `--kind obs4mips` snippet that fetches just this data. scripts/fetch-esgf.py maintains its request list by hand, and its TODO to derive the list from the data requirements is still open, so add tests that check the two against each other: every required source_id is either in the obs4REF registry or requested here, every dataset a requirement can select from the committed ESGF catalog snapshot is fetched, and no request names a source_id nothing requires. Each was confirmed to fail when the corresponding drift is introduced. The requirement side cannot be checked by taking the cross product of a requirement's facets: ESGF intersects facets, so the PMP ENSO requirement naming four sources and eight variables would appear to need 32 combinations when only 12 exist. The tests resolve against the catalog instead, which is also why the snapshot has to be filtered to its ESGF rows -- it is built from an archive that holds the obs4REF collection too. Constraint: The committed catalog snapshot only covers datasets someone has already downloaded, so the pair-level test cannot see a brand new source; the source-level test covers that case Rejected: Generate the request list from the data requirements now | the facet intersection means a generated list needs per-source grouping logic, which is the open TODO, not a docs change Confidence: high Scope-risk: narrow Directive: When adding an obs4MIPs data requirement, add or extend an Obs4MIPsRequest grouped by source_id -- one request naming several sources and several variables returns nothing
📝 WalkthroughWalkthroughThe change documents ESGF retrieval for obs4MIPs reference datasets and adds offline coverage tests. It clarifies diagnostic requirements, duplicate dataset handling, request grouping, and the Changesobs4MIPs ESGF fetching
Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches 💡 1⚔️ Resolve merge conflicts 💡
📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
…vers it The note added in the previous commit said fetching both the obs4REF and the ESGF copy of a dataset was safe because same-version copies share an instance_id and ingest as one dataset. They do ingest as one dataset, but that dataset then holds both sets of files. GPCP-Monthly-3-2 pr v20231205 ends up with 42: obs4REF's single 1983-2023 file plus ESGF's 41 yearly files covering the same period. A diagnostic reading it sees every time step twice, and both pmp/enso_tel and pmp/enso_proc read it. Recommend the five ESGF-only datasets by request id instead of --kind obs4mips, which also pulls the four duplicated ones, and keep --kind obs4mips for deployments that do not use the obs4REF registry. This is guidance for the current behaviour, not a fix. The fix is for the catalog to prefer the published obs4MIPs copy over the obs4REF one at the same version, which needs the obs4REF datasets to be ingested as their own source type first. Constraint: obs4REF and ESGF copies at the same version are indistinguishable once ingested, because both carry activity_id obs4MIPs and merge on instance_id before anything can choose between them Confidence: high Scope-risk: narrow Not-tested: The downstream effect on a diagnostic execution is inferred from the duplicated file list, not observed in a completed run
|
Correcting the note I added about fetching both copies of a dataset: it is not safe, and I have pushed a fix. Same-version copies do share an This means the workflow the tutorial previously implied (fetch the obs4REF registry, then That is guidance, not a fix. The fix is for the catalog to prefer the published obs4MIPs copy at the same version, which needs the obs4REF datasets ingested as their own source type first — tracked as follow-up work. |
There was a problem hiding this comment.
Actionable comments posted: 1
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 71c26ca2-a681-4877-8139-b5a51508b575
📒 Files selected for processing (4)
changelog/864.docs.mddocs/getting-started/02-download-datasets.mdscripts/fetch-esgf.pytests/scripts/test_fetch_esgf_coverage.py
🚧 Files skipped from review as they are similar to previous changes (3)
- changelog/864.docs.md
- tests/scripts/test_fetch_esgf_coverage.py
- scripts/fetch-esgf.py
Codecov Report✅ All modified and coverable lines are covered by tests.
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
1 participant
Summary
The download tutorial covered the obs4REF registry and the CMIP6 input data, but never said which reference datasets neither of those provides. This adds that, plus a snippet for fetching them, plus tests that stop
scripts/fetch-esgf.pydrifting from the diagnostics.What is actually needed from ESGF
Five obs4MIPs datasets are not in the obs4REF registry:
source_id20CR-V2pslpmp/extratropical-modes-of-variability-{nam,nao,npo,pna,sam}C3S-GTO-ECV-9-0tozesmvaltool/ozone-{annual-cycle,lat-time,nh-mar,sh-oct}CERES-EBAF-4-2-1rlut,rlutcs,rsut,rsutcsesmvaltool/cloud-radiative-effectsERA-5psl,ta,tas,uaesmvaltool/cloud-scatterplots-reference,esmvaltool/regional-historical-{annual-cycle,timeseries,trend}NOAA-NCEI-LAI-AVHRR-5-0laiilamb/lai-avh15c1An unmet reference requirement plans no executions rather than failing, so missing one of these shows up as a diagnostic that silently never runs.
Summary by CodeRabbit
Documentation
Tests