Reorganization of workflow repository

[hdpriest-ui]

Proposal: Repository Reorganization — Workflow Directory + Examples + Unified Manifest

Current Structure

In the current structure, there is no plain "workflow" that can be run on a user's provided data. There are a set of workflows which are bound to specific analyses or 'phases' of development. These are useful, as they are examples of particular analyses, but we need to also have a more agnostic workflow set of scripts which can, in theory, be used to analyze a set of user-provided data resources.

For clarity, the current structure is as follows:

workflows/
├── magic-ensemble
├── magic-ensemble-README.md
├── magic-ensemble-DEVELOPERS.md
├── README.md
├── data/
├── tools/
├── 1a_single_site/
│   ├── 01_prep_getERA5_met.R
│   ├── 02_prep_add_precip_to_clim_files.sh
│   ├── 03_prep_ic_build.R
│   ├── 04_run_model.R
│   ├── 05_validation.Rmd
│   ├── single_site_almond.xml
│   ├── README.md
│   ├── data_raw/
│   └── slurm_distributed_workflow/
├── 1b_statewide_woody/
│   ├── 00_install.sh
│   ├── 01_ERA5_nc_to_clim.R
│   ├── 02_ic_build.R
│   ├── 03_xml_build.R
│   ├── 04_run_model.R
│   ├── site_info.csv
│   ├── template.xml
│   ├── slurm_array_submit.sh
│   ├── *.Rmd
│   ├── README.md
│   └── site_irr_test/
└── 2a_grass/
    ├── 00_fetch_s3_and_prepare_run_dir.sh
    ├── 00_stage_external_inputs.sh
    ├── 01_ERA5_nc_to_clim.R
    ├── 02_ic_build.R
    ├── 03_xml_build.R
    ├── 04_run_model.R
    ├── workflow_manifest.yaml
    ├── example_user_config.yaml
    ├── site_info.csv
    ├── template.xml
    ├── slurm_array_submit.sh
    ├── *.Rmd
    └── README.md

We will create a 'workflow' directory which holds the canonical methods needed to run an ensemble analysis. Separately, the existing 1a, 1b, 2a, 3a, etc workflows will be subsumed into an 'examples' directory.

The 'magic-ensemble' CLI will be augmented to have 'prepare' methods which prepare a run directory to support execution of each of these example workflows. The R scripts in the example workflows will be directly invoked by the CLI command such as ./magic-ensemble prepare-example-1a --config ./examples/1a_single_site/example_user_config.yaml.

The CLI will require some modeling - there are paths in the manifest which utilize the current structure, and target scripts within the '2a_grass' directory. these will be updated to target the canonical script in the workflow directory.

Critically, the actual method for running ensembles will not be present in each workflow example directory. Those will be examples of different methods of data prep (primarily), and the actual execution of ensembles will run through ./workflow/04_run_model.R

I wonder to what extent we need to retain the numbered steps, but i have left them in, in the current proposal.

I would like to also make sure we have solidified and well-labeled s3 data artifacts for each workflow.

I would also like to see if we can deprecate some of the ancillary files in the example directory: these should be minimal examples without artifacts from development cycles in them. In the first iteration, we'll move these directories over as a single lift.

Proposed Structure

workflows/
├── magic-ensemble
├── magic-ensemble-README.md
├── magic-ensemble-DEVELOPERS.md
├── README.md
├── data/
├── tools/
├── workflow/                       
│   ├── workflow_manifest.yaml      
│   ├── 00_fetch_s3_and_prepare_run_dir.sh
│   ├── 00_stage_external_inputs.sh
│   ├── 01_ERA5_nc_to_clim.R
│   ├── 02_ic_build.R
│   ├── 03_xml_build.R
│   ├── 04_run_model.R
│   └── template.xml
└── examples/
    ├── 1a_single_site/
    │   ├── example_user_config.yaml
    │   ├── 01_prep_getERA5_met.R
    │   ├── 02_prep_add_precip_to_clim_files.sh
    │   ├── 03_prep_ic_build.R
    │   ├── single_site_almond.xml
    │   ├── 05_validation.Rmd
    │   ├── README.md
    │   ├── data_raw/
    │   └── slurm_distributed_workflow/
    ├── 1b_statewide_woody/
    │   ├── example_user_config.yaml 
    │   ├── 00_install.sh
    │   ├── 01_ERA5_nc_to_clim.R
    │   ├── 02_ic_build.R
    │   ├── 03_xml_build.R
    │   ├── site_info.csv
    │   ├── template.xml
    │   ├── slurm_array_submit.sh
    │   ├── *.Rmd
    │   ├── README.md
    │   └── site_irr_test/
    └── 2a_grass/
        ├── example_user_config.yaml
        ├── 00_fetch_s3_and_prepare_run_dir.sh
        ├── 00_stage_external_inputs.sh
        ├── 01_ERA5_nc_to_clim.R
        ├── 02_ic_build.R
        ├── 03_xml_build.R
        ├── site_info.csv
        ├── template.xml
        ├── slurm_array_submit.sh
        ├── *.Rmd
        └── README.md

Summarized changes:

1. 1a, 1b, and 2a move under examples/.
2. A new workflow/ directory holds the canonical data-agnostic versions of steps 00–03 (adapted from 2a_grass) and step 04. This is the implementation that backs the bare prepare, get-demo-data, and run-ensembles verbs.
3. workflow_manifest.yaml moves into workflow/ and expands to define all verbs: get-demo-data, prepare, prepare-example-1a, prepare-example-1b, prepare-example-2a, and run-ensembles. The example-specific prepare verbs point at scripts in their respective examples/ subdirectory; all other verbs point at workflow/.
4. 04_run_model.R is removed from all example directories. run-ensembles delegates to workflow/04_run_model.R regardless of which prepare verb was used.
5. Each example gets an example_user_config.yaml with defaults appropriate to that example.

What does not change:

• Steps 01–03 within each example remain example-specific.
• tools/ stays at the repo root.

[hdpriest-ui]

tagging @dlebauer and @infotroph for the notification

[dlebauer]

I think that this goes a long way to reducing redundancy, and associated maintenance and technical debt.

Would you envision the examples include examples from other components of the project (e.g. downscaling, uncertainty, etc)? Or would the examples/ approach be applied to each repository?

to what extent we need to retain the numbered steps

I think to the extend that the examples should be run in order, it makes sense to use ordered numbers 01-single-site-almond, 02-multisite-almond, - .... etc. But we don't need to stick with the phased design of the implementation plan as a reference, especially since we aren't planning to have an example for each phase.

[hdpriest-ui]

I think it would be useful to have any examples we have for each repository in an examples directory.

That being said, including the content in the repo itself may not be the most effective way to provide demo/examples. I wouldn't say I envision it as required - but, if we do have example/demo content to deliver in the repo, we should structure it consistently.

[dlebauer]

Are the two options to either keep repository specific examples in in each repository or a separate example repository?

[infotroph]

This seems like broadly the right approach. As long as we're moving things I suggest aspiring to pull steps 1-3 out to common versions too. Entirely possible that will turn out to be more work than we want, but conceptually they serve similar purposes in each example and removing the truly repeated steps would make it much easier to follow what does change between examples.

[hdpriest-ui]

Totally agree that the overall goal is that steps 1-3 become more generic, and are parameterized as-needed to fulfill their current function.

[infotroph]

I wonder to what extent we need to retain the numbered steps

I'd say what's there now is more strictly ordered than is needed, especially because xml_build, IC_build, and met handling are all intentionally ~independent of each others' execution status. The unavoidable sequencing is basically:

1. Define: Declare the site, dates, settings, scenarios. This stage is minimal in examples 1-2 and gets fancier in phase 3.
2. Get: Create/retrieve all the input files. Order mostly not important as long as all are present by end of step.
3. Setup: Instantiate all the the run directories and job scripts on disk. Lots of subtasks, but they're well-hidden inside PEcAn.
4. Compute: Actually run the model. I'd lump here in any post-run cleanup we need to automate.
5. Analyze.

[hdpriest-ui]

Over time, removing the numeric labeling and bringing that knowledge around the steps into a manual/readme is probably the way to go. You've summarized them well in those 5 items.