Skip to content

aws-healthomics: add S3 run storage, EXPRESS run mode, runtime platform, local scratch, image staging, run diagnosis/optimization and metrics steering docs - #216

Draft
ThandojuSruthi wants to merge 2 commits into
kirodotdev:mainfrom
ThandojuSruthi:healthomics-s3-express-arm64
Draft

ThandojuSruthi wants to merge 2 commits into
kirodotdev:mainfrom
ThandojuSruthi:healthomics-s3-express-arm64

Conversation

@ThandojuSruthi

Copy link
Copy Markdown

Issue #, if available: N/A

Description of changes:

Adds new steering docs to the aws-healthomics power and updates existing ones.

New steering docs

  • steering/s3-run-storage.md — S3 run storage (storageType: S3, workDirUri, taskWorkDirS3Uri): prerequisites, service-role permissions (work directory, output, and input buckets), starting and monitoring runs, run cache with S3 storage, reruns, local storage on S3 runs, and troubleshooting S3 storage failures.
  • steering/express-run-mode.md — EXPRESS run mode (runMode): when to use it, requirements (S3 run storage, runs under 8 hours, RESTRICTED networking), starting and verifying an EXPRESS run, and troubleshooting.
  • steering/runtime-platform.md — Choosing X86_64 or ARM64 compute with runtimePlatformPreference and task-level runtimePlatform (WDL, Nextflow, CWL): precedence, image-based inference, GPU and Region limitations, and troubleshooting.
  • steering/ephemeral-storage.md — Local scratch (ephemeral) storage for workflow tasks: scratchStorageMode, sizing with disk / disks / tmpdirMin, monitoring, billing, and troubleshooting /tmp exhaustion.
  • steering/image-staging.md — Staging container images from registries not supported by ECR pull-through cache (for example Seqera Wave, NVIDIA NGC, Google Artifact Registry).
  • steering/diagnose-run-failure.md — Diagnosing a specific failed run and guiding remediation, including routing feature-specific failure reasons to the matching steering doc.
  • steering/optimize-run.md — Analyzing a completed run for cost and performance optimizations and right-sizing.
  • steering/metrics-analysis.md — Answering run and workflow metrics questions with vended CloudWatch metrics: utilization, bottlenecks, stuck runs, right-sizing, and run comparisons.

Updated steering docs

  • POWER.md — Routing entries for all new steering docs.
  • steering/running-a-workflow.md — New "Storage Type, Run Mode, and Architecture" section with tooling guidance.
  • steering/batch-runs.md — storageType: S3, workDirUri, and runtimePlatformPreference in defaultRunSetting; note that runMode is not supported for batch runs and scratchStorageMode has no effect on S3 batches.
  • steering/workflow-development.md — Build images with an ARM64 variant (multi-architecture preferred) when ARM64 is wanted; x86_64 remains the default.
  • steering/migration-guide-for-wdl.md — Fix the container-tool check so it verifies every required binary, not just the first.
  • steering/regional-capabilities.md — Add g5.12xlarge and larger availability in Tokyo and Ohio.

Testing: Imported the power into Kiro IDE as a custom power and verified routing and answers for S3 run storage, EXPRESS run mode, ARM64, batch runs, run caching, local scratch, and run-failure diagnosis prompts.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant