Skip to content

feat: let climate-openeo-create cover the last N days - #9

Draft
knutdrand wants to merge 3 commits into
hisptz:mainfrom
knutdrand:feat/climate-openeo-last-days
Draft

knutdrand wants to merge 3 commits into
hisptz:mainfrom
knutdrand:feat/climate-openeo-last-days

Conversation

@knutdrand

Copy link
Copy Markdown

Summary

This is only an idea for how to make scheduled climate pipelines work, from setting CAPS up on our climate-demo server. We may well have misunderstood how pipelines are meant to get their periods, so please treat it as a question as much as a change.

What we ran into: climate-openeo-create takes a fixed period (id/endId). For a pipeline on a weekly schedule we could not find anything that moves that window forward (the scheduler's inputContext is static, and the step does not read the open-climate-service-sync output), so every run would fetch the same dates. prediction-trigger has periodOffset for this; the climate step has nothing similar.

The idea: let the period carry lastDays instead of id. Without an id, the step covers the lastDays days up to and including yesterday (UTC), worked out when it runs. A window wider than a week (we use 35, and 70 for CHIRPS, which publishes about a month late) also picks up late and revised source data, and re-uploading overlapping days is harmless with CREATE_AND_UPDATE. An id still takes precedence, so a manual run can override a rolling config through its context, which we used for a year-by-year backfill.

"period": { "periodType": "daily", "lastDays": 35 }

Builds on #8 (its first two commits are #8's); only the last commit is new here.

Type of change

  • feat — new capability (minor)
  • fix — bug fix (patch)
  • docs — documentation only
  • refactor / chore / ci / test
  • Breaking change (feat! / BREAKING CHANGE: in the commit)

How to verify

  • bunx --bun vitest run src/tests/unit (370 passed); tsc --noEmit, eslint, prettier --check clean
  • New tests in climateOpenEoConfig.test.ts: lastDays alone is valid, neither id nor lastDays is invalid, lastDays: 35 on 2026-10-09 gives 2026-09-04..2026-10-08, and id wins over lastDays.
  • Running on climate-demo since 2026-10-09 in a weekly pipeline (OCS sync, two climate chains with lastDays, analytics, chap prediction).

Checklist

  • Conventional Commit title (used by semantic-release on main)
  • bun run lint and bun run typecheck pass locally
  • Tests added or updated when behavior changed (bun run test / make test)
  • No secrets, .env, or production credentials in the diff
  • Handler keys / queue names / API paths match the code (no drive-by renames)
  • I have read CONTRIBUTING.md

Open Climate Service's aggregate_to_dhis2_json workflow no longer takes
data_element_id and period_type (dhis2/open-climate-service f548922). It takes
a named export from the instance config, which owns the data element mapping
and the period type. The step now passes that export instead.

BREAKING CHANGE: climate-openeo-create's config replaces variable.dataElement
with exportId, the id of an export under exports in the OCS instance config.
A named DHIS2 export is saved as export-<id>.json beside a manifest, so the
hardcoded result.json is a 404. Take the file name from the job's result
asset, or its first data asset.
A scheduled pipeline has a fixed config, so a fixed period id fetches the same
dates on every run. With lastDays and no id, the step covers the lastDays days
up to and including yesterday (UTC), worked out when it runs. An id still takes
precedence, so a manual run can override a rolling config through its context.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant