Skip to content

Storm detection and ERROR notification rate limiting - #3

Merged
phoeberepo merged 15 commits into
mainfrom
feature/psham-sigevent-storm-detection
Sep 9, 2026
Merged

phoeberepo merged 15 commits into
mainfrom
feature/psham-sigevent-storm-detection

Conversation

@phoeberepo

Copy link
Copy Markdown
Contributor
  • Storm detection. Added storm_threshold events for one collection in a rolling window,
    individual emails stop and a single summary carrying a running count is sent instead. The
    window doubles on each roll while the storm continues, capped at
    storm_summary_max_interval_minutes, so an unattended overnight storm backs off rather than
    summarizing forever.
  • Daily ERROR cap (max_daily_errors), with a one-time "limit reached" notice so a cap
    never turns a flood into silence. Storm summaries and the cap notice do not count against it.
  • Per-recipient send isolation. One bad address no longer aborts the invocation, which
    previously dead-lettered the message and re-sent to every address that had already succeeded.
  • Four new variables mirrored into SSM, defaulting to 20 / 10 / 5 min / 60 min.

@tloubrieu-jpl tloubrieu-jpl moved this to needs:triage in podaac Sep 3, 2026
@tloubrieu-jpl tloubrieu-jpl added the team:i&a Ingestion and Archive team label Sep 8, 2026
@hkryeung

hkryeung commented Sep 9, 2026

Copy link
Copy Markdown

@phoeberepo is there a reason there's no /terraform/tfvars/ops.tfvars file?

@phoeberepo

Copy link
Copy Markdown
Contributor Author

@phoeberepo is there a reason there's no /terraform/tfvars/ops.tfvars file?

Added ops.tfvars file f97d2d7

@phoeberepo
phoeberepo merged commit 2bca15a into main Sep 9, 2026
@tloubrieu-jpl tloubrieu-jpl moved this from needs:triage to closed in podaac Sep 9, 2026
@phoeberepo
phoeberepo deleted the feature/psham-sigevent-storm-detection branch September 9, 2026 20:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

team:i&a Ingestion and Archive team

Projects

Status: closed

Development

Successfully merging this pull request may close these issues.

3 participants