Make alert recovery (→ OK) notifications optional, or let automations subscribe to selected severity transitions #17513
Replies: 1 comment
|
I would filter notification delivery while still committing the new alert severity. Otherwise suppressing recovery could leave the alert internally stuck in ALERT, and the next real breach might never produce a fresh transition. A small contract could distinguish breach, recovery and renotification subscriptions, preserving current behavior by default. The useful regression sequence is: Then cover WARNING -> ALERT, ALERT -> WARNING, unchanged ALERT with renotify enabled, and NO_DATA separately. “Destination severity is ALERT” is not quite the same as “a new breach”: it can also include repeated notifications. The current alert documentation already distinguishes severity changes, renotification and no-data handling, so keeping those categories explicit would avoid surprising interactions. As an interim integration pattern, a webhook consumer can ignore payloads whose severity is OK while still acknowledging receipt, rather than treating intentionally suppressed messages as delivery failures. That only filters the downstream notification; it does not change Langfuse's event generation, and consumers needing exact transitions must track prior state because the documented payload exposes the current severity. I have reviewed the documented state/payload contract, not tested an automation deployment. Your event-window example is a good acceptance fixture because expiry of one matching event should not require inventing a meaning of operational recovery. |
Uh oh!
There was an error while loading. Please reload this page.
Describe the feature or potential improvement
Alerts currently notify on every severity transition, and recovery (WARNING | ALERT → OK) always notifies (https://langfuse.com/docs/observability/features/alerts#alert-states). There's no way to opt out — not on the alert, and not on the automation (a monitor-sourced trigger has no event selection).
For existence-style alerts, this makes the recovery message meaningless. For example:
count(scores-numeric.count) is above 0 over the last 5m
A single matching score fires [ALERT], and about five minutes later - once that score ages out of the window - you get
[OK]...count(scores-numeric.count) recovered. Nothing recovered - it only means no second occurrence within 5 minutes. Every real alert arrives as a pair, and the second half is noise.It would be nice if the recovery (OK) notification is optional or if there's a way to select the severity to alert.
Additional information
Support case 5709.
All reactions