Skip to content

feat: add Slack SR guard and resilient SR monitor#123

Open
warku123 wants to merge 7 commits into
tronprotocol:developfrom
warku123:feature/slack-sr-guard
Open

feat: add Slack SR guard and resilient SR monitor#123
warku123 wants to merge 7 commits into
tronprotocol:developfrom
warku123:feature/slack-sr-guard

Conversation

@warku123

@warku123 warku123 commented Jun 30, 2026

Copy link
Copy Markdown

What does this PR do?

Adds a new Slack SR guard tool alongside the existing Slack SR monitor, and refactors tools/slack_sr_monitor so both tools can run from one Docker image.

The monitor sends SR reports after each maintenance period. The new guard checks the current Top 27 SR set every minute and sends a Slack alert if any SR enters or leaves the Top 27 before maintenance. It also supports an optional phone-call escalation endpoint after the Slack alert is delivered successfully.

Why are these changes required?

The existing SR monitor only reports after maintenance. If community votes push an SR out of the Top 27 before the next maintenance period, there was no early warning.

This PR adds a lightweight guard for that risk, while also improving reliability around node fallback, state persistence, maintenance-time retries, and optional urgent phone escalation.

Changes overview:

  • Add sr_guard.go to check Top 27 membership every minute and alert on entered/left SRs
  • Add optional PHONE_ALERT_URL support for phone-call escalation after a successful Slack guard alert
  • Ensure phone alerts do not retry when the phone endpoint fails, while Slack failures still retry on the next guard poll
  • Refactor the tool into a shared main.go, sr_monitor.go, and sr_guard.go
  • Support monitor, guard, and default both run modes from one binary
  • Add multi-node fallback via TRON_NODES, with TRON_NODE kept as fallback
  • Add per-tool Slack webhooks with SLACK_WEBHOOK as common fallback
  • Persist monitor and guard state under logs/
  • Improve monitor retry behavior after maintenance before sending reports
  • Update Docker Compose, .env.example, and README usage docs

This PR has been tested by:

  • go test ./...
  • go build
  • docker compose config --quiet
  • Docker Compose startup for both services
  • Restart verification for monitor and guard state persistence
  • Manual phone endpoint POST test from the deployment server, with HTTP 200 and confirmed phone receipt

Extra details

  • State JSON files are loaded on startup and written after successful updates
  • Deleting state JSON files makes the tools create a fresh baseline on next successful fetch
  • The guard only alerts on Top 27 membership changes, not rank-only changes
  • Phone escalation is triggered only after the Slack guard alert succeeds and guard state is updated

warku123 and others added 7 commits June 30, 2026 11:33
Trigger a phone call via an optional PHONE_ALERT_URL webhook on the same
Top 27 membership change condition that already sends the Slack alert,
independent of and non-blocking to the existing Slack path.
Add phone-call escalation to slack_sr_guard
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant