feat: add Slack SR guard and resilient SR monitor

tools/slack_sr_monitor/.env.example

@@ -3,6 +3,20 @@
 # The Slack Webhook URL for sending notifications
 SLACK_WEBHOOK=your_slack_webhook_url_here
 
+# Optional per-tool Slack webhook URLs.
+# If not set, each tool falls back to SLACK_WEBHOOK.
+# SLACK_SR_MONITOR_WEBHOOK=your_monitor_slack_webhook_url_here
+# SLACK_SR_GUARD_WEBHOOK=your_guard_slack_webhook_url_here
+
 # The Tron node API endpoint
 # Default: https://api.trongrid.io
 TRON_NODE=https://api.trongrid.io
+
+# Optional comma-separated fallback Tron node API endpoints.
+# If set, TRON_NODES takes precedence over TRON_NODE.
+# TRON_NODES=https://api.trongrid.io,http://127.0.0.1:8090
+
+# Optional: have slack_sr_guard also trigger a phone/voice call alert when
+# the Top 27 changes, by POSTing a short message to this webhook endpoint.
+# If unset, this is skipped entirely and only the Slack alert is sent.
+# PHONE_ALERT_URL=your_phone_alert_webhook_url_here

tools/slack_sr_monitor/.gitignore

@@ -1,3 +1,4 @@
-slack_sr_monitor
+/slack_sr_monitor
+/slack_sr_guard
 logs/
 .env

tools/slack_sr_monitor/Dockerfile

@@ -11,7 +11,7 @@ RUN go mod download
 COPY . .
 
 # Build the application
-RUN CGO_ENABLED=0 GOOS=linux go build -o slack_sr_monitor main.go
+RUN CGO_ENABLED=0 GOOS=linux go build -o slack_sr_monitor .
 
 # Final stage
 FROM alpine:3.20@sha256:d9e853e87e55526f6b2917df91a2115c36dd7c696a35be12163d44e6e2a4b6bc

tools/slack_sr_monitor/README.md

@@ -2,8 +2,10 @@
 The Slack SR Monitor tool is designed to monitor TRON Super Representatives (SRs) and notify a Slack channel after every maintenance period.
 It automatically tracks vote changes and detects replacements in the top 27 SR positions, providing a clear and formatted report.
 
-### Build and Run the monitor
-To run the monitor tool, you can choose between native Go execution or Docker deployment.
+This directory also includes `slack_sr_guard`, a lightweight early-warning tool that polls the live SR list every minute and alerts Slack as soon as the current Top 27 membership changes.
+
+### Build and Run
+To run either tool, choose native Go execution or Docker deployment.
 
 #### Native Go Execution
 Make sure you have Go 1.25+ installed.
@@ -12,24 +14,49 @@ Make sure you have Go 1.25+ installed.
 cd tools/slack_sr_monitor
 # install dependencies
 go mod tidy
-# run the tool
-go run main.go
+# run both tools by default
+go run .
+
+# run only the maintenance-period monitor
+go run . monitor
+
+# run only the early-warning guard
+go run . guard
 ```
 
 #### Docker Deployment
 We provide a Docker-based deployment for easier management in production environments.
 ```shell
-# build and start the container
-docker-compose up -d --build
+# build and start both tools
+docker compose up -d --build
+
+# build and start only one tool
+docker compose up -d --build slack-sr-monitor
+docker compose up -d --build slack-sr-guard
+
 # check logs
 docker logs -f slack-sr-monitor
+docker logs -f slack-sr-guard
 ```
 
 ### Configuration
 All configurations are managed via environment variables or a `.env` file in the project root. Please refer to [.env.example](./.env.example) as an example.
 
 - `SLACK_WEBHOOK`: The Slack Incoming Webhook URL used to send notifications.
+- `SLACK_SR_MONITOR_WEBHOOK`: Optional Slack webhook URL for `slack_sr_monitor`. Falls back to `SLACK_WEBHOOK` if unset.
+- `SLACK_SR_GUARD_WEBHOOK`: Optional Slack webhook URL for `slack_sr_guard`. Falls back to `SLACK_WEBHOOK` if unset.
 - `TRON_NODE`: The TRON node HTTP API endpoint (e.g., `http://https://api.trongrid.io`). Default is Trongrid.
+- `TRON_NODES`: Optional comma-separated TRON node HTTP API endpoints. If set, the tools try each node in order for every TRON API request and only return an error after all nodes fail. This takes precedence over `TRON_NODE`.
+- `PHONE_ALERT_URL`: Optional webhook endpoint for phone/voice call escalation. When set, `slack_sr_guard` sends a short alert message here on every Top 27 change, in addition to the Slack alert, so on-call staff can be reached even outside Slack. Skipped entirely when unset.
+
+### State Persistence
+
+Both tools persist lightweight JSON state under `logs/`, which is already mounted by Docker Compose:
+
+- `logs/sr_monitor_state.json`: previous vote counts and Top 27 snapshot for maintenance-period reports.
+- `logs/sr_guard_state.json`: previous Top 27 snapshot and SR name cache for one-minute guard checks.
+
+If these files are deleted, the next run starts with a fresh baseline and recreates them after the first successful fetch.
 
 ### Key Features
 
@@ -43,7 +70,7 @@ Instead of a fixed interval, the tool queries `/wallet/getnextmaintenancetime` t
 The tool uses Go routines to fetch `account_name` for all 28 witnesses in parallel from the `/wallet/getaccount` interface, significantly reducing the collection time.
 
 #### Vote Change Tracking
-The tool maintains an in-memory snapshot of the previous period's votes. It calculates the `Change` for each SR:
+The tool persists a snapshot of the previous period's votes. It calculates the `Change` for each SR:
 ```text
 *1. Poloniex*
 Current: `3,228,089,488`  Change: `+89,488`
@@ -58,6 +85,13 @@ SR Replacement Detected:
 ```
 If no changes occur, it displays `Top 27 SRs remain unchanged.`
 
+#### Top 27 Early Warning Guard
+The `slack_sr_guard` process calls `/wallet/getpaginatednowwitnesslist` every minute with `limit=28`, keeps a persisted Top 27 snapshot, and sends a Slack alert only when an SR enters or leaves the Top 27 before the next maintenance period.
+
+The alert includes the entered SRs, left SRs, current 27th/28th vote gap, and a Top 28 vote table.
+
+When `PHONE_ALERT_URL` is configured, the same Top 27 change also triggers a phone call escalation, independent of the Slack alert (a failure to reach it is logged but never blocks or fails the Slack notification).
+
 ### Notifications
 
 This monitor only support java-tron node v4.8.1+, because of the API it used.

tools/slack_sr_monitor/docker-compose.yml

@@ -3,8 +3,25 @@ services:
     build: .
     container_name: slack-sr-monitor
     restart: unless-stopped
+    command: ["./slack_sr_monitor", "monitor"]
     environment:
       - SLACK_WEBHOOK=${SLACK_WEBHOOK}
+      - SLACK_SR_MONITOR_WEBHOOK=${SLACK_SR_MONITOR_WEBHOOK:-}
+      - TRON_NODES=${TRON_NODES:-}
       - TRON_NODE=${TRON_NODE:-https://api.trongrid.io}
     volumes:
       - ./logs:/home/monitor/logs
+
+  slack-sr-guard:
+    build: .
+    container_name: slack-sr-guard
+    restart: unless-stopped
+    command: ["./slack_sr_monitor", "guard"]
+    environment:
+      - SLACK_WEBHOOK=${SLACK_WEBHOOK}
+      - SLACK_SR_GUARD_WEBHOOK=${SLACK_SR_GUARD_WEBHOOK:-}
+      - TRON_NODES=${TRON_NODES:-}
+      - TRON_NODE=${TRON_NODE:-https://api.trongrid.io}
+      - PHONE_ALERT_URL=${PHONE_ALERT_URL:-}
+    volumes:
+      - ./logs:/home/monitor/logs