fix: throttle upstream RPC-provider error reporting to stop log amplification - #1453
Closed
tony8713 wants to merge 1 commit into
Closed
fix: throttle upstream RPC-provider error reporting to stop log amplification#1453tony8713 wants to merge 1 commit into
tony8713 wants to merge 1 commit into
Conversation
…fication When rpc.snapshot.org (the shared RPC proxy) returns HTTP 403/5xx, ethers throws a SERVER_ERROR/CALL_EXCEPTION for every failing strategy call. score-api was calling capture() (Sentry) and console.log once per failed request. During a provider outage that is thousands of failures per minute, each serializing the full ethers error object, which stalls the event loop and makes score-api miss the HTTP header timeout (the score.snapshot.org flapping). Add helpers/providerErrors with isProviderError(), summarizeError(), and a time-windowed shouldReport() throttle, and apply it in both error paths in rpc.ts so a provider outage produces a sampled trickle of reports instead of a self-inflicted flood. Behaviour for non-provider errors is unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
tony8713
force-pushed
the
fix/rpc-error-log-amplification
branch
from
August 16, 2026 03:15
a487824 to
90c4a10
Compare
Contributor
|
Closing: this approach isn't the right fit. The underlying problem is tracked in #1470 for a different fix. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
When the shared RPC proxy behind
rpc.snapshot.orgreturns 403s or 5xx, every affected strategy call throws, and score-api writes a log line per failed request. That line stringifies the full strategies payload along with the error, so a brief upstream fault turns into sustained log volume that blocks the event loop. score-api then stops answering inside the HTTP header timeout and the uptime check opens an incident, which clears as soon as the burst does.The upstream fault is short. The amplification is ours, and it is what turns the fault into an outage.
Fix
New
src/helpers/providerErrors.ts:isProviderErrorseparates upstream RPC failures from genuine input and validation errors.summarizeErrorbuilds a bounded single-line summary, so the strategies payload and the full error object are no longer stringified per request.shouldReportallows one report per key per time window, keyed on method or network plus the error code.src/rpc.tsapplies these on both error paths. An upstream-provider error gets the bounded, throttled line. Everything else behaves exactly as before, and the 500 sent to the client is unchanged in all cases.The Sentry capture is throttled on the same key, which makes the captures that survive a storm a predictable one per key per window rather than whichever ones happened to arrive first.
Non-goals
The 403 itself is upstream and is being handled separately. Per-strategy provider timeouts are left alone, since tightening them would change scoring for slow but valid strategies and deserves its own change.
🤖 Generated with Claude Code