Skip to content

fix: surface real error cause and make round-robin summary visible in prod#4

Open
ttomalak-bkpr wants to merge 1 commit into
masterfrom
feat/improve-error-logging
Open

fix: surface real error cause and make round-robin summary visible in prod#4
ttomalak-bkpr wants to merge 1 commit into
masterfrom
feat/improve-error-logging

Conversation

@ttomalak-bkpr

Copy link
Copy Markdown
Contributor

Summary

  • GitHubRestStream.get_records now logs the real exception (type + message) before re-raising. singer_sdk's own catch-all only logs "an unhandled error occurred while syncing '<stream>'" with zero detail, so prod runs currently give no way to tell why a stream died (rate limit, auth, transient API error, ...). FatalAPIError/RetriableAPIError already carry a rich message (HTTP status, GitHub-Request-Id, etc.) - this just makes sure it isn't lost.
  • The round-robin checkpoint's per-run summary (RepositoryStream._get_records_round_robin) was logged at info, but prod actually runs at warning - so despite already being deployed, this summary has never appeared in production logs. Bumped both the normal summary and the "0 repos returned" early-return to warning, so enumeration progress (repos processed, full lap or not, next resume point) is finally visible without any deploy-side log-level change.

This is scoped entirely to this repo - no changes to the bi-meltano launcher/image that runs it.

Test plan

  • ruff check / ruff format --check on both changed files - clean (pre-existing unrelated line-length issues on repository_streams.py left untouched, confirmed present on master too)
  • mypy on both changed files - clean
  • Full existing test suite (pytest tests/) - same 72 passed / 10 pre-existing network-dependent failures as on master, no regressions

… prod

GitHubRestStream.get_records now logs the actual exception (type + message)
before re-raising - singer_sdk's own catch-all only logs "an unhandled error
occurred while syncing '<stream>'" with no detail, which left prod runs with
no way to tell why a stream died (rate limit, auth, etc).

The round-robin checkpoint's per-run summary was logged at info, but prod
runs at warning - so it has never actually appeared in production logs since
it was added. Bumped to warning so enumeration progress is visible without
needing a log-level change.
@ttomalak-bkpr
ttomalak-bkpr requested review from a team and lumautomation as code owners July 24, 2026 15:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant