fix(worker): bound dispatcher tracing spans on 0.6 - #451
Merged
Conversation
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Backport companion to the dispatcher fix on this release line. `HeartbeatService::run` has the same shape as the poll loop: instrumented while owning a loop that lives as long as the worker, so every lease heartbeat and progress flush accumulated under one never-closing root. Left alone, #449 would recur on 0.6 at the heartbeat cadence. Removing the dispatcher's `run` span also dropped the `queue` field that its PG-listener `error!`/`warn!` events inherited, leaving listener failures unattributable on a multi-queue worker. Those events now carry `queue` explicitly. No regression test here: this line predates the in-memory span-exporter harness (ADR-039, 0.7) that the main-line test uses. The same code change is covered there. Refs #449. Main-line change: #450. Claude-Session: https://claude.ai/code/session_01Y5QbKR6tBzmn5u74U4Fxh2
Poll-only is degraded but fully functional, and the sibling LISTEN failure already warned for the same fallback. A transaction-mode pooler makes this the expected path (#374), so error-level would page on a supported deployment. Refs #449. Main-line change: #450. Claude-Session: https://claude.ai/code/session_01Y5QbKR6tBzmn5u74U4Fxh2
hardbyte
added
the
full-ci
Claude
hardbyte
marked this pull request as ready for review
July 28, 2026 10:27
Empty commit to re-run CI with the `full-ci` label set, so the worker test jobs run against this backport rather than being skipped. Claude-Session: https://claude.ai/code/session_01Y5QbKR6tBzmn5u74U4Fxh2
Documents the operational consequence for operators on this line: many
short traces instead of one unbounded one, ~5 poll traces/s per idle
queue-claimer at the default poll interval.
Records why the 0.7 span renaming is not backported: this line has no
ADR-039 producer-side messaging conventions for a `receive {queue}` span
to pair with, and a patch release should not move telemetry names.
No code change.
Refs #449.
Claude-Session: https://claude.ai/code/session_01Y5QbKR6tBzmn5u74U4Fxh2
Backports the level gating from #450. Without it this branch fixes one telemetry pathology and ships another: an idle dispatcher would open ~5 poll traces/s per queue-claimer, all recording "claimed nothing", where 0.6.4 had one growing trace. For a per-trace-priced backend that is a new problem arriving in a patch. Safe on this line because the spans it hides were largely not reaching users anyway — on the fleets that hit #449 the trace containing them was being rejected outright. Empty polls remain visible through awa.dispatch.empty_claim and the claim-duration histogram, which are metrics and unaffected by span filtering. Span renaming and the messaging.destination.name dual-emit stay on 0.7 only: this line has no ADR-039 producer-side conventions for them to pair with, and a patch release should not move or grow telemetry names. Refs #449, #455. Claude-Session: https://claude.ai/code/session_01Y5QbKR6tBzmn5u74U4Fxh2
Drops the warn-vs-error explanation entirely — the sibling LISTEN fallback three lines below already warns, so the code is self-consistent, and the reasoning belongs in the changelog and #374. Span comments reduce to a pointer at the issues. Claude-Session: https://claude.ai/code/session_01Y5QbKR6tBzmn5u74U4Fxh2
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backport of #450 to the release branch carrying 0.6.4.
What changed
#[tracing::instrument]fromDispatcher::runandHeartbeatService::runpoll_once/heartbeat_oncetick an explicit trace root, keeping the queue fieldqueueexplicitly on the dispatcher's PG-listener events;warnrather thanerroron the listener-create fallbackNo schema, migration, or API change.
Why
Production users on 0.6.4 otherwise produce dispatcher and heartbeat traces that grow for the lifetime of the worker: both
runmethods own loops that run until shutdown, so their spans never closed and every poll, claim, lease heartbeat, and progress flush accumulated under one root. Traces passed backend size limits (Tempo rejects at 5 MB withTRACE_TOO_LARGE) and the export pressure surfaced asBatchSpanProcessortimeouts. This gives that a patch-release path without adopting the 0.7 alpha line.HeartbeatService::runhas the same shape as the poll loop and is included for the same reason as on main — without it the issue recurs at the heartbeat cadence.Base branch
release/0.6.3is deliberate: it is the branch that contains tagv0.6.4(the tag is its tip).release/0.6.xis stale at 0.6.2.Differences from #450
No regression test. This line predates the ADR-039 in-memory span-exporter harness that the main-line test uses; the identical code change is covered there.
Validation
Against PostgreSQL 16, on a database separate from the 0.7 test runs:
cargo fmt --all --check;cargo clippy -p awa-worker --all-features --all-targets -- -D warningscargo test -p awa --test observability_test— 3 passedAddresses #449. Main-line change: #450 — land that first.
https://claude.ai/code/session_01Y5QbKR6tBzmn5u74U4Fxh2