Repository navigation
fix(3scale_batcher): read batched report keys while holding the lock - #1614
Open
borisurbanik wants to merge 1 commit into
Open
borisurbanik wants to merge 1 commit into
borisurbanik wants to merge 1 commit into
Conversation
Backport of 79f35ae (PR 3scale#1611) to 3scale-2.16-stable, without the timer_task.lua pcall change. ReportsBatcher:get_all() listed the shared dict keys before acquiring the lock and then read them while holding it. There is one reporting timer per worker and they all flush the same service, so a second worker could list a key, lose the race for the lock, and find the key already deleted by the time it ran. storage:get() then returned nil and the comparison below raised "attempt to compare number with nil". The raised error left the lock to expire on its own, which made every request for that service wait the full 10s lock timeout in the access phase before dropping its report. It also aborted the timer thread before TimerTask could schedule the next run, so reporting stopped on that worker until the configuration was reloaded. Read the keys under the lock so deletes cannot interleave, and skip entries that are gone anyway (eviction can still remove a key even under the lock). timer_task.lua is intentionally not changed in this backport: the nil guard makes the throw unreachable, so the pcall defence is not needed on the stable branch where the bar for generic-module behaviour changes is higher. Fixes: THREESCALE-16640 Cherry-picked-from: 79f35ae
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Backport of 79f35ae (PR #1611) to 3scale-2.16-stable
timer_task.lua is intentionally not changed in this backport: the nil guard
makes the throw unreachable, so the pcall defence is not needed on the
stable branch where the bar for generic-module behaviour changes is higher.
JIRA: https://redhat.atlassian.net/browse/THREESCALE-16640