Quiet the per-task worker logging - #77
Merged
Merged
Conversation
Every task emitted four JSON records: the executor's start and success messages plus Django's task started and finished records. Formatting and flushing them to stdout cost a large share of the worker's throughput. configure_logging forced the root logger to INFO, so deployments could not run a worker below that level. Stop setting the level and log on whatever Django's LOGGING configuration provides; the executor's task start and success records stay at INFO. The test app now runs its workers at WARNING, like the celery and dramatiq benchmark runs, while its own task logs stay at INFO. A 20,000-task drain on this machine drops from 7.03s to 5.02s (352 us to 251 us per task). Fixes #68
Regenerate docs/images/backend-comparison-*.svg from a fresh benchmark run: threadmill 5,069 tasks/s, celery 2,128, django-tasks-db 1,997, django-tasks-redis 1,313 with one worker process and one thread. The README alt text carries the same numbers.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.