docs: correct dynamic batching visualization and timings#153
Open
Lazytoucan wants to merge 1 commit into
Open
Conversation
Updated diagram PNGs to accurately reflect dynamic batching behavior. Inference triggers immediately upon reaching 'max_batch_size', rather then waiting for the timeout Ajusted launching time in the README to match the corrected visual flow.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR corrects the visualization and explanation of the Dynamic Batching behavior in the tutorial.
Previously, the diagrams and text incorrectly implied that the batcher waits for the full delay timeout even if max_batch_size has been reached. I have updated the visual diagrams to accurately show that inference execution triggers earlier — immediately when the queue reaches max_batch_size.
The text and specific timing examples in the README have been adjusted to seamlessly match the new diagrams.
Changes made
Visual Changes
1. Dynamic Batching scheme (
dynamic_batching.png)2. Multi-Instance scheme (
multi_instance.png)Type of change