Repository navigation
fix(arrow-avro): correctly read blocks by async reader when sync marker is cut by requested range - #11751
Open
mzabaluev wants to merge 4 commits into
Open
fix(arrow-avro): correctly read blocks by async reader when sync marker is cut by requested range#11751mzabaluev wants to merge 4 commits into
mzabaluev wants to merge 4 commits into
Conversation
Adjust the read logic so the block belongs to the range where its sync marker starts. Add test with split ranges that miss a block.
Contributor
Author
|
Note to self: update #11392 once this is PR done and merged. |
Jefffrey
reviewed
Nov 5, 2026
| (self.range.end + MAX_VLQ_HEADER_SIZE).min(self.file_size); | ||
|
|
||
| if fetch_end == self.range.end { | ||
| if matches!(self.block_decoder.state(), BlockDecoderState::Size) { |
Contributor
There was a problem hiding this comment.
here we narrow to only size state, instead of before where either count or size state would cause EOF; is this intentional?
here's an example test mocked by Codex which fails on this PR:
#[tokio::test]
async fn test_truncated_block_count_errors() {
use crate::writer::AvroWriter;
let schema = Schema::new(vec![Field::new("id", DataType::Int32, false)]);
let batch = RecordBatch::try_new(
Arc::new(schema.clone()),
vec![Arc::new(Int32Array::from(vec![1]))],
)
.unwrap();
let mut bytes = Vec::new();
let mut writer = AvroWriter::new(&mut bytes, schema).unwrap();
writer.write(&batch).unwrap();
writer.finish().unwrap();
// Start the next block's count varint without its terminating byte.
bytes.push(0x80);
let file_size = bytes.len() as u64;
let reader = AsyncAvroFileReader::builder(std::io::Cursor::new(bytes), file_size, 1024)
.try_build()
.await
.unwrap();
let result: Result<Vec<RecordBatch>, _> = reader.try_collect().await;
assert!(
result.is_err(),
"accepted truncated block count: {result:?}"
);
}it seems to suggest an EOF error should be occurring, but with changes here this isnt happening
A file ending partway through a block count varint was treated as a clean end after the last sync marker. Only end cleanly when no bytes of the next block have been consumed. Co-Authored-By: Jeffrey Vo <jeffreyvo@apache.org> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
The
async_readerimplementation has a bug where, if the requested range ends in the middle of a sync marker, the reader does not keep reading the block after the sync marker. If the range starts in the middle of a sync marker, the following block also gets skipped. Combined, this means that if a file is split into ranges so that the second range starts where the first range ends, and that split point happens to be inside a sync marker, the following block gets skipped entirely.What changes are included in this PR?
requested_endfield keeps the original end whilerange.endgrows with each extra fetch. After each block, the reader works out where the following sync marker starts and stops if that is at or pastrequested_end.DecodeVLQHeaderpath threw away any bytes after the block data, including the sync marker, whenever the 20-byte header fetch held the whole block. Blocks with less than about 20 bytes of data would then fail. The*every_split_point*test, which uses 1-row blocks, would hit this. Instead, header fetches go through normal decoding, and with itfinishing_partial_block. Every extra fetch movesrange.endforward, so the loop still terminates.Are these changes tested?
Added tests (created by Miha, modified by Claude) exercising cases when the read range splits a sync marker.
Are there any significant user-facing changes?
No.