Skip to content

fix(arrow-avro): correctly read blocks by async reader when sync marker is cut by requested range - #11751

Open
mzabaluev wants to merge 4 commits into
apache:mainfrom
mzabaluev:mikhail/avro-split-read-sync-marker
Open

mzabaluev wants to merge 4 commits into
apache:mainfrom
mzabaluev:mikhail/avro-split-read-sync-marker

Conversation

@mzabaluev

Copy link
Copy Markdown
Contributor

Rationale for this change

The async_reader implementation has a bug where, if the requested range ends in the middle of a sync marker, the reader does not keep reading the block after the sync marker. If the range starts in the middle of a sync marker, the following block also gets skipped. Combined, this means that if a file is split into ranges so that the second range starts where the first range ends, and that split point happens to be inside a sync marker, the following block gets skipped entirely.

What changes are included in this PR?

  • Marker cut by the range end: if no whole sync marker is found, the reader fetches up to 15 more bytes past the requested end. It searches them together with the last 15 bytes it already had, so any marker it finds there must have started inside the range.
  • Where a range stops: a new requested_end field keeps the original end while range.end grows with each extra fetch. After each block, the reader works out where the following sync marker starts and stops if that is at or past requested_end.
  • Builder: a range that ends exactly at the header's end no longer finishes immediately, because the header's own sync marker starts inside it. The doc comments are the shortened versions from the last commit.
  • Bug fix for small blocks: the DecodeVLQHeader path threw away any bytes after the block data, including the sync marker, whenever the 20-byte header fetch held the whole block. Blocks with less than about 20 bytes of data would then fail. The *every_split_point* test, which uses 1-row blocks, would hit this. Instead, header fetches go through normal decoding, and with it finishing_partial_block. Every extra fetch moves range.end forward, so the loop still terminates.

Are these changes tested?

Added tests (created by Miha, modified by Claude) exercising cases when the read range splits a sync marker.

Are there any significant user-facing changes?

No.

Adjust the read logic so the block belongs to the range
where its sync marker starts.

Add test with split ranges that miss a block.
@github-actions github-actions Bot added the arrow-avro arrow-avro crate label Oct 20, 2026
@mzabaluev

Copy link
Copy Markdown
Contributor Author

Note to self: update #11392 once this is PR done and merged.

(self.range.end + MAX_VLQ_HEADER_SIZE).min(self.file_size);

if fetch_end == self.range.end {
if matches!(self.block_decoder.state(), BlockDecoderState::Size) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

here we narrow to only size state, instead of before where either count or size state would cause EOF; is this intentional?

here's an example test mocked by Codex which fails on this PR:

    #[tokio::test]
    async fn test_truncated_block_count_errors() {
        use crate::writer::AvroWriter;

        let schema = Schema::new(vec![Field::new("id", DataType::Int32, false)]);
        let batch = RecordBatch::try_new(
            Arc::new(schema.clone()),
            vec![Arc::new(Int32Array::from(vec![1]))],
        )
        .unwrap();
        let mut bytes = Vec::new();
        let mut writer = AvroWriter::new(&mut bytes, schema).unwrap();
        writer.write(&batch).unwrap();
        writer.finish().unwrap();

        // Start the next block's count varint without its terminating byte.
        bytes.push(0x80);
        let file_size = bytes.len() as u64;
        let reader = AsyncAvroFileReader::builder(std::io::Cursor::new(bytes), file_size, 1024)
            .try_build()
            .await
            .unwrap();
        let result: Result<Vec<RecordBatch>, _> = reader.try_collect().await;
        assert!(
            result.is_err(),
            "accepted truncated block count: {result:?}"
        );
    }

it seems to suggest an EOF error should be occurring, but with changes here this isnt happening

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you, fixed in ed627cd

@Rich-T-kid Rich-T-kid added the bug label Nov 16, 2026
mzabaluev and others added 3 commits November 18, 2026 00:01
A file ending partway through a block count varint was treated as a
clean end after the last sync marker. Only end cleanly when no bytes
of the next block have been consumed.

Co-Authored-By: Jeffrey Vo <jeffreyvo@apache.org>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

arrow-avro arrow-avro crate bug

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants