Skip to content

HDDS-17399. Make positioned reads lock-free after initialization in BlockInputStream and StreamBlockInputStream - #11724

Open
peterxcli wants to merge 9 commits into
apache:masterfrom
peterxcli:HDDS-17399
Open

peterxcli wants to merge 9 commits into
apache:masterfrom
peterxcli:HDDS-17399

Conversation

@peterxcli

@peterxcli peterxcli commented Oct 23, 2026 •

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

Problem

BlockInputStream.readPositioned and StreamBlockInputStream.readPositioned take the stream lock on every call. The sequential read of each class holds the same lock while it waits for the datanode: BlockInputStream.readWithStrategy during the ReadChunk RPC, and StreamBlockInputStream.readFully while it waits for streamed data. So a positioned read waits for any sequential read in progress on the same block.

Changes

Both classes now follow one rule. After the one-time setup of a block stream, a positioned read takes no lock. The setup stays synchronized, and its last step is a volatile write: the BlockChunks object in BlockInputStream, and the needUpdateBlockLength flag in StreamBlockInputStream.

  • BlockInputStream: initialize() publishes the block data, the block length, the chunk offsets and the chunk streams as one immutable BlockChunks object in a volatile field. It replaces the initialized flag and the separate fields, so readPositioned reads one consistent snapshot without the lock, and final fields make sure that nothing changes after initialize(). This is the block and chunk part of the state split proposed in HDDS-16325. initialize() and refreshBlockInfo() (used only after a read error) stay synchronized. A block with no chunks still counts as not initialized, so the next read gets its block data again. A sequential read of such a block now returns EOF instead of throwing NullPointerException.
  • StreamBlockInputStream: needUpdateBlockLength is volatile. readPositioned calls the synchronized updateBlockLength() only while the flag is set, and then reads blockLength without the lock. xceiverClientFactory is final. close() sets a closed flag guarded by the lock, and checkOpen() tests that flag.
  • Neither block stream checks for close in readPositioned. MultipartInputStream.readPositioned, their only production caller, checks its own volatile closed flag first. EC readers use seek() and read() instead.
  • Sequential reads do not change.

One trade-off: the block stream no longer stops a positioned read that races with close(). On a short-circuit read, such a read retries under the read retry policy before it fails.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-17399

How was this patch tested?

New unit tests:

  • TestBlockInputStream#testPositionedReadDoesNotWaitForSequentialRead and TestStreamBlockInputStream#testPositionedReadDoesNotWaitForSequentialRead: a positioned read completes while a sequential read on the same block holds the lock. The StreamBlock test also covers an under-construction block. Both tests time out without the change.
  • TestBlockInputStream#testReadOnBlockWithNoChunks: positioned and sequential reads of a block with no chunks return EOF.
  • TestMultipartInputStream#testPositionedReadAfterCloseFails: after close(), a positioned read fails and does not reach either block stream type.

Benchmark: ozone freon ocpr on one host, 3 datanodes on one NVMe, warm page cache, a 256 MiB key (one block). All positioned-read threads share one stream with 1 sequential reader that reads 1 MiB at a time. Each run takes 30 s, after one 10 s warm-up run. Each value is the median of 3 runs. For example, the chunk row with 8 threads:

ozone freon ocpr -v vol1 -b bucket1 -k key1 -t 8 --read-size 65536 --sequential-readers 1 --duration 30s

The stream rows add -D ozone.client.stream.readblock.enable=true before ocpr.

Positioned reads per second with a sequential reader on the same stream, master vs this PR, for chunk and stream reads with 1 and 8 threads

Read path Positioned reads Before (ops/s, p50, p99) After (ops/s, p50, p99)
Chunk (default) 1 thread × 64 KiB 46, 5.1 ms, 552 ms 1,370, 0.50 ms, 1.8 ms
Chunk (default) 8 threads × 64 KiB 74, 44 ms, 1.38 s 6,440, 0.94 ms, 2.3 ms
Stream (ozone.client.stream.readblock.enable) 1 thread × 64 KiB 13, 21 ms, 446 ms 340, 2.4 ms, 8.5 ms
Stream (ozone.client.stream.readblock.enable) 8 threads × 64 KiB 35, 61 ms, 1.69 s 1,763, 4.0 ms, 10 ms
  • With positioned reads only (1 to 32 threads), throughput stays within run-to-run noise.
  • On the stream path, the sequential reader gets about 4% less bandwidth next to 8 positioned-read threads, and the total bytes read rise by 8-15%. All reads from one client to a datanode share one gRPC connection, which one client event-loop thread serves. With the positioned reads no longer waiting, that thread goes from about 70% to 95% busy. With the same positioned reads from a separate process, the sequential reader keeps its speed.

…itialization

readPositioned copied chunkOffsets, chunkStreams and length under the
BlockInputStream lock, and readChunkAt called the synchronized checkOpen()
before each attempt. readWithStrategy keeps the same lock during the
datanode RPC, so a positioned read waited for any sequential read on the
same block.

initialize() sets these fields before the volatile write of initialized
and nothing changes them after that, so read them without the lock.
Remove the checkOpen() calls from the positioned read path:
MultipartInputStream, the only caller, checks its own volatile closed flag
first. initialize() and refreshBlockInfo() stay synchronized.
readPositioned took the StreamBlockInputStream lock on each call: it
called the synchronized updateBlockLength(), then ran checkOpen() and
copied xceiverClientFactory in a synchronized block. The sequential read
path keeps the same lock while it waits for data from the datanode, so a
positioned read waited for any sequential read in progress on the same
stream.

- needUpdateBlockLength is now volatile. updateBlockLength() sets
  blockLength before it clears the flag, so readPositioned calls it only
  while the flag is set and then reads blockLength without the lock.
- The constructor sets xceiverClientFactory, so make it final and let
  readPositioned use it without the lock. close() now sets a closed flag
  that only the lock guards, instead of setting the factory to null.
- readPositioned does not check the flag: MultipartInputStream, its only
  caller, checks its own volatile closed flag before it calls the part.
Copilot AI balanced review requested due to automatic review settings October 23, 2026 19:11

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

@yandrey321

Copy link
Copy Markdown
Contributor

@peterxcli is the benchmark that you used under version control?

@jojochuang

Copy link
Copy Markdown
Contributor

@taklwu PTAL!

initialize() now publishes the block data, the block length, the chunk
offsets and the chunk streams as one BlockChunks object in a volatile
field. It replaces the initialized flag and the separate fields, so a
positioned read takes one consistent snapshot without the lock, and final
fields make sure that nothing changes after initialize(). This is the
block and chunk part of the state split proposed in HDDS-16325.

A block with no chunks still counts as not initialized, so the next read
gets its block data again. A sequential read of such a block now returns
EOF instead of throwing NullPointerException.
@peterxcli
peterxcli marked this pull request as ready for review November 5, 2026 08:10

@chihsuan chihsuan left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the patch! @peterxcli This looks good overall. I left a few inline comments.

private int readChunkAt(ChunkInputStream chunk, long chunkOffset, int readLength, ByteBuffer dst) throws IOException {
int preadRetries = 0;
while (true) {
checkOpen();

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any reason to drop checkOpen() here instead of making it lock-free? On EC failover, a short-circuit pread now retries and sleeps on a closed stream. A volatile closed check before shouldRetryRead() would avoid that. What do you think?

* Reuses each chunk's range reader, whose buffers and client references belong to the request.
* Takes this stream's lock only to initialize it or to refresh the block info after a read error,
* so it does not wait for a sequential read in progress.
* It does not check that this stream is open: {@link MultipartInputStream} checks that before it calls this method.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Could this state the contract instead of naming a caller? EC calls this directly too.

Suggested change
* It does not check that this stream is open: {@link MultipartInputStream} checks that before it calls this method.
* It does not check that this stream is open, so callers must check that first.

Comment on lines +190 to +191
* in progress. It does not check that this stream is open: {@link MultipartInputStream} checks that before it
* calls this method.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Same here, could this state the contract instead of naming a caller?

Suggested change
* in progress. It does not check that this stream is open: {@link MultipartInputStream} checks that before it
* calls this method.
* in progress. It does not check that this stream is open, so callers must check that first.

public synchronized void close() throws IOException {
releaseClient();
xceiverClientFactory = null;
closed = true;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we add a test for this flag? With the factory now final, it is the only guard against reading after close(), and removing it keeps all current tests green.

@szetszwo szetszwo left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@peterxcli , thanks for working on this!

Please see the comments inlined and also https://issues.apache.org/jira/secure/attachment/13086376/11724_review.patch

  • We should keep calling checkOpen() unless there are other reasons.
  • I have quite a few refactoring comments which can be done here or sparately.

Comment on lines 131 to 166
@@ -172,27 +158,28 @@ public synchronized void initialize() throws IOException {
}
} while (shouldRetryRead(catchEx, retryPolicy, ++retries));

if (chunks == null) {
if (blockData == null) {
throw catchEx;
} else {
// Reset retry count if we get chunks successfully.
retries = 0;
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's also add an getBlockDataWithRetries() method

    checkOpen();

    final BlockData blockData = getBlockDataWithRetries();
    final long length = blockInfo.isUnderConstruction() ? blockData.getSize() : blockInfo.getLength();

    // For each chunk in the block, create a ChunkInputStream and compute its chunkOffset
    final List<ChunkInfo> chunks = blockData.getChunksList();
  private BlockData getBlockDataWithRetries() throws IOException {
    for (;;) {
      final IOException exception;
      try {
        final BlockData data = getBlockData();
        retries = 0;
        return data;

        // If we get a StorageContainerException or an IOException due to datanodes are unreachable,
        // refresh pipeline and retry. Otherwise, retry according to the retry policy.
      } catch (SCMSecurityException ex) {
        throw ex;
      } catch (StorageContainerException e) {
        refreshBlockInfo(e);
        exception = e;
      } catch (IOException e) {
        LOG.debug("Retry to get chunk info fail", e);
        if (isConnectivityIssue(e)) {
          refreshBlockInfo(e);
        }
        exception = e;
      }

      if (!shouldRetryRead(exception, retryPolicy, ++retries)) {
        throw exception;
      }
    }
  }

Comment on lines +170 to +180
final List<ChunkInfo> chunks = blockData.getChunksList();
final long[] chunkOffsets = new long[chunks.size()];
long tempOffset = 0;

initialized = true;
final List<ChunkInputStream> chunkStreams = new ArrayList<>(chunks.size());
for (int i = 0; i < chunks.size(); i++) {
chunkStreams.add(createChunkInputStream(chunks.get(i)));
chunkOffsets[i] = tempOffset;
tempOffset += chunks.get(i).getLen();
}
blockChunks = new BlockChunks(blockData, length, chunkOffsets, chunkStreams);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's add a newBlockChunks(..) method:

  private BlockChunks newBlockChunks(BlockData blockData, long length) {
    final List<ChunkInfo> chunks = blockData.getChunksList();
    final long[] chunkOffsets = new long[chunks.size()];
    long tempOffset = 0;

    // For each chunk, create a ChunkInputStream and compute its chunkOffset
    final List<ChunkInputStream> chunkStreams = new ArrayList<>(chunks.size());
    for (int i = 0; i < chunks.size(); i++) {
      chunkStreams.add(createChunkInputStream(chunks.get(i)));
      chunkOffsets[i] = tempOffset;
      tempOffset += chunks.get(i).getLen();
    }
    return new BlockChunks(blockData, length, chunkOffsets, chunkStreams);
  }

Comment on lines -531 to +520
if (pos < 0 || pos > length) {
final BlockChunks chunks = blockChunks;
final long[] chunkOffsets = chunks.offsets;
final List<ChunkInputStream> chunkStreams = chunks.streams;
if (pos < 0 || pos > chunks.length) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's add some methods to BlockChunks:

    chunkIndex = chunks.searchChunkIndex(chunkIndex, pos);
    chunks.resetPosition(chunkIndex, chunkIndexOfPrevPosition, pos);
//BlockChunks
    void resetPosition(int chunkIndex, int chunkIndexOfPrevPosition, long pos) throws IOException {
      // Reset the previous chunkStream's position
      streams.get(chunkIndexOfPrevPosition).resetPosition();

      // Reset all the chunkStreams above the chunkIndex. We do this to reset
      // any previous reads which might have updated the chunkPosition.
      for (int index =  chunkIndex + 1; index < streams.size(); index++) {
        streams.get(index).seek(0);
      }
      // seek to the proper offset in the ChunkInputStream
      streams.get(chunkIndex).seek(pos - offsets[chunkIndex]);
    }

    int searchChunkIndex(int chunkIndex, long pos) {
      return searchStreamIndex(chunkIndex, pos, streams, offsets, ChunkInputStream::getLength);
    }
  }

  static <T extends Seekable> int searchStreamIndex(int index, long pos, List<T> streams, long[] offsets,
      ToLongFunction<T> getLength) {
    if (index >= streams.size()) {
      index = Arrays.binarySearch(offsets, pos);
    } else if (pos < offsets[index]) {
      index = Arrays.binarySearch(offsets, 0, index, pos);
    } else if (pos >= offsets[index] + getLength.applyAsLong(streams.get(index))) {
      index = Arrays.binarySearch(offsets, index + 1, streams.size(), pos);
    }
    if (index < 0) {
      // Binary search returns -insertionPoint - 1  if element is not present in the array.
      // Adjust the chunkIndex accordingly so that chunkIndex = insertionPoint - 1
      index = -index - 2;
    }
    return index;
  }

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MultipartInputStream can also use the new method

+++ b/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/MultipartInputStream.java
@@ -145,23 +145,7 @@ public synchronized void seek(long pos) throws IOException {
     }
 
     // 1. Update the partIndex
-    if (partIndex >= partStreams.size()) {
-      partIndex = Arrays.binarySearch(partOffsets, pos);
-    } else if (pos < partOffsets[partIndex]) {
-      partIndex =
-          Arrays.binarySearch(partOffsets, 0, partIndex, pos);
-    } else if (pos >= partOffsets[partIndex] + partStreams
-        .get(partIndex).getLength()) {
-      partIndex = Arrays.binarySearch(partOffsets, partIndex + 1,
-          partStreams.size(), pos);
-    }
-    if (partIndex < 0) {
-      // Binary search returns -insertionPoint - 1  if element is not present
-      // in the array. insertionPoint is the point at which element would be
-      // inserted in the sorted array. We need to adjust the blockIndex
-      // accordingly so that partIndex = insertionPoint - 1
-      partIndex = -partIndex - 2;
-    }
+    partIndex = BlockInputStream.searchStreamIndex(partIndex, pos, partStreams, partOffsets, PartInputStream::getLength);
 

return blockData;
final BlockChunks chunks = blockChunks;
return chunks != null ? chunks.blockData : null;
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

getStreamBlockData() is called multiple times in KeyValueHandler.reconcileChunksPerBlock(..) except for tests. reconcileChunksPerBlock(..) also calls getChunkStreams(). We should refactor the code to get BlockChunks only

We should get BlockChunks once and keep using it in reconcileChunksPerBlock(..).

Comment on lines -198 to +205
updateBlockLength();
// updateBlockLength() sets blockLength before it clears the volatile flag, so blockLength is safe to read here.
if (needUpdateBlockLength) {
updateBlockLength();
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • Just call updateBlockLength() as before -- The while (needUpdateBlockLength) loop will take care the if needUpdateBlockLength) here.
  • Then, we don't have to add volatile to needUpdateBlockLength.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

public synchronized void updateBlockLength() throws IOException:

updateBlockLength is already synchronized, and calling it would get a lock blindly and weaken or lose the optimization here.

To simplify the code and drop the if , maybe we move the lock into the while loop inside updateBlockLength and remove synchronized from updateBlockLength?

@szetszwo szetszwo Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@amaliujia Good point! I missed that this is a double checking lock. Then, let's keep it here and add "double checking lock" to the comment:

// needUpdateBlockLength is volatile. Use a double checking lock to avoid synchronization 

}
final XceiverClientFactory factory;
synchronized (this) {
checkOpen();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We probably need call checkOpen() somewhere.

private int readChunkAt(ChunkInputStream chunk, long chunkOffset, int readLength, ByteBuffer dst) throws IOException {
int preadRetries = 0;
while (true) {
checkOpen();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why removing checkOpen()?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants