Repository navigation
Conversation
…itialization readPositioned copied chunkOffsets, chunkStreams and length under the BlockInputStream lock, and readChunkAt called the synchronized checkOpen() before each attempt. readWithStrategy keeps the same lock during the datanode RPC, so a positioned read waited for any sequential read on the same block. initialize() sets these fields before the volatile write of initialized and nothing changes them after that, so read them without the lock. Remove the checkOpen() calls from the positioned read path: MultipartInputStream, the only caller, checks its own volatile closed flag first. initialize() and refreshBlockInfo() stay synchronized.
readPositioned took the StreamBlockInputStream lock on each call: it called the synchronized updateBlockLength(), then ran checkOpen() and copied xceiverClientFactory in a synchronized block. The sequential read path keeps the same lock while it waits for data from the datanode, so a positioned read waited for any sequential read in progress on the same stream. - needUpdateBlockLength is now volatile. updateBlockLength() sets blockLength before it clears the flag, so readPositioned calls it only while the flag is set and then reads blockLength without the lock. - The constructor sets xceiverClientFactory, so make it final and let readPositioned use it without the lock. close() now sets a closed flag that only the lock guards, instead of setting the factory to null. - readPositioned does not check the flag: MultipartInputStream, its only caller, checks its own volatile closed flag before it calls the part.
|
@peterxcli is the benchmark that you used under version control? |
|
@taklwu PTAL! |
initialize() now publishes the block data, the block length, the chunk offsets and the chunk streams as one BlockChunks object in a volatile field. It replaces the initialized flag and the separate fields, so a positioned read takes one consistent snapshot without the lock, and final fields make sure that nothing changes after initialize(). This is the block and chunk part of the state split proposed in HDDS-16325. A block with no chunks still counts as not initialized, so the next read gets its block data again. A sequential read of such a block now returns EOF instead of throwing NullPointerException.
chihsuan
left a comment
There was a problem hiding this comment.
Thanks for the patch! @peterxcli This looks good overall. I left a few inline comments.
| private int readChunkAt(ChunkInputStream chunk, long chunkOffset, int readLength, ByteBuffer dst) throws IOException { | ||
| int preadRetries = 0; | ||
| while (true) { | ||
| checkOpen(); |
There was a problem hiding this comment.
Any reason to drop checkOpen() here instead of making it lock-free? On EC failover, a short-circuit pread now retries and sleeps on a closed stream. A volatile closed check before shouldRetryRead() would avoid that. What do you think?
| * Reuses each chunk's range reader, whose buffers and client references belong to the request. | ||
| * Takes this stream's lock only to initialize it or to refresh the block info after a read error, | ||
| * so it does not wait for a sequential read in progress. | ||
| * It does not check that this stream is open: {@link MultipartInputStream} checks that before it calls this method. |
There was a problem hiding this comment.
nit: Could this state the contract instead of naming a caller? EC calls this directly too.
| * It does not check that this stream is open: {@link MultipartInputStream} checks that before it calls this method. | |
| * It does not check that this stream is open, so callers must check that first. |
| * in progress. It does not check that this stream is open: {@link MultipartInputStream} checks that before it | ||
| * calls this method. |
There was a problem hiding this comment.
nit: Same here, could this state the contract instead of naming a caller?
| * in progress. It does not check that this stream is open: {@link MultipartInputStream} checks that before it | |
| * calls this method. | |
| * in progress. It does not check that this stream is open, so callers must check that first. |
| public synchronized void close() throws IOException { | ||
| releaseClient(); | ||
| xceiverClientFactory = null; | ||
| closed = true; |
There was a problem hiding this comment.
Could we add a test for this flag? With the factory now final, it is the only guard against reading after close(), and removing it keeps all current tests green.
There was a problem hiding this comment.
@peterxcli , thanks for working on this!
Please see the comments inlined and also https://issues.apache.org/jira/secure/attachment/13086376/11724_review.patch
- We should keep calling checkOpen() unless there are other reasons.
- I have quite a few refactoring comments which can be done here or sparately.
| @@ -172,27 +158,28 @@ public synchronized void initialize() throws IOException { | |||
| } | |||
| } while (shouldRetryRead(catchEx, retryPolicy, ++retries)); | |||
|
|
|||
| if (chunks == null) { | |||
| if (blockData == null) { | |||
| throw catchEx; | |||
| } else { | |||
| // Reset retry count if we get chunks successfully. | |||
| retries = 0; | |||
| } | |||
There was a problem hiding this comment.
Let's also add an getBlockDataWithRetries() method
checkOpen();
final BlockData blockData = getBlockDataWithRetries();
final long length = blockInfo.isUnderConstruction() ? blockData.getSize() : blockInfo.getLength();
// For each chunk in the block, create a ChunkInputStream and compute its chunkOffset
final List<ChunkInfo> chunks = blockData.getChunksList(); private BlockData getBlockDataWithRetries() throws IOException {
for (;;) {
final IOException exception;
try {
final BlockData data = getBlockData();
retries = 0;
return data;
// If we get a StorageContainerException or an IOException due to datanodes are unreachable,
// refresh pipeline and retry. Otherwise, retry according to the retry policy.
} catch (SCMSecurityException ex) {
throw ex;
} catch (StorageContainerException e) {
refreshBlockInfo(e);
exception = e;
} catch (IOException e) {
LOG.debug("Retry to get chunk info fail", e);
if (isConnectivityIssue(e)) {
refreshBlockInfo(e);
}
exception = e;
}
if (!shouldRetryRead(exception, retryPolicy, ++retries)) {
throw exception;
}
}
}| final List<ChunkInfo> chunks = blockData.getChunksList(); | ||
| final long[] chunkOffsets = new long[chunks.size()]; | ||
| long tempOffset = 0; | ||
|
|
||
| initialized = true; | ||
| final List<ChunkInputStream> chunkStreams = new ArrayList<>(chunks.size()); | ||
| for (int i = 0; i < chunks.size(); i++) { | ||
| chunkStreams.add(createChunkInputStream(chunks.get(i))); | ||
| chunkOffsets[i] = tempOffset; | ||
| tempOffset += chunks.get(i).getLen(); | ||
| } | ||
| blockChunks = new BlockChunks(blockData, length, chunkOffsets, chunkStreams); |
There was a problem hiding this comment.
Let's add a newBlockChunks(..) method:
private BlockChunks newBlockChunks(BlockData blockData, long length) {
final List<ChunkInfo> chunks = blockData.getChunksList();
final long[] chunkOffsets = new long[chunks.size()];
long tempOffset = 0;
// For each chunk, create a ChunkInputStream and compute its chunkOffset
final List<ChunkInputStream> chunkStreams = new ArrayList<>(chunks.size());
for (int i = 0; i < chunks.size(); i++) {
chunkStreams.add(createChunkInputStream(chunks.get(i)));
chunkOffsets[i] = tempOffset;
tempOffset += chunks.get(i).getLen();
}
return new BlockChunks(blockData, length, chunkOffsets, chunkStreams);
}| if (pos < 0 || pos > length) { | ||
| final BlockChunks chunks = blockChunks; | ||
| final long[] chunkOffsets = chunks.offsets; | ||
| final List<ChunkInputStream> chunkStreams = chunks.streams; | ||
| if (pos < 0 || pos > chunks.length) { |
There was a problem hiding this comment.
Let's add some methods to BlockChunks:
chunkIndex = chunks.searchChunkIndex(chunkIndex, pos);
chunks.resetPosition(chunkIndex, chunkIndexOfPrevPosition, pos);//BlockChunks
void resetPosition(int chunkIndex, int chunkIndexOfPrevPosition, long pos) throws IOException {
// Reset the previous chunkStream's position
streams.get(chunkIndexOfPrevPosition).resetPosition();
// Reset all the chunkStreams above the chunkIndex. We do this to reset
// any previous reads which might have updated the chunkPosition.
for (int index = chunkIndex + 1; index < streams.size(); index++) {
streams.get(index).seek(0);
}
// seek to the proper offset in the ChunkInputStream
streams.get(chunkIndex).seek(pos - offsets[chunkIndex]);
}
int searchChunkIndex(int chunkIndex, long pos) {
return searchStreamIndex(chunkIndex, pos, streams, offsets, ChunkInputStream::getLength);
}
}
static <T extends Seekable> int searchStreamIndex(int index, long pos, List<T> streams, long[] offsets,
ToLongFunction<T> getLength) {
if (index >= streams.size()) {
index = Arrays.binarySearch(offsets, pos);
} else if (pos < offsets[index]) {
index = Arrays.binarySearch(offsets, 0, index, pos);
} else if (pos >= offsets[index] + getLength.applyAsLong(streams.get(index))) {
index = Arrays.binarySearch(offsets, index + 1, streams.size(), pos);
}
if (index < 0) {
// Binary search returns -insertionPoint - 1 if element is not present in the array.
// Adjust the chunkIndex accordingly so that chunkIndex = insertionPoint - 1
index = -index - 2;
}
return index;
}There was a problem hiding this comment.
MultipartInputStream can also use the new method
+++ b/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/MultipartInputStream.java
@@ -145,23 +145,7 @@ public synchronized void seek(long pos) throws IOException {
}
// 1. Update the partIndex
- if (partIndex >= partStreams.size()) {
- partIndex = Arrays.binarySearch(partOffsets, pos);
- } else if (pos < partOffsets[partIndex]) {
- partIndex =
- Arrays.binarySearch(partOffsets, 0, partIndex, pos);
- } else if (pos >= partOffsets[partIndex] + partStreams
- .get(partIndex).getLength()) {
- partIndex = Arrays.binarySearch(partOffsets, partIndex + 1,
- partStreams.size(), pos);
- }
- if (partIndex < 0) {
- // Binary search returns -insertionPoint - 1 if element is not present
- // in the array. insertionPoint is the point at which element would be
- // inserted in the sorted array. We need to adjust the blockIndex
- // accordingly so that partIndex = insertionPoint - 1
- partIndex = -partIndex - 2;
- }
+ partIndex = BlockInputStream.searchStreamIndex(partIndex, pos, partStreams, partOffsets, PartInputStream::getLength);
| return blockData; | ||
| final BlockChunks chunks = blockChunks; | ||
| return chunks != null ? chunks.blockData : null; | ||
| } |
There was a problem hiding this comment.
getStreamBlockData() is called multiple times in KeyValueHandler.reconcileChunksPerBlock(..) except for tests. reconcileChunksPerBlock(..) also calls getChunkStreams(). We should refactor the code to get BlockChunks only
We should get BlockChunks once and keep using it in reconcileChunksPerBlock(..).
| updateBlockLength(); | ||
| // updateBlockLength() sets blockLength before it clears the volatile flag, so blockLength is safe to read here. | ||
| if (needUpdateBlockLength) { | ||
| updateBlockLength(); | ||
| } |
There was a problem hiding this comment.
- Just call updateBlockLength() as before -- The while (needUpdateBlockLength) loop will take care the if needUpdateBlockLength) here.
- Then, we don't have to add volatile to needUpdateBlockLength.
There was a problem hiding this comment.
public synchronized void updateBlockLength() throws IOException:
updateBlockLength is already synchronized, and calling it would get a lock blindly and weaken or lose the optimization here.
To simplify the code and drop the if , maybe we move the lock into the while loop inside updateBlockLength and remove synchronized from updateBlockLength?
There was a problem hiding this comment.
@amaliujia Good point! I missed that this is a double checking lock. Then, let's keep it here and add "double checking lock" to the comment:
// needUpdateBlockLength is volatile. Use a double checking lock to avoid synchronization | } | ||
| final XceiverClientFactory factory; | ||
| synchronized (this) { | ||
| checkOpen(); |
There was a problem hiding this comment.
We probably need call checkOpen() somewhere.
| private int readChunkAt(ChunkInputStream chunk, long chunkOffset, int readLength, ByteBuffer dst) throws IOException { | ||
| int preadRetries = 0; | ||
| while (true) { | ||
| checkOpen(); |
What changes were proposed in this pull request?
Problem
BlockInputStream.readPositionedandStreamBlockInputStream.readPositionedtake the stream lock on every call. The sequential read of each class holds the same lock while it waits for the datanode:BlockInputStream.readWithStrategyduring the ReadChunk RPC, andStreamBlockInputStream.readFullywhile it waits for streamed data. So a positioned read waits for any sequential read in progress on the same block.Changes
Both classes now follow one rule. After the one-time setup of a block stream, a positioned read takes no lock. The setup stays synchronized, and its last step is a volatile write: the
BlockChunksobject inBlockInputStream, and theneedUpdateBlockLengthflag inStreamBlockInputStream.BlockInputStream:initialize()publishes the block data, the block length, the chunk offsets and the chunk streams as one immutableBlockChunksobject in a volatile field. It replaces theinitializedflag and the separate fields, soreadPositionedreads one consistent snapshot without the lock, andfinalfields make sure that nothing changes afterinitialize(). This is the block and chunk part of the state split proposed in HDDS-16325.initialize()andrefreshBlockInfo()(used only after a read error) stay synchronized. A block with no chunks still counts as not initialized, so the next read gets its block data again. A sequential read of such a block now returns EOF instead of throwingNullPointerException.StreamBlockInputStream:needUpdateBlockLengthis volatile.readPositionedcalls the synchronizedupdateBlockLength()only while the flag is set, and then readsblockLengthwithout the lock.xceiverClientFactoryis final.close()sets aclosedflag guarded by the lock, andcheckOpen()tests that flag.readPositioned.MultipartInputStream.readPositioned, their only production caller, checks its own volatileclosedflag first. EC readers useseek()andread()instead.One trade-off: the block stream no longer stops a positioned read that races with
close(). On a short-circuit read, such a read retries under the read retry policy before it fails.What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-17399
How was this patch tested?
New unit tests:
TestBlockInputStream#testPositionedReadDoesNotWaitForSequentialReadandTestStreamBlockInputStream#testPositionedReadDoesNotWaitForSequentialRead: a positioned read completes while a sequential read on the same block holds the lock. The StreamBlock test also covers an under-construction block. Both tests time out without the change.TestBlockInputStream#testReadOnBlockWithNoChunks: positioned and sequential reads of a block with no chunks return EOF.TestMultipartInputStream#testPositionedReadAfterCloseFails: afterclose(), a positioned read fails and does not reach either block stream type.Benchmark:
ozone freon ocpron one host, 3 datanodes on one NVMe, warm page cache, a 256 MiB key (one block). All positioned-read threads share one stream with 1 sequential reader that reads 1 MiB at a time. Each run takes 30 s, after one 10 s warm-up run. Each value is the median of 3 runs. For example, the chunk row with 8 threads:The stream rows add
-D ozone.client.stream.readblock.enable=truebeforeocpr.ozone.client.stream.readblock.enable)ozone.client.stream.readblock.enable)