What happened:
GetBlobChunk returns FailedPrecondition with the error:
number (1): rpc error: code = FailedPrecondition desc = found multiple chunks with number (1)
found for record (...)
Two ChunkRef entities with the same Number exist in Datastore for the same BlobRef. FindChunkRefByNumber finds more than one and rejects the request.
What you expected to happen:
At most one ChunkRef per (BlobRef, Number) tuple exists. Re-uploading a chunk (e.g. after a network error) replaces the previous one. GetBlobChunk succeeds.
How to reproduce it (as minimally and precisely as possible):
- Create a chunked blob upload session.
- Send two concurrent
UploadChunk RPCs for the same session ID and the same chunk Number (e.g. Number = 1). A client retry under load reproduces this naturally.
- Both complete without error.
- Call
GetBlobChunk for that chunk number.
Root cause:
UploadChunk ran deduplication and insert as two separate, non-atomic steps:
deleteSameNumberChunks(ctx, chunk) // query + delete in separate TX
InsertChunkRef(ctx, blob, chunk) // insert in separate TX
Two concurrent uploads both execute deleteSameNumberChunks before either calls InsertChunkRef. Both queries return empty, both deletes are no-ops, both inserts succeed — leaving two ChunkRef rows with the same Number.
The comment in InsertChunkRef promised atomic deduplication ("If the current session has another chunk with the same Number, it will be marked for deletion") but the implementation did not do it.
Fix:
Move the dedup query and delete inside InsertChunkRef's own Datastore transaction, making the find-old → delete-old → insert-new sequence atomic. Return the superseded chunks to the caller for GCS object cleanup.
Environment:
- Open Saves version: v1.2.0
- Kubernetes version (use
kubectl version): 1.33.9-gke.1060000
- Cloud provider or hardware configuration: GKE
- Install method (yaml/helm): yaml
- Troubleshooting guide log(s):
- Others:
What happened:
GetBlobChunkreturnsFailedPreconditionwith the error:Two
ChunkRefentities with the sameNumberexist in Datastore for the sameBlobRef.FindChunkRefByNumberfinds more than one and rejects the request.What you expected to happen:
At most one
ChunkRefper(BlobRef, Number)tuple exists. Re-uploading a chunk (e.g. after a network error) replaces the previous one.GetBlobChunksucceeds.How to reproduce it (as minimally and precisely as possible):
UploadChunkRPCs for the same session ID and the same chunkNumber(e.g.Number = 1). A client retry under load reproduces this naturally.GetBlobChunkfor that chunk number.Root cause:
UploadChunkran deduplication and insert as two separate, non-atomic steps:deleteSameNumberChunks(ctx, chunk) // query + delete in separate TX
InsertChunkRef(ctx, blob, chunk) // insert in separate TX
Two concurrent uploads both execute
deleteSameNumberChunksbefore either callsInsertChunkRef. Both queries return empty, both deletes are no-ops, both inserts succeed — leaving twoChunkRefrows with the sameNumber.The comment in
InsertChunkRefpromised atomic deduplication ("If the current session has another chunk with the same Number, it will be marked for deletion") but the implementation did not do it.Fix:
Move the dedup query and delete inside
InsertChunkRef's own Datastore transaction, making the find-old → delete-old → insert-new sequence atomic. Return the superseded chunks to the caller for GCS object cleanup.Environment:
kubectl version): 1.33.9-gke.1060000