Repository navigation
HDDS-17061. Recon drops the replicas of a DELETED container when it moves it back to SCM's live state. - #12096
Draft
ArafatKhan2198 wants to merge 1 commit into
Draft
ArafatKhan2198 wants to merge 1 commit into
ArafatKhan2198 wants to merge 1 commit into
Conversation
…oves it back to SCM's live state.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
When Recon brings a deleted container back to match SCM, it deletes its own record and adds a new one. Deleting the record also deletes the list of datanodes that hold copies. Recon then treats the container as missing even though the data is still there. SCM makes the same move in place and keeps that list. This change makes Recon do the same.
What goes wrong
rebuildContainerFromScmdeletes Recon's record and adds a new one from SCM's container info./api/v1/containers/unhealthythen returns HTTP 500 for every page that includes that row, until a later sync adds the container back. At the default interval that is 6 hours.Change
rebuildContainerFromScmstill checks that SCM lists the container in that state. It then callstransitionDeletingOrDeletedToTargetState, the same in-place move SCM's report handler uses, instead of deleting and re-adding the record. The replicas stay. If the update fails, the container stays in Recon as DELETED with its replicas, the unhealthy-containers API keeps working, and the next sync tries again. Recon keeps its own container details (sequence id, key count, used bytes). Replica reports already keep those up to date.Recon can still mark the container deleted from the stale page. It shows DELETED until the next sync moves it back. Skipping a missing container in the unhealthy-containers API, instead of failing the page, is left as separate hardening.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-17061
How was this patch tested?
Ran the
TestReconSCMContainerSyncIntegrationin:ozone-reconand had a green run -mvn -pl :ozone-recon test -Dtest=TestReconSCMContainerSyncIntegration -DskipShade -DskipReconcontainerDeletedInReconButClosedInScmKeepsReplicas: Recon has the container DELETED with one replica, and SCM lists it CLOSED. After sync the container is CLOSED and the replica is still there.failedCorrectionFromDeletedKeepsContainerInRecon: the in-place update throws. The container stays in Recon as DELETED, with its replica.