Skip to content

HDDS-17061. Recon drops the replicas of a DELETED container when it moves it back to SCM's live state. - #12096

Draft
ArafatKhan2198 wants to merge 1 commit into
apache:masterfrom
ArafatKhan2198:HDDS-17061
Draft

ArafatKhan2198 wants to merge 1 commit into
apache:masterfrom
ArafatKhan2198:HDDS-17061

Conversation

@ArafatKhan2198

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

When Recon brings a deleted container back to match SCM, it deletes its own record and adds a new one. Deleting the record also deletes the list of datanodes that hold copies. Recon then treats the container as missing even though the data is still there. SCM makes the same move in place and keeps that list. This change makes Recon do the same.

What goes wrong

  1. Recon reads a page of containers that SCM has marked deleted.
  2. Before Recon finishes that page, a datanode reports a copy that still has data. SCM brings the container back to QUASI_CLOSED or CLOSED.
  3. Recon finishes the old page and marks the container deleted anyway. It still knows which datanodes hold copies.
  4. On the next sync, SCM lists the container as alive. rebuildContainerFromScm deletes Recon's record and adds a new one from SCM's container info.
  5. The new record has no replicas. The next health check, every 5 minutes by default, marks the container MISSING. It stays MISSING until each datanode's next full report, up to 60 minutes by default, adds the replicas back. No data is lost.
  6. If adding the new record fails, the failure is only logged and the container is gone from Recon. An old MISSING row can still be in the unhealthy-containers table when no health check ran between the two syncs, for example after a restart. /api/v1/containers/unhealthy then returns HTTP 500 for every page that includes that row, until a later sync adds the container back. At the default interval that is 6 hours.

Change

rebuildContainerFromScm still checks that SCM lists the container in that state. It then calls transitionDeletingOrDeletedToTargetState, the same in-place move SCM's report handler uses, instead of deleting and re-adding the record. The replicas stay. If the update fails, the container stays in Recon as DELETED with its replicas, the unhealthy-containers API keeps working, and the next sync tries again. Recon keeps its own container details (sequence id, key count, used bytes). Replica reports already keep those up to date.

Recon can still mark the container deleted from the stale page. It shows DELETED until the next sync moves it back. Skipping a missing container in the unhealthy-containers API, instead of failing the page, is left as separate hardening.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-17061

How was this patch tested?

Ran the TestReconSCMContainerSyncIntegration in :ozone-recon and had a green run -

mvn -pl :ozone-recon test -Dtest=TestReconSCMContainerSyncIntegration -DskipShade -DskipRecon
  • containerDeletedInReconButClosedInScmKeepsReplicas: Recon has the container DELETED with one replica, and SCM lists it CLOSED. After sync the container is CLOSED and the replica is still there.
  • failedCorrectionFromDeletedKeepsContainerInRecon: the in-place update throws. The container stays in Recon as DELETED, with its replica.

@github-actions github-actions Bot added the recon label Nov 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant