Repository navigation
docs(pro): describe rehash progress and how a rehash survives a lost worker - #16325
Merged
Merged
Conversation
…worker Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Batches run in parallel, a lost batch is reported as Failed, redelivery happens at the latest after the visibility timeout, and the identity rehash item stays Running and cannot be canceled while its rehashes run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e retried A single-batch rehash is not retried after a kill, failed rehashes stay on the card for seven days, a rehash that has not moved for a day reads Failed, Run Again does not repeat a failed identity rehash, and manage.py dedupe --hash_code_only re-hashes a tool. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ry skips duplicates Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Maffooch
enabled auto-merge
October 14, 2026 23:08
blakeaowens
approved these changes
Oct 14, 2026
devGregA
approved these changes
Oct 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Documents two Pro changes to the background rehash that a Matching Configuration hash-field change (or the Locations identity rehash) queues:
The self-hosted note on the deduplication tuning page also covers worker loss. Batches run in parallel, each in its own short-lived process. A worker killed partway through hands its batches back, at the latest once the broker's visibility timeout passes, and a re-hash whose batches were all queued is not planned again. A batch that fails with an error is not retried; a killed batch is retried three times (once, with no retry, when the re-hash has a single batch). Once a batch gives up the re-hash shows Failed, and so does a re-hash that goes a whole day without finishing a batch. The page now names the recovery path,
manage.py dedupe --parser "<scan type>" --hash_code_only(it recomputes the tool's non-duplicate Findings), and the Locations page notes that running the identity rehash again does not repeat a failed rehash.DD_REHASH_SHARD_SIZE(default 50,000) sets the batch size.Pages changed:
docs/content/triage_findings/finding_deduplication/PRO__deduplication_tuning.md(Running Deduplication Retroactively on Existing Data)docs/content/asset_modelling/locations/PRO__migrating_from_endpoints.md(the data-migration suite)Test results
Docs only. Each statement was checked against the Pro implementation it describes.
Documentation
This PR is the documentation. It ships alongside the matching DefectDojo Pro change.