DAOS-17321 ddb: add csum_check command to verify data checksums - #18940
Draft
knard38 wants to merge 1 commit into
Draft
DAOS-17321 ddb: add csum_check command to verify data checksums#18940knard38 wants to merge 1 commit into
knard38 wants to merge 1 commit into
Conversation
Adds a new `ddb csum_check <path>` command that recomputes the checksum(s) stored at a VOS tree path from the currently stored data and compares them against what is stored on disk, without modifying anything. The path must be complete (including the akey, and the extent for an array value). The --epoch flag selects which version to check: for a single value it selects the value visible at or before that epoch; for an array value it bounds the visible record extent. By default the command is silent on a clean result (matching, or no checksum stored) and only prints corrupted entries plus a final error summary; --verbose prints every entry regardless of outcome and a final summary line even on success. The command exits non-zero (-DER_CSUM) if any checksum entry fails to match. Implementation: - ddb_vos.c/.h: adds dv_check_csum(), mirroring the structure of the existing dv_dump_csum() (open container, build the iod from path, dispatch by value type, close container) as an independent implementation that additionally fetches the live data, recomputes its checksum(s), and compares against what was fetched. Reports results through a callback with a got_csums array: got_csums[i] holds the recomputed checksum when entry i's stored checksum did not match, and is NULL when it matched or there was nothing to check. - ddb_commands.c: wires csum_check into the interactive/scripted command set, sharing print formatting with the existing csum_dump command (adding "State: OK"/"NOK", with the recomputed value shown as "(got=...)" on a mismatch). - src/control/cmd/ddb: adds the Go-side command/flag definitions and cgo wrapper so csum_check is available from the ddb binary. - README.md: documents the new command (and the previously undocumented csum_dump) in the command reference. Test coverage: unit tests for dv_check_csum() (single value and array, matching/mismatching/no-checksum cases, and per-entry mismatch isolation when one of several overlapping array segments is corrupted) and for the csum_check command's print output and verbosity behavior, using extended VOS test fixtures for corrupted single-value and array checksums. Validated locally (scons build under -Werror, full ddb_tests suite) and on hardware (brd-216: rebuilt, ran ddb_ut/ddb_tests/go unit tests, and a live functional pass with a real pool/container -- csum_check agreed with csum_dump on a clean value, and correctly detected corruption after directly modifying stored data). Signed-off-by: Cedric Koch-Hofer <cedric.koch-hofer@hpe.com>
|
Ticket title is 'Checksum management with ddb' |
This was referenced Aug 26, 2026
knard38
pushed a commit
that referenced
this pull request
Aug 26, 2026
Adds dv_check_csum() to the ddb VOS API (ddb_vos.c/.h), mirroring the structure of the existing dv_dump_csum() (open container, build the iod from path, dispatch by value type, close container) as an independent implementation that additionally fetches the live data, recomputes its checksum(s), and compares against what was fetched. Results are reported through a callback with a got_csums array: got_csums[i] holds the recomputed checksum when entry i's stored checksum did not match, and is NULL when it matched or there was nothing to check. The function returns -DER_CSUM if any checksum entry fails to match. Test coverage: unit tests for dv_check_csum() (single value and array, matching/mismatching/no-checksum cases, and per-entry mismatch isolation when one of several overlapping array segments is corrupted), using extended VOS test fixtures (ddb_test_driver.c/.h) for corrupted single-value and array checksums. This is patch 1/3 of the DAOS-17321 csum_check split (supersedes PR #18940): see patch-007 for the ddb command wiring and patch-008 for the Go CLI/documentation. Features: recovery Signed-off-by: Cedric Koch-Hofer <cedric.koch-hofer@hpe.com>
knard38
pushed a commit
that referenced
this pull request
Aug 26, 2026
Wires csum_check into the ddb interactive/scripted command set (ddb.h, ddb_commands.c) on top of dv_check_csum() from patch-006, sharing print formatting with the existing csum_dump command (adding "State: OK"/"NOK", with the recomputed value shown as "(got=...)" on a mismatch). ddb_run_csum_check(struct ddb_ctx *ctx, struct csum_check_options *opt) is the new top-level command function. The path must be complete (including the akey, and the extent for an array value). The --epoch flag selects which version to check: for a single value it selects the value visible at or before that epoch; for an array value it bounds the visible record extent. By default the command is silent on a clean result (matching, or no checksum stored) and only prints corrupted entries plus a final error summary; --verbose prints every entry regardless of outcome and a final summary line even on success. The command exits non-zero (-DER_CSUM) if any checksum entry fails to match. Test coverage: unit tests for the csum_check command's print output and verbosity behavior (ddb_commands_tests.c), plus a new assert_string_not_contains() cmocka helper (ddb_cmocka.h) used to confirm suppressed entries are absent from non-verbose output. This is patch 2/3 of the DAOS-17321 csum_check split (supersedes PR #18940): see patch-006 for the ddb_vos dv_check_csum() API and patch-008 for the Go CLI/documentation. Signed-off-by: Cedric Koch-Hofer <cedric.koch-hofer@hpe.com>
knard38
pushed a commit
that referenced
this pull request
Aug 26, 2026
Adds the Go-side command/flag definitions and cgo wrapper so csum_check is available from the ddb binary, wired to ddb_run_csum_check() from patch-007: - ddb_commands.go: registers the csum_check command (required `path` positional, `-e/--epoch` default EPOCH_MAX, `-v/--verbose`). - commands_wrapper.go: CsumCheck() on DdbContext builds the C csum_check_options struct and calls through to ddb_run_csum_check(). - libddb.go / libddb_stubs.go: real + stub ddb_run_csum_check() wrappers, following the existing per-command _RC/_Fn hook pattern so csum_check can be unit tested without a live VOS environment. - ddb_commands_test.go: help text, missing/invalid argument handling, default/short/long epoch forms, and verbose short/long forms. Also documents the new command (and the previously undocumented csum_dump) in the ddb command reference (README.md). This is patch 3/3 of the DAOS-17321 csum_check split (supersedes PR #18940): see patch-006 for the ddb_vos dv_check_csum() API and patch-007 for the ddb C command wiring. This completes the user-facing `ddb csum_check` feature. Signed-off-by: Cedric Koch-Hofer <cedric.koch-hofer@hpe.com>
Collaborator
|
Test stage Functional Hardware Large MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18940/1/execution/node/1762/log |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds a new
ddb csum_check <path>command that recomputes the checksum(s) stored at a VOS tree path from the currently stored data and compares them against what is stored on disk, without modifying anything. The path must be complete (including the akey, and the extent for an array value). The --epoch flag selects which version to check: for a single value it selects the value visible at or before that epoch; for an array value it bounds the visible record extent. By default the command is silent on a clean result (matching, or no checksum stored) and only prints corrupted entries plus a final error summary; --verbose prints every entry regardless of outcome and a final summary line even on success. The command exits non-zero (-DER_CSUM) if any checksum entry fails to match.Implementation:
Test coverage: unit tests for dv_check_csum() (single value and array, matching/mismatching/no-checksum cases, and per-entry mismatch isolation when one of several overlapping array segments is corrupted) and for the csum_check command's print output and verbosity behavior, using extended VOS test fixtures for corrupted single-value and array checksums.
Validated locally (scons build under -Werror, full ddb_tests suite) and on hardware (brd-216: rebuilt, ran ddb_ut/ddb_tests/go unit tests, and a live functional pass with a real pool/container -- csum_check agreed with csum_dump on a clean value, and correctly detected corruption after directly modifying stored data).
Steps for the author:
After all prior steps are complete: