Skip to content

DAOS-17519 test: Automate dlck testing (basic/fault_injection) - #17307

Open
rpadma2 wants to merge 65 commits into
masterfrom
rpadma2/daos_17519
Open

DAOS-17519 test: Automate dlck testing (basic/fault_injection)#17307
rpadma2 wants to merge 65 commits into
masterfrom
rpadma2/daos_17519

Conversation

@rpadma2

@rpadma2 rpadma2 commented Dec 23, 2025 •
edited
Loading

Copy link
Copy Markdown
Contributor

Skip-func-hw-test-medium: false
Skip-func-hw-test-medium-md-on-ssd: false
Skip-fault-injection-test: true
Skip-unit-test: true
Skip-nlt: true
Test-tag: DlckBasicFaultTest DlckBasicTest

Steps for the author:

  • Commit message follows the guidelines.
  • Appropriate Features or Test-tag pragmas were used.
  • Appropriate Functional Test Stages were run.
  • At least two positive code reviews including at least one code owner from each category referenced in the PR.
  • Testing is complete. If necessary, forced-landing label added and a reason added in a comment.

After all prior steps are complete:

  • Gatekeeper requested (daos-gatekeeper added as a reviewer).

@rpadma2
DAOS-17519 test: Automate dlck testing (basic/fault_injection)
e44ee36 
Test-tag: DlckBasicFaultTest DlckBasicTest

Signed-off-by: rpadma2 <ravindran.padmanabhan@hpe.com>
@github-actions

github-actions Bot commented Dec 23, 2025 •
edited
Loading

Copy link
Copy Markdown

Ticket title is 'Automation - Basic dlck test: scan the DAOS system by running the dlck tool.'
Status is 'In Progress'
Labels: 'test_2.8,testp2'
https://daosio.atlassian.net/browse/DAOS-17519

@rpadma2
DAOS-17519 test: Fix linting issues.
1dfe0ef 
Test-tag: DlckBasicTest DlckBasicFaultTest
Skip-func-hw-test-medium: false
Skip-func-hw-test-medium-md-on-ssd: false
Skip-unit-test: true
Skip-fault-injection-test: true

Signed-off-by: rpadma2 <ravindran.padmanabhan@hpe.com>
janekmi
janekmi requested changes Dec 29, 2025
View reviewed changes
Comment thread src/tests/ftest/util/fault_config_utils.py Outdated
Comment thread src/tests/ftest/util/fault_config_utils.py Outdated
Comment thread src/tests/ftest/util/fault_config_utils.py Outdated
Comment thread src/tests/ftest/util/fault_config_utils.py Outdated
Comment thread src/tests/ftest/util/fault_config_utils.py Outdated
Comment thread src/tests/ftest/recovery/dlck_basic_fault.py Outdated
Comment thread src/tests/ftest/recovery/dlck_basic_fault.py Outdated
Comment thread src/tests/ftest/recovery/dlck_basic_fault.py Outdated
Comment thread src/tests/ftest/recovery/dlck_basic.yaml Outdated
Comment thread src/tests/ftest/recovery/dlck_basic.py Outdated
@rpadma2
DAOS-17519 test: Merge branch 'master' into rpadma2/daos_17519
a376c98 
Skip-func-hw-test-medium: false
Skip-func-hw-test-medium-md-on-ssd: false
Skip-unit-test: true
Skip-fault-injection-test: true
Skip-nlt: true
Test-tag: DlckBasicFaultTest DlckBasicTest
@daosbuild3

daosbuild3 commented Jan 8, 2026

Copy link
Copy Markdown
Collaborator

Test stage Functional Hardware Medium completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17307/3/execution/node/932/log

@daosbuild3

daosbuild3 commented Jan 8, 2026

Copy link
Copy Markdown
Collaborator

Test stage Functional Hardware Medium MD on SSD completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-17307/3/testReport/

rpadma2 added 2 commits March 12, 2026 11:26
@rpadma2
DAOS-17519 test: Merge branch 'master' into rpadma2/daos_17519
1f16a51
@rpadma2
DAOS-17519 test: Some of the review updates.
aa8ede9 
Skip-func-hw-test-medium: false
Skip-func-hw-test-medium-md-on-ssd: false
Skip-unit-test: true
Skip-fault-injection-test: true
Skip-nlt: true
Test-tag: DlckBasicFaultTest DlckBasicTest

Signed-off-by: rpadma2 <ravindran.padmanabhan@hpe.com>
@daosbuild3

daosbuild3 commented Mar 12, 2026

Copy link
Copy Markdown
Collaborator

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17307/4/execution/node/966/log

rpadma2 added 3 commits March 31, 2026 10:49
@rpadma2
DAOS-17519 test: Merge branch 'master' into rpadma2/daos_17519
5213ba6
@rpadma2
DAOS-17519 test: Fix Linting issues.
0cea354 
Skip-func-hw-test-medium: false
Skip-func-hw-test-medium-md-on-ssd: false
Skip-unit-test: true
Skip-fault-injection-test: true
Skip-nlt: true
Test-tag: DlckBasicFaultTest DlckBasicTest

Signed-off-by: rpadma2 <ravindran.padmanabhan@hpe.com>
@rpadma2
DAOS-17519 test: Fix python/flake8 lint and copyright issues.
585528a 
Skip-func-hw-test-medium: false
Skip-func-hw-test-medium-md-on-ssd: false
Skip-unit-test: true
Skip-fault-injection-test: true
Skip-nlt: true
Test-tag: DlckBasicFaultTest DlckBasicTest

Signed-off-by: rpadma2 <ravindran.padmanabhan@hpe.com>
@daosbuild3

daosbuild3 commented Mar 31, 2026

Copy link
Copy Markdown
Collaborator

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17307/5/execution/node/988/log

rpadma2 added 2 commits May 22, 2026 13:18
@rpadma2
DAOS-17519 test: Merge branch 'master' into rpadma2/daos_17519
593f473
@rpadma2
DAOS-17519 test: Update the control_meta_dir.
e4508aa 
Skip-func-hw-test-medium: false
Skip-func-hw-test-medium-md-on-ssd: false
Skip-unit-test: true
Skip-fault-injection-test: true
Skip-nlt: true
Test-tag: DlckBasicFaultTest DlckBasicTest

Signed-off-by: rpadma2 <ravindran.padmanabhan@hpe.com>
@daosbuild3

daosbuild3 commented May 23, 2026

Copy link
Copy Markdown
Collaborator

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17307/6/execution/node/867/log

@rpadma2
DAOS-17519 test: Fix typo on control_metadata_dir path.
10920ac 
Skip-func-hw-test-medium: false
Skip-func-hw-test-medium-md-on-ssd: false
Skip-unit-test: true
Skip-fault-injection-test: true
Skip-nlt: true
Test-tag: DlckBasicFaultTest DlckBasicTest

Signed-off-by: rpadma2 <ravindran.padmanabhan@hpe.com>
@daosbuild3

daosbuild3 commented May 23, 2026

Copy link
Copy Markdown
Collaborator

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17307/7/execution/node/866/log

rpadma2 added 2 commits August 12, 2026 14:18
@rpadma2
DAOS-17519 test: Merge branch 'master' into rpadma2/daos_17519
2b4bc8a
@rpadma2
DAOS-17519 test: Fix some of the pylint errors.
4103769 
Skip-func-hw-test-medium: false
Skip-func-hw-test-medium-md-on-ssd: false
Skip-unit-test: true
Skip-fault-injection-test: true
Skip-nlt: true
Test-tag: DlckBasicFaultTest DlckBasicTest

Signed-off-by: rpadma2 <ravindran.padmanabhan@hpe.com>
@rpadma2
rpadma2 requested a review from janekmi August 12, 2026 18:40
@daosbuild3

daosbuild3 commented Aug 12, 2026

Copy link
Copy Markdown
Collaborator

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17307/8/execution/node/1073/log

@daosbuild3

daosbuild3 commented Aug 12, 2026

Copy link
Copy Markdown
Collaborator

Test stage Functional Hardware Medium completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17307/8/execution/node/1083/log

rpadma2 added 2 commits August 12, 2026 15:25
@rpadma2
DAOS-17519 test: Fix more pylint issues.
2191472 
Skip-func-hw-test-medium: false
Skip-func-hw-test-medium-md-on-ssd: false
Skip-unit-test: true
Skip-fault-injection-test: true
Skip-nlt: true
Test-tag: DlckBasicFaultTest DlckBasicTest

Signed-off-by: rpadma2 <ravindran.padmanabhan@hpe.com>
@rpadma2
DAOS-17519 test: Update test timeouts.
fb11310 
Skip-func-hw-test-medium: false
Skip-func-hw-test-medium-md-on-ssd: false
Skip-unit-test: true
Skip-fault-injection-test: true
Skip-nlt: true
Test-tag: DlckBasicFaultTest DlckBasicTest

Signed-off-by: rpadma2 <ravindran.padmanabhan@hpe.com>
@daosbuild3

daosbuild3 commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17307/9/execution/node/1072/log

@daosbuild3

daosbuild3 commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator

Test stage Functional Hardware Medium completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17307/9/execution/node/1082/log

rpadma2 added 2 commits August 12, 2026 21:55
@rpadma2
DAOS-17519 test: add_pool needed parameter.
7284649 
Skip-func-hw-test-medium: false
Skip-func-hw-test-medium-md-on-ssd: false
Skip-unit-test: true
Skip-fault-injection-test: true
Skip-nlt: true
Test-tag: DlckBasicFaultTest DlckBasicTest

Signed-off-by: rpadma2 <ravindran.padmanabhan@hpe.com>
@rpadma2
DAOS-17519 test: Fix flake8 newline issue.
3b870dc 
Skip-func-hw-test-medium: false
Skip-func-hw-test-medium-md-on-ssd: false
Skip-unit-test: true
Skip-fault-injection-test: true
Skip-nlt: true
Test-tag: DlckBasicFaultTest DlckBasicTest

Signed-off-by: rpadma2 <ravindran.padmanabhan@hpe.com>
@rpadma2

rpadma2 commented Aug 27, 2026

Copy link
Copy Markdown
Contributor Author

I have addressed most of the issues on the latest update....

Since you wrote the above I went through all of them. Probably you are aware but just in case I wrote down the list:

I have made the changes. Also, added comment to the changes which I haven't done due to some fault injection limitation with the python ftest infrastructure.

@daosbuild3

daosbuild3 commented Aug 27, 2026

Copy link
Copy Markdown
Collaborator

Test stage Build on EL 9 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17307/39/execution/node/520/log

rpadma2 added 2 commits August 26, 2026 21:19
@rpadma2
DAOS-17519 test: Remove unwanted code.
e8a4309 
Skip-func-hw-test-medium: false
Skip-func-hw-test-medium-md-on-ssd: false
Skip-fault-injection-test: true
Skip-unit-test: true
Skip-nlt: true
Test-tag: DlckBasicFaultTest DlckBasicTest

Signed-off-by: rpadma2 <ravindran.padmanabhan@hpe.com>
@rpadma2
DAOS-17519 test: Address the dlck build error.
f32bea4 
Skip-func-hw-test-medium: false
Skip-func-hw-test-medium-md-on-ssd: false
Skip-fault-injection-test: true
Skip-unit-test: true
Skip-nlt: true
Test-tag: DlckBasicFaultTest DlckBasicTest

Signed-off-by: rpadma2 <ravindran.padmanabhan@hpe.com>
@daosbuild3

daosbuild3 commented Aug 27, 2026

Copy link
Copy Markdown
Collaborator

Test stage Build on EL 9 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17307/40/execution/node/531/log

rpadma2 added 3 commits August 26, 2026 21:27
@rpadma2
DAOS-17519 test: Minor space issue.
82a724d 
Skip-func-hw-test-medium: false
Skip-func-hw-test-medium-md-on-ssd: false
Skip-fault-injection-test: true
Skip-unit-test: true
Skip-nlt: true
Test-tag: DlckBasicFaultTest DlckBasicTest

Signed-off-by: rpadma2 <ravindran.padmanabhan@hpe.com>
@rpadma2
DAOS-17519 test: Fix clang errors.
3d4781f 
Skip-func-hw-test-medium: false
Skip-func-hw-test-medium-md-on-ssd: false
Skip-fault-injection-test: true
Skip-unit-test: true
Skip-nlt: true
Test-tag: DlckBasicFaultTest DlckBasicTest

Signed-off-by: rpadma2 <ravindran.padmanabhan@hpe.com>
@rpadma2
DAOS-17519 test: More clang errors.
33590aa 
Skip-func-hw-test-medium: false
Skip-func-hw-test-medium-md-on-ssd: false
Skip-fault-injection-test: true
Skip-unit-test: true
Skip-nlt: true
Test-tag: DlckBasicFaultTest DlckBasicTest

Signed-off-by: rpadma2 <ravindran.padmanabhan@hpe.com>
janekmi
janekmi requested changes Aug 27, 2026
View reviewed changes
Comment thread src/utils/dlck/cmds/dlck_cmd_check.c Outdated
Comment thread src/tests/ftest/recovery/dlck_basic.yaml Outdated
rpadma2 added 3 commits August 27, 2026 09:20
@rpadma2
DAOS-17519 test: Merge branch 'master' into rpadma2/daos_17519
3479901
@rpadma2
DAOS-17519 test: Reduce the test_servers. Review updates.
e061e56 
Skip-func-hw-test-medium: false
Skip-func-hw-test-medium-md-on-ssd: false
Skip-fault-injection-test: true
Skip-unit-test: true
Skip-nlt: true
Test-tag: DlckBasicFaultTest DlckBasicTest

Signed-off-by: rpadma2 <ravindran.padmanabhan@hpe.com>
@rpadma2
DAOS-17519 test: Modify fault injection mechanism for dlck utility.
767de9a 
Skip-func-hw-test-medium: false
Skip-func-hw-test-medium-md-on-ssd: false
Skip-fault-injection-test: true
Skip-unit-test: true
Skip-nlt: true
Test-tag: DlckBasicFaultTest DlckBasicTest

Signed-off-by: rpadma2 <ravindran.padmanabhan@hpe.com>
@rpadma2

rpadma2 commented Aug 28, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @janekmi , @phender , @daltonbohning and @shimizukko for your reviews... The dlck test framework seems to be much better now. Take a look at the latest changes.

rpadma2 added 2 commits August 28, 2026 13:24
@rpadma2
DAOS-17519 test: Remove the unwanted comment.
0891a3c 
Skip-func-hw-test-medium: false
Skip-func-hw-test-medium-md-on-ssd: false
Skip-fault-injection-test: true
Skip-unit-test: true
Skip-nlt: true
Test-tag: DlckBasicFaultTest DlckBasicTest

Signed-off-by: rpadma2 <ravindran.padmanabhan@hpe.com>
@rpadma2
DAOS-17519 test: Fix pylint issue.
db479c6 
Skip-func-hw-test-medium: false
Skip-func-hw-test-medium-md-on-ssd: false
Skip-fault-injection-test: true
Skip-unit-test: true
Skip-nlt: true
Test-tag: DlckBasicFaultTest DlckBasicTest

Signed-off-by: rpadma2 <ravindran.padmanabhan@hpe.com>
daltonbohning
daltonbohning reviewed Aug 28, 2026
View reviewed changes
Comment thread src/tests/ftest/recovery/dlck_basic.py Outdated
errors = []
dmg = self.get_dmg_command()
self.log_step("Create a pool to run dlck")
pool = add_pool(self)

@daltonbohning daltonbohning Aug 28, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We actually want to go back to using ONLY self.get_pool so if you need to push again, please change this to use self.get_pool.

@rpadma2 rpadma2 Aug 28, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

Comment thread src/tests/ftest/recovery/dlck_basic.py Outdated
Comment on lines +43 to +44
errors.append(f"dlck failed on {result.failed_hosts}")
self.fail(f"dlck basic test failed with errors: {errors}")

@daltonbohning daltonbohning Aug 28, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the only error in the list, so I think we should just print it directly

Suggested change
errors.append(f"dlck failed on {result.failed_hosts}")
self.fail(f"dlck basic test failed with errors: {errors}")
self.fail(f"dlck failed on {result.failed_hosts}")

@rpadma2 rpadma2 Aug 28, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

Comment thread src/tests/ftest/recovery/dlck_basic.yaml Outdated
test_servers: 1
test_clients: 1

timeout: 600

@daltonbohning daltonbohning Aug 28, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The last run completed in under 5 minutes so maybe this can be reduce to like 7 minutes?

Suggested change
timeout: 600
timeout: 420

@rpadma2 rpadma2 Aug 28, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

Comment thread src/tests/ftest/recovery/dlck_basic_fault.py Outdated
self.log.info("Faults: %s", fault_list)
self.log.info("Faults dict: %s", faults_dict)
dmg = self.get_dmg_command()
pool = add_pool(self)

@daltonbohning daltonbohning Aug 28, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
pool = add_pool(self)
pool = self.get_pool(self)

@rpadma2 rpadma2 Aug 28, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Comment thread src/tests/ftest/recovery/dlck_basic_fault.py Outdated
Comment thread src/tests/ftest/recovery/dlck_basic_fault.py Outdated
Comment on lines +70 to +76
count = 0
for key, value in faults_dict[test_fault].items():
if count == 0:
f.write(f"- {key}: \'{value}\'\n")
else:
f.write(f" {key}: \'{value}\'\n")
count += 1

@daltonbohning daltonbohning Aug 28, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can use enumerate instead of keeping track of the count

Suggested change
count = 0
for key, value in faults_dict[test_fault].items():
if count == 0:
f.write(f"- {key}: \'{value}\'\n")
else:
f.write(f" {key}: \'{value}\'\n")
count += 1
for count, (key, value) in enumerate(d.items()):
if count == 0:
f.write(f"- {key}: \'{value}\'\n")
else:
f.write(f" {key}: \'{value}\'\n")

@rpadma2 rpadma2 Aug 28, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done. Good suggestion !

Comment thread src/tests/ftest/util/dlck_utils.py Outdated
super().__init__(namespace, "dlck", path)
self.pool_uuid = FormattedParameter("--file={}", None)
self.nvme = FormattedParameter("--nvme={}", None)
self.storage_mount = FormattedParameter("--storage={}", None)

@daltonbohning daltonbohning Aug 28, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any reason to not call this self.storage to match the command line param name?

@rpadma2 rpadma2 Aug 28, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure. Done.

Comment thread src/tests/ftest/recovery/dlck_basic_fault.py Outdated

@daltonbohning daltonbohning Aug 28, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since the server and pool config is the same for each test, I would suggest combining these two tests into a single python file.

@rpadma2 rpadma2 Aug 28, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. Merged both the test cases to a single python file...

janekmi
janekmi requested changes Aug 28, 2026
View reviewed changes
Comment thread src/tests/ftest/recovery/dlck_basic_fault.py Outdated
Comment thread src/tests/ftest/recovery/dlck_basic_fault.py Outdated
@rpadma2
DAOS-17519 test: Merged both the test cases to a single python file.
ee205a0 
Skip-func-hw-test-medium: false
Skip-func-hw-test-medium-md-on-ssd: false
Skip-fault-injection-test: true
Skip-unit-test: true
Skip-nlt: true
Test-tag: DlckBasicFaultTest DlckBasicTest

Signed-off-by: rpadma2 <ravindran.padmanabhan@hpe.com>
rpadma2 added 2 commits August 28, 2026 19:52
@rpadma2
DAOS-17519 test: Missed to remove add_pool import.
9c8b347 
Skip-func-hw-test-medium: false
Skip-func-hw-test-medium-md-on-ssd: false
Skip-fault-injection-test: true
Skip-unit-test: true
Skip-nlt: true
Test-tag: DlckBasicFaultTest DlckBasicTest

Signed-off-by: rpadma2 <ravindran.padmanabhan@hpe.com>
@rpadma2
DAOS-17519 test: Removed unused variable.
9d199de 
Skip-func-hw-test-medium: false
Skip-func-hw-test-medium-md-on-ssd: false
Skip-fault-injection-test: true
Skip-unit-test: true
Skip-nlt: true
Test-tag: DlckBasicFaultTest DlckBasicTest

Signed-off-by: rpadma2 <ravindran.padmanabhan@hpe.com>
@rpadma2

rpadma2 commented Aug 29, 2026

Copy link
Copy Markdown
Contributor Author

Hmm... Something doesn't work after I changed to use get_pool instead of add_pool... Let me debug it later...

@rpadma2
DAOS-17519 test: Fixed the get_pool call.
f78c858 
Skip-func-hw-test-medium: false
Skip-func-hw-test-medium-md-on-ssd: false
Skip-fault-injection-test: true
Skip-unit-test: true
Skip-nlt: true
Test-tag: DlckBasicFaultTest DlckBasicTest

Signed-off-by: rpadma2 <ravindran.padmanabhan@hpe.com>
shimizukko
shimizukko approved these changes Aug 29, 2026
View reviewed changes
Comment thread src/tests/ftest/util/command_utils.py
raise TypeError(f"Invalid {self.command} host NodeSet: {value} ({type(value)})")
self._hosts = value.copy()

def _run_process(self, raise_exception=None):

@shimizukko shimizukko Aug 29, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should the default for raise_exception be False instead of None?

@rpadma2 rpadma2 Aug 29, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@shimizukko : Presently I am using this file based on @phender PR (related to his branch : hendersp/DAOS-19532 . Hopefully, all issues related to command_utils.py is addressed in that PR which I merge to my branch later.

Comment thread src/tests/ftest/util/command_utils.py
Returns:
CommandResult: result from running the command
"""
if raise_exception is None:

@shimizukko shimizukko Aug 29, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If it's bool shouldn't we use False?

Suggested change
if raise_exception is None:
if raise_exception is False:

Comment thread src/tests/ftest/util/command_utils.py
detected, running = stop_processes(
self.log, hosts, regex, full_command=self.full_command_regex)
if not detected:
self.log.info(

@shimizukko shimizukko Aug 29, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't we collect error here (and elif and else block below) and return it at the end?

Comment thread src/tests/ftest/util/dlck_utils.py
from command_utils_base import CommandWithParameters, FormattedParameter
from run_utils import run_remote

class TestDlck(TestWithServers):

@shimizukko shimizukko Aug 29, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe the pattern is to separate util file and test base file. If this is a test base file, the class name should be:

Suggested change
class TestDlck(TestWithServers):
class DlckTestBase(TestWithServers):

The file name should be dlck_test_base.py

@rpadma2 rpadma2 Aug 29, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@daltonbohning / @phender : Any comments? I can do whatever we decide. This is an initial version... I am pretty sure we will be create lot of utility functions later....

Comment thread src/tests/ftest/recovery/dlck_basic.py
errors.append(error)
self.log_step("Run the dmg start command")
dmg.system_start()

@shimizukko shimizukko Aug 29, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is no line break from beginning of the test to here. IMO, it would be easier to read by adding a few line breaks to separate many lines of code into blocks of code that runs similar tasks.

@rpadma2 rpadma2 Aug 29, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will do it later... Let the present PR run complete.

Comment thread src/tests/ftest/recovery/dlck_basic.py
:avocado: recursive
"""
def check_dlck_result(self, result, fault_name):
"""Check that dlck reports the expected error for an injected fault."""

@shimizukko shimizukko Aug 29, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You might want to add Args section to explain the arguments.
Also, the description would be more clear if you say "Check that dlck reported the expected error..."

janekmi
janekmi requested changes Aug 29, 2026
View reviewed changes
Comment thread src/tests/ftest/recovery/dlck_basic.py
Comment on lines +91 to +97
with open(fault_inject_file, 'w') as f:
f.write("fault_config:\n")
for count, (key, value) in enumerate(faults_dict[test_fault].items()):
if count == 0:
f.write(f"- {key}: \'{value}\'\n")
else:
f.write(f" {key}: \'{value}\'\n")

@janekmi janekmi Aug 29, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just realized it makes little sense we generate a YAML file line by line when we could use the pyyaml package. From what I see we already use it in functional tests. Please look out for import yaml.

@janekmi janekmi Aug 29, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe it would look like something like:

            fault_config = {
                'fault_config': [
                    {key: str(value) for key, value in faults_dict[test_fault].items()}]}
            with open(fault_inject_file, 'w') as f:
                yaml.dump(fault_config, f, default_flow_style=False)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@janekmi janekmi janekmi requested changes
@shimizukko shimizukko shimizukko approved these changes
@jamesanunez jamesanunez Awaiting requested review from jamesanunez
@phender phender Awaiting requested review from phender
@daltonbohning daltonbohning Awaiting requested review from daltonbohning

Requested changes must be addressed to merge this pull request.

Assignees

No one assigned

Labels

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@rpadma2 @daosbuild3 @janekmi @shimizukko @daltonbohning @phender