SRE-3989 test: add NVMe reset recovery - #18956
Conversation
|
Errors are Unable to load ticket data |
cbe79c3 to
486b154
Compare
|
Is this meant to be ready for review? |
|
Test stage Functional Hardware Large MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18956/3/execution/node/773/log |
486b154 to
4f3b0c0
Compare
Unfortunately it is not. It still needs some debug, and hdr-114 is also still going into this state. |
|
Test stage Functional Hardware Large MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18956/4/execution/node/772/log |
Log the NVMe IOMMU baseline to hardware prep so post-reboot PCI, driver, IOMMU, and nvme-cli state is recorded before functional tests run. Add NVMe diagnostics around the ftest reset paths used by server manager cleanup and launch-level post-test cleanup. - Jenkinsfile: Fix minor groovylint issues. - ci/functional/test_main_prep_node.sh: Log NVMe IOMMU settings and health before tests are run Take cluster offline if VT-D/IOMMU is not enabled. - src/tests/ftest/util/collection_utils.py: - src/tests/ftest/util/server_utils.py: - src/tests/ftest/util/storage_utils.py: Add NVMe diagnostics for reset of NVMe devices. Signed-off-by: John E. Malmberg <john.malmberg@hpe.com>
4f3b0c0 to
d9ceb97
Compare
|
Test stage Functional Hardware Large MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18956/5/execution/node/772/log |
|
Test stage Functional Hardware Large MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18956/6/execution/node/773/log |
3 participants
Log the NVMe IOMMU baseline to hardware prep so post-reboot PCI, driver, IOMMU, and nvme-cli state is recorded before functional tests run.
Add NVMe diagnostics around the ftest reset paths used by server manager cleanup and launch-level post-test cleanup. When daos_server nvme reset fails, attempt non-reboot recovery with daos_server nvme scan --ignore-config and continue if the scan recovers the devices.
Jenkinsfile: Fix minor groovylint issues.
ci/functional/test_main_prep_node.sh: Log NVMe IOMMU settings and health before tests are run
src/tests/ftest/util/collection_utils.py:
src/tests/ftest/util/server_utils.py:
src/tests/ftest/util/storage_utils.py: Add NVMe diagnostics for reset of NVMe devices and attempt recovery of the NVMe devices if the reset fails.
Steps for the author:
After all prior steps are complete: