tests(storage): add LvmHotAddSuite for hot-add + LVM pvcreate stress - #4674
tests(storage): add LvmHotAddSuite for hot-add + LVM pvcreate stress#4674rabdulfaizy wants to merge 1 commit into
Conversation
There was a problem hiding this comment.
Pull request overview
Adds a new functional storage test suite (LvmHotAddSuite) under testsuites/storage to stress Azure data-disk hot-add followed by LVM pvcreate, aiming to reproduce/observe NVMe namespace initialization failures (including “Identify … failed” patterns) while gating PASS/FAIL on functional disk usability.
Changes:
- Introduces
LvmHotAddSuitewith 4 test variants (parallel/serial × StandardSSD/PremiumSSD) usingsimple_requirement(... max_data_disk_count=IntRange(min=1))to preserve all free LUNs for the hot-add burst. - Implements hot-add enumeration +
fdisk -l+pvcreategates, plus a post-pvcreatelsblkcheck for missing/zero-sized devices. - Adds dmesg scanning for NVMe Identify-failure fingerprints as informational warnings.
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| timeout=60, | ||
| interval=2, |
There was a problem hiding this comment.
Done in 2680b46 - promoted to _HOT_ADD_LSBLK_POLL_TIMEOUT = 60 and _HOT_ADD_LSBLK_POLL_INTERVAL = 2 class constants with inline rationale explaining why these values were chosen (max_data_disk_count fanout + typical udev settle time), so future tuning is intentional and reviewable.
AI Test Case SelectionSelected 3 test case(s): verify_dm_cache_setup,verify_disk_with_nobarrier,verify_disk_with_fio_verify_option Marketplace image: Result: Succeeded |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.
Suppressed comments (2)
lisa/microsoft/testsuites/storage/lvm_hot_add.py:390
- The cleanup path catches a broad
Exceptionaroundpvremove.remove_pv(..., ignore_errors=True). Sinceignore_errors=Truealready suppresses non-zero exit codes, the remaining expected failure mode is typically a LISA-level execution problem (e.g. command timeout) which raisesLisaException. Narrowing the exception type avoids hiding unexpected programming errors while still keeping cleanup best-effort.
try:
pvremove.remove_pv(*new_device_paths, ignore_errors=True)
except Exception as ex: # noqa: BLE001
log.warning(f"pvremove cleanup raised: {ex}")
lisa/microsoft/testsuites/storage/lvm_hot_add.py:256
max_disks_to_addis parsed withint(...)directly, so a non-numeric value (e.g. a typo in the runbook variable) will raiseValueErrorwithout a clear, actionable message. Please validate and fail with a helpful error that shows the received value and an example of the correct-v max_disks_to_add:<N>usage.
requested_cap = int(variables.get("max_disks_to_add", 0) or 0)
AI Test Case SelectionSelected 3 test case(s): verify_dm_cache_setup,verify_disk_with_nobarrier,verify_disk_with_fio_verify_option Marketplace image: Result: Succeeded |
AI Test Case SelectionSelected 3 test case(s): verify_dm_cache_setup,verify_disk_with_nobarrier,verify_disk_with_fio_verify_option Marketplace image: Result: Succeeded |
… LVM
Adds a new functional storage suite that stresses hot-add of data disks
followed by LVM pvcreate to reproduce a class of host-side NVMe VF
initialization failures where the guest kernel logs
nvme nvmeN: Identify NS List failed (status=0xb)
on the newly attached namespace, leaving pvcreate to fail with
'Error reading device ...' and lsblk to report the affected devices
with size 0. The suite covers four permutations:
- verify_hot_add_disks_pvcreate_parallel_standard_ssd (P1)
- verify_hot_add_disks_pvcreate_parallel_premium_ssd (P1)
- verify_hot_add_disks_pvcreate_serial_standard_ssd (P2)
- verify_hot_add_disks_pvcreate_serial_premium_ssd (P2)
Parallel mode hot-adds every free LUN up to max_data_disk_count in a
single call and then runs a single-shot pvcreate on all newly attached
devices - this is the primary repro variant since the burst of
NS-Changed AENs floods udev before pvcreate opens the devices.
Serial mode adds one LUN at a time and is a baseline that should
almost always pass.
Each variant snapshots lsblk, performs the hot-add, runs 'fdisk -l',
runs pvcreate, re-snapshots lsblk to confirm the devices did not
collapse to size 0, and scans dmesg for the Identify-family bad-status
codes (0xb / 0x2 / 0xa) associated with the NVMe 1.4 CNS=02h / CNS=03h
spec violations. Cleanup always runs pvremove and detaches all
attached data disks.
Supports an optional 'max_disks_to_add' runbook variable to cap the
hot-add burst below max_data_disk_count for targeted triage; unset or
0 keeps the full-burst behaviour (highest fidelity repro).
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.
Suppressed comments (5)
lisa/microsoft/testsuites/storage/lvm_hot_add.py:399
- Avoid catching broad Exception here; prefer a narrower set (e.g. LisaException / AssertionError) so unexpected programming errors are not masked. This also keeps the 'BadEnvironmentStateException' path focused on expected detach failures.
try:
disk.remove_data_disk(disks_added)
except Exception as ex: # noqa: BLE001
# A failed detach can leak Azure managed disks and
# leave the environment in an unknown state. Raise
# BadEnvironmentStateException so LISA recycles the
# environment instead of handing a partially-cleaned
# node to the next test.
raise BadEnvironmentStateException(
f"remove_data_disk cleanup failed for {disks_added}: {ex}"
) from ex
lisa/microsoft/testsuites/storage/lvm_hot_add.py:53
- _DATA_DISK_SIZE_IN_GB and _DEFAULT_TIMEOUT are test-behavior knobs; add a brief inline rationale so future tuning is intentional and reviewable (per LISA guidelines on magic numbers).
_DATA_DISK_SIZE_IN_GB = 10
_DEFAULT_TIMEOUT = 3600
lisa/microsoft/testsuites/storage/lvm_hot_add.py:459
- The f-string passed to check_till_timeout is evaluated immediately, so if the wait times out the raised message will always show the initial empty state (not the final lsblk state). Consider catching LisaTimeoutException and re-raising with the dynamically captured state for accurate diagnostics.
check_till_timeout(
_check,
timeout_message=(
f"expected {expected_count} new disk(s) in lsblk after "
f"hot-add, found {len(state['added'])}: "
f"{[d.name for d in state['added']]}"
),
lisa/microsoft/testsuites/storage/lvm_hot_add.py:253
- Parsing max_disks_to_add via int(...) can raise ValueError/TypeError and fail the test with a less actionable traceback when the runbook variable is malformed. Raise a LisaException with the invalid value to make this easier to diagnose.
requested_cap = int(variables.get("max_disks_to_add", 0) or 0)
if 0 < requested_cap < len(free_luns):
lisa/microsoft/testsuites/storage/lvm_hot_add.py:386
- Avoid catching broad Exception in cleanup paths. Catch the specific exception types expected from tool execution so genuine programming errors still surface.
This issue also appears on line 389 of the same file.
try:
pvremove.remove_pv(*new_device_paths, ignore_errors=True)
except Exception as ex: # noqa: BLE001
log.warning(f"pvremove cleanup raised: {ex}")
AI Test Case SelectionSelected 3 test case(s): verify_dm_cache_setup,verify_disk_with_nobarrier,verify_disk_with_fio_verify_option Marketplace image: Result: Succeeded |
AI Test Case SelectionSelected 3 test case(s): verify_dm_cache_setup,verify_disk_with_nobarrier,verify_disk_with_fio_verify_option Marketplace image: Result: Succeeded |
Description
Adds a new functional storage suite,
LvmHotAddSuite, that stresses hot-add of data disks followed by LVMpvcreateto reproduce a class of NVMe-VF initialization failures where the guest kernel logs:on a freshly attached namespace, leaving
pvcreateto fail withError reading device ...andlsblkto report the affected devices with size 0.The suite covers four permutations:
verify_hot_add_disks_pvcreate_parallel_standard_ssdverify_hot_add_disks_pvcreate_parallel_premium_ssdverify_hot_add_disks_pvcreate_serial_standard_ssdverify_hot_add_disks_pvcreate_serial_premium_ssdEach variant snapshots
lsblk, performs the hot-add, runsfdisk -l, runspvcreate, re-snapshotslsblkto confirm the devices did not collapse to size 0, and scansdmesgfor the Identify-family bad-status codes (0xb/0x2/0xa) associated with NVMe 1.4 CNS=02h / CNS=03h. Cleanup always runspvremoveand detaches every attached data disk.Mode differences
max_data_disk_countin one call, then runs a single bulkpvcreate a b c ...so LVM opens every device in one syscall burst. This is the primary repro path — the burst of NS-Changed AENs floods udev before pvcreate opens the devices.pvcreate <dev>loop so each device gets its own exit code / stderr. Baseline that should almost always pass; useful for isolating which disk fails.Design notes
lsblk,fdisk -llisting,pvcreateexit 0, and post-pvcreatelsblkzero-size / missing-device check.simple_requirement(disk=Disk*SSDLRS(max_data_disk_count=IntRange(min=1)))so it filters at capability level and leaves all LUNs free for the hot-add burst — usingdata_disk_countwould burn LUN 0..N-1 with placeholder disks at deploy time and shrink the burst.max_disks_to_addvariable (read fromvariables) to cap the hot-add burst belowmax_data_disk_countfor targeted triage. Unset or0keeps the full-burst behaviour (highest fidelity repro).Related Issue
None.
Type of Change
Checklist
Test Validation
Key Test Cases:
verify_hot_add_disks_pvcreate_serial_premium_ssd|verify_hot_add_disks_pvcreate_parallel_premium_ssd|verify_hot_add_disks_pvcreate_serial_standard_ssd|verify_hot_add_disks_pvcreate_parallel_standard_ssd
Impacted LISA Features:
Disk, DiskPremiumSSDLRS, DiskStandardSSDLRS
Tested Azure Marketplace Images:
Test Results
All matrix cells validated with
max_disks_to_add=5(runbook cap; nativemax_data_disk_countper SKU noted for reference). Every cell passes both the standard-SSD and premium-SSD variants — nonvme <ctrl>: Identify ... failedmessages were observed indmesgon any cell.Total: 12 / 12 test cases PASSED across 6 SKU/mode combinations spanning three VM generations (v5, v6, v7), both AMD (
D*a*_v5) and Intel (D*_v6,E*_v7) platforms, and two guest distros (Ubuntu 24.04, SLES 15 SP7). Both storage attach paths are exercised: SCSI (/dev/sd*) on v5 and NVMe (/dev/nvme0n*) on v6/v7. Runbook capmax_disks_to_add=5is honored on every SKU (native caps 16/24/32 → 5).Individual test durations range from 152 sec (parallel, premium) to 355 sec (serial, premium); wall time per config also includes deployment and cleanup.