Skip to content

Wipe: pick the erase primitive from device class, fail loud when it doesn't run#127

Merged
mastacontrola merged 2 commits into
masterfrom
secure-wipe-device-class
Jul 16, 2026
Merged

Wipe: pick the erase primitive from device class, fail loud when it doesn't run#127
mastacontrola merged 2 commits into
masterfrom
secure-wipe-device-class

Conversation

@mastacontrola

Copy link
Copy Markdown
Member

Fixes #125.

FOG's NVMe wipe has not erased anything since Feb 2023, and would report success either way. Both halves are fixed here, and the NVMe sanitize path is now validated on real hardware.

The no-op erase

fog.wipe ran nvme format $hd --force with no --ses. That field defaults to 0 — "no secure erase operation requested" — so the controller reformats namespace metadata and is under no obligation to erase user data. Many drives deallocate on format and a deallocated read commonly returns zeros, which is exactly why this looked right in testing. Every NVMe wipe FOG has performed through this path should be assumed non-erasing.

This is a regression of a fix, not a missing feature. #61 (closed 2023-02-28) had the reasoning right — NVMe isn't frozen by UEFI, so nvme-cli is the correct tool — and missed one flag. The pre-#61 behaviour was shred: slow and hard on the drive, but it erased. The 2023 change traded a real wipe for a fast one and the speed was mistaken for the improvement.

Nothing checked whether the wipe ran

No branch of the case inspected an exit status. A failed format, a shred that died on a bad sector, or an unknown wipemode matching no branch at all each fell through to echo "Wiping complete." and reported success to the server. Same class as ADR-0003, worse consequence: a mis-imaged disk is found when the machine won't boot; a disk wrongly believed wiped is found by whoever finds the data on it.

Compounding it, wipemode="nvme" was forced for NVMe targets before the case, so an operator choosing full — the strongest option — got the metadata-only format. The strongest choice silently became the weakest.

What changed

The primitive is selected from device class, not a name match, in wipeDisk(). fog.wipe is reduced to countdown, one call, and handleError on non-zero.

class fast normal full
nvme format --ses=2 (crypto) if FNA bit 2, else --ses=1 format --ses=1 sanitize --sanact=2 if SANICAP bit 1, else format --ses=1
ssd / unknown dd zeros over metadata shred -n 1 + warning shred -n 3 -z + warning
hdd dd zeros over metadata shred -n 1 shred -n 3 -z

unknown is treated as possibly-flash, because the failure that matters is assuming flash is a platter, not the reverse. A bare nvme format with no --ses is never issued by any path.

Sanitize is preferred but never blindly

format --ses=1 is the floor everywhere: universally supported, synchronous, meaningful exit status, NIST SP 800-88r1 Purge. Sanitize is preferred for full because it covers over-provisioned and unmapped blocks no namespace-level op can reach — but it's optional, async, and cannot be canceled once started.

That last point drives a hard rule: once a sanitize starts, falling back to format is not allowed. The controller rejects format with 0x1d SANITIZE_IN_PROGRESS, so the fallback cannot succeed — attempting it converts a working wipe into a reported failure. nvmeSanitize() returns 0 (confirmed complete), 1 (nothing running, fallback legal), or 2 (started, unconfirmable). Only rc 1 falls back. rc 2 means the drive is unknown, not unwiped, so FOS claims neither outcome and points the operator at nvme sanitize-log. SSTAT 3 needs sanitize --sanact=1 Exit Failure Mode before format is legal at all, since 0x1c is defined as "no recovery action has been successfully completed".

Hardware validation

Tested on a real NVMe drive. Old build:

Starting disk wipe of /dev/nvme0n1 using nvme format …
Success formatting namespace:1
Wiping complete.

Seconds, no erase — the SES=0 no-op reproducing exactly. New build:

Erasing /dev/nvme0n1 with an NVMe sanitize block erase (sanact 2)
Sanitize cannot be canceled once started; it will resume after a power cycle.
Sanitizing /dev/nvme0n1 … 100%
Wiping complete.

An earlier hardware run caught a real bug this PR also fixes: the poll parsed .sstat expecting a flat {"sstat":2,...}, but nvme-cli nests under a device-name key, makes sstat an object, and renders status as a string. It bailed on poll 1, fell back to a format the controller rejected, and told the operator "This disk still holds its data" about a drive that was sanitizing correctly. The stub had encoded the same wrong assumption as the parser, so 24/24 tests passed against code that could never work on hardware. The stub is now transcribed from json_sanitize_log() in nvme-cli's nvme-print-json.c, and the harness requires real jq.

Verification

tests/checks/wipe.sh — 29 passed, 0 failed. Negative controls confirm each fix is load-bearing and independent: bare nvme format --force fails 6, swallowing format's exit status fails 2, swallowing shred's fails 1, moving mode validation behind the class dispatch fails 4, reverting the sanitize-log parse fails 5, allowing the post-sanitize fallback fails 3.

Validated: the NVMe full sanitize happy path, end to end on hardware.
Not exercised on hardware: the format fallback paths, SSTAT 3 recovery, rc 2, and the SATA/HDD paths. These are covered by stubs only.

Known gap: SATA SSD is disclosed, not fixed

shred on a SATA SSD is not a guaranteed erase, for the same wear-levelling reason the format wasn't. The right primitive is ATA SANITIZE, and it isn't here. #40 (2020) tried and found drives frozen despite advertising enhanced erase; #61 (2023) reached the sharper conclusion that the standard unfreeze is suspend-to-RAM and FOS has no suspend to offer — it's a login shell in an initramfs, not a system with a power-management stack. That's an open problem, not an unwritten function. So SATA SSD users now get a loud warning that the overwrite may leave data recoverable and that the drive's own secure erase is the remedy. Rotational disks deliberately don't get the warning: overwriting genuinely erases them, and a warning shown everywhere is one operators learn to ignore.

Consequences

  • NVMe wipes now take real time. Minutes for format, potentially much longer for sanitize, where the old no-op returned in seconds. That speed was the bug.
  • full on NVMe may be uncancelable once sanitize starts. The countdown is the cancel window.
  • A failed wipe is now a failed task. Runs that previously reported success while erasing nothing will now handleError. These were always failures; they'll now look like new ones.
  • Unknown/empty wipemode refuses rather than silently doing nothing. Deliberately not defaulted to normal — guessing a destructive action on malformed input is its own hazard. The legacy fastwipe alias still works.
  • No FOG-server change needed. Mode values and the Post_Wipe.php contract are unchanged.

🤖 Generated with Claude Code

mastacontrola and others added 2 commits July 16, 2026 06:25
…oesn't run

fog.wipe forced wipemode="nvme" for any NVMe target and ran
`nvme format $hd --force`. The Secure Erase Settings field (--ses)
defaults to 0 -- "no secure erase operation requested" -- so that
reformats the namespace's LBA metadata and returns in seconds without
the spec requiring the controller to erase user data. Drives commonly
deallocate on format and deallocated reads commonly return zeros, which
is why this looked correct; that is implementation-defined per drive,
and "reads back as zeros" is not "the data is gone from the NAND".
Every NVMe wipe performed through this path should be assumed
non-erasing.

This is a regression of a fix, not a never-implemented feature. The NVMe
path came from #61 (closed as fixed 2023-02-28), whose reasoning was
sound -- "UEFI does not freeze an NVMe. NVMe drives have their own way to
format using nvme-cli" is correct, and is still the basis of what this
commit does. The approach was right; one flag was missing. The pre-#61
code shredded NVMe: slow, hard on the drive, and the reason #61 was
opened -- but it erased. The 2023 change traded a real wipe for a fast
one and the speed was mistaken for the improvement. Believed fixed for
three years.

Nothing checked whether the wipe ran, either. No branch of the case
inspected an exit status, so a failed format, a shred that died on a bad
sector, or an unknown wipemode matching no branch all fell through to
`echo "Wiping complete."` and reported success to the server. Same class
of silent failure ADR-0003 removed from the partition path, with a worse
consequence: a mis-imaged disk is found when the machine won't boot, a
disk wrongly believed wiped is found by whoever finds the data on it.

The mode override compounded it: forcing "nvme" ahead of the case meant
an operator choosing `full`, the strongest option offered, got the
metadata-only format instead of the 3-pass shred.

The primitive is now selected from the device class in wipeDisk(), and
fog.wipe reduces to countdown, one call, handleError on non-zero:

  nvme        fast   -> format --ses=2 (crypto) if FNA bit 2, else --ses=1
              normal -> format --ses=1
              full   -> sanitize --sanact=2 if SANICAP bit 1, else --ses=1
  ssd/unknown fast   -> dd zeros over metadata
              normal -> shred -n 1, with a not-a-guaranteed-erase warning
              full   -> shred -n 3 -z, same warning
  hdd         same as ssd without the warning (overwrite genuinely erases)

format --ses=1 is the floor everywhere: universally supported,
synchronous, meaningful exit status, and NIST SP 800-88r1 Purge for NVMe.
sanitize is preferred for full because it covers over-provisioned and
unmapped blocks no namespace-level operation can reach, but it is gated
on a SANICAP probe issued *before* sanitize starts, since sanitize cannot
be canceled once begun. Falling back to --ses=1 is safe because both are
real erases: the fallback degrades thoroughness, never to "no erase".

A bare `nvme format` with no explicit --ses is never issued by any path.

The SATA SSD path still overwrites and is disclosed rather than fixed.
shred on a SATA SSD is not a guaranteed erase (wear levelling, over-
provisioning), but removing it would leave those users with less than
they have today, and the ATA SANITIZE alternative is blocked by more than
effort: #40 (2020) found drives frozen despite advertising enhanced
erase, and #61 (2023) established that the standard unfreeze is a
suspend-to-RAM cycle which FOS cannot perform, being a login shell in an
initramfs rather than a system with a power-management stack. That is an
open problem, not an unwritten function. Operators get an explicit
warning instead; anyone picking it up should start at #40 and #61.

An unknown or empty wipemode now refuses rather than doing nothing and
reporting success; it is deliberately not defaulted, since guessing a
destructive action on malformed input is its own hazard. The legacy
fastwipe alias is still accepted. The server only sends fast/normal/full
(commons/schema.php), so no FOG-server change is needed.

tests/checks/wipe.sh locks the mapping with 24 cases. Four negative
controls confirm each fix is load-bearing: restoring the bare
`nvme format --force` fails 6, swallowing the format exit status fails 2,
swallowing shred's fails 1, and moving the mode validation back behind
the class dispatch fails 4. That last guards a real ordering constraint:
nvmeSecureErase() treats an unrecognised mode as "neither full nor fast"
and issues --ses=1, so with validation after the dispatch an unknown mode
refused on /dev/sda but erased /dev/nvme0n1.

Not yet validated on real hardware. The harness stubs nvme-cli, so it
proves which commands FOS issues and how it reacts to exit statuses; it
cannot prove a physical drive erases, nor that the sanitize-log JSON
field names match every nvme-cli build. The sanitize polling loop in
particular has never run against a real controller.

See docs/adr/0008-secure-wipe-by-device-class.md.

Refs #125, #61, #40

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…nitize starts

The first hardware test of the NVMe full wipe failed, and the failure message
was backwards: the operator was told "This disk still holds its data" about a
drive that was almost certainly sanitizing correctly.

Two defects, both fixed here.

The sanitize poll parsed `jq -r '.sstat // empty'`, expecting a flat
{"sstat":2,"sprog":32768}. nvme-cli emits three things different at once:

  {"/dev/nvme0":{"sprog":32768,"sstat":{"status":"(2) Sanitize in Progress."}}}

nested under a device-name key, sstat an object, status a string with the code
in parens. So .sstat was null on the very first poll and nvmeSanitize() bailed.
nvmeSanitizeStatus() now reads `.[] | .sstat.status` and extracts the leading
int, transcribed from json_sanitize_log() in nvme-cli's nvme-print-json.c
(identical in 2.15, which FOS ships, and 2.16). id-ctrl really is flat, so the
sanicap/fna probes were already correct and are unchanged.

Having bailed, the code then fell back to `format --ses=1`, which a controller
with a sanitize in progress rejects with 0x1d - the fallback could not have
succeeded. nvmeSanitize() now returns 0 (confirmed complete), 1 (nothing
running, fallback is legal) or 2 (started but unconfirmable). Only rc 1 falls
back. A sanitize cannot be canceled and resumes across power cycles, so rc 2
means the drive's state is unknown, not unwiped: FOS claims neither outcome and
tells the operator to read `nvme sanitize-log` on the drive itself, where SSTAT
1 or 4 is proof of completion. SSTAT 3 (failed) needs `sanitize --sanact=1`
Exit Failure Mode before format is legal at all, since 0x1c is defined as "no
recovery action has been successfully completed"; if that clear fails, the wipe
refuses rather than guessing.

Also surface nvme format's stderr, which was going to /dev/null - the tester's
report would have named 0x1d directly had it been shown.

The tests deserve the blame here and are rebuilt. The old stub printed the flat
shape I *believed* nvme-cli emitted, so 24/24 passed against a parser that could
never work on hardware: the stub and the parser shared one author and one
misreading, and the suite only ever confirmed the code agreed with me. The stub
is now transcribed from the emitting source, and the harness hard-requires jq
instead of a sed shim that would reintroduce the same half-right-parser hazard.

29 passed, 0 failed. Negative controls confirm both fixes are load-bearing and
independent: reverting the parse fails 5 cases and reproduces the tester's call
log exactly (id-ctrl, sanitize, one poll, stop); allowing the post-sanitize
format fallback fails 3 different cases.

ADR-0008 is updated with the rc contract, the fallback rules, and a post-mortem
replacing its old "not yet validated on hardware" caveat - which predicted this
exact failure and is kept rather than quietly dropped. Still outstanding: the
corrected loop has reasoned semantics and passing tests, but has still never
completed a sanitize end to end on a real controller, which is what the last
version of it also had.

Refs #125

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@mastacontrola
mastacontrola merged commit 293eb77 into master Jul 16, 2026
9 checks passed
@mastacontrola
mastacontrola deleted the secure-wipe-device-class branch July 16, 2026 18:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

NVMe wipe erases nothing: nvme format defaults to SES=0, and every wipe reports success regardless

1 participant