Skip to content

feat: add --split-exp-no-overwrite flag to refuse overwriting existing files#2717

Open
toller892 wants to merge 1 commit into
mikefarah:masterfrom
toller892:feat/split-exp-no-overwrite
Open

feat: add --split-exp-no-overwrite flag to refuse overwriting existing files#2717
toller892 wants to merge 1 commit into
mikefarah:masterfrom
toller892:feat/split-exp-no-overwrite

Conversation

@toller892
Copy link
Copy Markdown

What

Adds an opt-in --split-exp-no-overwrite flag that makes --split-exp refuse to overwrite any pre-existing file at a target path, instead failing fast with a clear error message. The existing default behaviour (silent overwrite via os.Create) is unchanged.

Closes #2028.

Why

When yq splits documents into per-file outputs, file names are usually derived from the input data itself, e.g.:

yq -s '.metadata.name + "_" + .kind' all.yaml

If two input documents map to the same name, or if a target path happens to already exist on disk, the second write silently clobbers whatever was there because os.Create truncates. That's a footgun for CI pipelines and bulk-conversion scripts where the user would much rather see an error and fix the expression than discover a missing/overwritten file later.

The issue author asked for an explicit opt-in flag rather than changing the default, which is what this PR does.

How

  • Added a new yqlib constructor NewMultiPrinterWriterWithOptions(expression, format, noOverwrite) that carries the new option.

  • The original NewMultiPrinterWriter is preserved and now just delegates to the new constructor with noOverwrite=false, so the public API is fully backwards compatible.

  • When noOverwrite is true, the writer uses os.OpenFile(name, O_WRONLY|O_CREATE|O_EXCL, 0600). If the target exists, yq returns:

    Error: refusing to overwrite existing file "file_0.yml" (--no-overwrite is set)
    

    and leaves the existing file's contents untouched.

  • Wired up a new --split-exp-no-overwrite persistent flag on the root command, defaulting to false.

  • Updated the --help snippet in README.md to include the new flag.

Tests

Added pkg/yqlib/printer_writer_test.go covering three scenarios:

  1. Default behaviour — existing files are still silently truncated/overwritten (regression guard).
  2. --no-overwrite refuses existing filesGetWriter returns an error containing "refusing to overwrite", and the pre-existing file's contents are verified to be untouched.
  3. --no-overwrite still creates new files — confirms the flag doesn't break the normal "first write" path.

All existing tests pass:

$ go test ./...
ok  	github.com/mikefarah/yq/v4	1.385s
ok  	github.com/mikefarah/yq/v4/cmd	0.620s
ok  	github.com/mikefarah/yq/v4/pkg/yqlib	1.049s
ok  	github.com/mikefarah/yq/v4/test	1.806s

Manual smoke test

$ printf 'a: 1\n---\nb: 2\n' | yq -s '"file_" + $index'
$ ls
file_0.yml  file_1.yml

# Default: silent overwrite (existing behaviour)
$ printf 'X: 9\n---\nY: 8\n' | yq -s '"file_" + $index'
$ cat file_0.yml
X: 9

# With --split-exp-no-overwrite: fails fast, leaves file alone
$ printf 'A: 7\n---\nB: 6\n' | yq -s '"file_" + $index' --split-exp-no-overwrite
Error: refusing to overwrite existing file "file_0.yml" (--no-overwrite is set)
$ cat file_0.yml
X: 9    # untouched

…g files

When using --split-exp to split documents into per-file outputs, yq
silently overwrites any pre-existing files at the target paths because
os.Create truncates. For workflows that generate filenames from input
data (e.g. '.metadata.name + ".yml"'), this can clobber unrelated files
when two documents map to the same name, or when a target path collides
with something already on disk.

This change adds an opt-in --split-exp-no-overwrite flag (and a
matching yqlib constructor NewMultiPrinterWriterWithOptions) that uses
O_WRONLY|O_CREATE|O_EXCL so existing files are left untouched and yq
exits with a clear error message instead.

The default behaviour (overwrite) is unchanged; the original
NewMultiPrinterWriter constructor still exists and now delegates to the
new options-aware constructor with noOverwrite=false.

Fixes mikefarah#2028
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

--split-exp should have an option to refuse overwriting existing files.

1 participant