feat: add --split-exp-no-overwrite flag to refuse overwriting existing files#2717
Open
toller892 wants to merge 1 commit into
Open
feat: add --split-exp-no-overwrite flag to refuse overwriting existing files#2717toller892 wants to merge 1 commit into
toller892 wants to merge 1 commit into
Conversation
…g files When using --split-exp to split documents into per-file outputs, yq silently overwrites any pre-existing files at the target paths because os.Create truncates. For workflows that generate filenames from input data (e.g. '.metadata.name + ".yml"'), this can clobber unrelated files when two documents map to the same name, or when a target path collides with something already on disk. This change adds an opt-in --split-exp-no-overwrite flag (and a matching yqlib constructor NewMultiPrinterWriterWithOptions) that uses O_WRONLY|O_CREATE|O_EXCL so existing files are left untouched and yq exits with a clear error message instead. The default behaviour (overwrite) is unchanged; the original NewMultiPrinterWriter constructor still exists and now delegates to the new options-aware constructor with noOverwrite=false. Fixes mikefarah#2028
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds an opt-in
--split-exp-no-overwriteflag that makes--split-exprefuse to overwrite any pre-existing file at a target path, instead failing fast with a clear error message. The existing default behaviour (silent overwrite viaos.Create) is unchanged.Closes #2028.
Why
When yq splits documents into per-file outputs, file names are usually derived from the input data itself, e.g.:
yq -s '.metadata.name + "_" + .kind' all.yamlIf two input documents map to the same name, or if a target path happens to already exist on disk, the second write silently clobbers whatever was there because
os.Createtruncates. That's a footgun for CI pipelines and bulk-conversion scripts where the user would much rather see an error and fix the expression than discover a missing/overwritten file later.The issue author asked for an explicit opt-in flag rather than changing the default, which is what this PR does.
How
Added a new yqlib constructor
NewMultiPrinterWriterWithOptions(expression, format, noOverwrite)that carries the new option.The original
NewMultiPrinterWriteris preserved and now just delegates to the new constructor withnoOverwrite=false, so the public API is fully backwards compatible.When
noOverwriteis true, the writer usesos.OpenFile(name, O_WRONLY|O_CREATE|O_EXCL, 0600). If the target exists, yq returns:and leaves the existing file's contents untouched.
Wired up a new
--split-exp-no-overwritepersistent flag on the root command, defaulting tofalse.Updated the
--helpsnippet inREADME.mdto include the new flag.Tests
Added
pkg/yqlib/printer_writer_test.gocovering three scenarios:--no-overwriterefuses existing files —GetWriterreturns an error containing"refusing to overwrite", and the pre-existing file's contents are verified to be untouched.--no-overwritestill creates new files — confirms the flag doesn't break the normal "first write" path.All existing tests pass:
Manual smoke test