Repository navigation
Make URL normalization idempotent - #48
Open
binggao1230 wants to merge 1 commit into
Open
binggao1230 wants to merge 1 commit into
binggao1230 wants to merge 1 commit into
Conversation
NormalizeURL ran the flag pass exactly once, so Normalize(Normalize(x))
could differ from Normalize(x) for the recommended greedy flag sets:
- removeTrailingSlash stripped only one trailing slash, so a path ending
in two or more slashes needed several passes to fully collapse; and
- flags such as removeDirectoryIndex run before removeTrailingSlash, so a
trailing slash (or a %2F decoded to one) could hide a segment that a
later pass would then remove.
Strip every trailing slash in removeTrailingSlash (a root "/" still
collapses to ""), and iterate the flag pass in NormalizeURL to a bounded
fixed point. The converged value equals the one repeated normalization
already reached, so no existing behaviour changes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
NormalizeURLString/NormalizeURLare not idempotent for the library's own recommended greedy flag sets:Normalize(Normalize(x)) != Normalize(x). Since purell exists to produce stable dedup/cache keys, idempotence is a defining property, and the value a second pass reaches is exactly the one already intended (all trailing slashes stripped) — so this is a convergence bug, not a behaviour change.Two causes, both in
purell.go:removeTrailingSlashstripped a single trailing slash, so a path ending in two or more slashes needed several passes to collapse:http://host//→http://host/→http://host. This breaks idempotence with the standaloneFlagRemoveTrailingSlash.NormalizeURLapplied the flag pass once. Flags such asremoveDirectoryIndexrun beforeremoveTrailingSlash, and%2Fdecodes to/on parse, so a trailing slash can hide a segment a later pass would remove:http://host/index.html/(UnsafeGreedy) →http://host/index.html→http://host.Minimal repro:
Fix:
removeTrailingSlashstrips every trailing slash (a root/still collapses to""), andNormalizeURLiterates the flag pass to a bounded fixed point.The converged output equals the value repeated normalization already reached, so no existing expected value changes — the full existing suite passes unmodified. A property fuzz over ~325k random URLs per flag set goes from:
TestNormalizeIsIdempotentcovers the standalone flag, both greedy sets, and the directory-index /%2Fsub-cases.