Repository navigation
json 3.0 compatibility: json 3 semantics, drop-in follows the installed json - #7
Merged
Merged
Conversation
json 3.0 rejects duplicate object keys by default (allow_duplicate_key:
true restores last-wins) and rejects lone trailing surrogates, not only
leading ones. NOSJ now does the same everywhere it accepts documents:
- RubyValueSink (parse, load_file, dig/at_pointer materialization,
lazy values, each_line): a repeated key collapses in
rb_hash_bulk_insert, so the hash comes out smaller than the pair
count; one size read per object (parse cost measured 0-2%).
- valid? and minify/reformat build no hash: sink::DupKeys remembers a
64-bit fingerprint per key and checks each object as it closes
(pairwise up to 8 keys, a reused epoch-stamped open-addressing table
up to 128, sort beyond). A fingerprint hit is confirmed exactly, so
a collision only costs a re-run. Measured overhead on key-heavy
documents: twitter valid? +4%, minify +7%; citm/gsoc/ohai 4-12%.
Comparing key bytes instead of fingerprints measured up to 93% slower;
a leaner hand-rolled hash measured no faster (the close-time check,
not hashing, was the cost, hence the table).
- Lone surrogates: every sink's str_bytes/key_bytes refuses.
- Sinks see no offsets, so locate.rs re-walks the document with the
pull Reader on the cold path: duplicate keys report the repeating
object's '{' (json 3's position; innermost first), lone surrogates
the string. The reformat pipe's WTF-8 re-escaper is gone.
- stats keeps describing such documents (diagnostic); the fuzz
reference checks now treat it as accepting a superset.
Specs pinned to the json-2-era behavior are rewritten, and valid?
coverage exercises all three close strategies.
json 3 raises GeneratorError ("detected duplicate key \"a\" in {...}")
when two keys of a hash render the same, e.g. "a" and :a, or 1 and "1"
(json 2.21 only warns). It keeps the check cheap: keys of one kind
cannot collide, so only a String or Symbol key in a hash whose first key
had another kind triggers the full check (every key's to_s into a set).
NOSJ mirrors that rule exactly, including what it does not catch (two
objects with the same to_s, compare_by_identity string keys), and the
message; allow_duplicate_key: true emits them as before. Rails-mode
configs keep ActiveSupport's own key handling.
Per pair: the key's kind is computed once and shared with the key-cache
lookup (which tested the same flags), and the tracker costs one byte
compare; a hash's first key one more; only actual mixing goes out of
line, into a cold check using protected to_s/inspect. Measured non-PGO
before/after: twitter +1.3%, citm +1.7%, gsoc -2%, activitypub +1.5%;
ohai (many tiny hashes) about +7%, to be judged by the PGO sweep. All
cases checked against json 3.0.2 byte-for-byte.
json 3 made JSON.parse keyword-only, fixed JSON.dump's defaults
({allow_nan: true}, nesting capped at 100), removed _dump_default_options,
and raises for options json 2 ignored (quirks_mode, escape_slash,
create_additions, anything unknown). The drop-in now picks the installed
version's conventions at require time: a **opts parse on json 3 (so a
positional options hash raises exactly as json 3 does), the version's dump
defaults, and fast-path option lists without the json-2-only keys.
Semantics follow the same way. NOSJ implements json 3's (duplicate keys,
lone surrogates and comments are errors), so whenever the fast path
refuses a call, the installed gem runs it again and has the last word:
json 2 accepts what it always accepted, and every exception is the gem's
own (message, json_path, invalid_object) instead of NOSJ's message
re-raised as a JSON class. The second pass costs failures only.
Also fixes JSON.dump raising NameError under json older than 2.11 (the
json Ruby 3.3 and 3.4 bundle), which has no _dump_default_options; json
2.10 and older also accept only strict: in dump's options hash, so an
options hash takes the gem's dump there.
The new differential spec compares every case with the installed gem's
own outcome, so it holds on either version.
json 3 raises ArgumentError for any options key it does not know
("unknown keyword: foo", or "unknown keywords: a, b"). NOSJ used to
ignore them, so a typo or an option nosj lacks (sort_keys, on_load,
allow_comments) silently did nothing. Every entry point now follows json
3, through one reader shared by the parse and generate decoders
(opt_reader.rs): it counts the keys it finds, so a clean hash costs one
length compare, and only a shortfall walks the hash to name the
leftovers. Reformat reads both option sets through one reader, so a key
either set knows is accepted.
json options nosj does not implement (object_class, array_class,
decimal_class, on_load, create_additions, allow_comments,
allow_control_characters, allow_invalid_escape, sort_keys, as_json) are
accepted while falsy, since their default is nosj's behavior, and raise
otherwise. They are checked only on that cold path, so the hot path now
does fewer lookups than before (parse: 6, was 9). escape_slash, which
json 3 removed, is gone; quirks_mode is no longer ignored. An empty
options hash returns early (json 3's keyword parse hands the drop-in one
per call).
The drop-in follows whichever json is installed and the parity specs compare against it, so the whole suite also runs with json ~> 3.0 (Ruby 3.3 and 4.0). NOSJ_JSON_VERSION pins json in the Gemfile; a pinned run leaves out the development group, since rubocop (under standard) requires json 2.x, and the Rakefile tolerates standard's absence.
The cold-path duplicate search re-indexed every nested container as its own slice (a fresh Reader and stage-1 pass per container, so bytes times depth: a repeat deep inside a large document cost up to max_nesting parses), recursed, and gave up past 4096 levels. valid? and minify read that give-up as "no repeat after all, a fingerprint collision" and accepted real duplicates nested that deep under max_nesting: false (parse, whose hash-size check is exact, raised). Now one Reader walks the whole document with an explicit stack of open containers, stops at the first object to close with a repeated key and records the member indices down to it; a second partial walk follows that path, skipping earlier siblings, and skips the container itself to get its offset. Both linear, neither recursive, so every depth is reachable. The result has three states: found, absent (a collision), and undecided (the Reader stopped first), which callers refuse. The lone surrogate walk is the same loop without key tracking, so it now positions refusals at any depth too.
Three drivers mapped sink aborts onto Ruby exceptions separately: finish_drive, the reformat pipe (with its own TooDeep, LoneSurrogate and GeneratorError arms plus a catch-all) and stats (TooDeep plus a catch-all), and valid? and minify each wrote out the confirm-or-retry protocol for fingerprint matches. Now parse.rs has drive_error, the one mapping (NonFiniteFloat now GeneratorError there, the only thing it could be), and drive_hashless, the one protocol: drive with the check, confirm a refusal exactly, drive again without it after a collision. valid? is `drive_hashless(..).is_ok()`, the pipe and stats map_err through drive_error, and the two error constructors fold into it.
Every option lookup built its Symbol with Ruby::to_symbol, which
allocates a String and runs rb_to_symbol under protect: a parse with
{symbolize_names: true} allocated 10 objects per call, a generate with
four formatting options 14 (main). Ruby::sym_new interns to a
StaticSymbol without allocating, and once every key of the hash has
been found the remaining reads are absent without a lookup, unless the
option was found already (reformat reads a few twice). Both now
allocate only their result.
Also: `read` was derivable (leftover keys are present, and a present
key's read bit equals its found bit), opt_bool duplicated truthy (every
default is false), and opt_bytes repeated each option's name, which
Opt::name already knows.
stats borrowed parse's decoder, so under the unknown-key rule it still accepted symbolize_names, freeze and allow_duplicate_key, which it ignores, and it looked max_nesting up a second time (allocating a Symbol) only to learn whether the key was present. It now reads exactly the options it documents through its own OptReader, where get's Option already says whether max_nesting was given; the max_nesting decoding is shared with parse as max_nesting_of. Decoding also moves ahead of the source borrow in stats and stats_file: it ran inside stats_over while the input slice was held, and a hash lookup can run a key's own hash/eql?, which the memory-safety rules keep away from borrowed source bytes.
DEFAULT_CONFIG, the four Rails statics and Default each spelled out all 13 fields (so adding allow_duplicate_key took six edits), with the nesting limit as a bare 100. They now come from one const fn over the two things that vary, the Rails walk and the escape mode, with MAX_NESTING named. (Struct-update syntax from a base const is not allowed in statics for a type with Vec fields.)
MixedKeys packed "no key yet" into a u8 by hand (a repr(u8) enum with discriminants from 1, a NO_KEY_YET constant, `as u8` casts), which is exactly what Option<KeyKind> already is: one byte, one compare. The gem's key coercion (Strings as they are, Symbols by name, else to_s) was written twice, in emit_key and in the duplicate check, and emit_key re-derived the type the callers had already classified. Now key_string holds the rule, emit_key takes the caller's kind, and the duplicate check calls key_string too.
The fingerprint stack and the close-time table always travel together, yet PullState held them as two fields and DupKeys took two &mut borrows at every construction. One DupScratch owns both; SeenTable is private to sink.rs again.
quirks_mode can only reach the fast path on json 2 (json 3's option list leaves it out), yet every parse on either version tested for it, and stripping it allocated a Hash per call for Rails 7.x's lone `quirks_mode: true`. The strip now sits in json 2's parse and returns nil for that case. The dump defaults' reader was chosen per call with a respond_to? check; it is now chosen once, by defining dump_defaults per version, next to the other load-time decisions. A new spec pins the fast-path option lists to what NOSJ reads: a listed key NOSJ refused would escape as ArgumentError, since the drop-in hands the gem only parse and generate refusals.
NOSJ refuses that pair with ArgumentError (no crate escape mode does both), json supports it, and the fast path let it through, so JSON.generate(obj, ascii_only: true, script_safe: true) raised under the drop-in (also on main). The drop-in hands the gem parse and generate refusals, not option errors, so the pair now skips the fast path: generate_supported? is the one check generate, pretty_generate and dump share.
Only standard pins json to 2.x (through rubocop); rbs, yard and lefthook have no json dependency, so the json-compat bundle keeps them. The Rakefile skips standard's tasks on the same variable instead of swallowing any LoadError, which would also have hidden a broken development checkout.
patch_case skipped specs holding broken strings and documents whose stats key count differed from the tree's (duplicate keys). The prelude parses with default options, which now refuse lone surrogates and duplicate keys, so neither guard could fire any more; the second cost a stats pass on every patch case. Both go, with their helpers.
parse, reformat and valid specs each built the same k1..kN object by hand, twice more with a repeated key appended.
json 3 raises "detected duplicate key #{key.inspect} in #{hash.inspect}"
and so does NOSJ, but the spec spelled out Ruby 3.4's Hash#inspect
({"a" => 1, a: 2}); Ruby 3.3 prints {"a"=>1, :a=>2}, failing CI on 3.3.
The expectation now uses #inspect too.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Matches the json gem 3.0 and stays compatible with both the json 2.x and 3.0 interfaces:
NOSJ.*follows json 3.0 semantics, while thenosj/jsondrop-in follows whichever json the application has installed. Intended as 0.5.0 (behavior changes for input that was accepted before; see the CHANGELOG's[Unreleased]section).NOSJ's own API: json 3.0 semantics
NOSJ::ParserError, positioned at the repeating object's{like json 3, inparse,load_file,valid?(false),minify/reformat, and every entry point that materializes values.allow_duplicate_key: truerestores last-key-wins (minifypasses repeats through).generateraises json 3's exactdetected duplicate key ...GeneratorError for keys that render alike ("a"and:a), using json 3's own trigger (only a String/Symbol key in a hash whose first key had another kind). The Rails encoder is unchanged.unknown keyword: foo) in every entry point. Unimplemented json options (object_class,on_load,allow_comments,sort_keys, ...) raise unless falsy.escape_slashis gone,quirks_moderefused, andstatstakes only its documented options.The drop-in follows the installed json
JSON.parse(a positional options hash raises like json 3), json 3'sdumpdefaults, and whatever json 3 refuses raises exactly as json 3 does.json_path,invalid_object). The second pass costs failures only.main):JSON.dumpraised NameError with json older than 2.11, i.e. Ruby 3.3/3.4's bundled json;ascii_onlytogether withscript_saferaised ArgumentError.How
valid?andminifybuild no hash: they keep 64-bit key fingerprints checked when each object closes (pairwise, epoch table, or sort by object size); a refusal is confirmed exactly, and a fingerprint collision redoes the pass without the check.locate.rs): two linear, iterativeReaderpasses with no depth limit.OptReaderserves the parse and generate decoders: interned symbols, and no lookups once every key is found. A parse with options allocated 10 objects per call onmain, now 1; a generate with four options 14, now 1.Verification
json-compatruns the whole suite against json~> 3.0on Ruby 3.3 and 4.0 (NOSJ_JSON_VERSIONpins json; onlystandard, whose rubocop needs json 2.x, drops out).main(fresh profiles on both sides, interleavedbench_sweeprounds): parity OK, 13/13 + 13/13 wins, parse and generate at parity.valid?is 9-19% andminify5-12% slower on key-heavy files (canada unchanged): the cost of duplicate detection without a hash.Not changed, to decide separately:
Lazy#keys/size/eachdon't check for duplicates;patch'stestop parses with default options; error messages inside sub-documents carry slice-relative byte numbers (as onmain). json older than 2.15 formats floats differently, so the drop-in's byte parity holds from json 2.15 on.