config: `BoolOr<T>` hides the real deserialization error behind "expected boolean or object" (e.g. unknown `reactCompiler` fields)

[PhilMeyr]

Describe the bug

Any invalid field inside an option typed BoolOrDataConfig<T> (jsc.transform.reactCompiler, jsc.minify.compress, jsc.minify.mangle, …) is reported as expected boolean or object, even though an object was passed. The real serde error (unknown field name, invalid enum variant, wrong type) is discarded, so users cannot tell which key is wrong.

Cause: BoolOr<T>::deserialize in crates/swc_config/src/types/bool_or_data.rs drops the inner error:

other => {
    T::deserialize(other)
        .map(BoolOr::Data)
        .map_err(|_| serde::de::Error::custom("expected boolean or object"))
}

This is especially confusing for reactCompiler, whose structs use #[serde(deny_unknown_fields)]: a React Compiler option that swc does not expose (e.g. copied from the Babel plugin docs) produces this message instead of unknown field ..., expected one of ....

Input code

// npm i @swc/core@1.16.12
const swc = require('@swc/core');

const cases = {
  'unknown environment field': { reactCompiler: { environment: { enableTreatFunctionDepsAsConditional: true } } },
  'unknown top-level field': { reactCompiler: { compilationMode: 'infer', typo: true } },
  'invalid enum value': { reactCompiler: { compilationMode: 'bogus' } },
};
for (const [name, transform] of Object.entries(cases)) {
  try {
    swc.transformSync('export default function A() { return <div />; }', {
      filename: 'a.jsx',
      jsc: { parser: { syntax: 'ecmascript', jsx: true }, transform },
    });
  } catch (e) {
    console.log(`${name}: ${e.message.split('Caused by:')[1].trim()}`);
  }
}

// Same class outside the React Compiler:
for (const minify of [{ compress: { typo: true } }, { mangle: { typo: true } }]) {
  try {
    swc.transformSync('export const a = 1', { filename: 'a.js', minify: true, jsc: { minify } });
  } catch (e) {
    console.log(`${Object.keys(minify)[0]}: ${e.message.split('Caused by:')[1].trim()}`);
  }
}

Config

See the options passed inline in the input code.

Link to the code that reproduces this issue

N/A (self-contained script above)

SWC Info output

@swc/core@1.16.12 (Linux x64, Node 24.21.0). Also reproduced through @rspack/core@2.2.7 (experiments.swc.transformSync, builtin:swc-loader). bool_or_data.rs is unchanged on main (68586f07feb1).

Expected behavior

The underlying serde error is surfaced when an object was provided, e.g.:

unknown field `enableTreatFunctionDepsAsConditional`, expected `enableFunctionOutlining`
unknown field `typo`, expected one of `compilationMode`, `panicThreshold`, ...
unknown variant `bogus`, expected one of `infer`, `syntax`, `annotation`, `all`

and expected boolean or object is kept only for values that are neither (string, number, array, null).

Actual behavior

unknown environment field: expected boolean or object at line 1 column 164
unknown top-level field: expected boolean or object at line 1 column 142
invalid enum value: expected boolean or object at line 1 column 130
compress: expected boolean or object at line 1 column 77
mangle: expected boolean or object at line 1 column 75

Version

1.16.12

Additional context

Suggested fix, in BoolOr<T>::deserialize: forward the inner error for objects and keep the generic message for other value types:

match value {
    Value::Bool(b) => Ok(BoolOr::Bool(b)),
    Value::Object(map) if map.is_empty() => Ok(BoolOr::Bool(true)),
    obj @ Value::Object(_) => T::deserialize(obj)
        .map(BoolOr::Data)
        .map_err(serde::de::Error::custom),
    other => T::deserialize(other)
        .map(BoolOr::Data)
        .map_err(|_| serde::de::Error::custom("expected boolean or object")),
}

(The last arm keeps accepting Ts that deserialize from non-object values.)

A test in crates/swc_config asserting that serde_json::from_str::<BoolOrDataConfig<S>>(r#"{"typo":true}"#) for a deny_unknown_fields struct S returns an error containing unknown field would lock this in.

Context: we hit this while trying the React Compiler workaround discussed in react/react#35762. That option no longer exists upstream, so this issue is only about the error message, not about exposing the option.