Skip to content

fix(core): decode UTF-16 BOMs in readJson - #54307

Closed
reynold-hu wants to merge 1 commit into
anomalyco:devfrom
reynold-hu:import-utf16-bom
Closed

reynold-hu wants to merge 1 commit into
anomalyco:devfrom
reynold-hu:import-utf16-bom

Conversation

@reynold-hu

Copy link
Copy Markdown

Issue for this PR

Closes #54306

Type of change

  • Bug fix
  • New feature
  • Refactor / code improvement
  • Documentation

What does this PR do?

FSUtil.readJson decoded every file as UTF-8. JSON written as UTF-16 with a BOM — the default output of Windows shell redirection, e.g. PowerShell 5.1 opencode export <id> > session.json — turned into replacement characters and failed with Invalid JSON ... Unrecognized token.

It now decodes BOM-prefixed UTF-16LE/UTF-16BE files and otherwise keeps the previous behavior (UTF-8, with a UTF-8 BOM stripped). UTF-16 without a BOM stays unsupported because the encoding is ambiguous.

How did you verify your code works?

  • Added tests in packages/core/test/filesystem/filesystem.test.ts covering UTF-16LE, UTF-16BE and UTF-8 BOM files; they fail before the change and pass after.
  • End-to-end on Windows 10: bun dev export <id> > session.json then bun dev import session.json prints Imported session: ...; before the change it failed with Invalid JSON ... Unrecognized token.
  • bun typecheck in packages/core passes.

Screenshots / recordings

Not a UI change.

Checklist

  • I have tested my changes locally
  • I have not included unrelated changes in this PR

Windows shells write redirected output as UTF-16 with a BOM (PowerShell 5.1 uses UTF-16LE), which made 'opencode import' reject JSON exported from the same shell. Decode BOM-prefixed UTF-16LE/BE files; UTF-8 handling is unchanged.
@github-actions

Copy link
Copy Markdown
Contributor

The following comment was made by an LLM, it may be inaccurate:

@opencode-agent

Copy link
Copy Markdown
Contributor

Thanks for the clear write-up and the tests, @reynold-hu!

v1 (the dev branch) only takes critical fixes. This bug has a simple workaround: write or convert the export as UTF-8 (for example opencode export <id> | Out-File -Encoding utf8 session.json, or use PowerShell 7), so it doesn't qualify.

The same issue seems to exist in v2: readJson in packages/util/src/fs-util.ts on the v2 branch also reads files with readFileString. You're welcome to open this fix against v2. It's a separate implementation, so the change would need to be adapted there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

import fails on UTF-16 files with BOM (Windows PowerShell redirection)

1 participant