Repository navigation
test: add cross-PR scenario suite - #594
Open
LukasParke wants to merge 4 commits into
Open
LukasParke wants to merge 4 commits into
LukasParke wants to merge 4 commits into
Conversation
Adds 126 scenario tests under src/tests/scenarios/ that exercise the provider through generateText/streamText/evaluate/registry with a mocked fetch. A global fetch guard fails any request that bypasses the mock. The 20k-delta streaming test asserts on the number of large JSON re-parses instead of wall time. Four known bugs stay as it.fails. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
Removes the four it.fails scenarios. Each is covered by a regression test in the PR that fixes it (#596, #598, #601), so main no longer goes red when one of those fixes merges. Adds the property suite's biome override after the scenario one, in the same form as #599, so the two PRs merge without a conflict. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
LukasParke
added a commit
that referenced
this pull request
Oct 23, 2026
…Text Moved from the scenario suite's known-bug repros (#594): a file part URL with the top-level media type 'image' is sent as image_url with its detail, and the AI SDK does not download it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
LukasParke
added a commit
that referenced
this pull request
Oct 23, 2026
Removes known-bugs.test.ts. Each it.fails repro is covered by a regression test in the PR that fixes it (#590, #591, #592, #595, #600), so main no longer goes red when one of those fixes merges. The convert-messages invariant on assistant content shape no longer assumes that any truthy cacheControl makes the content an array: it asserts a string when no text part has a truthy value, an array when some part has a valid value, and skips the shape check when only invalid values are present (these are dropped once cache control is validated, #590). Adds the scenario suite's biome override next to the property one, in the same form as #594, so the two PRs merge without a conflict. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
4 of 5 tasks
An empty reasoning part that carries late reasoning_details (#608) is allowed, so the signature-only scenario now passes whatever order the PRs merge in. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds a cross-PR scenario suite (122 tests over 8 files) under
src/tests/scenarios/. It drives the provider throughai's public APIs (generateText,streamText,experimental_evaluate, provider registry, workflow serde) with a mocked fetch. It covers cache control, Decisions, media (imageDetail, videoProcessing, image routing), reasoning params and streaming, tool-call streaming and workflow serialization.src/**/*.test.ts, sopnpm testruns the suite in both environments. No new config.helpers.ts(imported by every scenario file) replacesglobalThis.fetchwith a guard that throws on any request that bypasses the per-test mock.< 1500ms). It now counts re-parses of the large accumulated arguments by spying onJSON.parse(< 10). I checked that this catches the pre-perf(chat): avoid re-parsing streamed tool-call arguments on every delta #586 behaviour: reverting the scanner gate gives 19904 re-parses.biome.json: added an override forsrc/tests/scenarios/**/*.tsthat turns offnoExplicitAny. The existing*.test.tsoverride only matches root-level files in Biome 2, andhelpers.tsisn't a test file either.biome.jsonalso carries the property suite's override (src/tests/properties/**) in exactly the form test: add property-based fuzz suite with fast-check #599 adds it, so whichever of test: add cross-PR scenario suite #594 and test: add property-based fuzz suite with fast-check #599 merges second merges without a conflict.it.failstests. Coverage for the known bugs the suite found lives in the PRs that fix them, so main stays green whichever order they merge in:detail: fix: handle bare top-level image/video/audio media types #598 (src/chat/bare-media-types.test.ts)'image'isn't mapped toimage_url: fix: handle bare top-level image/video/audio media types #598 (converter test, plus thegenerateTextscenario moved intosrc/chat/bare-media-types.test.ts)supportedUrlshas novideo/*: fix(chat): pass video URLs through instead of downloading them #601 (src/chat/video-url.test.ts)WORKFLOW_SERIALIZEthrows withoutOPENROUTER_API_KEYin the environment: fix: serialize workflow models without loading the API key #596 (src/utils/workflow-serde.test.ts)I ran the suite 3 times in each of node and edge: 126/126 passed every time (before the known-bug tests moved out), in about 4-5s per run.
Checklist
pnpm stylecheckandpnpm typecheckpnpm testand all tests passChangeset
pnpm changesetto create a changeset file🤖 Generated with Claude Code