Repository navigation
refactor(core): restructure Jev integration and planning delegation - #3295
Merged
Merged
Conversation
Separate generative and System One requests and resolve model fallbacks by task type. Add Jev localization, boolean insight, and action-aware planning with default-model delegation. Update model configuration documentation and regression tests.
Deploying midscene with
|
| Latest commit: |
88a00ef
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://707e5167.midscene.pages.dev |
| Branch Preview URL: | https://zzy-jev-model.midscene.pages.dev |
yuyutaotao
approved these changes
Oct 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Refactor the existing Jev integration into dedicated System One calling, localization, boolean insight, and planning modules. Jev derives planning choices from the available Action Space: simple actions are planned directly, while complex actions are delegated to the default model with an advisory suggestion.
When Planning and Locate both use Jev with the same model name and Base URL, simple single-target actions return an action choice and target box in one request. Execution reuses the coordinates instead of making another localization request. This optimization uses the Planning request configuration and is disabled for deepThink; different models or providers retain separate localization.
callGenerativeAIandcallSystemOneAIwhile sharing retries, cancellation, recording, and usage accounting. Preserve SIWC authentication and token refresh.supportedBySystemOneaction metadata, prefixed dynamic choices, and a fallback when the Choice limit is exceeded. Forward support metadata through the system-input action helper.screenshotShrinkFactorand provider Base URL conventions in both languages.Validation
For the latest changes:
pnpm run lint— passed with one existing unused-suppression warning.pnpm exec nx test @midscene/core jev task-executor-concurrency bbox-locate-cache— 83 passed.pnpm exec nx test @midscene/core task-executor-concurrency— 25 passed after refining the matching-model conditions.pnpm exec tsc -p packages/core/tsconfig.json --noEmit --pretty false— passed.pnpm exec nx build @midscene/core— passed before the final condition-only refinement.Earlier validation of the refactor:
pnpm exec nx test @midscene/core— 2,453 passed, 8 skipped.pnpm exec nx test @midscene/shared env— 175 passed.pnpm exec nx test @midscene/web web-actions-navigation proxy-deps-resolution— 16 passed.pnpm exec nx test doc model-config— 19 passed.Manual live-model reports confirm planning coordinates are reused without a separate Locate model request. Jev can still prematurely mark unfinished tasks complete; this change does not add completion verification.