fix: prepare OpenGUI model and bound phone screenshots

Author: HarveySang

deepseek-harness-plugin/README.md

@@ -177,7 +177,7 @@ Keep the phone unlocked and authorized for USB debugging, then send:
 /opengui Open Settings and report the Android version
 ```
 
-The native **OpenGUI** tab owns phone selection and status. One authorized phone is selected automatically. With multiple phones, check any subset in that workbench before sending `/opengui` or `@OpenGUI`. A non-empty task pauses before model routing until a phone is ready. Device selection remains editable while waiting, then locks after detection until the whole batch finishes.
+The native **OpenGUI** tab owns phone selection and status. One authorized phone is selected automatically. With multiple phones, check any subset in that workbench before sending `/opengui` or `@OpenGUI`. A non-empty task resolves the current-model choice or dedicated fallback setup before it waits for a phone, but it sends no provider request until the selected-device snapshot is ready. Device selection remains editable while waiting, then locks after detection until the whole batch finishes.
 
 The native **OpenGUI** tab is a full-width device wall. It renders every visible phone in Host order and appends one connection guide, without reserving empty slots. Connected phones expand automatically into a complete, low-latency H.264 view while the tab, card, and page remain visible. First use asks before downloading and verifying the pinned scrcpy server; unsupported browsers or stream failures fall back to an uncropped JPEG preview. Each phone can also be opened in an optional independent scrcpy window.
 
@@ -286,6 +286,7 @@ If you used a profile patch, also remove its `coremate-mobile` row. Leave `[]` w
 - Phone text accepts up to 500 Unicode characters. Safe ASCII uses `adb input text`; Chinese, emoji, and other Unicode text uses scrcpy's standard UTF-8 clipboard control message and must receive the matching device ACK. This is independent of the phone vendor and active input method, and never silently falls back to simulated keyboard typing.
 - `browser_control` exposes no arbitrary CDP, JavaScript execution, filesystem access, or non-HTTP(S) URL. Page mutations require the latest observation.
 - The current DSH route is proxied without copying credentials. Provider requests keep tool schemas, cancellation, streaming, and at most the latest phone screenshot.
+- Model-facing phone screenshots are JPEG quality 65 with preserved aspect ratio and a maximum 2048px long edge. Smaller frames are never enlarged; tap and swipe coordinates are mapped from that bounded screenshot back to the Android logical display size.
 - A capability failure never automatically retries a task that may already have caused phone or browser side effects.
 - Direct `/opengui` results are text-only; screenshots remain in the phone child session.
 - Native mirroring is view-only and opens only on the graphical computer running Harness; a headless Host cannot display the window.

deepseek-harness-plugin/README.zh.md

@@ -179,7 +179,7 @@ Usage: /opengui <task>
 /opengui 打开设置并报告 Android 版本
 ```
 
-手机选择和状态统一放在原生 **OpenGUI** Tab。只有一台已授权手机时会自动选中；有多台时，可在工作台中勾选任意一台或多台后再发送 `/opengui` 或 `@OpenGUI`。非空任务会先等待手机，不会提前调用模型；等待期间仍可连接、选择和投屏，检测成功后才锁定本次设备快照，直到整批结束。
+手机选择和状态统一放在原生 **OpenGUI** Tab。只有一台已授权手机时会自动选中；有多台时，可在工作台中勾选任意一台或多台后再发送 `/opengui` 或 `@OpenGUI`。非空任务会先确定当前模型或完成专用回退配置，再等待手机；在选定设备快照就绪前不会向 provider 发起模型请求。等待期间仍可连接、选择和投屏，检测成功后才锁定本次设备快照，直到整批结束。
 
 原生 **OpenGUI** Tab 是全宽设备照片墙。界面按 Host 顺序展示全部可见手机，并在末尾追加唯一的连接说明卡，不会为凑列数预留空卡。新检测到的手机默认展开完整、低延迟的 H.264 实时画面；只有 Tab、设备卡和页面都可见时才保持连接。首次使用会先征得同意，再下载并校验固定版本的 scrcpy server；浏览器不支持或视频流失败时，会降级为不裁切的 JPEG 截图预览。每台手机还可按需打开独立的 scrcpy 窗口。
 
@@ -288,6 +288,7 @@ dsh plugin --profile web remove dsh-coremate-mobile
 - 手机文本输入最多 500 个 Unicode 字符。安全 ASCII 直接使用 `adb input text`；中文、emoji 等文本使用 scrcpy 标准控制协议发送 UTF-8 剪贴板消息，并必须收到对应的设备 ACK。该路径与手机厂商和当前输入法无关，也不会静默退化为模拟屏幕键盘逐字输入。
 - `browser_control` 不暴露任意 CDP、JavaScript 执行、文件系统访问或非 HTTP(S) URL；页面修改必须引用最新观察。
 - 当前 DSH 路由通过内部代理转发，不复制凭据；provider 请求保留工具 schema、取消信号和流式结果，并且最多只保留最新手机截图。
+- 发给模型的手机截图保持原始宽高比，统一编码为质量 65、最长边不超过 2048px 的 JPEG；小图不会放大。点击和滑动坐标会从该截图空间换算回 Android 逻辑显示尺寸。
 - 能力错误不会自动重试可能已经产生手机或浏览器副作用的任务。
 - 直接 `/opengui` 的最终结果只显示文本；截图保留在对应控制子会话中。
 - 原生投屏只读且只在运行 Harness 的桌面主机上打开；无图形桌面的主机无法显示窗口。

deepseek-harness-plugin/src/image.ts

@@ -3,13 +3,22 @@ import sharp from 'sharp'
 /** JPEG quality used for every phone screenshot; intentionally within 60–70. */
 export const PHONE_SCREENSHOT_JPEG_QUALITY = 65
 
+/** Conservative provider-compatible bound for the longest model-facing image edge. */
+export const PHONE_SCREENSHOT_MAX_LONG_EDGE = 2_048
+
 /**
- * Encode an ADB screenshot as JPEG without changing its pixel dimensions.
+ * Encode an ADB screenshot as a bounded JPEG for model input.
  * @param source Encoded source screenshot returned by ADB.
- * @returns JPEG bytes at the source image width and height.
+ * @returns JPEG bytes whose aspect ratio is preserved and longest edge is at most 2048px.
  */
 export async function encodePhoneScreenshot(source: Buffer): Promise<Buffer> {
   return sharp(source, { failOn: 'error', limitInputPixels: false })
+    .resize({
+      width: PHONE_SCREENSHOT_MAX_LONG_EDGE,
+      height: PHONE_SCREENSHOT_MAX_LONG_EDGE,
+      fit: 'inside',
+      withoutEnlargement: true,
+    })
     .jpeg({ quality: PHONE_SCREENSHOT_JPEG_QUALITY })
     .toBuffer()
 }

deepseek-harness-plugin/src/index.ts

@@ -764,9 +764,9 @@ export function apply(ctx: Context, baseConfig: Config): void {
     interaction: TaskInteraction,
     operation: (lease: OpenGuiTaskLease<OpenGuiExecutionContext>) => Promise<CoremateTaskResult>,
   ): Promise<CoremateTaskResult> => tasks.runRoot<CoremateTaskResult>(interaction.agent, interaction.signal, 'waiting-for-device', async lease => {
+    const route = await prepareTask(interaction)
     const targets = await waitForSelectedPhone(interaction)
     lease.setPhase('routing')
-    const route = await prepareTask(interaction)
     lease.context = { targets, route }
     lease.setPhase('running')
     try {

deepseek-harness-plugin/src/phone-task.ts

@@ -60,7 +60,7 @@ export type CoremateTaskStart = (
 /** Resolve the model route before a non-empty direct command starts. */
 export type CoremateTaskPrepare = (invocation: CommandInvocation) => Promise<AgentOptions>
 
-/** Pause a non-empty command until its external prerequisites are ready. */
+/** Pause a non-empty command until its external device prerequisites are ready. */
 export type CoremateTaskPreflight = (invocation: CommandInvocation, signal: AbortSignal) => Promise<void>
 
 /** Convert an inherited-model capability failure into a safe next action. */
@@ -145,9 +145,9 @@ export class CoremateTaskCoordinator {
         if (task.length === 0) return { kind: 'success', text: OPENGUI_USAGE }
         try {
           const result = await this.tasks.runRoot(invocation.agent, invocation.signal, 'waiting-for-device', async lease => {
+            const agentOptions = await prepare?.({ ...invocation, signal: lease.signal })
             await preflight?.(invocation, lease.signal)
             lease.setPhase('routing')
-            const agentOptions = await prepare?.({ ...invocation, signal: lease.signal })
             lease.setPhase('running')
             try {
               return await this.start(task, invocation.agent, lease.signal, 'parent-chat', agentOptions)

deepseek-harness-plugin/tests/adb.spec.ts

@@ -53,6 +53,34 @@ describe('coremate-mobile ADB policy', () => {
     })).toEqual(['shell', 'input', 'tap', '990', '155'])
   })
 
+  it('maps M153 model coordinates from the bounded screenshot back to the device', () => {
+    expect(actionCommand({
+      action: 'tap',
+      observationId: ObservationId('m153-observation'),
+      targetBBox: { left: 440, top: 990, right: 484, bottom: 1_058 },
+    }, {
+      width: 1_264,
+      height: 2_800,
+      screenshotWidth: 925,
+      screenshotHeight: 2_048,
+    })).toEqual(['shell', 'input', 'tap', '631', '1400'])
+
+    expect(actionCommand({
+      action: 'swipe',
+      observationId: ObservationId('m153-observation'),
+      x1: 462,
+      y1: 1_536,
+      x2: 462,
+      y2: 512,
+      durationMs: 300,
+    }, {
+      width: 1_264,
+      height: 2_800,
+      screenshotWidth: 925,
+      screenshotHeight: 2_048,
+    })).toEqual(['shell', 'input', 'swipe', '631', '2100', '631', '700', '300'])
+  })
+
   it('does not impose a fixed delay after every device mutation', () => {
     expect(Config({})).not.toHaveProperty('settleDelayMs')
   })

deepseek-harness-plugin/tests/bundle.spec.ts

@@ -65,6 +65,16 @@ describe('standalone DeepSeek Harness bundle', () => {
     expect(clientSource).not.toContain("'shell.overlay'")
   })
 
+  it('prepares the model before waiting for an explicit phone selection', async () => {
+    const hostSource = await readFile(new URL('src/index.ts', root), 'utf8')
+    const prepareIndex = hostSource.indexOf('const route = await prepareTask(interaction)')
+    const deviceIndex = hostSource.indexOf('const targets = await waitForSelectedPhone(interaction)')
+
+    expect(prepareIndex).toBeGreaterThan(-1)
+    expect(deviceIndex).toBeGreaterThan(-1)
+    expect(prepareIndex).toBeLessThan(deviceIndex)
+  })
+
   it('keeps the stop glyph visible independently of the host foreground color', async () => {
     const stopButtonSource = await readFile(new URL('src/client/TaskStopButton.tsx', root), 'utf8')
 

deepseek-harness-plugin/tests/image.spec.ts

@@ -2,6 +2,7 @@ import sharp from 'sharp'
 import { describe, expect, it } from 'vitest'
 import {
   PHONE_SCREENSHOT_JPEG_QUALITY,
+  PHONE_SCREENSHOT_MAX_LONG_EDGE,
   encodePhoneScreenshot,
 } from '../src/image.ts'
 
@@ -24,4 +25,21 @@ describe('coremate-mobile screenshot encoding', () => {
     expect([...jpeg.subarray(0, 3)]).toEqual([0xFF, 0xD8, 0xFF])
     expect(metadata).toMatchObject({ format: 'jpeg', width: 321, height: 654 })
   })
+
+  it('bounds Nubia M153 portrait and landscape screenshots without enlarging smaller frames', async () => {
+    expect(PHONE_SCREENSHOT_MAX_LONG_EDGE).toBe(2_048)
+    for (const [width, height, expectedWidth, expectedHeight] of [
+      [1_264, 2_800, 925, 2_048],
+      [2_800, 1_264, 2_048, 925],
+      [1_080, 2_400, 922, 2_048],
+      [321, 654, 321, 654],
+    ] as const) {
+      const png = await sharp({
+        create: { width, height, channels: 3, background: { r: 16, g: 32, b: 48 } },
+      }).png().toBuffer()
+
+      const metadata = await sharp(await encodePhoneScreenshot(png)).metadata()
+      expect(metadata).toMatchObject({ format: 'jpeg', width: expectedWidth, height: expectedHeight })
+    }
+  })
 })

deepseek-harness-plugin/tests/phone-task.spec.ts

@@ -120,7 +120,7 @@ describe('OpenGUI task entry points', () => {
     expect(start).toHaveBeenCalledWith('open settings', call.agent, expect.any(AbortSignal), 'parent-chat', route)
   })
 
-  it('waits for a phone before resolving the model and locks selection only afterwards', async () => {
+  it('resolves the model before waiting for a phone and locks selection only afterwards', async () => {
     let allowPhone!: () => void
     const phoneReady = new Promise<void>(resolve => { allowPhone = resolve })
     const order: string[] = []
@@ -139,16 +139,16 @@ describe('OpenGUI task entry points', () => {
     })
     const running = coordinator.command(preflight, prepare).handler(invocation('检查手机'))
 
-    await vi.waitFor(() => { expect(preflight).toHaveBeenCalledTimes(1) })
+    await vi.waitFor(() => { expect(prepare).toHaveBeenCalledTimes(1) })
     expect(coordinator.state()).toEqual({ active: true, phase: 'waiting-for-device', selectionLocked: false, ownerSessionId: 'session-owner' })
-    expect(prepare).not.toHaveBeenCalled()
+    expect(preflight).toHaveBeenCalledTimes(1)
     allowPhone()
     await expect(running).resolves.toEqual({ kind: 'success', text: 'done' })
-    expect(order).toEqual(['preflight', 'prepare', 'start'])
+    expect(order).toEqual(['prepare', 'preflight', 'start'])
     expect(coordinator.state()).toEqual({ active: false, phase: 'idle', selectionLocked: false })
   })
 
-  it('can cancel while waiting for a phone without resolving a model or starting work', async () => {
+  it('can cancel while waiting for a phone after resolving the model without starting work', async () => {
     const prepare = vi.fn(async () => ({ provider: 'current', model: 'vision' }))
     const start = vi.fn(async () => result('unused'))
     const coordinator = new CoremateTaskCoordinator(start)
@@ -160,7 +160,7 @@ describe('OpenGUI task entry points', () => {
     await vi.waitFor(() => { expect(coordinator.state().phase).toBe('waiting-for-device') })
     expect(coordinator.cancel()).toBe(true)
     await expect(running).resolves.toEqual({ kind: 'error', text: 'coremate-mobile: OpenGUI task stopped by user' })
-    expect(prepare).not.toHaveBeenCalled()
+    expect(prepare).toHaveBeenCalledTimes(1)
     expect(start).not.toHaveBeenCalled()
   })