diff --git a/CHANGELOG.md b/CHANGELOG.md index c8148de8d..780e0abb7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## Unreleased +- Accept signed global coordinates for foreground MCP move and drag within existing magnitude bounds, preserving target guards and before-delivery validation. Thanks @rudycelekli! #963. - Prepare the final-qualification launcher's atomic receipt publisher before spawning the suspended coordinator, so loaded release Macs no longer spend the identity-acknowledgement window compiling Swift; each use still re-verifies the exact executable, and the exit receipt keeps a fresh build. - Preserve documentation source text through syntax highlighting, parse CRLF metadata, render unfinished EOF fences, retain link entities/query parameters, and assign unique heading targets. Thanks @rudycelekli! #931, #939, #940, #969, #971. - Add exact app/PID-scoped menu-extra clicks with bounded AX owner preparation and revalidation, preserving displayed-CG names and indices without scanning unrelated applications. Thanks @buremba! #953. diff --git a/Core/PeekabooAutomationKit/Tests/PeekabooAutomationKitTests/SnapshotDetectionMetadataTests.swift b/Core/PeekabooAutomationKit/Tests/PeekabooAutomationKitTests/SnapshotDetectionMetadataTests.swift index e9b254213..0467cb78a 100644 --- a/Core/PeekabooAutomationKit/Tests/PeekabooAutomationKitTests/SnapshotDetectionMetadataTests.swift +++ b/Core/PeekabooAutomationKit/Tests/PeekabooAutomationKitTests/SnapshotDetectionMetadataTests.swift @@ -21,6 +21,25 @@ struct SnapshotDetectionMetadataTests { #expect(loaded.metadata.truncationInfo == truncation) } + @Test + func `complete detection replaces prior dialog and truncation classification`() async throws { + let storage = Self.storage() + defer { try? FileManager.default.removeItem(at: storage) } + let manager = SnapshotManager(snapshotStorageURL: storage) + let id = try await manager.createExplicitSnapshot() + try await manager.storeDetectionResult( + snapshotId: id, + result: Self.result(id, dialog: true, truncation: DetectionTruncationInfo(deadlineReached: true))) + try await manager.storeDetectionResult( + snapshotId: id, + result: Self.result(id, dialog: false, truncation: nil)) + + let loaded = try #require(try await SnapshotManager(snapshotStorageURL: storage) + .getDetectionResult(snapshotId: id)) + #expect(loaded.metadata.isDialog == false) + #expect(loaded.metadata.truncationInfo == nil) + } + @Test func `legacy version one record without classification remains readable`() async throws { let storage = Self.storage() diff --git a/Core/PeekabooCore/Sources/PeekabooAgentRuntime/MCP/Tools/DragTool+Resolution.swift b/Core/PeekabooCore/Sources/PeekabooAgentRuntime/MCP/Tools/DragTool+Resolution.swift index a8a6dc41b..9150a355d 100644 --- a/Core/PeekabooCore/Sources/PeekabooAgentRuntime/MCP/Tools/DragTool+Resolution.swift +++ b/Core/PeekabooCore/Sources/PeekabooAgentRuntime/MCP/Tools/DragTool+Resolution.swift @@ -72,15 +72,10 @@ extension DragTool { message: "Invalid \(parameterName) coordinates. Both x and y must be valid numbers") } - // Coordinates outside the desktop are nearly always malformed tool input. - guard background || (x >= 0 && y >= 0) else { + // Displays left of or above the primary display have negative global coordinates. + guard background || ((-20000...20000).contains(x) && (-20000...20000).contains(y)) else { throw CoordinateParseError( - message: "Invalid \(parameterName) coordinates. Both x and y must be non-negative") - } - - guard background || (x <= 20000 && y <= 20000) else { - throw CoordinateParseError( - message: "Invalid \(parameterName) coordinates. Both x and y must be 20000 or less") + message: "Invalid \(parameterName) coordinates. Both x and y must be between -20000 and 20000") } return CGPoint(x: x, y: y) diff --git a/Core/PeekabooCore/Sources/PeekabooAgentRuntime/MCP/Tools/MoveTool+Parsing.swift b/Core/PeekabooCore/Sources/PeekabooAgentRuntime/MCP/Tools/MoveTool+Parsing.swift index 276dc07c6..c65b7439c 100644 --- a/Core/PeekabooCore/Sources/PeekabooAgentRuntime/MCP/Tools/MoveTool+Parsing.swift +++ b/Core/PeekabooCore/Sources/PeekabooAgentRuntime/MCP/Tools/MoveTool+Parsing.swift @@ -15,12 +15,9 @@ extension MoveTool { throw CoordinateParseError(message: "Invalid \(parameterName). Both x and y must be valid numbers") } - guard x >= 0, y >= 0 else { - throw CoordinateParseError(message: "Invalid \(parameterName). Both x and y must be non-negative") - } - - guard x <= 20000, y <= 20000 else { - throw CoordinateParseError(message: "Invalid \(parameterName). Both x and y must be 20000 or less") + guard (-20000...20000).contains(x), (-20000...20000).contains(y) else { + throw CoordinateParseError( + message: "Invalid \(parameterName). Both x and y must be between -20000 and 20000") } return CGPoint(x: x, y: y) diff --git a/Core/PeekabooCore/Tests/PeekabooTests/MCP/MCPGlobalPointerActionResultTests.swift b/Core/PeekabooCore/Tests/PeekabooTests/MCP/MCPGlobalPointerActionResultTests.swift index 04b39e5d8..5c4419fad 100644 --- a/Core/PeekabooCore/Tests/PeekabooTests/MCP/MCPGlobalPointerActionResultTests.swift +++ b/Core/PeekabooCore/Tests/PeekabooTests/MCP/MCPGlobalPointerActionResultTests.swift @@ -18,7 +18,8 @@ struct MCPGlobalPointerActionResultTests { func `move response reports direction in screen coordinate space`(_ vector: (Double, Double, String)) async throws { let start = CGPoint(x: 100, y: 200) let automation = MockAutomationService(accessibilityGranted: true, currentMouseLocation: start) - let context = await MCPToolTestHelpers.makeLegacyContext(automation: automation) + let context = await MCPToolTestHelpers.makeContext( + automation: automation, executionPolicy: .foregroundAllowed) let target = CGPoint(x: start.x + vector.0, y: start.y + vector.1) let response = try await MoveTool(context: context).execute(arguments: ToolArguments(raw: [ "to": "\(target.x),\(target.y)", "foreground": true, diff --git a/Core/PeekabooCore/Tests/PeekabooTests/MCP/MCPHumanMoveParameterTests.swift b/Core/PeekabooCore/Tests/PeekabooTests/MCP/MCPHumanMoveParameterTests.swift index 0b78b1b58..408ec8dc4 100644 --- a/Core/PeekabooCore/Tests/PeekabooTests/MCP/MCPHumanMoveParameterTests.swift +++ b/Core/PeekabooCore/Tests/PeekabooTests/MCP/MCPHumanMoveParameterTests.swift @@ -21,7 +21,8 @@ struct MCPHumanMoveParameterTests { ]) func `human movement refuses invalid overrides before pointer dispatch`(_ input: (String, Int)) async throws { let automation = MockAutomationService(accessibilityGranted: true) - let context = await MCPToolTestHelpers.makeLegacyContext(automation: automation) + let context = await MCPToolTestHelpers.makeContext( + automation: automation, executionPolicy: .foregroundAllowed) let response = try await MoveTool(context: context).execute(arguments: ToolArguments(raw: [ "to": "50,60", "foreground": true, "profile": "human", input.0: input.1, ])) @@ -35,7 +36,8 @@ struct MCPHumanMoveParameterTests { @Test(arguments: ["linear", "human"]) func `valid smooth overrides still execute for both profiles`(_ profile: String) async throws { let automation = MockAutomationService(accessibilityGranted: true) - let context = await MCPToolTestHelpers.makeLegacyContext(automation: automation) + let context = await MCPToolTestHelpers.makeContext( + automation: automation, executionPolicy: .foregroundAllowed) let response = try await MoveTool(context: context).execute(arguments: ToolArguments(raw: [ "to": "50,60", "foreground": true, "smooth": true, "profile": profile, "duration": 1000, "steps": 50, ])) diff --git a/Core/PeekabooCore/Tests/PeekabooTests/MCP/MCPSecondaryDisplayPointerTests.swift b/Core/PeekabooCore/Tests/PeekabooTests/MCP/MCPSecondaryDisplayPointerTests.swift new file mode 100644 index 000000000..782158a8d --- /dev/null +++ b/Core/PeekabooCore/Tests/PeekabooTests/MCP/MCPSecondaryDisplayPointerTests.swift @@ -0,0 +1,59 @@ +import CoreGraphics +import MCP +import PeekabooAutomationKit +import TachikomaMCP +import Testing +@testable import PeekabooAgentRuntime +@testable import PeekabooAutomation +@testable import PeekabooCore + +@MainActor +struct MCPSecondaryDisplayPointerTests { + @Test(arguments: [ + ("-100,200", CGPoint(x: -100, y: 200)), + ("100,-200", CGPoint(x: 100, y: -200)), + ("-100,-200", CGPoint(x: -100, y: -200)), + ]) + func `move accepts signed global coordinates for secondary displays`(_ input: (String, CGPoint)) async throws { + let automation = MockAutomationService(accessibilityGranted: true) + let context = await MCPToolTestHelpers.makeContext( + automation: automation, executionPolicy: .foregroundAllowed) + let response = try await MoveTool(context: context).execute(arguments: ToolArguments(raw: [ + "to": input.0, "foreground": true, + ])) + #expect(!response.isError) + #expect(automation.lastMoveTarget == input.1) + } + + @Test + func `foreground drag accepts endpoints left of and above the primary display`() async throws { + let automation = MockAutomationService(accessibilityGranted: true) + let context = await MCPToolTestHelpers.makeContext( + automation: automation, executionPolicy: .foregroundAllowed) + let response = try await DragTool(context: context).execute(arguments: ToolArguments(raw: [ + "from_coords": "-100,200", "to_coords": "100,-200", "foreground": true, + ])) + #expect(!response.isError) + #expect(automation.dragRequests.count == 1) + let request = try #require(automation.dragRequests.first) + #expect(request.from == CGPoint(x: -100, y: 200)) + #expect(request.to == CGPoint(x: 100, y: -200)) + } + + @Test(arguments: ["-20001,1", "1,-20001", "20001,1", "nan,1", "inf,1"]) + func `malformed or extreme foreground coordinates are refused`(_ coordinates: String) async throws { + let automation = MockAutomationService(accessibilityGranted: true) + let context = await MCPToolTestHelpers.makeContext( + automation: automation, executionPolicy: .foregroundAllowed) + let move = try await MoveTool(context: context).execute(arguments: ToolArguments(raw: [ + "to": coordinates, "foreground": true, + ])) + let drag = try await DragTool(context: context).execute(arguments: ToolArguments(raw: [ + "from_coords": "0,0", "to_coords": coordinates, "foreground": true, + ])) + #expect(move.isError) + #expect(drag.isError) + #expect(automation.lastMoveTarget == nil) + #expect(automation.dragRequests.isEmpty) + } +} diff --git a/Core/PeekabooCore/Tests/PeekabooTests/MCP/MCPToolExecutionTests.swift b/Core/PeekabooCore/Tests/PeekabooTests/MCP/MCPToolExecutionTests.swift index 26bf89803..5b4312532 100644 --- a/Core/PeekabooCore/Tests/PeekabooTests/MCP/MCPToolExecutionTests.swift +++ b/Core/PeekabooCore/Tests/PeekabooTests/MCP/MCPToolExecutionTests.swift @@ -1106,6 +1106,7 @@ class MockAutomationService: ExactWindowTargetedClickServiceProtocol, TargetedHo private(set) var lastHotkeyKeys: String? private(set) var lastHotkeyHoldDuration: Int? private(set) var lastMoveTarget: CGPoint? + private(set) var dragRequests: [DragOperationRequest] = [] private(set) var lastMoveDuration: Int? private(set) var lastWindowContext: WindowContext? var supportsTargetedHotkeys = true @@ -1307,7 +1308,9 @@ class MockAutomationService: ExactWindowTargetedClickServiceProtocol, TargetedHo WaitForElementResult(found: false, element: nil, waitTime: 0) } - func drag(_: DragOperationRequest) async throws {} + func drag(_ request: DragOperationRequest) async throws { + self.dragRequests.append(request) + } func moveMouse( to: CGPoint, diff --git a/docs/commands/see.md b/docs/commands/see.md index f84dfb015..a0768dc02 100644 --- a/docs/commands/see.md +++ b/docs/commands/see.md @@ -26,6 +26,10 @@ means the evidence is unknown or not applicable. Freshness does not mean complet retain their existing warnings and targeting restrictions. Text-selection range readback remains limited to the focused field; `--fresh` does not make unfocused ranges available. +Disk-backed snapshot readback preserves the dialog and truncation information recorded by detection. +Replacing a raw screenshot clears this classification together with the old element map; storing another detection +replaces those values. Older stored records without these fields keep their existing defaults. + Pixels and Accessibility metadata are collected at different times, not as one atomic application-state snapshot. During asynchronous UI updates, an image can still show the old value while AX already reports the new value. Atomic snapshot publication binds the returned artifacts; it does not make their acquisition simultaneous. After diff --git a/docs/human-mouse-move.md b/docs/human-mouse-move.md index e435322c6..bf4bdb7a9 100644 --- a/docs/human-mouse-move.md +++ b/docs/human-mouse-move.md @@ -18,6 +18,11 @@ Peekaboo's `human` profile makes cursor motion look hand-driven without forcing - **CLI**: use `--smooth` for a natural `peekaboo move`, or add `--profile human` to `move` or `drag`. Duration/sample counts pick sensible defaults per distance. Explicit `--duration` and `--steps` values are honored; human paths never emit more than 96 samples. - **Agents / MCP**: include `"profile": "human"` in move/drag tool arguments. Optional `duration` and `steps` fields work the same way as in the CLI—you only need them when you want to clamp the adaptive heuristics. +MCP `move` requires `foreground: true`. Smooth movement accepts a duration of 1–30000 ms and 1–100 steps, +including the human profile; invalid overrides are rejected before delivery. Foreground MCP move/drag coordinates +accept values from -20000 to 20000 on either axis, including displays left of or above the primary display. +Compass direction metadata uses screen coordinates: positive x points east and positive y points south. + ## Defaults at a glance | Distance | Typical Duration | Typical Steps | Notes |