Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@

## Unreleased

- Accept signed global coordinates for foreground MCP move and drag within existing magnitude bounds, preserving target guards and before-delivery validation. Thanks @rudycelekli! #963.
- Prepare the final-qualification launcher's atomic receipt publisher before spawning the suspended coordinator, so loaded release Macs no longer spend the identity-acknowledgement window compiling Swift; each use still re-verifies the exact executable, and the exit receipt keeps a fresh build.
- Preserve documentation source text through syntax highlighting, parse CRLF metadata, render unfinished EOF fences, retain link entities/query parameters, and assign unique heading targets. Thanks @rudycelekli! #931, #939, #940, #969, #971.
- Add exact app/PID-scoped menu-extra clicks with bounded AX owner preparation and revalidation, preserving displayed-CG names and indices without scanning unrelated applications. Thanks @buremba! #953.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,25 @@ struct SnapshotDetectionMetadataTests {
#expect(loaded.metadata.truncationInfo == truncation)
}

@Test
func `complete detection replaces prior dialog and truncation classification`() async throws {
let storage = Self.storage()
defer { try? FileManager.default.removeItem(at: storage) }
let manager = SnapshotManager(snapshotStorageURL: storage)
let id = try await manager.createExplicitSnapshot()
try await manager.storeDetectionResult(
snapshotId: id,
result: Self.result(id, dialog: true, truncation: DetectionTruncationInfo(deadlineReached: true)))
try await manager.storeDetectionResult(
snapshotId: id,
result: Self.result(id, dialog: false, truncation: nil))

let loaded = try #require(try await SnapshotManager(snapshotStorageURL: storage)
.getDetectionResult(snapshotId: id))
#expect(loaded.metadata.isDialog == false)
#expect(loaded.metadata.truncationInfo == nil)
}

@Test
func `legacy version one record without classification remains readable`() async throws {
let storage = Self.storage()
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -72,15 +72,10 @@ extension DragTool {
message: "Invalid \(parameterName) coordinates. Both x and y must be valid numbers")
}

// Coordinates outside the desktop are nearly always malformed tool input.
guard background || (x >= 0 && y >= 0) else {
// Displays left of or above the primary display have negative global coordinates.
guard background || ((-20000...20000).contains(x) && (-20000...20000).contains(y)) else {
throw CoordinateParseError(
message: "Invalid \(parameterName) coordinates. Both x and y must be non-negative")
}

guard background || (x <= 20000 && y <= 20000) else {
throw CoordinateParseError(
message: "Invalid \(parameterName) coordinates. Both x and y must be 20000 or less")
message: "Invalid \(parameterName) coordinates. Both x and y must be between -20000 and 20000")
}

return CGPoint(x: x, y: y)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,12 +15,9 @@ extension MoveTool {
throw CoordinateParseError(message: "Invalid \(parameterName). Both x and y must be valid numbers")
}

guard x >= 0, y >= 0 else {
throw CoordinateParseError(message: "Invalid \(parameterName). Both x and y must be non-negative")
}

guard x <= 20000, y <= 20000 else {
throw CoordinateParseError(message: "Invalid \(parameterName). Both x and y must be 20000 or less")
guard (-20000...20000).contains(x), (-20000...20000).contains(y) else {
throw CoordinateParseError(
message: "Invalid \(parameterName). Both x and y must be between -20000 and 20000")
}

return CGPoint(x: x, y: y)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,8 @@ struct MCPGlobalPointerActionResultTests {
func `move response reports direction in screen coordinate space`(_ vector: (Double, Double, String)) async throws {
let start = CGPoint(x: 100, y: 200)
let automation = MockAutomationService(accessibilityGranted: true, currentMouseLocation: start)
let context = await MCPToolTestHelpers.makeLegacyContext(automation: automation)
let context = await MCPToolTestHelpers.makeContext(
automation: automation, executionPolicy: .foregroundAllowed)
let target = CGPoint(x: start.x + vector.0, y: start.y + vector.1)
let response = try await MoveTool(context: context).execute(arguments: ToolArguments(raw: [
"to": "\(target.x),\(target.y)", "foreground": true,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,8 @@ struct MCPHumanMoveParameterTests {
])
func `human movement refuses invalid overrides before pointer dispatch`(_ input: (String, Int)) async throws {
let automation = MockAutomationService(accessibilityGranted: true)
let context = await MCPToolTestHelpers.makeLegacyContext(automation: automation)
let context = await MCPToolTestHelpers.makeContext(
automation: automation, executionPolicy: .foregroundAllowed)
let response = try await MoveTool(context: context).execute(arguments: ToolArguments(raw: [
"to": "50,60", "foreground": true, "profile": "human", input.0: input.1,
]))
Expand All @@ -35,7 +36,8 @@ struct MCPHumanMoveParameterTests {
@Test(arguments: ["linear", "human"])
func `valid smooth overrides still execute for both profiles`(_ profile: String) async throws {
let automation = MockAutomationService(accessibilityGranted: true)
let context = await MCPToolTestHelpers.makeLegacyContext(automation: automation)
let context = await MCPToolTestHelpers.makeContext(
automation: automation, executionPolicy: .foregroundAllowed)
let response = try await MoveTool(context: context).execute(arguments: ToolArguments(raw: [
"to": "50,60", "foreground": true, "smooth": true, "profile": profile, "duration": 1000, "steps": 50,
]))
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
import CoreGraphics
import MCP
import PeekabooAutomationKit
import TachikomaMCP
import Testing
@testable import PeekabooAgentRuntime
@testable import PeekabooAutomation
@testable import PeekabooCore

@MainActor
struct MCPSecondaryDisplayPointerTests {
@Test(arguments: [
("-100,200", CGPoint(x: -100, y: 200)),
("100,-200", CGPoint(x: 100, y: -200)),
("-100,-200", CGPoint(x: -100, y: -200)),
])
func `move accepts signed global coordinates for secondary displays`(_ input: (String, CGPoint)) async throws {
let automation = MockAutomationService(accessibilityGranted: true)
let context = await MCPToolTestHelpers.makeContext(
automation: automation, executionPolicy: .foregroundAllowed)
let response = try await MoveTool(context: context).execute(arguments: ToolArguments(raw: [
"to": input.0, "foreground": true,
]))
#expect(!response.isError)
#expect(automation.lastMoveTarget == input.1)
}

@Test
func `foreground drag accepts endpoints left of and above the primary display`() async throws {
let automation = MockAutomationService(accessibilityGranted: true)
let context = await MCPToolTestHelpers.makeContext(
automation: automation, executionPolicy: .foregroundAllowed)
let response = try await DragTool(context: context).execute(arguments: ToolArguments(raw: [
"from_coords": "-100,200", "to_coords": "100,-200", "foreground": true,
]))
#expect(!response.isError)
#expect(automation.dragRequests.count == 1)
let request = try #require(automation.dragRequests.first)
#expect(request.from == CGPoint(x: -100, y: 200))
#expect(request.to == CGPoint(x: 100, y: -200))
}

@Test(arguments: ["-20001,1", "1,-20001", "20001,1", "nan,1", "inf,1"])
func `malformed or extreme foreground coordinates are refused`(_ coordinates: String) async throws {
let automation = MockAutomationService(accessibilityGranted: true)
let context = await MCPToolTestHelpers.makeContext(
automation: automation, executionPolicy: .foregroundAllowed)
let move = try await MoveTool(context: context).execute(arguments: ToolArguments(raw: [
"to": coordinates, "foreground": true,
]))
let drag = try await DragTool(context: context).execute(arguments: ToolArguments(raw: [
"from_coords": "0,0", "to_coords": coordinates, "foreground": true,
]))
#expect(move.isError)
#expect(drag.isError)
#expect(automation.lastMoveTarget == nil)
#expect(automation.dragRequests.isEmpty)
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -1106,6 +1106,7 @@ class MockAutomationService: ExactWindowTargetedClickServiceProtocol, TargetedHo
private(set) var lastHotkeyKeys: String?
private(set) var lastHotkeyHoldDuration: Int?
private(set) var lastMoveTarget: CGPoint?
private(set) var dragRequests: [DragOperationRequest] = []
private(set) var lastMoveDuration: Int?
private(set) var lastWindowContext: WindowContext?
var supportsTargetedHotkeys = true
Expand Down Expand Up @@ -1307,7 +1308,9 @@ class MockAutomationService: ExactWindowTargetedClickServiceProtocol, TargetedHo
WaitForElementResult(found: false, element: nil, waitTime: 0)
}

func drag(_: DragOperationRequest) async throws {}
func drag(_ request: DragOperationRequest) async throws {
self.dragRequests.append(request)
}

func moveMouse(
to: CGPoint,
Expand Down
4 changes: 4 additions & 0 deletions docs/commands/see.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,10 @@ means the evidence is unknown or not applicable. Freshness does not mean complet
retain their existing warnings and targeting restrictions. Text-selection range readback remains limited to the
focused field; `--fresh` does not make unfocused ranges available.

Disk-backed snapshot readback preserves the dialog and truncation information recorded by detection.
Replacing a raw screenshot clears this classification together with the old element map; storing another detection
replaces those values. Older stored records without these fields keep their existing defaults.

Pixels and Accessibility metadata are collected at different times, not as one atomic application-state snapshot.
During asynchronous UI updates, an image can still show the old value while AX already reports the new value.
Atomic snapshot publication binds the returned artifacts; it does not make their acquisition simultaneous. After
Expand Down
5 changes: 5 additions & 0 deletions docs/human-mouse-move.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,11 @@ Peekaboo's `human` profile makes cursor motion look hand-driven without forcing
- **CLI**: use `--smooth` for a natural `peekaboo move`, or add `--profile human` to `move` or `drag`. Duration/sample counts pick sensible defaults per distance. Explicit `--duration` and `--steps` values are honored; human paths never emit more than 96 samples.
- **Agents / MCP**: include `"profile": "human"` in move/drag tool arguments. Optional `duration` and `steps` fields work the same way as in the CLI—you only need them when you want to clamp the adaptive heuristics.

MCP `move` requires `foreground: true`. Smooth movement accepts a duration of 1–30000 ms and 1–100 steps,
including the human profile; invalid overrides are rejected before delivery. Foreground MCP move/drag coordinates
accept values from -20000 to 20000 on either axis, including displays left of or above the primary display.
Compass direction metadata uses screen coordinates: positive x points east and positive y points south.

## Defaults at a glance

| Distance | Typical Duration | Typical Steps | Notes |
Expand Down
Loading