feat(cua): restore windows without activation and reject stale coordinates
This commit is contained in:
1 parent
4e5234ac9c
commit
a83cebc4cf
20 files changed
+336
-30
No files matched your search
@@ -15,11 +15,11 @@
|
||||
en: d152484eb41ac6b4
|
||||
zh: 7b30b973a849247d
|
||||
/deepseek-ai-dsh-cua/use-this-package:
|
||||
en: d8c3e57d3f825b2d
|
||||
zh: 9dc16af1e8b738b5
|
||||
en: d9f53efd0a975464
|
||||
zh: c8b826321c5bc91d
|
||||
/deepseek-ai-dsh-cua/use-this-package/connect-an-external-execution-environment:
|
||||
en: a9a65a2953478d19
|
||||
zh: ef06dc75607e1724
|
||||
en: 52302a8fccc8fe38
|
||||
zh: facf0b651e90e2be
|
||||
/deepseek-ai-dsh-cua/understand-the-implementation:
|
||||
en: 0ab772a0dce859f6
|
||||
zh: 7851201c70372a25
|
||||
|
||||
@@ -34,7 +34,7 @@ export DSH_CUA_HELPER="$cua_native_root/.build/release/dsh-cua-native"
|
||||
pnpm dsh headless --patch packages/computer-use/cua/examples/macos.cordis.yml --dump-config
|
||||
```
|
||||
|
||||
The dump command checks profile composition without a model request. For a real task, replace `--dump-config` with a prompt and configure the profile’s model credentials. The overlay mounts the computer-use registry and this provider; remove an existing competing provider before using it. The responsible host/helper needs Accessibility permission for AX/input and Screen Recording permission for capture. `getCapabilities` reports permissions and session lock state without opening a system dialog. Locked or unavailable desktop sessions reject UI observations and input before dispatch; the user must unlock macOS before resuming. `listApps` and `cua.getState` list running applications. `findApps(query)` searches standard macOS application directories by name, bundle ID or absolute path. `getApp` selects a unique name, bundle ID, path or running PID, obtains application consent, and launches a stopped app in the background when needed. A stopped executable is identified before approval and checked again before and after launch; unsigned code only receives task access. The returned object reads initial AX without activating the app. Launch cancellation may leave the application running. Model guidance starts with background observation and input, without activation during initialization or before each action. If a hidden window, another Space or a Stage Manager thumbnail prevents observation, explain the foreground-dependent step before `app.activate()` or an observed window’s `raise` action, then read fresh state. Applications may change focus after input; background delivery does not guarantee an undisturbed user session.
|
||||
The dump command checks profile composition without a model request. For a real task, replace `--dump-config` with a prompt and configure the profile’s model credentials. The overlay mounts the computer-use registry and this provider; remove an existing competing provider before using it. The responsible host/helper needs Accessibility permission for AX/input and Screen Recording permission for capture. `getCapabilities` reports permissions and session lock state without opening a system dialog. Locked or unavailable desktop sessions reject UI observations and input before dispatch; the user must unlock macOS before resuming. `listApps` and `cua.getState` list running applications. `findApps(query)` searches standard macOS application directories by name, bundle ID or absolute path. `getApp` selects a unique name, bundle ID, path or running PID, obtains application consent, and launches a stopped app in the background when needed. A stopped executable is identified before approval and checked again before and after launch; unsigned code only receives task access. The returned object reads initial AX without activating the app. Launch cancellation may leave the application running. Model guidance starts with background observation and input, without activation during initialization or before each action. For hidden or minimized windows, try `app.show()` or `window.show()` and read fresh state. This requests unhide and unminimize without activation, raising or switching Spaces; app-level restoration selects the focused, main or sole AX window. The result names the selected `windowId`, or null when no window was selected, and reports `foregroundChanged` by comparing the front app and top window before and after the request. Null means foreground evidence was unavailable; later focus changes remain possible. If restoration fails or the window remains unusable, explain the needed foreground step, call `app.activate()` or an observed window’s `raise` action, then observe again. Necessary foreground access takes priority over preserving focus. Applications may change focus after input; background delivery does not guarantee an undisturbed user session.
|
||||
|
||||
| Field | Default | Meaning |
|
||||
| --- | --- | --- |
|
||||
@@ -89,7 +89,7 @@ Use app methods directly for single-window tasks. `getApp` reads initial AX stat
|
||||
|
||||
Screenshot data must use canonical Base64. Response byte limits apply before parsing, and attachment admission validates the raster and its configured capacity.
|
||||
|
||||
Capture and pointer input reject known disagreement between the selected window’s accessibility frame and displayed bounds with `WINDOW_CHANGED`, before posting pointer events. This includes Stage Manager transformations when AX retains the full window frame. Read fresh state; any activation or window raise remains explicit. Advertised AX actions remain available. Missing AX geometry retains the AXless path and does not establish that a transformed window can be operated.
|
||||
Coordinate clicks, scrolling and dragging require a successful AX read or screenshot of that window. A width or height change since the latest observation returns `WINDOW_CHANGED` before input, including raw calls without a snapshot ID; read fresh AX or take a new screenshot before proceeding. Window movement alone preserves window-local coordinates. Capture and pointer input reject known disagreement between the selected window’s accessibility frame and displayed bounds with `WINDOW_CHANGED`, before posting pointer events. This includes Stage Manager transformations when AX retains the full window frame. Read fresh state; any activation or window raise remains explicit. Advertised AX actions remain available. Missing AX geometry retains the AXless path and does not establish that a transformed window can be operated.
|
||||
|
||||
Retained objects resolve the current tool binding on every call. The external runtime owns persistence, imports, cancellation and completion-value serialization. Screenshots return PNG bytes; the standard tool pipeline admits the image attachment exactly once. `emit: false` suppresses facade output only. A host that uses another dispatcher can build objects through `createCuaApi` from `./api`.
|
||||
|
||||
@@ -161,6 +161,7 @@ Object API signatures (options default to { emit: true }):
|
||||
|
||||
App and Window methods:
|
||||
|
||||
- show() -> { dispatched: true, verified: false, windowId: number | null, foregroundChanged: boolean | null }
|
||||
- getAXState(options?: { emit?: boolean, disableDiffing?: boolean, includeMenus?: boolean, details?: boolean }) -> string
|
||||
- getScreenshot(options?: { emit?: boolean }) -> Uint8Array
|
||||
- getAXStateAndScreenshot(options?) -> { state: string, screenshot: Uint8Array }
|
||||
@@ -171,9 +172,9 @@ App and Window methods:
|
||||
- scroll({ x, y, deltaLinesX, deltaLinesY }) -> action result
|
||||
- drag(from: [x, y], to: [x, y], options?: { mouseButton?: "left" | "right" | "middle", modifiers?: Array<"command" | "control" | "option" | "shift" | "fn"> }) -> action result
|
||||
|
||||
Raw operation tools use { method, params }, including params: {} for parameterless methods. getApp selects without activating the app. Keep applications in the background for observation and routine input. Do not activate or raise an app during initialization, before each action, or merely because another app is foreground. Prefer advertised AX actions. If a hidden window, another Space or Stage Manager prevents usable observation, explain the foreground-dependent step before app.activate() or an advertised raise action, then read fresh state. Avoid repeated activation when the user switches away; continue background-capable steps and report what requires foreground access. Applications may still change focus in response to input.
|
||||
Raw operation tools use { method, params }, including params: {} for parameterless methods. getApp selects without activating the app. Keep applications in the background for observation and routine input. Do not activate or raise an app during initialization, before each action, or merely because another app is foreground. Prefer advertised AX actions. If a window is hidden or minimized, try app.show() or window.show() and read fresh state. show requests unhide and unminimize without activation, raising or switching Spaces; it chooses the explicit, focused, main or sole AX window. A null windowId means no window was selected for restoration. foregroundChanged compares the front app and top window before and after the request; null means unavailable, and delayed focus changes remain possible. If restoration is unsupported or the window remains unusable, including another Space or Stage Manager, explain the needed foreground step, use app.activate() or an advertised raise action, and read fresh state. Completing the task takes priority when foreground access is necessary. Avoid repeated activation when the user switches away; continue background-capable steps and report what requires foreground access. Applications may still change focus in response to input.
|
||||
|
||||
Await operations sequentially. After input, read fresh state to verify the effect. A successful action means dispatched, not verified completion. Cancellation may leave partial effects; never retry uncertain input without observing. STALE_APP requires getApp again; STALE_SNAPSHOT requires getState again. WINDOW_CHANGED requires fresh state before any retry; use an advertised AX action or explicitly restore the window before screenshot or coordinate input, then observe again. PERMISSION_DENIED requires the user to grant macOS Accessibility or Screen Recording access in System Settings. After granting access, observe fresh state before continuing; do not automatically replay prior input.
|
||||
Await operations sequentially. After input, read fresh state to verify the effect. A successful action means dispatched, not verified completion. Cancellation may leave partial effects; never retry uncertain input without observing. STALE_APP requires getApp again; STALE_SNAPSHOT requires getState again. Coordinate clicks, scrolling and dragging require a successful AX observation or screenshot of that window. Window resizing since the latest observation rejects coordinate input; movement alone preserves window-local coordinates. WINDOW_CHANGED requires fresh state before any retry; use an advertised AX action or explicitly restore the window before screenshot or coordinate input, then observe again. PERMISSION_DENIED requires the user to grant macOS Accessibility or Screen Recording access in System Settings. After granting access, observe fresh state before continuing; do not automatically replay prior input.
|
||||
|
||||
Use selectText only on controls advertising selectionWritable. It selects a unique literal occurrence, or places the cursor before/after it; prefix and suffix must be immediately adjacent. AMBIGUOUS_TEXT requires more context; TEXT_NOT_FOUND or SELECTION_CHANGED requires a fresh observation before deciding on input. Left clicks support single/double clicks; right and middle clicks support single clicks. Modified clicks use pointer input. Keypad names such as KP_0 and KP_Enter retain their physical key identity. Window objects expose observation and pointer/AX actions; keyboard input remains app-scoped and follows current focus. Scroll deltas are integer lines from -10 to 10; positive values reveal content above or left. Paste delivers plain text or HTML through the application's enabled standard Paste menu command. Original clipboard formats are restored if the clipboard still belongs to this paste; a later copy is preserved. Unsupported Paste commands fail without typing fallback. CLIPBOARD_CHANGED means another writer intervened; observe before deciding on further input. Browser applications can be controlled with getApp like other native apps; browser tabs are unavailable. Installed-app search covers standard macOS application directories; use an absolute application path for another location. Launch does not restore a window or document, and cancellation may leave an application already started. The execution host must install the object API and own variable persistence; this plugin does not preload a cua global. In persistent PTC, use the host-provided initialization snippet below once. Its installation-owned file URL works independently of the project directory. Retain app objects between calls. Observation methods emit by default and also return their value. Use void await app.getAXState() to avoid a duplicate REPL completion, or pass emit:false and print the returned text once. End screenshot calls with void await app.getScreenshot() or void await app.getAXStateAndScreenshot() so image bytes are not serialized as the program completion value.
|
||||
```
|
||||
|
||||
@@ -34,7 +34,7 @@ export DSH_CUA_HELPER="$cua_native_root/.build/release/dsh-cua-native"
|
||||
pnpm dsh headless --patch packages/computer-use/cua/examples/macos.cordis.yml --dump-config
|
||||
```
|
||||
|
||||
配置输出命令不发起模型请求。执行真实任务时,将 `--dump-config` 替换为任务提示,并配置该 profile 的模型凭据。配置挂载 computer-use 注册表和本 provider;使用前需移除已有的竞争 provider。负责执行的宿主或辅助进程需要辅助功能权限来读取 AX 和输入,需要屏幕录制权限来截图。`getCapabilities` 查询权限和锁屏状态,不弹出系统授权窗口。桌面会话锁定或不可用时,界面观察与输入会在分派前拒绝,需由用户解锁 macOS 后继续。`listApps` 和 `cua.getState` 列出运行中的应用。`findApps(query)` 按名称、bundle ID 或绝对路径搜索 macOS 标准应用目录。`getApp` 按唯一名称、bundle ID、路径或运行中的 PID 选择应用,取得应用许可后,在需要时后台启动尚未运行的应用。启动前先确认可执行文件身份,授权后在启动前后再次核对;未签名代码只取得任务许可。返回的对象读取初始 AX,不自动激活应用。启动被取消时,应用可能已经运行。模型指引要求先在后台观察和输入,不在初始化或每次动作前激活应用。窗口隐藏、位于其他桌面空间或被台前调度缩为缩略图而无法观察时,先说明需要前台的步骤,再显式调用 `app.activate()` 或窗口的 `raise` 动作,之后重新读取状态。应用收到输入后仍可能改变焦点,后台投递不保证用户操作始终不受干扰。
|
||||
配置输出命令不发起模型请求。执行真实任务时,将 `--dump-config` 替换为任务提示,并配置该 profile 的模型凭据。配置挂载 computer-use 注册表和本 provider;使用前需移除已有的竞争 provider。负责执行的宿主或辅助进程需要辅助功能权限来读取 AX 和输入,需要屏幕录制权限来截图。`getCapabilities` 查询权限和锁屏状态,不弹出系统授权窗口。桌面会话锁定或不可用时,界面观察与输入会在分派前拒绝,需由用户解锁 macOS 后继续。`listApps` 和 `cua.getState` 列出运行中的应用。`findApps(query)` 按名称、bundle ID 或绝对路径搜索 macOS 标准应用目录。`getApp` 按唯一名称、bundle ID、路径或运行中的 PID 选择应用,取得应用许可后,在需要时后台启动尚未运行的应用。启动前先确认可执行文件身份,授权后在启动前后再次核对;未签名代码只取得任务许可。返回的对象读取初始 AX,不自动激活应用。启动被取消时,应用可能已经运行。模型指引要求先在后台观察和输入,不在初始化或每次动作前激活应用。窗口隐藏或最小化时,先尝试 `app.show()` 或 `window.show()`,再重新观察。它请求取消隐藏和恢复最小化,不主动激活、抬起窗口或切换桌面空间;应用级恢复选择焦点窗口、主窗口或唯一 AX 窗口。结果通过 `windowId` 指明选中的窗口,未选中时为 null;`foregroundChanged` 比较操作前后的前台应用和最上层窗口,无法取得完整观察时为 null,返回后仍可能发生焦点变化。恢复失败或窗口仍不可用时,说明需要前台的步骤,再调用 `app.activate()` 或窗口的 `raise` 动作并重新观察。必须使用前台时,以完成任务为先。应用收到输入后仍可能改变焦点,后台投递不保证用户操作始终不受干扰。
|
||||
|
||||
| 字段 | 默认值 | 含义 |
|
||||
| --- | --- | --- |
|
||||
@@ -89,7 +89,7 @@ void await app.getScreenshot()
|
||||
|
||||
截图数据必须使用规范 Base64。响应字节上限在解析前执行,附件接入校验图像数据及配置的容量。
|
||||
|
||||
截图和鼠标输入会核对所选窗口的辅助功能框与显示边界;已知两者不一致时,在发送任何鼠标事件前返回 `WINDOW_CHANGED`。当 AX 保留完整窗口框时,这也会拦住 Stage Manager 变换后的窗口。需要重新读取状态;激活应用或抬起窗口仍须显式调用。元素声明的 AX 动作保持可用。缺少 AX 几何时保留无 AX 路径,不代表变换后的窗口一定可以操作。
|
||||
坐标点击、滚动和拖动要求先成功读取该窗口的 AX 或截图。窗口宽高与最近一次观察不同时,在输入前返回 `WINDOW_CHANGED`,原始调用即使不传快照 ID 也会检查;重新读取 AX 或截图后再操作。单纯移动窗口不使窗口内坐标失效。截图和鼠标输入会核对所选窗口的辅助功能框与显示边界;已知两者不一致时,在发送任何鼠标事件前返回 `WINDOW_CHANGED`。当 AX 保留完整窗口框时,这也会拦住 Stage Manager 变换后的窗口。需要重新读取状态;激活应用或抬起窗口仍须显式调用。元素声明的 AX 动作保持可用。缺少 AX 几何时保留无 AX 路径,不代表变换后的窗口一定可以操作。
|
||||
|
||||
保留的对象会在每次调用时获取当前工具绑定。外部运行时负责持久化、导入、取消和最终表达式值的序列化。截图返回 PNG 字节,标准工具链负责将图片作为附件传递一次。`emit: false` 只控制对象层的输出。使用其他分派器的宿主可以通过 `./api` 的 `createCuaApi` 创建对象。
|
||||
|
||||
@@ -161,6 +161,7 @@ Object API signatures (options default to { emit: true }):
|
||||
|
||||
App and Window methods:
|
||||
|
||||
- show() -> { dispatched: true, verified: false, windowId: number | null, foregroundChanged: boolean | null }
|
||||
- getAXState(options?: { emit?: boolean, disableDiffing?: boolean, includeMenus?: boolean, details?: boolean }) -> string
|
||||
- getScreenshot(options?: { emit?: boolean }) -> Uint8Array
|
||||
- getAXStateAndScreenshot(options?) -> { state: string, screenshot: Uint8Array }
|
||||
@@ -171,9 +172,9 @@ App and Window methods:
|
||||
- scroll({ x, y, deltaLinesX, deltaLinesY }) -> action result
|
||||
- drag(from: [x, y], to: [x, y], options?: { mouseButton?: "left" | "right" | "middle", modifiers?: Array<"command" | "control" | "option" | "shift" | "fn"> }) -> action result
|
||||
|
||||
Raw operation tools use { method, params }, including params: {} for parameterless methods. getApp selects without activating the app. Keep applications in the background for observation and routine input. Do not activate or raise an app during initialization, before each action, or merely because another app is foreground. Prefer advertised AX actions. If a hidden window, another Space or Stage Manager prevents usable observation, explain the foreground-dependent step before app.activate() or an advertised raise action, then read fresh state. Avoid repeated activation when the user switches away; continue background-capable steps and report what requires foreground access. Applications may still change focus in response to input.
|
||||
Raw operation tools use { method, params }, including params: {} for parameterless methods. getApp selects without activating the app. Keep applications in the background for observation and routine input. Do not activate or raise an app during initialization, before each action, or merely because another app is foreground. Prefer advertised AX actions. If a window is hidden or minimized, try app.show() or window.show() and read fresh state. show requests unhide and unminimize without activation, raising or switching Spaces; it chooses the explicit, focused, main or sole AX window. A null windowId means no window was selected for restoration. foregroundChanged compares the front app and top window before and after the request; null means unavailable, and delayed focus changes remain possible. If restoration is unsupported or the window remains unusable, including another Space or Stage Manager, explain the needed foreground step, use app.activate() or an advertised raise action, and read fresh state. Completing the task takes priority when foreground access is necessary. Avoid repeated activation when the user switches away; continue background-capable steps and report what requires foreground access. Applications may still change focus in response to input.
|
||||
|
||||
Await operations sequentially. After input, read fresh state to verify the effect. A successful action means dispatched, not verified completion. Cancellation may leave partial effects; never retry uncertain input without observing. STALE_APP requires getApp again; STALE_SNAPSHOT requires getState again. WINDOW_CHANGED requires fresh state before any retry; use an advertised AX action or explicitly restore the window before screenshot or coordinate input, then observe again. PERMISSION_DENIED requires the user to grant macOS Accessibility or Screen Recording access in System Settings. After granting access, observe fresh state before continuing; do not automatically replay prior input.
|
||||
Await operations sequentially. After input, read fresh state to verify the effect. A successful action means dispatched, not verified completion. Cancellation may leave partial effects; never retry uncertain input without observing. STALE_APP requires getApp again; STALE_SNAPSHOT requires getState again. Coordinate clicks, scrolling and dragging require a successful AX observation or screenshot of that window. Window resizing since the latest observation rejects coordinate input; movement alone preserves window-local coordinates. WINDOW_CHANGED requires fresh state before any retry; use an advertised AX action or explicitly restore the window before screenshot or coordinate input, then observe again. PERMISSION_DENIED requires the user to grant macOS Accessibility or Screen Recording access in System Settings. After granting access, observe fresh state before continuing; do not automatically replay prior input.
|
||||
|
||||
Use selectText only on controls advertising selectionWritable. It selects a unique literal occurrence, or places the cursor before/after it; prefix and suffix must be immediately adjacent. AMBIGUOUS_TEXT requires more context; TEXT_NOT_FOUND or SELECTION_CHANGED requires a fresh observation before deciding on input. Left clicks support single/double clicks; right and middle clicks support single clicks. Modified clicks use pointer input. Keypad names such as KP_0 and KP_Enter retain their physical key identity. Window objects expose observation and pointer/AX actions; keyboard input remains app-scoped and follows current focus. Scroll deltas are integer lines from -10 to 10; positive values reveal content above or left. Paste delivers plain text or HTML through the application's enabled standard Paste menu command. Original clipboard formats are restored if the clipboard still belongs to this paste; a later copy is preserved. Unsupported Paste commands fail without typing fallback. CLIPBOARD_CHANGED means another writer intervened; observe before deciding on further input. Browser applications can be controlled with getApp like other native apps; browser tabs are unavailable. Installed-app search covers standard macOS application directories; use an absolute application path for another location. Launch does not restore a window or document, and cancellation may leave an application already started. The execution host must install the object API and own variable persistence; this plugin does not preload a cua global. In persistent PTC, use the host-provided initialization snippet below once. Its installation-owned file URL works independently of the project directory. Retain app objects between calls. Observation methods emit by default and also return their value. Use void await app.getAXState() to avoid a duplicate REPL completion, or pass emit:false and print the returned text once. End screenshot calls with void await app.getScreenshot() or void await app.getAXStateAndScreenshot() so image bytes are not serialized as the program completion value.
|
||||
```
|
||||
|
||||
@@ -112,6 +112,29 @@ final class AXAccess {
|
||||
guard let focused = try element(AXUIElementCreateApplication(pid), "AXFocusedWindow") else { return nil }
|
||||
return try windowID(focused)
|
||||
}
|
||||
/// Restore a selected or unambiguous window without raising it or requesting application activation.
|
||||
func restoreWindow(_ requested: UInt32?, owner: ProcessIdentity) throws -> UInt32? {
|
||||
let app = AXUIElementCreateApplication(owner.pid)
|
||||
let windows = try children(app, "AXWindows")
|
||||
let selected: AXUIElement?
|
||||
if let requested {
|
||||
_ = try NativePlatform.ownedWindow(owner.pid, requested)
|
||||
selected = try windows.first { try windowID($0) == requested }
|
||||
guard selected != nil else { throw NativeFailure(code: "WINDOW_UNAVAILABLE", message: "Window is not exposed for restoration; read fresh state before deciding whether to activate") }
|
||||
} else {
|
||||
selected = try element(app, "AXFocusedWindow") ?? element(app, "AXMainWindow") ?? (windows.count == 1 ? windows[0] : nil)
|
||||
}
|
||||
guard let selected else { return nil }
|
||||
if try boolean(selected, "AXMinimized") == true {
|
||||
guard try writable(selected, "AXMinimized") == true else {
|
||||
throw NativeFailure(code: "UNSUPPORTED", message: "Window cannot be restored without activation; read fresh state before deciding whether to activate")
|
||||
}
|
||||
try prepare(selected)
|
||||
let error = AXUIElementSetAttributeValue(selected, "AXMinimized" as CFString, kCFBooleanFalse)
|
||||
guard error == .success else { throw failure(error, "restore minimized window") }
|
||||
}
|
||||
return try windowID(selected)
|
||||
}
|
||||
func actions(_ element: AXUIElement) throws -> [String] {
|
||||
try prepare(element)
|
||||
var names: CFArray?
|
||||
|
||||
@@ -24,6 +24,7 @@ struct CaptureTarget: Sendable {
|
||||
let process: ProcessIdentity
|
||||
let window: WindowInfo
|
||||
}
|
||||
struct ShowPlan: Sendable { let appHandle: String; let identity: ProcessIdentity; let windowID: UInt32? }
|
||||
struct ForegroundPlan: Sendable {
|
||||
let appHandle: String
|
||||
let identity: ProcessIdentity
|
||||
@@ -64,7 +65,7 @@ private enum ApplicationTarget {
|
||||
case running(ProcessIdentity)
|
||||
case pending(InstalledApplication, JSONValue)
|
||||
}
|
||||
enum DriverOutcome: Sendable { case result(JSONValue), capture(CaptureTarget), foreground(ForegroundPlan), launch(ApplicationLaunch), paste(PastePlan) }
|
||||
enum DriverOutcome: Sendable { case result(JSONValue), capture(CaptureTarget), foreground(ForegroundPlan), show(ShowPlan), launch(ApplicationLaunch), paste(PastePlan) }
|
||||
|
||||
actor AXWorker {
|
||||
nonisolated let executor = NativeSerialExecutor()
|
||||
@@ -73,6 +74,7 @@ actor AXWorker {
|
||||
private var generation: UInt64 = 0
|
||||
private var targets: [String: ApplicationTarget] = [:]
|
||||
private var snapshots: [SnapshotScope: NativeSnapshot] = [:]
|
||||
private var observedWindowBounds: [String: [UInt32: WindowBounds]] = [:]
|
||||
private let cancellation: CancellationLatch
|
||||
private let axTimeout: Float
|
||||
private let textEventInterval: Double
|
||||
@@ -108,12 +110,13 @@ actor AXWorker {
|
||||
}
|
||||
private func observeSession() -> Bool? {
|
||||
let state = NativePlatform.sessionLocked()
|
||||
if state != false { snapshots.removeAll() }
|
||||
if state != false { snapshots.removeAll(); observedWindowBounds.removeAll() }
|
||||
return state
|
||||
}
|
||||
private func forget(_ handle: String) {
|
||||
targets.removeValue(forKey: handle)
|
||||
snapshots = snapshots.filter { $0.key.appHandle != handle }
|
||||
observedWindowBounds.removeValue(forKey: handle)
|
||||
}
|
||||
func validateIdentity(_ handle: String) throws -> ProcessIdentity {
|
||||
guard !closed else { throw NativeFailure(code: "STALE_APP", message: "Helper is closing; acquire fresh application handles") }
|
||||
@@ -157,6 +160,11 @@ actor AXWorker {
|
||||
return try ax().windowBounds(windowID, owner: identity)
|
||||
}
|
||||
}
|
||||
private func observedCoordinateWindow(_ handle: String, _ identity: ProcessIdentity, _ windowID: UInt32) throws -> WindowInfo {
|
||||
let window = try coordinateWindow(identity, windowID)
|
||||
try WindowGeometry.validateObservedSize(observed: observedWindowBounds[handle]?[windowID], current: window.bounds)
|
||||
return window
|
||||
}
|
||||
static func actionKey(_ value: String) -> String {
|
||||
let name = AXTextFormatter.actionName(value)
|
||||
return name.filter { !$0.isWhitespace }.lowercased()
|
||||
@@ -233,11 +241,21 @@ actor AXWorker {
|
||||
// Remove closed window scopes using the current WindowServer identity set.
|
||||
let ids = Set(windows.map(\.windowID))
|
||||
snapshots = snapshots.filter { key, _ in key.appHandle != handle || (key.windowID.map { ids.contains($0) } ?? true) }
|
||||
observedWindowBounds[handle] = observedWindowBounds[handle]?.filter { ids.contains($0.key) }
|
||||
return .result(.array(windows.map(\.json)))
|
||||
}
|
||||
if method == "getState" { snapshots.removeValue(forKey: SnapshotScope(appHandle: handle, windowID: try params.optionalWindowID())) }
|
||||
if method == "getState" {
|
||||
let windowID = try params.optionalWindowID()
|
||||
snapshots.removeValue(forKey: SnapshotScope(appHandle: handle, windowID: windowID))
|
||||
if let windowID { observedWindowBounds[handle]?.removeValue(forKey: windowID) }
|
||||
else { observedWindowBounds.removeValue(forKey: handle) }
|
||||
}
|
||||
if method != "screenshot" { try NativePlatform.requireAccessibility() }
|
||||
switch method {
|
||||
case "show":
|
||||
let windowID = try params.optionalWindowID()
|
||||
if let windowID { _ = try NativePlatform.ownedWindow(identity.pid, windowID) }
|
||||
return .show(ShowPlan(appHandle: handle, identity: identity, windowID: windowID))
|
||||
case "activate":
|
||||
return .foreground(ForegroundPlan(appHandle: handle, identity: identity, snapshotID: nil, index: nil, targetWindowID: nil, role: nil, main: nil, minimized: nil))
|
||||
case "getState":
|
||||
@@ -260,6 +278,9 @@ actor AXWorker {
|
||||
let id = try identifier("snapshot")
|
||||
let scope = SnapshotScope(appHandle: handle, windowID: window)
|
||||
snapshots[scope] = NativeSnapshot(id: id, scope: scope, elements: projection.indexed)
|
||||
for observed in observedWindows where window == nil || observed.windowID == window {
|
||||
observedWindowBounds[handle, default: [:]][observed.windowID] = observed.bounds
|
||||
}
|
||||
return .result(.object(["appHandle": .string(handle), "pid": .number(Double(identity.pid)),
|
||||
"windowId": window.map { .number(Double($0)) } ?? .null,
|
||||
"focusedWindowId": projection.focusedWindowID.map { .number(Double($0)) } ?? .null,
|
||||
@@ -315,13 +336,13 @@ actor AXWorker {
|
||||
} else {
|
||||
let id = try params.windowID()
|
||||
try coordinateSnapshot(handle, params, id)
|
||||
let window = try coordinateWindow(identity, id)
|
||||
let window = try observedCoordinateWindow(handle, identity, id)
|
||||
try inputs().1.click(in: WindowRoute(owner: identity, windowID: id, bounds: window.bounds), at: params.point(), button: button, count: count, modifiers: modifiers)
|
||||
}
|
||||
case "scroll", "drag":
|
||||
let id = try params.windowID()
|
||||
try coordinateSnapshot(handle, params, id)
|
||||
let window = try coordinateWindow(identity, id)
|
||||
let window = try observedCoordinateWindow(handle, identity, id)
|
||||
let route = WindowRoute(owner: identity, windowID: id, bounds: window.bounds)
|
||||
if method == "scroll" {
|
||||
let dx = params.contains("deltaLinesX") ? try params.integer("deltaLinesX", in: -10...10) : 0
|
||||
@@ -378,6 +399,14 @@ actor AXWorker {
|
||||
return ForegroundObservation(frontPID: NativePlatform.frontPID(), focusedWindowID: try ax().focusedWindowID(plan.identity.pid),
|
||||
frontWindowID: top?.windowID, frontWindowPID: top?.pid)
|
||||
}
|
||||
func restoreWindow(_ plan: ShowPlan) throws -> UInt32? {
|
||||
try cancellation.check()
|
||||
let identity = try validateIdentity(plan.appHandle)
|
||||
guard identity == plan.identity else { throw NativeFailure(code: "STALE_APP", message: "Application changed before restoring its window") }
|
||||
try NativePlatform.requireUnlocked(observeSession())
|
||||
try NativePlatform.requireAccessibility()
|
||||
return try ax().restoreWindow(plan.windowID, owner: identity)
|
||||
}
|
||||
static func shouldRaise(_ plan: ForegroundPlan, before: ForegroundObservation) -> Bool {
|
||||
guard let id = plan.targetWindowID else { return true }
|
||||
return !(plan.role == "AXWindow" && plan.main == true && plan.minimized == false && before.frontPID == plan.identity.pid
|
||||
@@ -400,9 +429,10 @@ actor AXWorker {
|
||||
try NativePlatform.requireScreenRecording()
|
||||
let after = try coordinateWindow(identity, target.window.windowID)
|
||||
guard after.bounds == target.window.bounds else { throw NativeFailure(code: "WINDOW_CHANGED", message: "Window moved or resized during capture; observe again") }
|
||||
observedWindowBounds[target.appHandle, default: [:]][after.windowID] = after.bounds
|
||||
return .object(["mimeType": .string("image/png"), "data": .string(image.bytes.base64EncodedString()),
|
||||
"pixelWidth": .number(Double(image.width)), "pixelHeight": .number(Double(image.height)), "bounds": after.bounds.json,
|
||||
"coordinateSpace": .string("window-points"), "windowId": .number(Double(after.windowID))])
|
||||
}
|
||||
func shutdown() { closed = true; snapshots.removeAll(); targets.removeAll(); access = nil; keyboard = nil; pointer = nil }
|
||||
func shutdown() { closed = true; snapshots.removeAll(); observedWindowBounds.removeAll(); targets.removeAll(); access = nil; keyboard = nil; pointer = nil }
|
||||
}
|
||||
@@ -29,6 +29,13 @@ struct ApplicationCatalogSnapshot: Sendable {
|
||||
func application(_ pid: Int32) -> ApplicationInfo? { applications.first { $0.pid == pid } }
|
||||
}
|
||||
@MainActor enum ApplicationCatalog {
|
||||
static func unhide(_ identity: ProcessIdentity) throws {
|
||||
try NativePlatform.verify(identity)
|
||||
guard let app = NSRunningApplication(processIdentifier: identity.pid), !app.isTerminated else {
|
||||
throw NativeFailure(code: "APP_UNAVAILABLE", message: "Application exited before showing its windows")
|
||||
}
|
||||
if app.isHidden && !app.unhide() { throw NativeFailure(code: "NATIVE_ERROR", message: "macOS refused to unhide the application") }
|
||||
}
|
||||
static func snapshot() -> ApplicationCatalogSnapshot {
|
||||
let apps = NSWorkspace.shared.runningApplications.filter { !$0.isTerminated }.map { app in
|
||||
ApplicationInfo(pid: app.processIdentifier, name: app.localizedName, bundleID: app.bundleIdentifier,
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
// Generated by native/generate-protocol.ts from src/protocol.ts. Do not edit.
|
||||
enum NativeProtocol {
|
||||
static let modelMethods: [String] = ["activate", "click", "drag", "findApps", "getApp", "getCapabilities", "getState", "listApps", "listWindows", "paste", "performSecondaryAction", "pressKey", "screenshot", "scroll", "selectText", "setValue", "typeText"]
|
||||
static let modelMethods: [String] = ["activate", "click", "drag", "findApps", "getApp", "getCapabilities", "getState", "listApps", "listWindows", "paste", "performSecondaryAction", "pressKey", "screenshot", "scroll", "selectText", "setValue", "show", "typeText"]
|
||||
static let fields: [String: Set<String>] = [
|
||||
"activate": ["appHandle"],
|
||||
"click": ["appHandle", "clickCount", "elementIndex", "modifiers", "mouseButton", "snapshotId", "windowId", "x", "y"],
|
||||
@@ -19,6 +19,7 @@ enum NativeProtocol {
|
||||
"scroll": ["appHandle", "deltaLinesX", "deltaLinesY", "snapshotId", "windowId", "x", "y"],
|
||||
"selectText": ["appHandle", "elementIndex", "prefix", "selectionType", "snapshotId", "suffix", "text"],
|
||||
"setValue": ["appHandle", "elementIndex", "snapshotId", "value"],
|
||||
"show": ["appHandle", "windowId"],
|
||||
"typeText": ["appHandle", "text"],
|
||||
]
|
||||
static let stringEnums: [String: [String: Set<String>]] = [
|
||||
|
||||
@@ -86,6 +86,7 @@ import Darwin
|
||||
let image = try await self.capture.capture(target, cancellation: self.cancellation)
|
||||
result = try await self.worker.finishCapture(target, image: image)
|
||||
case .foreground(let plan): result = try await self.performForeground(plan, requestID: request.id)
|
||||
case .show(let plan): result = try await self.performShow(plan)
|
||||
case .paste(let plan):
|
||||
try await self.clipboard.perform(text: plan.text, format: plan.format, maxBytes: plan.maxBytes,
|
||||
checkCancellation: { try self.cancellation.check() }) { requireClipboard in
|
||||
@@ -143,6 +144,21 @@ import Darwin
|
||||
"before": before.json, "after": after.json, "activationRequested": .bool(activationRequested), "raiseRequested": .bool(raiseRequested),
|
||||
])])
|
||||
}
|
||||
private func performShow(_ plan: ShowPlan) async throws -> JSONValue {
|
||||
try cancellation.check()
|
||||
let beforePID = NativePlatform.frontPID()
|
||||
let beforeWindow = NativePlatform.windows().first { $0.isOnScreen }?.windowID
|
||||
try ApplicationCatalog.unhide(plan.identity)
|
||||
let windowID = try await worker.restoreWindow(plan)
|
||||
let afterPID = NativePlatform.frontPID()
|
||||
let afterWindow = NativePlatform.windows().first { $0.isOnScreen }?.windowID
|
||||
let changed: JSONValue
|
||||
if let beforePID, let afterPID, let beforeWindow, let afterWindow {
|
||||
changed = .bool(beforePID != afterPID || beforeWindow != afterWindow)
|
||||
} else { changed = .null }
|
||||
return .object(["dispatched": .bool(true), "verified": .bool(false),
|
||||
"windowId": windowID.map { .number(Double($0)) } ?? .null, "foregroundChanged": changed])
|
||||
}
|
||||
private func transportFailed(_ failure: NativeFailure) {
|
||||
if let bytes = "\(failure.code): \(failure.message)\n".data(using: .utf8) { try? FileHandle.standardError.write(contentsOf: bytes) }
|
||||
shutdown(status: 1)
|
||||
|
||||
@@ -4,6 +4,16 @@ import CoreGraphics
|
||||
enum WindowGeometry {
|
||||
static let conflictMessage = "Window display and accessibility bounds disagree. Read fresh state; if unchanged, use AX actions or explicitly restore the window before capture or coordinate input"
|
||||
|
||||
/// Window-local coordinates survive movement, but require a fresh observation after resizing.
|
||||
static func validateObservedSize(observed: WindowBounds?, current: WindowBounds) throws {
|
||||
guard let observed else {
|
||||
throw NativeFailure(code: "STALE_SNAPSHOT", message: "Window has no observation; read getAXState or take a screenshot before coordinate input")
|
||||
}
|
||||
guard observed.width == current.width, observed.height == current.height else {
|
||||
throw NativeFailure(code: "WINDOW_CHANGED", message: "Window size changed since observation; read getAXState or take a screenshot before coordinate input")
|
||||
}
|
||||
}
|
||||
|
||||
/// Read CG bounds around the AX query; missing AX geometry does not disable an owned window.
|
||||
static func resolve(
|
||||
owner: ProcessIdentity, windowID: UInt32,
|
||||
|
||||
@@ -150,6 +150,42 @@
|
||||
],
|
||||
"additionalProperties": false
|
||||
},
|
||||
{
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"id": {
|
||||
"type": "string"
|
||||
},
|
||||
"method": {
|
||||
"type": "string",
|
||||
"const": "show"
|
||||
},
|
||||
"params": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"appHandle": {
|
||||
"type": "string",
|
||||
"minLength": 1
|
||||
},
|
||||
"windowId": {
|
||||
"type": "integer",
|
||||
"exclusiveMinimum": 0,
|
||||
"maximum": 4294967295
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"appHandle"
|
||||
],
|
||||
"additionalProperties": false
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"id",
|
||||
"method",
|
||||
"params"
|
||||
],
|
||||
"additionalProperties": false
|
||||
},
|
||||
{
|
||||
"type": "object",
|
||||
"properties": {
|
||||
|
||||
@@ -2,19 +2,22 @@ import AppKit
|
||||
import Darwin
|
||||
|
||||
/// Removes this app's AX roots while keeping its CG window and native button alive.
|
||||
final class AxlessFixture: NSObject, NSApplicationDelegate {
|
||||
final class AxlessFixture: NSObject, NSApplicationDelegate, NSWindowDelegate {
|
||||
let statePath: String
|
||||
var window: NSWindow!
|
||||
var button: NSButton!
|
||||
var savedChildren: [Any]?
|
||||
var hiddenAX = false
|
||||
var count = 0
|
||||
var resizeCount = 0
|
||||
var moved = false
|
||||
var controls: [DispatchSourceSignal] = []
|
||||
|
||||
init(statePath: String) { self.statePath = statePath }
|
||||
func applicationDidFinishLaunching(_ notification: Notification) {
|
||||
window = NSWindow(contentRect: NSRect(x: 120, y: 120, width: 360, height: 180),
|
||||
styleMask: [.titled, .closable], backing: .buffered, defer: false)
|
||||
styleMask: [.titled, .closable, .miniaturizable], backing: .buffered, defer: false)
|
||||
window.delegate = self
|
||||
window.title = "DSH AXless window fixture"
|
||||
window.isReleasedWhenClosed = false
|
||||
window.animationBehavior = .none
|
||||
@@ -25,10 +28,25 @@ final class AxlessFixture: NSObject, NSApplicationDelegate {
|
||||
window.makeKeyAndOrderFront(nil)
|
||||
NSApplication.shared.activate(ignoringOtherApps: true)
|
||||
savedChildren = NSApplication.shared.accessibilityChildren()
|
||||
for number in [SIGUSR1, SIGUSR2] {
|
||||
for number in [SIGUSR1, SIGUSR2, SIGWINCH, SIGURG, SIGHUP, SIGALRM] {
|
||||
signal(number, SIG_IGN)
|
||||
let control = DispatchSource.makeSignalSource(signal: number, queue: .main)
|
||||
control.setEventHandler { [weak self] in self?.setAXHidden(number == SIGUSR1) }
|
||||
control.setEventHandler { [weak self] in
|
||||
guard let self else { return }
|
||||
switch number {
|
||||
case SIGUSR1, SIGUSR2: self.setAXHidden(number == SIGUSR1)
|
||||
case SIGWINCH:
|
||||
self.resizeCount += 1
|
||||
self.window.setContentSize(NSSize(width: 360 + 40 * self.resizeCount, height: 180 + 20 * self.resizeCount))
|
||||
self.persist()
|
||||
case SIGHUP: self.window.miniaturize(nil)
|
||||
case SIGALRM: NSApplication.shared.hide(nil)
|
||||
default:
|
||||
self.window.setFrameOrigin(NSPoint(x: 180, y: 180))
|
||||
self.moved = true
|
||||
self.persist()
|
||||
}
|
||||
}
|
||||
control.resume()
|
||||
controls.append(control)
|
||||
}
|
||||
@@ -37,6 +55,12 @@ final class AxlessFixture: NSObject, NSApplicationDelegate {
|
||||
fflush(stdout)
|
||||
}
|
||||
@objc func increment() { count += 1; persist() }
|
||||
func applicationDidBecomeActive(_ notification: Notification) { persist() }
|
||||
func applicationDidResignActive(_ notification: Notification) { persist() }
|
||||
func applicationDidHide(_ notification: Notification) { persist() }
|
||||
func applicationDidUnhide(_ notification: Notification) { persist() }
|
||||
func windowDidMiniaturize(_ notification: Notification) { persist() }
|
||||
func windowDidDeminiaturize(_ notification: Notification) { persist() }
|
||||
func setAXHidden(_ hidden: Bool) {
|
||||
let app = NSApplication.shared
|
||||
app.setAccessibilityChildren(hidden ? [] : savedChildren)
|
||||
@@ -47,11 +71,14 @@ final class AxlessFixture: NSObject, NSApplicationDelegate {
|
||||
persist()
|
||||
}
|
||||
func persist() {
|
||||
guard window != nil, button != nil else { return }
|
||||
let rect = button.convert(button.bounds, to: nil)
|
||||
let data = try! JSONSerialization.data(withJSONObject: [
|
||||
"pid": ProcessInfo.processInfo.processIdentifier,
|
||||
"windowId": window.windowNumber,
|
||||
"hiddenAX": hiddenAX, "count": count,
|
||||
"hiddenAX": hiddenAX, "count": count, "resizeCount": resizeCount, "moved": moved,
|
||||
"hidden": NSApplication.shared.isHidden, "minimized": window.isMiniaturized,
|
||||
"active": NSApplication.shared.isActive,
|
||||
"size": [window.frame.width, window.frame.height],
|
||||
"buttonPoint": [rect.midX, window.frame.height - rect.midY],
|
||||
])
|
||||
|
||||
@@ -24,6 +24,7 @@ def main():
|
||||
raise RuntimeError('this opt-in check requires macOS')
|
||||
helper = NativeHelper([], args.helper)
|
||||
fixture = None
|
||||
foreground = None
|
||||
try:
|
||||
capabilities = helper.call('getCapabilities', {})
|
||||
if capabilities['sessionLocked'] is not False:
|
||||
@@ -56,6 +57,7 @@ def main():
|
||||
owner = {'appHandle': app['appHandle']}
|
||||
window_id = actual['windowId']
|
||||
params = {**owner, 'windowId': window_id, 'maxElements': 100, 'maxDepth': 15}
|
||||
helper.call('click', {**owner, 'windowId': window_id, 'x': 1, 'y': 1}, 'STALE_SNAPSHOT')
|
||||
first = helper.call('getState', params)
|
||||
buttons = [element for element in first['elements'] if element['title'] == 'Increment AXless fixture']
|
||||
assert len(buttons) == 1, first
|
||||
@@ -77,12 +79,57 @@ def main():
|
||||
recovered = helper.call('getState', params)
|
||||
assert 'Increment AXless fixture' in recovered['tree']
|
||||
helper.call('click', {**owner, 'windowId': window_id, 'snapshotId': empty['snapshotId'], 'x': x, 'y': y}, 'STALE_SNAPSHOT')
|
||||
|
||||
os.kill(fixture.pid, signal.SIGWINCH)
|
||||
actual = wait_state(lambda state: state['resizeCount'] == 1, 'fixture did not resize')
|
||||
for snapshot in [{}, {'snapshotId': recovered['snapshotId']}]:
|
||||
target = {**owner, 'windowId': window_id, **snapshot}
|
||||
helper.call('click', {**target, 'x': x, 'y': y}, 'WINDOW_CHANGED')
|
||||
helper.call('scroll', {**target, 'x': x, 'y': y, 'deltaLinesY': -1}, 'WINDOW_CHANGED')
|
||||
helper.call('drag', {**target, 'fromX': x, 'fromY': y, 'toX': x + 1, 'toY': y,
|
||||
'durationMs': 100, 'steps': 2}, 'WINDOW_CHANGED')
|
||||
assert json.loads(state_path.read_text())['count'] == 1
|
||||
helper.call('getState', params)
|
||||
x, y = actual['buttonPoint']
|
||||
helper.call('click', {**owner, 'windowId': window_id, 'x': x, 'y': y})
|
||||
wait_state(lambda state: state['count'] == 2, 'fresh AX did not admit resized-window input')
|
||||
|
||||
os.kill(fixture.pid, signal.SIGURG)
|
||||
wait_state(lambda state: state['moved'], 'fixture did not move')
|
||||
helper.call('click', {**owner, 'windowId': window_id, 'x': x, 'y': y})
|
||||
wait_state(lambda state: state['count'] == 3, 'window movement invalidated local coordinates')
|
||||
|
||||
os.kill(fixture.pid, signal.SIGWINCH)
|
||||
actual = wait_state(lambda state: state['resizeCount'] == 2, 'fixture did not resize again')
|
||||
helper.call('screenshot', {**owner, 'windowId': window_id})
|
||||
x, y = actual['buttonPoint']
|
||||
helper.call('click', {**owner, 'windowId': window_id, 'x': x, 'y': y})
|
||||
wait_state(lambda state: state['count'] == 4, 'fresh screenshot did not admit resized-window input')
|
||||
|
||||
foreground_path = root / 'foreground.json'
|
||||
foreground = subprocess.Popen([str(binary), str(foreground_path)], stdout=subprocess.PIPE,
|
||||
text=True, start_new_session=True)
|
||||
assert line(foreground).strip() == 'READY'
|
||||
wait_state(lambda state: not state['active'], 'foreground fixture did not take focus')
|
||||
for number, key, target in [(signal.SIGALRM, 'hidden', owner),
|
||||
(signal.SIGHUP, 'minimized', {**owner, 'windowId': window_id})]:
|
||||
os.kill(fixture.pid, number)
|
||||
wait_state(lambda state: state[key], 'fixture did not ' + key)
|
||||
restored = helper.call('show', target)
|
||||
actual = wait_state(lambda state: not state[key], 'show did not restore ' + key)
|
||||
assert restored['windowId'] == window_id, restored
|
||||
assert restored['dispatched'] and not restored['verified'], restored
|
||||
assert restored['foregroundChanged'] is False, restored
|
||||
assert not actual['active'] and json.loads(foreground_path.read_text())['active']
|
||||
helper.call('show', {**owner, 'windowId': json.loads(foreground_path.read_text())['windowId']}, 'WINDOW_UNAVAILABLE')
|
||||
helper.eof()
|
||||
print('PASS: AXless owned window, new empty generation, stale indices, native screenshot/click, restored AX')
|
||||
print('PASS: AXless input, resize guards, fresh observation, movement and background unhide/unminimize')
|
||||
finally:
|
||||
stop_group(fixture)
|
||||
fixture = None
|
||||
finally:
|
||||
if foreground is not None:
|
||||
stop_group(foreground)
|
||||
if fixture is not None:
|
||||
stop_group(fixture)
|
||||
helper.stop()
|
||||
|
||||
@@ -6,6 +6,28 @@ private func frame(_ x: Double, _ y: Double, _ width: Double, _ height: Double)
|
||||
WindowBounds(x: x, y: y, width: width, height: height)
|
||||
}
|
||||
|
||||
@Test func coordinateInputRequiresAnObservedWindowSize() {
|
||||
do {
|
||||
try WindowGeometry.validateObservedSize(observed: nil, current: frame(0, 0, 320, 200))
|
||||
Issue.record("Unobserved coordinates were accepted")
|
||||
} catch let error as NativeFailure { #expect(error.code == "STALE_SNAPSHOT") }
|
||||
catch { Issue.record("Unexpected error: \(error)") }
|
||||
}
|
||||
|
||||
@Test(arguments: [frame(0, 0, 321, 200), frame(0, 0, 320, 201), frame(0, 0, 320.5, 200)])
|
||||
func coordinateInputRejectsAWindowResizedSinceObservation(_ current: WindowBounds) {
|
||||
do {
|
||||
try WindowGeometry.validateObservedSize(observed: frame(0, 0, 320, 200), current: current)
|
||||
Issue.record("Coordinates from before resizing were accepted")
|
||||
} catch let error as NativeFailure { #expect(error.code == "WINDOW_CHANGED") }
|
||||
catch { Issue.record("Unexpected error: \(error)") }
|
||||
}
|
||||
|
||||
@Test func coordinateInputAllowsWindowMovementAndFreshlyObservedSizes() throws {
|
||||
try WindowGeometry.validateObservedSize(observed: frame(0, 0, 320, 200), current: frame(-900, 120, 320, 200))
|
||||
try WindowGeometry.validateObservedSize(observed: frame(0, 0, 500, 400), current: frame(0, 0, 500, 400))
|
||||
}
|
||||
|
||||
@Test(arguments: [
|
||||
(frame(221, 50, 1200, 832), frame(16, 512, 131, 145)),
|
||||
(frame(-600, -100, 320, 200), frame(-599, -100, 320, 200)),
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
/** Object facade over logged native operations; its caller owns evaluator persistence. */
|
||||
import { Buffer } from 'node:buffer'
|
||||
import type { JsonValue } from '@deepseek-ai/dsh-util-values'
|
||||
import { ActionResult, AppResult, AppsResult, CapabilitiesResult, FoundAppsResult, Request, ScreenshotResult, StateResult, WindowsResult } from './protocol.ts'
|
||||
import { ActionResult, AppResult, AppsResult, CapabilitiesResult, FoundAppsResult, Request, ScreenshotResult, ShowResult, StateResult, WindowsResult } from './protocol.ts'
|
||||
import type { CuaApi, CuaApp, CuaAppHandle, CuaDispatch, CuaObservationOptions, CuaObservationSink, CuaOperation, CuaSurface, CuaWindow } from './types.ts'
|
||||
export type * from './types.ts'
|
||||
|
||||
@@ -55,6 +55,11 @@ export function createCuaApi(dispatch: CuaDispatch, sink?: CuaObservationSink):
|
||||
return first.windowId
|
||||
}
|
||||
const result: CuaSurface = {
|
||||
async show() {
|
||||
return ShowResult.parse(await call('show', { appHandle,
|
||||
...selectedWindow === undefined ? {} : { windowId: selectedWindow },
|
||||
}))
|
||||
},
|
||||
async getAXState(options) {
|
||||
observed = undefined
|
||||
const state = StateResult.parse(await call('getState', { appHandle,
|
||||
|
||||
@@ -109,6 +109,7 @@ Object API signatures (options default to { emit: true }):
|
||||
|
||||
App and Window methods:
|
||||
|
||||
- show() -> { dispatched: true, verified: false, windowId: number | null, foregroundChanged: boolean | null }
|
||||
- getAXState(options?: { emit?: boolean, disableDiffing?: boolean, includeMenus?: boolean, details?: boolean }) -> string
|
||||
- getScreenshot(options?: { emit?: boolean }) -> Uint8Array
|
||||
- getAXStateAndScreenshot(options?) -> { state: string, screenshot: Uint8Array }
|
||||
@@ -119,9 +120,9 @@ App and Window methods:
|
||||
- scroll({ x, y, deltaLinesX, deltaLinesY }) -> action result
|
||||
- drag(from: [x, y], to: [x, y], options?: { mouseButton?: "left" | "right" | "middle", modifiers?: Array<"command" | "control" | "option" | "shift" | "fn"> }) -> action result
|
||||
|
||||
Raw operation tools use { method, params }, including params: {} for parameterless methods. getApp selects without activating the app. Keep applications in the background for observation and routine input. Do not activate or raise an app during initialization, before each action, or merely because another app is foreground. Prefer advertised AX actions. If a hidden window, another Space or Stage Manager prevents usable observation, explain the foreground-dependent step before app.activate() or an advertised raise action, then read fresh state. Avoid repeated activation when the user switches away; continue background-capable steps and report what requires foreground access. Applications may still change focus in response to input.
|
||||
Raw operation tools use { method, params }, including params: {} for parameterless methods. getApp selects without activating the app. Keep applications in the background for observation and routine input. Do not activate or raise an app during initialization, before each action, or merely because another app is foreground. Prefer advertised AX actions. If a window is hidden or minimized, try app.show() or window.show() and read fresh state. show requests unhide and unminimize without activation, raising or switching Spaces; it chooses the explicit, focused, main or sole AX window. A null windowId means no window was selected for restoration. foregroundChanged compares the front app and top window before and after the request; null means unavailable, and delayed focus changes remain possible. If restoration is unsupported or the window remains unusable, including another Space or Stage Manager, explain the needed foreground step, use app.activate() or an advertised raise action, and read fresh state. Completing the task takes priority when foreground access is necessary. Avoid repeated activation when the user switches away; continue background-capable steps and report what requires foreground access. Applications may still change focus in response to input.
|
||||
|
||||
Await operations sequentially. After input, read fresh state to verify the effect. A successful action means dispatched, not verified completion. Cancellation may leave partial effects; never retry uncertain input without observing. STALE_APP requires getApp again; STALE_SNAPSHOT requires getState again. WINDOW_CHANGED requires fresh state before any retry; use an advertised AX action or explicitly restore the window before screenshot or coordinate input, then observe again. PERMISSION_DENIED requires the user to grant macOS Accessibility or Screen Recording access in System Settings. After granting access, observe fresh state before continuing; do not automatically replay prior input.
|
||||
Await operations sequentially. After input, read fresh state to verify the effect. A successful action means dispatched, not verified completion. Cancellation may leave partial effects; never retry uncertain input without observing. STALE_APP requires getApp again; STALE_SNAPSHOT requires getState again. Coordinate clicks, scrolling and dragging require a successful AX observation or screenshot of that window. Window resizing since the latest observation rejects coordinate input; movement alone preserves window-local coordinates. WINDOW_CHANGED requires fresh state before any retry; use an advertised AX action or explicitly restore the window before screenshot or coordinate input, then observe again. PERMISSION_DENIED requires the user to grant macOS Accessibility or Screen Recording access in System Settings. After granting access, observe fresh state before continuing; do not automatically replay prior input.
|
||||
|
||||
Use selectText only on controls advertising selectionWritable. It selects a unique literal occurrence, or places the cursor before/after it; prefix and suffix must be immediately adjacent. AMBIGUOUS_TEXT requires more context; TEXT_NOT_FOUND or SELECTION_CHANGED requires a fresh observation before deciding on input. Left clicks support single/double clicks; right and middle clicks support single clicks. Modified clicks use pointer input. Keypad names such as KP_0 and KP_Enter retain their physical key identity. Window objects expose observation and pointer/AX actions; keyboard input remains app-scoped and follows current focus. Scroll deltas are integer lines from -10 to 10; positive values reveal content above or left. Paste delivers plain text or HTML through the application's enabled standard Paste menu command. Original clipboard formats are restored if the clipboard still belongs to this paste; a later copy is preserved. Unsupported Paste commands fail without typing fallback. CLIPBOARD_CHANGED means another writer intervened; observe before deciding on further input. Browser applications can be controlled with getApp like other native apps; browser tabs are unavailable. Installed-app search covers standard macOS application directories; use an absolute application path for another location. Launch does not restore a window or document, and cancellation may leave an application already started. The execution host must install the object API and own variable persistence; this plugin does not preload a cua global. In persistent PTC, use the host-provided initialization snippet below once. Its installation-owned file URL works independently of the project directory. Retain app objects between calls. Observation methods emit by default and also return their value. Use void await app.getAXState() to avoid a duplicate REPL completion, or pass emit:false and print the returned text once. End screenshot calls with void await app.getScreenshot() or void await app.getAXStateAndScreenshot() so image bytes are not serialized as the program completion value.`
|
||||
|
||||
|
||||
@@ -29,6 +29,7 @@ export const Request = z.discriminatedUnion('method', [
|
||||
operation('getCapabilities', {}), operation('listApps', {}), operation('findApps', { query: z.string().min(1).regex(/\S/) }),
|
||||
operation('getApp', { selector: z.string().min(1) }),
|
||||
operation('activate', app),
|
||||
operation('show', { ...app, windowId: id.optional() }),
|
||||
operation('listWindows', app),
|
||||
operation('getState', { ...app, windowId: id.optional(),
|
||||
includeMenus: z.boolean().optional().describe('Expand menu contents; omitted or false shows menu-bar entries only.'),
|
||||
@@ -97,3 +98,5 @@ export const CapabilitiesResult = z.object({
|
||||
})
|
||||
/** An action has reached native dispatch; application state was not verified. */
|
||||
export const ActionResult = z.object({ dispatched: z.literal(true), verified: z.literal(false) })
|
||||
/** Background restoration dispatch and sampled foreground changes; null means the observation was unavailable. */
|
||||
export const ShowResult = ActionResult.extend({ windowId: id.nullable(), foregroundChanged: z.boolean().nullable() })
|
||||
@@ -2,7 +2,7 @@
|
||||
import type { Branded } from '@deepseek-ai/dsh-brand'
|
||||
import type { JsonValue } from '@deepseek-ai/dsh-util-values'
|
||||
import type { z } from 'zod'
|
||||
import type { ActionResult, CapabilitiesResult, FoundAppsResult, Request, WindowsResult } from './protocol.ts'
|
||||
import type { ActionResult, CapabilitiesResult, FoundAppsResult, Request, ShowResult, WindowsResult } from './protocol.ts'
|
||||
|
||||
/** Helper-issued application selection; public app objects bind it to one process lifetime. */
|
||||
export type CuaAppHandle = Branded<'CuaAppHandle'>
|
||||
@@ -49,6 +49,8 @@ export type CuaBounds = CuaWindowInfo['bounds']
|
||||
export type CuaWindowInfo = z.output<typeof WindowsResult>[number]
|
||||
/** Successful input dispatch; application outcome still requires observation. */
|
||||
export type CuaActionResult = z.output<typeof ActionResult>
|
||||
/** Restoration dispatch; foregroundChanged compares the front app and top window before and after the request. */
|
||||
export type CuaShowResult = z.output<typeof ShowResult>
|
||||
/** Modifiers attached to a window-directed pointer gesture. */
|
||||
export type CuaPointerModifier = NonNullable<CuaParameters<'click'>['modifiers']>[number]
|
||||
/** Buttons supported by window-directed clicks and drags. */
|
||||
@@ -68,9 +70,15 @@ export interface CuaObservation { state: string; screenshot: Uint8Array }
|
||||
/**
|
||||
* App-wide or explicitly scoped window observations and actions.
|
||||
* Capture and pointer delivery reject known AX/display frame conflicts with WINDOW_CHANGED.
|
||||
* Coordinate input also rejects resizing since the latest successful AX read or screenshot.
|
||||
* Advertised AX actions remain available; failed actions are not retried through pointer input.
|
||||
*/
|
||||
export interface CuaSurface {
|
||||
/**
|
||||
* Unhide the app and restore a minimized selected, focused, main or sole window without requesting activation.
|
||||
* @returns Dispatch evidence, the selected window when available, and sampled foreground change; read fresh state before input.
|
||||
*/
|
||||
show(): Promise<CuaShowResult>
|
||||
/**
|
||||
* Read fresh AX text, with menu contents collapsed by default.
|
||||
* @param options - emission, menu expansion and field detail options.
|
||||
|
||||
@@ -25,6 +25,7 @@ function fixture(withSink = true) {
|
||||
case 'listApps': return [selected, { pid: 18, name: 'Utility' }]
|
||||
case 'getApp': return selected
|
||||
case 'listWindows': return windows
|
||||
case 'show': return { ...delivered, windowId: params.windowId ?? 23, foregroundChanged: false }
|
||||
case 'getState': return {
|
||||
appHandle: selected.appHandle, pid: 17, windowId: params.windowId ?? null,
|
||||
snapshotId: `snapshot-${++generation}`, tree: `Tree ${generation}`, truncated: false,
|
||||
@@ -104,6 +105,39 @@ describe('application discovery', () => {
|
||||
})
|
||||
})
|
||||
|
||||
describe('background window restoration', () => {
|
||||
it('shows an app or selected window without dispatching activation or AXRaise', async () => {
|
||||
const f = fixture()
|
||||
const app = await f.api.getApp('Editor')
|
||||
const window = await app.getWindow(24)
|
||||
f.dispatch.mockClear()
|
||||
expect(await app.show()).toEqual({ ...delivered, windowId: 23, foregroundChanged: false })
|
||||
expect(await window.show()).toEqual({ ...delivered, windowId: 24, foregroundChanged: false })
|
||||
expect(f.dispatch.mock.calls.map(([operation]) => operation)).toEqual([
|
||||
{ method: 'show', params: { appHandle: 'app-17' } },
|
||||
{ method: 'show', params: { appHandle: 'app-17', windowId: 24 } },
|
||||
])
|
||||
})
|
||||
|
||||
it.each([true, null])('preserves foreground change evidence %j without switching focus back', async (foregroundChanged) => {
|
||||
const f = fixture()
|
||||
const app = await f.api.getApp('Editor')
|
||||
f.dispatch.mockClear()
|
||||
f.dispatch.mockResolvedValueOnce({ ...delivered, windowId: null, foregroundChanged })
|
||||
expect(await app.show()).toEqual({ ...delivered, windowId: null, foregroundChanged })
|
||||
expect(f.dispatch).toHaveBeenCalledTimes(1)
|
||||
})
|
||||
|
||||
it('leaves foreground fallback to the caller after restoration fails', async () => {
|
||||
const f = fixture()
|
||||
const app = await f.api.getApp('Editor')
|
||||
f.dispatch.mockClear()
|
||||
f.dispatch.mockRejectedValueOnce(new Error('UNSUPPORTED: Window cannot be restored'))
|
||||
await expect(app.show()).rejects.toThrow('UNSUPPORTED')
|
||||
expect(f.dispatch).toHaveBeenCalledTimes(1)
|
||||
})
|
||||
})
|
||||
|
||||
describe('pointer buttons and modifiers', () => {
|
||||
it('retains middle-button and modifier intent through click and drag dispatch', async () => {
|
||||
const f = fixture()
|
||||
|
||||
@@ -362,6 +362,39 @@
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"kind": "chunks",
|
||||
"chunks": [
|
||||
{
|
||||
"type": "block-start",
|
||||
"index": 0,
|
||||
"blockType": "tool-call"
|
||||
},
|
||||
{
|
||||
"type": "block-end",
|
||||
"index": 0,
|
||||
"block": {
|
||||
"type": "tool-call",
|
||||
"id": "cua-native-show",
|
||||
"name": "cua",
|
||||
"arguments": "{\"method\":\"show\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "usage",
|
||||
"usage": {
|
||||
"inputTokens": 3,
|
||||
"outputTokens": 3
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "finish",
|
||||
"reason": {
|
||||
"kind": "tool-calls"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"kind": "chunks",
|
||||
"chunks": [
|
||||
|
||||
@@ -17,7 +17,7 @@ const window = { windowId: 7, pid: 9, appName: 'Fixture', title: 'CUA fixture',
|
||||
const axlessWindow = { ...window, windowId: 8, title: 'AXless canvas' }
|
||||
const transformedWindow = { ...window, windowId: 9, title: 'Transformed window', bounds: { x: 16, y: 512, width: 131, height: 145 } }
|
||||
const snapshots = new Map()
|
||||
const methods = ['getCapabilities', 'listApps', 'getApp', 'activate', 'listWindows', 'getState', 'click', 'setValue', 'typeText', 'pressKey', 'scroll', 'drag', 'screenshot']
|
||||
const methods = ['getCapabilities', 'listApps', 'getApp', 'activate', 'show', 'listWindows', 'getState', 'click', 'setValue', 'typeText', 'pressKey', 'scroll', 'drag', 'screenshot']
|
||||
let acquired = false
|
||||
let generation = 0
|
||||
let count = 0
|
||||
@@ -43,6 +43,7 @@ function call(requestId, method, params) {
|
||||
return { ...app, authorization: {"id": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", "name": "Fixture", "executablePath": "/fixture/app", "persistable": true} }
|
||||
}
|
||||
if (!acquired || params.appHandle !== app.appHandle) refuse('APP_NOT_FOUND', 'Acquire an application in this helper first')
|
||||
if (method === 'show') return { dispatched: true, verified: false, windowId: params.windowId ?? window.windowId, foregroundChanged: false }
|
||||
if (method === 'activate') {
|
||||
const before = { frontPid, focusedWindowId: window.windowId, frontWindowId, frontWindowPid: frontPid }
|
||||
const activationRequested = frontPid !== app.pid
|
||||
|
||||
Reference in new issue
Block a user