diff --git a/.agents/notes/implemented/architecture/2026-11-05-cua-scoped-surfaces-and-input.i18n.yaml b/.agents/notes/implemented/architecture/2026-11-05-cua-scoped-surfaces-and-input.i18n.yaml index 3e32b6eb92..81912accfe 100644 --- a/.agents/notes/implemented/architecture/2026-11-05-cua-scoped-surfaces-and-input.i18n.yaml +++ b/.agents/notes/implemented/architecture/2026-11-05-cua-scoped-surfaces-and-input.i18n.yaml @@ -9,8 +9,8 @@ en: f72db0b79aa02fb5 zh: b91c1f177dc93ab4 /agent-note-cua-scoped-surfaces-and-separated-input-capabilities/decision: - en: ec115e11f96ffe60 - zh: c1fe9eb2866196d0 + en: 41fd533d5be7642b + zh: c6fcbabec22bdbb7 /agent-note-cua-scoped-surfaces-and-separated-input-capabilities/alternatives-considered: en: 32c970ae153e0f6c zh: 8431be0a10ffed15 diff --git a/.agents/notes/implemented/architecture/2026-11-05-cua-scoped-surfaces-and-input.md b/.agents/notes/implemented/architecture/2026-11-05-cua-scoped-surfaces-and-input.md index cce266e52b..6d9aea0408 100644 --- a/.agents/notes/implemented/architecture/2026-11-05-cua-scoped-surfaces-and-input.md +++ b/.agents/notes/implemented/architecture/2026-11-05-cua-scoped-surfaces-and-input.md @@ -16,7 +16,7 @@ Provider startup capabilities are static. Platform, supported methods, applicati Keyboard input keeps keys and modifiers separate: `pressKey(key, modifiers?)` presses and releases one key while holding the listed modifiers, and its keyboard modifier list is a plain string array defined by the provider. `sendKeyEvent(key, direction)` sends exactly one key-down or key-up without an implicit counterpart. A caller can therefore hold a modifier or split a gesture; neither method packs a chord string, and the shared schema does not enumerate platform key names. -Within one API instance, the object facade keeps one current accessibility snapshot per application handle, shared by every application and window object for that handle, including repeated selections of the same application. `getWindow` reads its initial tree silently; `getApp` does the same when the provider reports application-level accessibility. A later `getAccessibilityTree` read invalidates the previous snapshot before dispatch. The facade automatically attaches the latest snapshot ID to element operations; callers use element IDs from the latest tree and do not assume an older ID still identifies the same control. Element IDs are opaque provider strings. Only an explicit `CuaWindow` captures screenshots or sends pointer input. The native helper retains its own application and optional-window snapshots behind that facade; the native layer and the facade snapshot remain separate. The external runtime continues to own variable persistence and completion serialization; the facade adds no evaluator. The [native object decision](2026-09-17-experimental-cua-object-api.md) owns evaluator independence, and the [shared helper decision](2026-10-09-cua-shared-native-helper.md) owns connection lifetime. +Within one API instance, the object facade keeps one current accessibility snapshot per application handle, shared by every application and window object for that handle, including repeated selections of the same application. `getApp` and `getWindow` select objects without reading accessibility or replacing that snapshot; native window operations reject closed or foreign windows. An explicit `getAccessibilityTree` read invalidates the previous snapshot before dispatch. The facade automatically attaches the latest snapshot ID to element operations; callers use element IDs from the latest tree and do not assume an older ID still identifies the same control. Element IDs are opaque provider strings. Only an explicit `CuaWindow` captures screenshots or sends pointer input. The native helper retains its own application and optional-window snapshots behind that facade; the native layer and the facade snapshot remain separate. The external runtime continues to own variable persistence and completion serialization; the facade adds no evaluator. The [native object decision](2026-09-17-experimental-cua-object-api.md) owns evaluator independence, and the [shared helper decision](2026-10-09-cua-shared-native-helper.md) owns connection lifetime. ## Alternatives considered diff --git a/.agents/notes/implemented/architecture/2026-11-05-cua-scoped-surfaces-and-input.zh.md b/.agents/notes/implemented/architecture/2026-11-05-cua-scoped-surfaces-and-input.zh.md index 5135dc9be8..ceed375ff3 100644 --- a/.agents/notes/implemented/architecture/2026-11-05-cua-scoped-surfaces-and-input.zh.md +++ b/.agents/notes/implemented/architecture/2026-11-05-cua-scoped-surfaces-and-input.zh.md @@ -16,7 +16,7 @@ Provider 的启动能力是静态的。平台、支持的方法、应用级辅 键盘输入把按键与修饰键分开：`pressKey(key, modifiers?)` 在按住所列修饰键时按下并释放一个键，其键盘修饰键列表是 provider 定义的普通字符串数组。`sendKeyEvent(key, direction)` 只发送一次按下或抬起，不隐式发送相反事件。因此调用方可以保持修饰键或拆分手势；两个方法都不接受组合键字符串，共享 schema 也不枚举平台按键名。 -在单个 API 实例内，对象门面按应用句柄保存一份当前辅助功能快照，该句柄的所有应用对象和窗口对象共享这份快照，包括对同一应用的重复选择。`getWindow` 静默执行首次树读取；当 provider 报告应用级辅助功能时，`getApp` 也静默读取。之后的 `getAccessibilityTree` 读取会在分派前使上一份快照失效。门面会为元素操作自动附加最新快照 ID；调用方应使用最新树中的元素 ID，不应假定旧 ID 仍指向同一控件。元素 ID 是 provider 定义的不透明字符串。只有显式 `CuaWindow` 可以截图或发送指针输入。原生 helper 在该门面之后保留自己的应用快照和可选窗口快照；原生层与门面快照相互独立。外部运行时继续负责变量持久化和最终表达式序列化；门面不加入 evaluator。[原生对象决定](2026-09-17-experimental-cua-object-api.zh.md)负责 evaluator 独立性，[共享 helper 决定](2026-10-09-cua-shared-native-helper.zh.md)负责连接生命期。 +在单个 API 实例内，对象门面按应用句柄保存一份当前辅助功能快照，该句柄的所有应用对象和窗口对象共享这份快照，包括对同一应用的重复选择。`getApp` 和 `getWindow` 只选择对象，不读取辅助功能或替换该快照；原生窗口操作会拒绝已关闭或属于其他应用的窗口。显式 `getAccessibilityTree` 读取会在分派前使上一份快照失效。门面会为元素操作自动附加最新快照 ID；调用方应使用最新树中的元素 ID，不应假定旧 ID 仍指向同一控件。元素 ID 是 provider 定义的不透明字符串。只有显式 `CuaWindow` 可以截图或发送指针输入。原生 helper 在该门面之后保留自己的应用快照和可选窗口快照；原生层与门面快照相互独立。外部运行时继续负责变量持久化和最终表达式序列化；门面不加入 evaluator。[原生对象决定](2026-09-17-experimental-cua-object-api.zh.md)负责 evaluator 独立性，[共享 helper 决定](2026-10-09-cua-shared-native-helper.zh.md)负责连接生命期。 ## 考虑过的替代方案 diff --git a/docs/upgrade-guide/v0.2.2-rc.1/cua-native-platform/guide.i18n.yaml b/docs/upgrade-guide/v0.2.2-rc.1/cua-native-platform/guide.i18n.yaml index 6dfabdc004..acdb30680e 100644 --- a/docs/upgrade-guide/v0.2.2-rc.1/cua-native-platform/guide.i18n.yaml +++ b/docs/upgrade-guide/v0.2.2-rc.1/cua-native-platform/guide.i18n.yaml @@ -12,5 +12,5 @@ en: e5d1027faa55adba zh: 4b88ad6a70977fb3 /native-cua-application-and-window-api/migration: - en: 106eab81fd078fec - zh: 6fb30738af8bc621 + en: 071b67613c512b84 + zh: 7ee32d8dbcf7a0f2 diff --git a/docs/upgrade-guide/v0.2.2-rc.1/cua-native-platform/guide.md b/docs/upgrade-guide/v0.2.2-rc.1/cua-native-platform/guide.md index 2f4eea3c91..12d949e8b5 100644 --- a/docs/upgrade-guide/v0.2.2-rc.1/cua-native-platform/guide.md +++ b/docs/upgrade-guide/v0.2.2-rc.1/cua-native-platform/guide.md @@ -16,7 +16,7 @@ Providers supply startup capabilities alongside their platform guidance. The nat ## Migration 1. After `cua.getApp(selector)`, select the target from `app.getWindows()` with `app.getWindow(windowId)`. Use `window.getScreenshot()`, `window.click()`, `window.scroll()`, and `window.drag()`. Replace `hover` with `window.mouseMove()`. On macOS, `window.click(id)` sends mouse events at the element center. Switch to `invoke(id)` for the default accessibility action, or `performAction(id, "AXShowMenu")` to request an advertised menu action. -2. Replace `getAXState()` with `getAccessibilityTree()`. Replace `getAXStateAndScreenshot()` with separate tree and screenshot reads. Selection reads the initial tree silently for a window, and for an app when `appLevelAccessibility` is true. Read the tree explicitly to display it. See [macOS AX read limits](../cua-ax-read-limits/guide.md) for changes to traversal defaults and configuration. +2. Replace `getAXState()` with `getAccessibilityTree()`. Replace `getAXStateAndScreenshot()` with separate tree and screenshot reads. Application and window selection does not read AX. Before using element IDs, call `getAccessibilityTree()` explicitly; use `{ emit: false }` to retain the snapshot without displaying it. A screenshot does not require a successful AX read. Before coordinate input, including relative coordinates, obtain a successful AX observation covering the window or a screenshot; absolute coordinates require a screenshot. See [macOS AX read limits](../cua-ax-read-limits/guide.md) for changes to traversal defaults and configuration. 3. Pass the opaque string ID from the latest tree wherever you passed a numeric element index. In raw requests use `elementId` with `snapshotId`. Replace `performSecondaryAction` with `listActions(id)` and `performAction(id, exactName, argument?)`, or use `invoke(id)` for the default action. `setValue` accepts the provider’s supported JSON values. On macOS, `performAction(id, 'AXRaise')` performs the advertised raise action. Call `app.activate()` first when the application must become active. 4. Replace a packed chord such as `pressKey('shift+Tab')` with `pressKey('Tab', ['shift'])`. Check `(await cua.getCapabilities()).methods` before using `sendKeyEvent`. On macOS, call it inside one `run_code` to hold a key across tool calls. To request one repeat, send another down for a non-modifier key held by that execution. Returning, throwing or pausing releases any remaining keys. If your script handles unknown macOS key names as `NATIVE_ERROR`, handle `INVALID_ARGUMENT` instead. Correct the key name before retrying. 5. Read `dispatched` from action results, then observe the application to verify the outcome. `invoke` returns `dispatched: false` when the element has no default action. Remove reads of the old `verified` field and of `pid` on selected objects or window records. Use `listApps()` or running `findApps()` matches when you need a PID. diff --git a/docs/upgrade-guide/v0.2.2-rc.1/cua-native-platform/guide.zh.md b/docs/upgrade-guide/v0.2.2-rc.1/cua-native-platform/guide.zh.md index 77e65e7ba4..c74b879337 100644 --- a/docs/upgrade-guide/v0.2.2-rc.1/cua-native-platform/guide.zh.md +++ b/docs/upgrade-guide/v0.2.2-rc.1/cua-native-platform/guide.zh.md @@ -16,7 +16,7 @@ CUA 脚本在指定窗口上截图和操作鼠标。App 和 Window 对象共享 ## 迁移 1. 调用 `cua.getApp(selector)` 后，从 `app.getWindows()` 返回的列表选定目标，再调用 `app.getWindow(windowId)`。在窗口上调用 `getScreenshot()`、`click()`、`scroll()` 和 `drag()`，将原有的 `hover` 调用改为 `window.mouseMove()`。在 macOS 上，`window.click(id)` 在元素中心发送鼠标事件。需要执行默认辅助功能动作时，改用 `invoke(id)`。需要打开控件菜单且控件声明了 `AXShowMenu` 时，调用 `performAction(id, "AXShowMenu")`。 -2. 将 `getAXState()` 改为 `getAccessibilityTree()`，将 `getAXStateAndScreenshot()` 拆成读取树和截图两次调用。选择窗口时会静默读取初始树。选择应用时，只有 `appLevelAccessibility` 为真才读取。要显示树，请显式读取。[读取上限的迁移说明](../cua-ax-read-limits/guide.zh.md)列出了 macOS 默认值和配置含义的变化。 +2. 将 `getAXState()` 改为 `getAccessibilityTree()`，将 `getAXStateAndScreenshot()` 拆成读取树和截图两次调用。选择应用或窗口不读取 AX。使用元素 ID 前须显式调用 `getAccessibilityTree()`；使用 `{ emit: false }` 可保留快照但不显示树。截图不要求 AX 读取成功。所有坐标输入（包括相对坐标）之前都须成功读取覆盖该窗口的 AX 或取得截图；绝对坐标要求截图。[读取上限的迁移说明](../cua-ax-read-limits/guide.zh.md)列出了 macOS 默认值和配置含义的变化。 3. 将数字元素索引改为从最新树中取得的字符串 ID。直接发送请求时使用 `elementId` 和 `snapshotId`。原先调用 `performSecondaryAction` 的地方，先用 `listActions(id)` 取得完整动作名，再调用 `performAction(id, exactName, argument?)`。要执行默认动作，可调用 `invoke(id)`。`setValue` 接受提供方支持的 JSON 值。 在 macOS 上，`performAction(id, 'AXRaise')` 只执行元素声明的抬起窗口动作。需要把应用切到前台时先调用 `app.activate()`。 4. 将 `pressKey('shift+Tab')` 之类的组合键写法改为 `pressKey('Tab', ['shift'])`。调用 `sendKeyEvent` 前检查 `(await cua.getCapabilities()).methods`。在 macOS 上，同一次 `run_code` 中可调用此方法跨工具调用保持按键。如果需要一次重复按下，可在这次执行中对已按住的普通键再发一次 `down`。执行返回、异常退出或用户暂停时会释放仍按着的键。如果脚本用 `NATIVE_ERROR` 识别 macOS 上的未知键名，请改为处理 `INVALID_ARGUMENT`，并在重试前修正键名。 5. 从动作结果读取 `dispatched`，再观察应用确认效果。元素没有默认动作时，`invoke` 返回 `dispatched: false`。移除读取旧字段 `verified` 的代码，以及对已选对象和窗口记录中 `pid` 的读取。需要 PID 时，使用 `listApps()` 或 `findApps()` 返回的运行中应用记录。 diff --git a/packages/experimental/cua-macos/README.i18n.yaml b/packages/experimental/cua-macos/README.i18n.yaml index ca9b29d4d7..fc9fa048bc 100644 --- a/packages/experimental/cua-macos/README.i18n.yaml +++ b/packages/experimental/cua-macos/README.i18n.yaml @@ -15,11 +15,11 @@ en: 1777e7bf6785f986 zh: e3daac1074ebb74d /deepseek-ai-dsh-experimental-cua-macos/use-this-package: - en: 2d6987af8fe64389 - zh: 848d3f4c2702f16f + en: a1327058ca336056 + zh: f2f7738b95ac8fde /deepseek-ai-dsh-experimental-cua-macos/use-this-package/connect-an-external-execution-environment: - en: 79fd97d4e082861a - zh: 24552b77ad5bf236 + en: bc90a7d2a6f82036 + zh: 48af5ed1d5c2abdc /deepseek-ai-dsh-experimental-cua-macos/understand-the-implementation: en: 4e82b85a5aa72eee zh: 4e004b7a205d8688 diff --git a/packages/experimental/cua-macos/README.md b/packages/experimental/cua-macos/README.md index e96739b263..2a72bc88a3 100644 --- a/packages/experimental/cua-macos/README.md +++ b/packages/experimental/cua-macos/README.md @@ -41,7 +41,7 @@ For repository Web use, enable [Computer Use](../cua-bundle/README.md) in Plugin pnpm dsh headless --patch packages/experimental/cua-macos/examples/macos.cordis.yml --dump-config ``` -The dump command checks profile composition without a model request. For a real task, replace `--dump-config` with a prompt and configure the profile’s model credentials. The overlay mounts the computer-use registry and this provider; remove an existing competing provider before using it. The helper needs Accessibility permission for AX/input and Screen Recording permission for capture; use the Computer Use permission page to open System Settings and add the helper. Source builds use the bundle filename `DSH Computer Use Dev.app` and declare `DSH Computer Use Dev` in Info.plist; select the development entry when granting access. `getCapabilities` returns the provider’s declared capabilities with observed permissions and session lock state. Locked or unavailable desktop sessions reject UI observations and input before dispatch; the user must unlock macOS before resuming. `listApps` and `cua.getState` list running applications. `findApps(query)` searches standard macOS application directories by name, bundle ID or absolute path. `getApp` selects a unique name, bundle ID, path or running PID, obtains application consent, and launches a stopped app in the background when needed. A stopped application’s bundle identifier and executable path are read from disk before approval and rechecked before and after launch; applications without a bundle identifier use their canonical executable path for reusable consent. Selecting a stopped installation requests a new instance of that copy. The helper waits for launch to complete before reporting the running state. A post-launch verification failure expires the pending handle but can leave the process running; inspect running applications before selecting a fresh handle. The returned object reads initial AX silently without activating the app. Launch cancellation may leave the application running. Model guidance starts with background observation and input, without activation during initialization or before each action. For hidden or minimized windows, try `app.show()` or `window.show()` and read fresh state. This requests unhide and unminimize without activation, raising or switching Spaces; app-level restoration selects the focused, main or sole AX window. The result names the selected `windowId`, or null when no window was selected, and reports `foregroundChanged` by comparing the front app and top window before and after the request. Null means foreground evidence was unavailable; later focus changes remain possible. If restoration fails or the window remains unusable, explain the needed foreground step, call `app.activate()` or an observed window’s advertised `AXRaise` action, then observe again. Necessary foreground access takes priority over preserving focus. Applications may change focus after input; background delivery does not guarantee an undisturbed user session. +The dump command checks profile composition without a model request. For a real task, replace `--dump-config` with a prompt and configure the profile’s model credentials. The overlay mounts the computer-use registry and this provider; remove an existing competing provider before using it. The helper needs Accessibility permission for AX/input and Screen Recording permission for capture; use the Computer Use permission page to open System Settings and add the helper. Source builds use the bundle filename `DSH Computer Use Dev.app` and declare `DSH Computer Use Dev` in Info.plist; select the development entry when granting access. `getCapabilities` returns the provider’s declared capabilities with observed permissions and session lock state. Locked or unavailable desktop sessions reject UI observations and input before dispatch; the user must unlock macOS before resuming. `listApps` and `cua.getState` list running applications. `findApps(query)` searches standard macOS application directories by name, bundle ID or absolute path. `getApp` selects a unique name, bundle ID, path or running PID, obtains application consent, and launches a stopped app in the background when needed. A stopped application’s bundle identifier and executable path are read from disk before approval and rechecked before and after launch; applications without a bundle identifier use their canonical executable path for reusable consent. Selecting a stopped installation requests a new instance of that copy. The helper waits for launch to complete before reporting the running state. A post-launch verification failure expires the pending handle but can leave the process running; inspect running applications before selecting a fresh handle. The returned object does not read AX or activate the app. Launch cancellation may leave the application running. Model guidance starts with background observation and input, without activation during initialization or before each action. For hidden or minimized windows, try `app.show()` or `window.show()` and read fresh state. This requests unhide and unminimize without activation, raising or switching Spaces; app-level restoration selects the focused, main or sole AX window. The result names the selected `windowId`, or null when no window was selected, and reports `foregroundChanged` by comparing the front app and top window before and after the request. Null means foreground evidence was unavailable; later focus changes remain possible. If restoration fails or the window remains unusable, explain the needed foreground step, call `app.activate()` or an observed window’s advertised `AXRaise` action, then observe again. Necessary foreground access takes priority over preserving focus. Applications may change focus after input; background delivery does not guarantee an undisturbed user session. Source builds default to `sourceSigning: ad-hoc`, without creating certificates or keychains. Changed binaries can require new permission grants. To reduce repeated grants during rebuilds and across worktrees, select an existing certificate and its absolute keychain path in the DSH home’s `cordis.patch.yml`; `cua` is the entry ID loading `@deepseek-ai/dsh-experimental-cua-macos`: @@ -91,7 +91,7 @@ Raw `getState` exposes validated state fields and the complete text for the requ AX observations show menu-bar entries by default. Use `app.getAccessibilityTree({ includeMenus: true })` to expand menu contents and `app.getAccessibilityTree({ details: true })` for identifiers, global-point bounds and complete action names. These options also apply to raw `getState`. Every read replaces the application's current snapshot; use element IDs from that read. Pass `maxElements` or `maxDepth` to override either configured limit for one read. Truncation identifies the reached limit. -Values and placeholders are separate. Named, stateful or actionable groups and scroll areas retain their own rows. Repeated names, descriptions and help text on one node are shown once. Values, placeholders and URLs keep their distinct meanings even when their text matches; body text shared by distinct nodes remains intact. Writability is explicit on relevant controls, including `AXIncrementor`; `unknown` means the system did not report it. Missing optional state fields do not imply false. UTF-16 ranges outside the observed value are marked unavailable. Compact tree labels can omit AppKit target and selector details. Pass the exact name returned by `listActions(elementId)` to `performAction`. +Values and placeholders are separate. Named, stateful or actionable groups and scroll areas retain their own rows. Repeated names, descriptions and help text on one node are shown once. Values, placeholders and URLs keep their distinct meanings even when their text matches; body text shared by distinct nodes remains intact. Writability is explicit on relevant controls, including `AXIncrementor`; `unknown` means the system did not report it. Missing optional state fields do not imply false. During tree observation, an AXValue or AXSelectedTextRange writability probe returning -25200 leaves that field unknown and preserves the remaining nodes. Actual writes remain strict; permission, communication and invalid-element errors still propagate. UTF-16 ranges outside the observed value are marked unavailable. Compact tree labels can omit AppKit target and selector details. Pass the exact name returned by `listActions(elementId)` to `performAction`. Observations emit through `inspect` by default and also return their value. Use `await app.getAccessibilityTree()` to emit an observation, or set `emit: false` to retain its value for later inspection. Capturing a screenshot after an emitted tree can fail independently; retain the successful AX output. @@ -115,9 +115,9 @@ inspect(await window.getScreenshot()) An independent floating panel has its own window ID. Select it from `app.getWindows()` to capture or click the panel. -Use the `cua` API to discover and select the application (`listApps`, `getState`, `findApps`, `getApp`); use the selected app to activate it, enumerate its windows, and select one. Window selection is required for screenshots and pointer input, not only when several windows exist. `getApp` and `app.getWindow(windowId)` read their initial trees silently; choose the window by its observed title and bounds. +Use the `cua` API to discover and select the application (`listApps`, `getState`, `findApps`, `getApp`); use the selected app to activate it, enumerate its windows, and select one. Window selection is required for screenshots and pointer input, not only when several windows exist. `getApp` and `app.getWindow(windowId)` return without reading AX; choose the window from `getWindows()` by its title and bounds. -`getWindow` reads its initial tree silently. An owned window without an AX surface returns an empty snapshot and remains available for screenshots and coordinate input. Within one API instance, the facade keeps one current accessibility snapshot per application handle, shared by every app and window object for that handle, including repeated selections. Reading any window replaces that snapshot; element operations use the latest snapshot ID, so use IDs from the latest tree and do not assume an older ID still identifies the same control. A failed read publishes no partial snapshot; read again without repeating the preceding input. Window objects own screenshots and pointer input. Keyboard methods stay on `app` because input follows the app’s current focus. +`getWindow` sends no native request; screenshots and pointer input reject a closed or foreign window. Read `getAccessibilityTree()` explicitly before element operations. Screenshots do not require a successful AX read. All coordinate input, including relative coordinates, requires a successful AX observation covering the window or a screenshot; absolute pixel coordinates require a delivered screenshot. An owned window without an AX surface returns an empty snapshot when explicitly observed. Within one API instance, the facade keeps one current accessibility snapshot per application handle, shared by every app and window object for that handle, including repeated selections. Reading any window replaces that snapshot; element operations use the latest snapshot ID, so use IDs from the latest tree and do not assume an older ID still identifies the same control. A failed read publishes no partial snapshot; read again without repeating the preceding input. Window objects own screenshots and pointer input. Keyboard methods stay on `app` because input follows the app’s current focus. Ordinary `mouseMove(target)` defaults to AX hover. Without both `AXShowAlternateUI` and `AXShowDefaultUI` on an enabled target or ancestor, it returns `HOVER_UNAVAILABLE` without prompting. Only when the task requires physical movement should the main task call `win.mouseMove(target, { useSystemPointer: true })`. On approval the Host rechecks the window and point and dispatches this move. Pass `useSystemPointer: true` on every physical move. Later explicit moves reuse the decision until the session closes or DSH restarts. Omitting the option or passing `false` uses AX. Refusal sends no input. @@ -193,9 +193,9 @@ Setup and discovery: - cua.listApps(options?: { emit?: boolean }) -> { id: string, displayName: string, isRunning: true, pid: number }[] — enumerate running applications. - cua.findApps(query: string, options?: { emit?: boolean }) -> ({ id: string, displayName: string, path: string } & ({ isRunning: true, pid: number } | { isRunning: false }))[] — search installed applications, including stopped matches; only running matches carry pid. - cua.getState(options?: { emit?: boolean }) -> { apps: { id: string, displayName: string, isRunning: true, pid: number }[], browsers: [] } — summarize running applications; this is not an accessibility observation. -- cua.getApp(selector: string) -> App — select an application by name, bundle ID, absolute path, or running PID string, obtain consent, launch in the background if needed, and read initial AX state silently; it does not activate the app. +- cua.getApp(selector: string) -> App — select an application by name, bundle ID, absolute path, or running PID string, obtain consent, launch in the background if needed, and return without reading AX or activating the app. - app.getWindows(options?: { emit?: boolean }) -> { windowId: number, appName: string, title: string, bounds: { x: number, y: number, width: number, height: number }, isOnScreen: boolean, onCurrentSpace: boolean | null }[] — enumerate windows with global native bounds; null onCurrentSpace means unknown Space membership. -- app.getWindow(windowId: number) -> Window — select one window and read its initial AX state silently; do this before any window screenshot or coordinate input. +- app.getWindow(windowId: number) -> Window — select a window by an ID from getWindows without reading AX. Providers reject closed or foreign windows when operating on them. - app.activate() -> ActionResult — request explicit foreground activation; use only when foreground access is required. - app.show() -> { dispatched: boolean, windowId: number | null, foregroundChanged: boolean | null } — unhide the app and restore its selected, focused, main, or sole window without activation. - window.show() -> { dispatched: boolean, windowId: number | null, foregroundChanged: boolean | null } — unhide and restore that explicit window without activation. @@ -203,7 +203,7 @@ Setup and discovery: ## Accessibility tree Available at app or window level depending on the platform. -Element ids are opaque strings from the latest successful tree; the object facade keeps one current snapshot per app across its repeated app and window objects, a new read replaces it, and a failed read invalidates the previous one. Repeated strings can appear for different elements, so do not reuse an old id blindly, and element operations do not fall back to pointer input or typing. +Read getAccessibilityTree explicitly before using element ids. Selection and screenshots do not read or replace the current AX snapshot. Element ids are opaque strings from the latest successful tree; the object facade keeps one current snapshot per app across its repeated app and window objects, a new read replaces it, and a failed read invalidates the previous one. Repeated strings can appear for different elements, so do not reuse an old id blindly, and element operations do not fall back to pointer input or typing. - getAccessibilityTree(options?: { emit?: boolean, includeMenus?: boolean, details?: boolean, maxElements?: number, maxDepth?: number }) -> string — set the node count or depth for one read. Omitted limits use the provider settings. includeMenus expands menu contents and details adds identifiers, native bounds, and complete action names. macOS accepts maxElements up to 100000 and maxDepth up to 512. If the tree is truncated, select a narrower window or raise the reported limit on the next read. In persistent PTC, capture a large tree with const tree = await win.getAccessibilityTree({ emit: false }). Filter its lines in memory, or save it with await tools.write({ file_path: path, content: tree }) to a new workspace file. Search its contents with tools.grep({ path, pattern }) and use tools.read around a matching line. Print only the matches you need. Use element ids from the file only while that observation is current. - listActions(elementId: string) -> string[] — list the actions an element advertises. - invoke(elementId: string) -> ActionResult — invoke the element's default action; dispatched is false when it has none. @@ -212,7 +212,7 @@ Element ids are opaque strings from the latest successful tree; the object facad - selectText(elementId: string, text: string, options?: { prefix?: string, suffix?: string, selectionType?: 'text' | 'cursor_before' | 'cursor_after' }) -> ActionResult — select one literal occurrence or place the cursor beside it, using adjacent context to disambiguate. ## Simulated mouse -Use an explicit window for screenshots and pointer input. +Use an explicit window for screenshots and pointer input. Before coordinate input, including relative coordinates, obtain a successful getAccessibilityTree observation covering that window or take its screenshot. Absolute pixel coordinates require a returned screenshot. - getScreenshot() -> Image — capture the selected window as a stored image with readonly path, width and height; pass it to inspect to display it. - mouseMove(target: string | [number, number], options?: { relative?: boolean, useSystemPointer?: boolean }) -> { dispatched: boolean, foregroundChanged?: boolean | null, hoverRoute: 'accessibility' | 'system-pointer', systemPointerMoved: boolean } — request a mouse move to the target without pressing a button; inspect hoverRoute and systemPointerMoved, and expect no automatic pointer restoration. Default hover requires both AXShowAlternateUI and AXShowDefaultUI on an enabled element or ancestor. Without that route, HOVER_UNAVAILABLE is returned before sending input. Physical movement requires an unobscured foreground target. Explain any necessary activation and observe the target again. - click(target: string | [number, number], options?: { mouseButton?: 'left' | 'right' | 'middle', clickCount?: 1 | 2, modifiers?: ('command' | 'control' | 'option' | 'shift' | 'fn')[], holdMs?: number, relative?: boolean }) -> ActionResult — click an element id or point, or perform a left-button double-click; holdMs controls the press duration of each press. click is a Window method: select a window with const win = await app.getWindow(windowId), then call win.click(target). For a background control’s default action, use app.invoke(id). For a mouse click on a background app, call app.activate() first and re-read the target. Both target forms send mouse-down and mouse-up events. click(id) uses the center of the full element bounds. If that center is outside the window, take a fresh win.getScreenshot() and use click([x, y]) on a visible part of the control in screenshot pixels. A background app can ignore the events even when dispatched is true. Verify the expected effect, such as a changed selection, before typing or retrying. @@ -228,7 +228,7 @@ Available at app or window level depending on the platform. - pressKey(key: string, modifiers?: string[]) -> ActionResult — press and release one complete keystroke. For example pressKey('Tab', ['shift']) or pressKey('+'). Key and modifier names are platform-defined strings. Pass key and modifiers separately; do not pack them into a chord string. - sendKeyEvent(key: string, direction: 'down' | 'up') -> ActionResult — send 'down' to hold a key across tool calls in one run_code, for example W down, D down, D up, W up. Holding a key on a physical keyboard can produce repeated key-down events after a delay. To send one such repeat for a non-modifier key this run already holds, send 'down' again. Each call posts one event. A control can filter repeats. Use the app's response to decide whether and when to repeat or release the key. When this run_code returns, throws, or the user pauses it, macOS releases only its remaining keys at their original recipients. -Raw tools use { method, params }: include appHandle for app calls and windowId for window calls, and use snapshotId with elementId for element routes. The emit option is facade-only: emit is local output control, so raw callers must follow the raw operation schema for admitted params. getAccessibilityTree and getWindow read through raw getState; getWindows maps to listWindows; getScreenshot maps to screenshot; cua.getState maps to listApps; every other named operation uses the same method name. +Raw tools use { method, params }: include appHandle for app calls and windowId for window calls, and use snapshotId with elementId for element routes. The emit option is facade-only: emit is local output control, so raw callers must follow the raw operation schema for admitted params. getAccessibilityTree maps to getState; getWindow has no raw method; getWindows maps to listWindows; getScreenshot maps to screenshot; cua.getState maps to listApps; every other named operation uses the same method name. Only if the task cannot be completed without moving the user’s pointer, call await win.mouseMove(target, { useSystemPointer: true }). Prefer an advertised accessibility action, a click on a visible control, or browser tools when possible. The question interrupts the user. When approval is disabled an explicit move can act immediately. Pass true on every physical move, even after approval. Omitting the option or passing false uses the default hover route without prompting. Approval or refusal lasts until this session closes or DSH restarts. A refusal sends no input. If the user dismisses the request, ask again only if they request it. Wait for an outstanding request to finish. Subagents cannot request physical movement. ``` diff --git a/packages/experimental/cua-macos/README.zh.md b/packages/experimental/cua-macos/README.zh.md index 4613b4452e..96e6a04c03 100644 --- a/packages/experimental/cua-macos/README.zh.md +++ b/packages/experimental/cua-macos/README.zh.md @@ -41,7 +41,7 @@ provider 按当前界面语言生成引导和恢复文案，不支持的语言 pnpm dsh headless --patch packages/experimental/cua-macos/examples/macos.cordis.yml --dump-config ``` -配置输出命令不发起模型请求。执行真实任务时，将 `--dump-config` 替换为任务提示，并配置该 profile 的模型凭据。配置挂载 computer-use 注册表和本 provider；使用前需移除已有的竞争 provider。helper 需要辅助功能权限来读取 AX 和输入，需要屏幕录制权限来截图；通过 Computer Use 权限页打开系统设置并添加 helper。源码构建的应用包名为 `DSH Computer Use Dev.app`，Info.plist 声明名称 `DSH Computer Use Dev`；授权时选择开发版条目。`getCapabilities` 返回提供方声明的平台能力，以及实际查询的权限和锁屏状态。桌面会话锁定或不可用时，界面观察与输入会在分派前拒绝，需由用户解锁 macOS 后继续。`listApps` 和 `cua.getState` 列出运行中的应用。`findApps(query)` 按名称、bundle ID 或绝对路径搜索 macOS 标准应用目录。`getApp` 按唯一名称、bundle ID、路径或运行中的 PID 选择应用，取得应用许可后，在需要时后台启动尚未运行的应用。授权前从磁盘读取应用的 Bundle ID 和可执行路径，在启动前后重新读取并核对；没有 Bundle ID 的应用使用规范化的可执行路径复用授权。选择尚未运行的安装副本时，会请求为该副本创建新实例。helper 等待应用完成启动后返回运行状态。启动后验证失败会使待启动句柄失效，但进程可能仍在运行；应先查看运行中的应用，再取得新句柄。返回的对象静默读取初始 AX，不自动激活应用。启动被取消时，应用可能已经运行。模型指引要求先在后台观察和输入，不在初始化或每次动作前激活应用。窗口隐藏或最小化时，先尝试 `app.show()` 或 `window.show()`，再重新观察。它请求取消隐藏和恢复最小化，不主动激活、抬起窗口或切换桌面空间；应用级恢复选择焦点窗口、主窗口或唯一 AX 窗口。结果通过 `windowId` 指明选中的窗口，未选中时为 null；`foregroundChanged` 比较操作前后的前台应用和最上层窗口，无法取得完整观察时为 null，返回后仍可能发生焦点变化。恢复失败或窗口仍不可用时，说明需要前台的步骤，再调用 `app.activate()` 或窗口声明的 `AXRaise` 动作并重新观察。必须使用前台时，以完成任务为先。应用收到输入后仍可能改变焦点，后台投递不保证用户操作始终不受干扰。 +配置输出命令不发起模型请求。执行真实任务时，将 `--dump-config` 替换为任务提示，并配置该 profile 的模型凭据。配置挂载 computer-use 注册表和本 provider；使用前需移除已有的竞争 provider。helper 需要辅助功能权限来读取 AX 和输入，需要屏幕录制权限来截图；通过 Computer Use 权限页打开系统设置并添加 helper。源码构建的应用包名为 `DSH Computer Use Dev.app`，Info.plist 声明名称 `DSH Computer Use Dev`；授权时选择开发版条目。`getCapabilities` 返回提供方声明的平台能力，以及实际查询的权限和锁屏状态。桌面会话锁定或不可用时，界面观察与输入会在分派前拒绝，需由用户解锁 macOS 后继续。`listApps` 和 `cua.getState` 列出运行中的应用。`findApps(query)` 按名称、bundle ID 或绝对路径搜索 macOS 标准应用目录。`getApp` 按唯一名称、bundle ID、路径或运行中的 PID 选择应用，取得应用许可后，在需要时后台启动尚未运行的应用。授权前从磁盘读取应用的 Bundle ID 和可执行路径，在启动前后重新读取并核对；没有 Bundle ID 的应用使用规范化的可执行路径复用授权。选择尚未运行的安装副本时，会请求为该副本创建新实例。helper 等待应用完成启动后返回运行状态。启动后验证失败会使待启动句柄失效，但进程可能仍在运行；应先查看运行中的应用，再取得新句柄。返回对象时不读取 AX，也不激活应用。启动被取消时，应用可能已经运行。模型指引要求先在后台观察和输入，不在初始化或每次动作前激活应用。窗口隐藏或最小化时，先尝试 `app.show()` 或 `window.show()`，再重新观察。它请求取消隐藏和恢复最小化，不主动激活、抬起窗口或切换桌面空间；应用级恢复选择焦点窗口、主窗口或唯一 AX 窗口。结果通过 `windowId` 指明选中的窗口，未选中时为 null；`foregroundChanged` 比较操作前后的前台应用和最上层窗口，无法取得完整观察时为 null，返回后仍可能发生焦点变化。恢复失败或窗口仍不可用时，说明需要前台的步骤，再调用 `app.activate()` 或窗口声明的 `AXRaise` 动作并重新观察。必须使用前台时，以完成任务为先。应用收到输入后仍可能改变焦点，后台投递不保证用户操作始终不受干扰。 源码构建默认使用 `sourceSigning: ad-hoc` 临时签名，不创建证书或钥匙串。二进制变化后可能需要重新授权。需要减少开发重建和跨 worktree 的授权次数时，在 DSH home 的 `cordis.patch.yml` 中选择已有证书及其钥匙串绝对路径；`cua` 是加载 `@deepseek-ai/dsh-experimental-cua-macos` 的节点 ID： @@ -91,7 +91,7 @@ CUA 工具在读取 AX、窗口标题、截图、激活和输入前检查应用 AX 观察默认只展示菜单栏入口。使用 `app.getAccessibilityTree({ includeMenus: true })` 展开菜单内容，使用 `app.getAccessibilityTree({ details: true })` 读取标识符、全局 point 坐标和完整动作名。原始 `getState` 也接受这两个选项。每次读取都会替换应用当前的快照；应使用该次读取返回的元素 ID。传入 `maxElements` 或 `maxDepth` 可调整本次读取的上限。树被截断时会提示触发的上限。 -实际值与占位文字分别展示。有名称、状态或可用动作的分组和滚动区域保留独立行。同一节点中重复的名称、描述和帮助只显示一次。实际值、占位文字和 URL 即使文字相同也保留各自含义，不删除不同节点共有的正文。相关控件明确展示可写性，包括 `AXIncrementor`；`unknown` 表示系统未报告。缺少可选状态字段不代表 false。超出当前值长度的 UTF-16 选区标为不可用。树中的动作简称可能省略 AppKit 的 target 和 selector 信息。调用 `performAction` 时使用 `listActions(elementId)` 返回的完整名称。 +实际值与占位文字分别展示。有名称、状态或可用动作的分组和滚动区域保留独立行。同一节点中重复的名称、描述和帮助只显示一次。实际值、占位文字和 URL 即使文字相同也保留各自含义，不删除不同节点共有的正文。相关控件明确展示可写性，包括 `AXIncrementor`；`unknown` 表示系统未报告。缺少可选状态字段不代表 false。读树时，AXValue 或 AXSelectedTextRange 的可写性探测返回 -25200，会将该字段保留为未知并继续读取其他节点。实际写入仍严格校验；权限、通信和失效元素错误仍向上传播。超出当前值长度的 UTF-16 选区标为不可用。树中的动作简称可能省略 AppKit 的 target 和 selector 信息。调用 `performAction` 时使用 `listActions(elementId)` 返回的完整名称。 观察方法默认通过 `inspect` 输出，同时返回结果。使用 `await app.getAccessibilityTree()` 输出观察，或设置 `emit: false` 保留结果供之后 inspect。AX 输出后的截图可能单独失败，已成功取得的 AX 输出仍保留。 @@ -115,9 +115,9 @@ inspect(await window.getScreenshot()) 独立浮层有自己的窗口 ID。需要截图或点击浮层时，从 `app.getWindows()` 中选中它。 -使用 `cua` API 发现并选择应用（`listApps`、`getState`、`findApps`、`getApp`）；使用选定的应用来激活它、枚举其窗口并选择窗口。截图和指针输入都要求选择窗口，不只在存在多个窗口时才需要。`getApp` 和 `app.getWindow(windowId)` 静默读取初始树；根据观察到的标题与位置选择窗口。 +使用 `cua` API 发现并选择应用（`listApps`、`getState`、`findApps`、`getApp`）；使用选定的应用来激活它、枚举其窗口并选择窗口。截图和指针输入都要求选择窗口，不只在存在多个窗口时才需要。`getApp` 和 `app.getWindow(windowId)` 返回时不读取 AX；根据 `getWindows()` 中的标题与位置选择窗口。 -`getWindow` 静默读取初始树。已确认归属但没有 AX 界面的窗口返回空快照，仍可截图和发送坐标输入。在单个 API 实例内，门面按应用句柄保存一份当前辅助功能快照，该句柄的所有应用对象和窗口对象共享它，包括重复选择。读取任一窗口都会替换该快照；元素操作使用最新快照 ID，因此应使用最新树中的元素 ID，不应假定旧 ID 仍指向同一控件。读取失败不会发布部分快照；可以重新读取，但不要重复此前的输入。窗口对象负责截图和指针输入。键盘方法保留在 `app` 上，因为输入跟随应用的当前焦点。 +`getWindow` 不发送原生请求；截图和指针输入会拒绝已关闭或属于其他应用的窗口。元素操作前须显式读取 `getAccessibilityTree()`。截图不要求 AX 读取成功。所有坐标输入（包括相对坐标）都要求先成功读取覆盖该窗口的 AX 或取得截图；绝对像素坐标要求已交付的截图。已确认归属但没有 AX 界面的窗口在显式观察时返回空快照。在单个 API 实例内，门面按应用句柄保存一份当前辅助功能快照，该句柄的所有应用对象和窗口对象共享它，包括重复选择。读取任一窗口都会替换该快照；元素操作使用最新快照 ID，因此应使用最新树中的元素 ID，不应假定旧 ID 仍指向同一控件。读取失败不会发布部分快照；可以重新读取，但不要重复此前的输入。窗口对象负责截图和指针输入。键盘方法保留在 `app` 上，因为输入跟随应用的当前焦点。 普通 `mouseMove(target)` 默认通过辅助功能悬停。如果目标或其祖先没有同时声明 `AXShowAlternateUI` 和 `AXShowDefaultUI`，则返回 `HOVER_UNAVAILABLE`，不会弹窗。只有任务确实需要移动用户的鼠标时，主任务才调用 `win.mouseMove(target, { useSystemPointer: true })`。用户允许后，Host 重新检查窗口和目标位置，再执行本次移动。每次移动系统指针都要显式传 `useSystemPointer: true`。省略参数或传 `false` 时使用辅助功能悬停。拒绝时不派发输入。会话记住此次选择，直到关闭会话或重启 DSH。 @@ -193,9 +193,9 @@ Setup and discovery: - cua.listApps(options?: { emit?: boolean }) -> { id: string, displayName: string, isRunning: true, pid: number }[] — enumerate running applications. - cua.findApps(query: string, options?: { emit?: boolean }) -> ({ id: string, displayName: string, path: string } & ({ isRunning: true, pid: number } | { isRunning: false }))[] — search installed applications, including stopped matches; only running matches carry pid. - cua.getState(options?: { emit?: boolean }) -> { apps: { id: string, displayName: string, isRunning: true, pid: number }[], browsers: [] } — summarize running applications; this is not an accessibility observation. -- cua.getApp(selector: string) -> App — select an application by name, bundle ID, absolute path, or running PID string, obtain consent, launch in the background if needed, and read initial AX state silently; it does not activate the app. +- cua.getApp(selector: string) -> App — select an application by name, bundle ID, absolute path, or running PID string, obtain consent, launch in the background if needed, and return without reading AX or activating the app. - app.getWindows(options?: { emit?: boolean }) -> { windowId: number, appName: string, title: string, bounds: { x: number, y: number, width: number, height: number }, isOnScreen: boolean, onCurrentSpace: boolean | null }[] — enumerate windows with global native bounds; null onCurrentSpace means unknown Space membership. -- app.getWindow(windowId: number) -> Window — select one window and read its initial AX state silently; do this before any window screenshot or coordinate input. +- app.getWindow(windowId: number) -> Window — select a window by an ID from getWindows without reading AX. Providers reject closed or foreign windows when operating on them. - app.activate() -> ActionResult — request explicit foreground activation; use only when foreground access is required. - app.show() -> { dispatched: boolean, windowId: number | null, foregroundChanged: boolean | null } — unhide the app and restore its selected, focused, main, or sole window without activation. - window.show() -> { dispatched: boolean, windowId: number | null, foregroundChanged: boolean | null } — unhide and restore that explicit window without activation. @@ -203,7 +203,7 @@ Setup and discovery: ## Accessibility tree Available at app or window level depending on the platform. -Element ids are opaque strings from the latest successful tree; the object facade keeps one current snapshot per app across its repeated app and window objects, a new read replaces it, and a failed read invalidates the previous one. Repeated strings can appear for different elements, so do not reuse an old id blindly, and element operations do not fall back to pointer input or typing. +Read getAccessibilityTree explicitly before using element ids. Selection and screenshots do not read or replace the current AX snapshot. Element ids are opaque strings from the latest successful tree; the object facade keeps one current snapshot per app across its repeated app and window objects, a new read replaces it, and a failed read invalidates the previous one. Repeated strings can appear for different elements, so do not reuse an old id blindly, and element operations do not fall back to pointer input or typing. - getAccessibilityTree(options?: { emit?: boolean, includeMenus?: boolean, details?: boolean, maxElements?: number, maxDepth?: number }) -> string — set the node count or depth for one read. Omitted limits use the provider settings. includeMenus expands menu contents and details adds identifiers, native bounds, and complete action names. macOS accepts maxElements up to 100000 and maxDepth up to 512. If the tree is truncated, select a narrower window or raise the reported limit on the next read. In persistent PTC, capture a large tree with const tree = await win.getAccessibilityTree({ emit: false }). Filter its lines in memory, or save it with await tools.write({ file_path: path, content: tree }) to a new workspace file. Search its contents with tools.grep({ path, pattern }) and use tools.read around a matching line. Print only the matches you need. Use element ids from the file only while that observation is current. - listActions(elementId: string) -> string[] — list the actions an element advertises. - invoke(elementId: string) -> ActionResult — invoke the element's default action; dispatched is false when it has none. @@ -212,7 +212,7 @@ Element ids are opaque strings from the latest successful tree; the object facad - selectText(elementId: string, text: string, options?: { prefix?: string, suffix?: string, selectionType?: 'text' | 'cursor_before' | 'cursor_after' }) -> ActionResult — select one literal occurrence or place the cursor beside it, using adjacent context to disambiguate. ## Simulated mouse -Use an explicit window for screenshots and pointer input. +Use an explicit window for screenshots and pointer input. Before coordinate input, including relative coordinates, obtain a successful getAccessibilityTree observation covering that window or take its screenshot. Absolute pixel coordinates require a returned screenshot. - getScreenshot() -> Image — capture the selected window as a stored image with readonly path, width and height; pass it to inspect to display it. - mouseMove(target: string | [number, number], options?: { relative?: boolean, useSystemPointer?: boolean }) -> { dispatched: boolean, foregroundChanged?: boolean | null, hoverRoute: 'accessibility' | 'system-pointer', systemPointerMoved: boolean } — request a mouse move to the target without pressing a button; inspect hoverRoute and systemPointerMoved, and expect no automatic pointer restoration. Default hover requires both AXShowAlternateUI and AXShowDefaultUI on an enabled element or ancestor. Without that route, HOVER_UNAVAILABLE is returned before sending input. Physical movement requires an unobscured foreground target. Explain any necessary activation and observe the target again. - click(target: string | [number, number], options?: { mouseButton?: 'left' | 'right' | 'middle', clickCount?: 1 | 2, modifiers?: ('command' | 'control' | 'option' | 'shift' | 'fn')[], holdMs?: number, relative?: boolean }) -> ActionResult — click an element id or point, or perform a left-button double-click; holdMs controls the press duration of each press. click is a Window method: select a window with const win = await app.getWindow(windowId), then call win.click(target). For a background control’s default action, use app.invoke(id). For a mouse click on a background app, call app.activate() first and re-read the target. Both target forms send mouse-down and mouse-up events. click(id) uses the center of the full element bounds. If that center is outside the window, take a fresh win.getScreenshot() and use click([x, y]) on a visible part of the control in screenshot pixels. A background app can ignore the events even when dispatched is true. Verify the expected effect, such as a changed selection, before typing or retrying. @@ -228,7 +228,7 @@ Available at app or window level depending on the platform. - pressKey(key: string, modifiers?: string[]) -> ActionResult — press and release one complete keystroke. For example pressKey('Tab', ['shift']) or pressKey('+'). Key and modifier names are platform-defined strings. Pass key and modifiers separately; do not pack them into a chord string. - sendKeyEvent(key: string, direction: 'down' | 'up') -> ActionResult — send 'down' to hold a key across tool calls in one run_code, for example W down, D down, D up, W up. Holding a key on a physical keyboard can produce repeated key-down events after a delay. To send one such repeat for a non-modifier key this run already holds, send 'down' again. Each call posts one event. A control can filter repeats. Use the app's response to decide whether and when to repeat or release the key. When this run_code returns, throws, or the user pauses it, macOS releases only its remaining keys at their original recipients. -Raw tools use { method, params }: include appHandle for app calls and windowId for window calls, and use snapshotId with elementId for element routes. The emit option is facade-only: emit is local output control, so raw callers must follow the raw operation schema for admitted params. getAccessibilityTree and getWindow read through raw getState; getWindows maps to listWindows; getScreenshot maps to screenshot; cua.getState maps to listApps; every other named operation uses the same method name. +Raw tools use { method, params }: include appHandle for app calls and windowId for window calls, and use snapshotId with elementId for element routes. The emit option is facade-only: emit is local output control, so raw callers must follow the raw operation schema for admitted params. getAccessibilityTree maps to getState; getWindow has no raw method; getWindows maps to listWindows; getScreenshot maps to screenshot; cua.getState maps to listApps; every other named operation uses the same method name. Only if the task cannot be completed without moving the user’s pointer, call await win.mouseMove(target, { useSystemPointer: true }). Prefer an advertised accessibility action, a click on a visible control, or browser tools when possible. The question interrupts the user. When approval is disabled an explicit move can act immediately. Pass true on every physical move, even after approval. Omitting the option or passing false uses the default hover route without prompting. Approval or refusal lasts until this session closes or DSH restarts. A refusal sends no input. If the user dismisses the request, ask again only if they request it. Wait for an outstanding request to finish. Subagents cannot request physical movement. ``` diff --git a/packages/experimental/cua-macos/native/Sources/CuaNative/AXAccess.swift b/packages/experimental/cua-macos/native/Sources/CuaNative/AXAccess.swift index 46a9c1e3b4..9c166c04c6 100644 --- a/packages/experimental/cua-macos/native/Sources/CuaNative/AXAccess.swift +++ b/packages/experimental/cua-macos/native/Sources/CuaNative/AXAccess.swift @@ -208,6 +208,12 @@ final class AXAccess { guard error == .success else { throw failure(error, "read writability of \(name)") } return value } + /// Observation keeps unsupported writability unknown; mutation callers use strict writable checks. + static func observationWritability(role: String, probe: (String, Bool) throws -> Bool?) rethrows -> (value: Bool?, selection: Bool?) { + let value = AXTextNode.valueRoles.contains(role) ? try probe("AXValue", true) : nil + let selection = AXTextNode.textRoles.contains(role) ? try probe("AXSelectedTextRange", true) : nil + return (value, selection) + } func requireEnabled(_ element: AXUIElement) throws { if try boolean(element, "AXEnabled") == false { throw NativeFailure(code: "NATIVE_ERROR", message: "Accessibility element is disabled") } } @@ -590,10 +596,11 @@ final class AXAccess { let placeholder = try stringValue(observed("AXPlaceholderValue"), attribute: "AXPlaceholderValue") let description = try stringValue(observed("AXDescription"), attribute: "AXDescription") let actionNames = try self.actions(element) - let valueWritable = AXTextNode.valueRoles.contains(role) ? try writable(element, "AXValue") : nil + let writability = try Self.observationWritability(role: role) { try writable(element, $0, forDiscovery: $1) } + let valueWritable = writability.value let enabled = try booleanValue(observed("AXEnabled"), attribute: "AXEnabled") let selected = try booleanValue(observed("AXSelected"), attribute: "AXSelected") - let selectionWritable = AXTextNode.textRoles.contains(role) ? try writable(element, "AXSelectedTextRange") : nil + let selectionWritable = writability.selection let selectedRange = AXTextNode.textRoles.contains(role) ? try textRange(observed("AXSelectedTextRange")) : nil let selectedText = AXTextNode.textRoles.contains(role) ? try stringValue(observed("AXSelectedText"), attribute: "AXSelectedText") : nil let identifier = try stringValue(observed("AXIdentifier"), attribute: "AXIdentifier") diff --git a/packages/experimental/cua-macos/native/Sources/CuaNative/AXWorker.swift b/packages/experimental/cua-macos/native/Sources/CuaNative/AXWorker.swift index 2720ca0643..6aea749256 100644 --- a/packages/experimental/cua-macos/native/Sources/CuaNative/AXWorker.swift +++ b/packages/experimental/cua-macos/native/Sources/CuaNative/AXWorker.swift @@ -157,6 +157,9 @@ actor AXWorker { return result } private func element(_ handle: String, _ params: RequestParameters) throws -> AXUIElement { + if let windowID = try params.optionalWindowID() { + _ = try NativePlatform.ownedWindow(validateIdentity(handle).pid, windowID) + } let value = try snapshot(handle, params.string("snapshotId")) let index = try params.elementIdIndex() guard let element = value.elements[index] else { throw RequestParameters.invalid("elementId is absent from the selected snapshot") } diff --git a/packages/experimental/cua-macos/native/tests/e2e.py b/packages/experimental/cua-macos/native/tests/e2e.py index 83782db1a2..68dc4f1344 100644 --- a/packages/experimental/cua-macos/native/tests/e2e.py +++ b/packages/experimental/cua-macos/native/tests/e2e.py @@ -109,6 +109,7 @@ def main(): parser = argparse.ArgumentParser(description=__doc__) parser.add_argument("--helper", type=Path, help="Existing native helper executable for this run") parser.add_argument("--artifacts", type=Path, help="Save fixture screenshots and private JSON protocol transcript") + parser.add_argument("--selection-only", action="store_true", help="Exercise observation and selected-window admission without pointer input") parser.add_argument("--panels-only", action="store_true", help="Exercise system file panels without unrelated pointer gestures") parser.add_argument("--hover-only", action="store_true", help="Exercise native hover without keyboard or file-panel scenarios") parser.add_argument("--alternate-hover", action="store_true", help="Use AX alternate UI and verify the system pointer stays unchanged") @@ -129,6 +130,7 @@ def main(): fixture_args += ["35", "--alternate-hover"] fixture = subprocess.Popen(fixture_args, stdout=subprocess.PIPE, text=True) helper = None + foreign_fixture = None try: assert line(fixture).strip() == "READY" helper = NativeHelper(transcript, args.helper) @@ -167,6 +169,37 @@ def main(): time.sleep(0.025) primary_params = {**owner, "windowId": primary["windowId"], "maxElements": 300, "maxDepth": 20} secondary_params = {**owner, "windowId": secondary["windowId"], "maxElements": 300, "maxDepth": 20} + if args.selection_only: + call("click", {**owner, "windowId": primary["windowId"], "x": 20, "y": 20}, "STALE_SNAPSHOT") + observed = call("getState", {**owner, "maxElements": 300, "maxDepth": 20}) + button = next(e for e in observed["elements"] if e["role"] == "AXButton" and e["title"] == "Increment primary") + field = next(e for e in observed["elements"] if e["role"] == "AXTextField") + target = {**owner, "snapshotId": observed["snapshotId"], "elementId": button["elementId"]} + call("invoke", {**target, "windowId": primary["windowId"]}) + wait_state(lambda state: state["count"] == 1, "application snapshot did not invoke on selected window") + # A second fixture window is closed while the application snapshot stays live. + close_button = next(e for e in observed["elements"] if e["role"] == "AXButton" and e["title"] == "Close secondary") + call("invoke", {**target, "elementId": close_button["elementId"]}) + deadline = time.monotonic() + 5 + while any(w["windowId"] == secondary["windowId"] for w in call("listWindows", owner)): + if time.monotonic() >= deadline: + raise AssertionError("secondary window did not close") + time.sleep(0.025) + foreign_fixture = subprocess.Popen([str(fixture_binary), str(Path(directory) / "foreign.json")], stdout=subprocess.PIPE, text=True) + assert line(foreign_fixture).strip() == "READY" + foreign_app = call("getApp", {"selector": str(foreign_fixture.pid)}) + foreign_windows = call("listWindows", {"appHandle": foreign_app["appHandle"]}) + assert foreign_windows, "Foreign fixture did not publish its windows" + invalid_ids = [secondary["windowId"], foreign_windows[0]["windowId"]] + for window_id in invalid_ids: + for method, extra in [("invoke", {}), ("listActions", {}), ("performAction", {"action": "AXPress"}), + ("setValue", {"elementId": field["elementId"], "value": "must not be written"}), + ("selectText", {"elementId": field["elementId"], "text": "missing"})]: + call(method, {**target, "windowId": window_id, **extra}, "WINDOW_UNAVAILABLE") + assert json.loads(state_path.read_text())["count"] == 1 + helper.eof() + print("PASS: coordinate observation, application snapshot, and selected-window rejection") + return limited = call("getState", {**primary_params, "maxElements": 1}) assert limited["truncated"] and "maxElements=1" in limited["tree"], limited first = call("getState", {**primary_params, "maxElements": 100000, "maxDepth": 512}) @@ -368,7 +401,7 @@ def main(): if state_path.exists(): (artifacts / "fixture-state.json").write_bytes(state_path.read_bytes()) - run_cleanup(save_artifacts, *([helper.stop] if helper is not None else []), lambda: stop(fixture)) + run_cleanup(save_artifacts, *([helper.stop] if helper is not None else []), lambda: stop(fixture), *([lambda: stop(foreign_fixture)] if foreign_fixture is not None else [])) if __name__ == "__main__": diff --git a/packages/experimental/cua-macos/native/tests/swift/AXFailureTests.swift b/packages/experimental/cua-macos/native/tests/swift/AXFailureTests.swift index 2c25c943e4..5ededcff6e 100644 --- a/packages/experimental/cua-macos/native/tests/swift/AXFailureTests.swift +++ b/packages/experimental/cua-macos/native/tests/swift/AXFailureTests.swift @@ -183,3 +183,48 @@ func consumedContainerMetadataPreservesStaleAndCommunicationErrors(role: String) try access.nodeObservation(attributes: ["AXRole": "AXGroup" as CFString, "AXChildren": try wrappedAXError(.illegalArgument)]) { _ in nil }.attribute("AXChildren") } } + +@Test(arguments: ["AXValue", "AXSelectedTextRange"]) +func observationRetainsUnknownWritabilityAndFollowingNodes(failedAttribute: String) throws { + let access = AXAccess(timeout: 1, cancellation: CancellationLatch()) + var lines: [String] = [] + for title in ["Affected control", "Following control"] { + let writable = try AXAccess.observationWritability(role: "AXTextField") { name, discovery in + try access.attributeWritability(true, error: title == "Affected control" && name == failedAttribute ? .failure : .success, + name: name, forDiscovery: discovery) + } + let node = AXTextNode(index: nil, role: "AXTextField", title: title, value: "Text", description: nil, + actions: [], enabled: true, valueWritable: writable.value, selectionWritable: writable.selection, + selected: nil, selectedRange: nil, selectedText: nil) + lines.append(AXTextFormatter.line(node, level: 0)) + } + let unknown = failedAttribute == "AXValue" ? "valueWritable" : "selectionWritable" + #expect(lines[0].contains("\(unknown)=unknown")) + #expect(!lines[0].contains("\(unknown)=true")) + #expect(lines[0].contains("Affected control")) + #expect(lines[1].contains("Following control")) + #expect(lines[1].contains("valueWritable=true selectionWritable=true")) + do { + _ = try access.attributeWritability(true, error: .failure, name: failedAttribute, forDiscovery: false) + Issue.record("Mutation admitted unknown writability") + } catch let error as NativeFailure { + #expect(error.accessibilityError == -25200) + #expect(error.code == "NATIVE_ERROR") + } +} + +@Test(arguments: ["AXValue", "AXSelectedTextRange"]) +func observationWritabilityPreservesPermissionCommunicationAndInvalidElementErrors(attribute: String) { + let access = AXAccess(timeout: 1, cancellation: CancellationLatch()) + for code in [AXError.apiDisabled, .cannotComplete, .invalidUIElement, .illegalArgument] { + do { + _ = try AXAccess.observationWritability(role: "AXTextField") { name, discovery in + try access.attributeWritability(true, error: name == attribute ? code : .success, name: name, forDiscovery: discovery) + } + Issue.record("Observation hid an actionable AX error") + } catch let error as NativeFailure { + #expect(error.accessibilityError == code.rawValue) + #expect(error.code == (code == .apiDisabled ? "PERMISSION_DENIED" : "NATIVE_ERROR")) + } catch { Issue.record("Unexpected error: \(error)") } + } +} diff --git a/packages/experimental/cua-macos/tests/loader-composition.spec.ts b/packages/experimental/cua-macos/tests/loader-composition.spec.ts index 0a64537f0d..71a1da141a 100644 --- a/packages/experimental/cua-macos/tests/loader-composition.spec.ts +++ b/packages/experimental/cua-macos/tests/loader-composition.spec.ts @@ -195,7 +195,7 @@ it.skipIf(process.platform === 'win32')('shares one application between retained const screenshot = await peerWindow.getScreenshot() expect(screenshot).toMatchObject({ width: 1, height: 1 }) expect((await readFile(screenshot.path)).toString('base64')).toBe(png) - await peerApp.getWindow(8) + await (await peerApp.getWindow(8)).getAccessibilityTree({ emit: false }) const recovery = 'Snapshot belongs to window 8; read getAccessibilityTree on window 7' await expect(peerWindow.click(element('0'))).rejects.toThrow(recovery) await expect(peerWindow.mouseMove(element('0'))).rejects.toThrow(recovery) diff --git a/packages/experimental/cua-macos/tests/persistent-bridge.built.e2e.ts b/packages/experimental/cua-macos/tests/persistent-bridge.built.e2e.ts index c3d3e6b275..c34fdb3666 100644 --- a/packages/experimental/cua-macos/tests/persistent-bridge.built.e2e.ts +++ b/packages/experimental/cua-macos/tests/persistent-bridge.built.e2e.ts @@ -85,6 +85,7 @@ it.skipIf(process.platform === 'win32' || !existsSync(bridge) || !existsSync(boo const first = await run(`${initialization} var app = await cua.getApp("test.fixture"); var win = await app.getWindow(7); + await win.getAccessibilityTree({ emit: false }); await win.click("0"); var shot = await win.getScreenshot(); var { readFile } = await import("node:fs/promises"); diff --git a/packages/experimental/cua/README.i18n.yaml b/packages/experimental/cua/README.i18n.yaml index 9fd0241285..f4bed0ddf5 100644 --- a/packages/experimental/cua/README.i18n.yaml +++ b/packages/experimental/cua/README.i18n.yaml @@ -15,14 +15,14 @@ en: 5db4d4fea25493b9 zh: 65ab64fc24eb0d09 /deepseek-ai-dsh-cua/use-this-package: - en: 0d06ebeb481da5ad - zh: cc0c797377544d22 + en: bc2b9775affe2c72 + zh: 26b27aeb53177ab9 /deepseek-ai-dsh-cua/use-this-package/permission-controls: en: d92d5948a7941e8d zh: d43491ae6521b87e /deepseek-ai-dsh-cua/understand-the-implementation: - en: f04dde99ab5ee0a3 - zh: 1ea8bf39528666dd + en: a106924b505ff8b0 + zh: ad4fc148ef65e0e7 /deepseek-ai-dsh-cua/further-exploration: en: c174fb5a00ea3681 zh: 039340042676881d diff --git a/packages/experimental/cua/README.md b/packages/experimental/cua/README.md index b1501dc7ae..d6aedee8f6 100644 --- a/packages/experimental/cua/README.md +++ b/packages/experimental/cua/README.md @@ -31,7 +31,7 @@ Application consent supplies English, Chinese, and Japanese display text from `s Mount the [macOS provider](../cua-macos/README.md) to enable native operations on macOS. External PTC hosts import `connectCua` from `@deepseek-ai/dsh-cua/bridge` and supply the current `tools.cua` binding. Source compositions and native preparation belong to the provider. -Use `cua.getApp(selector)` to select an application; it obtains consent, optionally launches a stopped application in the background, and reads its initial accessibility tree silently when the provider supports application-level accessibility. `cua.listApps` and `cua.getState` list running applications; `cua.findApps(query)` searches installed applications, including stopped ones. `app.activate()`, `app.getWindows()`, and `app.getWindow(windowId)` operate on the selected application; `getWindow` returns the object for screenshots and pointer input. Select an application and one of its windows before any capture or pointer input. Call `app.getAccessibilityTree()` to emit the tree. App and window reads accept `maxElements` and `maxDepth` independently. Omitted limits use the provider configuration. Within one API instance, the facade keeps one current accessibility snapshot per application handle across app and window objects, including repeated selections; each new read invalidates the previous snapshot. Element operations use the latest snapshot ID, so callers use IDs from the latest tree and do not assume an older ID still identifies the same control. Element IDs are opaque strings. +Use `cua.getApp(selector)` to select an application; it obtains consent and optionally launches a stopped application in the background, without reading its accessibility tree. `cua.listApps` and `cua.getState` list running applications; `cua.findApps(query)` searches installed applications, including stopped ones. `app.activate()`, `app.getWindows()`, and `app.getWindow(windowId)` operate on the selected application; `getWindow` returns the object without a native request; window operations reject closed or foreign windows. Select an application and one of its windows before any capture or pointer input. Read `app.getAccessibilityTree()` or `window.getAccessibilityTree()` explicitly before element operations. Screenshots remain available when AX reading fails. App and window reads accept `maxElements` and `maxDepth` independently. Omitted limits use the provider configuration. Within one API instance, the facade keeps one current accessibility snapshot per application handle across app and window objects, including repeated selections; each new read invalidates the previous snapshot. Element operations use the latest snapshot ID, so callers use IDs from the latest tree and do not assume an older ID still identifies the same control. Element IDs are opaque strings. Keyboard methods are available in the scopes a provider declares; the provider owns how input is delivered within each scope. A lost connection invalidates its handles and coordinates. Failed cleanup is reported before that connection is retired; select the application again and inspect fresh state, because earlier input may have partially executed. Screenshot image errors retain the failing stage and attachment error code. Invalid request arguments reject before dispatch with a readable method, parameter and requirement. @@ -41,7 +41,7 @@ Observation methods emit and return their result. Connect PTC text output with ` Application consent precedes launch, observation, capture, and input. Session access is stored by conversation and application identity and survives DSH restart and Computer Use disablement. Other conversations require their own consent. Saved grants require a platform-provided reusable application identity. Refusal and revocation close affected Agent connections, so their old objects require fresh selection. No input is replayed after cancellation, timeout, or reconnect. -Actions return a `dispatched` boolean; `mouseMove` also reports `hoverRoute` and `systemPointerMoved`, and `show()` adds `windowId` and `foregroundChanged`. Read fresh state to verify the result. Screenshots and pointer actions are available only on explicit window objects. Coordinate actions default to pixels in the latest returned screenshot; `relative: true` accepts integers from 0 to 999 after a successful window observation. Values are divided by 1000 and scaled to the window dimensions, then clamped inside the window. The Host resolves attachment normalization and the selected model route’s image projection before returning the screenshot, then converts input to the provider’s native window coordinates. Resizing invalidates the screenshot mapping; `SCREENSHOT_REQUIRED` requires a new screenshot for absolute coordinates. Resizing rejects old coordinates with `WINDOW_CHANGED`; movement alone preserves window-local coordinates. +Actions return a `dispatched` boolean; `mouseMove` also reports `hoverRoute` and `systemPointerMoved`, and `show()` adds `windowId` and `foregroundChanged`. Read fresh state to verify the result. Screenshots and pointer actions are available only on explicit window objects. Coordinate actions default to pixels in the latest returned screenshot; `relative: true` accepts integers from 0 to 999 after a successful AX observation covering the window or a screenshot. Selecting a window alone does not establish this observation. Values are divided by 1000 and scaled to the window dimensions, then clamped inside the window. The Host resolves attachment normalization and the selected model route’s image projection before returning the screenshot, then converts input to the provider’s native window coordinates. Resizing invalidates the screenshot mapping; `SCREENSHOT_REQUIRED` requires a new screenshot for absolute coordinates. Resizing rejects old coordinates with `WINDOW_CHANGED`; movement alone preserves window-local coordinates. `click(target, { holdMs: 800 })` holds the mouse button for 800 milliseconds before release. `holdMs` applies only to `click`; it accepts nonnegative integers and applies to each press of a double-click; zero releases without an additional hold. The provider config `maxClickHoldMs` limits explicit holds and defaults to 5000 ms. Explicit holds use pointer input; omission preserves ordinary click behavior. The operation deadline includes the hold, and cancellation or timeout releases the button. Received input cannot be undone; forced helper termination cannot guarantee release. @@ -89,7 +89,7 @@ The provider calls [installCua](src/index.ts) with its prepared helper, connecti [The operation schema](src/protocol.ts) is the authority for model parameters and Host-supplied request fields. [Native observations](src/native-observation.ts) decode private window reply fields before updating coordinates and returning public results. Native implementations enforce target identity, window geometry, resource limits, and operating-system access at execution. The permission store owns durable consent separately from the native protocol. -One object facade ([src/api.ts](src/api.ts)) builds a shared `CuaSurface` and derives `CuaApp` and `CuaWindow` from it. The derived factories inject scope parameters and the accessibility and keyboard checks for their scope; the shared base does not branch on application-only capabilities. Within one API instance, the facade keeps one current accessibility snapshot per `appHandle` across every app and window object for that handle. `getWindow` reads its initial tree silently; `getApp` does the same when the provider reports application-level accessibility. Each later `getAccessibilityTree()` invalidates the previous snapshot before dispatch, and element operations use the latest snapshot ID. Only a `CuaWindow` captures screenshots or sends pointer input. +One object facade ([src/api.ts](src/api.ts)) builds a shared `CuaSurface` and derives `CuaApp` and `CuaWindow` from it. The derived factories inject scope parameters and the accessibility and keyboard checks for their scope; the shared base does not branch on application-only capabilities. Within one API instance, the facade keeps one current accessibility snapshot per `appHandle` across every app and window object for that handle. Selection leaves the current accessibility snapshot unchanged. Each explicit `getAccessibilityTree()` invalidates the previous snapshot before dispatch, and element operations use the latest snapshot ID. Only a `CuaWindow` captures screenshots or sends pointer input. Provider startup capabilities are fixed at installation: platform, methods, app-level accessibility, keyboard scopes, browser support, and coordinate space. The native `getCapabilities` RPC reports runtime permissions and session lock only; `installCua` merges both layers for the model-visible result. The raw RPC remains a common operation representation rather than a capability inventory, so provider guidance states the actions its platform supports. diff --git a/packages/experimental/cua/README.zh.md b/packages/experimental/cua/README.zh.md index d7e645b989..4422b22b2e 100644 --- a/packages/experimental/cua/README.zh.md +++ b/packages/experimental/cua/README.zh.md @@ -31,7 +31,7 @@ kind: "package-reference" 挂载 [macOS 提供方](../cua-macos/README.zh.md)以在 macOS 上启用原生操作。外部 PTC 宿主从 `@deepseek-ai/dsh-cua/bridge` 导入 `connectCua`，并提供当前 `tools.cua` 绑定。源码组合与原生组件准备由提供方负责。 -使用 `cua.getApp(selector)` 选择应用；它会获取授权、按需在后台启动已停止的应用，并在 provider 支持应用级辅助功能时静默读取初始辅助功能树。`cua.listApps` 和 `cua.getState` 列出运行中的应用；`cua.findApps(query)` 搜索已安装应用，包括未运行的应用。`app.activate()`、`app.getWindows()` 和 `app.getWindow(windowId)` 操作选定的应用；`getWindow` 返回用于截图和指针输入的对象。进行任何截图或指针输入前，必须选择应用及其一个窗口。调用 `app.getAccessibilityTree()` 输出树。读取应用或窗口时，可以单独传入 `maxElements` 或 `maxDepth`。省略的参数使用当前平台配置。在单个 API 实例内，门面按应用句柄保存一份当前辅助功能快照，该句柄的所有应用对象和窗口对象共享它，包括重复选择；每次新读取都会使上一份快照失效。元素操作使用最新快照 ID，因此调用方应使用最新树中的元素 ID，不应假定旧 ID 仍指向同一控件。元素 ID 是不透明字符串。 +使用 `cua.getApp(selector)` 选择应用；它会获取授权、按需在后台启动已停止的应用，不读取辅助功能树。`cua.listApps` 和 `cua.getState` 列出运行中的应用；`cua.findApps(query)` 搜索已安装应用，包括未运行的应用。`app.activate()`、`app.getWindows()` 和 `app.getWindow(windowId)` 操作选定的应用；`getWindow` 返回对象时不发送原生请求；窗口操作会拒绝已关闭或属于其他应用的窗口。进行任何截图或指针输入前，必须选择应用及其一个窗口。元素操作前须显式读取 `app.getAccessibilityTree()` 或 `window.getAccessibilityTree()`。AX 读取失败时仍可截图。读取应用或窗口时，可以单独传入 `maxElements` 或 `maxDepth`。省略的参数使用当前平台配置。在单个 API 实例内，门面按应用句柄保存一份当前辅助功能快照，该句柄的所有应用对象和窗口对象共享它，包括重复选择；每次新读取都会使上一份快照失效。元素操作使用最新快照 ID，因此调用方应使用最新树中的元素 ID，不应假定旧 ID 仍指向同一控件。元素 ID 是不透明字符串。 键盘方法可在 provider 声明的作用域中使用；每个作用域内的输入投递由 provider 负责。连接丢失会使其句柄和坐标失效。清理失败会先报告，再释放该连接；应重新选择应用并读取新状态，因为先前的输入可能已部分执行。截图图像错误会保留失败阶段和附件错误码。请求参数校验失败时，在分派前返回包含方法、参数和要求的可读消息。 @@ -41,7 +41,7 @@ kind: "package-reference" 应用授权发生在启动、观察、截图和输入之前。会话许可按对话和应用身份持久化，重启 DSH 或关闭 Computer Use 后仍可复用，其他对话需要单独授权。长期许可要求平台提供可复用的应用身份。拒绝或撤销会关闭受影响的 Agent 连接，因此旧对象必须重新选择目标。取消、超时或重新连接后均不重放输入。 -动作返回 `dispatched` 布尔值；`mouseMove` 还会返回 `hoverRoute` 和 `systemPointerMoved`，`show()` 还会返回 `windowId` 和 `foregroundChanged`。请读取新状态以验证结果。截图和指针操作只能通过显式窗口对象使用。坐标操作默认使用最近返回截图中的像素；成功观察窗口后，传入 `relative: true` 可使用 0～999 的整数坐标。数值除以 1000 后按窗口尺寸缩放，并限制在窗口内。Host 在返回截图前解析附件归一化和当前模型的图片缩放规则，再将输入换算为提供方的原生窗口坐标。窗口尺寸变化会使截图映射失效，绝对坐标遇到 `SCREENSHOT_REQUIRED` 时需要重新截图。窗口缩放后以 `WINDOW_CHANGED` 拒绝旧坐标；单纯移动窗口不改变窗口内坐标。 +动作返回 `dispatched` 布尔值；`mouseMove` 还会返回 `hoverRoute` 和 `systemPointerMoved`，`show()` 还会返回 `windowId` 和 `foregroundChanged`。请读取新状态以验证结果。截图和指针操作只能通过显式窗口对象使用。坐标操作默认使用最近返回截图中的像素；成功读取覆盖该窗口的 AX 或取得截图后，传入 `relative: true` 可使用 0～999 的整数坐标。仅选择窗口不会建立该观察。数值除以 1000 后按窗口尺寸缩放，并限制在窗口内。Host 在返回截图前解析附件归一化和当前模型的图片缩放规则，再将输入换算为提供方的原生窗口坐标。窗口尺寸变化会使截图映射失效，绝对坐标遇到 `SCREENSHOT_REQUIRED` 时需要重新截图。窗口缩放后以 `WINDOW_CHANGED` 拒绝旧坐标；单纯移动窗口不改变窗口内坐标。 `click(target, { holdMs: 800 })` 按住鼠标按钮 800 毫秒后松开。`holdMs` 仅适用于 `click`；它接受非负整数，双击时应用于每次按下；0 表示按下后立即松开，不额外等待。提供方配置 `maxClickHoldMs` 限制显式按住时长，默认 5000 毫秒。指定时长后使用鼠标事件；省略时保留普通点击行为。按住时间计入操作超时，取消或超时会松开按钮。已接收的输入无法撤销；辅助进程被强制终止时无法保证松开。 @@ -89,7 +89,7 @@ Computer Use 详情页的权限卡片、组件列表和来源信息使用 l3 边 [操作 schema](src/protocol.ts) 定义模型参数，以及 Host 补充的请求字段。[原生观察](src/native-observation.ts) 解析窗口回复的私有字段，更新坐标记录后返回公开结果。原生实现在执行时检查目标身份、窗口几何、资源上限和操作系统访问条件。授权存储独立于原生协议保存持久许可。 -对象门面（[src/api.ts](src/api.ts)）构建共享的 `CuaSurface`，并从它派生 `CuaApp` 和 `CuaWindow`。派生工厂注入作用域参数以及该作用域的辅助功能和键盘检查；共享基类不为仅属于应用的能力分支。在单个 API 实例内，门面按 `appHandle` 保存一份当前辅助功能快照，该句柄的所有应用对象和窗口对象共享它。`getWindow` 静默读取初始树；当 provider 报告应用级辅助功能时，`getApp` 也静默读取。之后每次 `getAccessibilityTree()` 在分派前使上一份快照失效，元素操作使用最新快照 ID。只有 `CuaWindow` 可以截图或发送指针输入。 +对象门面（[src/api.ts](src/api.ts)）构建共享的 `CuaSurface`，并从它派生 `CuaApp` 和 `CuaWindow`。派生工厂注入作用域参数以及该作用域的辅助功能和键盘检查；共享基类不为仅属于应用的能力分支。在单个 API 实例内，门面按 `appHandle` 保存一份当前辅助功能快照，该句柄的所有应用对象和窗口对象共享它。选择对象不改变当前辅助功能快照。每次显式调用 `getAccessibilityTree()` 在分派前使上一份快照失效，元素操作使用最新快照 ID。只有 `CuaWindow` 可以截图或发送指针输入。 提供方启动能力在安装时固定：平台、方法、应用级辅助功能、键盘作用域、浏览器支持和坐标空间。原生 `getCapabilities` RPC 只报告运行时权限和会话锁定状态；`installCua` 为模型可见结果合并两层。原生 RPC 仍是通用操作表示，而不是能力清单，因此 provider 指引说明其平台支持的动作。 diff --git a/packages/experimental/cua/src/api.ts b/packages/experimental/cua/src/api.ts index 704a4b8fc8..587a4bfd90 100644 --- a/packages/experimental/cua/src/api.ts +++ b/packages/experimental/cua/src/api.ts @@ -86,9 +86,7 @@ export function createCuaApi(dispatch: CuaDispatch, onText?: (text: string) => v if (!request.success) throw new Error(`getApp ${requestIssueMessage(request.error.issues)}`) const capabilities = await currentCapabilities() const { appHandle } = AppResult.parse(await call('getApp', { selector })) - const app = objects.createCuaApp(appHandle, capabilities) - if (capabilities.appLevelAccessibility) await app.getAccessibilityTree({ emit: false }) - return app + return objects.createCuaApp(appHandle, capabilities) }, } return api @@ -263,10 +261,10 @@ function createCuaObjects(call: ReturnType, onText: ((text await emit(onText, windows, options) return windows }, - async getWindow(windowId): Promise { - const window = createCuaWindow(appHandle, windowId, capabilities) - await window.getAccessibilityTree({ emit: false }) - return window + getWindow(windowId): Promise { + const result = WindowsResult.element.shape.windowId.safeParse(windowId) + if (!result.success) return Promise.reject(new Error(`getWindow ${requestIssueMessage(result.error.issues, ['params', 'windowId'])}`)) + return Promise.resolve(createCuaWindow(appHandle, result.data, capabilities)) }, } } diff --git a/packages/experimental/cua/src/guidance.ts b/packages/experimental/cua/src/guidance.ts index ddc59fc438..dfe27314c2 100644 --- a/packages/experimental/cua/src/guidance.ts +++ b/packages/experimental/cua/src/guidance.ts @@ -13,9 +13,9 @@ Setup and discovery: - cua.listApps(options?: { emit?: boolean }) -> { id: string, displayName: string, isRunning: true, pid: number }[] — enumerate running applications. - cua.findApps(query: string, options?: { emit?: boolean }) -> ({ id: string, displayName: string, path: string } & ({ isRunning: true, pid: number } | { isRunning: false }))[] — search installed applications, including stopped matches; only running matches carry pid. - cua.getState(options?: { emit?: boolean }) -> { apps: { id: string, displayName: string, isRunning: true, pid: number }[], browsers: [] } — summarize running applications; this is not an accessibility observation. -- cua.getApp(selector: string) -> App — select an application by name, bundle ID, absolute path, or running PID string, obtain consent, launch in the background if needed, and read initial AX state silently; it does not activate the app. +- cua.getApp(selector: string) -> App — select an application by name, bundle ID, absolute path, or running PID string, obtain consent, launch in the background if needed, and return without reading AX or activating the app. - app.getWindows(options?: { emit?: boolean }) -> { windowId: number, appName: string, title: string, bounds: { x: number, y: number, width: number, height: number }, isOnScreen: boolean, onCurrentSpace: boolean | null }[] — enumerate windows with global native bounds; null onCurrentSpace means unknown Space membership. -- app.getWindow(windowId: number) -> Window — select one window and read its initial AX state silently; do this before any window screenshot or coordinate input. +- app.getWindow(windowId: number) -> Window — select a window by an ID from getWindows without reading AX. Providers reject closed or foreign windows when operating on them. - app.activate() -> ActionResult — request explicit foreground activation; use only when foreground access is required. - app.show() -> { dispatched: boolean, windowId: number | null, foregroundChanged: boolean | null } — unhide the app and restore its selected, focused, main, or sole window without activation. - window.show() -> { dispatched: boolean, windowId: number | null, foregroundChanged: boolean | null } — unhide and restore that explicit window without activation. @@ -23,7 +23,7 @@ Setup and discovery: ## Accessibility tree Available at app or window level depending on the platform. -Element ids are opaque strings from the latest successful tree; the object facade keeps one current snapshot per app across its repeated app and window objects, a new read replaces it, and a failed read invalidates the previous one. Repeated strings can appear for different elements, so do not reuse an old id blindly, and element operations do not fall back to pointer input or typing. +Read getAccessibilityTree explicitly before using element ids. Selection and screenshots do not read or replace the current AX snapshot. Element ids are opaque strings from the latest successful tree; the object facade keeps one current snapshot per app across its repeated app and window objects, a new read replaces it, and a failed read invalidates the previous one. Repeated strings can appear for different elements, so do not reuse an old id blindly, and element operations do not fall back to pointer input or typing. - getAccessibilityTree(options?: { emit?: boolean, includeMenus?: boolean, details?: boolean, maxElements?: number, maxDepth?: number }) -> string — set the node count or depth for one read. Omitted limits use the provider settings. includeMenus expands menu contents and details adds identifiers, native bounds, and complete action names. - listActions(elementId: string) -> string[] — list the actions an element advertises. - invoke(elementId: string) -> ActionResult — invoke the element's default action; dispatched is false when it has none. @@ -32,7 +32,7 @@ Element ids are opaque strings from the latest successful tree; the object facad - selectText(elementId: string, text: string, options?: { prefix?: string, suffix?: string, selectionType?: 'text' | 'cursor_before' | 'cursor_after' }) -> ActionResult — select one literal occurrence or place the cursor beside it, using adjacent context to disambiguate. ## Simulated mouse -Use an explicit window for screenshots and pointer input. +Use an explicit window for screenshots and pointer input. Before coordinate input, including relative coordinates, obtain a successful getAccessibilityTree observation covering that window or take its screenshot. Absolute pixel coordinates require a returned screenshot. - getScreenshot() -> Image — capture the selected window as a stored image with readonly path, width, and height. Pass selected screenshots to inspect(...values) to display them; read the window ID from window.windowId. - ${hoverSignature} -> { dispatched: boolean, foregroundChanged?: boolean | null, hoverRoute: 'accessibility' | 'system-pointer', systemPointerMoved: boolean } — request a mouse move to the target without pressing a button; inspect hoverRoute and systemPointerMoved, and expect no automatic pointer restoration. - click(target: string | [number, number], options?: { mouseButton?: 'left' | 'right' | 'middle', clickCount?: 1 | 2, modifiers?: ('command' | 'control' | 'option' | 'shift' | 'fn')[], holdMs?: number, relative?: boolean }) -> ActionResult — click an element id or point, or perform a left-button double-click; holdMs controls the press duration of each press. @@ -47,7 +47,7 @@ Available at app or window level depending on the platform. - paste(text: string, options?: { format?: 'text' | 'html' }) -> ActionResult — deliver text or HTML through the app's Paste command, restoring prior clipboard contents unless another writer changes them; a Paste failure is surfaced without substituting typeText, and PASTE_NOT_CONSUMED means inspect before any retry. - pressKey(key: string, modifiers?: string[]) -> ActionResult — press and release one complete keystroke. For example pressKey('Tab', ['shift']) or pressKey('+'). Key and modifier names are platform-defined strings. Pass key and modifiers separately; do not pack them into a chord string. -Raw tools use { method, params }: include appHandle for app calls and windowId for window calls, and use snapshotId with elementId for element routes. The emit option is facade-only: emit is local output control, so raw callers must follow the raw operation schema for admitted params. getAccessibilityTree and getWindow read through raw getState; getWindows maps to listWindows; getScreenshot maps to screenshot; cua.getState maps to listApps; every other named operation uses the same method name.` +Raw tools use { method, params }: include appHandle for app calls and windowId for window calls, and use snapshotId with elementId for element routes. The emit option is facade-only: emit is local output control, so raw callers must follow the raw operation schema for admitted params. getAccessibilityTree maps to getState; getWindow has no raw method; getWindows maps to listWindows; getScreenshot maps to screenshot; cua.getState maps to listApps; every other named operation uses the same method name.` /** * Describe the system-pointer option for a provider that declares the capability. diff --git a/packages/experimental/cua/src/types.ts b/packages/experimental/cua/src/types.ts index 6d3ff7f2bd..799ae2e942 100644 --- a/packages/experimental/cua/src/types.ts +++ b/packages/experimental/cua/src/types.ts @@ -169,7 +169,7 @@ export interface CuaSurface { /** * An explicit window with screenshot and pointer input operations. * Capture and pointer delivery reject known AX/display frame conflicts with WINDOW_CHANGED. - * Coordinate input also rejects resizing since the latest successful AX read or screenshot. + * Coordinate input requires a successful AX observation covering the window or a screenshot, and rejects resizing since that observation. */ export interface CuaWindow extends CuaSurface { readonly windowId: number @@ -228,9 +228,10 @@ export interface CuaApp extends CuaSurface { */ getWindows(options?: CuaObservationOptions): Promise /** - * Observe a specific window. + * Select a specific window. * @param windowId - ID obtained from getWindows. - * @returns A window object with its initial accessibility tree read silently; this updates the application's current snapshot. + * @returns A window object without a native request; selection leaves the accessibility snapshot unchanged. + * Providers reject a closed or foreign window when operating on it. */ getWindow(windowId: number): Promise } @@ -262,9 +263,9 @@ export interface CuaApi { */ getState(options?: CuaObservationOptions): Promise<{ apps: CuaAppInfo[]; browsers: [] }> /** - * Select an app, obtain consent, launch it in the background if necessary, and silently retain initial AX state. + * Select an app, obtain consent, and launch it in the background if necessary, without reading AX. * @param selector - Exact name, bundle ID, absolute application path or running PID string; ambiguous names reject. - * @returns An application handle tied to this process lifetime; its initial tree read becomes the current snapshot. + * @returns An application handle tied to this process lifetime; call getAccessibilityTree explicitly before element operations. */ getApp(selector: string): Promise } diff --git a/packages/experimental/cua/tests/api.spec.ts b/packages/experimental/cua/tests/api.spec.ts index e8c7cf1721..4d069c78de 100644 --- a/packages/experimental/cua/tests/api.spec.ts +++ b/packages/experimental/cua/tests/api.spec.ts @@ -78,7 +78,7 @@ describe('application discovery', () => { const window = await app.getWindow(23) expect(await window.getScreenshot()).toEqual(IMAGE) expect(f.dispatch.mock.calls.map(([operation]) => operation.method)).toEqual([ - 'getCapabilities', 'listApps', 'getApp', 'getState', 'getState', 'screenshot', + 'getCapabilities', 'listApps', 'getApp', 'screenshot', ]) expect(await app.getWindows()).toEqual([windowInfo(23)]) expect(f.sink.text).not.toHaveBeenCalled() @@ -118,6 +118,7 @@ describe('application discovery', () => { describe('background window restoration', () => { it('shows an app or selected window without dispatching activation or AXRaise', async () => { const f = fixture() + f.setWindows([23, 24]) const app = await f.api.getApp('Editor') const window = await app.getWindow(24) f.dispatch.mockClear() @@ -151,8 +152,11 @@ describe('background window restoration', () => { describe('mouseMove across observations', () => { it('moves to an AX element then into a popup without inserting clicks or restoring a point', async () => { const f = fixture() + f.setWindows([23, 24]) const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) const window = await app.getWindow(23) + await window.getAccessibilityTree({ emit: false }) f.dispatch.mockClear() expect(await window.mouseMove(element('0'))).toEqual({ ...delivered, hoverRoute: 'accessibility', systemPointerMoved: false }) await window.getAccessibilityTree({ emit: false }) @@ -165,6 +169,7 @@ describe('mouseMove across observations', () => { { method: 'mouseMove', params: { appHandle: 'app-17', snapshotId: 'snapshot-3', windowId: 23, x: 30, y: 40 } }, ]) const other = await app.getWindow(24) + await other.getAccessibilityTree({ emit: false }) await other.mouseMove([5, 6]) expect(f.dispatch).toHaveBeenLastCalledWith({ method: 'mouseMove', params: { appHandle: 'app-17', snapshotId: 'snapshot-4', windowId: 24, x: 5, y: 6, @@ -179,7 +184,9 @@ describe('pointer buttons and modifiers', () => { it.each([0, 800])('preserves %d ms for window clicks at elements and coordinates', async (holdMs) => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) const window = await app.getWindow(23) + await window.getAccessibilityTree({ emit: false }) for (const target of [element('0'), [10, 20] as const]) { await window.click(target, { holdMs, mouseButton: 'right' }) expect(f.dispatch.mock.calls.at(-1)?.[0]).toMatchObject({ method: 'click', params: { holdMs, mouseButton: 'right' } }) @@ -192,7 +199,9 @@ describe('pointer buttons and modifiers', () => { it('retains middle-button and modifier intent through click and drag dispatch', async () => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) const window = await app.getWindow(23) + await window.getAccessibilityTree({ emit: false }) await window.click(element('0'), { mouseButton: 'middle', modifiers: ['shift'] }) expect(f.dispatch).toHaveBeenLastCalledWith({ method: 'click', params: { appHandle: 'app-17', snapshotId: 'snapshot-2', elementId: '0', windowId: 23, mouseButton: 'middle', modifiers: ['shift'], @@ -217,6 +226,7 @@ describe('text selections', () => { it('binds literal text and optional disambiguation to the latest observed control', async () => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) await app.selectText(element('1'), 'target') expect(f.dispatch).toHaveBeenLastCalledWith({ method: 'selectText', params: { appHandle: 'app-17', snapshotId: 'snapshot-1', elementId: '1', text: 'target', @@ -236,6 +246,7 @@ describe('element action queries', () => { it('returns raw provider action names including attribute-setting operations without reading new state', async () => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) f.dispatch.mockClear() expect(await app.listActions(element('42.12345'))).toEqual(['AXPress', 'AXShowMenu', 'AXTextArea.AXValue.Set']) expect(f.dispatch.mock.calls.map(([operation]) => operation)).toEqual([ @@ -251,6 +262,7 @@ describe('element actions', () => { it('dispatches advertised actions with omitted, null or direct values without wrapping the value', async () => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) f.dispatch.mockClear() expect(await app.performAction(element('42.12345'), 'AXPress')).toEqual(delivered) expect(await app.performAction(element('42.12345'), 'AXPress', null)).toEqual(delivered) @@ -268,6 +280,7 @@ describe('element actions', () => { it('rejects an empty action before dispatch', async () => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) f.dispatch.mockClear() await expect(app.performAction(element('0'), '')).rejects.toThrow('performAction action') expect(f.dispatch).not.toHaveBeenCalled() @@ -278,6 +291,7 @@ describe('element default invocation', () => { it('uses the latest snapshot and opaque element id without reading fresh state', async () => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) f.dispatch.mockClear() expect(await app.invoke(element('42.12345'))).toEqual(delivered) expect(f.dispatch.mock.calls.map(([operation]) => operation)).toEqual([ @@ -289,6 +303,7 @@ describe('element default invocation', () => { it('returns confirmed absence of a default action as un-dispatched evidence', async () => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) f.dispatch.mockResolvedValueOnce({ dispatched: false }) await expect(app.invoke(element('9'))).resolves.toEqual({ dispatched: false }) }) @@ -296,6 +311,7 @@ describe('element default invocation', () => { it('rejects provider failures and malformed dispatch evidence', async () => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) const failure = new Error('PERMISSION_DENIED') f.dispatch.mockRejectedValueOnce(failure) await expect(app.invoke(element('0'))).rejects.toBe(failure) @@ -310,9 +326,12 @@ describe('application and window observations', () => { const f = fixture() f.setWindows([23, 24]) const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) expect(await app.getWindows({ emit: false })).toEqual([windowInfo(23), windowInfo(24)]) const first = await app.getWindow(23) + await first.getAccessibilityTree({ emit: false }) const second = await app.getWindow(24) + await second.getAccessibilityTree({ emit: false }) await app.setValue(element('1'), 'First document') await first.click(element('2')) await second.click(element('3')) @@ -336,9 +355,11 @@ describe('application and window observations', () => { it('keeps separate snapshots for different application handles', async () => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) f.dispatch.mockResolvedValueOnce({ appHandle: 'app-18', name: 'Utility', running: true }) f.dispatch.mockResolvedValueOnce(state({ appHandle: 'app-18', snapshotId: 'other', windows: [] })) const other = await f.api.getApp('Utility') + await other.getAccessibilityTree({ emit: false }) await app.setValue(element('0'), 'Editor') await other.setValue(element('0'), 'Utility') expect(f.dispatch.mock.calls.slice(-2).map(([operation]) => operation)).toEqual([ @@ -350,7 +371,15 @@ describe('application and window observations', () => { it('shares one current snapshot between repeated handles for the same application', async () => { const f = fixture() const first = await f.api.getApp('Editor') + await first.getAccessibilityTree({ emit: false }) const second = await f.api.getApp('Editor') + const window = await second.getWindow(23) + await window.getScreenshot() + await first.setValue(element('1'), 'Preserved') + expect(f.dispatch).toHaveBeenLastCalledWith({ method: 'setValue', params: { + appHandle: 'app-17', snapshotId: 'snapshot-1', elementId: '1', value: 'Preserved', + } }) + await second.getAccessibilityTree({ emit: false }) expect(second.appHandle).toBe(first.appHandle) await first.setValue(element('1'), 'Shared') expect(f.dispatch).toHaveBeenLastCalledWith({ method: 'setValue', params: { @@ -361,9 +390,11 @@ describe('application and window observations', () => { it('exposes screenshots and pointer input only on explicit windows', async () => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) expect(app).toHaveProperty('getAccessibilityTree') for (const method of ['getScreenshot', 'click', 'mouseMove', 'scroll', 'drag']) expect(app).not.toHaveProperty(method) const window = await app.getWindow(23) + await window.getAccessibilityTree({ emit: false }) for (const method of ['getScreenshot', 'click', 'mouseMove', 'scroll', 'drag']) expect(window).toHaveProperty(method) expect(window).not.toHaveProperty('hover') expect(window).toHaveProperty('mouseMove') @@ -371,13 +402,17 @@ describe('application and window observations', () => { it('attaches the shared snapshot to coordinates only within its window scope', async () => { const f = fixture() + f.setWindows([23, 24]) const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) const first = await app.getWindow(23) + await first.getAccessibilityTree({ emit: false }) await first.click([10, 20]) expect(f.dispatch).toHaveBeenLastCalledWith({ method: 'click', params: { appHandle: 'app-17', windowId: 23, snapshotId: 'snapshot-2', x: 10, y: 20, mouseButton: 'left', clickCount: 1, } }) const second = await app.getWindow(24) + await second.getAccessibilityTree({ emit: false }) await first.click([10, 20]) expect(f.dispatch).toHaveBeenLastCalledWith({ method: 'click', params: { appHandle: 'app-17', windowId: 23, x: 10, y: 20, mouseButton: 'left', clickCount: 1, @@ -392,8 +427,10 @@ describe('application and window observations', () => { it('keeps a validated AXless window usable for screenshots and coordinate input', async () => { const f = fixture() const app = await f.api.getApp('Editor') - f.dispatch.mockResolvedValueOnce(state({ windowId: 23, snapshotId: 'axless', tree: '', elements: [] })) + await app.getAccessibilityTree({ emit: false }) const window = await app.getWindow(23) + f.dispatch.mockResolvedValueOnce(state({ windowId: 23, snapshotId: 'axless', tree: '', elements: [] })) + await window.getAccessibilityTree({ emit: false }) expect(f.sink.text).not.toHaveBeenCalled() await expect(window.getScreenshot()).resolves.toEqual(IMAGE) await window.click([10, 20]) @@ -405,14 +442,15 @@ describe('application and window observations', () => { } }) - it.each(['WINDOW_UNAVAILABLE', 'PERMISSION_DENIED', 'STALE_APP', 'SESSION_LOCKED'])('propagates %s while selecting a window without attempting image or input fallback', async (code) => { + it.each(['WINDOW_UNAVAILABLE', 'PERMISSION_DENIED', 'STALE_APP', 'SESSION_LOCKED'])('propagates %s from coordinate input on a selected window without image or input fallback', async (code) => { const f = fixture() const app = await f.api.getApp('Editor') + const window = await app.getWindow(23) const failure = new Error(code) f.dispatch.mockRejectedValueOnce(failure) const calls = f.dispatch.mock.calls.length - await expect(app.getWindow(23)).rejects.toBe(failure) - expect(f.dispatch.mock.calls.slice(calls).map(([operation]) => operation.method)).toEqual(['getState']) + await expect(window.click([1, 2])).rejects.toBe(failure) + expect(f.dispatch.mock.calls.slice(calls).map(([operation]) => operation.method)).toEqual(['click']) }) it('captures and clicks the explicitly selected window among several', async () => { @@ -425,8 +463,10 @@ describe('application and window observations', () => { ] f.setWindowRecords(windows) const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) expect(await app.getWindows({ emit: false })).toEqual(windows) const window = await app.getWindow(23) + await window.getAccessibilityTree({ emit: false }) await window.getScreenshot() expect(f.dispatch).toHaveBeenLastCalledWith({ method: 'screenshot', params: { appHandle: 'app-17', windowId: 23 } }) await window.click([10, 20]) @@ -434,6 +474,7 @@ describe('application and window observations', () => { appHandle: 'app-17', windowId: 23, snapshotId: 'snapshot-2', x: 10, y: 20, mouseButton: 'left', clickCount: 1, } }) const hidden = await app.getWindow(25) + await hidden.getAccessibilityTree({ emit: false }) await hidden.click([10, 20]) expect(f.dispatch).toHaveBeenLastCalledWith({ method: 'click', params: { appHandle: 'app-17', windowId: 25, snapshotId: 'snapshot-3', x: 10, y: 20, mouseButton: 'left', clickCount: 1, @@ -465,6 +506,7 @@ describe('application and window observations', () => { it('returns provider window records without process filtering', async () => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) const observed = windowInfo(24) f.dispatch.mockResolvedValueOnce(state({ windows: [observed] })) await expect(app.getAccessibilityTree()).resolves.toBe('Changed tree') @@ -490,11 +532,11 @@ describe('application and window observations', () => { { method: 'typeText', params: { appHandle: 'app-17', text: 'keyboard without accessibility' } }, ]) const window = await app.getWindow(23) - await expect(window.getAccessibilityTree({ emit: false })).resolves.toBe('Tree 2') + await expect(window.getAccessibilityTree({ emit: false })).resolves.toBe('Tree 1') await expect(app.getAccessibilityTree()).rejects.toThrow('Application-level accessibility is not supported on current platform; observe a window instead.') await expect(window.invoke(element('0'))).resolves.toEqual(delivered) expect(f.dispatch).toHaveBeenLastCalledWith({ method: 'invoke', params: { - appHandle: 'app-17', windowId: 23, snapshotId: 'snapshot-2', elementId: '0', + appHandle: 'app-17', windowId: 23, snapshotId: 'snapshot-1', elementId: '0', } }) }) @@ -502,7 +544,9 @@ describe('application and window observations', () => { const f = fixture() f.setCapabilities({ ...capabilities, keyboard: ['app', 'window'] }) const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) const window = await app.getWindow(23) + await window.getAccessibilityTree({ emit: false }) f.dispatch.mockClear() await window.typeText('hello') await window.paste('bold', { format: 'html' }) @@ -524,7 +568,9 @@ describe('application and window observations', () => { const f = fixture() f.setCapabilities({ ...capabilities, keyboard: ['window'] }) const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) const window = await app.getWindow(23) + await window.getAccessibilityTree({ emit: false }) f.dispatch.mockClear() await expect(app.typeText('app text')).rejects.toThrow('App-scoped keyboard input is not supported on current platform.') await expect(app.paste('app text')).rejects.toThrow('App-scoped keyboard input is not supported on current platform.') @@ -538,7 +584,9 @@ describe('application and window observations', () => { it('accepts coordinate input without a current snapshot', async () => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) const window = await app.getWindow(23) + await window.getAccessibilityTree({ emit: false }) f.dispatch.mockRejectedValueOnce(new Error('Native app is busy')) await expect(window.getAccessibilityTree()).rejects.toThrow('Native app is busy') await window.click([10, 20]) @@ -550,7 +598,9 @@ describe('application and window observations', () => { it('clears a failed refresh and requires a new read before element operations', async () => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) const window = await app.getWindow(23) + await window.getAccessibilityTree({ emit: false }) const failure = new Error('Native app is busy') f.dispatch.mockRejectedValueOnce(failure) await expect(app.getAccessibilityTree()).rejects.toBe(failure) @@ -568,11 +618,51 @@ describe('application and window observations', () => { }) describe('screenshots', () => { + it('selects windows without a native request and rejects invalid IDs locally', async () => { + const f = fixture() + const app = await f.api.getApp('Editor') + expect((await app.getWindow(24)).windowId).toBe(24) + await expect(app.getWindow(0)).rejects.toThrow('getWindow windowId: Too small: expected number to be >0') + expect(f.dispatch.mock.calls.map(([request]) => request.method)).toEqual(['getCapabilities', 'getApp']) + }) + + it.each(['APP_ACCESS_REVOKED', 'WINDOW_UNAVAILABLE', 'STALE_APP'])('preserves %s when capturing a previously selected window', async (code) => { + const f = fixture() + const app = await f.api.getApp('Editor') + const window = await app.getWindow(23) + f.dispatch.mockRejectedValueOnce(new Error(code)) + await expect(window.getScreenshot()).rejects.toThrow(code) + expect(f.sink.text).not.toHaveBeenCalled() + }) + + it.each(['application', 'window'])('captures the first screenshot when %s AX reads fail', async (scope) => { + const f = fixture() + const dispatch = f.dispatch.getMockImplementation()! + const failure = new Error('AX read failed') + f.dispatch.mockImplementation(async (request) => { + if (request.method === 'getState' && (scope === 'application' ? request.params.windowId === undefined : request.params.windowId !== undefined)) throw failure + return dispatch(request) + }) + const app = await f.api.getApp('Editor') + expect(await app.getWindows({ emit: false })).toEqual([windowInfo(23)]) + const window = await app.getWindow(23) + const target = scope === 'application' ? app : window + await expect(target.getAccessibilityTree()).rejects.toBe(failure) + await expect(window.getScreenshot()).resolves.toBe(IMAGE) + await expect(window.invoke(element('0'))).rejects.toThrow('Read getAccessibilityTree') + await window.click([0, 0]) + expect(f.dispatch).toHaveBeenLastCalledWith({ method: 'click', params: { + appHandle: 'app-17', windowId: 23, x: 0, y: 0, mouseButton: 'left', clickCount: 1, + } }) + }) + it('returns the stored screenshot Image and emits selected accessibility text', async () => { const f = fixture() f.setWindows([23, 24]) const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) const window = await app.getWindow(24) + await window.getAccessibilityTree({ emit: false }) f.sink.text.mockClear() const tree = await window.getAccessibilityTree() const image = await window.getScreenshot() @@ -588,18 +678,31 @@ describe('screenshots', () => { it.each([null, 'text only', {}, { path: '/image.png', width: 1 }, { path: '/image.png', width: 1, height: 1 }])('rejects an invalid screenshot result %j', async (value) => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) const window = await app.getWindow(23) f.dispatch.mockResolvedValueOnce(value) await expect(window.getScreenshot()).rejects.toThrow('must return an Image') }) + it.each(['jpeg', 'webp', 'gif'])('retains the Host Image for a %s screenshot', async (format) => { + const f = fixture() + const app = await f.api.getApp('Editor') + const window = await app.getWindow(23) + const image = Object.freeze({ ...IMAGE, path: '/images/screenshot.' + format, width: 320, height: 240 }) + f.dispatch.mockResolvedValueOnce(image) + expect(await window.getScreenshot()).toBe(image) + expect(f.sink.text).not.toHaveBeenCalled() + }) + }) describe('input actions', () => { it('passes explicit relative mode to every coordinate gesture and rejects it on element ids', async () => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) const window = await app.getWindow(23) + await window.getAccessibilityTree({ emit: false }) await window.click([500, 250], { relative: true }) await window.mouseMove([500, 250], { relative: true }) await window.drag([0, 0], [999, 999], { relative: true }) @@ -617,7 +720,9 @@ describe('input actions', () => { it('keeps the selected window when scroll options include another target identity', async () => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) const window = await app.getWindow(23) + await window.getAccessibilityTree({ emit: false }) const options = { x: 10, y: 20, deltaLinesX: 0, deltaLinesY: 1, appHandle: 'another-app', snapshotId: 'another-snapshot', windowId: 24, @@ -633,7 +738,9 @@ describe('input actions', () => { it('dispatches window-point gestures and reports delivery without claiming the action succeeded', async () => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) const window = await app.getWindow(23) + await window.getAccessibilityTree({ emit: false }) expect(await window.click([10, 20], { mouseButton: 'right' })).toEqual(delivered) expect(await window.click([11, 21], { clickCount: 2 })).toEqual(delivered) expect(await window.scroll({ x: 10, y: 20, deltaLinesX: -2, deltaLinesY: 3 })).toEqual(delivered) @@ -664,7 +771,6 @@ describe('input actions', () => { ['empty key', ({ app }) => app.pressKey('')], ] it.each<[string, (surfaces: Surfaces) => Promise]>([ - ['getState windowId: Too small: expected number to be >0', ({ app }) => app.getWindow(0)], ['selectText text: Too small: expected string to have >=1 characters', ({ app }) => app.selectText(element('0'), '')], ['mouseMove x: Invalid input: expected number, received NaN', ({ window }) => window.mouseMove([NaN, 0])], ['mouseMove x: Too big: expected number to be <=999', ({ window }) => window.mouseMove([1000, 0], { relative: true })], @@ -672,7 +778,9 @@ describe('input actions', () => { ])('reports a readable validation failure: %s', async (message, run) => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) const window = await app.getWindow(23) + await window.getAccessibilityTree({ emit: false }) f.dispatch.mockClear() await expect(run({ app, window })).rejects.toMatchObject({ name: 'Error', message }) expect(f.dispatch).not.toHaveBeenCalled() @@ -681,7 +789,9 @@ describe('input actions', () => { it.each(invalidActions)('rejects %s without invoking native input', async (_label, run) => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) const window = await app.getWindow(23) + await window.getAccessibilityTree({ emit: false }) const calls = f.dispatch.mock.calls.length await expect(run({ app, window })).rejects.toThrow() expect(f.dispatch).toHaveBeenCalledTimes(calls) @@ -727,6 +837,7 @@ describe('input actions', () => { it('passes literal text and keys to the selected app identity', async () => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) await app.typeText('中文\nline two') await app.paste('粘贴\nplain text') await app.paste('bold', { format: 'html' }) @@ -744,7 +855,9 @@ describe('input actions', () => { it('rejects keyboard input on a window surface without dispatching an app-level fallback', async () => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) const window = await app.getWindow(23) + await window.getAccessibilityTree({ emit: false }) f.dispatch.mockClear() for (const run of [ () => window.typeText('hello'), @@ -766,6 +879,7 @@ describe('input actions', () => { const f = fixture() f.setCapabilities({ ...capabilities, keyboard: ['app', 'window'] }) const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) const surface = scope === 'window' ? await app.getWindow(23) : app f.dispatch.mockClear() f.dispatch.mockResolvedValueOnce(delivered) @@ -779,6 +893,7 @@ describe('input actions', () => { it('returns an un-dispatched key-event result from one RPC without retrying', async () => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) f.dispatch.mockClear() f.dispatch.mockResolvedValueOnce({ dispatched: false }) await expect(app.sendKeyEvent('Command', 'down')).resolves.toEqual({ dispatched: false }) @@ -808,7 +923,9 @@ describe('input actions', () => { it('retains the snapshot identity for element right and double clicks', async () => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) const window = await app.getWindow(23) + await window.getAccessibilityTree({ emit: false }) await window.click(element('12'), { mouseButton: 'right' }) await window.click(element('13'), { clickCount: 2 }) expect(f.dispatch.mock.calls.slice(-2).map(([operation]) => operation)).toEqual([ @@ -820,7 +937,9 @@ describe('input actions', () => { it('dispatches an opaque provider element id unchanged', async () => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) const window = await app.getWindow(23) + await window.getAccessibilityTree({ emit: false }) await window.click(element('42.12345')) await window.mouseMove(element('42.12345')) expect(f.dispatch.mock.calls.slice(-2).map(([operation]) => operation)).toEqual([ @@ -832,7 +951,7 @@ describe('input actions', () => { it('does not activate on selection and dispatches only an explicit activation request', async () => { const f = fixture() const app = await f.api.getApp('Editor') - expect(f.dispatch.mock.calls.map(([call]) => call.method)).toEqual(['getCapabilities', 'getApp', 'getState']) + expect(f.dispatch.mock.calls.map(([call]) => call.method)).toEqual(['getCapabilities', 'getApp']) await app.activate() expect(f.dispatch).toHaveBeenLastCalledWith({ method: 'activate', params: { appHandle: 'app-17' } }) }) @@ -840,7 +959,9 @@ describe('input actions', () => { it('propagates native failures once and rejects malformed delivery claims', async () => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) const window = await app.getWindow(23) + await window.getAccessibilityTree({ emit: false }) const failure = new Error('Application was restarted; select it again') f.dispatch.mockRejectedValueOnce(failure) await expect(app.typeText('hello')).rejects.toBe(failure) @@ -855,6 +976,7 @@ describe('input actions', () => { await expect(f.api.getApp('')).rejects.toThrow() expect(f.dispatch).not.toHaveBeenCalled() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) const calls = f.dispatch.mock.calls.length await expect(app.getWindow(0)).rejects.toThrow() expect(f.dispatch).toHaveBeenCalledTimes(calls) @@ -873,22 +995,23 @@ it('expands menu contents and details only for the requested observation', async it('forwards provider options while retaining the selected target and coordinate mode', async () => { const f = fixture() const win = await (await f.api.getApp('Editor')).getWindow(23) + await win.getAccessibilityTree({ emit: false }) const defaultCoordinates: CuaCoordinateOptions = {} const elementOptions = { ...defaultCoordinates, trackingMode: 'precise', appHandle: 'another-app', windowId: 99, unused: undefined } await win.mouseMove(element('0'), elementOptions) expect(f.dispatch).toHaveBeenLastCalledWith({ method: 'mouseMove', params: { - appHandle: 'app-17', windowId: 23, snapshotId: 'snapshot-2', elementId: '0', trackingMode: 'precise', + appHandle: 'app-17', windowId: 23, snapshotId: 'snapshot-1', elementId: '0', trackingMode: 'precise', } }) await win.getScreenshot() const pixelOptions = { relative: false, useSystemPointer: true, trackingMode: 'precise', x: 222 } await win.mouseMove([30, 40], pixelOptions) expect(f.dispatch).toHaveBeenLastCalledWith({ method: 'mouseMove', params: { - appHandle: 'app-17', snapshotId: 'snapshot-2', windowId: 23, x: 30, y: 40, relative: false, useSystemPointer: true, trackingMode: 'precise', + appHandle: 'app-17', snapshotId: 'snapshot-1', windowId: 23, x: 30, y: 40, relative: false, useSystemPointer: true, trackingMode: 'precise', } }) const relativeOptions = { relative: true, trackingMode: 'coarse' } await win.mouseMove([250, 750], relativeOptions) expect(f.dispatch).toHaveBeenLastCalledWith({ method: 'mouseMove', params: { - appHandle: 'app-17', snapshotId: 'snapshot-2', windowId: 23, x: 250, y: 750, relative: true, trackingMode: 'coarse', + appHandle: 'app-17', snapshotId: 'snapshot-1', windowId: 23, x: 250, y: 750, relative: true, trackingMode: 'coarse', } }) }) diff --git a/packages/experimental/cua/tests/bridge.spec.ts b/packages/experimental/cua/tests/bridge.spec.ts index 984c89c661..bcf569ece3 100644 --- a/packages/experimental/cua/tests/bridge.spec.ts +++ b/packages/experimental/cua/tests/bridge.spec.ts @@ -33,11 +33,12 @@ it('uses the current tool binding when a retained app is called by another execu const cua = connectCua(() => current, text) const app = await cua.getApp('Editor') const window = await app.getWindow(23) + await window.getAccessibilityTree({ emit: false }) const second = vi.fn(async (): Promise => ({ dispatched: true })) current = second await expect(window.click(element('0'))).resolves.toEqual({ dispatched: true }) - expect(first).toHaveBeenCalledTimes(4) + expect(first).toHaveBeenCalledTimes(3) expect(second).toHaveBeenCalledWith({ method: 'click', params: { appHandle: 'app-17', snapshotId: 'snapshot-1', elementId: '0', windowId: 23 } }) expect(text).not.toHaveBeenCalled() current = first @@ -77,6 +78,6 @@ it('returns the program Image unchanged from one screenshot tool call', async () text.mockClear() expect(await window.getScreenshot()).toBe(image) - expect(tool.mock.calls.map(([operation]) => operation.method)).toEqual(['getCapabilities', 'getApp', 'getState', 'getState', 'screenshot']) + expect(tool.mock.calls.map(([operation]) => operation.method)).toEqual(['getCapabilities', 'getApp', 'screenshot']) expect(text).not.toHaveBeenCalled() }) diff --git a/packages/llm/llm-pi-ai/README.i18n.yaml b/packages/llm/llm-pi-ai/README.i18n.yaml index f7c103d726..fd6a360e8b 100644 --- a/packages/llm/llm-pi-ai/README.i18n.yaml +++ b/packages/llm/llm-pi-ai/README.i18n.yaml @@ -93,8 +93,8 @@ en: b6bd570c0159ba08 zh: 7d2d91eea6a8d830 /deepseek-ai-dsh-llm-pi-ai/known-limitations-and-deferred-work: - en: 3a0cf0d369d25d37 - zh: 5974f328aea06aa8 + en: fdbe8ce850250b66 + zh: d8b9a06a996850c1 /deepseek-ai-dsh-llm-pi-ai/known-limitations-and-deferred-work/dev-note: en: 3678dd0f6642c2a5 zh: 75cbdd064cc64282 diff --git a/packages/llm/llm-pi-ai/README.md b/packages/llm/llm-pi-ai/README.md index c063137301..2bba84d774 100644 --- a/packages/llm/llm-pi-ai/README.md +++ b/packages/llm/llm-pi-ai/README.md @@ -271,7 +271,7 @@ These limits define where the adapter stops and future work begins. They are cur - **`GenerateOptions.stop` is unsupported** — pi-ai's common stream options cannot guarantee stop-sequence behavior across providers. - **Conversation update support comes only from the installed pi-ai catalog** — native system and tool updates require the catalog model’s mid-conversation compat flags, which profiles cannot set. Hand-declared routes and unflagged models receive later system text as user-role reminders and the complete current tool list at the start of each request. - **Images are accepted only in user and tool-result messages** — images in system, developer, or assistant history fail with `UNSUPPORTED_CONTENT` on both conversion paths. -- **Provider HTTP status is unavailable** — pi-ai error events do not expose a stable HTTP status across providers. +- **Provider HTTP status is unavailable** — pi-ai error events do not expose a stable HTTP status across providers, so failures are classified from the error text. When a message leads with a 4xx or 5xx status, as Anthropic and OpenAI SDK errors do, the status rules read only that status. Other formats, such as `Mistral API error (): `, still match a status quoted anywhere in the message, and the quota, rate-limit, and invalid-request wording rules read the whole message, so a gateway 5xx that relays an upstream `insufficient_quota` or `invalid_request_error` becomes a non-retryable `QUOTA` or `INVALID_REQUEST` failure. - **Retry policy is provider-owned, not an SDK retry** — pi-ai SDK retries stay disabled so durable agent steps and `llm/retry` events own every visible attempt, and direct `ctx.llm.stream()` calls remain single-attempt. - **Streamed JSON tool-call arguments avoid per-delta reparsing** — the installed pi-ai carries [`patches/@earendil-works__pi-ai@1.3.0.patch`](../../../patches/@earendil-works__pi-ai@1.3.0.patch), which removes the per-delta re-parse of the whole accumulated argument JSON in every stream adapter (upstream [earendil-works/pi#9265](https://github.com/earendil-works/pi/issues/9265)); unpatched, a multi-megabyte argument stream costs O(n²) CPU on the event loop and stalls every session in the process. Completion events supply parsed arguments; Responses may parse at both argument completion and output-item completion. This adapter reads only delta strings and finalized arguments. Re-apply or retire the patch on every pi-ai upgrade. - **Stale Anthropic thinking is omitted even where the endpoint would accept it** — prefix checks depend on the model, the account, and the request, which the adapter cannot observe, so every prefix change and every replay state without `prefixDigest` costs the model that reasoning, and Anthropic regenerates it as output tokens. The digest covers the pi-ai context objects, not the wire body: switching authentication or wire-changing compat settings mid-conversation is not detected, a pi-ai upgrade that changes Anthropic message serialization requires a new `DIGEST_SEED` in [`src/prefix-binding.ts`](src/prefix-binding.ts), and any change to how this package builds those objects drops recorded thinking once. diff --git a/packages/llm/llm-pi-ai/README.zh.md b/packages/llm/llm-pi-ai/README.zh.md index a8ee6f2eb6..9a964b8681 100644 --- a/packages/llm/llm-pi-ai/README.zh.md +++ b/packages/llm/llm-pi-ai/README.zh.md @@ -271,7 +271,7 @@ pi-ai 事件变成 harness 的推理、文本、工具调用、用量与 finish - **不支持 `GenerateOptions.stop`**——pi-ai 的通用流式选项无法跨提供方保证停止序列行为。 - **对话更新支持只来自已安装的 pi-ai 目录**——原生系统与工具更新需要目录模型的对话中 compat 标志，profile 不能设置这些标志。手工声明的路由与未标记的模型会以用户角色提醒接收后续系统文本，并在每次请求开头接收完整的当前工具列表。 - **图片只在 user 与工具结果消息中被接受**——system、developer 或 assistant 历史中的图片在两条转换路径上都会以 `UNSUPPORTED_CONTENT` 失败。 -- **提供方 HTTP 状态不可用**——pi-ai 错误事件不跨提供方暴露稳定 HTTP 状态。 +- **提供方 HTTP 状态不可用**——pi-ai 错误事件不跨提供方暴露稳定 HTTP 状态，失败按错误文本分类。信息以 4xx 或 5xx 状态码开头时（Anthropic 与 OpenAI SDK 的错误即如此），状态码规则只读取该状态码。其他格式（如 `Mistral API error (): `）仍会匹配信息中任意位置引用的状态码；配额、限流与无效请求的措辞规则读取整条信息，因此网关在 5xx 响应体中转述上游的 `insufficient_quota` 或 `invalid_request_error` 时，会分类为不重试的 `QUOTA` 或 `INVALID_REQUEST`。 - **重试策略由提供方自有，而非 SDK 重试**——pi-ai SDK 重试保持禁用，因此持久 agent（智能体）步骤与 `llm/retry` 事件拥有每个可见尝试，直接 `ctx.llm.stream()` 调用仍是单次尝试。 - **流式 JSON 工具调用参数避免逐 delta 重复解析**——安装的 pi-ai 带有 [`patches/@earendil-works__pi-ai@1.3.0.patch`](../../../patches/@earendil-works__pi-ai@1.3.0.patch)，它移除了每个流适配器中对整段累计参数 JSON 的逐 delta 重新解析（上游 [earendil-works/pi#9265](https://github.com/earendil-works/pi/issues/9265)）；未打补丁时，数 MB 的参数流会在事件循环上消耗 O(n²) CPU，并使进程内所有会话停滞。完成事件提供解析后的参数；Responses 可能在参数完成和输出项完成时分别解析。本适配器只读取 delta 字符串与最终参数。每次升级 pi-ai 时都要重新应用或撤销该补丁。 - **即使端点会接受，过期的 Anthropic thinking 也会被省略**——前缀检查取决于模型、账户和请求，而适配器无法观测这些因素，因此每次前缀变化以及每个不含 `prefixDigest` 的回放状态都会让模型失去这部分推理，Anthropic 会以输出 token 重新生成它。摘要覆盖 pi-ai 上下文对象，而不是线上请求体：对话中途切换认证或会改变线上请求的 compat 设置不会被检测到；改变 Anthropic 消息序列化的 pi-ai 升级需要在 [`src/prefix-binding.ts`](src/prefix-binding.ts) 中使用新的 `DIGEST_SEED`；本包构建这些对象的方式发生任何变化，都会使已记录的 thinking 被丢弃一次。 diff --git a/packages/llm/llm-pi-ai/src/stream.ts b/packages/llm/llm-pi-ai/src/stream.ts index 58bd2e8e3e..c3bfc1cd3f 100644 --- a/packages/llm/llm-pi-ai/src/stream.ts +++ b/packages/llm/llm-pi-ai/src/stream.ts @@ -31,6 +31,13 @@ export function mapUsage(usage: PiUsage): TokenUsage { } } +// pi-ai's SDK-backed providers render an HTTP failure as ` ` +// (Anthropic and OpenAI SDK `APIError`, pi-messages ` : …`). +// The body can quote another hop's status, such as a gateway 500 that embeds its +// upstream's 401, so when a message leads with a status the status rules read +// only that status; other wordings scan the whole message. +const LEADING_HTTP_STATUS = /^[45]\d\d\b/ + // XXX(pi-ai upstream): pi-ai flattens the caught error to `error.message` // (api/anthropic-messages.js: `errorMessage = error instanceof Error ? // error.message : JSON.stringify(error)`), discarding the original Error and its @@ -40,14 +47,15 @@ export function mapUsage(usage: PiUsage): TokenUsage { // If pi-ai ever forwards the original Error (or a fetch/dispatcher hook that lets // us capture the cause ourselves), classify on `code`/`cause` instead of text. function classifyPiAiError(message: string): BuiltinLlmFailureCode { - if (/\b(?:401|403)\b/.test(message)) return 'AUTH' + const statusSource = LEADING_HTTP_STATUS.exec(message)?.[0] ?? message + if (/\b(?:401|403)\b/.test(statusSource)) return 'AUTH' if (isQuotaExceededError(message)) return QUOTA_EXCEEDED_CODE - if (/\b429\b|rate.?limit/i.test(message)) return 'RATE_LIMIT' + if (/\b429\b/.test(statusSource) || /rate.?limit/i.test(message)) return 'RATE_LIMIT' // A rejected request body (gateway or provider size cap): resending the // same request cannot succeed, so it is invalid, not transient. - if (/\b413\b|failed to buffer the request body:\s*length limit exceeded|payload too large|request body too large/i.test(message)) return 'INVALID_REQUEST' - if (/\b400\b|invalid.?request/i.test(message)) return 'INVALID_REQUEST' - if (/\b5\d\d\b/.test(message)) return 'SERVER' + if (/\b413\b/.test(statusSource) || /failed to buffer the request body:\s*length limit exceeded|payload too large|request body too large/i.test(message)) return 'INVALID_REQUEST' + if (/\b400\b/.test(statusSource) || /invalid.?request/i.test(message)) return 'INVALID_REQUEST' + if (/\b5\d\d\b/.test(statusSource)) return 'SERVER' if (/\btime(?:d)?\s*out\b|timeout/i.test(message)) return 'TIMEOUT' // A stream truncated before the provider's terminal event: each pi-ai provider // throws its own wording when the wire closes mid-response without a terminal diff --git a/packages/llm/llm-pi-ai/tests/adapter.spec.ts b/packages/llm/llm-pi-ai/tests/adapter.spec.ts index 4810a236c2..076f0fdf5f 100644 --- a/packages/llm/llm-pi-ai/tests/adapter.spec.ts +++ b/packages/llm/llm-pi-ai/tests/adapter.spec.ts @@ -474,6 +474,29 @@ describe('PiAiAdapter provider routing', () => { expect(server.paths).toEqual(['/chat/completions']) }) + it('classifies a gateway 500 that relays an upstream 401 as SERVER with the rendered message', async () => { + vi.stubEnv('PI_TEST_KEY', 'test-key') + // The Anthropic SDK renders the whole JSON body after the gateway's own status. + const body = JSON.stringify({ + error: { + origin: 'upstream', + message: 'upstream returned an error response: User not found.', + type: 'upstream_error', + upstream: { provider: 'custom', status: 401, body: { error: { message: 'User not found.' } } }, + }, + }) + const server = await mockServer([{ status: 500, body }]) + const ctx = new Context() + await ctx.plugin(LlmRuntime) + await ctx.plugin(LlmPiAi, { + providers: { + gateway: { apiKeyEnv: 'PI_TEST_KEY', api: 'anthropic-messages', baseURL: server.url, models: [{ id: 'gateway-model' }] }, + }, + }) + const result = await assemble(ctx, { provider: 'gateway', model: 'gateway-model', messages: [] }) + expect(result.finish).toEqual({ kind: 'error', failure: { message: `500 ${body}`, code: 'SERVER' } }) + }) + it('uses the resolved catalog context window for usage-based overflow detection', async () => { const model = getBuiltinModels('deepseek').find(candidate => candidate.id === 'deepseek-flash') if (model === undefined) throw new Error('deepseek-flash missing from pi-ai test catalog') diff --git a/packages/llm/llm-pi-ai/tests/convert.spec.ts b/packages/llm/llm-pi-ai/tests/convert.spec.ts index fe6d72a2b4..ff5a65870b 100644 --- a/packages/llm/llm-pi-ai/tests/convert.spec.ts +++ b/packages/llm/llm-pi-ai/tests/convert.spec.ts @@ -939,6 +939,19 @@ describe('mapStopReason / mapUsage', () => { }))).toMatchObject({ kind: 'error', failure: { code: 'PI_AI_ERROR' } }) }) + it.each([ + // The gateway body as the Anthropic SDK renders it: the gateway's own 500 leads. + ['a gateway 500 relaying an upstream 401', '500 {"error":{"origin":"upstream","message":"upstream returned an error response: User not found.","type":"upstream_error","code":"upstream_response_error","upstream":{"provider":"custom","status":401,"body":{"error":{"type":"","message":"User not found."},"type":"error"}}}}', 'SERVER'], + ['a 502 quoting a non-retryable 400', '502 {"error":{"message":"upstream status 400"}}', 'SERVER'], + ['a 400 quoting a retryable 429', '400 {"error":{"message":"upstream returned 429 Too Many Requests"}}', 'INVALID_REQUEST'], + ['a 404 quoting a 403', '404 {"error":{"message":"upstream status 403"}}', 'PI_AI_ERROR'], + ['a leading 401', '401 {"type":"error","error":{"type":"authentication_error","message":"invalid x-api-key"}}', 'AUTH'], + ['a leading 429', '429 Too Many Requests: slow down', 'RATE_LIMIT'], + ] as const)('classifies %s by its leading response status', (_label, errorMessage, code) => { + expect(mapStopReason(assistant({ stopReason: 'error', errorMessage }))) + .toEqual({ kind: 'error', failure: { message: errorMessage, code } }) + }) + it.each([ 'other side closed', 'HTTP2 request did not get a response', diff --git a/snapshots/session/cua-native/system-prompt.expected.md b/snapshots/session/cua-native/system-prompt.expected.md index e3528c757e..3b3eeaa1e0 100644 --- a/snapshots/session/cua-native/system-prompt.expected.md +++ b/snapshots/session/cua-native/system-prompt.expected.md @@ -46,9 +46,9 @@ Setup and discovery: - cua.listApps(options?: { emit?: boolean }) -> { id: string, displayName: string, isRunning: true, pid: number }[] — enumerate running applications. - cua.findApps(query: string, options?: { emit?: boolean }) -> ({ id: string, displayName: string, path: string } & ({ isRunning: true, pid: number } | { isRunning: false }))[] — search installed applications, including stopped matches; only running matches carry pid. - cua.getState(options?: { emit?: boolean }) -> { apps: { id: string, displayName: string, isRunning: true, pid: number }[], browsers: [] } — summarize running applications; this is not an accessibility observation. -- cua.getApp(selector: string) -> App — select an application by name, bundle ID, absolute path, or running PID string, obtain consent, launch in the background if needed, and read initial AX state silently; it does not activate the app. +- cua.getApp(selector: string) -> App — select an application by name, bundle ID, absolute path, or running PID string, obtain consent, launch in the background if needed, and return without reading AX or activating the app. - app.getWindows(options?: { emit?: boolean }) -> { windowId: number, appName: string, title: string, bounds: { x: number, y: number, width: number, height: number }, isOnScreen: boolean, onCurrentSpace: boolean | null }[] — enumerate windows with global native bounds; null onCurrentSpace means unknown Space membership. -- app.getWindow(windowId: number) -> Window — select one window and read its initial AX state silently; do this before any window screenshot or coordinate input. +- app.getWindow(windowId: number) -> Window — select a window by an ID from getWindows without reading AX. Providers reject closed or foreign windows when operating on them. - app.activate() -> ActionResult — request explicit foreground activation; use only when foreground access is required. - app.show() -> { dispatched: boolean, windowId: number | null, foregroundChanged: boolean | null } — unhide the app and restore its selected, focused, main, or sole window without activation. - window.show() -> { dispatched: boolean, windowId: number | null, foregroundChanged: boolean | null } — unhide and restore that explicit window without activation. @@ -56,7 +56,7 @@ Setup and discovery: ## Accessibility tree Available at app or window level depending on the platform. -Element ids are opaque strings from the latest successful tree; the object facade keeps one current snapshot per app across its repeated app and window objects, a new read replaces it, and a failed read invalidates the previous one. Repeated strings can appear for different elements, so do not reuse an old id blindly, and element operations do not fall back to pointer input or typing. +Read getAccessibilityTree explicitly before using element ids. Selection and screenshots do not read or replace the current AX snapshot. Element ids are opaque strings from the latest successful tree; the object facade keeps one current snapshot per app across its repeated app and window objects, a new read replaces it, and a failed read invalidates the previous one. Repeated strings can appear for different elements, so do not reuse an old id blindly, and element operations do not fall back to pointer input or typing. - getAccessibilityTree(options?: { emit?: boolean, includeMenus?: boolean, details?: boolean, maxElements?: number, maxDepth?: number }) -> string — set the node count or depth for one read. Omitted limits use the provider settings. includeMenus expands menu contents and details adds identifiers, native bounds, and complete action names. macOS accepts maxElements up to 100000 and maxDepth up to 512. If the tree is truncated, select a narrower window or raise the reported limit on the next read. In persistent PTC, capture a large tree with const tree = await win.getAccessibilityTree({ emit: false }). Filter its lines in memory, or save it with await tools.write({ file_path: path, content: tree }) to a new workspace file. Search its contents with tools.grep({ path, pattern }) and use tools.read around a matching line. Print only the matches you need. Use element ids from the file only while that observation is current. - listActions(elementId: string) -> string[] — list the actions an element advertises. - invoke(elementId: string) -> ActionResult — invoke the element's default action; dispatched is false when it has none. @@ -65,7 +65,7 @@ Element ids are opaque strings from the latest successful tree; the object facad - selectText(elementId: string, text: string, options?: { prefix?: string, suffix?: string, selectionType?: 'text' | 'cursor_before' | 'cursor_after' }) -> ActionResult — select one literal occurrence or place the cursor beside it, using adjacent context to disambiguate. ## Simulated mouse -Use an explicit window for screenshots and pointer input. +Use an explicit window for screenshots and pointer input. Before coordinate input, including relative coordinates, obtain a successful getAccessibilityTree observation covering that window or take its screenshot. Absolute pixel coordinates require a returned screenshot. - getScreenshot() -> Image — capture the selected window as a stored image with readonly path, width, and height. Pass selected screenshots to inspect(...values) to display them; read the window ID from window.windowId. - mouseMove(target: string | [number, number], options?: { relative?: boolean, useSystemPointer?: boolean }) -> { dispatched: boolean, foregroundChanged?: boolean | null, hoverRoute: 'accessibility' | 'system-pointer', systemPointerMoved: boolean } — request a mouse move to the target without pressing a button; inspect hoverRoute and systemPointerMoved, and expect no automatic pointer restoration. Default hover requires both AXShowAlternateUI and AXShowDefaultUI on an enabled element or ancestor. Without that route, HOVER_UNAVAILABLE is returned before sending input. Physical movement requires an unobscured foreground target. Explain any necessary activation and observe the target again. - click(target: string | [number, number], options?: { mouseButton?: 'left' | 'right' | 'middle', clickCount?: 1 | 2, modifiers?: ('command' | 'control' | 'option' | 'shift' | 'fn')[], holdMs?: number, relative?: boolean }) -> ActionResult — click an element id or point, or perform a left-button double-click; holdMs controls the press duration of each press. click is a Window method: select a window with const win = await app.getWindow(windowId), then call win.click(target). For a background control’s default action, use app.invoke(id). For a mouse click on a background app, call app.activate() first and re-read the target. Both target forms send mouse-down and mouse-up events. click(id) uses the center of the full element bounds. If that center is outside the window, take a fresh win.getScreenshot() and use click([x, y]) on a visible part of the control in screenshot pixels. A background app can ignore the events even when dispatched is true. Verify the expected effect, such as a changed selection, before typing or retrying. @@ -81,7 +81,7 @@ Available at app or window level depending on the platform. - pressKey(key: string, modifiers?: string[]) -> ActionResult — press and release one complete keystroke. For example pressKey('Tab', ['shift']) or pressKey('+'). Key and modifier names are platform-defined strings. Pass key and modifiers separately; do not pack them into a chord string. - sendKeyEvent(key: string, direction: 'down' | 'up') -> ActionResult — send 'down' to hold a key across tool calls in one run_code, for example W down, D down, D up, W up. Holding a key on a physical keyboard can produce repeated key-down events after a delay. To send one such repeat for a non-modifier key this run already holds, send 'down' again. Each call posts one event. A control can filter repeats. Use the app's response to decide whether and when to repeat or release the key. When this run_code returns, throws, or the user pauses it, macOS releases only its remaining keys at their original recipients. -Raw tools use { method, params }: include appHandle for app calls and windowId for window calls, and use snapshotId with elementId for element routes. The emit option is facade-only: emit is local output control, so raw callers must follow the raw operation schema for admitted params. getAccessibilityTree and getWindow read through raw getState; getWindows maps to listWindows; getScreenshot maps to screenshot; cua.getState maps to listApps; every other named operation uses the same method name. +Raw tools use { method, params }: include appHandle for app calls and windowId for window calls, and use snapshotId with elementId for element routes. The emit option is facade-only: emit is local output control, so raw callers must follow the raw operation schema for admitted params. getAccessibilityTree maps to getState; getWindow has no raw method; getWindows maps to listWindows; getScreenshot maps to screenshot; cua.getState maps to listApps; every other named operation uses the same method name. Only if the task cannot be completed without moving the user’s pointer, call await win.mouseMove(target, { useSystemPointer: true }). Prefer an advertised accessibility action, a click on a visible control, or browser tools when possible. The question interrupts the user. When approval is disabled an explicit move can act immediately. Pass true on every physical move, even after approval. Omitting the option or passing false uses the default hover route without prompting. Approval or refusal lasts until this session closes or DSH restarts. A refusal sends no input. If the user dismisses the request, ask again only if they request it. Wait for an outstanding request to finish. Subagents cannot request physical movement. diff --git a/snapshots/session/cua-native/workspace/native-helper.mjs b/snapshots/session/cua-native/workspace/native-helper.mjs index 771eec41e1..3a3e90065f 100755 --- a/snapshots/session/cua-native/workspace/native-helper.mjs +++ b/snapshots/session/cua-native/workspace/native-helper.mjs @@ -120,6 +120,7 @@ function createApplicationState(socket, record, application, captureBroker, conn const screenshotConfig = new URL('./fixture-screenshot.json', import.meta.url) const customScreenshot = existsSync(screenshotConfig) ? JSON.parse(readFileSync(screenshotConfig, 'utf8')) : undefined const snapshots = new Map() + const observedWindows = new Set() let acquired = false let generation = 0 let observationChanged = false @@ -161,12 +162,19 @@ function createApplicationState(socket, record, application, captureBroker, conn } } } if (method === 'listWindows') return windows() + if (params.windowId !== undefined && !windows().some(window => window.windowId === params.windowId)) refuse('WINDOW_UNAVAILABLE', 'Window is closed or belongs to another application') + const writableFailureFile = new URL('./fixture-ax-writability-failure', import.meta.url) + const writableFailure = existsSync(writableFailureFile) ? readFileSync(writableFailureFile, 'utf8') : undefined if (method === 'getState') { snapshots.delete(params.windowId ?? null) + if (params.windowId === undefined) observedWindows.clear() + else observedWindows.delete(params.windowId) + if (existsSync(new URL('./fixture-ax-failure', import.meta.url))) refuse('NATIVE_ERROR', 'Fixture accessibility read failed') if (observationChanged) { observationChanged = false refuse('STALE_SNAPSHOT', 'Accessibility tree changed during observation; read fresh getAccessibilityTree') } + for (const id of params.windowId === undefined ? windows().map(window => window.windowId) : [params.windowId]) observedWindows.add(id) generation++ const axless = params.windowId === 8 const snapshotId = `${scope}-snapshot-${generation}` @@ -182,7 +190,7 @@ function createApplicationState(socket, record, application, captureBroker, conn tree: `AX scope=${params.windowId === undefined ? 'application' : 'window'} offsets=UTF-16\n- [0] AXButton "Increment"\nAccessibility tree truncated (maxElements=1).`, elements: [], windows: windows(), truncated: true } return { appHandle: app.appHandle, windowId: params.windowId ?? null, - snapshotId: `${scope}-snapshot-${generation}`, tree: `AX scope=${params.windowId === undefined ? 'application' : 'window'} offsets=UTF-16${params.details ? ' bounds=global-points' : ''}\n- [0] AXButton "Increment"${params.details ? ' id="increment" actions=["AXPress"] bounds=[60,80,100,20]' : ''}\n- [1] AXTextField valueWritable=true selectionWritable=true value=${JSON.stringify(application.value)} placeholder="Fixture text"\n- AXStaticText "Count: ${application.count}"${application.hovering ? '\n- AXStaticText "Hover information"' : ''}\n- [2] AXMenuBarItem "File"${params.includeMenus ? '\n - [3] AXMenuItem "Fixture command"' : '\nMenu contents omitted; request includeMenus=true to expand.'}`, + snapshotId: `${scope}-snapshot-${generation}`, tree: `AX scope=${params.windowId === undefined ? 'application' : 'window'} offsets=UTF-16${params.details ? ' bounds=global-points' : ''}\n- [0] AXButton "Increment"${params.details ? ' id="increment" actions=["AXPress"] bounds=[60,80,100,20]' : ''}\n- [1] AXTextField valueWritable=${writableFailure === 'AXValue' ? 'unknown' : 'true'} selectionWritable=${writableFailure === 'AXSelectedTextRange' ? 'unknown' : 'true'} value=${JSON.stringify(application.value)} placeholder="Fixture text"\n- AXStaticText "Count: ${application.count}"${application.hovering ? '\n- AXStaticText "Hover information"' : ''}\n- [2] AXMenuBarItem "File"${params.includeMenus ? '\n - [3] AXMenuItem "Fixture command"' : '\nMenu contents omitted; request includeMenus=true to expand.'}`, elements: [{ elementId: '0', role: 'AXButton', title: 'Increment', value: null, description: null, identifier: null, bounds: [60, 80, 100, 20], actions: ['AXPress'], enabled: true }], focusedWindowId: window.windowId, windows: windows(), truncated: false } @@ -198,19 +206,27 @@ function createApplicationState(socket, record, application, captureBroker, conn socket.write(JSON.stringify({ type: 'diagnostic', requestId, stage: 'capture-image.begin', elapsedMs: 1 }) + '\ninvalid screenshot response\n') return } - if (method === 'screenshot') return customScreenshot - ? { pixelWidth: customScreenshot.pixelWidth, pixelHeight: customScreenshot.pixelHeight, bounds: customScreenshot.bounds, + if (method === 'screenshot') { + observedWindows.add(params.windowId) + return customScreenshot + ? { pixelWidth: customScreenshot.pixelWidth, pixelHeight: customScreenshot.pixelHeight, bounds: customScreenshot.bounds, windowId: params.windowId, mimeType: 'image/png', coordinateSpace: 'window-points', data: readFileSync(new URL(customScreenshot.file, import.meta.url)).toString('base64') } - : { windowId: params.windowId, mimeType: 'image/png', data: png, pixelWidth: 1, pixelHeight: 1, - bounds: params.windowId === panel.windowId ? panel.bounds : bounds, coordinateSpace: 'window-points' } + : { windowId: params.windowId, mimeType: 'image/png', data: png, pixelWidth: 1, pixelHeight: 1, + bounds: params.windowId === panel.windowId ? panel.bounds : bounds, coordinateSpace: 'window-points' } + } + if (['click', 'mouseMove', 'scroll', 'drag'].includes(method) && params.elementId === undefined + && !observedWindows.has(params.windowId)) refuse('STALE_SNAPSHOT', 'Window has no observation; read getAccessibilityTree or take a screenshot before coordinate input') + if ((method === 'setValue' && writableFailure === 'AXValue') || (method === 'selectText' && writableFailure === 'AXSelectedTextRange')) { + refuse('NATIVE_ERROR', `Accessibility read writability of ${writableFailure} failed with error -25200`) + } if (method === 'click' || method === 'setValue' || method === 'mouseMove') { const observed = params.snapshotId === undefined && params.elementId === undefined ? snapshots.get(params.windowId) ?? snapshots.get(null) : [...snapshots.values()].find(snapshot => snapshot.snapshotId === params.snapshotId) - if (!observed) refuse('STALE_SNAPSHOT', 'Read fresh state before acting') - if (params.windowId !== undefined && observed.windowId !== null && observed.windowId !== params.windowId) refuse('STALE_SNAPSHOT', `Snapshot belongs to window ${observed.windowId}; read getAccessibilityTree on window ${params.windowId}`) - if (observed.axless && params.elementId !== undefined) refuse('INVALID_ARGUMENT', 'Element id is absent from the selected snapshot') + if (!observed && (params.snapshotId !== undefined || params.elementId !== undefined)) refuse('STALE_SNAPSHOT', 'Read fresh state before acting') + if (params.windowId !== undefined && observed?.windowId != null && observed.windowId !== params.windowId) refuse('STALE_SNAPSHOT', `Snapshot belongs to window ${observed.windowId}; read getAccessibilityTree on window ${params.windowId}`) + if (observed?.axless && params.elementId !== undefined) refuse('INVALID_ARGUMENT', 'Element id is absent from the selected snapshot') let systemPointerMoved = false if (method === 'mouseMove') { const alternate = params.elementId === '0' || (params.x >= 20 && params.x <= 40) diff --git a/snapshots/session/cua-ptc/consent-fixture.mjs b/snapshots/session/cua-ptc/consent-fixture.mjs index 85e83bcf69..e93ba645f8 100644 --- a/snapshots/session/cua-ptc/consent-fixture.mjs +++ b/snapshots/session/cua-ptc/consent-fixture.mjs @@ -1,5 +1,5 @@ -/** Own a private native substitute for the PTC result replay. */ -import { chmod, copyFile, unlink } from 'node:fs/promises' +/** Own the bridge module and private native substitute; the replay owns its AX-failure markers through finally. */ +import { chmod, copyFile, unlink, writeFile } from 'node:fs/promises' import { join } from 'node:path' import { startFixtureHost } from '../cua-native/fixture-host.mjs' export const name = 'cua-ptc-fixture' @@ -7,6 +7,10 @@ export const inject = ['userQuestions'] /** Prepare the helper and retire its service before removing the private executable. */ export async function apply(ctx) { + const bridge = join(process.cwd(), 'cua-bridge.mjs') + const face = process.env.DSH_EXAMPLE_MODE === 'lib' ? 'lib/bridge.js' : 'src/bridge.ts' + await writeFile(bridge, `export { connectCua } from ${JSON.stringify(new URL('../../../packages/experimental/cua/' + face, import.meta.url).href)}\n`) + ctx.effect(() => () => unlink(bridge)) const helper = join(process.cwd(), 'native-helper.mjs') await copyFile(new URL('../cua-native/workspace/native-helper.mjs', import.meta.url), helper) await chmod(helper, 0o700) diff --git a/snapshots/session/cua-ptc/replay.override.json b/snapshots/session/cua-ptc/replay.override.json index a6ff126329..dbf87cd0b1 100644 --- a/snapshots/session/cua-ptc/replay.override.json +++ b/snapshots/session/cua-ptc/replay.override.json @@ -32,6 +32,72 @@ } ] }, + { + "kind": "chunks", + "chunks": [ + { + "type": "block-start", + "index": 0, + "blockType": "tool-call" + }, + { + "type": "block-end", + "index": 0, + "block": { + "type": "tool-call", + "id": "cua-ptc-ax-failure", + "name": "run_code", + "arguments": "{\"code\":\"const { readFile, writeFile, unlink } = await import('node:fs/promises');\\nconst { resolve } = await import('node:path');\\nconst { pathToFileURL } = await import('node:url');\\nconst { connectCua } = await import(pathToFileURL(resolve('cua-bridge.mjs')).href);\\nawait writeFile('fixture-ax-failure', '');\\ntry {\\n const cua = connectCua(() => tools.cua);\\n const app = await cua.getApp('test.fixture');\\n const windows = await app.getWindows({ emit: false });\\n const window = await app.getWindow(windows[0].windowId);\\n const failures = [];\\n for (const surface of [app, window]) {\\n try { await surface.getAccessibilityTree(); throw new Error('Expected AX failure'); }\\n catch (error) { if (!error.message.includes('Fixture accessibility read failed')) throw error; failures.push(error.message); }\\n }\\n let coordinateRefused = false;\\n try { await window.click([100, 100], { relative: true }); }\\n catch (error) { if (!error.message.includes('Window has no observation')) throw error; coordinateRefused = true; }\\n if (!coordinateRefused) throw new Error('Coordinate input accepted without observation');\\n const shot = await window.getScreenshot();\\n const coordinateDispatched = (await window.click([100, 100], { relative: true })).dispatched;\\n let elementRefused = false;\\n try { await window.invoke('0'); }\\n catch (error) { if (!error.message.includes('Read getAccessibilityTree')) throw error; elementRefused = true; }\\n if (!elementRefused) throw new Error('Element input accepted without an AX snapshot');\\n inspect({ axFailures: failures.length, screenshotWindow: window.windowId, pngBytes: (await readFile(shot.path)).byteLength, elementRefused, coordinateRefused, coordinateDispatched }, shot);\\n} finally { await unlink('fixture-ax-failure'); }\",\"description\":\"Capture through the object API despite application and window AX failures\"}" + } + }, + { + "type": "usage", + "usage": { + "inputTokens": 3, + "outputTokens": 3 + } + }, + { + "type": "finish", + "reason": { + "kind": "tool-calls" + } + } + ] + }, + { + "kind": "chunks", + "chunks": [ + { + "type": "block-start", + "index": 0, + "blockType": "tool-call" + }, + { + "type": "block-end", + "index": 0, + "block": { + "type": "tool-call", + "id": "cua-ptc-writability", + "name": "run_code", + "arguments": "{\"code\":\"const { writeFile, unlink } = await import('node:fs/promises');\\nconst { resolve } = await import('node:path');\\nconst { pathToFileURL } = await import('node:url');\\nconst { connectCua } = await import(pathToFileURL(resolve('cua-bridge.mjs')).href);\\nconst cua = connectCua(() => tools.cua);\\nconst app = await cua.getApp('test.fixture');\\nconst results = [];\\nfor (const attribute of ['AXValue', 'AXSelectedTextRange']) {\\n await writeFile('fixture-ax-writability-failure', attribute);\\n try {\\n const tree = await app.getAccessibilityTree({ emit: false });\\n const field = attribute === 'AXValue' ? 'valueWritable' : 'selectionWritable';\\n if (!tree.includes(field + '=unknown') || tree.includes(field + '=true')) throw new Error('Unknown writability was lost');\\n if (!tree.includes('Increment') || !tree.includes('Count:')) throw new Error('Other nodes were lost');\\n let writeRefused = false;\\n try {\\n if (attribute === 'AXValue') await app.setValue('1', 'must not be written');\\n else await app.selectText('1', 'Fixture');\\n } catch (error) {\\n if (!error.message.includes('writability of ' + attribute + ' failed with error -25200')) throw error;\\n writeRefused = true;\\n }\\n if (!writeRefused) throw new Error('Write accepted with unknown writability');\\n results.push({ attribute, unknown: true, otherNodesRetained: true, writeRefused });\\n } finally { await unlink('fixture-ax-writability-failure'); }\\n}\\ninspect(results);\",\"description\":\"Retain AX trees with unknown writability and reject writes\"}" + } + }, + { + "type": "usage", + "usage": { + "inputTokens": 3, + "outputTokens": 3 + } + }, + { + "type": "finish", + "reason": { + "kind": "tool-calls" + } + } + ] + }, { "kind": "chunks", "chunks": [ @@ -45,7 +111,7 @@ "index": 0, "block": { "type": "text", - "text": "Received native CUA values and one PNG screenshot through PTC. The PNG is 90 bytes with signature 89504e470d0a1a0a." + "text": "Received native CUA values and PNG screenshots through PTC. The object API captured a 90-byte PNG after two AX read failures and refused element input without an accessibility snapshot. Both writability probe failures left the tree readable with unknown fields and kept writes strict." } }, { diff --git a/snapshots/session/cua-ptc/session.v4.jsonl b/snapshots/session/cua-ptc/session.v4.jsonl index f78543f624..d196dbfe94 100644 --- a/snapshots/session/cua-ptc/session.v4.jsonl +++ b/snapshots/session/cua-ptc/session.v4.jsonl @@ -12,7 +12,7 @@ {"type":"request/header","data":{"header":{"config":{"provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"tools":"{{tools}}"},"reason":"initial"}} {"type":"request/context","data":{"provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"}} {"type":"session/title","data":{"title":"Use tools.cua from Node PTC","messageSeqs":[8],"source":{"kind":"fallback"}}} -{"type":"assistant/message","data":{"turn":1,"step":1,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-ptc-native-result","name":"run_code","arguments":"{\"code\":\"const { readFile } = await import('node:fs/promises');\\nconst app = await tools.cua({ method: 'getApp', params: { selector: 'test.fixture' } });\\nconst target = { appHandle: app.appHandle, windowId: 7 };\\nconst first = await tools.cua({ method: 'getState', params: { ...target, maxElements: 1 } });\\nconst state = first.truncated ? await tools.cua({ method: 'getState', params: { ...target, maxElements: 16 } }) : first;\\nconst shot = await tools.cua({ method: 'screenshot', params: { appHandle: app.appHandle, windowId: 7 } });\\nconst bytes = await readFile(shot.path);\\ninspect({ appHandle: app.appHandle, initiallyTruncated: first.truncated, recoveredTextField: !first.tree.includes('AXTextField') && state.tree.includes('AXTextField'), hasIncrement: state.tree.includes('Increment'), pngBytes: bytes.length, signature: bytes.subarray(0, 8).toString('hex'), imageWidth: shot.width, imageHeight: shot.height }, shot); return;\",\"description\":\"Read native CUA values and receive one screenshot\"}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:4}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1794218542463,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1794218542463,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-ptc-native-result","name":"run_code","arguments":"{\"code\":\"const { readFile } = await import('node:fs/promises');\\nconst app = await tools.cua({ method: 'getApp', params: { selector: 'test.fixture' } });\\nconst target = { appHandle: app.appHandle, windowId: 7 };\\nconst first = await tools.cua({ method: 'getState', params: { ...target, maxElements: 1 } });\\nconst state = first.truncated ? await tools.cua({ method: 'getState', params: { ...target, maxElements: 16 } }) : first;\\nconst shot = await tools.cua({ method: 'screenshot', params: { appHandle: app.appHandle, windowId: 7 } });\\nconst bytes = await readFile(shot.path);\\ninspect({ appHandle: app.appHandle, initiallyTruncated: first.truncated, recoveredTextField: !first.tree.includes('AXTextField') && state.tree.includes('AXTextField'), hasIncrement: state.tree.includes('Increment'), pngBytes: bytes.length, signature: bytes.subarray(0, 8).toString('hex'), imageWidth: shot.width, imageHeight: shot.height }, shot); return;\",\"description\":\"Read native CUA values and receive one screenshot\"}"}}},{"type":"chunk","time":1794218542463,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1794218542463,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} +{"type":"assistant/message","data":{"turn":1,"step":1,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-ptc-native-result","name":"run_code","arguments":"{\"code\":\"const { readFile } = await import('node:fs/promises');\\nconst app = await tools.cua({ method: 'getApp', params: { selector: 'test.fixture' } });\\nconst target = { appHandle: app.appHandle, windowId: 7 };\\nconst first = await tools.cua({ method: 'getState', params: { ...target, maxElements: 1 } });\\nconst state = first.truncated ? await tools.cua({ method: 'getState', params: { ...target, maxElements: 16 } }) : first;\\nconst shot = await tools.cua({ method: 'screenshot', params: { appHandle: app.appHandle, windowId: 7 } });\\nconst bytes = await readFile(shot.path);\\ninspect({ appHandle: app.appHandle, initiallyTruncated: first.truncated, recoveredTextField: !first.tree.includes('AXTextField') && state.tree.includes('AXTextField'), hasIncrement: state.tree.includes('Increment'), pngBytes: bytes.length, signature: bytes.subarray(0, 8).toString('hex'), imageWidth: shot.width, imageHeight: shot.height }, shot); return;\",\"description\":\"Read native CUA values and receive one screenshot\"}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:4}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1794226987185,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1794226987185,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-ptc-native-result","name":"run_code","arguments":"{\"code\":\"const { readFile } = await import('node:fs/promises');\\nconst app = await tools.cua({ method: 'getApp', params: { selector: 'test.fixture' } });\\nconst target = { appHandle: app.appHandle, windowId: 7 };\\nconst first = await tools.cua({ method: 'getState', params: { ...target, maxElements: 1 } });\\nconst state = first.truncated ? await tools.cua({ method: 'getState', params: { ...target, maxElements: 16 } }) : first;\\nconst shot = await tools.cua({ method: 'screenshot', params: { appHandle: app.appHandle, windowId: 7 } });\\nconst bytes = await readFile(shot.path);\\ninspect({ appHandle: app.appHandle, initiallyTruncated: first.truncated, recoveredTextField: !first.tree.includes('AXTextField') && state.tree.includes('AXTextField'), hasIncrement: state.tree.includes('Increment'), pngBytes: bytes.length, signature: bytes.subarray(0, 8).toString('hex'), imageWidth: shot.width, imageHeight: shot.height }, shot); return;\",\"description\":\"Read native CUA values and receive one screenshot\"}"}}},{"type":"chunk","time":1794226987185,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1794226987185,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} {"type":"tool/call","data":{"turn":1,"step":1,"callId":"cua-ptc-native-result","name":"run_code","arguments":"{\"code\":\"const { readFile } = await import('node:fs/promises');\\nconst app = await tools.cua({ method: 'getApp', params: { selector: 'test.fixture' } });\\nconst target = { appHandle: app.appHandle, windowId: 7 };\\nconst first = await tools.cua({ method: 'getState', params: { ...target, maxElements: 1 } });\\nconst state = first.truncated ? await tools.cua({ method: 'getState', params: { ...target, maxElements: 16 } }) : first;\\nconst shot = await tools.cua({ method: 'screenshot', params: { appHandle: app.appHandle, windowId: 7 } });\\nconst bytes = await readFile(shot.path);\\ninspect({ appHandle: app.appHandle, initiallyTruncated: first.truncated, recoveredTextField: !first.tree.includes('AXTextField') && state.tree.includes('AXTextField'), hasIncrement: state.tree.includes('Increment'), pngBytes: bytes.length, signature: bytes.subarray(0, 8).toString('hex'), imageWidth: shot.width, imageHeight: shot.height }, shot); return;\",\"description\":\"Read native CUA values and receive one screenshot\"}"}} {"type":"tool/ptc-dispatch-start","data":{"rootCallId":"cua-ptc-native-result","parentCallId":"cua-ptc-native-result","subCallId":"cua-ptc-native-result:ptc:1","name":"cua","arguments":{"method":"getApp","params":{"selector":"test.fixture"}}}} {"type":"tool/ptc-dispatch","data":{"rootCallId":"cua-ptc-native-result","parentCallId":"cua-ptc-native-result","subCallId":"cua-ptc-native-result:ptc:1","name":"cua","arguments":{"method":"getApp","params":{"selector":"test.fixture"}},"isError":false,"content":[{"type":"text","text":"{\"appHandle\":\"fixture-app\",\"name\":\"Fixture\",\"bundleId\":\"test.fixture\",\"running\":true}"}]}} @@ -25,6 +25,44 @@ {"type":"tool/result","data":{"turn":1,"step":1,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-ptc-native-result"},"toolCallId":"cua-ptc-native-result","content":[{"type":"text","text":"{\n appHandle: 'fixture-app',\n initiallyTruncated: true,\n recoveredTextField: true,\n hasIncrement: true,\n pngBytes: 90,\n signature: '89504e470d0a1a0a',\n imageWidth: 1,\n imageHeight: 1\n}"},{"type":"image","attachment":{"attachmentId":"sha256:999f1d1527ee7e79266f16add5430fff76b1225d742464a5b1ff1f02971bb8ee","mediaType":"image/png","width":1,"height":1,"bytes":90}}],"isError":false,"id":"{{message:5}}"}},"sourceEventSeqs":[14],"surfaceOp":"append"} {"type":"step/end","data":{"turn":1,"step":1}} {"type":"step/start","data":{"turn":1,"step":2}} -{"type":"assistant/message","data":{"turn":1,"step":2,"message":{"role":"assistant","content":[{"type":"text","text":"Received native CUA values and one PNG screenshot through PTC. The PNG is 90 bytes with signature 89504e470d0a1a0a."}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:6}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1794218543290,"chunk":{"type":"block-start","index":0,"blockType":"text"}},{"type":"chunk","time":1794218543290,"chunk":{"type":"block-end","index":0,"block":{"type":"text","text":"Received native CUA values and one PNG screenshot through PTC. The PNG is 90 bytes with signature 89504e470d0a1a0a."}}},{"type":"chunk","time":1794218543290,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1794218543290,"chunk":{"type":"finish","reason":{"kind":"stop"}}}]},"surfaceOp":"append"} +{"type":"assistant/message","data":{"turn":1,"step":2,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-ptc-ax-failure","name":"run_code","arguments":"{\"code\":\"const { readFile, writeFile, unlink } = await import('node:fs/promises');\\nconst { resolve } = await import('node:path');\\nconst { pathToFileURL } = await import('node:url');\\nconst { connectCua } = await import(pathToFileURL(resolve('cua-bridge.mjs')).href);\\nawait writeFile('fixture-ax-failure', '');\\ntry {\\n const cua = connectCua(() => tools.cua);\\n const app = await cua.getApp('test.fixture');\\n const windows = await app.getWindows({ emit: false });\\n const window = await app.getWindow(windows[0].windowId);\\n const failures = [];\\n for (const surface of [app, window]) {\\n try { await surface.getAccessibilityTree(); throw new Error('Expected AX failure'); }\\n catch (error) { if (!error.message.includes('Fixture accessibility read failed')) throw error; failures.push(error.message); }\\n }\\n let coordinateRefused = false;\\n try { await window.click([100, 100], { relative: true }); }\\n catch (error) { if (!error.message.includes('Window has no observation')) throw error; coordinateRefused = true; }\\n if (!coordinateRefused) throw new Error('Coordinate input accepted without observation');\\n const shot = await window.getScreenshot();\\n const coordinateDispatched = (await window.click([100, 100], { relative: true })).dispatched;\\n let elementRefused = false;\\n try { await window.invoke('0'); }\\n catch (error) { if (!error.message.includes('Read getAccessibilityTree')) throw error; elementRefused = true; }\\n if (!elementRefused) throw new Error('Element input accepted without an AX snapshot');\\n inspect({ axFailures: failures.length, screenshotWindow: window.windowId, pngBytes: (await readFile(shot.path)).byteLength, elementRefused, coordinateRefused, coordinateDispatched }, shot);\\n} finally { await unlink('fixture-ax-failure'); }\",\"description\":\"Capture through the object API despite application and window AX failures\"}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:6}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1794226987900,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1794226987900,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-ptc-ax-failure","name":"run_code","arguments":"{\"code\":\"const { readFile, writeFile, unlink } = await import('node:fs/promises');\\nconst { resolve } = await import('node:path');\\nconst { pathToFileURL } = await import('node:url');\\nconst { connectCua } = await import(pathToFileURL(resolve('cua-bridge.mjs')).href);\\nawait writeFile('fixture-ax-failure', '');\\ntry {\\n const cua = connectCua(() => tools.cua);\\n const app = await cua.getApp('test.fixture');\\n const windows = await app.getWindows({ emit: false });\\n const window = await app.getWindow(windows[0].windowId);\\n const failures = [];\\n for (const surface of [app, window]) {\\n try { await surface.getAccessibilityTree(); throw new Error('Expected AX failure'); }\\n catch (error) { if (!error.message.includes('Fixture accessibility read failed')) throw error; failures.push(error.message); }\\n }\\n let coordinateRefused = false;\\n try { await window.click([100, 100], { relative: true }); }\\n catch (error) { if (!error.message.includes('Window has no observation')) throw error; coordinateRefused = true; }\\n if (!coordinateRefused) throw new Error('Coordinate input accepted without observation');\\n const shot = await window.getScreenshot();\\n const coordinateDispatched = (await window.click([100, 100], { relative: true })).dispatched;\\n let elementRefused = false;\\n try { await window.invoke('0'); }\\n catch (error) { if (!error.message.includes('Read getAccessibilityTree')) throw error; elementRefused = true; }\\n if (!elementRefused) throw new Error('Element input accepted without an AX snapshot');\\n inspect({ axFailures: failures.length, screenshotWindow: window.windowId, pngBytes: (await readFile(shot.path)).byteLength, elementRefused, coordinateRefused, coordinateDispatched }, shot);\\n} finally { await unlink('fixture-ax-failure'); }\",\"description\":\"Capture through the object API despite application and window AX failures\"}"}}},{"type":"chunk","time":1794226987901,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1794226987901,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} +{"type":"tool/call","data":{"turn":1,"step":2,"callId":"cua-ptc-ax-failure","name":"run_code","arguments":"{\"code\":\"const { readFile, writeFile, unlink } = await import('node:fs/promises');\\nconst { resolve } = await import('node:path');\\nconst { pathToFileURL } = await import('node:url');\\nconst { connectCua } = await import(pathToFileURL(resolve('cua-bridge.mjs')).href);\\nawait writeFile('fixture-ax-failure', '');\\ntry {\\n const cua = connectCua(() => tools.cua);\\n const app = await cua.getApp('test.fixture');\\n const windows = await app.getWindows({ emit: false });\\n const window = await app.getWindow(windows[0].windowId);\\n const failures = [];\\n for (const surface of [app, window]) {\\n try { await surface.getAccessibilityTree(); throw new Error('Expected AX failure'); }\\n catch (error) { if (!error.message.includes('Fixture accessibility read failed')) throw error; failures.push(error.message); }\\n }\\n let coordinateRefused = false;\\n try { await window.click([100, 100], { relative: true }); }\\n catch (error) { if (!error.message.includes('Window has no observation')) throw error; coordinateRefused = true; }\\n if (!coordinateRefused) throw new Error('Coordinate input accepted without observation');\\n const shot = await window.getScreenshot();\\n const coordinateDispatched = (await window.click([100, 100], { relative: true })).dispatched;\\n let elementRefused = false;\\n try { await window.invoke('0'); }\\n catch (error) { if (!error.message.includes('Read getAccessibilityTree')) throw error; elementRefused = true; }\\n if (!elementRefused) throw new Error('Element input accepted without an AX snapshot');\\n inspect({ axFailures: failures.length, screenshotWindow: window.windowId, pngBytes: (await readFile(shot.path)).byteLength, elementRefused, coordinateRefused, coordinateDispatched }, shot);\\n} finally { await unlink('fixture-ax-failure'); }\",\"description\":\"Capture through the object API despite application and window AX failures\"}"}} +{"type":"tool/ptc-dispatch-start","data":{"rootCallId":"cua-ptc-ax-failure","parentCallId":"cua-ptc-ax-failure","subCallId":"cua-ptc-ax-failure:ptc:1","name":"cua","arguments":{"method":"getCapabilities","params":{}}}} +{"type":"tool/ptc-dispatch","data":{"rootCallId":"cua-ptc-ax-failure","parentCallId":"cua-ptc-ax-failure","subCallId":"cua-ptc-ax-failure:ptc:1","name":"cua","arguments":{"method":"getCapabilities","params":{}},"isError":false,"content":[{"type":"text","text":"{\"sessionLocked\":false,\"permissions\":{\"accessibility\":true,\"screenRecording\":true},\"platform\":\"macos\",\"systemPointer\":true,\"methods\":[\"activate\",\"click\",\"drag\",\"findApps\",\"getApp\",\"getCapabilities\",\"getState\",\"invoke\",\"listActions\",\"listApps\",\"listWindows\",\"mouseMove\",\"paste\",\"performAction\",\"pressKey\",\"screenshot\",\"scroll\",\"selectText\",\"sendKeyEvent\",\"setValue\",\"show\",\"typeText\"],\"appLevelAccessibility\":true,\"keyboard\":[\"app\"],\"browser\":false,\"coordinateSpace\":\"screenshot-pixels\"}"}]}} +{"type":"tool/ptc-dispatch-start","data":{"rootCallId":"cua-ptc-ax-failure","parentCallId":"cua-ptc-ax-failure","subCallId":"cua-ptc-ax-failure:ptc:2","name":"cua","arguments":{"method":"getApp","params":{"selector":"test.fixture"}}}} +{"type":"tool/ptc-dispatch","data":{"rootCallId":"cua-ptc-ax-failure","parentCallId":"cua-ptc-ax-failure","subCallId":"cua-ptc-ax-failure:ptc:2","name":"cua","arguments":{"method":"getApp","params":{"selector":"test.fixture"}},"isError":false,"content":[{"type":"text","text":"{\"appHandle\":\"fixture-app\",\"name\":\"Fixture\",\"bundleId\":\"test.fixture\",\"running\":true}"}]}} +{"type":"tool/ptc-dispatch-start","data":{"rootCallId":"cua-ptc-ax-failure","parentCallId":"cua-ptc-ax-failure","subCallId":"cua-ptc-ax-failure:ptc:3","name":"cua","arguments":{"method":"listWindows","params":{"appHandle":"fixture-app"}}}} +{"type":"tool/ptc-dispatch","data":{"rootCallId":"cua-ptc-ax-failure","parentCallId":"cua-ptc-ax-failure","subCallId":"cua-ptc-ax-failure:ptc:3","name":"cua","arguments":{"method":"listWindows","params":{"appHandle":"fixture-app"}},"isError":false,"content":[{"type":"text","text":"[{\"windowId\":7,\"appName\":\"Fixture\",\"title\":\"CUA fixture\",\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"isOnScreen\":true,\"onCurrentSpace\":true},{\"windowId\":8,\"appName\":\"Fixture\",\"title\":\"AXless canvas\",\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"isOnScreen\":true,\"onCurrentSpace\":true},{\"windowId\":9,\"appName\":\"Fixture\",\"title\":\"Transformed window\",\"bounds\":{\"x\":16,\"y\":512,\"width\":131,\"height\":145},\"isOnScreen\":true,\"onCurrentSpace\":true}]"}]}} +{"type":"tool/ptc-dispatch-start","data":{"rootCallId":"cua-ptc-ax-failure","parentCallId":"cua-ptc-ax-failure","subCallId":"cua-ptc-ax-failure:ptc:4","name":"cua","arguments":{"method":"getState","params":{"appHandle":"fixture-app"}}}} +{"type":"tool/ptc-dispatch","data":{"rootCallId":"cua-ptc-ax-failure","parentCallId":"cua-ptc-ax-failure","subCallId":"cua-ptc-ax-failure:ptc:4","name":"cua","arguments":{"method":"getState","params":{"appHandle":"fixture-app"}},"isError":true,"content":[{"type":"text","text":"Error: NATIVE_ERROR: Fixture accessibility read failed"}]}} +{"type":"tool/ptc-dispatch-start","data":{"rootCallId":"cua-ptc-ax-failure","parentCallId":"cua-ptc-ax-failure","subCallId":"cua-ptc-ax-failure:ptc:5","name":"cua","arguments":{"method":"getState","params":{"appHandle":"fixture-app","windowId":7}}}} +{"type":"tool/ptc-dispatch","data":{"rootCallId":"cua-ptc-ax-failure","parentCallId":"cua-ptc-ax-failure","subCallId":"cua-ptc-ax-failure:ptc:5","name":"cua","arguments":{"method":"getState","params":{"appHandle":"fixture-app","windowId":7}},"isError":true,"content":[{"type":"text","text":"Error: NATIVE_ERROR: Fixture accessibility read failed"}]}} +{"type":"tool/ptc-dispatch-start","data":{"rootCallId":"cua-ptc-ax-failure","parentCallId":"cua-ptc-ax-failure","subCallId":"cua-ptc-ax-failure:ptc:6","name":"cua","arguments":{"method":"click","params":{"appHandle":"fixture-app","windowId":7,"x":100,"y":100,"relative":true,"mouseButton":"left","clickCount":1}}}} +{"type":"tool/ptc-dispatch","data":{"rootCallId":"cua-ptc-ax-failure","parentCallId":"cua-ptc-ax-failure","subCallId":"cua-ptc-ax-failure:ptc:6","name":"cua","arguments":{"method":"click","params":{"appHandle":"fixture-app","windowId":7,"x":100,"y":100,"relative":true,"mouseButton":"left","clickCount":1}},"isError":true,"content":[{"type":"text","text":"Error: STALE_SNAPSHOT: Window has no observation; read getAccessibilityTree or take a screenshot before coordinate input"}]}} +{"type":"tool/ptc-dispatch-start","data":{"rootCallId":"cua-ptc-ax-failure","parentCallId":"cua-ptc-ax-failure","subCallId":"cua-ptc-ax-failure:ptc:7","name":"cua","arguments":{"method":"screenshot","params":{"appHandle":"fixture-app","windowId":7}}}} +{"type":"tool/ptc-dispatch","data":{"rootCallId":"cua-ptc-ax-failure","parentCallId":"cua-ptc-ax-failure","subCallId":"cua-ptc-ax-failure:ptc:7","name":"cua","arguments":{"method":"screenshot","params":{"appHandle":"fixture-app","windowId":7}},"isError":false,"content":[{"type":"text","text":"{\"pixelWidth\":1,\"pixelHeight\":1,\"windowId\":7,\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"coordinateSpace\":\"screenshot-pixels\"}"},{"type":"image","attachment":{"attachmentId":"sha256:999f1d1527ee7e79266f16add5430fff76b1225d742464a5b1ff1f02971bb8ee","mediaType":"image/png","width":1,"height":1,"bytes":90}}]}} +{"type":"tool/ptc-dispatch-start","data":{"rootCallId":"cua-ptc-ax-failure","parentCallId":"cua-ptc-ax-failure","subCallId":"cua-ptc-ax-failure:ptc:8","name":"cua","arguments":{"method":"click","params":{"appHandle":"fixture-app","windowId":7,"x":100,"y":100,"relative":true,"mouseButton":"left","clickCount":1}}}} +{"type":"tool/ptc-dispatch","data":{"rootCallId":"cua-ptc-ax-failure","parentCallId":"cua-ptc-ax-failure","subCallId":"cua-ptc-ax-failure:ptc:8","name":"cua","arguments":{"method":"click","params":{"appHandle":"fixture-app","windowId":7,"x":100,"y":100,"relative":true,"mouseButton":"left","clickCount":1}},"isError":false,"content":[{"type":"text","text":"{\"dispatched\":true}"}]}} +{"type":"tool/result","data":{"turn":1,"step":2,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-ptc-ax-failure"},"toolCallId":"cua-ptc-ax-failure","content":[{"type":"text","text":"{\n axFailures: 2,\n screenshotWindow: 7,\n pngBytes: 90,\n elementRefused: true,\n coordinateRefused: true,\n coordinateDispatched: true\n}"},{"type":"image","attachment":{"attachmentId":"sha256:999f1d1527ee7e79266f16add5430fff76b1225d742464a5b1ff1f02971bb8ee","mediaType":"image/png","width":1,"height":1,"bytes":90}}],"isError":false,"id":"{{message:7}}"}},"sourceEventSeqs":[27],"surfaceOp":"append"} {"type":"step/end","data":{"turn":1,"step":2}} +{"type":"step/start","data":{"turn":1,"step":3}} +{"type":"assistant/message","data":{"turn":1,"step":3,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-ptc-writability","name":"run_code","arguments":"{\"code\":\"const { writeFile, unlink } = await import('node:fs/promises');\\nconst { resolve } = await import('node:path');\\nconst { pathToFileURL } = await import('node:url');\\nconst { connectCua } = await import(pathToFileURL(resolve('cua-bridge.mjs')).href);\\nconst cua = connectCua(() => tools.cua);\\nconst app = await cua.getApp('test.fixture');\\nconst results = [];\\nfor (const attribute of ['AXValue', 'AXSelectedTextRange']) {\\n await writeFile('fixture-ax-writability-failure', attribute);\\n try {\\n const tree = await app.getAccessibilityTree({ emit: false });\\n const field = attribute === 'AXValue' ? 'valueWritable' : 'selectionWritable';\\n if (!tree.includes(field + '=unknown') || tree.includes(field + '=true')) throw new Error('Unknown writability was lost');\\n if (!tree.includes('Increment') || !tree.includes('Count:')) throw new Error('Other nodes were lost');\\n let writeRefused = false;\\n try {\\n if (attribute === 'AXValue') await app.setValue('1', 'must not be written');\\n else await app.selectText('1', 'Fixture');\\n } catch (error) {\\n if (!error.message.includes('writability of ' + attribute + ' failed with error -25200')) throw error;\\n writeRefused = true;\\n }\\n if (!writeRefused) throw new Error('Write accepted with unknown writability');\\n results.push({ attribute, unknown: true, otherNodesRetained: true, writeRefused });\\n } finally { await unlink('fixture-ax-writability-failure'); }\\n}\\ninspect(results);\",\"description\":\"Retain AX trees with unknown writability and reject writes\"}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:8}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1794226988058,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1794226988058,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-ptc-writability","name":"run_code","arguments":"{\"code\":\"const { writeFile, unlink } = await import('node:fs/promises');\\nconst { resolve } = await import('node:path');\\nconst { pathToFileURL } = await import('node:url');\\nconst { connectCua } = await import(pathToFileURL(resolve('cua-bridge.mjs')).href);\\nconst cua = connectCua(() => tools.cua);\\nconst app = await cua.getApp('test.fixture');\\nconst results = [];\\nfor (const attribute of ['AXValue', 'AXSelectedTextRange']) {\\n await writeFile('fixture-ax-writability-failure', attribute);\\n try {\\n const tree = await app.getAccessibilityTree({ emit: false });\\n const field = attribute === 'AXValue' ? 'valueWritable' : 'selectionWritable';\\n if (!tree.includes(field + '=unknown') || tree.includes(field + '=true')) throw new Error('Unknown writability was lost');\\n if (!tree.includes('Increment') || !tree.includes('Count:')) throw new Error('Other nodes were lost');\\n let writeRefused = false;\\n try {\\n if (attribute === 'AXValue') await app.setValue('1', 'must not be written');\\n else await app.selectText('1', 'Fixture');\\n } catch (error) {\\n if (!error.message.includes('writability of ' + attribute + ' failed with error -25200')) throw error;\\n writeRefused = true;\\n }\\n if (!writeRefused) throw new Error('Write accepted with unknown writability');\\n results.push({ attribute, unknown: true, otherNodesRetained: true, writeRefused });\\n } finally { await unlink('fixture-ax-writability-failure'); }\\n}\\ninspect(results);\",\"description\":\"Retain AX trees with unknown writability and reject writes\"}"}}},{"type":"chunk","time":1794226988058,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1794226988058,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} +{"type":"tool/call","data":{"turn":1,"step":3,"callId":"cua-ptc-writability","name":"run_code","arguments":"{\"code\":\"const { writeFile, unlink } = await import('node:fs/promises');\\nconst { resolve } = await import('node:path');\\nconst { pathToFileURL } = await import('node:url');\\nconst { connectCua } = await import(pathToFileURL(resolve('cua-bridge.mjs')).href);\\nconst cua = connectCua(() => tools.cua);\\nconst app = await cua.getApp('test.fixture');\\nconst results = [];\\nfor (const attribute of ['AXValue', 'AXSelectedTextRange']) {\\n await writeFile('fixture-ax-writability-failure', attribute);\\n try {\\n const tree = await app.getAccessibilityTree({ emit: false });\\n const field = attribute === 'AXValue' ? 'valueWritable' : 'selectionWritable';\\n if (!tree.includes(field + '=unknown') || tree.includes(field + '=true')) throw new Error('Unknown writability was lost');\\n if (!tree.includes('Increment') || !tree.includes('Count:')) throw new Error('Other nodes were lost');\\n let writeRefused = false;\\n try {\\n if (attribute === 'AXValue') await app.setValue('1', 'must not be written');\\n else await app.selectText('1', 'Fixture');\\n } catch (error) {\\n if (!error.message.includes('writability of ' + attribute + ' failed with error -25200')) throw error;\\n writeRefused = true;\\n }\\n if (!writeRefused) throw new Error('Write accepted with unknown writability');\\n results.push({ attribute, unknown: true, otherNodesRetained: true, writeRefused });\\n } finally { await unlink('fixture-ax-writability-failure'); }\\n}\\ninspect(results);\",\"description\":\"Retain AX trees with unknown writability and reject writes\"}"}} +{"type":"tool/ptc-dispatch-start","data":{"rootCallId":"cua-ptc-writability","parentCallId":"cua-ptc-writability","subCallId":"cua-ptc-writability:ptc:1","name":"cua","arguments":{"method":"getCapabilities","params":{}}}} +{"type":"tool/ptc-dispatch","data":{"rootCallId":"cua-ptc-writability","parentCallId":"cua-ptc-writability","subCallId":"cua-ptc-writability:ptc:1","name":"cua","arguments":{"method":"getCapabilities","params":{}},"isError":false,"content":[{"type":"text","text":"{\"sessionLocked\":false,\"permissions\":{\"accessibility\":true,\"screenRecording\":true},\"platform\":\"macos\",\"systemPointer\":true,\"methods\":[\"activate\",\"click\",\"drag\",\"findApps\",\"getApp\",\"getCapabilities\",\"getState\",\"invoke\",\"listActions\",\"listApps\",\"listWindows\",\"mouseMove\",\"paste\",\"performAction\",\"pressKey\",\"screenshot\",\"scroll\",\"selectText\",\"sendKeyEvent\",\"setValue\",\"show\",\"typeText\"],\"appLevelAccessibility\":true,\"keyboard\":[\"app\"],\"browser\":false,\"coordinateSpace\":\"screenshot-pixels\"}"}]}} +{"type":"tool/ptc-dispatch-start","data":{"rootCallId":"cua-ptc-writability","parentCallId":"cua-ptc-writability","subCallId":"cua-ptc-writability:ptc:2","name":"cua","arguments":{"method":"getApp","params":{"selector":"test.fixture"}}}} +{"type":"tool/ptc-dispatch","data":{"rootCallId":"cua-ptc-writability","parentCallId":"cua-ptc-writability","subCallId":"cua-ptc-writability:ptc:2","name":"cua","arguments":{"method":"getApp","params":{"selector":"test.fixture"}},"isError":false,"content":[{"type":"text","text":"{\"appHandle\":\"fixture-app\",\"name\":\"Fixture\",\"bundleId\":\"test.fixture\",\"running\":true}"}]}} +{"type":"tool/ptc-dispatch-start","data":{"rootCallId":"cua-ptc-writability","parentCallId":"cua-ptc-writability","subCallId":"cua-ptc-writability:ptc:3","name":"cua","arguments":{"method":"getState","params":{"appHandle":"fixture-app"}}}} +{"type":"tool/ptc-dispatch","data":{"rootCallId":"cua-ptc-writability","parentCallId":"cua-ptc-writability","subCallId":"cua-ptc-writability:ptc:3","name":"cua","arguments":{"method":"getState","params":{"appHandle":"fixture-app"}},"isError":false,"content":[{"type":"text","text":"{\"appHandle\":\"fixture-app\",\"windowId\":null,\"focusedWindowId\":7,\"snapshotId\":\"fixture-snapshot-3\",\"tree\":\"AX scope=application offsets=UTF-16\\n- [0] AXButton \\\"Increment\\\"\\n- [1] AXTextField valueWritable=unknown selectionWritable=true value=\\\"\\\" placeholder=\\\"Fixture text\\\"\\n- AXStaticText \\\"Count: 1\\\"\\n- [2] AXMenuBarItem \\\"File\\\"\\nMenu contents omitted; request includeMenus=true to expand.\",\"windows\":[{\"windowId\":7,\"appName\":\"Fixture\",\"title\":\"CUA fixture\",\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"isOnScreen\":true,\"onCurrentSpace\":true},{\"windowId\":8,\"appName\":\"Fixture\",\"title\":\"AXless canvas\",\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"isOnScreen\":true,\"onCurrentSpace\":true},{\"windowId\":9,\"appName\":\"Fixture\",\"title\":\"Transformed window\",\"bounds\":{\"x\":16,\"y\":512,\"width\":131,\"height\":145},\"isOnScreen\":true,\"onCurrentSpace\":true}],\"truncated\":false}"}]}} +{"type":"tool/ptc-dispatch-start","data":{"rootCallId":"cua-ptc-writability","parentCallId":"cua-ptc-writability","subCallId":"cua-ptc-writability:ptc:4","name":"cua","arguments":{"method":"setValue","params":{"appHandle":"fixture-app","snapshotId":"fixture-snapshot-3","elementId":"1","value":"must not be written"}}}} +{"type":"tool/ptc-dispatch","data":{"rootCallId":"cua-ptc-writability","parentCallId":"cua-ptc-writability","subCallId":"cua-ptc-writability:ptc:4","name":"cua","arguments":{"method":"setValue","params":{"appHandle":"fixture-app","snapshotId":"fixture-snapshot-3","elementId":"1","value":"must not be written"}},"isError":true,"content":[{"type":"text","text":"Error: NATIVE_ERROR: Accessibility read writability of AXValue failed with error -25200"}]}} +{"type":"tool/ptc-dispatch-start","data":{"rootCallId":"cua-ptc-writability","parentCallId":"cua-ptc-writability","subCallId":"cua-ptc-writability:ptc:5","name":"cua","arguments":{"method":"getState","params":{"appHandle":"fixture-app"}}}} +{"type":"tool/ptc-dispatch","data":{"rootCallId":"cua-ptc-writability","parentCallId":"cua-ptc-writability","subCallId":"cua-ptc-writability:ptc:5","name":"cua","arguments":{"method":"getState","params":{"appHandle":"fixture-app"}},"isError":false,"content":[{"type":"text","text":"{\"appHandle\":\"fixture-app\",\"windowId\":null,\"focusedWindowId\":7,\"snapshotId\":\"fixture-snapshot-4\",\"tree\":\"AX scope=application offsets=UTF-16\\n- [0] AXButton \\\"Increment\\\"\\n- [1] AXTextField valueWritable=true selectionWritable=unknown value=\\\"\\\" placeholder=\\\"Fixture text\\\"\\n- AXStaticText \\\"Count: 1\\\"\\n- [2] AXMenuBarItem \\\"File\\\"\\nMenu contents omitted; request includeMenus=true to expand.\",\"windows\":[{\"windowId\":7,\"appName\":\"Fixture\",\"title\":\"CUA fixture\",\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"isOnScreen\":true,\"onCurrentSpace\":true},{\"windowId\":8,\"appName\":\"Fixture\",\"title\":\"AXless canvas\",\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"isOnScreen\":true,\"onCurrentSpace\":true},{\"windowId\":9,\"appName\":\"Fixture\",\"title\":\"Transformed window\",\"bounds\":{\"x\":16,\"y\":512,\"width\":131,\"height\":145},\"isOnScreen\":true,\"onCurrentSpace\":true}],\"truncated\":false}"}]}} +{"type":"tool/ptc-dispatch-start","data":{"rootCallId":"cua-ptc-writability","parentCallId":"cua-ptc-writability","subCallId":"cua-ptc-writability:ptc:6","name":"cua","arguments":{"method":"selectText","params":{"appHandle":"fixture-app","snapshotId":"fixture-snapshot-4","elementId":"1","text":"Fixture"}}}} +{"type":"tool/ptc-dispatch","data":{"rootCallId":"cua-ptc-writability","parentCallId":"cua-ptc-writability","subCallId":"cua-ptc-writability:ptc:6","name":"cua","arguments":{"method":"selectText","params":{"appHandle":"fixture-app","snapshotId":"fixture-snapshot-4","elementId":"1","text":"Fixture"}},"isError":true,"content":[{"type":"text","text":"Error: NATIVE_ERROR: Accessibility read writability of AXSelectedTextRange failed with error -25200"}]}} +{"type":"tool/result","data":{"turn":1,"step":3,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-ptc-writability"},"toolCallId":"cua-ptc-writability","content":[{"type":"text","text":"[\n {\n attribute: 'AXValue',\n unknown: true,\n otherNodesRetained: true,\n writeRefused: true\n },\n {\n attribute: 'AXSelectedTextRange',\n unknown: true,\n otherNodesRetained: true,\n writeRefused: true\n }\n]"}],"isError":false,"id":"{{message:9}}"}},"sourceEventSeqs":[48],"surfaceOp":"append"} +{"type":"step/end","data":{"turn":1,"step":3}} +{"type":"step/start","data":{"turn":1,"step":4}} +{"type":"assistant/message","data":{"turn":1,"step":4,"message":{"role":"assistant","content":[{"type":"text","text":"Received native CUA values and PNG screenshots through PTC. The object API captured a 90-byte PNG after two AX read failures and refused element input without an accessibility snapshot. Both writability probe failures left the tree readable with unknown fields and kept writes strict."}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:10}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1794226988201,"chunk":{"type":"block-start","index":0,"blockType":"text"}},{"type":"chunk","time":1794226988201,"chunk":{"type":"block-end","index":0,"block":{"type":"text","text":"Received native CUA values and PNG screenshots through PTC. The object API captured a 90-byte PNG after two AX read failures and refused element input without an accessibility snapshot. Both writability probe failures left the tree readable with unknown fields and kept writes strict."}}},{"type":"chunk","time":1794226988201,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1794226988201,"chunk":{"type":"finish","reason":{"kind":"stop"}}}]},"surfaceOp":"append"} +{"type":"step/end","data":{"turn":1,"step":4}} {"type":"turn/end","data":{"turn":1,"reason":{"kind":"completed"}}} diff --git a/snapshots/session/cua-ptc/system-prompt.expected.md b/snapshots/session/cua-ptc/system-prompt.expected.md index d0be91f90e..4e79b4d334 100644 --- a/snapshots/session/cua-ptc/system-prompt.expected.md +++ b/snapshots/session/cua-ptc/system-prompt.expected.md @@ -48,9 +48,9 @@ Setup and discovery: - cua.listApps(options?: { emit?: boolean }) -> { id: string, displayName: string, isRunning: true, pid: number }[] — enumerate running applications. - cua.findApps(query: string, options?: { emit?: boolean }) -> ({ id: string, displayName: string, path: string } & ({ isRunning: true, pid: number } | { isRunning: false }))[] — search installed applications, including stopped matches; only running matches carry pid. - cua.getState(options?: { emit?: boolean }) -> { apps: { id: string, displayName: string, isRunning: true, pid: number }[], browsers: [] } — summarize running applications; this is not an accessibility observation. -- cua.getApp(selector: string) -> App — select an application by name, bundle ID, absolute path, or running PID string, obtain consent, launch in the background if needed, and read initial AX state silently; it does not activate the app. +- cua.getApp(selector: string) -> App — select an application by name, bundle ID, absolute path, or running PID string, obtain consent, launch in the background if needed, and return without reading AX or activating the app. - app.getWindows(options?: { emit?: boolean }) -> { windowId: number, appName: string, title: string, bounds: { x: number, y: number, width: number, height: number }, isOnScreen: boolean, onCurrentSpace: boolean | null }[] — enumerate windows with global native bounds; null onCurrentSpace means unknown Space membership. -- app.getWindow(windowId: number) -> Window — select one window and read its initial AX state silently; do this before any window screenshot or coordinate input. +- app.getWindow(windowId: number) -> Window — select a window by an ID from getWindows without reading AX. Providers reject closed or foreign windows when operating on them. - app.activate() -> ActionResult — request explicit foreground activation; use only when foreground access is required. - app.show() -> { dispatched: boolean, windowId: number | null, foregroundChanged: boolean | null } — unhide the app and restore its selected, focused, main, or sole window without activation. - window.show() -> { dispatched: boolean, windowId: number | null, foregroundChanged: boolean | null } — unhide and restore that explicit window without activation. @@ -58,7 +58,7 @@ Setup and discovery: ## Accessibility tree Available at app or window level depending on the platform. -Element ids are opaque strings from the latest successful tree; the object facade keeps one current snapshot per app across its repeated app and window objects, a new read replaces it, and a failed read invalidates the previous one. Repeated strings can appear for different elements, so do not reuse an old id blindly, and element operations do not fall back to pointer input or typing. +Read getAccessibilityTree explicitly before using element ids. Selection and screenshots do not read or replace the current AX snapshot. Element ids are opaque strings from the latest successful tree; the object facade keeps one current snapshot per app across its repeated app and window objects, a new read replaces it, and a failed read invalidates the previous one. Repeated strings can appear for different elements, so do not reuse an old id blindly, and element operations do not fall back to pointer input or typing. - getAccessibilityTree(options?: { emit?: boolean, includeMenus?: boolean, details?: boolean, maxElements?: number, maxDepth?: number }) -> string — set the node count or depth for one read. Omitted limits use the provider settings. includeMenus expands menu contents and details adds identifiers, native bounds, and complete action names. macOS accepts maxElements up to 100000 and maxDepth up to 512. If the tree is truncated, select a narrower window or raise the reported limit on the next read. In persistent PTC, capture a large tree with const tree = await win.getAccessibilityTree({ emit: false }). Filter its lines in memory, or save it with await tools.write({ file_path: path, content: tree }) to a new workspace file. Search its contents with tools.grep({ path, pattern }) and use tools.read around a matching line. Print only the matches you need. Use element ids from the file only while that observation is current. - listActions(elementId: string) -> string[] — list the actions an element advertises. - invoke(elementId: string) -> ActionResult — invoke the element's default action; dispatched is false when it has none. @@ -67,7 +67,7 @@ Element ids are opaque strings from the latest successful tree; the object facad - selectText(elementId: string, text: string, options?: { prefix?: string, suffix?: string, selectionType?: 'text' | 'cursor_before' | 'cursor_after' }) -> ActionResult — select one literal occurrence or place the cursor beside it, using adjacent context to disambiguate. ## Simulated mouse -Use an explicit window for screenshots and pointer input. +Use an explicit window for screenshots and pointer input. Before coordinate input, including relative coordinates, obtain a successful getAccessibilityTree observation covering that window or take its screenshot. Absolute pixel coordinates require a returned screenshot. - getScreenshot() -> Image — capture the selected window as a stored image with readonly path, width, and height. Pass selected screenshots to inspect(...values) to display them; read the window ID from window.windowId. - mouseMove(target: string | [number, number], options?: { relative?: boolean, useSystemPointer?: boolean }) -> { dispatched: boolean, foregroundChanged?: boolean | null, hoverRoute: 'accessibility' | 'system-pointer', systemPointerMoved: boolean } — request a mouse move to the target without pressing a button; inspect hoverRoute and systemPointerMoved, and expect no automatic pointer restoration. Default hover requires both AXShowAlternateUI and AXShowDefaultUI on an enabled element or ancestor. Without that route, HOVER_UNAVAILABLE is returned before sending input. Physical movement requires an unobscured foreground target. Explain any necessary activation and observe the target again. - click(target: string | [number, number], options?: { mouseButton?: 'left' | 'right' | 'middle', clickCount?: 1 | 2, modifiers?: ('command' | 'control' | 'option' | 'shift' | 'fn')[], holdMs?: number, relative?: boolean }) -> ActionResult — click an element id or point, or perform a left-button double-click; holdMs controls the press duration of each press. click is a Window method: select a window with const win = await app.getWindow(windowId), then call win.click(target). For a background control’s default action, use app.invoke(id). For a mouse click on a background app, call app.activate() first and re-read the target. Both target forms send mouse-down and mouse-up events. click(id) uses the center of the full element bounds. If that center is outside the window, take a fresh win.getScreenshot() and use click([x, y]) on a visible part of the control in screenshot pixels. A background app can ignore the events even when dispatched is true. Verify the expected effect, such as a changed selection, before typing or retrying. @@ -83,7 +83,7 @@ Available at app or window level depending on the platform. - pressKey(key: string, modifiers?: string[]) -> ActionResult — press and release one complete keystroke. For example pressKey('Tab', ['shift']) or pressKey('+'). Key and modifier names are platform-defined strings. Pass key and modifiers separately; do not pack them into a chord string. - sendKeyEvent(key: string, direction: 'down' | 'up') -> ActionResult — send 'down' to hold a key across tool calls in one run_code, for example W down, D down, D up, W up. Holding a key on a physical keyboard can produce repeated key-down events after a delay. To send one such repeat for a non-modifier key this run already holds, send 'down' again. Each call posts one event. A control can filter repeats. Use the app's response to decide whether and when to repeat or release the key. When this run_code returns, throws, or the user pauses it, macOS releases only its remaining keys at their original recipients. -Raw tools use { method, params }: include appHandle for app calls and windowId for window calls, and use snapshotId with elementId for element routes. The emit option is facade-only: emit is local output control, so raw callers must follow the raw operation schema for admitted params. getAccessibilityTree and getWindow read through raw getState; getWindows maps to listWindows; getScreenshot maps to screenshot; cua.getState maps to listApps; every other named operation uses the same method name. +Raw tools use { method, params }: include appHandle for app calls and windowId for window calls, and use snapshotId with elementId for element routes. The emit option is facade-only: emit is local output control, so raw callers must follow the raw operation schema for admitted params. getAccessibilityTree maps to getState; getWindow has no raw method; getWindows maps to listWindows; getScreenshot maps to screenshot; cua.getState maps to listApps; every other named operation uses the same method name. Only if the task cannot be completed without moving the user’s pointer, call await win.mouseMove(target, { useSystemPointer: true }). Prefer an advertised accessibility action, a click on a visible control, or browser tools when possible. The question interrupts the user. When approval is disabled an explicit move can act immediately. Pass true on every physical move, even after approval. Omitting the option or passing false uses the default hover route without prompting. Approval or refusal lasts until this session closes or DSH restarts. A refusal sends no input. If the user dismisses the request, ask again only if they request it. Wait for an outstanding request to finish. Subagents cannot request physical movement. diff --git a/snapshots/session/ptc-script-retained/system-prompt.expected.md b/snapshots/session/ptc-script-retained/system-prompt.expected.md index b9559f014a..5dd8efdccc 100644 --- a/snapshots/session/ptc-script-retained/system-prompt.expected.md +++ b/snapshots/session/ptc-script-retained/system-prompt.expected.md @@ -48,9 +48,9 @@ Setup and discovery: - cua.listApps(options?: { emit?: boolean }) -> { id: string, displayName: string, isRunning: true, pid: number }[] — enumerate running applications. - cua.findApps(query: string, options?: { emit?: boolean }) -> ({ id: string, displayName: string, path: string } & ({ isRunning: true, pid: number } | { isRunning: false }))[] — search installed applications, including stopped matches; only running matches carry pid. - cua.getState(options?: { emit?: boolean }) -> { apps: { id: string, displayName: string, isRunning: true, pid: number }[], browsers: [] } — summarize running applications; this is not an accessibility observation. -- cua.getApp(selector: string) -> App — select an application by name, bundle ID, absolute path, or running PID string, obtain consent, launch in the background if needed, and read initial AX state silently; it does not activate the app. +- cua.getApp(selector: string) -> App — select an application by name, bundle ID, absolute path, or running PID string, obtain consent, launch in the background if needed, and return without reading AX or activating the app. - app.getWindows(options?: { emit?: boolean }) -> { windowId: number, appName: string, title: string, bounds: { x: number, y: number, width: number, height: number }, isOnScreen: boolean, onCurrentSpace: boolean | null }[] — enumerate windows with global native bounds; null onCurrentSpace means unknown Space membership. -- app.getWindow(windowId: number) -> Window — select one window and read its initial AX state silently; do this before any window screenshot or coordinate input. +- app.getWindow(windowId: number) -> Window — select a window by an ID from getWindows without reading AX. Providers reject closed or foreign windows when operating on them. - app.activate() -> ActionResult — request explicit foreground activation; use only when foreground access is required. - app.show() -> { dispatched: boolean, windowId: number | null, foregroundChanged: boolean | null } — unhide the app and restore its selected, focused, main, or sole window without activation. - window.show() -> { dispatched: boolean, windowId: number | null, foregroundChanged: boolean | null } — unhide and restore that explicit window without activation. @@ -58,7 +58,7 @@ Setup and discovery: ## Accessibility tree Available at app or window level depending on the platform. -Element ids are opaque strings from the latest successful tree; the object facade keeps one current snapshot per app across its repeated app and window objects, a new read replaces it, and a failed read invalidates the previous one. Repeated strings can appear for different elements, so do not reuse an old id blindly, and element operations do not fall back to pointer input or typing. +Read getAccessibilityTree explicitly before using element ids. Selection and screenshots do not read or replace the current AX snapshot. Element ids are opaque strings from the latest successful tree; the object facade keeps one current snapshot per app across its repeated app and window objects, a new read replaces it, and a failed read invalidates the previous one. Repeated strings can appear for different elements, so do not reuse an old id blindly, and element operations do not fall back to pointer input or typing. - getAccessibilityTree(options?: { emit?: boolean, includeMenus?: boolean, details?: boolean, maxElements?: number, maxDepth?: number }) -> string — set the node count or depth for one read. Omitted limits use the provider settings. includeMenus expands menu contents and details adds identifiers, native bounds, and complete action names. macOS accepts maxElements up to 100000 and maxDepth up to 512. If the tree is truncated, select a narrower window or raise the reported limit on the next read. In persistent PTC, capture a large tree with const tree = await win.getAccessibilityTree({ emit: false }). Filter its lines in memory, or save it with await tools.write({ file_path: path, content: tree }) to a new workspace file. Search its contents with tools.grep({ path, pattern }) and use tools.read around a matching line. Print only the matches you need. Use element ids from the file only while that observation is current. - listActions(elementId: string) -> string[] — list the actions an element advertises. - invoke(elementId: string) -> ActionResult — invoke the element's default action; dispatched is false when it has none. @@ -67,7 +67,7 @@ Element ids are opaque strings from the latest successful tree; the object facad - selectText(elementId: string, text: string, options?: { prefix?: string, suffix?: string, selectionType?: 'text' | 'cursor_before' | 'cursor_after' }) -> ActionResult — select one literal occurrence or place the cursor beside it, using adjacent context to disambiguate. ## Simulated mouse -Use an explicit window for screenshots and pointer input. +Use an explicit window for screenshots and pointer input. Before coordinate input, including relative coordinates, obtain a successful getAccessibilityTree observation covering that window or take its screenshot. Absolute pixel coordinates require a returned screenshot. - getScreenshot() -> Image — capture the selected window as a stored image with readonly path, width, and height. Pass selected screenshots to inspect(...values) to display them; read the window ID from window.windowId. - mouseMove(target: string | [number, number], options?: { relative?: boolean }) -> { dispatched: boolean, foregroundChanged?: boolean | null, hoverRoute: 'accessibility' | 'system-pointer', systemPointerMoved: boolean } — request a mouse move to the target without pressing a button; inspect hoverRoute and systemPointerMoved, and expect no automatic pointer restoration. Default hover requires both AXShowAlternateUI and AXShowDefaultUI on an enabled element or ancestor. Without that route, HOVER_UNAVAILABLE is returned before sending input. Physical movement requires an unobscured foreground target. Explain any necessary activation and observe the target again. - click(target: string | [number, number], options?: { mouseButton?: 'left' | 'right' | 'middle', clickCount?: 1 | 2, modifiers?: ('command' | 'control' | 'option' | 'shift' | 'fn')[], holdMs?: number, relative?: boolean }) -> ActionResult — click an element id or point, or perform a left-button double-click; holdMs controls the press duration of each press. click is a Window method: select a window with const win = await app.getWindow(windowId), then call win.click(target). For a background control’s default action, use app.invoke(id). For a mouse click on a background app, call app.activate() first and re-read the target. Both target forms send mouse-down and mouse-up events. click(id) uses the center of the full element bounds. If that center is outside the window, take a fresh win.getScreenshot() and use click([x, y]) on a visible part of the control in screenshot pixels. A background app can ignore the events even when dispatched is true. Verify the expected effect, such as a changed selection, before typing or retrying. @@ -83,7 +83,7 @@ Available at app or window level depending on the platform. - pressKey(key: string, modifiers?: string[]) -> ActionResult — press and release one complete keystroke. For example pressKey('Tab', ['shift']) or pressKey('+'). Key and modifier names are platform-defined strings. Pass key and modifiers separately; do not pack them into a chord string. - sendKeyEvent(key: string, direction: 'down' | 'up') -> ActionResult — send 'down' to hold a key across tool calls in one run_code, for example W down, D down, D up, W up. Holding a key on a physical keyboard can produce repeated key-down events after a delay. To send one such repeat for a non-modifier key this run already holds, send 'down' again. Each call posts one event. A control can filter repeats. Use the app's response to decide whether and when to repeat or release the key. When this run_code returns, throws, or the user pauses it, macOS releases only its remaining keys at their original recipients. -Raw tools use { method, params }: include appHandle for app calls and windowId for window calls, and use snapshotId with elementId for element routes. The emit option is facade-only: emit is local output control, so raw callers must follow the raw operation schema for admitted params. getAccessibilityTree and getWindow read through raw getState; getWindows maps to listWindows; getScreenshot maps to screenshot; cua.getState maps to listApps; every other named operation uses the same method name. +Raw tools use { method, params }: include appHandle for app calls and windowId for window calls, and use snapshotId with elementId for element routes. The emit option is facade-only: emit is local output control, so raw callers must follow the raw operation schema for admitted params. getAccessibilityTree maps to getState; getWindow has no raw method; getWindows maps to listWindows; getScreenshot maps to screenshot; cua.getState maps to listApps; every other named operation uses the same method name. AX read defaults: maxElements=2000, maxDepth=25.