From 2ef5f9dc534bcc935f3dfbcebdd39d9280f078a6 Mon Sep 17 00:00:00 2001 From: Dudu-0223 Date: Sat, 7 Nov 2026 12:20:20 +0800 Subject: [PATCH 1/9] fix(cua): select applications and windows without reading AX --- ...05-cua-scoped-surfaces-and-input.i18n.yaml | 4 +- ...026-11-05-cua-scoped-surfaces-and-input.md | 2 +- ...-11-05-cua-scoped-surfaces-and-input.zh.md | 2 +- .../cua-native-platform/guide.i18n.yaml | 4 +- .../v0.2.2-rc.1/cua-native-platform/guide.md | 2 +- .../cua-native-platform/guide.zh.md | 2 +- .../experimental/cua-macos/README.i18n.yaml | 8 +- packages/experimental/cua-macos/README.md | 14 +- packages/experimental/cua-macos/README.zh.md | 14 +- .../tests/loader-composition.spec.ts | 2 +- .../tests/persistent-bridge.built.e2e.ts | 2 +- packages/experimental/cua/README.i18n.yaml | 8 +- packages/experimental/cua/README.md | 4 +- packages/experimental/cua/README.zh.md | 4 +- packages/experimental/cua/package.json | 3 +- packages/experimental/cua/src/api.ts | 15 +- packages/experimental/cua/src/guidance.ts | 8 +- packages/experimental/cua/src/types.ts | 8 +- packages/experimental/cua/tests/api.spec.ts | 145 ++++++++++++++++-- .../experimental/cua/tests/bridge.spec.ts | 5 +- packages/experimental/cua/tsconfig.host.json | 3 + pnpm-lock.yaml | 3 + .../cua-native/system-prompt.expected.md | 8 +- .../cua-native/workspace/native-helper.mjs | 1 + snapshots/session/cua-ptc/consent-fixture.mjs | 6 +- .../session/cua-ptc/replay.override.json | 35 ++++- snapshots/session/cua-ptc/session.v4.jsonl | 26 +++- .../session/cua-ptc/system-prompt.expected.md | 8 +- .../system-prompt.expected.md | 8 +- 29 files changed, 276 insertions(+), 78 deletions(-) diff --git a/.agents/notes/implemented/architecture/2026-11-05-cua-scoped-surfaces-and-input.i18n.yaml b/.agents/notes/implemented/architecture/2026-11-05-cua-scoped-surfaces-and-input.i18n.yaml index f3a0d8f1d3..5f1ff64a4d 100644 --- a/.agents/notes/implemented/architecture/2026-11-05-cua-scoped-surfaces-and-input.i18n.yaml +++ b/.agents/notes/implemented/architecture/2026-11-05-cua-scoped-surfaces-and-input.i18n.yaml @@ -9,8 +9,8 @@ en: f72db0b79aa02fb5 zh: b91c1f177dc93ab4 /agent-note-cua-scoped-surfaces-and-separated-input-capabilities/decision: - en: ec115e11f96ffe60 - zh: c1fe9eb2866196d0 + en: 4fa61247fe33edb0 + zh: afa76f74841eb586 /agent-note-cua-scoped-surfaces-and-separated-input-capabilities/alternatives-considered: en: 32c970ae153e0f6c zh: 8431be0a10ffed15 diff --git a/.agents/notes/implemented/architecture/2026-11-05-cua-scoped-surfaces-and-input.md b/.agents/notes/implemented/architecture/2026-11-05-cua-scoped-surfaces-and-input.md index 609e5bf901..3888527013 100644 --- a/.agents/notes/implemented/architecture/2026-11-05-cua-scoped-surfaces-and-input.md +++ b/.agents/notes/implemented/architecture/2026-11-05-cua-scoped-surfaces-and-input.md @@ -16,7 +16,7 @@ Provider startup capabilities are static. Platform, supported methods, applicati Keyboard input keeps keys and modifiers separate: `pressKey(key, modifiers?)` presses and releases one key while holding the listed modifiers, and its keyboard modifier list is a plain string array defined by the provider. `sendKeyEvent(key, direction)` sends exactly one key-down or key-up without an implicit counterpart. A caller can therefore hold a modifier or split a gesture; neither method packs a chord string, and the shared schema does not enumerate platform key names. -Within one API instance, the object facade keeps one current accessibility snapshot per application handle, shared by every application and window object for that handle, including repeated selections of the same application. `getWindow` reads its initial tree silently; `getApp` does the same when the provider reports application-level accessibility. A later `getAccessibilityTree` read invalidates the previous snapshot before dispatch. The facade automatically attaches the latest snapshot ID to element operations; callers use element IDs from the latest tree and do not assume an older ID still identifies the same control. Element IDs are opaque provider strings. Only an explicit `CuaWindow` captures screenshots or sends pointer input. The native helper retains its own application and optional-window snapshots behind that facade; the native layer and the facade snapshot remain separate. The external runtime continues to own variable persistence and completion serialization; the facade adds no evaluator. The [native object decision](2026-09-17-experimental-cua-object-api.md) owns evaluator independence, and the [shared helper decision](2026-10-09-cua-shared-native-helper.md) owns connection lifetime. +Within one API instance, the object facade keeps one current accessibility snapshot per application handle, shared by every application and window object for that handle, including repeated selections of the same application. `getApp` and `getWindow` select objects without reading accessibility or replacing that snapshot; `getWindow` validates membership in the application’s current window list. An explicit `getAccessibilityTree` read invalidates the previous snapshot before dispatch. The facade automatically attaches the latest snapshot ID to element operations; callers use element IDs from the latest tree and do not assume an older ID still identifies the same control. Element IDs are opaque provider strings. Only an explicit `CuaWindow` captures screenshots or sends pointer input. The native helper retains its own application and optional-window snapshots behind that facade; the native layer and the facade snapshot remain separate. The external runtime continues to own variable persistence and completion serialization; the facade adds no evaluator. The [native object decision](2026-09-17-experimental-cua-object-api.md) owns evaluator independence, and the [shared helper decision](2026-10-09-cua-shared-native-helper.md) owns connection lifetime. ## Alternatives considered diff --git a/.agents/notes/implemented/architecture/2026-11-05-cua-scoped-surfaces-and-input.zh.md b/.agents/notes/implemented/architecture/2026-11-05-cua-scoped-surfaces-and-input.zh.md index 081f02986e..8a3f152e67 100644 --- a/.agents/notes/implemented/architecture/2026-11-05-cua-scoped-surfaces-and-input.zh.md +++ b/.agents/notes/implemented/architecture/2026-11-05-cua-scoped-surfaces-and-input.zh.md @@ -16,7 +16,7 @@ Provider 的启动能力是静态的。平台、支持的方法、应用级辅 键盘输入把按键与修饰键分开：`pressKey(key, modifiers?)` 在按住所列修饰键时按下并释放一个键，其键盘修饰键列表是 provider 定义的普通字符串数组。`sendKeyEvent(key, direction)` 只发送一次按下或抬起，不隐式发送相反事件。因此调用方可以保持修饰键或拆分手势；两个方法都不接受组合键字符串，共享 schema 也不枚举平台按键名。 -在单个 API 实例内，对象门面按应用句柄保存一份当前辅助功能快照，该句柄的所有应用对象和窗口对象共享这份快照，包括对同一应用的重复选择。`getWindow` 静默执行首次树读取；当 provider 报告应用级辅助功能时，`getApp` 也静默读取。之后的 `getAccessibilityTree` 读取会在分派前使上一份快照失效。门面会为元素操作自动附加最新快照 ID；调用方应使用最新树中的元素 ID，不应假定旧 ID 仍指向同一控件。元素 ID 是 provider 定义的不透明字符串。只有显式 `CuaWindow` 可以截图或发送指针输入。原生 helper 在该门面之后保留自己的应用快照和可选窗口快照；原生层与门面快照相互独立。外部运行时继续负责变量持久化和最终表达式序列化；门面不加入 evaluator。[原生对象决定](2026-09-17-experimental-cua-object-api.zh.md)负责 evaluator 独立性，[共享 helper 决定](2026-10-09-cua-shared-native-helper.zh.md)负责连接生命期。 +在单个 API 实例内，对象门面按应用句柄保存一份当前辅助功能快照，该句柄的所有应用对象和窗口对象共享这份快照，包括对同一应用的重复选择。`getApp` 和 `getWindow` 只选择对象，不读取辅助功能或替换该快照；`getWindow` 会核验窗口属于应用的当前窗口列表。显式 `getAccessibilityTree` 读取会在分派前使上一份快照失效。门面会为元素操作自动附加最新快照 ID；调用方应使用最新树中的元素 ID，不应假定旧 ID 仍指向同一控件。元素 ID 是 provider 定义的不透明字符串。只有显式 `CuaWindow` 可以截图或发送指针输入。原生 helper 在该门面之后保留自己的应用快照和可选窗口快照；原生层与门面快照相互独立。外部运行时继续负责变量持久化和最终表达式序列化；门面不加入 evaluator。[原生对象决定](2026-09-17-experimental-cua-object-api.zh.md)负责 evaluator 独立性，[共享 helper 决定](2026-10-09-cua-shared-native-helper.zh.md)负责连接生命期。 ## 考虑过的替代方案 diff --git a/docs/upgrade-guide/v0.2.2-rc.1/cua-native-platform/guide.i18n.yaml b/docs/upgrade-guide/v0.2.2-rc.1/cua-native-platform/guide.i18n.yaml index a76d4de320..c907006371 100644 --- a/docs/upgrade-guide/v0.2.2-rc.1/cua-native-platform/guide.i18n.yaml +++ b/docs/upgrade-guide/v0.2.2-rc.1/cua-native-platform/guide.i18n.yaml @@ -12,5 +12,5 @@ en: e5d1027faa55adba zh: 4b88ad6a70977fb3 /native-cua-application-and-window-api/migration: - en: b09e3dde32391786 - zh: f702e1e25b260b29 + en: b4d5bf5c30f42dfb + zh: 7ec59dc6a90165ef diff --git a/docs/upgrade-guide/v0.2.2-rc.1/cua-native-platform/guide.md b/docs/upgrade-guide/v0.2.2-rc.1/cua-native-platform/guide.md index 3179dae0d9..a49a251c37 100644 --- a/docs/upgrade-guide/v0.2.2-rc.1/cua-native-platform/guide.md +++ b/docs/upgrade-guide/v0.2.2-rc.1/cua-native-platform/guide.md @@ -16,7 +16,7 @@ Providers supply startup capabilities alongside their platform guidance. The nat ## Migration 1. After `cua.getApp(selector)`, select the target from `app.getWindows()` with `app.getWindow(windowId)`. Use `window.getScreenshot()`, `window.click()`, `window.scroll()`, and `window.drag()`. Replace `hover` with `window.mouseMove()`. On macOS, `window.click(id)` sends mouse events at the element center. Switch to `invoke(id)` for the default accessibility action, or `performAction(id, "AXShowMenu")` to request an advertised menu action. -2. Replace `getAXState()` with `getAccessibilityTree()`. Replace `getAXStateAndScreenshot()` with separate tree and screenshot reads. Selection reads the initial tree silently for a window, and for an app when `appLevelAccessibility` is true. Read the tree explicitly to display it. +2. Replace `getAXState()` with `getAccessibilityTree()`. Replace `getAXStateAndScreenshot()` with separate tree and screenshot reads. Application and window selection no longer read AX implicitly. Before using element IDs, call `getAccessibilityTree()` explicitly; use `{ emit: false }` to retain the snapshot without displaying it. A screenshot does not require a successful AX read. 3. Pass the opaque string ID from the latest tree wherever you passed a numeric element index. In raw requests use `elementId` with `snapshotId`. Replace `performSecondaryAction` with `listActions(id)` and `performAction(id, exactName, argument?)`, or use `invoke(id)` for the default action. `setValue` accepts the provider’s supported JSON values. On macOS, `performAction(id, 'AXRaise')` performs the advertised raise action. Call `app.activate()` first when the application must become active. 4. Replace a packed chord such as `pressKey('shift+Tab')` with `pressKey('Tab', ['shift'])`. Check `(await cua.getCapabilities()).methods` before using `sendKeyEvent`: macOS supports complete keystrokes and rejects directional events with `NOT_IMPLEMENTED`. 5. Read `dispatched` from action results, then observe the application to verify the outcome. `invoke` returns `dispatched: false` when the element has no default action. Remove reads of the old `verified` field and of `pid` on selected objects or window records. Use `listApps()` or running `findApps()` matches when you need a PID. diff --git a/docs/upgrade-guide/v0.2.2-rc.1/cua-native-platform/guide.zh.md b/docs/upgrade-guide/v0.2.2-rc.1/cua-native-platform/guide.zh.md index f0a4a905e0..fae92008f1 100644 --- a/docs/upgrade-guide/v0.2.2-rc.1/cua-native-platform/guide.zh.md +++ b/docs/upgrade-guide/v0.2.2-rc.1/cua-native-platform/guide.zh.md @@ -16,7 +16,7 @@ CUA 脚本在指定窗口上截图和操作鼠标。App 和 Window 对象共享 ## 迁移 1. 调用 `cua.getApp(selector)` 后，从 `app.getWindows()` 返回的列表选定目标，再调用 `app.getWindow(windowId)`。在窗口上调用 `getScreenshot()`、`click()`、`scroll()` 和 `drag()`，将原有的 `hover` 调用改为 `window.mouseMove()`。在 macOS 上，`window.click(id)` 在元素中心发送鼠标事件。需要执行默认辅助功能动作时，改用 `invoke(id)`。需要打开控件菜单且控件声明了 `AXShowMenu` 时，调用 `performAction(id, "AXShowMenu")`。 -2. 将 `getAXState()` 改为 `getAccessibilityTree()`，将 `getAXStateAndScreenshot()` 拆成读取树和截图两次调用。选择窗口时会静默读取初始树。选择应用时，只有 `appLevelAccessibility` 为真才读取。要显示树，请显式读取。 +2. 将 `getAXState()` 改为 `getAccessibilityTree()`，将 `getAXStateAndScreenshot()` 拆成读取树和截图两次调用。选择应用或窗口不再隐式读取 AX。使用元素 ID 前须显式调用 `getAccessibilityTree()`；使用 `{ emit: false }` 可保留快照但不显示树。截图不要求 AX 读取成功。 3. 将数字元素索引改为从最新树中取得的字符串 ID。直接发送请求时使用 `elementId` 和 `snapshotId`。原先调用 `performSecondaryAction` 的地方，先用 `listActions(id)` 取得完整动作名，再调用 `performAction(id, exactName, argument?)`。要执行默认动作，可调用 `invoke(id)`。`setValue` 接受提供方支持的 JSON 值。 在 macOS 上，`performAction(id, 'AXRaise')` 只执行元素声明的抬起窗口动作。需要把应用切到前台时先调用 `app.activate()`。 4. 将 `pressKey('shift+Tab')` 之类的组合键写法改为 `pressKey('Tab', ['shift'])`。调用 `sendKeyEvent` 前检查 `(await cua.getCapabilities()).methods`。macOS 支持完整按键，单独发送按下或松开事件会返回 `NOT_IMPLEMENTED`。 5. 从动作结果读取 `dispatched`，再观察应用确认效果。元素没有默认动作时，`invoke` 返回 `dispatched: false`。移除读取旧字段 `verified` 的代码，以及对已选对象和窗口记录中 `pid` 的读取。需要 PID 时，使用 `listApps()` 或 `findApps()` 返回的运行中应用记录。 diff --git a/packages/experimental/cua-macos/README.i18n.yaml b/packages/experimental/cua-macos/README.i18n.yaml index 0d83c6e538..7571c8004d 100644 --- a/packages/experimental/cua-macos/README.i18n.yaml +++ b/packages/experimental/cua-macos/README.i18n.yaml @@ -15,11 +15,11 @@ en: 1777e7bf6785f986 zh: e3daac1074ebb74d /deepseek-ai-dsh-experimental-cua-macos/use-this-package: - en: 533ed4dd11af5126 - zh: 03036e8c949d6aca + en: e9c78a8cbd68147c + zh: 98c64b09f858edf7 /deepseek-ai-dsh-experimental-cua-macos/use-this-package/connect-an-external-execution-environment: - en: b734a0907747580e - zh: 8110c5951b70e100 + en: 7c460511bb1cee4c + zh: 2eba4fc997427e18 /deepseek-ai-dsh-experimental-cua-macos/understand-the-implementation: en: 4e82b85a5aa72eee zh: 4e004b7a205d8688 diff --git a/packages/experimental/cua-macos/README.md b/packages/experimental/cua-macos/README.md index 46b035112e..d139d988a3 100644 --- a/packages/experimental/cua-macos/README.md +++ b/packages/experimental/cua-macos/README.md @@ -41,7 +41,7 @@ For repository Web use, enable [Computer Use](../cua-bundle/README.md) in Plugin pnpm dsh headless --patch packages/experimental/cua-macos/examples/macos.cordis.yml --dump-config ``` -The dump command checks profile composition without a model request. For a real task, replace `--dump-config` with a prompt and configure the profile’s model credentials. The overlay mounts the computer-use registry and this provider; remove an existing competing provider before using it. The helper needs Accessibility permission for AX/input and Screen Recording permission for capture; use the Computer Use permission page to open System Settings and add the helper. Source builds use the bundle filename `DSH Computer Use Dev.app` and declare `DSH Computer Use Dev` in Info.plist; select the development entry when granting access. `getCapabilities` returns the provider’s declared capabilities with observed permissions and session lock state. Locked or unavailable desktop sessions reject UI observations and input before dispatch; the user must unlock macOS before resuming. `listApps` and `cua.getState` list running applications. `findApps(query)` searches standard macOS application directories by name, bundle ID or absolute path. `getApp` selects a unique name, bundle ID, path or running PID, obtains application consent, and launches a stopped app in the background when needed. A stopped application’s bundle identifier and executable path are read from disk before approval and rechecked before and after launch; applications without a bundle identifier use their canonical executable path for reusable consent. Selecting a stopped installation requests a new instance of that copy. The helper waits for launch to complete before reporting the running state. A post-launch verification failure expires the pending handle but can leave the process running; inspect running applications before selecting a fresh handle. The returned object reads initial AX silently without activating the app. Launch cancellation may leave the application running. Model guidance starts with background observation and input, without activation during initialization or before each action. For hidden or minimized windows, try `app.show()` or `window.show()` and read fresh state. This requests unhide and unminimize without activation, raising or switching Spaces; app-level restoration selects the focused, main or sole AX window. The result names the selected `windowId`, or null when no window was selected, and reports `foregroundChanged` by comparing the front app and top window before and after the request. Null means foreground evidence was unavailable; later focus changes remain possible. If restoration fails or the window remains unusable, explain the needed foreground step, call `app.activate()` or an observed window’s advertised `AXRaise` action, then observe again. Necessary foreground access takes priority over preserving focus. Applications may change focus after input; background delivery does not guarantee an undisturbed user session. +The dump command checks profile composition without a model request. For a real task, replace `--dump-config` with a prompt and configure the profile’s model credentials. The overlay mounts the computer-use registry and this provider; remove an existing competing provider before using it. The helper needs Accessibility permission for AX/input and Screen Recording permission for capture; use the Computer Use permission page to open System Settings and add the helper. Source builds use the bundle filename `DSH Computer Use Dev.app` and declare `DSH Computer Use Dev` in Info.plist; select the development entry when granting access. `getCapabilities` returns the provider’s declared capabilities with observed permissions and session lock state. Locked or unavailable desktop sessions reject UI observations and input before dispatch; the user must unlock macOS before resuming. `listApps` and `cua.getState` list running applications. `findApps(query)` searches standard macOS application directories by name, bundle ID or absolute path. `getApp` selects a unique name, bundle ID, path or running PID, obtains application consent, and launches a stopped app in the background when needed. A stopped application’s bundle identifier and executable path are read from disk before approval and rechecked before and after launch; applications without a bundle identifier use their canonical executable path for reusable consent. Selecting a stopped installation requests a new instance of that copy. The helper waits for launch to complete before reporting the running state. A post-launch verification failure expires the pending handle but can leave the process running; inspect running applications before selecting a fresh handle. The returned object does not read AX or activate the app. Launch cancellation may leave the application running. Model guidance starts with background observation and input, without activation during initialization or before each action. For hidden or minimized windows, try `app.show()` or `window.show()` and read fresh state. This requests unhide and unminimize without activation, raising or switching Spaces; app-level restoration selects the focused, main or sole AX window. The result names the selected `windowId`, or null when no window was selected, and reports `foregroundChanged` by comparing the front app and top window before and after the request. Null means foreground evidence was unavailable; later focus changes remain possible. If restoration fails or the window remains unusable, explain the needed foreground step, call `app.activate()` or an observed window’s advertised `AXRaise` action, then observe again. Necessary foreground access takes priority over preserving focus. Applications may change focus after input; background delivery does not guarantee an undisturbed user session. Source builds default to `sourceSigning: ad-hoc`, without creating certificates or keychains. Changed binaries can require new permission grants. To reduce repeated grants during rebuilds and across worktrees, select an existing certificate and its absolute keychain path in the DSH home’s `cordis.patch.yml`; `cua` is the entry ID loading `@deepseek-ai/dsh-experimental-cua-macos`: @@ -111,9 +111,9 @@ const window = await app.getWindow(windows[0].windowId) void await window.getScreenshot() ``` -Use the `cua` API to discover and select the application (`listApps`, `getState`, `findApps`, `getApp`); use the selected app to activate it, enumerate its windows, and select one. Window selection is required for screenshots and pointer input, not only when several windows exist. `getApp` and `app.getWindow(windowId)` read their initial trees silently; choose the window by its observed title and bounds. +Use the `cua` API to discover and select the application (`listApps`, `getState`, `findApps`, `getApp`); use the selected app to activate it, enumerate its windows, and select one. Window selection is required for screenshots and pointer input, not only when several windows exist. `getApp` and `app.getWindow(windowId)` return without reading AX; choose the window from `getWindows()` by its title and bounds. -`getWindow` reads its initial tree silently. An owned window without an AX surface returns an empty snapshot and remains available for screenshots and coordinate input. Within one API instance, the facade keeps one current accessibility snapshot per application handle, shared by every app and window object for that handle, including repeated selections. Reading any window replaces that snapshot; element operations use the latest snapshot ID, so use IDs from the latest tree and do not assume an older ID still identifies the same control. A failed read publishes no partial snapshot; read again without repeating the preceding input. Window objects own screenshots and pointer input. Keyboard methods stay on `app` because input follows the app’s current focus. +`getWindow` validates the ID against the current application window list. Read `getAccessibilityTree()` explicitly before element operations. Screenshots and coordinate input do not require a successful AX read; absolute pixel coordinates require a delivered screenshot. An owned window without an AX surface returns an empty snapshot when explicitly observed. Within one API instance, the facade keeps one current accessibility snapshot per application handle, shared by every app and window object for that handle, including repeated selections. Reading any window replaces that snapshot; element operations use the latest snapshot ID, so use IDs from the latest tree and do not assume an older ID still identifies the same control. A failed read publishes no partial snapshot; read again without repeating the preceding input. Window objects own screenshots and pointer input. Keyboard methods stay on `app` because input follows the app’s current focus. `mouseMove(target)` defaults to AX hover. An enabled target or ancestor must advertise both `AXShowAlternateUI` and `AXShowDefaultUI`. Moving into its descendants keeps the UI open, and leaving restores it. Without an AX route the helper returns `HOVER_UNAVAILABLE` before sending input. Click a visible control or use `performAction` with an advertised action. For browser content without these AX actions, use browser tools. Enable `systemPointerHover` only when moving the user’s visible cursor is acceptable. That route requires a foreground, unobscured target and leaves the cursor at the destination. @@ -189,9 +189,9 @@ Setup and discovery: - cua.listApps(options?: { emit?: boolean }) -> { id: string, displayName: string, isRunning: true, pid: number }[] — enumerate running applications. - cua.findApps(query: string, options?: { emit?: boolean }) -> ({ id: string, displayName: string, path: string } & ({ isRunning: true, pid: number } | { isRunning: false }))[] — search installed applications, including stopped matches; only running matches carry pid. - cua.getState(options?: { emit?: boolean }) -> { apps: { id: string, displayName: string, isRunning: true, pid: number }[], browsers: [] } — summarize running applications; this is not an accessibility observation. -- cua.getApp(selector: string) -> App — select an application by name, bundle ID, absolute path, or running PID string, obtain consent, launch in the background if needed, and read initial AX state silently; it does not activate the app. +- cua.getApp(selector: string) -> App — select an application by name, bundle ID, absolute path, or running PID string, obtain consent, launch in the background if needed, and return without reading AX or activating the app. - app.getWindows(options?: { emit?: boolean }) -> { windowId: number, appName: string, title: string, bounds: { x: number, y: number, width: number, height: number }, isOnScreen: boolean, onCurrentSpace: boolean | null }[] — enumerate windows with global native bounds; null onCurrentSpace means unknown Space membership. -- app.getWindow(windowId: number) -> Window — select one window and read its initial AX state silently; do this before any window screenshot or coordinate input. +- app.getWindow(windowId: number) -> Window — select a window from the application’s current window list without reading AX; do this before any window screenshot or coordinate input. - app.activate() -> ActionResult — request explicit foreground activation; use only when foreground access is required. - app.show() -> { dispatched: boolean, windowId: number | null, foregroundChanged: boolean | null } — unhide the app and restore its selected, focused, main, or sole window without activation. - window.show() -> { dispatched: boolean, windowId: number | null, foregroundChanged: boolean | null } — unhide and restore that explicit window without activation. @@ -199,7 +199,7 @@ Setup and discovery: ## Accessibility tree Available at app or window level depending on the platform. -Element ids are opaque strings from the latest successful tree; the object facade keeps one current snapshot per app across its repeated app and window objects, a new read replaces it, and a failed read invalidates the previous one. Repeated strings can appear for different elements, so do not reuse an old id blindly, and element operations do not fall back to pointer input or typing. +Read getAccessibilityTree explicitly before using element ids. Selection and screenshots do not read or replace the current AX snapshot. Element ids are opaque strings from the latest successful tree; the object facade keeps one current snapshot per app across its repeated app and window objects, a new read replaces it, and a failed read invalidates the previous one. Repeated strings can appear for different elements, so do not reuse an old id blindly, and element operations do not fall back to pointer input or typing. - getAccessibilityTree(options?: { emit?: boolean, includeMenus?: boolean, details?: boolean }) -> string — read the full tree text; includeMenus expands menu contents and details adds identifiers, native bounds, and complete action names. - listActions(elementId: string) -> string[] — list the actions an element advertises. - invoke(elementId: string) -> ActionResult — invoke the element's default action; dispatched is false when it has none. @@ -223,7 +223,7 @@ Available at app or window level depending on the platform. - paste(text: string, options?: { format?: 'text' | 'html' }) -> ActionResult — deliver text or HTML through the app's Paste command, restoring prior clipboard contents unless another writer changes them; a Paste failure is surfaced without substituting typeText, and PASTE_NOT_CONSUMED means inspect before any retry. - pressKey(key: string, modifiers?: string[]) -> ActionResult — press and release one complete keystroke. For example pressKey('Tab', ['shift']) or pressKey('+'). Key and modifier names are platform-defined strings. Pass key and modifiers separately; do not pack them into a chord string. -Raw tools use { method, params }: include appHandle for app calls and windowId for window calls, and use snapshotId with elementId for element routes. The emit option is facade-only: emit is local output control, so raw callers must follow the raw operation schema for admitted params. getAccessibilityTree and getWindow read through raw getState; getWindows maps to listWindows; getScreenshot maps to screenshot; cua.getState maps to listApps; every other named operation uses the same method name. +Raw tools use { method, params }: include appHandle for app calls and windowId for window calls, and use snapshotId with elementId for element routes. The emit option is facade-only: emit is local output control, so raw callers must follow the raw operation schema for admitted params. getAccessibilityTree maps to getState; getWindow validates against listWindows; getWindows maps to listWindows; getScreenshot maps to screenshot; cua.getState maps to listApps; every other named operation uses the same method name. ``` #### Token effect diff --git a/packages/experimental/cua-macos/README.zh.md b/packages/experimental/cua-macos/README.zh.md index 341796b8c4..f8fed7b624 100644 --- a/packages/experimental/cua-macos/README.zh.md +++ b/packages/experimental/cua-macos/README.zh.md @@ -41,7 +41,7 @@ provider 按当前界面语言生成引导和恢复文案，不支持的语言 pnpm dsh headless --patch packages/experimental/cua-macos/examples/macos.cordis.yml --dump-config ``` -配置输出命令不发起模型请求。执行真实任务时，将 `--dump-config` 替换为任务提示，并配置该 profile 的模型凭据。配置挂载 computer-use 注册表和本 provider；使用前需移除已有的竞争 provider。helper 需要辅助功能权限来读取 AX 和输入，需要屏幕录制权限来截图；通过 Computer Use 权限页打开系统设置并添加 helper。源码构建的应用包名为 `DSH Computer Use Dev.app`，Info.plist 声明名称 `DSH Computer Use Dev`；授权时选择开发版条目。`getCapabilities` 返回提供方声明的平台能力，以及实际查询的权限和锁屏状态。桌面会话锁定或不可用时，界面观察与输入会在分派前拒绝，需由用户解锁 macOS 后继续。`listApps` 和 `cua.getState` 列出运行中的应用。`findApps(query)` 按名称、bundle ID 或绝对路径搜索 macOS 标准应用目录。`getApp` 按唯一名称、bundle ID、路径或运行中的 PID 选择应用，取得应用许可后，在需要时后台启动尚未运行的应用。授权前从磁盘读取应用的 Bundle ID 和可执行路径，在启动前后重新读取并核对；没有 Bundle ID 的应用使用规范化的可执行路径复用授权。选择尚未运行的安装副本时，会请求为该副本创建新实例。helper 等待应用完成启动后返回运行状态。启动后验证失败会使待启动句柄失效，但进程可能仍在运行；应先查看运行中的应用，再取得新句柄。返回的对象静默读取初始 AX，不自动激活应用。启动被取消时，应用可能已经运行。模型指引要求先在后台观察和输入，不在初始化或每次动作前激活应用。窗口隐藏或最小化时，先尝试 `app.show()` 或 `window.show()`，再重新观察。它请求取消隐藏和恢复最小化，不主动激活、抬起窗口或切换桌面空间；应用级恢复选择焦点窗口、主窗口或唯一 AX 窗口。结果通过 `windowId` 指明选中的窗口，未选中时为 null；`foregroundChanged` 比较操作前后的前台应用和最上层窗口，无法取得完整观察时为 null，返回后仍可能发生焦点变化。恢复失败或窗口仍不可用时，说明需要前台的步骤，再调用 `app.activate()` 或窗口声明的 `AXRaise` 动作并重新观察。必须使用前台时，以完成任务为先。应用收到输入后仍可能改变焦点，后台投递不保证用户操作始终不受干扰。 +配置输出命令不发起模型请求。执行真实任务时，将 `--dump-config` 替换为任务提示，并配置该 profile 的模型凭据。配置挂载 computer-use 注册表和本 provider；使用前需移除已有的竞争 provider。helper 需要辅助功能权限来读取 AX 和输入，需要屏幕录制权限来截图；通过 Computer Use 权限页打开系统设置并添加 helper。源码构建的应用包名为 `DSH Computer Use Dev.app`，Info.plist 声明名称 `DSH Computer Use Dev`；授权时选择开发版条目。`getCapabilities` 返回提供方声明的平台能力，以及实际查询的权限和锁屏状态。桌面会话锁定或不可用时，界面观察与输入会在分派前拒绝，需由用户解锁 macOS 后继续。`listApps` 和 `cua.getState` 列出运行中的应用。`findApps(query)` 按名称、bundle ID 或绝对路径搜索 macOS 标准应用目录。`getApp` 按唯一名称、bundle ID、路径或运行中的 PID 选择应用，取得应用许可后，在需要时后台启动尚未运行的应用。授权前从磁盘读取应用的 Bundle ID 和可执行路径，在启动前后重新读取并核对；没有 Bundle ID 的应用使用规范化的可执行路径复用授权。选择尚未运行的安装副本时，会请求为该副本创建新实例。helper 等待应用完成启动后返回运行状态。启动后验证失败会使待启动句柄失效，但进程可能仍在运行；应先查看运行中的应用，再取得新句柄。返回对象时不读取 AX，也不激活应用。启动被取消时，应用可能已经运行。模型指引要求先在后台观察和输入，不在初始化或每次动作前激活应用。窗口隐藏或最小化时，先尝试 `app.show()` 或 `window.show()`，再重新观察。它请求取消隐藏和恢复最小化，不主动激活、抬起窗口或切换桌面空间；应用级恢复选择焦点窗口、主窗口或唯一 AX 窗口。结果通过 `windowId` 指明选中的窗口，未选中时为 null；`foregroundChanged` 比较操作前后的前台应用和最上层窗口，无法取得完整观察时为 null，返回后仍可能发生焦点变化。恢复失败或窗口仍不可用时，说明需要前台的步骤，再调用 `app.activate()` 或窗口声明的 `AXRaise` 动作并重新观察。必须使用前台时，以完成任务为先。应用收到输入后仍可能改变焦点，后台投递不保证用户操作始终不受干扰。 源码构建默认使用 `sourceSigning: ad-hoc` 临时签名，不创建证书或钥匙串。二进制变化后可能需要重新授权。需要减少开发重建和跨 worktree 的授权次数时，在 DSH home 的 `cordis.patch.yml` 中选择已有证书及其钥匙串绝对路径；`cua` 是加载 `@deepseek-ai/dsh-experimental-cua-macos` 的节点 ID： @@ -111,9 +111,9 @@ const window = await app.getWindow(windows[0].windowId) void await window.getScreenshot() ``` -使用 `cua` API 发现并选择应用（`listApps`、`getState`、`findApps`、`getApp`）；使用选定的应用来激活它、枚举其窗口并选择窗口。截图和指针输入都要求选择窗口，不只在存在多个窗口时才需要。`getApp` 和 `app.getWindow(windowId)` 静默读取初始树；根据观察到的标题与位置选择窗口。 +使用 `cua` API 发现并选择应用（`listApps`、`getState`、`findApps`、`getApp`）；使用选定的应用来激活它、枚举其窗口并选择窗口。截图和指针输入都要求选择窗口，不只在存在多个窗口时才需要。`getApp` 和 `app.getWindow(windowId)` 返回时不读取 AX；根据 `getWindows()` 中的标题与位置选择窗口。 -`getWindow` 静默读取初始树。已确认归属但没有 AX 界面的窗口返回空快照，仍可截图和发送坐标输入。在单个 API 实例内，门面按应用句柄保存一份当前辅助功能快照，该句柄的所有应用对象和窗口对象共享它，包括重复选择。读取任一窗口都会替换该快照；元素操作使用最新快照 ID，因此应使用最新树中的元素 ID，不应假定旧 ID 仍指向同一控件。读取失败不会发布部分快照；可以重新读取，但不要重复此前的输入。窗口对象负责截图和指针输入。键盘方法保留在 `app` 上，因为输入跟随应用的当前焦点。 +`getWindow` 根据当前应用窗口清单验证 ID。元素操作前须显式读取 `getAccessibilityTree()`。截图和坐标输入不要求 AX 读取成功；绝对像素坐标要求已交付的截图。已确认归属但没有 AX 界面的窗口在显式观察时返回空快照。在单个 API 实例内，门面按应用句柄保存一份当前辅助功能快照，该句柄的所有应用对象和窗口对象共享它，包括重复选择。读取任一窗口都会替换该快照；元素操作使用最新快照 ID，因此应使用最新树中的元素 ID，不应假定旧 ID 仍指向同一控件。读取失败不会发布部分快照；可以重新读取，但不要重复此前的输入。窗口对象负责截图和指针输入。键盘方法保留在 `app` 上，因为输入跟随应用的当前焦点。 `mouseMove(target)` 默认通过 AX 悬停。目标或其祖先必须已启用，并同时声明 `AXShowAlternateUI` 和 `AXShowDefaultUI`。进入子控件会保持展开，离开后恢复默认界面。如果没有可用的 AX 路径，helper 返回 `HOVER_UNAVAILABLE`，不发送输入。可点击已经可见的控件，或用 `performAction` 执行控件声明的动作。浏览器内容缺少这些 AX 动作时，请使用浏览器工具。只有允许移动用户可见光标时才开启 `systemPointerHover`。这条路径要求目标在前台且未被遮挡，操作后光标停在目标位置。 @@ -189,9 +189,9 @@ Setup and discovery: - cua.listApps(options?: { emit?: boolean }) -> { id: string, displayName: string, isRunning: true, pid: number }[] — enumerate running applications. - cua.findApps(query: string, options?: { emit?: boolean }) -> ({ id: string, displayName: string, path: string } & ({ isRunning: true, pid: number } | { isRunning: false }))[] — search installed applications, including stopped matches; only running matches carry pid. - cua.getState(options?: { emit?: boolean }) -> { apps: { id: string, displayName: string, isRunning: true, pid: number }[], browsers: [] } — summarize running applications; this is not an accessibility observation. -- cua.getApp(selector: string) -> App — select an application by name, bundle ID, absolute path, or running PID string, obtain consent, launch in the background if needed, and read initial AX state silently; it does not activate the app. +- cua.getApp(selector: string) -> App — select an application by name, bundle ID, absolute path, or running PID string, obtain consent, launch in the background if needed, and return without reading AX or activating the app. - app.getWindows(options?: { emit?: boolean }) -> { windowId: number, appName: string, title: string, bounds: { x: number, y: number, width: number, height: number }, isOnScreen: boolean, onCurrentSpace: boolean | null }[] — enumerate windows with global native bounds; null onCurrentSpace means unknown Space membership. -- app.getWindow(windowId: number) -> Window — select one window and read its initial AX state silently; do this before any window screenshot or coordinate input. +- app.getWindow(windowId: number) -> Window — select a window from the application’s current window list without reading AX; do this before any window screenshot or coordinate input. - app.activate() -> ActionResult — request explicit foreground activation; use only when foreground access is required. - app.show() -> { dispatched: boolean, windowId: number | null, foregroundChanged: boolean | null } — unhide the app and restore its selected, focused, main, or sole window without activation. - window.show() -> { dispatched: boolean, windowId: number | null, foregroundChanged: boolean | null } — unhide and restore that explicit window without activation. @@ -199,7 +199,7 @@ Setup and discovery: ## Accessibility tree Available at app or window level depending on the platform. -Element ids are opaque strings from the latest successful tree; the object facade keeps one current snapshot per app across its repeated app and window objects, a new read replaces it, and a failed read invalidates the previous one. Repeated strings can appear for different elements, so do not reuse an old id blindly, and element operations do not fall back to pointer input or typing. +Read getAccessibilityTree explicitly before using element ids. Selection and screenshots do not read or replace the current AX snapshot. Element ids are opaque strings from the latest successful tree; the object facade keeps one current snapshot per app across its repeated app and window objects, a new read replaces it, and a failed read invalidates the previous one. Repeated strings can appear for different elements, so do not reuse an old id blindly, and element operations do not fall back to pointer input or typing. - getAccessibilityTree(options?: { emit?: boolean, includeMenus?: boolean, details?: boolean }) -> string — read the full tree text; includeMenus expands menu contents and details adds identifiers, native bounds, and complete action names. - listActions(elementId: string) -> string[] — list the actions an element advertises. - invoke(elementId: string) -> ActionResult — invoke the element's default action; dispatched is false when it has none. @@ -223,7 +223,7 @@ Available at app or window level depending on the platform. - paste(text: string, options?: { format?: 'text' | 'html' }) -> ActionResult — deliver text or HTML through the app's Paste command, restoring prior clipboard contents unless another writer changes them; a Paste failure is surfaced without substituting typeText, and PASTE_NOT_CONSUMED means inspect before any retry. - pressKey(key: string, modifiers?: string[]) -> ActionResult — press and release one complete keystroke. For example pressKey('Tab', ['shift']) or pressKey('+'). Key and modifier names are platform-defined strings. Pass key and modifiers separately; do not pack them into a chord string. -Raw tools use { method, params }: include appHandle for app calls and windowId for window calls, and use snapshotId with elementId for element routes. The emit option is facade-only: emit is local output control, so raw callers must follow the raw operation schema for admitted params. getAccessibilityTree and getWindow read through raw getState; getWindows maps to listWindows; getScreenshot maps to screenshot; cua.getState maps to listApps; every other named operation uses the same method name. +Raw tools use { method, params }: include appHandle for app calls and windowId for window calls, and use snapshotId with elementId for element routes. The emit option is facade-only: emit is local output control, so raw callers must follow the raw operation schema for admitted params. getAccessibilityTree maps to getState; getWindow validates against listWindows; getWindows maps to listWindows; getScreenshot maps to screenshot; cua.getState maps to listApps; every other named operation uses the same method name. ``` #### Token 影响 diff --git a/packages/experimental/cua-macos/tests/loader-composition.spec.ts b/packages/experimental/cua-macos/tests/loader-composition.spec.ts index e92f138d87..e1395145d4 100644 --- a/packages/experimental/cua-macos/tests/loader-composition.spec.ts +++ b/packages/experimental/cua-macos/tests/loader-composition.spec.ts @@ -181,7 +181,7 @@ it.skipIf(process.platform === 'win32')('shares one application between retained const screenshot = await peerWindow.getScreenshot({ emit: false }) expect(screenshot).toMatchObject({ mimeType: 'image/png', pixelWidth: 1, pixelHeight: 1, windowId: 7 }) expect(Buffer.from(screenshot.bytes).toString('base64')).toBe(png) - await peerApp.getWindow(8) + await (await peerApp.getWindow(8)).getAccessibilityTree({ emit: false }) const recovery = 'Snapshot belongs to window 8; read getAccessibilityTree on window 7' await expect(peerWindow.click(element('0'))).rejects.toThrow(recovery) await expect(peerWindow.mouseMove(element('0'))).rejects.toThrow(recovery) diff --git a/packages/experimental/cua-macos/tests/persistent-bridge.built.e2e.ts b/packages/experimental/cua-macos/tests/persistent-bridge.built.e2e.ts index b37b79b6c1..61fe3b3bdb 100644 --- a/packages/experimental/cua-macos/tests/persistent-bridge.built.e2e.ts +++ b/packages/experimental/cua-macos/tests/persistent-bridge.built.e2e.ts @@ -74,7 +74,7 @@ it.skipIf(process.platform === 'win32' || !existsSync(bridge) || !existsSync(boo const section = (await ctx.systemPrompt.assemble()).sections.find(value => value.name === 'computer-use:dsh-cua') const initialization = section?.text.match(/Persistent PTC initialization:\n\n```ts\n([\s\S]*?)\n```/u)?.[1] expect(initialization).toContain('/lib/bridge.js') - const first = await run(`${initialization}\nvar app = await cua.getApp("test.fixture"); var win = await app.getWindow(7); await win.click("0"); var shot = await win.getScreenshot(); ({ mimeType: shot.mimeType, pixelWidth: shot.pixelWidth, pixelHeight: shot.pixelHeight, windowId: shot.windowId, byteLength: shot.bytes.length });`, 'first') + const first = await run(`${initialization}\nvar app = await cua.getApp("test.fixture"); var win = await app.getWindow(7); await win.getAccessibilityTree({emit:false}); await win.click("0"); var shot = await win.getScreenshot(); ({ mimeType: shot.mimeType, pixelWidth: shot.pixelWidth, pixelHeight: shot.pixelHeight, windowId: shot.windowId, byteLength: shot.bytes.length });`, 'first') expect(first.error).toBeUndefined() expect(first.value).toEqual({ mimeType: 'image/png', pixelWidth: 1, pixelHeight: 1, windowId: 7, byteLength: 90 }) expect(images).toBe(1) diff --git a/packages/experimental/cua/README.i18n.yaml b/packages/experimental/cua/README.i18n.yaml index f4abb13c43..fffdfaa5c6 100644 --- a/packages/experimental/cua/README.i18n.yaml +++ b/packages/experimental/cua/README.i18n.yaml @@ -15,14 +15,14 @@ en: 5db4d4fea25493b9 zh: 65ab64fc24eb0d09 /deepseek-ai-dsh-cua/use-this-package: - en: ba1f8e6ae51f69d7 - zh: 8f4fde9857bae319 + en: 77dbaaa91d5b92c8 + zh: 94c3a758a91476bd /deepseek-ai-dsh-cua/use-this-package/permission-controls: en: d92d5948a7941e8d zh: d43491ae6521b87e /deepseek-ai-dsh-cua/understand-the-implementation: - en: 2ab74a1ff120459e - zh: 39b3ef198989f133 + en: fbd52f5ae2ca44f8 + zh: 386fb22fa7f7703d /deepseek-ai-dsh-cua/further-exploration: en: c174fb5a00ea3681 zh: 039340042676881d diff --git a/packages/experimental/cua/README.md b/packages/experimental/cua/README.md index c95e4ae8ec..0b3d39fdd1 100644 --- a/packages/experimental/cua/README.md +++ b/packages/experimental/cua/README.md @@ -31,7 +31,7 @@ Application consent supplies English, Chinese, and Japanese display text from `s Mount the [macOS provider](../cua-macos/README.md) to enable native operations on macOS. External PTC hosts import `connectCua` from `@deepseek-ai/dsh-cua/bridge` and supply the current `tools.cua` binding. Source compositions and native preparation belong to the provider. -Use `cua.getApp(selector)` to select an application; it obtains consent, optionally launches a stopped application in the background, and reads its initial accessibility tree silently when the provider supports application-level accessibility. `cua.listApps` and `cua.getState` list running applications; `cua.findApps(query)` searches installed applications, including stopped ones. `app.activate()`, `app.getWindows()`, and `app.getWindow(windowId)` operate on the selected application; `getWindow` returns the object for screenshots and pointer input. Select an application and one of its windows before any capture or pointer input. Call `app.getAccessibilityTree()` to emit the tree. Within one API instance, the facade keeps one current accessibility snapshot per application handle across app and window objects, including repeated selections; each new read invalidates the previous snapshot. Element operations use the latest snapshot ID, so callers use IDs from the latest tree and do not assume an older ID still identifies the same control. Element IDs are opaque strings. +Use `cua.getApp(selector)` to select an application; it obtains consent and optionally launches a stopped application in the background, without reading its accessibility tree. `cua.listApps` and `cua.getState` list running applications; `cua.findApps(query)` searches installed applications, including stopped ones. `app.activate()`, `app.getWindows()`, and `app.getWindow(windowId)` operate on the selected application; `getWindow` validates the ID against the current application window list and returns the object without reading AX. Select an application and one of its windows before any capture or pointer input. Read `app.getAccessibilityTree()` or `window.getAccessibilityTree()` explicitly before element operations. Screenshots remain available when AX reading fails. Within one API instance, the facade keeps one current accessibility snapshot per application handle across app and window objects, including repeated selections; each new read invalidates the previous snapshot. Element operations use the latest snapshot ID, so callers use IDs from the latest tree and do not assume an older ID still identifies the same control. Element IDs are opaque strings. Keyboard methods are available in the scopes a provider declares; the provider owns how input is delivered within each scope. A lost connection invalidates its handles and coordinates. Failed cleanup is reported before that connection is retired; select the application again and inspect fresh state, because earlier input may have partially executed. Screenshot image errors retain the failing stage and attachment error code. Invalid request arguments reject before dispatch with a readable method, parameter and requirement. @@ -87,7 +87,7 @@ The provider calls [installCua](src/index.ts) with its prepared helper, connecti [The operation schema](src/protocol.ts) is the authority for model parameters and separately declared Host-only fields. Native implementations enforce target identity, window geometry, resource limits, and operating-system access at execution. The permission store owns durable consent separately from the native protocol. -One object facade ([src/api.ts](src/api.ts)) builds a shared `CuaSurface` and derives `CuaApp` and `CuaWindow` from it. The derived factories inject scope parameters and the accessibility and keyboard checks for their scope; the shared base does not branch on application-only capabilities. Within one API instance, the facade keeps one current accessibility snapshot per `appHandle` across every app and window object for that handle. `getWindow` reads its initial tree silently; `getApp` does the same when the provider reports application-level accessibility. Each later `getAccessibilityTree()` invalidates the previous snapshot before dispatch, and element operations use the latest snapshot ID. Only a `CuaWindow` captures screenshots or sends pointer input. +One object facade ([src/api.ts](src/api.ts)) builds a shared `CuaSurface` and derives `CuaApp` and `CuaWindow` from it. The derived factories inject scope parameters and the accessibility and keyboard checks for their scope; the shared base does not branch on application-only capabilities. Within one API instance, the facade keeps one current accessibility snapshot per `appHandle` across every app and window object for that handle. Selection leaves the current accessibility snapshot unchanged. Each explicit `getAccessibilityTree()` invalidates the previous snapshot before dispatch, and element operations use the latest snapshot ID. Only a `CuaWindow` captures screenshots or sends pointer input. Provider startup capabilities are fixed at installation: platform, methods, app-level accessibility, keyboard scopes, browser support, and coordinate space. The native `getCapabilities` RPC reports runtime permissions and session lock only; `installCua` merges both layers for the model-visible result. The raw RPC remains a common operation representation rather than a capability inventory, so provider guidance states the actions its platform supports. diff --git a/packages/experimental/cua/README.zh.md b/packages/experimental/cua/README.zh.md index c1c5ed863c..b79cc6fdb5 100644 --- a/packages/experimental/cua/README.zh.md +++ b/packages/experimental/cua/README.zh.md @@ -31,7 +31,7 @@ kind: "package-reference" 挂载 [macOS 提供方](../cua-macos/README.zh.md)以在 macOS 上启用原生操作。外部 PTC 宿主从 `@deepseek-ai/dsh-cua/bridge` 导入 `connectCua`，并提供当前 `tools.cua` 绑定。源码组合与原生组件准备由提供方负责。 -使用 `cua.getApp(selector)` 选择应用；它会获取授权、按需在后台启动已停止的应用，并在 provider 支持应用级辅助功能时静默读取初始辅助功能树。`cua.listApps` 和 `cua.getState` 列出运行中的应用；`cua.findApps(query)` 搜索已安装应用，包括未运行的应用。`app.activate()`、`app.getWindows()` 和 `app.getWindow(windowId)` 操作选定的应用；`getWindow` 返回用于截图和指针输入的对象。进行任何截图或指针输入前，必须选择应用及其一个窗口。调用 `app.getAccessibilityTree()` 输出树。在单个 API 实例内，门面按应用句柄保存一份当前辅助功能快照，该句柄的所有应用对象和窗口对象共享它，包括重复选择；每次新读取都会使上一份快照失效。元素操作使用最新快照 ID，因此调用方应使用最新树中的元素 ID，不应假定旧 ID 仍指向同一控件。元素 ID 是不透明字符串。 +使用 `cua.getApp(selector)` 选择应用；它会获取授权、按需在后台启动已停止的应用，不读取辅助功能树。`cua.listApps` 和 `cua.getState` 列出运行中的应用；`cua.findApps(query)` 搜索已安装应用，包括未运行的应用。`app.activate()`、`app.getWindows()` 和 `app.getWindow(windowId)` 操作选定的应用；`getWindow` 根据当前应用窗口清单验证 ID，然后返回对象，不读取 AX。进行任何截图或指针输入前，必须选择应用及其一个窗口。元素操作前须显式读取 `app.getAccessibilityTree()` 或 `window.getAccessibilityTree()`。AX 读取失败时仍可截图。在单个 API 实例内，门面按应用句柄保存一份当前辅助功能快照，该句柄的所有应用对象和窗口对象共享它，包括重复选择；每次新读取都会使上一份快照失效。元素操作使用最新快照 ID，因此调用方应使用最新树中的元素 ID，不应假定旧 ID 仍指向同一控件。元素 ID 是不透明字符串。 键盘方法可在 provider 声明的作用域中使用；每个作用域内的输入投递由 provider 负责。连接丢失会使其句柄和坐标失效。清理失败会先报告，再释放该连接；应重新选择应用并读取新状态，因为先前的输入可能已部分执行。截图图像错误会保留失败阶段和附件错误码。请求参数校验失败时，在分派前返回包含方法、参数和要求的可读消息。 @@ -87,7 +87,7 @@ Computer Use 详情页的权限卡片、组件列表和来源信息使用 l3 边 [操作 schema](src/protocol.ts) 定义模型参数，并单独声明 Host 专用字段。原生实现在执行时检查目标身份、窗口几何、资源上限和操作系统访问条件。授权存储独立于原生协议保存持久许可。 -对象门面（[src/api.ts](src/api.ts)）构建共享的 `CuaSurface`，并从它派生 `CuaApp` 和 `CuaWindow`。派生工厂注入作用域参数以及该作用域的辅助功能和键盘检查；共享基类不为仅属于应用的能力分支。在单个 API 实例内，门面按 `appHandle` 保存一份当前辅助功能快照，该句柄的所有应用对象和窗口对象共享它。`getWindow` 静默读取初始树；当 provider 报告应用级辅助功能时，`getApp` 也静默读取。之后每次 `getAccessibilityTree()` 在分派前使上一份快照失效，元素操作使用最新快照 ID。只有 `CuaWindow` 可以截图或发送指针输入。 +对象门面（[src/api.ts](src/api.ts)）构建共享的 `CuaSurface`，并从它派生 `CuaApp` 和 `CuaWindow`。派生工厂注入作用域参数以及该作用域的辅助功能和键盘检查；共享基类不为仅属于应用的能力分支。在单个 API 实例内，门面按 `appHandle` 保存一份当前辅助功能快照，该句柄的所有应用对象和窗口对象共享它。选择对象不改变当前辅助功能快照。每次显式调用 `getAccessibilityTree()` 在分派前使上一份快照失效，元素操作使用最新快照 ID。只有 `CuaWindow` 可以截图或发送指针输入。 提供方启动能力在安装时固定：平台、方法、应用级辅助功能、键盘作用域、浏览器支持和坐标空间。原生 `getCapabilities` RPC 只报告运行时权限和会话锁定状态；`installCua` 为模型可见结果合并两层。原生 RPC 仍是通用操作表示，而不是能力清单，因此 provider 指引说明其平台支持的动作。 diff --git a/packages/experimental/cua/package.json b/packages/experimental/cua/package.json index 440f321935..0d62908a8a 100644 --- a/packages/experimental/cua/package.json +++ b/packages/experimental/cua/package.json @@ -107,7 +107,8 @@ "@deepseek-ai/dsh-atomic-write": "workspace:*", "@deepseek-ai/dsh-typert-protocol": "workspace:*", "@deepseek-ai/dsh-computer-use": "workspace:*", - "@deepseek-ai/dsh-storage-domain": "workspace:*" + "@deepseek-ai/dsh-storage-domain": "workspace:*", + "@deepseek-ai/dsh-error": "workspace:*" }, "devDependencies": { "@deepseek-ai/dsh-working-directory": "workspace:*", diff --git a/packages/experimental/cua/src/api.ts b/packages/experimental/cua/src/api.ts index 22039c6591..509b9b8676 100644 --- a/packages/experimental/cua/src/api.ts +++ b/packages/experimental/cua/src/api.ts @@ -1,5 +1,6 @@ /** Object facade over logged native operations; its caller owns evaluator persistence. */ import { Buffer } from 'node:buffer' +import { HarnessError } from '@deepseek-ai/dsh-error' import type { z } from 'zod' import type { JsonValue } from '@deepseek-ai/dsh-util-values' import { ActionNamesResult, ActionResult, AppResult, AppsResult, CapabilitiesResult, FoundAppsResult, HoverResult, Request, ScreenshotResult, ShowResult, StateResult, WindowsResult } from './protocol.ts' @@ -79,9 +80,7 @@ export function createCuaApi(dispatch: CuaDispatch, sink?: CuaObservationSink): if (!request.success) throw new Error(`getApp ${requestIssueMessage(request.error.issues)}`) const capabilities = await currentCapabilities() const { appHandle } = AppResult.parse(await call('getApp', { selector })) - const app = objects.createCuaApp(appHandle, capabilities) - if (capabilities.appLevelAccessibility) await app.getAccessibilityTree({ emit: false }) - return app + return objects.createCuaApp(appHandle, capabilities) }, } return api @@ -259,9 +258,13 @@ function createCuaObjects(call: ReturnType, sink: CuaObser return windows }, async getWindow(windowId): Promise { - const window = createCuaWindow(appHandle, windowId, capabilities) - await window.getAccessibilityTree({ emit: false }) - return window + const parsed = WindowsResult.element.shape.windowId.safeParse(windowId) + if (!parsed.success) throw new HarnessError(`getWindow ${requestIssueMessage(parsed.error.issues, ['params', 'windowId'])}`, 'INVALID_ARGS') + const windows = WindowsResult.parse(await base.call('listWindows', {})) + if (!windows.some(window => window.windowId === windowId)) { + throw new HarnessError('Selected window is unavailable for this application; read getWindows again', 'CUA_WINDOW_UNAVAILABLE') + } + return createCuaWindow(appHandle, windowId, capabilities) }, } } diff --git a/packages/experimental/cua/src/guidance.ts b/packages/experimental/cua/src/guidance.ts index 745ccfe9be..89a32bd21a 100644 --- a/packages/experimental/cua/src/guidance.ts +++ b/packages/experimental/cua/src/guidance.ts @@ -11,9 +11,9 @@ Setup and discovery: - cua.listApps(options?: { emit?: boolean }) -> { id: string, displayName: string, isRunning: true, pid: number }[] — enumerate running applications. - cua.findApps(query: string, options?: { emit?: boolean }) -> ({ id: string, displayName: string, path: string } & ({ isRunning: true, pid: number } | { isRunning: false }))[] — search installed applications, including stopped matches; only running matches carry pid. - cua.getState(options?: { emit?: boolean }) -> { apps: { id: string, displayName: string, isRunning: true, pid: number }[], browsers: [] } — summarize running applications; this is not an accessibility observation. -- cua.getApp(selector: string) -> App — select an application by name, bundle ID, absolute path, or running PID string, obtain consent, launch in the background if needed, and read initial AX state silently; it does not activate the app. +- cua.getApp(selector: string) -> App — select an application by name, bundle ID, absolute path, or running PID string, obtain consent, launch in the background if needed, and return without reading AX or activating the app. - app.getWindows(options?: { emit?: boolean }) -> { windowId: number, appName: string, title: string, bounds: { x: number, y: number, width: number, height: number }, isOnScreen: boolean, onCurrentSpace: boolean | null }[] — enumerate windows with global native bounds; null onCurrentSpace means unknown Space membership. -- app.getWindow(windowId: number) -> Window — select one window and read its initial AX state silently; do this before any window screenshot or coordinate input. +- app.getWindow(windowId: number) -> Window — select a window from the application’s current window list without reading AX; do this before any window screenshot or coordinate input. - app.activate() -> ActionResult — request explicit foreground activation; use only when foreground access is required. - app.show() -> { dispatched: boolean, windowId: number | null, foregroundChanged: boolean | null } — unhide the app and restore its selected, focused, main, or sole window without activation. - window.show() -> { dispatched: boolean, windowId: number | null, foregroundChanged: boolean | null } — unhide and restore that explicit window without activation. @@ -21,7 +21,7 @@ Setup and discovery: ## Accessibility tree Available at app or window level depending on the platform. -Element ids are opaque strings from the latest successful tree; the object facade keeps one current snapshot per app across its repeated app and window objects, a new read replaces it, and a failed read invalidates the previous one. Repeated strings can appear for different elements, so do not reuse an old id blindly, and element operations do not fall back to pointer input or typing. +Read getAccessibilityTree explicitly before using element ids. Selection and screenshots do not read or replace the current AX snapshot. Element ids are opaque strings from the latest successful tree; the object facade keeps one current snapshot per app across its repeated app and window objects, a new read replaces it, and a failed read invalidates the previous one. Repeated strings can appear for different elements, so do not reuse an old id blindly, and element operations do not fall back to pointer input or typing. - getAccessibilityTree(options?: { emit?: boolean, includeMenus?: boolean, details?: boolean }) -> string — read the full tree text; includeMenus expands menu contents and details adds identifiers, native bounds, and complete action names. - listActions(elementId: string) -> string[] — list the actions an element advertises. - invoke(elementId: string) -> ActionResult — invoke the element's default action; dispatched is false when it has none. @@ -45,4 +45,4 @@ Available at app or window level depending on the platform. - paste(text: string, options?: { format?: 'text' | 'html' }) -> ActionResult — deliver text or HTML through the app's Paste command, restoring prior clipboard contents unless another writer changes them; a Paste failure is surfaced without substituting typeText, and PASTE_NOT_CONSUMED means inspect before any retry. - pressKey(key: string, modifiers?: string[]) -> ActionResult — press and release one complete keystroke. For example pressKey('Tab', ['shift']) or pressKey('+'). Key and modifier names are platform-defined strings. Pass key and modifiers separately; do not pack them into a chord string. -Raw tools use { method, params }: include appHandle for app calls and windowId for window calls, and use snapshotId with elementId for element routes. The emit option is facade-only: emit is local output control, so raw callers must follow the raw operation schema for admitted params. getAccessibilityTree and getWindow read through raw getState; getWindows maps to listWindows; getScreenshot maps to screenshot; cua.getState maps to listApps; every other named operation uses the same method name.` +Raw tools use { method, params }: include appHandle for app calls and windowId for window calls, and use snapshotId with elementId for element routes. The emit option is facade-only: emit is local output control, so raw callers must follow the raw operation schema for admitted params. getAccessibilityTree maps to getState; getWindow validates against listWindows; getWindows maps to listWindows; getScreenshot maps to screenshot; cua.getState maps to listApps; every other named operation uses the same method name.` diff --git a/packages/experimental/cua/src/types.ts b/packages/experimental/cua/src/types.ts index b84663f9e2..0e8624b6ea 100644 --- a/packages/experimental/cua/src/types.ts +++ b/packages/experimental/cua/src/types.ts @@ -238,9 +238,9 @@ export interface CuaApp extends CuaSurface { */ getWindows(options?: CuaObservationOptions): Promise /** - * Observe a specific window. + * Select a specific window. * @param windowId - ID obtained from getWindows. - * @returns A window object with its initial accessibility tree read silently; this updates the application's current snapshot. + * @returns A window object from the app’s current window list, without reading or replacing its accessibility snapshot. */ getWindow(windowId: number): Promise } @@ -272,9 +272,9 @@ export interface CuaApi { */ getState(options?: CuaObservationOptions): Promise<{ apps: CuaAppInfo[]; browsers: [] }> /** - * Select an app, obtain consent, launch it in the background if necessary, and silently retain initial AX state. + * Select an app, obtain consent, and launch it in the background if necessary, without reading AX. * @param selector - Exact name, bundle ID, absolute application path or running PID string; ambiguous names reject. - * @returns An application handle tied to this process lifetime; its initial tree read becomes the current snapshot. + * @returns An application handle tied to this process lifetime; call getAccessibilityTree explicitly before element operations. */ getApp(selector: string): Promise } diff --git a/packages/experimental/cua/tests/api.spec.ts b/packages/experimental/cua/tests/api.spec.ts index cbaa419951..fb2bf5a204 100644 --- a/packages/experimental/cua/tests/api.spec.ts +++ b/packages/experimental/cua/tests/api.spec.ts @@ -83,7 +83,7 @@ describe('application discovery', () => { const window = await app.getWindow(23) expect(await window.getScreenshot()).toEqual({ bytes: Uint8Array.from(Buffer.from(PNG, 'base64')), mimeType: 'image/png', pixelWidth: 1, pixelHeight: 1, windowId: 23 }) expect(f.dispatch.mock.calls.map(([operation]) => operation.method)).toEqual([ - 'getCapabilities', 'listApps', 'getApp', 'getState', 'getState', 'screenshot', + 'getCapabilities', 'listApps', 'getApp', 'listWindows', 'screenshot', ]) expect(await app.getWindows()).toEqual([windowInfo(23)]) expect(f.sink.text).not.toHaveBeenCalled() @@ -124,8 +124,11 @@ describe('application discovery', () => { describe('background window restoration', () => { it('shows an app or selected window without dispatching activation or AXRaise', async () => { const f = fixture() + f.setWindows([23, 24]) const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) const window = await app.getWindow(24) + await window.getAccessibilityTree({ emit: false }) f.dispatch.mockClear() expect(await app.show()).toEqual({ ...delivered, windowId: 23, foregroundChanged: false }) expect(await window.show()).toEqual({ ...delivered, windowId: 24, foregroundChanged: false }) @@ -138,6 +141,7 @@ describe('background window restoration', () => { it.each([true, null])('preserves foreground change evidence %j without switching focus back', async (foregroundChanged) => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) f.dispatch.mockClear() f.dispatch.mockResolvedValueOnce({ ...delivered, windowId: null, foregroundChanged }) expect(await app.show()).toEqual({ ...delivered, windowId: null, foregroundChanged }) @@ -147,6 +151,7 @@ describe('background window restoration', () => { it('leaves foreground fallback to the caller after restoration fails', async () => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) f.dispatch.mockClear() f.dispatch.mockRejectedValueOnce(new Error('UNSUPPORTED: Window cannot be restored')) await expect(app.show()).rejects.toThrow('UNSUPPORTED') @@ -157,8 +162,11 @@ describe('background window restoration', () => { describe('mouseMove across observations', () => { it('moves to an AX element then into a popup without inserting clicks or restoring a point', async () => { const f = fixture() + f.setWindows([23, 24]) const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) const window = await app.getWindow(23) + await window.getAccessibilityTree({ emit: false }) f.dispatch.mockClear() expect(await window.mouseMove(element('0'))).toEqual({ ...delivered, hoverRoute: 'accessibility', systemPointerMoved: false }) await window.getAccessibilityTree({ emit: false }) @@ -171,13 +179,14 @@ describe('mouseMove across observations', () => { { method: 'mouseMove', params: { appHandle: 'app-17', snapshotId: 'snapshot-3', windowId: 23, x: 30, y: 40 } }, ]) const other = await app.getWindow(24) + await other.getAccessibilityTree({ emit: false }) await other.mouseMove([5, 6]) expect(f.dispatch).toHaveBeenLastCalledWith({ method: 'mouseMove', params: { appHandle: 'app-17', snapshotId: 'snapshot-4', windowId: 24, x: 5, y: 6, } }) f.dispatch.mockRejectedValueOnce(new Error('WINDOW_CHANGED')) await expect(other.mouseMove([5, 6])).rejects.toThrow('WINDOW_CHANGED') - expect(f.dispatch).toHaveBeenCalledTimes(7) + expect(f.dispatch).toHaveBeenCalledTimes(8) }) }) @@ -185,7 +194,9 @@ describe('pointer buttons and modifiers', () => { it.each([0, 800])('preserves %d ms for window clicks at elements and coordinates', async (holdMs) => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) const window = await app.getWindow(23) + await window.getAccessibilityTree({ emit: false }) for (const target of [element('0'), [10, 20] as const]) { await window.click(target, { holdMs, mouseButton: 'right' }) expect(f.dispatch.mock.calls.at(-1)?.[0]).toMatchObject({ method: 'click', params: { holdMs, mouseButton: 'right' } }) @@ -198,7 +209,9 @@ describe('pointer buttons and modifiers', () => { it('retains middle-button and modifier intent through click and drag dispatch', async () => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) const window = await app.getWindow(23) + await window.getAccessibilityTree({ emit: false }) await window.click(element('0'), { mouseButton: 'middle', modifiers: ['shift'] }) expect(f.dispatch).toHaveBeenLastCalledWith({ method: 'click', params: { appHandle: 'app-17', snapshotId: 'snapshot-2', elementId: '0', windowId: 23, mouseButton: 'middle', modifiers: ['shift'], @@ -223,6 +236,7 @@ describe('text selections', () => { it('binds literal text and optional disambiguation to the latest observed control', async () => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) await app.selectText(element('1'), 'target') expect(f.dispatch).toHaveBeenLastCalledWith({ method: 'selectText', params: { appHandle: 'app-17', snapshotId: 'snapshot-1', elementId: '1', text: 'target', @@ -242,6 +256,7 @@ describe('element action queries', () => { it('returns raw provider action names including attribute-setting operations without reading new state', async () => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) f.dispatch.mockClear() expect(await app.listActions(element('42.12345'))).toEqual(['AXPress', 'AXShowMenu', 'AXTextArea.AXValue.Set']) expect(f.dispatch.mock.calls.map(([operation]) => operation)).toEqual([ @@ -257,6 +272,7 @@ describe('element actions', () => { it('dispatches advertised actions with omitted, null or direct values without wrapping the value', async () => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) f.dispatch.mockClear() expect(await app.performAction(element('42.12345'), 'AXPress')).toEqual(delivered) expect(await app.performAction(element('42.12345'), 'AXPress', null)).toEqual(delivered) @@ -274,6 +290,7 @@ describe('element actions', () => { it('rejects an empty action before dispatch', async () => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) f.dispatch.mockClear() await expect(app.performAction(element('0'), '')).rejects.toThrow('performAction action') expect(f.dispatch).not.toHaveBeenCalled() @@ -284,6 +301,7 @@ describe('element default invocation', () => { it('uses the latest snapshot and opaque element id without reading fresh state', async () => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) f.dispatch.mockClear() expect(await app.invoke(element('42.12345'))).toEqual(delivered) expect(f.dispatch.mock.calls.map(([operation]) => operation)).toEqual([ @@ -295,6 +313,7 @@ describe('element default invocation', () => { it('returns confirmed absence of a default action as un-dispatched evidence', async () => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) f.dispatch.mockResolvedValueOnce({ dispatched: false }) await expect(app.invoke(element('9'))).resolves.toEqual({ dispatched: false }) }) @@ -302,6 +321,7 @@ describe('element default invocation', () => { it('rejects provider failures and malformed dispatch evidence', async () => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) const failure = new Error('PERMISSION_DENIED') f.dispatch.mockRejectedValueOnce(failure) await expect(app.invoke(element('0'))).rejects.toBe(failure) @@ -316,9 +336,12 @@ describe('application and window observations', () => { const f = fixture() f.setWindows([23, 24]) const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) expect(await app.getWindows({ emit: false })).toEqual([windowInfo(23), windowInfo(24)]) const first = await app.getWindow(23) + await first.getAccessibilityTree({ emit: false }) const second = await app.getWindow(24) + await second.getAccessibilityTree({ emit: false }) await app.setValue(element('1'), 'First document') await first.click(element('2')) await second.click(element('3')) @@ -342,9 +365,11 @@ describe('application and window observations', () => { it('keeps separate snapshots for different application handles', async () => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) f.dispatch.mockResolvedValueOnce({ appHandle: 'app-18', name: 'Utility', running: true }) f.dispatch.mockResolvedValueOnce(state({ appHandle: 'app-18', snapshotId: 'other', windows: [] })) const other = await f.api.getApp('Utility') + await other.getAccessibilityTree({ emit: false }) await app.setValue(element('0'), 'Editor') await other.setValue(element('0'), 'Utility') expect(f.dispatch.mock.calls.slice(-2).map(([operation]) => operation)).toEqual([ @@ -356,7 +381,15 @@ describe('application and window observations', () => { it('shares one current snapshot between repeated handles for the same application', async () => { const f = fixture() const first = await f.api.getApp('Editor') + await first.getAccessibilityTree({ emit: false }) const second = await f.api.getApp('Editor') + const window = await second.getWindow(23) + await window.getScreenshot({ emit: false }) + await first.setValue(element('1'), 'Preserved') + expect(f.dispatch).toHaveBeenLastCalledWith({ method: 'setValue', params: { + appHandle: 'app-17', snapshotId: 'snapshot-1', elementId: '1', value: 'Preserved', + } }) + await second.getAccessibilityTree({ emit: false }) expect(second.appHandle).toBe(first.appHandle) await first.setValue(element('1'), 'Shared') expect(f.dispatch).toHaveBeenLastCalledWith({ method: 'setValue', params: { @@ -367,9 +400,11 @@ describe('application and window observations', () => { it('exposes screenshots and pointer input only on explicit windows', async () => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) expect(app).toHaveProperty('getAccessibilityTree') for (const method of ['getScreenshot', 'click', 'mouseMove', 'scroll', 'drag']) expect(app).not.toHaveProperty(method) const window = await app.getWindow(23) + await window.getAccessibilityTree({ emit: false }) for (const method of ['getScreenshot', 'click', 'mouseMove', 'scroll', 'drag']) expect(window).toHaveProperty(method) expect(window).not.toHaveProperty('hover') expect(window).toHaveProperty('mouseMove') @@ -377,13 +412,17 @@ describe('application and window observations', () => { it('attaches the shared snapshot to coordinates only within its window scope', async () => { const f = fixture() + f.setWindows([23, 24]) const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) const first = await app.getWindow(23) + await first.getAccessibilityTree({ emit: false }) await first.click([10, 20]) expect(f.dispatch).toHaveBeenLastCalledWith({ method: 'click', params: { appHandle: 'app-17', windowId: 23, snapshotId: 'snapshot-2', x: 10, y: 20, mouseButton: 'left', clickCount: 1, } }) const second = await app.getWindow(24) + await second.getAccessibilityTree({ emit: false }) await first.click([10, 20]) expect(f.dispatch).toHaveBeenLastCalledWith({ method: 'click', params: { appHandle: 'app-17', windowId: 23, x: 10, y: 20, mouseButton: 'left', clickCount: 1, @@ -398,8 +437,10 @@ describe('application and window observations', () => { it('keeps a validated AXless window usable for screenshots and coordinate input', async () => { const f = fixture() const app = await f.api.getApp('Editor') - f.dispatch.mockResolvedValueOnce(state({ windowId: 23, snapshotId: 'axless', tree: '', elements: [] })) + await app.getAccessibilityTree({ emit: false }) const window = await app.getWindow(23) + f.dispatch.mockResolvedValueOnce(state({ windowId: 23, snapshotId: 'axless', tree: '', elements: [] })) + await window.getAccessibilityTree({ emit: false }) expect(f.sink.text).not.toHaveBeenCalled() await expect(window.getScreenshot()).resolves.toEqual({ bytes: Uint8Array.from(Buffer.from(PNG, 'base64')), mimeType: 'image/png', pixelWidth: 1, pixelHeight: 1, windowId: 23 }) await window.click([10, 20]) @@ -414,11 +455,12 @@ describe('application and window observations', () => { it.each(['WINDOW_UNAVAILABLE', 'PERMISSION_DENIED', 'STALE_APP', 'SESSION_LOCKED'])('propagates %s while selecting a window without attempting image or input fallback', async (code) => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) const failure = new Error(code) f.dispatch.mockRejectedValueOnce(failure) const calls = f.dispatch.mock.calls.length await expect(app.getWindow(23)).rejects.toBe(failure) - expect(f.dispatch.mock.calls.slice(calls).map(([operation]) => operation.method)).toEqual(['getState']) + expect(f.dispatch.mock.calls.slice(calls).map(([operation]) => operation.method)).toEqual(['listWindows']) }) it('captures and clicks the explicitly selected window among several', async () => { @@ -431,8 +473,10 @@ describe('application and window observations', () => { ] f.setWindowRecords(windows) const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) expect(await app.getWindows({ emit: false })).toEqual(windows) const window = await app.getWindow(23) + await window.getAccessibilityTree({ emit: false }) await window.getScreenshot() expect(f.dispatch).toHaveBeenLastCalledWith({ method: 'screenshot', params: { appHandle: 'app-17', windowId: 23 } }) await window.click([10, 20]) @@ -440,6 +484,7 @@ describe('application and window observations', () => { appHandle: 'app-17', windowId: 23, snapshotId: 'snapshot-2', x: 10, y: 20, mouseButton: 'left', clickCount: 1, } }) const hidden = await app.getWindow(25) + await hidden.getAccessibilityTree({ emit: false }) await hidden.click([10, 20]) expect(f.dispatch).toHaveBeenLastCalledWith({ method: 'click', params: { appHandle: 'app-17', windowId: 25, snapshotId: 'snapshot-3', x: 10, y: 20, mouseButton: 'left', clickCount: 1, @@ -471,6 +516,7 @@ describe('application and window observations', () => { it('returns provider window records without process filtering', async () => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) const observed = windowInfo(24) f.dispatch.mockResolvedValueOnce(state({ windows: [observed] })) await expect(app.getAccessibilityTree()).resolves.toBe('Changed tree') @@ -496,11 +542,11 @@ describe('application and window observations', () => { { method: 'typeText', params: { appHandle: 'app-17', text: 'keyboard without accessibility' } }, ]) const window = await app.getWindow(23) - await expect(window.getAccessibilityTree({ emit: false })).resolves.toBe('Tree 2') + await expect(window.getAccessibilityTree({ emit: false })).resolves.toBe('Tree 1') await expect(app.getAccessibilityTree()).rejects.toThrow('Application-level accessibility is not supported on current platform; observe a window instead.') await expect(window.invoke(element('0'))).resolves.toEqual(delivered) expect(f.dispatch).toHaveBeenLastCalledWith({ method: 'invoke', params: { - appHandle: 'app-17', windowId: 23, snapshotId: 'snapshot-2', elementId: '0', + appHandle: 'app-17', windowId: 23, snapshotId: 'snapshot-1', elementId: '0', } }) }) @@ -508,7 +554,9 @@ describe('application and window observations', () => { const f = fixture() f.setCapabilities({ ...capabilities, keyboard: ['app', 'window'] }) const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) const window = await app.getWindow(23) + await window.getAccessibilityTree({ emit: false }) f.dispatch.mockClear() await window.typeText('hello') await window.paste('bold', { format: 'html' }) @@ -530,7 +578,9 @@ describe('application and window observations', () => { const f = fixture() f.setCapabilities({ ...capabilities, keyboard: ['window'] }) const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) const window = await app.getWindow(23) + await window.getAccessibilityTree({ emit: false }) f.dispatch.mockClear() await expect(app.typeText('app text')).rejects.toThrow('App-scoped keyboard input is not supported on current platform.') await expect(app.paste('app text')).rejects.toThrow('App-scoped keyboard input is not supported on current platform.') @@ -544,7 +594,9 @@ describe('application and window observations', () => { it('accepts coordinate input without a current snapshot', async () => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) const window = await app.getWindow(23) + await window.getAccessibilityTree({ emit: false }) f.dispatch.mockRejectedValueOnce(new Error('Native app is busy')) await expect(window.getAccessibilityTree()).rejects.toThrow('Native app is busy') await window.click([10, 20]) @@ -556,7 +608,9 @@ describe('application and window observations', () => { it('clears a failed refresh and requires a new read before element operations', async () => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) const window = await app.getWindow(23) + await window.getAccessibilityTree({ emit: false }) const failure = new Error('Native app is busy') f.dispatch.mockRejectedValueOnce(failure) await expect(app.getAccessibilityTree()).rejects.toBe(failure) @@ -575,11 +629,54 @@ describe('application and window observations', () => { }) describe('screenshots', () => { + it('rejects a missing or foreign window without reading AX or taking a screenshot', async () => { + const f = fixture() + const app = await f.api.getApp('Editor') + await expect(app.getWindow(24)).rejects.toMatchObject({ name: 'HarnessError', code: 'CUA_WINDOW_UNAVAILABLE' }) + f.setWindows([]) + await expect(app.getWindow(23)).rejects.toMatchObject({ name: 'HarnessError', code: 'CUA_WINDOW_UNAVAILABLE' }) + expect(f.dispatch.mock.calls.map(([request]) => request.method)).toEqual(['getCapabilities', 'getApp', 'listWindows', 'listWindows']) + await expect(app.getWindow(0)).rejects.toMatchObject({ name: 'HarnessError', code: 'INVALID_ARGS', message: 'getWindow windowId: Too small: expected number to be >0' }) + expect(f.dispatch).toHaveBeenCalledTimes(4) + }) + + it.each(['APP_ACCESS_REVOKED', 'WINDOW_UNAVAILABLE', 'STALE_APP'])('preserves %s when capturing a previously selected window', async (code) => { + const f = fixture() + const app = await f.api.getApp('Editor') + const window = await app.getWindow(23) + f.dispatch.mockRejectedValueOnce(new Error(code)) + await expect(window.getScreenshot()).rejects.toThrow(code) + expect(f.sink.image).not.toHaveBeenCalled() + }) + + it.each(['application', 'window'])('captures the first screenshot when %s AX reads fail', async (scope) => { + const f = fixture() + const dispatch = f.dispatch.getMockImplementation()! + const failure = new Error('AX read failed') + f.dispatch.mockImplementation(async (request) => { + if (request.method === 'getState' && (scope === 'application' ? request.params.windowId === undefined : request.params.windowId !== undefined)) throw failure + return dispatch(request) + }) + const app = await f.api.getApp('Editor') + expect(await app.getWindows({ emit: false })).toEqual([windowInfo(23)]) + const window = await app.getWindow(23) + const target = scope === 'application' ? app : window + await expect(target.getAccessibilityTree()).rejects.toBe(failure) + await expect(window.getScreenshot()).resolves.toMatchObject({ windowId: 23, pixelWidth: 1 }) + await expect(window.invoke(element('0'))).rejects.toThrow('Read getAccessibilityTree') + await window.click([0, 0]) + expect(f.dispatch).toHaveBeenLastCalledWith({ method: 'click', params: { + appHandle: 'app-17', windowId: 23, x: 0, y: 0, mouseButton: 'left', clickCount: 1, + } }) + }) + it('returns screenshot metadata independently of input point bounds and supports selective output', async () => { const f = fixture() f.setWindows([23, 24]) const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) const window = await app.getWindow(24) + await window.getAccessibilityTree({ emit: false }) f.sink.text.mockClear() const tree = await window.getAccessibilityTree() const image = await window.getScreenshot() @@ -597,7 +694,9 @@ describe('screenshots', () => { it('rejects a screenshot from another window before emitting its image', async () => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) const window = await app.getWindow(23) + await window.getAccessibilityTree({ emit: false }) f.dispatch.mockResolvedValueOnce(screenshot({ windowId: 24 })) await expect(window.getScreenshot()).rejects.toThrow('belongs to another window') expect(f.sink.image).not.toHaveBeenCalled() @@ -606,7 +705,9 @@ describe('screenshots', () => { it.each(['image/jpeg', 'image/webp', 'image/gif'])('preserves Host-admitted %s bytes and metadata in the return value and image sink', async (mimeType) => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) const window = await app.getWindow(23) + await window.getAccessibilityTree({ emit: false }) const data = Buffer.from('admitted request image').toString('base64') f.dispatch.mockResolvedValueOnce(screenshot({ mimeType, data, pixelWidth: 320, pixelHeight: 240 })) const image = await window.getScreenshot() @@ -620,7 +721,9 @@ describe('input actions', () => { it('passes explicit relative mode to every coordinate gesture and rejects it on element ids', async () => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) const window = await app.getWindow(23) + await window.getAccessibilityTree({ emit: false }) await window.click([500, 250], { relative: true }) await window.mouseMove([500, 250], { relative: true }) await window.drag([0, 0], [999, 999], { relative: true }) @@ -638,7 +741,9 @@ describe('input actions', () => { it('keeps the selected window when scroll options include another target identity', async () => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) const window = await app.getWindow(23) + await window.getAccessibilityTree({ emit: false }) const options = { x: 10, y: 20, deltaLinesX: 0, deltaLinesY: 1, appHandle: 'another-app', snapshotId: 'another-snapshot', windowId: 24, @@ -654,7 +759,9 @@ describe('input actions', () => { it('dispatches window-point gestures and reports delivery without claiming the action succeeded', async () => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) const window = await app.getWindow(23) + await window.getAccessibilityTree({ emit: false }) expect(await window.click([10, 20], { mouseButton: 'right' })).toEqual(delivered) expect(await window.click([11, 21], { clickCount: 2 })).toEqual(delivered) expect(await window.scroll({ x: 10, y: 20, deltaLinesX: -2, deltaLinesY: 3 })).toEqual(delivered) @@ -685,7 +792,6 @@ describe('input actions', () => { ['empty key', ({ app }) => app.pressKey('')], ] it.each<[string, (surfaces: Surfaces) => Promise]>([ - ['getState windowId: Too small: expected number to be >0', ({ app }) => app.getWindow(0)], ['selectText text: Too small: expected string to have >=1 characters', ({ app }) => app.selectText(element('0'), '')], ['mouseMove x: Invalid input: expected number, received NaN', ({ window }) => window.mouseMove([NaN, 0])], ['mouseMove x: Too big: expected number to be <=999', ({ window }) => window.mouseMove([1000, 0], { relative: true })], @@ -693,7 +799,9 @@ describe('input actions', () => { ])('reports a readable validation failure: %s', async (message, run) => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) const window = await app.getWindow(23) + await window.getAccessibilityTree({ emit: false }) f.dispatch.mockClear() await expect(run({ app, window })).rejects.toMatchObject({ name: 'Error', message }) expect(f.dispatch).not.toHaveBeenCalled() @@ -702,7 +810,9 @@ describe('input actions', () => { it.each(invalidActions)('rejects %s without invoking native input', async (_label, run) => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) const window = await app.getWindow(23) + await window.getAccessibilityTree({ emit: false }) const calls = f.dispatch.mock.calls.length await expect(run({ app, window })).rejects.toThrow() expect(f.dispatch).toHaveBeenCalledTimes(calls) @@ -716,6 +826,7 @@ describe('input actions', () => { ])('forwards literal key %s with modifiers %j without splitting', async (key, modifiers, expected) => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) if (modifiers === undefined) await app.pressKey(key) else await app.pressKey(key, modifiers) expect(f.dispatch).toHaveBeenLastCalledWith({ method: 'pressKey', params: { appHandle: 'app-17', key, modifiers: expected } }) @@ -724,6 +835,7 @@ describe('input actions', () => { it('forwards the literal key unsplit in one RPC without mutating modifiers', async () => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) const modifiers: string[] = ['control', 'shift'] f.dispatch.mockClear() await app.pressKey('+', modifiers) @@ -737,6 +849,7 @@ describe('input actions', () => { it('forwards arbitrary provider-defined key and modifier names in one RPC', async () => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) f.dispatch.mockClear() await app.pressKey('PlatformKey', ['platform-modifier']) expect(f.dispatch).toHaveBeenCalledTimes(1) @@ -748,6 +861,7 @@ describe('input actions', () => { it('passes literal text and keys to the selected app identity', async () => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) await app.typeText('中文\nline two') await app.paste('粘贴\nplain text') await app.paste('bold', { format: 'html' }) @@ -765,7 +879,9 @@ describe('input actions', () => { it('rejects keyboard input on a window surface without dispatching an app-level fallback', async () => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) const window = await app.getWindow(23) + await window.getAccessibilityTree({ emit: false }) f.dispatch.mockClear() for (const run of [ () => window.typeText('hello'), @@ -787,6 +903,7 @@ describe('input actions', () => { const f = fixture() f.setCapabilities({ ...capabilities, keyboard: ['app', 'window'] }) const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) const surface = scope === 'window' ? await app.getWindow(23) : app f.dispatch.mockClear() f.dispatch.mockResolvedValueOnce(delivered) @@ -800,6 +917,7 @@ describe('input actions', () => { it('returns an un-dispatched key-event result from one RPC without retrying', async () => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) f.dispatch.mockClear() f.dispatch.mockResolvedValueOnce({ dispatched: false }) await expect(app.sendKeyEvent('Command', 'down')).resolves.toEqual({ dispatched: false }) @@ -829,7 +947,9 @@ describe('input actions', () => { it('retains the snapshot identity for element right and double clicks', async () => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) const window = await app.getWindow(23) + await window.getAccessibilityTree({ emit: false }) await window.click(element('12'), { mouseButton: 'right' }) await window.click(element('13'), { clickCount: 2 }) expect(f.dispatch.mock.calls.slice(-2).map(([operation]) => operation)).toEqual([ @@ -841,7 +961,9 @@ describe('input actions', () => { it('dispatches an opaque provider element id unchanged', async () => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) const window = await app.getWindow(23) + await window.getAccessibilityTree({ emit: false }) await window.click(element('42.12345')) await window.mouseMove(element('42.12345')) expect(f.dispatch.mock.calls.slice(-2).map(([operation]) => operation)).toEqual([ @@ -853,7 +975,7 @@ describe('input actions', () => { it('does not activate on selection and dispatches only an explicit activation request', async () => { const f = fixture() const app = await f.api.getApp('Editor') - expect(f.dispatch.mock.calls.map(([call]) => call.method)).toEqual(['getCapabilities', 'getApp', 'getState']) + expect(f.dispatch.mock.calls.map(([call]) => call.method)).toEqual(['getCapabilities', 'getApp']) await app.activate() expect(f.dispatch).toHaveBeenLastCalledWith({ method: 'activate', params: { appHandle: 'app-17' } }) }) @@ -861,14 +983,16 @@ describe('input actions', () => { it('propagates native failures once and rejects malformed delivery claims', async () => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) const window = await app.getWindow(23) + await window.getAccessibilityTree({ emit: false }) const failure = new Error('Application was restarted; select it again') f.dispatch.mockRejectedValueOnce(failure) await expect(app.typeText('hello')).rejects.toBe(failure) - expect(f.dispatch).toHaveBeenCalledTimes(5) + expect(f.dispatch).toHaveBeenCalledTimes(6) f.dispatch.mockResolvedValueOnce({ dispatched: 'yes' }) await expect(window.click(element('0'))).rejects.toThrow() - expect(f.dispatch).toHaveBeenCalledTimes(6) + expect(f.dispatch).toHaveBeenCalledTimes(7) }) it('rejects invalid selection inputs before dispatch', async () => { @@ -876,6 +1000,7 @@ describe('input actions', () => { await expect(f.api.getApp('')).rejects.toThrow() expect(f.dispatch).not.toHaveBeenCalled() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) const calls = f.dispatch.mock.calls.length await expect(app.getWindow(0)).rejects.toThrow() expect(f.dispatch).toHaveBeenCalledTimes(calls) diff --git a/packages/experimental/cua/tests/bridge.spec.ts b/packages/experimental/cua/tests/bridge.spec.ts index 14a36baaba..7bba3c4e0b 100644 --- a/packages/experimental/cua/tests/bridge.spec.ts +++ b/packages/experimental/cua/tests/bridge.spec.ts @@ -27,6 +27,7 @@ it('uses the current tool binding when a retained app is called by another execu const first = vi.fn(async (operation: CuaOperation): Promise => { if (operation.method === 'getCapabilities') return capabilities if (operation.method === 'getApp') return selected + if (operation.method === 'listWindows') return snapshot.windows if (operation.method === 'getState') return { ...snapshot, windowId: operation.params.windowId ?? null } return snapshot }) @@ -35,6 +36,7 @@ it('uses the current tool binding when a retained app is called by another execu const cua = connectCua(() => current, text) const app = await cua.getApp('Editor') const window = await app.getWindow(23) + await window.getAccessibilityTree({ emit: false }) const second = vi.fn(async (): Promise => ({ dispatched: true })) current = second @@ -69,6 +71,7 @@ it('returns screenshot bytes and metadata with one image-bearing tool call and n const tool = vi.fn(async (operation: CuaOperation): Promise => { if (operation.method === 'getCapabilities') return capabilities if (operation.method === 'getApp') return selected + if (operation.method === 'listWindows') return snapshot.windows if (operation.method === 'getState') return { ...snapshot, windowId: operation.params.windowId ?? null } return { mimeType: 'image/png', data: PNG, pixelWidth: 1, pixelHeight: 1, windowId: 23, bounds, coordinateSpace: 'screenshot-pixels', @@ -80,6 +83,6 @@ it('returns screenshot bytes and metadata with one image-bearing tool call and n text.mockClear() expect(await window.getScreenshot()).toEqual({ bytes: Uint8Array.from(Buffer.from(PNG, 'base64')), mimeType: 'image/png', pixelWidth: 1, pixelHeight: 1, windowId: 23 }) - expect(tool.mock.calls.map(([operation]) => operation.method)).toEqual(['getCapabilities', 'getApp', 'getState', 'getState', 'screenshot']) + expect(tool.mock.calls.map(([operation]) => operation.method)).toEqual(['getCapabilities', 'getApp', 'listWindows', 'screenshot']) expect(text).not.toHaveBeenCalled() }) diff --git a/packages/experimental/cua/tsconfig.host.json b/packages/experimental/cua/tsconfig.host.json index 2386f83640..4bf0b7ddda 100644 --- a/packages/experimental/cua/tsconfig.host.json +++ b/packages/experimental/cua/tsconfig.host.json @@ -74,6 +74,9 @@ }, { "path": "../../boot/config-editor" + }, + { + "path": "../../util/error" } ], "exclude": [ diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index d016fe8deb..83d9bd763d 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -9800,6 +9800,9 @@ importers: '@deepseek-ai/dsh-computer-use': specifier: workspace:* version: link:../../computer-use/computer-use + '@deepseek-ai/dsh-error': + specifier: workspace:* + version: link:../../util/error '@deepseek-ai/dsh-storage-domain': specifier: workspace:* version: link:../../storage/storage-domain diff --git a/snapshots/session/cua-native/system-prompt.expected.md b/snapshots/session/cua-native/system-prompt.expected.md index 0b9c1b29f6..04fb520f49 100644 --- a/snapshots/session/cua-native/system-prompt.expected.md +++ b/snapshots/session/cua-native/system-prompt.expected.md @@ -46,9 +46,9 @@ Setup and discovery: - cua.listApps(options?: { emit?: boolean }) -> { id: string, displayName: string, isRunning: true, pid: number }[] — enumerate running applications. - cua.findApps(query: string, options?: { emit?: boolean }) -> ({ id: string, displayName: string, path: string } & ({ isRunning: true, pid: number } | { isRunning: false }))[] — search installed applications, including stopped matches; only running matches carry pid. - cua.getState(options?: { emit?: boolean }) -> { apps: { id: string, displayName: string, isRunning: true, pid: number }[], browsers: [] } — summarize running applications; this is not an accessibility observation. -- cua.getApp(selector: string) -> App — select an application by name, bundle ID, absolute path, or running PID string, obtain consent, launch in the background if needed, and read initial AX state silently; it does not activate the app. +- cua.getApp(selector: string) -> App — select an application by name, bundle ID, absolute path, or running PID string, obtain consent, launch in the background if needed, and return without reading AX or activating the app. - app.getWindows(options?: { emit?: boolean }) -> { windowId: number, appName: string, title: string, bounds: { x: number, y: number, width: number, height: number }, isOnScreen: boolean, onCurrentSpace: boolean | null }[] — enumerate windows with global native bounds; null onCurrentSpace means unknown Space membership. -- app.getWindow(windowId: number) -> Window — select one window and read its initial AX state silently; do this before any window screenshot or coordinate input. +- app.getWindow(windowId: number) -> Window — select a window from the application’s current window list without reading AX; do this before any window screenshot or coordinate input. - app.activate() -> ActionResult — request explicit foreground activation; use only when foreground access is required. - app.show() -> { dispatched: boolean, windowId: number | null, foregroundChanged: boolean | null } — unhide the app and restore its selected, focused, main, or sole window without activation. - window.show() -> { dispatched: boolean, windowId: number | null, foregroundChanged: boolean | null } — unhide and restore that explicit window without activation. @@ -56,7 +56,7 @@ Setup and discovery: ## Accessibility tree Available at app or window level depending on the platform. -Element ids are opaque strings from the latest successful tree; the object facade keeps one current snapshot per app across its repeated app and window objects, a new read replaces it, and a failed read invalidates the previous one. Repeated strings can appear for different elements, so do not reuse an old id blindly, and element operations do not fall back to pointer input or typing. +Read getAccessibilityTree explicitly before using element ids. Selection and screenshots do not read or replace the current AX snapshot. Element ids are opaque strings from the latest successful tree; the object facade keeps one current snapshot per app across its repeated app and window objects, a new read replaces it, and a failed read invalidates the previous one. Repeated strings can appear for different elements, so do not reuse an old id blindly, and element operations do not fall back to pointer input or typing. - getAccessibilityTree(options?: { emit?: boolean, includeMenus?: boolean, details?: boolean }) -> string — read the full tree text; includeMenus expands menu contents and details adds identifiers, native bounds, and complete action names. - listActions(elementId: string) -> string[] — list the actions an element advertises. - invoke(elementId: string) -> ActionResult — invoke the element's default action; dispatched is false when it has none. @@ -80,6 +80,6 @@ Available at app or window level depending on the platform. - paste(text: string, options?: { format?: 'text' | 'html' }) -> ActionResult — deliver text or HTML through the app's Paste command, restoring prior clipboard contents unless another writer changes them; a Paste failure is surfaced without substituting typeText, and PASTE_NOT_CONSUMED means inspect before any retry. - pressKey(key: string, modifiers?: string[]) -> ActionResult — press and release one complete keystroke. For example pressKey('Tab', ['shift']) or pressKey('+'). Key and modifier names are platform-defined strings. Pass key and modifiers separately; do not pack them into a chord string. -Raw tools use { method, params }: include appHandle for app calls and windowId for window calls, and use snapshotId with elementId for element routes. The emit option is facade-only: emit is local output control, so raw callers must follow the raw operation schema for admitted params. getAccessibilityTree and getWindow read through raw getState; getWindows maps to listWindows; getScreenshot maps to screenshot; cua.getState maps to listApps; every other named operation uses the same method name. +Raw tools use { method, params }: include appHandle for app calls and windowId for window calls, and use snapshotId with elementId for element routes. The emit option is facade-only: emit is local output control, so raw callers must follow the raw operation schema for admitted params. getAccessibilityTree maps to getState; getWindow validates against listWindows; getWindows maps to listWindows; getScreenshot maps to screenshot; cua.getState maps to listApps; every other named operation uses the same method name. Maximum click holdMs: 5000 ms. diff --git a/snapshots/session/cua-native/workspace/native-helper.mjs b/snapshots/session/cua-native/workspace/native-helper.mjs index 1e148d7ab5..c39f62f769 100755 --- a/snapshots/session/cua-native/workspace/native-helper.mjs +++ b/snapshots/session/cua-native/workspace/native-helper.mjs @@ -162,6 +162,7 @@ function createApplicationState(socket, record, application, captureBroker, conn if (method === 'listWindows') return [window, axlessWindow] if (method === 'getState') { snapshots.delete(params.windowId ?? null) + if (existsSync(new URL('./fixture-ax-failure', import.meta.url))) refuse('NATIVE_ERROR', 'Fixture accessibility read failed') if (observationChanged) { observationChanged = false refuse('STALE_SNAPSHOT', 'Accessibility tree changed during observation; read fresh getAccessibilityTree') diff --git a/snapshots/session/cua-ptc/consent-fixture.mjs b/snapshots/session/cua-ptc/consent-fixture.mjs index 85e83bcf69..6fdbdbbfc2 100644 --- a/snapshots/session/cua-ptc/consent-fixture.mjs +++ b/snapshots/session/cua-ptc/consent-fixture.mjs @@ -1,5 +1,5 @@ /** Own a private native substitute for the PTC result replay. */ -import { chmod, copyFile, unlink } from 'node:fs/promises' +import { chmod, copyFile, unlink, writeFile } from 'node:fs/promises' import { join } from 'node:path' import { startFixtureHost } from '../cua-native/fixture-host.mjs' export const name = 'cua-ptc-fixture' @@ -7,6 +7,10 @@ export const inject = ['userQuestions'] /** Prepare the helper and retire its service before removing the private executable. */ export async function apply(ctx) { + const bridge = join(process.cwd(), 'cua-bridge.mjs') + const face = process.env.DSH_EXAMPLE_MODE === 'src' ? 'src/bridge.ts' : 'lib/bridge.js' + await writeFile(bridge, `export { connectCua } from ${JSON.stringify(new URL('../../../packages/experimental/cua/' + face, import.meta.url).href)}\n`) + ctx.effect(() => () => unlink(bridge)) const helper = join(process.cwd(), 'native-helper.mjs') await copyFile(new URL('../cua-native/workspace/native-helper.mjs', import.meta.url), helper) await chmod(helper, 0o700) diff --git a/snapshots/session/cua-ptc/replay.override.json b/snapshots/session/cua-ptc/replay.override.json index 63e4009990..5ea62fee03 100644 --- a/snapshots/session/cua-ptc/replay.override.json +++ b/snapshots/session/cua-ptc/replay.override.json @@ -32,6 +32,39 @@ } ] }, + { + "kind": "chunks", + "chunks": [ + { + "type": "block-start", + "index": 0, + "blockType": "tool-call" + }, + { + "type": "block-end", + "index": 0, + "block": { + "type": "tool-call", + "id": "cua-ptc-ax-failure", + "name": "run_code", + "arguments": "{\"code\":\"const { writeFile, unlink } = await import('node:fs/promises');\\nconst { resolve } = await import('node:path');\\nconst { pathToFileURL } = await import('node:url');\\nconst { connectCua } = await import(pathToFileURL(resolve('cua-bridge.mjs')).href);\\nawait writeFile('fixture-ax-failure', '');\\ntry {\\n const cua = connectCua(() => tools.cua);\\n const app = await cua.getApp('test.fixture');\\n const windows = await app.getWindows({ emit: false });\\n const window = await app.getWindow(windows[0].windowId);\\n const failures = [];\\n for (const surface of [app, window]) {\\n try { await surface.getAccessibilityTree(); throw new Error('Expected AX failure'); }\\n catch (error) { if (!error.message.includes('Fixture accessibility read failed')) throw error; failures.push(error.message); }\\n }\\n const shot = await window.getScreenshot();\\n let elementRefused = false;\\n try { await window.invoke('0'); }\\n catch (error) { if (!error.message.includes('Read getAccessibilityTree')) throw error; elementRefused = true; }\\n if (!elementRefused) throw new Error('Element input accepted without an AX snapshot');\\n return { axFailures: failures.length, screenshotWindow: shot.windowId, pngBytes: shot.bytes.length, elementRefused };\\n} finally { await unlink('fixture-ax-failure'); }\",\"description\":\"Capture through the object API despite application and window AX failures\"}" + } + }, + { + "type": "usage", + "usage": { + "inputTokens": 3, + "outputTokens": 3 + } + }, + { + "type": "finish", + "reason": { + "kind": "tool-calls" + } + } + ] + }, { "kind": "chunks", "chunks": [ @@ -45,7 +78,7 @@ "index": 0, "block": { "type": "text", - "text": "Received native CUA values and one PNG screenshot through PTC. The PNG is 90 bytes with signature 89504e470d0a1a0a." + "text": "Received native CUA values and PNG screenshots through PTC. The object API captured a 90-byte PNG after two AX read failures and refused element input without an accessibility snapshot." } }, { diff --git a/snapshots/session/cua-ptc/session.v4.jsonl b/snapshots/session/cua-ptc/session.v4.jsonl index 87791413b2..f9d25d914b 100644 --- a/snapshots/session/cua-ptc/session.v4.jsonl +++ b/snapshots/session/cua-ptc/session.v4.jsonl @@ -12,7 +12,7 @@ {"type":"request/header","data":{"header":{"config":{"provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"tools":"{{tools}}"},"reason":"initial"}} {"type":"request/context","data":{"provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"}} {"type":"session/title","data":{"title":"Use tools.cua from Node PTC","messageSeqs":[8],"source":{"kind":"fallback"}}} -{"type":"assistant/message","data":{"turn":1,"step":1,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-ptc-native-result","name":"run_code","arguments":"{\"code\":\"const { Buffer } = await import('node:buffer');\\nconst app = await tools.cua({ method: 'getApp', params: { selector: 'test.fixture' } });\\nconst state = await tools.cua({ method: 'getState', params: { appHandle: app.appHandle, windowId: 7 } });\\nconst shot = await tools.cua({ method: 'screenshot', params: { appHandle: app.appHandle, windowId: 7 } });\\nconst bytes = Buffer.from(shot.data, 'base64');\\nreturn { appHandle: app.appHandle, hasIncrement: state.tree.includes('Increment'), pngBytes: bytes.length, signature: bytes.subarray(0, 8).toString('hex'), imageWidth: shot.attachment.width, imageHeight: shot.attachment.height };\",\"description\":\"Read native CUA values and receive one screenshot\"}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:4}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791864698727,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791864698727,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-ptc-native-result","name":"run_code","arguments":"{\"code\":\"const { Buffer } = await import('node:buffer');\\nconst app = await tools.cua({ method: 'getApp', params: { selector: 'test.fixture' } });\\nconst state = await tools.cua({ method: 'getState', params: { appHandle: app.appHandle, windowId: 7 } });\\nconst shot = await tools.cua({ method: 'screenshot', params: { appHandle: app.appHandle, windowId: 7 } });\\nconst bytes = Buffer.from(shot.data, 'base64');\\nreturn { appHandle: app.appHandle, hasIncrement: state.tree.includes('Increment'), pngBytes: bytes.length, signature: bytes.subarray(0, 8).toString('hex'), imageWidth: shot.attachment.width, imageHeight: shot.attachment.height };\",\"description\":\"Read native CUA values and receive one screenshot\"}"}}},{"type":"chunk","time":1791864698727,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791864698727,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} +{"type":"assistant/message","data":{"turn":1,"step":1,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-ptc-native-result","name":"run_code","arguments":"{\"code\":\"const { Buffer } = await import('node:buffer');\\nconst app = await tools.cua({ method: 'getApp', params: { selector: 'test.fixture' } });\\nconst state = await tools.cua({ method: 'getState', params: { appHandle: app.appHandle, windowId: 7 } });\\nconst shot = await tools.cua({ method: 'screenshot', params: { appHandle: app.appHandle, windowId: 7 } });\\nconst bytes = Buffer.from(shot.data, 'base64');\\nreturn { appHandle: app.appHandle, hasIncrement: state.tree.includes('Increment'), pngBytes: bytes.length, signature: bytes.subarray(0, 8).toString('hex'), imageWidth: shot.attachment.width, imageHeight: shot.attachment.height };\",\"description\":\"Read native CUA values and receive one screenshot\"}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:4}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1794024973829,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1794024973829,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-ptc-native-result","name":"run_code","arguments":"{\"code\":\"const { Buffer } = await import('node:buffer');\\nconst app = await tools.cua({ method: 'getApp', params: { selector: 'test.fixture' } });\\nconst state = await tools.cua({ method: 'getState', params: { appHandle: app.appHandle, windowId: 7 } });\\nconst shot = await tools.cua({ method: 'screenshot', params: { appHandle: app.appHandle, windowId: 7 } });\\nconst bytes = Buffer.from(shot.data, 'base64');\\nreturn { appHandle: app.appHandle, hasIncrement: state.tree.includes('Increment'), pngBytes: bytes.length, signature: bytes.subarray(0, 8).toString('hex'), imageWidth: shot.attachment.width, imageHeight: shot.attachment.height };\",\"description\":\"Read native CUA values and receive one screenshot\"}"}}},{"type":"chunk","time":1794024973829,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1794024973829,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} {"type":"tool/call","data":{"turn":1,"step":1,"callId":"cua-ptc-native-result","name":"run_code","arguments":"{\"code\":\"const { Buffer } = await import('node:buffer');\\nconst app = await tools.cua({ method: 'getApp', params: { selector: 'test.fixture' } });\\nconst state = await tools.cua({ method: 'getState', params: { appHandle: app.appHandle, windowId: 7 } });\\nconst shot = await tools.cua({ method: 'screenshot', params: { appHandle: app.appHandle, windowId: 7 } });\\nconst bytes = Buffer.from(shot.data, 'base64');\\nreturn { appHandle: app.appHandle, hasIncrement: state.tree.includes('Increment'), pngBytes: bytes.length, signature: bytes.subarray(0, 8).toString('hex'), imageWidth: shot.attachment.width, imageHeight: shot.attachment.height };\",\"description\":\"Read native CUA values and receive one screenshot\"}"}} {"type":"tool/ptc-dispatch-start","data":{"rootCallId":"cua-ptc-native-result","parentCallId":"cua-ptc-native-result","subCallId":"cua-ptc-native-result:ptc:1","name":"cua","arguments":{"method":"getApp","params":{"selector":"test.fixture"}}}} {"type":"tool/ptc-dispatch","data":{"rootCallId":"cua-ptc-native-result","parentCallId":"cua-ptc-native-result","subCallId":"cua-ptc-native-result:ptc:1","name":"cua","arguments":{"method":"getApp","params":{"selector":"test.fixture"}},"isError":false,"content":[{"type":"text","text":"{\"appHandle\":\"fixture-app\",\"name\":\"Fixture\",\"bundleId\":\"test.fixture\",\"running\":true}"}]}} @@ -26,6 +26,28 @@ {"type":"agent/inbox/spliced","data":{"target":"next-step","start":0,"removedCount":1,"inserted":[]}} {"type":"step/start","data":{"turn":1,"step":2}} {"type":"user/message","data":{"content":[{"type":"text","text":"{\"pixelWidth\":1,\"pixelHeight\":1,\"windowId\":7,\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"coordinateSpace\":\"screenshot-pixels\"}"},{"type":"image","attachment":{"attachmentId":"sha256:999f1d1527ee7e79266f16add5430fff76b1225d742464a5b1ff1f02971bb8ee","mediaType":"image/png","width":1,"height":1,"bytes":90}}],"source":{"kind":"ptc-mode"},"role":"user","id":"{{message:6}}"},"surfaceOp":"append"} -{"type":"assistant/message","data":{"turn":1,"step":2,"message":{"role":"assistant","content":[{"type":"text","text":"Received native CUA values and one PNG screenshot through PTC. The PNG is 90 bytes with signature 89504e470d0a1a0a."}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:7}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791864699463,"chunk":{"type":"block-start","index":0,"blockType":"text"}},{"type":"chunk","time":1791864699463,"chunk":{"type":"block-end","index":0,"block":{"type":"text","text":"Received native CUA values and one PNG screenshot through PTC. The PNG is 90 bytes with signature 89504e470d0a1a0a."}}},{"type":"chunk","time":1791864699463,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791864699463,"chunk":{"type":"finish","reason":{"kind":"stop"}}}]},"surfaceOp":"append"} +{"type":"assistant/message","data":{"turn":1,"step":2,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-ptc-ax-failure","name":"run_code","arguments":"{\"code\":\"const { writeFile, unlink } = await import('node:fs/promises');\\nconst { resolve } = await import('node:path');\\nconst { pathToFileURL } = await import('node:url');\\nconst { connectCua } = await import(pathToFileURL(resolve('cua-bridge.mjs')).href);\\nawait writeFile('fixture-ax-failure', '');\\ntry {\\n const cua = connectCua(() => tools.cua);\\n const app = await cua.getApp('test.fixture');\\n const windows = await app.getWindows({ emit: false });\\n const window = await app.getWindow(windows[0].windowId);\\n const failures = [];\\n for (const surface of [app, window]) {\\n try { await surface.getAccessibilityTree(); throw new Error('Expected AX failure'); }\\n catch (error) { if (!error.message.includes('Fixture accessibility read failed')) throw error; failures.push(error.message); }\\n }\\n const shot = await window.getScreenshot();\\n let elementRefused = false;\\n try { await window.invoke('0'); }\\n catch (error) { if (!error.message.includes('Read getAccessibilityTree')) throw error; elementRefused = true; }\\n if (!elementRefused) throw new Error('Element input accepted without an AX snapshot');\\n return { axFailures: failures.length, screenshotWindow: shot.windowId, pngBytes: shot.bytes.length, elementRefused };\\n} finally { await unlink('fixture-ax-failure'); }\",\"description\":\"Capture through the object API despite application and window AX failures\"}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:7}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1794024974646,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1794024974646,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-ptc-ax-failure","name":"run_code","arguments":"{\"code\":\"const { writeFile, unlink } = await import('node:fs/promises');\\nconst { resolve } = await import('node:path');\\nconst { pathToFileURL } = await import('node:url');\\nconst { connectCua } = await import(pathToFileURL(resolve('cua-bridge.mjs')).href);\\nawait writeFile('fixture-ax-failure', '');\\ntry {\\n const cua = connectCua(() => tools.cua);\\n const app = await cua.getApp('test.fixture');\\n const windows = await app.getWindows({ emit: false });\\n const window = await app.getWindow(windows[0].windowId);\\n const failures = [];\\n for (const surface of [app, window]) {\\n try { await surface.getAccessibilityTree(); throw new Error('Expected AX failure'); }\\n catch (error) { if (!error.message.includes('Fixture accessibility read failed')) throw error; failures.push(error.message); }\\n }\\n const shot = await window.getScreenshot();\\n let elementRefused = false;\\n try { await window.invoke('0'); }\\n catch (error) { if (!error.message.includes('Read getAccessibilityTree')) throw error; elementRefused = true; }\\n if (!elementRefused) throw new Error('Element input accepted without an AX snapshot');\\n return { axFailures: failures.length, screenshotWindow: shot.windowId, pngBytes: shot.bytes.length, elementRefused };\\n} finally { await unlink('fixture-ax-failure'); }\",\"description\":\"Capture through the object API despite application and window AX failures\"}"}}},{"type":"chunk","time":1794024974646,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1794024974646,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} +{"type":"tool/call","data":{"turn":1,"step":2,"callId":"cua-ptc-ax-failure","name":"run_code","arguments":"{\"code\":\"const { writeFile, unlink } = await import('node:fs/promises');\\nconst { resolve } = await import('node:path');\\nconst { pathToFileURL } = await import('node:url');\\nconst { connectCua } = await import(pathToFileURL(resolve('cua-bridge.mjs')).href);\\nawait writeFile('fixture-ax-failure', '');\\ntry {\\n const cua = connectCua(() => tools.cua);\\n const app = await cua.getApp('test.fixture');\\n const windows = await app.getWindows({ emit: false });\\n const window = await app.getWindow(windows[0].windowId);\\n const failures = [];\\n for (const surface of [app, window]) {\\n try { await surface.getAccessibilityTree(); throw new Error('Expected AX failure'); }\\n catch (error) { if (!error.message.includes('Fixture accessibility read failed')) throw error; failures.push(error.message); }\\n }\\n const shot = await window.getScreenshot();\\n let elementRefused = false;\\n try { await window.invoke('0'); }\\n catch (error) { if (!error.message.includes('Read getAccessibilityTree')) throw error; elementRefused = true; }\\n if (!elementRefused) throw new Error('Element input accepted without an AX snapshot');\\n return { axFailures: failures.length, screenshotWindow: shot.windowId, pngBytes: shot.bytes.length, elementRefused };\\n} finally { await unlink('fixture-ax-failure'); }\",\"description\":\"Capture through the object API despite application and window AX failures\"}"}} +{"type":"tool/ptc-dispatch-start","data":{"rootCallId":"cua-ptc-ax-failure","parentCallId":"cua-ptc-ax-failure","subCallId":"cua-ptc-ax-failure:ptc:1","name":"cua","arguments":{"method":"getCapabilities","params":{}}}} +{"type":"tool/ptc-dispatch","data":{"rootCallId":"cua-ptc-ax-failure","parentCallId":"cua-ptc-ax-failure","subCallId":"cua-ptc-ax-failure:ptc:1","name":"cua","arguments":{"method":"getCapabilities","params":{}},"isError":false,"content":[{"type":"text","text":"{\"sessionLocked\":false,\"permissions\":{\"accessibility\":true,\"screenRecording\":true},\"platform\":\"macos\",\"methods\":[\"activate\",\"click\",\"drag\",\"findApps\",\"getApp\",\"getCapabilities\",\"getState\",\"invoke\",\"listActions\",\"listApps\",\"listWindows\",\"mouseMove\",\"paste\",\"performAction\",\"pressKey\",\"screenshot\",\"scroll\",\"selectText\",\"setValue\",\"show\",\"typeText\"],\"appLevelAccessibility\":true,\"keyboard\":[\"app\"],\"browser\":false,\"coordinateSpace\":\"screenshot-pixels\"}"}]}} +{"type":"tool/ptc-dispatch-start","data":{"rootCallId":"cua-ptc-ax-failure","parentCallId":"cua-ptc-ax-failure","subCallId":"cua-ptc-ax-failure:ptc:2","name":"cua","arguments":{"method":"getApp","params":{"selector":"test.fixture"}}}} +{"type":"tool/ptc-dispatch","data":{"rootCallId":"cua-ptc-ax-failure","parentCallId":"cua-ptc-ax-failure","subCallId":"cua-ptc-ax-failure:ptc:2","name":"cua","arguments":{"method":"getApp","params":{"selector":"test.fixture"}},"isError":false,"content":[{"type":"text","text":"{\"appHandle\":\"fixture-app\",\"name\":\"Fixture\",\"bundleId\":\"test.fixture\",\"running\":true}"}]}} +{"type":"tool/ptc-dispatch-start","data":{"rootCallId":"cua-ptc-ax-failure","parentCallId":"cua-ptc-ax-failure","subCallId":"cua-ptc-ax-failure:ptc:3","name":"cua","arguments":{"method":"listWindows","params":{"appHandle":"fixture-app"}}}} +{"type":"tool/ptc-dispatch","data":{"rootCallId":"cua-ptc-ax-failure","parentCallId":"cua-ptc-ax-failure","subCallId":"cua-ptc-ax-failure:ptc:3","name":"cua","arguments":{"method":"listWindows","params":{"appHandle":"fixture-app"}},"isError":false,"content":[{"type":"text","text":"[{\"windowId\":7,\"appName\":\"Fixture\",\"title\":\"CUA fixture\",\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"isOnScreen\":true,\"onCurrentSpace\":true},{\"windowId\":8,\"appName\":\"Fixture\",\"title\":\"AXless canvas\",\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"isOnScreen\":true,\"onCurrentSpace\":true}]"}]}} +{"type":"tool/ptc-dispatch-start","data":{"rootCallId":"cua-ptc-ax-failure","parentCallId":"cua-ptc-ax-failure","subCallId":"cua-ptc-ax-failure:ptc:4","name":"cua","arguments":{"method":"listWindows","params":{"appHandle":"fixture-app"}}}} +{"type":"tool/ptc-dispatch","data":{"rootCallId":"cua-ptc-ax-failure","parentCallId":"cua-ptc-ax-failure","subCallId":"cua-ptc-ax-failure:ptc:4","name":"cua","arguments":{"method":"listWindows","params":{"appHandle":"fixture-app"}},"isError":false,"content":[{"type":"text","text":"[{\"windowId\":7,\"appName\":\"Fixture\",\"title\":\"CUA fixture\",\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"isOnScreen\":true,\"onCurrentSpace\":true},{\"windowId\":8,\"appName\":\"Fixture\",\"title\":\"AXless canvas\",\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"isOnScreen\":true,\"onCurrentSpace\":true}]"}]}} +{"type":"tool/ptc-dispatch-start","data":{"rootCallId":"cua-ptc-ax-failure","parentCallId":"cua-ptc-ax-failure","subCallId":"cua-ptc-ax-failure:ptc:5","name":"cua","arguments":{"method":"getState","params":{"appHandle":"fixture-app"}}}} +{"type":"tool/ptc-dispatch","data":{"rootCallId":"cua-ptc-ax-failure","parentCallId":"cua-ptc-ax-failure","subCallId":"cua-ptc-ax-failure:ptc:5","name":"cua","arguments":{"method":"getState","params":{"appHandle":"fixture-app"}},"isError":true,"content":[{"type":"text","text":"Error: NATIVE_ERROR: Fixture accessibility read failed"}]}} +{"type":"tool/ptc-dispatch-start","data":{"rootCallId":"cua-ptc-ax-failure","parentCallId":"cua-ptc-ax-failure","subCallId":"cua-ptc-ax-failure:ptc:6","name":"cua","arguments":{"method":"getState","params":{"appHandle":"fixture-app","windowId":7}}}} +{"type":"tool/ptc-dispatch","data":{"rootCallId":"cua-ptc-ax-failure","parentCallId":"cua-ptc-ax-failure","subCallId":"cua-ptc-ax-failure:ptc:6","name":"cua","arguments":{"method":"getState","params":{"appHandle":"fixture-app","windowId":7}},"isError":true,"content":[{"type":"text","text":"Error: NATIVE_ERROR: Fixture accessibility read failed"}]}} +{"type":"tool/ptc-dispatch-start","data":{"rootCallId":"cua-ptc-ax-failure","parentCallId":"cua-ptc-ax-failure","subCallId":"cua-ptc-ax-failure:ptc:7","name":"cua","arguments":{"method":"screenshot","params":{"appHandle":"fixture-app","windowId":7}}}} +{"type":"tool/ptc-dispatch","data":{"rootCallId":"cua-ptc-ax-failure","parentCallId":"cua-ptc-ax-failure","subCallId":"cua-ptc-ax-failure:ptc:7","name":"cua","arguments":{"method":"screenshot","params":{"appHandle":"fixture-app","windowId":7}},"isError":false,"content":[{"type":"text","text":"{\"pixelWidth\":1,\"pixelHeight\":1,\"windowId\":7,\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"coordinateSpace\":\"screenshot-pixels\"}"},{"type":"image","attachment":{"attachmentId":"sha256:999f1d1527ee7e79266f16add5430fff76b1225d742464a5b1ff1f02971bb8ee","mediaType":"image/png","width":1,"height":1,"bytes":90}}]}} +{"type":"tool/result","data":{"turn":1,"step":2,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-ptc-ax-failure"},"toolCallId":"cua-ptc-ax-failure","content":[{"type":"text","text":"{\n \"axFailures\": 2,\n \"screenshotWindow\": 7,\n \"pngBytes\": 90,\n \"elementRefused\": true\n}"}],"isError":false,"id":"{{message:8}}"}},"sourceEventSeqs":[28],"surfaceOp":"append"} +{"type":"agent/inbox/spliced","data":{"target":"next-step","start":0,"inserted":[{"content":[{"type":"text","text":"{\"pixelWidth\":1,\"pixelHeight\":1,\"windowId\":7,\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"coordinateSpace\":\"screenshot-pixels\"}"},{"type":"image","attachment":{"attachmentId":"sha256:999f1d1527ee7e79266f16add5430fff76b1225d742464a5b1ff1f02971bb8ee","mediaType":"image/png","width":1,"height":1,"bytes":90}}],"source":{"kind":"ptc-mode"},"role":"user","id":"{{message:9}}"}]}} {"type":"step/end","data":{"turn":1,"step":2}} +{"type":"agent/inbox/spliced","data":{"target":"next-step","start":0,"removedCount":1,"inserted":[]}} +{"type":"step/start","data":{"turn":1,"step":3}} +{"type":"user/message","data":{"content":[{"type":"text","text":"{\"pixelWidth\":1,\"pixelHeight\":1,\"windowId\":7,\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"coordinateSpace\":\"screenshot-pixels\"}"},{"type":"image","attachment":{"attachmentId":"sha256:999f1d1527ee7e79266f16add5430fff76b1225d742464a5b1ff1f02971bb8ee","mediaType":"image/png","width":1,"height":1,"bytes":90}}],"source":{"kind":"ptc-mode"},"role":"user","id":"{{message:9}}"},"surfaceOp":"append"} +{"type":"assistant/message","data":{"turn":1,"step":3,"message":{"role":"assistant","content":[{"type":"text","text":"Received native CUA values and PNG screenshots through PTC. The object API captured a 90-byte PNG after two AX read failures and refused element input without an accessibility snapshot."}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:10}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1794024974851,"chunk":{"type":"block-start","index":0,"blockType":"text"}},{"type":"chunk","time":1794024974851,"chunk":{"type":"block-end","index":0,"block":{"type":"text","text":"Received native CUA values and PNG screenshots through PTC. The object API captured a 90-byte PNG after two AX read failures and refused element input without an accessibility snapshot."}}},{"type":"chunk","time":1794024974851,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1794024974851,"chunk":{"type":"finish","reason":{"kind":"stop"}}}]},"surfaceOp":"append"} +{"type":"step/end","data":{"turn":1,"step":3}} {"type":"turn/end","data":{"turn":1,"reason":{"kind":"completed"}}} diff --git a/snapshots/session/cua-ptc/system-prompt.expected.md b/snapshots/session/cua-ptc/system-prompt.expected.md index 587436227a..be8ab5aed8 100644 --- a/snapshots/session/cua-ptc/system-prompt.expected.md +++ b/snapshots/session/cua-ptc/system-prompt.expected.md @@ -48,9 +48,9 @@ Setup and discovery: - cua.listApps(options?: { emit?: boolean }) -> { id: string, displayName: string, isRunning: true, pid: number }[] — enumerate running applications. - cua.findApps(query: string, options?: { emit?: boolean }) -> ({ id: string, displayName: string, path: string } & ({ isRunning: true, pid: number } | { isRunning: false }))[] — search installed applications, including stopped matches; only running matches carry pid. - cua.getState(options?: { emit?: boolean }) -> { apps: { id: string, displayName: string, isRunning: true, pid: number }[], browsers: [] } — summarize running applications; this is not an accessibility observation. -- cua.getApp(selector: string) -> App — select an application by name, bundle ID, absolute path, or running PID string, obtain consent, launch in the background if needed, and read initial AX state silently; it does not activate the app. +- cua.getApp(selector: string) -> App — select an application by name, bundle ID, absolute path, or running PID string, obtain consent, launch in the background if needed, and return without reading AX or activating the app. - app.getWindows(options?: { emit?: boolean }) -> { windowId: number, appName: string, title: string, bounds: { x: number, y: number, width: number, height: number }, isOnScreen: boolean, onCurrentSpace: boolean | null }[] — enumerate windows with global native bounds; null onCurrentSpace means unknown Space membership. -- app.getWindow(windowId: number) -> Window — select one window and read its initial AX state silently; do this before any window screenshot or coordinate input. +- app.getWindow(windowId: number) -> Window — select a window from the application’s current window list without reading AX; do this before any window screenshot or coordinate input. - app.activate() -> ActionResult — request explicit foreground activation; use only when foreground access is required. - app.show() -> { dispatched: boolean, windowId: number | null, foregroundChanged: boolean | null } — unhide the app and restore its selected, focused, main, or sole window without activation. - window.show() -> { dispatched: boolean, windowId: number | null, foregroundChanged: boolean | null } — unhide and restore that explicit window without activation. @@ -58,7 +58,7 @@ Setup and discovery: ## Accessibility tree Available at app or window level depending on the platform. -Element ids are opaque strings from the latest successful tree; the object facade keeps one current snapshot per app across its repeated app and window objects, a new read replaces it, and a failed read invalidates the previous one. Repeated strings can appear for different elements, so do not reuse an old id blindly, and element operations do not fall back to pointer input or typing. +Read getAccessibilityTree explicitly before using element ids. Selection and screenshots do not read or replace the current AX snapshot. Element ids are opaque strings from the latest successful tree; the object facade keeps one current snapshot per app across its repeated app and window objects, a new read replaces it, and a failed read invalidates the previous one. Repeated strings can appear for different elements, so do not reuse an old id blindly, and element operations do not fall back to pointer input or typing. - getAccessibilityTree(options?: { emit?: boolean, includeMenus?: boolean, details?: boolean }) -> string — read the full tree text; includeMenus expands menu contents and details adds identifiers, native bounds, and complete action names. - listActions(elementId: string) -> string[] — list the actions an element advertises. - invoke(elementId: string) -> ActionResult — invoke the element's default action; dispatched is false when it has none. @@ -82,7 +82,7 @@ Available at app or window level depending on the platform. - paste(text: string, options?: { format?: 'text' | 'html' }) -> ActionResult — deliver text or HTML through the app's Paste command, restoring prior clipboard contents unless another writer changes them; a Paste failure is surfaced without substituting typeText, and PASTE_NOT_CONSUMED means inspect before any retry. - pressKey(key: string, modifiers?: string[]) -> ActionResult — press and release one complete keystroke. For example pressKey('Tab', ['shift']) or pressKey('+'). Key and modifier names are platform-defined strings. Pass key and modifiers separately; do not pack them into a chord string. -Raw tools use { method, params }: include appHandle for app calls and windowId for window calls, and use snapshotId with elementId for element routes. The emit option is facade-only: emit is local output control, so raw callers must follow the raw operation schema for admitted params. getAccessibilityTree and getWindow read through raw getState; getWindows maps to listWindows; getScreenshot maps to screenshot; cua.getState maps to listApps; every other named operation uses the same method name. +Raw tools use { method, params }: include appHandle for app calls and windowId for window calls, and use snapshotId with elementId for element routes. The emit option is facade-only: emit is local output control, so raw callers must follow the raw operation schema for admitted params. getAccessibilityTree maps to getState; getWindow validates against listWindows; getWindows maps to listWindows; getScreenshot maps to screenshot; cua.getState maps to listApps; every other named operation uses the same method name. Maximum click holdMs: 5000 ms. diff --git a/snapshots/session/ptc-script-retained/system-prompt.expected.md b/snapshots/session/ptc-script-retained/system-prompt.expected.md index dd88fa8d9e..63e91315c0 100644 --- a/snapshots/session/ptc-script-retained/system-prompt.expected.md +++ b/snapshots/session/ptc-script-retained/system-prompt.expected.md @@ -48,9 +48,9 @@ Setup and discovery: - cua.listApps(options?: { emit?: boolean }) -> { id: string, displayName: string, isRunning: true, pid: number }[] — enumerate running applications. - cua.findApps(query: string, options?: { emit?: boolean }) -> ({ id: string, displayName: string, path: string } & ({ isRunning: true, pid: number } | { isRunning: false }))[] — search installed applications, including stopped matches; only running matches carry pid. - cua.getState(options?: { emit?: boolean }) -> { apps: { id: string, displayName: string, isRunning: true, pid: number }[], browsers: [] } — summarize running applications; this is not an accessibility observation. -- cua.getApp(selector: string) -> App — select an application by name, bundle ID, absolute path, or running PID string, obtain consent, launch in the background if needed, and read initial AX state silently; it does not activate the app. +- cua.getApp(selector: string) -> App — select an application by name, bundle ID, absolute path, or running PID string, obtain consent, launch in the background if needed, and return without reading AX or activating the app. - app.getWindows(options?: { emit?: boolean }) -> { windowId: number, appName: string, title: string, bounds: { x: number, y: number, width: number, height: number }, isOnScreen: boolean, onCurrentSpace: boolean | null }[] — enumerate windows with global native bounds; null onCurrentSpace means unknown Space membership. -- app.getWindow(windowId: number) -> Window — select one window and read its initial AX state silently; do this before any window screenshot or coordinate input. +- app.getWindow(windowId: number) -> Window — select a window from the application’s current window list without reading AX; do this before any window screenshot or coordinate input. - app.activate() -> ActionResult — request explicit foreground activation; use only when foreground access is required. - app.show() -> { dispatched: boolean, windowId: number | null, foregroundChanged: boolean | null } — unhide the app and restore its selected, focused, main, or sole window without activation. - window.show() -> { dispatched: boolean, windowId: number | null, foregroundChanged: boolean | null } — unhide and restore that explicit window without activation. @@ -58,7 +58,7 @@ Setup and discovery: ## Accessibility tree Available at app or window level depending on the platform. -Element ids are opaque strings from the latest successful tree; the object facade keeps one current snapshot per app across its repeated app and window objects, a new read replaces it, and a failed read invalidates the previous one. Repeated strings can appear for different elements, so do not reuse an old id blindly, and element operations do not fall back to pointer input or typing. +Read getAccessibilityTree explicitly before using element ids. Selection and screenshots do not read or replace the current AX snapshot. Element ids are opaque strings from the latest successful tree; the object facade keeps one current snapshot per app across its repeated app and window objects, a new read replaces it, and a failed read invalidates the previous one. Repeated strings can appear for different elements, so do not reuse an old id blindly, and element operations do not fall back to pointer input or typing. - getAccessibilityTree(options?: { emit?: boolean, includeMenus?: boolean, details?: boolean }) -> string — read the full tree text; includeMenus expands menu contents and details adds identifiers, native bounds, and complete action names. - listActions(elementId: string) -> string[] — list the actions an element advertises. - invoke(elementId: string) -> ActionResult — invoke the element's default action; dispatched is false when it has none. @@ -82,7 +82,7 @@ Available at app or window level depending on the platform. - paste(text: string, options?: { format?: 'text' | 'html' }) -> ActionResult — deliver text or HTML through the app's Paste command, restoring prior clipboard contents unless another writer changes them; a Paste failure is surfaced without substituting typeText, and PASTE_NOT_CONSUMED means inspect before any retry. - pressKey(key: string, modifiers?: string[]) -> ActionResult — press and release one complete keystroke. For example pressKey('Tab', ['shift']) or pressKey('+'). Key and modifier names are platform-defined strings. Pass key and modifiers separately; do not pack them into a chord string. -Raw tools use { method, params }: include appHandle for app calls and windowId for window calls, and use snapshotId with elementId for element routes. The emit option is facade-only: emit is local output control, so raw callers must follow the raw operation schema for admitted params. getAccessibilityTree and getWindow read through raw getState; getWindows maps to listWindows; getScreenshot maps to screenshot; cua.getState maps to listApps; every other named operation uses the same method name. +Raw tools use { method, params }: include appHandle for app calls and windowId for window calls, and use snapshotId with elementId for element routes. The emit option is facade-only: emit is local output control, so raw callers must follow the raw operation schema for admitted params. getAccessibilityTree maps to getState; getWindow validates against listWindows; getWindows maps to listWindows; getScreenshot maps to screenshot; cua.getState maps to listApps; every other named operation uses the same method name. Maximum click holdMs: 5000 ms. From f2dc7347adbcb37322265e727ce09b134018b3fd Mon Sep 17 00:00:00 2001 From: Dudu-0223 Date: Sat, 7 Nov 2026 12:29:31 +0800 Subject: [PATCH 2/9] fix(cua): share HarnessError through a peer dependency --- packages/experimental/cua/package.json | 9 +++++---- pnpm-lock.yaml | 6 +++--- 2 files changed, 8 insertions(+), 7 deletions(-) diff --git a/packages/experimental/cua/package.json b/packages/experimental/cua/package.json index 0d62908a8a..410dd23cf7 100644 --- a/packages/experimental/cua/package.json +++ b/packages/experimental/cua/package.json @@ -99,7 +99,8 @@ "@deepseek-ai/dsh-user-questions": "workspace:*", "@deepseek-ai/dsh-attachment": "workspace:*", "@deepseek-ai/dsh-scope": "workspace:*", - "@deepseek-ai/dsh-ptc-runtime-node": "workspace:*" + "@deepseek-ai/dsh-ptc-runtime-node": "workspace:*", + "@deepseek-ai/dsh-error": "workspace:*" }, "dependencies": { "@deepseek-ai/schemastery": "workspace:~", @@ -107,8 +108,7 @@ "@deepseek-ai/dsh-atomic-write": "workspace:*", "@deepseek-ai/dsh-typert-protocol": "workspace:*", "@deepseek-ai/dsh-computer-use": "workspace:*", - "@deepseek-ai/dsh-storage-domain": "workspace:*", - "@deepseek-ai/dsh-error": "workspace:*" + "@deepseek-ai/dsh-storage-domain": "workspace:*" }, "devDependencies": { "@deepseek-ai/dsh-working-directory": "workspace:*", @@ -162,7 +162,8 @@ "@deepseek-ai/dsh-storage-json": "workspace:*", "@deepseek-ai/dsh-client-ui-user-questions": "workspace:*", "@deepseek-ai/dsh-client-ui-approval": "workspace:*", - "@deepseek-ai/dsh-client-ui-conversation": "workspace:*" + "@deepseek-ai/dsh-client-ui-conversation": "workspace:*", + "@deepseek-ai/dsh-error": "workspace:*" }, "dsh": { "client": { diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 83d9bd763d..f33fef7d42 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -9800,9 +9800,6 @@ importers: '@deepseek-ai/dsh-computer-use': specifier: workspace:* version: link:../../computer-use/computer-use - '@deepseek-ai/dsh-error': - specifier: workspace:* - version: link:../../util/error '@deepseek-ai/dsh-storage-domain': specifier: workspace:* version: link:../../storage/storage-domain @@ -9897,6 +9894,9 @@ importers: '@deepseek-ai/dsh-config-editor': specifier: workspace:* version: link:../../boot/config-editor + '@deepseek-ai/dsh-error': + specifier: workspace:* + version: link:../../util/error '@deepseek-ai/dsh-fs-local': specifier: workspace:* version: link:../../fs/fs-local From 67561b163eebf322d39f6f87c27eac93b4ebaf39 Mon Sep 17 00:00:00 2001 From: Dudu-0223 Date: Sat, 7 Nov 2026 12:35:17 +0800 Subject: [PATCH 3/9] docs(cua): regenerate shared dependency graph --- docs/module-graph.md | 3 ++- docs/module-graph.zh.md | 3 ++- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/module-graph.md b/docs/module-graph.md index a3a28f92fc..cc0d1e4f2c 100644 --- a/docs/module-graph.md +++ b/docs/module-graph.md @@ -952,6 +952,7 @@ flowchart TD pkg_creation_recommendations --> pkg_skill pkg_creation_recommendations --> pkg_tools pkg_cua --> pkg_attachment + pkg_cua --> pkg_error pkg_cua --> pkg_ptc_runtime_node pkg_cua --> pkg_scope pkg_cua --> pkg_user_questions @@ -1821,7 +1822,7 @@ flowchart TD | [`workspace-changes`](../packages/deliverables/workspace-changes) | `deliverables` | [`agent`](../packages/core/agent), [`fs`](../packages/fs/fs), [`session`](../packages/core/session), [`subprocess`](../packages/subprocess/subprocess), [`tools`](../packages/core/tools) | | [`action-cards`](../packages/experimental/action-cards) | `experimental` | [`llm`](../packages/llm/llm), [`session`](../packages/core/session), [`tools`](../packages/core/tools), [`typert-protocol`](../packages/typert/protocol) | | [`creation-recommendations`](../packages/experimental/creation-recommendations) | `experimental` | [`llm`](../packages/llm/llm), [`session`](../packages/core/session), [`skill`](../packages/skill/skill), [`tools`](../packages/core/tools) | -| [`cua`](../packages/experimental/cua) | `experimental` | [`attachment`](../packages/attachment/attachment), [`ptc-runtime-node`](../packages/ptc-runtime/ptc-runtime-node), [`scope`](../packages/core/scope), [`user-questions`](../packages/interaction/user-questions) | +| [`cua`](../packages/experimental/cua) | `experimental` | [`attachment`](../packages/attachment/attachment), [`error`](../packages/util/error), [`ptc-runtime-node`](../packages/ptc-runtime/ptc-runtime-node), [`scope`](../packages/core/scope), [`user-questions`](../packages/interaction/user-questions) | | [`experimental-browser-use-chrome-devtools-mcp`](../packages/experimental/browser-use-chrome-devtools-mcp) | `experimental` | [`agent`](../packages/core/agent), [`browser-use`](../packages/browser-use/browser-use), [`system-prompt`](../packages/core/system-prompt), [`tools`](../packages/core/tools), [`working-directory`](../packages/session/working-directory) | | [`experimental-browser-use-playwright-mcp`](../packages/experimental/browser-use-playwright-mcp) | `experimental` | [`agent`](../packages/core/agent), [`browser-use`](../packages/browser-use/browser-use), [`system-prompt`](../packages/core/system-prompt), [`tools`](../packages/core/tools), [`working-directory`](../packages/session/working-directory) | | [`experimental-browser-use-stagehand-native`](../packages/experimental/browser-use-stagehand-native) | `experimental` | [`agent`](../packages/core/agent), [`browser-use`](../packages/browser-use/browser-use), [`system-prompt`](../packages/core/system-prompt), [`tools`](../packages/core/tools) | diff --git a/docs/module-graph.zh.md b/docs/module-graph.zh.md index 9e337229b4..c9cf1d8d93 100644 --- a/docs/module-graph.zh.md +++ b/docs/module-graph.zh.md @@ -954,6 +954,7 @@ flowchart TD pkg_creation_recommendations --> pkg_skill pkg_creation_recommendations --> pkg_tools pkg_cua --> pkg_attachment + pkg_cua --> pkg_error pkg_cua --> pkg_ptc_runtime_node pkg_cua --> pkg_scope pkg_cua --> pkg_user_questions @@ -1823,7 +1824,7 @@ flowchart TD | [`workspace-changes`](../packages/deliverables/workspace-changes) | `deliverables` | [`agent`](../packages/core/agent), [`fs`](../packages/fs/fs), [`session`](../packages/core/session), [`subprocess`](../packages/subprocess/subprocess), [`tools`](../packages/core/tools) | | [`action-cards`](../packages/experimental/action-cards) | `experimental` | [`llm`](../packages/llm/llm), [`session`](../packages/core/session), [`tools`](../packages/core/tools), [`typert-protocol`](../packages/typert/protocol) | | [`creation-recommendations`](../packages/experimental/creation-recommendations) | `experimental` | [`llm`](../packages/llm/llm), [`session`](../packages/core/session), [`skill`](../packages/skill/skill), [`tools`](../packages/core/tools) | -| [`cua`](../packages/experimental/cua) | `experimental` | [`attachment`](../packages/attachment/attachment), [`ptc-runtime-node`](../packages/ptc-runtime/ptc-runtime-node), [`scope`](../packages/core/scope), [`user-questions`](../packages/interaction/user-questions) | +| [`cua`](../packages/experimental/cua) | `experimental` | [`attachment`](../packages/attachment/attachment), [`error`](../packages/util/error), [`ptc-runtime-node`](../packages/ptc-runtime/ptc-runtime-node), [`scope`](../packages/core/scope), [`user-questions`](../packages/interaction/user-questions) | | [`experimental-browser-use-chrome-devtools-mcp`](../packages/experimental/browser-use-chrome-devtools-mcp) | `experimental` | [`agent`](../packages/core/agent), [`browser-use`](../packages/browser-use/browser-use), [`system-prompt`](../packages/core/system-prompt), [`tools`](../packages/core/tools), [`working-directory`](../packages/session/working-directory) | | [`experimental-browser-use-playwright-mcp`](../packages/experimental/browser-use-playwright-mcp) | `experimental` | [`agent`](../packages/core/agent), [`browser-use`](../packages/browser-use/browser-use), [`system-prompt`](../packages/core/system-prompt), [`tools`](../packages/core/tools), [`working-directory`](../packages/session/working-directory) | | [`experimental-browser-use-stagehand-native`](../packages/experimental/browser-use-stagehand-native) | `experimental` | [`agent`](../packages/core/agent), [`browser-use`](../packages/browser-use/browser-use), [`system-prompt`](../packages/core/system-prompt), [`tools`](../packages/core/tools) | From facebeb270d44d5d20143e850c7605b09b23ab29 Mon Sep 17 00:00:00 2001 From: Dudu-0223 Date: Sat, 7 Nov 2026 14:12:45 +0800 Subject: [PATCH 4/9] fix(cua): let native window operations own window validation getWindow now creates the window object without a listWindows request. Screenshots and pointer input already reject closed and foreign windows natively, so the facade keeps only local windowId validation and drops the dsh-error dependency and CUA_WINDOW_UNAVAILABLE code. --- ...05-cua-scoped-surfaces-and-input.i18n.yaml | 4 +-- ...026-11-05-cua-scoped-surfaces-and-input.md | 2 +- ...-11-05-cua-scoped-surfaces-and-input.zh.md | 2 +- docs/module-graph.md | 3 +-- docs/module-graph.zh.md | 3 +-- .../experimental/cua-macos/README.i18n.yaml | 4 +-- packages/experimental/cua-macos/README.md | 6 ++--- packages/experimental/cua-macos/README.zh.md | 6 ++--- packages/experimental/cua/README.i18n.yaml | 4 +-- packages/experimental/cua/README.md | 2 +- packages/experimental/cua/README.zh.md | 2 +- packages/experimental/cua/package.json | 6 ++--- packages/experimental/cua/src/api.ts | 14 ++++------ packages/experimental/cua/src/guidance.ts | 4 +-- packages/experimental/cua/src/types.ts | 3 ++- packages/experimental/cua/tests/api.spec.ts | 27 +++++++++---------- .../experimental/cua/tests/bridge.spec.ts | 6 ++--- packages/experimental/cua/tsconfig.host.json | 3 --- pnpm-lock.yaml | 3 --- .../cua-native/system-prompt.expected.md | 4 +-- snapshots/session/cua-ptc/session.v4.jsonl | 20 +++++++------- .../session/cua-ptc/system-prompt.expected.md | 4 +-- .../system-prompt.expected.md | 4 +-- 23 files changed, 58 insertions(+), 78 deletions(-) diff --git a/.agents/notes/implemented/architecture/2026-11-05-cua-scoped-surfaces-and-input.i18n.yaml b/.agents/notes/implemented/architecture/2026-11-05-cua-scoped-surfaces-and-input.i18n.yaml index 5f1ff64a4d..53d7ee394c 100644 --- a/.agents/notes/implemented/architecture/2026-11-05-cua-scoped-surfaces-and-input.i18n.yaml +++ b/.agents/notes/implemented/architecture/2026-11-05-cua-scoped-surfaces-and-input.i18n.yaml @@ -9,8 +9,8 @@ en: f72db0b79aa02fb5 zh: b91c1f177dc93ab4 /agent-note-cua-scoped-surfaces-and-separated-input-capabilities/decision: - en: 4fa61247fe33edb0 - zh: afa76f74841eb586 + en: 41fd533d5be7642b + zh: c6fcbabec22bdbb7 /agent-note-cua-scoped-surfaces-and-separated-input-capabilities/alternatives-considered: en: 32c970ae153e0f6c zh: 8431be0a10ffed15 diff --git a/.agents/notes/implemented/architecture/2026-11-05-cua-scoped-surfaces-and-input.md b/.agents/notes/implemented/architecture/2026-11-05-cua-scoped-surfaces-and-input.md index 3888527013..23780d7e88 100644 --- a/.agents/notes/implemented/architecture/2026-11-05-cua-scoped-surfaces-and-input.md +++ b/.agents/notes/implemented/architecture/2026-11-05-cua-scoped-surfaces-and-input.md @@ -16,7 +16,7 @@ Provider startup capabilities are static. Platform, supported methods, applicati Keyboard input keeps keys and modifiers separate: `pressKey(key, modifiers?)` presses and releases one key while holding the listed modifiers, and its keyboard modifier list is a plain string array defined by the provider. `sendKeyEvent(key, direction)` sends exactly one key-down or key-up without an implicit counterpart. A caller can therefore hold a modifier or split a gesture; neither method packs a chord string, and the shared schema does not enumerate platform key names. -Within one API instance, the object facade keeps one current accessibility snapshot per application handle, shared by every application and window object for that handle, including repeated selections of the same application. `getApp` and `getWindow` select objects without reading accessibility or replacing that snapshot; `getWindow` validates membership in the application’s current window list. An explicit `getAccessibilityTree` read invalidates the previous snapshot before dispatch. The facade automatically attaches the latest snapshot ID to element operations; callers use element IDs from the latest tree and do not assume an older ID still identifies the same control. Element IDs are opaque provider strings. Only an explicit `CuaWindow` captures screenshots or sends pointer input. The native helper retains its own application and optional-window snapshots behind that facade; the native layer and the facade snapshot remain separate. The external runtime continues to own variable persistence and completion serialization; the facade adds no evaluator. The [native object decision](2026-09-17-experimental-cua-object-api.md) owns evaluator independence, and the [shared helper decision](2026-10-09-cua-shared-native-helper.md) owns connection lifetime. +Within one API instance, the object facade keeps one current accessibility snapshot per application handle, shared by every application and window object for that handle, including repeated selections of the same application. `getApp` and `getWindow` select objects without reading accessibility or replacing that snapshot; native window operations reject closed or foreign windows. An explicit `getAccessibilityTree` read invalidates the previous snapshot before dispatch. The facade automatically attaches the latest snapshot ID to element operations; callers use element IDs from the latest tree and do not assume an older ID still identifies the same control. Element IDs are opaque provider strings. Only an explicit `CuaWindow` captures screenshots or sends pointer input. The native helper retains its own application and optional-window snapshots behind that facade; the native layer and the facade snapshot remain separate. The external runtime continues to own variable persistence and completion serialization; the facade adds no evaluator. The [native object decision](2026-09-17-experimental-cua-object-api.md) owns evaluator independence, and the [shared helper decision](2026-10-09-cua-shared-native-helper.md) owns connection lifetime. ## Alternatives considered diff --git a/.agents/notes/implemented/architecture/2026-11-05-cua-scoped-surfaces-and-input.zh.md b/.agents/notes/implemented/architecture/2026-11-05-cua-scoped-surfaces-and-input.zh.md index 8a3f152e67..f65ec242cb 100644 --- a/.agents/notes/implemented/architecture/2026-11-05-cua-scoped-surfaces-and-input.zh.md +++ b/.agents/notes/implemented/architecture/2026-11-05-cua-scoped-surfaces-and-input.zh.md @@ -16,7 +16,7 @@ Provider 的启动能力是静态的。平台、支持的方法、应用级辅 键盘输入把按键与修饰键分开：`pressKey(key, modifiers?)` 在按住所列修饰键时按下并释放一个键，其键盘修饰键列表是 provider 定义的普通字符串数组。`sendKeyEvent(key, direction)` 只发送一次按下或抬起，不隐式发送相反事件。因此调用方可以保持修饰键或拆分手势；两个方法都不接受组合键字符串，共享 schema 也不枚举平台按键名。 -在单个 API 实例内，对象门面按应用句柄保存一份当前辅助功能快照，该句柄的所有应用对象和窗口对象共享这份快照，包括对同一应用的重复选择。`getApp` 和 `getWindow` 只选择对象，不读取辅助功能或替换该快照；`getWindow` 会核验窗口属于应用的当前窗口列表。显式 `getAccessibilityTree` 读取会在分派前使上一份快照失效。门面会为元素操作自动附加最新快照 ID；调用方应使用最新树中的元素 ID，不应假定旧 ID 仍指向同一控件。元素 ID 是 provider 定义的不透明字符串。只有显式 `CuaWindow` 可以截图或发送指针输入。原生 helper 在该门面之后保留自己的应用快照和可选窗口快照；原生层与门面快照相互独立。外部运行时继续负责变量持久化和最终表达式序列化；门面不加入 evaluator。[原生对象决定](2026-09-17-experimental-cua-object-api.zh.md)负责 evaluator 独立性，[共享 helper 决定](2026-10-09-cua-shared-native-helper.zh.md)负责连接生命期。 +在单个 API 实例内，对象门面按应用句柄保存一份当前辅助功能快照，该句柄的所有应用对象和窗口对象共享这份快照，包括对同一应用的重复选择。`getApp` 和 `getWindow` 只选择对象，不读取辅助功能或替换该快照；原生窗口操作会拒绝已关闭或属于其他应用的窗口。显式 `getAccessibilityTree` 读取会在分派前使上一份快照失效。门面会为元素操作自动附加最新快照 ID；调用方应使用最新树中的元素 ID，不应假定旧 ID 仍指向同一控件。元素 ID 是 provider 定义的不透明字符串。只有显式 `CuaWindow` 可以截图或发送指针输入。原生 helper 在该门面之后保留自己的应用快照和可选窗口快照；原生层与门面快照相互独立。外部运行时继续负责变量持久化和最终表达式序列化；门面不加入 evaluator。[原生对象决定](2026-09-17-experimental-cua-object-api.zh.md)负责 evaluator 独立性，[共享 helper 决定](2026-10-09-cua-shared-native-helper.zh.md)负责连接生命期。 ## 考虑过的替代方案 diff --git a/docs/module-graph.md b/docs/module-graph.md index cc0d1e4f2c..a3a28f92fc 100644 --- a/docs/module-graph.md +++ b/docs/module-graph.md @@ -952,7 +952,6 @@ flowchart TD pkg_creation_recommendations --> pkg_skill pkg_creation_recommendations --> pkg_tools pkg_cua --> pkg_attachment - pkg_cua --> pkg_error pkg_cua --> pkg_ptc_runtime_node pkg_cua --> pkg_scope pkg_cua --> pkg_user_questions @@ -1822,7 +1821,7 @@ flowchart TD | [`workspace-changes`](../packages/deliverables/workspace-changes) | `deliverables` | [`agent`](../packages/core/agent), [`fs`](../packages/fs/fs), [`session`](../packages/core/session), [`subprocess`](../packages/subprocess/subprocess), [`tools`](../packages/core/tools) | | [`action-cards`](../packages/experimental/action-cards) | `experimental` | [`llm`](../packages/llm/llm), [`session`](../packages/core/session), [`tools`](../packages/core/tools), [`typert-protocol`](../packages/typert/protocol) | | [`creation-recommendations`](../packages/experimental/creation-recommendations) | `experimental` | [`llm`](../packages/llm/llm), [`session`](../packages/core/session), [`skill`](../packages/skill/skill), [`tools`](../packages/core/tools) | -| [`cua`](../packages/experimental/cua) | `experimental` | [`attachment`](../packages/attachment/attachment), [`error`](../packages/util/error), [`ptc-runtime-node`](../packages/ptc-runtime/ptc-runtime-node), [`scope`](../packages/core/scope), [`user-questions`](../packages/interaction/user-questions) | +| [`cua`](../packages/experimental/cua) | `experimental` | [`attachment`](../packages/attachment/attachment), [`ptc-runtime-node`](../packages/ptc-runtime/ptc-runtime-node), [`scope`](../packages/core/scope), [`user-questions`](../packages/interaction/user-questions) | | [`experimental-browser-use-chrome-devtools-mcp`](../packages/experimental/browser-use-chrome-devtools-mcp) | `experimental` | [`agent`](../packages/core/agent), [`browser-use`](../packages/browser-use/browser-use), [`system-prompt`](../packages/core/system-prompt), [`tools`](../packages/core/tools), [`working-directory`](../packages/session/working-directory) | | [`experimental-browser-use-playwright-mcp`](../packages/experimental/browser-use-playwright-mcp) | `experimental` | [`agent`](../packages/core/agent), [`browser-use`](../packages/browser-use/browser-use), [`system-prompt`](../packages/core/system-prompt), [`tools`](../packages/core/tools), [`working-directory`](../packages/session/working-directory) | | [`experimental-browser-use-stagehand-native`](../packages/experimental/browser-use-stagehand-native) | `experimental` | [`agent`](../packages/core/agent), [`browser-use`](../packages/browser-use/browser-use), [`system-prompt`](../packages/core/system-prompt), [`tools`](../packages/core/tools) | diff --git a/docs/module-graph.zh.md b/docs/module-graph.zh.md index c9cf1d8d93..9e337229b4 100644 --- a/docs/module-graph.zh.md +++ b/docs/module-graph.zh.md @@ -954,7 +954,6 @@ flowchart TD pkg_creation_recommendations --> pkg_skill pkg_creation_recommendations --> pkg_tools pkg_cua --> pkg_attachment - pkg_cua --> pkg_error pkg_cua --> pkg_ptc_runtime_node pkg_cua --> pkg_scope pkg_cua --> pkg_user_questions @@ -1824,7 +1823,7 @@ flowchart TD | [`workspace-changes`](../packages/deliverables/workspace-changes) | `deliverables` | [`agent`](../packages/core/agent), [`fs`](../packages/fs/fs), [`session`](../packages/core/session), [`subprocess`](../packages/subprocess/subprocess), [`tools`](../packages/core/tools) | | [`action-cards`](../packages/experimental/action-cards) | `experimental` | [`llm`](../packages/llm/llm), [`session`](../packages/core/session), [`tools`](../packages/core/tools), [`typert-protocol`](../packages/typert/protocol) | | [`creation-recommendations`](../packages/experimental/creation-recommendations) | `experimental` | [`llm`](../packages/llm/llm), [`session`](../packages/core/session), [`skill`](../packages/skill/skill), [`tools`](../packages/core/tools) | -| [`cua`](../packages/experimental/cua) | `experimental` | [`attachment`](../packages/attachment/attachment), [`error`](../packages/util/error), [`ptc-runtime-node`](../packages/ptc-runtime/ptc-runtime-node), [`scope`](../packages/core/scope), [`user-questions`](../packages/interaction/user-questions) | +| [`cua`](../packages/experimental/cua) | `experimental` | [`attachment`](../packages/attachment/attachment), [`ptc-runtime-node`](../packages/ptc-runtime/ptc-runtime-node), [`scope`](../packages/core/scope), [`user-questions`](../packages/interaction/user-questions) | | [`experimental-browser-use-chrome-devtools-mcp`](../packages/experimental/browser-use-chrome-devtools-mcp) | `experimental` | [`agent`](../packages/core/agent), [`browser-use`](../packages/browser-use/browser-use), [`system-prompt`](../packages/core/system-prompt), [`tools`](../packages/core/tools), [`working-directory`](../packages/session/working-directory) | | [`experimental-browser-use-playwright-mcp`](../packages/experimental/browser-use-playwright-mcp) | `experimental` | [`agent`](../packages/core/agent), [`browser-use`](../packages/browser-use/browser-use), [`system-prompt`](../packages/core/system-prompt), [`tools`](../packages/core/tools), [`working-directory`](../packages/session/working-directory) | | [`experimental-browser-use-stagehand-native`](../packages/experimental/browser-use-stagehand-native) | `experimental` | [`agent`](../packages/core/agent), [`browser-use`](../packages/browser-use/browser-use), [`system-prompt`](../packages/core/system-prompt), [`tools`](../packages/core/tools) | diff --git a/packages/experimental/cua-macos/README.i18n.yaml b/packages/experimental/cua-macos/README.i18n.yaml index 7571c8004d..6ddbd9a57f 100644 --- a/packages/experimental/cua-macos/README.i18n.yaml +++ b/packages/experimental/cua-macos/README.i18n.yaml @@ -18,8 +18,8 @@ en: e9c78a8cbd68147c zh: 98c64b09f858edf7 /deepseek-ai-dsh-experimental-cua-macos/use-this-package/connect-an-external-execution-environment: - en: 7c460511bb1cee4c - zh: 2eba4fc997427e18 + en: 3cd638c0c59fd9cc + zh: 594bacf314d4cd08 /deepseek-ai-dsh-experimental-cua-macos/understand-the-implementation: en: 4e82b85a5aa72eee zh: 4e004b7a205d8688 diff --git a/packages/experimental/cua-macos/README.md b/packages/experimental/cua-macos/README.md index d139d988a3..2e57fbaf13 100644 --- a/packages/experimental/cua-macos/README.md +++ b/packages/experimental/cua-macos/README.md @@ -113,7 +113,7 @@ void await window.getScreenshot() Use the `cua` API to discover and select the application (`listApps`, `getState`, `findApps`, `getApp`); use the selected app to activate it, enumerate its windows, and select one. Window selection is required for screenshots and pointer input, not only when several windows exist. `getApp` and `app.getWindow(windowId)` return without reading AX; choose the window from `getWindows()` by its title and bounds. -`getWindow` validates the ID against the current application window list. Read `getAccessibilityTree()` explicitly before element operations. Screenshots and coordinate input do not require a successful AX read; absolute pixel coordinates require a delivered screenshot. An owned window without an AX surface returns an empty snapshot when explicitly observed. Within one API instance, the facade keeps one current accessibility snapshot per application handle, shared by every app and window object for that handle, including repeated selections. Reading any window replaces that snapshot; element operations use the latest snapshot ID, so use IDs from the latest tree and do not assume an older ID still identifies the same control. A failed read publishes no partial snapshot; read again without repeating the preceding input. Window objects own screenshots and pointer input. Keyboard methods stay on `app` because input follows the app’s current focus. +`getWindow` sends no native request; screenshots and pointer input reject a closed or foreign window. Read `getAccessibilityTree()` explicitly before element operations. Screenshots and coordinate input do not require a successful AX read; absolute pixel coordinates require a delivered screenshot. An owned window without an AX surface returns an empty snapshot when explicitly observed. Within one API instance, the facade keeps one current accessibility snapshot per application handle, shared by every app and window object for that handle, including repeated selections. Reading any window replaces that snapshot; element operations use the latest snapshot ID, so use IDs from the latest tree and do not assume an older ID still identifies the same control. A failed read publishes no partial snapshot; read again without repeating the preceding input. Window objects own screenshots and pointer input. Keyboard methods stay on `app` because input follows the app’s current focus. `mouseMove(target)` defaults to AX hover. An enabled target or ancestor must advertise both `AXShowAlternateUI` and `AXShowDefaultUI`. Moving into its descendants keeps the UI open, and leaving restores it. Without an AX route the helper returns `HOVER_UNAVAILABLE` before sending input. Click a visible control or use `performAction` with an advertised action. For browser content without these AX actions, use browser tools. Enable `systemPointerHover` only when moving the user’s visible cursor is acceptable. That route requires a foreground, unobscured target and leaves the cursor at the destination. @@ -191,7 +191,7 @@ Setup and discovery: - cua.getState(options?: { emit?: boolean }) -> { apps: { id: string, displayName: string, isRunning: true, pid: number }[], browsers: [] } — summarize running applications; this is not an accessibility observation. - cua.getApp(selector: string) -> App — select an application by name, bundle ID, absolute path, or running PID string, obtain consent, launch in the background if needed, and return without reading AX or activating the app. - app.getWindows(options?: { emit?: boolean }) -> { windowId: number, appName: string, title: string, bounds: { x: number, y: number, width: number, height: number }, isOnScreen: boolean, onCurrentSpace: boolean | null }[] — enumerate windows with global native bounds; null onCurrentSpace means unknown Space membership. -- app.getWindow(windowId: number) -> Window — select a window from the application’s current window list without reading AX; do this before any window screenshot or coordinate input. +- app.getWindow(windowId: number) -> Window — select a window by an ID from getWindows without reading AX; do this before any window screenshot or coordinate input, which rejects a closed or foreign window. - app.activate() -> ActionResult — request explicit foreground activation; use only when foreground access is required. - app.show() -> { dispatched: boolean, windowId: number | null, foregroundChanged: boolean | null } — unhide the app and restore its selected, focused, main, or sole window without activation. - window.show() -> { dispatched: boolean, windowId: number | null, foregroundChanged: boolean | null } — unhide and restore that explicit window without activation. @@ -223,7 +223,7 @@ Available at app or window level depending on the platform. - paste(text: string, options?: { format?: 'text' | 'html' }) -> ActionResult — deliver text or HTML through the app's Paste command, restoring prior clipboard contents unless another writer changes them; a Paste failure is surfaced without substituting typeText, and PASTE_NOT_CONSUMED means inspect before any retry. - pressKey(key: string, modifiers?: string[]) -> ActionResult — press and release one complete keystroke. For example pressKey('Tab', ['shift']) or pressKey('+'). Key and modifier names are platform-defined strings. Pass key and modifiers separately; do not pack them into a chord string. -Raw tools use { method, params }: include appHandle for app calls and windowId for window calls, and use snapshotId with elementId for element routes. The emit option is facade-only: emit is local output control, so raw callers must follow the raw operation schema for admitted params. getAccessibilityTree maps to getState; getWindow validates against listWindows; getWindows maps to listWindows; getScreenshot maps to screenshot; cua.getState maps to listApps; every other named operation uses the same method name. +Raw tools use { method, params }: include appHandle for app calls and windowId for window calls, and use snapshotId with elementId for element routes. The emit option is facade-only: emit is local output control, so raw callers must follow the raw operation schema for admitted params. getAccessibilityTree maps to getState; getWindow has no raw method; getWindows maps to listWindows; getScreenshot maps to screenshot; cua.getState maps to listApps; every other named operation uses the same method name. ``` #### Token effect diff --git a/packages/experimental/cua-macos/README.zh.md b/packages/experimental/cua-macos/README.zh.md index f8fed7b624..4a6be7b8ef 100644 --- a/packages/experimental/cua-macos/README.zh.md +++ b/packages/experimental/cua-macos/README.zh.md @@ -113,7 +113,7 @@ void await window.getScreenshot() 使用 `cua` API 发现并选择应用（`listApps`、`getState`、`findApps`、`getApp`）；使用选定的应用来激活它、枚举其窗口并选择窗口。截图和指针输入都要求选择窗口，不只在存在多个窗口时才需要。`getApp` 和 `app.getWindow(windowId)` 返回时不读取 AX；根据 `getWindows()` 中的标题与位置选择窗口。 -`getWindow` 根据当前应用窗口清单验证 ID。元素操作前须显式读取 `getAccessibilityTree()`。截图和坐标输入不要求 AX 读取成功；绝对像素坐标要求已交付的截图。已确认归属但没有 AX 界面的窗口在显式观察时返回空快照。在单个 API 实例内，门面按应用句柄保存一份当前辅助功能快照，该句柄的所有应用对象和窗口对象共享它，包括重复选择。读取任一窗口都会替换该快照；元素操作使用最新快照 ID，因此应使用最新树中的元素 ID，不应假定旧 ID 仍指向同一控件。读取失败不会发布部分快照；可以重新读取，但不要重复此前的输入。窗口对象负责截图和指针输入。键盘方法保留在 `app` 上，因为输入跟随应用的当前焦点。 +`getWindow` 不发送原生请求；截图和指针输入会拒绝已关闭或属于其他应用的窗口。元素操作前须显式读取 `getAccessibilityTree()`。截图和坐标输入不要求 AX 读取成功；绝对像素坐标要求已交付的截图。已确认归属但没有 AX 界面的窗口在显式观察时返回空快照。在单个 API 实例内，门面按应用句柄保存一份当前辅助功能快照，该句柄的所有应用对象和窗口对象共享它，包括重复选择。读取任一窗口都会替换该快照；元素操作使用最新快照 ID，因此应使用最新树中的元素 ID，不应假定旧 ID 仍指向同一控件。读取失败不会发布部分快照；可以重新读取，但不要重复此前的输入。窗口对象负责截图和指针输入。键盘方法保留在 `app` 上，因为输入跟随应用的当前焦点。 `mouseMove(target)` 默认通过 AX 悬停。目标或其祖先必须已启用，并同时声明 `AXShowAlternateUI` 和 `AXShowDefaultUI`。进入子控件会保持展开，离开后恢复默认界面。如果没有可用的 AX 路径，helper 返回 `HOVER_UNAVAILABLE`，不发送输入。可点击已经可见的控件，或用 `performAction` 执行控件声明的动作。浏览器内容缺少这些 AX 动作时，请使用浏览器工具。只有允许移动用户可见光标时才开启 `systemPointerHover`。这条路径要求目标在前台且未被遮挡，操作后光标停在目标位置。 @@ -191,7 +191,7 @@ Setup and discovery: - cua.getState(options?: { emit?: boolean }) -> { apps: { id: string, displayName: string, isRunning: true, pid: number }[], browsers: [] } — summarize running applications; this is not an accessibility observation. - cua.getApp(selector: string) -> App — select an application by name, bundle ID, absolute path, or running PID string, obtain consent, launch in the background if needed, and return without reading AX or activating the app. - app.getWindows(options?: { emit?: boolean }) -> { windowId: number, appName: string, title: string, bounds: { x: number, y: number, width: number, height: number }, isOnScreen: boolean, onCurrentSpace: boolean | null }[] — enumerate windows with global native bounds; null onCurrentSpace means unknown Space membership. -- app.getWindow(windowId: number) -> Window — select a window from the application’s current window list without reading AX; do this before any window screenshot or coordinate input. +- app.getWindow(windowId: number) -> Window — select a window by an ID from getWindows without reading AX; do this before any window screenshot or coordinate input, which rejects a closed or foreign window. - app.activate() -> ActionResult — request explicit foreground activation; use only when foreground access is required. - app.show() -> { dispatched: boolean, windowId: number | null, foregroundChanged: boolean | null } — unhide the app and restore its selected, focused, main, or sole window without activation. - window.show() -> { dispatched: boolean, windowId: number | null, foregroundChanged: boolean | null } — unhide and restore that explicit window without activation. @@ -223,7 +223,7 @@ Available at app or window level depending on the platform. - paste(text: string, options?: { format?: 'text' | 'html' }) -> ActionResult — deliver text or HTML through the app's Paste command, restoring prior clipboard contents unless another writer changes them; a Paste failure is surfaced without substituting typeText, and PASTE_NOT_CONSUMED means inspect before any retry. - pressKey(key: string, modifiers?: string[]) -> ActionResult — press and release one complete keystroke. For example pressKey('Tab', ['shift']) or pressKey('+'). Key and modifier names are platform-defined strings. Pass key and modifiers separately; do not pack them into a chord string. -Raw tools use { method, params }: include appHandle for app calls and windowId for window calls, and use snapshotId with elementId for element routes. The emit option is facade-only: emit is local output control, so raw callers must follow the raw operation schema for admitted params. getAccessibilityTree maps to getState; getWindow validates against listWindows; getWindows maps to listWindows; getScreenshot maps to screenshot; cua.getState maps to listApps; every other named operation uses the same method name. +Raw tools use { method, params }: include appHandle for app calls and windowId for window calls, and use snapshotId with elementId for element routes. The emit option is facade-only: emit is local output control, so raw callers must follow the raw operation schema for admitted params. getAccessibilityTree maps to getState; getWindow has no raw method; getWindows maps to listWindows; getScreenshot maps to screenshot; cua.getState maps to listApps; every other named operation uses the same method name. ``` #### Token 影响 diff --git a/packages/experimental/cua/README.i18n.yaml b/packages/experimental/cua/README.i18n.yaml index fffdfaa5c6..dee3c61085 100644 --- a/packages/experimental/cua/README.i18n.yaml +++ b/packages/experimental/cua/README.i18n.yaml @@ -15,8 +15,8 @@ en: 5db4d4fea25493b9 zh: 65ab64fc24eb0d09 /deepseek-ai-dsh-cua/use-this-package: - en: 77dbaaa91d5b92c8 - zh: 94c3a758a91476bd + en: 01344bd3eba672cd + zh: e26f37a224899a46 /deepseek-ai-dsh-cua/use-this-package/permission-controls: en: d92d5948a7941e8d zh: d43491ae6521b87e diff --git a/packages/experimental/cua/README.md b/packages/experimental/cua/README.md index 0b3d39fdd1..dd262aede7 100644 --- a/packages/experimental/cua/README.md +++ b/packages/experimental/cua/README.md @@ -31,7 +31,7 @@ Application consent supplies English, Chinese, and Japanese display text from `s Mount the [macOS provider](../cua-macos/README.md) to enable native operations on macOS. External PTC hosts import `connectCua` from `@deepseek-ai/dsh-cua/bridge` and supply the current `tools.cua` binding. Source compositions and native preparation belong to the provider. -Use `cua.getApp(selector)` to select an application; it obtains consent and optionally launches a stopped application in the background, without reading its accessibility tree. `cua.listApps` and `cua.getState` list running applications; `cua.findApps(query)` searches installed applications, including stopped ones. `app.activate()`, `app.getWindows()`, and `app.getWindow(windowId)` operate on the selected application; `getWindow` validates the ID against the current application window list and returns the object without reading AX. Select an application and one of its windows before any capture or pointer input. Read `app.getAccessibilityTree()` or `window.getAccessibilityTree()` explicitly before element operations. Screenshots remain available when AX reading fails. Within one API instance, the facade keeps one current accessibility snapshot per application handle across app and window objects, including repeated selections; each new read invalidates the previous snapshot. Element operations use the latest snapshot ID, so callers use IDs from the latest tree and do not assume an older ID still identifies the same control. Element IDs are opaque strings. +Use `cua.getApp(selector)` to select an application; it obtains consent and optionally launches a stopped application in the background, without reading its accessibility tree. `cua.listApps` and `cua.getState` list running applications; `cua.findApps(query)` searches installed applications, including stopped ones. `app.activate()`, `app.getWindows()`, and `app.getWindow(windowId)` operate on the selected application; `getWindow` returns the object without a native request; window operations reject closed or foreign windows. Select an application and one of its windows before any capture or pointer input. Read `app.getAccessibilityTree()` or `window.getAccessibilityTree()` explicitly before element operations. Screenshots remain available when AX reading fails. Within one API instance, the facade keeps one current accessibility snapshot per application handle across app and window objects, including repeated selections; each new read invalidates the previous snapshot. Element operations use the latest snapshot ID, so callers use IDs from the latest tree and do not assume an older ID still identifies the same control. Element IDs are opaque strings. Keyboard methods are available in the scopes a provider declares; the provider owns how input is delivered within each scope. A lost connection invalidates its handles and coordinates. Failed cleanup is reported before that connection is retired; select the application again and inspect fresh state, because earlier input may have partially executed. Screenshot image errors retain the failing stage and attachment error code. Invalid request arguments reject before dispatch with a readable method, parameter and requirement. diff --git a/packages/experimental/cua/README.zh.md b/packages/experimental/cua/README.zh.md index b79cc6fdb5..ee274413ef 100644 --- a/packages/experimental/cua/README.zh.md +++ b/packages/experimental/cua/README.zh.md @@ -31,7 +31,7 @@ kind: "package-reference" 挂载 [macOS 提供方](../cua-macos/README.zh.md)以在 macOS 上启用原生操作。外部 PTC 宿主从 `@deepseek-ai/dsh-cua/bridge` 导入 `connectCua`，并提供当前 `tools.cua` 绑定。源码组合与原生组件准备由提供方负责。 -使用 `cua.getApp(selector)` 选择应用；它会获取授权、按需在后台启动已停止的应用，不读取辅助功能树。`cua.listApps` 和 `cua.getState` 列出运行中的应用；`cua.findApps(query)` 搜索已安装应用，包括未运行的应用。`app.activate()`、`app.getWindows()` 和 `app.getWindow(windowId)` 操作选定的应用；`getWindow` 根据当前应用窗口清单验证 ID，然后返回对象，不读取 AX。进行任何截图或指针输入前，必须选择应用及其一个窗口。元素操作前须显式读取 `app.getAccessibilityTree()` 或 `window.getAccessibilityTree()`。AX 读取失败时仍可截图。在单个 API 实例内，门面按应用句柄保存一份当前辅助功能快照，该句柄的所有应用对象和窗口对象共享它，包括重复选择；每次新读取都会使上一份快照失效。元素操作使用最新快照 ID，因此调用方应使用最新树中的元素 ID，不应假定旧 ID 仍指向同一控件。元素 ID 是不透明字符串。 +使用 `cua.getApp(selector)` 选择应用；它会获取授权、按需在后台启动已停止的应用，不读取辅助功能树。`cua.listApps` 和 `cua.getState` 列出运行中的应用；`cua.findApps(query)` 搜索已安装应用，包括未运行的应用。`app.activate()`、`app.getWindows()` 和 `app.getWindow(windowId)` 操作选定的应用；`getWindow` 返回对象时不发送原生请求；窗口操作会拒绝已关闭或属于其他应用的窗口。进行任何截图或指针输入前，必须选择应用及其一个窗口。元素操作前须显式读取 `app.getAccessibilityTree()` 或 `window.getAccessibilityTree()`。AX 读取失败时仍可截图。在单个 API 实例内，门面按应用句柄保存一份当前辅助功能快照，该句柄的所有应用对象和窗口对象共享它，包括重复选择；每次新读取都会使上一份快照失效。元素操作使用最新快照 ID，因此调用方应使用最新树中的元素 ID，不应假定旧 ID 仍指向同一控件。元素 ID 是不透明字符串。 键盘方法可在 provider 声明的作用域中使用；每个作用域内的输入投递由 provider 负责。连接丢失会使其句柄和坐标失效。清理失败会先报告，再释放该连接；应重新选择应用并读取新状态，因为先前的输入可能已部分执行。截图图像错误会保留失败阶段和附件错误码。请求参数校验失败时，在分派前返回包含方法、参数和要求的可读消息。 diff --git a/packages/experimental/cua/package.json b/packages/experimental/cua/package.json index 410dd23cf7..440f321935 100644 --- a/packages/experimental/cua/package.json +++ b/packages/experimental/cua/package.json @@ -99,8 +99,7 @@ "@deepseek-ai/dsh-user-questions": "workspace:*", "@deepseek-ai/dsh-attachment": "workspace:*", "@deepseek-ai/dsh-scope": "workspace:*", - "@deepseek-ai/dsh-ptc-runtime-node": "workspace:*", - "@deepseek-ai/dsh-error": "workspace:*" + "@deepseek-ai/dsh-ptc-runtime-node": "workspace:*" }, "dependencies": { "@deepseek-ai/schemastery": "workspace:~", @@ -162,8 +161,7 @@ "@deepseek-ai/dsh-storage-json": "workspace:*", "@deepseek-ai/dsh-client-ui-user-questions": "workspace:*", "@deepseek-ai/dsh-client-ui-approval": "workspace:*", - "@deepseek-ai/dsh-client-ui-conversation": "workspace:*", - "@deepseek-ai/dsh-error": "workspace:*" + "@deepseek-ai/dsh-client-ui-conversation": "workspace:*" }, "dsh": { "client": { diff --git a/packages/experimental/cua/src/api.ts b/packages/experimental/cua/src/api.ts index 509b9b8676..b28d4e9f1c 100644 --- a/packages/experimental/cua/src/api.ts +++ b/packages/experimental/cua/src/api.ts @@ -1,6 +1,5 @@ /** Object facade over logged native operations; its caller owns evaluator persistence. */ import { Buffer } from 'node:buffer' -import { HarnessError } from '@deepseek-ai/dsh-error' import type { z } from 'zod' import type { JsonValue } from '@deepseek-ai/dsh-util-values' import { ActionNamesResult, ActionResult, AppResult, AppsResult, CapabilitiesResult, FoundAppsResult, HoverResult, Request, ScreenshotResult, ShowResult, StateResult, WindowsResult } from './protocol.ts' @@ -257,14 +256,11 @@ function createCuaObjects(call: ReturnType, sink: CuaObser await emit(sink, windows, options) return windows }, - async getWindow(windowId): Promise { - const parsed = WindowsResult.element.shape.windowId.safeParse(windowId) - if (!parsed.success) throw new HarnessError(`getWindow ${requestIssueMessage(parsed.error.issues, ['params', 'windowId'])}`, 'INVALID_ARGS') - const windows = WindowsResult.parse(await base.call('listWindows', {})) - if (!windows.some(window => window.windowId === windowId)) { - throw new HarnessError('Selected window is unavailable for this application; read getWindows again', 'CUA_WINDOW_UNAVAILABLE') - } - return createCuaWindow(appHandle, windowId, capabilities) + getWindow(windowId): Promise { + // Native window operations reject closed and foreign windows; selection only admits the ID they accept. + const request = Request.safeParse({ method: 'screenshot', params: { appHandle, windowId } }) + if (!request.success) return Promise.reject(new Error(`getWindow ${requestIssueMessage(request.error.issues)}`)) + return Promise.resolve(createCuaWindow(appHandle, windowId, capabilities)) }, } } diff --git a/packages/experimental/cua/src/guidance.ts b/packages/experimental/cua/src/guidance.ts index 89a32bd21a..c1ede9f30b 100644 --- a/packages/experimental/cua/src/guidance.ts +++ b/packages/experimental/cua/src/guidance.ts @@ -13,7 +13,7 @@ Setup and discovery: - cua.getState(options?: { emit?: boolean }) -> { apps: { id: string, displayName: string, isRunning: true, pid: number }[], browsers: [] } — summarize running applications; this is not an accessibility observation. - cua.getApp(selector: string) -> App — select an application by name, bundle ID, absolute path, or running PID string, obtain consent, launch in the background if needed, and return without reading AX or activating the app. - app.getWindows(options?: { emit?: boolean }) -> { windowId: number, appName: string, title: string, bounds: { x: number, y: number, width: number, height: number }, isOnScreen: boolean, onCurrentSpace: boolean | null }[] — enumerate windows with global native bounds; null onCurrentSpace means unknown Space membership. -- app.getWindow(windowId: number) -> Window — select a window from the application’s current window list without reading AX; do this before any window screenshot or coordinate input. +- app.getWindow(windowId: number) -> Window — select a window by an ID from getWindows without reading AX; do this before any window screenshot or coordinate input, which rejects a closed or foreign window. - app.activate() -> ActionResult — request explicit foreground activation; use only when foreground access is required. - app.show() -> { dispatched: boolean, windowId: number | null, foregroundChanged: boolean | null } — unhide the app and restore its selected, focused, main, or sole window without activation. - window.show() -> { dispatched: boolean, windowId: number | null, foregroundChanged: boolean | null } — unhide and restore that explicit window without activation. @@ -45,4 +45,4 @@ Available at app or window level depending on the platform. - paste(text: string, options?: { format?: 'text' | 'html' }) -> ActionResult — deliver text or HTML through the app's Paste command, restoring prior clipboard contents unless another writer changes them; a Paste failure is surfaced without substituting typeText, and PASTE_NOT_CONSUMED means inspect before any retry. - pressKey(key: string, modifiers?: string[]) -> ActionResult — press and release one complete keystroke. For example pressKey('Tab', ['shift']) or pressKey('+'). Key and modifier names are platform-defined strings. Pass key and modifiers separately; do not pack them into a chord string. -Raw tools use { method, params }: include appHandle for app calls and windowId for window calls, and use snapshotId with elementId for element routes. The emit option is facade-only: emit is local output control, so raw callers must follow the raw operation schema for admitted params. getAccessibilityTree maps to getState; getWindow validates against listWindows; getWindows maps to listWindows; getScreenshot maps to screenshot; cua.getState maps to listApps; every other named operation uses the same method name.` +Raw tools use { method, params }: include appHandle for app calls and windowId for window calls, and use snapshotId with elementId for element routes. The emit option is facade-only: emit is local output control, so raw callers must follow the raw operation schema for admitted params. getAccessibilityTree maps to getState; getWindow has no raw method; getWindows maps to listWindows; getScreenshot maps to screenshot; cua.getState maps to listApps; every other named operation uses the same method name.` diff --git a/packages/experimental/cua/src/types.ts b/packages/experimental/cua/src/types.ts index 0e8624b6ea..a9c4776acf 100644 --- a/packages/experimental/cua/src/types.ts +++ b/packages/experimental/cua/src/types.ts @@ -240,7 +240,8 @@ export interface CuaApp extends CuaSurface { /** * Select a specific window. * @param windowId - ID obtained from getWindows. - * @returns A window object from the app’s current window list, without reading or replacing its accessibility snapshot. + * @returns A window object without a native request; selection leaves the accessibility snapshot unchanged. + * Window operations reject a closed or foreign window. */ getWindow(windowId: number): Promise } diff --git a/packages/experimental/cua/tests/api.spec.ts b/packages/experimental/cua/tests/api.spec.ts index fb2bf5a204..2b023db37d 100644 --- a/packages/experimental/cua/tests/api.spec.ts +++ b/packages/experimental/cua/tests/api.spec.ts @@ -83,7 +83,7 @@ describe('application discovery', () => { const window = await app.getWindow(23) expect(await window.getScreenshot()).toEqual({ bytes: Uint8Array.from(Buffer.from(PNG, 'base64')), mimeType: 'image/png', pixelWidth: 1, pixelHeight: 1, windowId: 23 }) expect(f.dispatch.mock.calls.map(([operation]) => operation.method)).toEqual([ - 'getCapabilities', 'listApps', 'getApp', 'listWindows', 'screenshot', + 'getCapabilities', 'listApps', 'getApp', 'screenshot', ]) expect(await app.getWindows()).toEqual([windowInfo(23)]) expect(f.sink.text).not.toHaveBeenCalled() @@ -186,7 +186,7 @@ describe('mouseMove across observations', () => { } }) f.dispatch.mockRejectedValueOnce(new Error('WINDOW_CHANGED')) await expect(other.mouseMove([5, 6])).rejects.toThrow('WINDOW_CHANGED') - expect(f.dispatch).toHaveBeenCalledTimes(8) + expect(f.dispatch).toHaveBeenCalledTimes(7) }) }) @@ -452,15 +452,15 @@ describe('application and window observations', () => { } }) - it.each(['WINDOW_UNAVAILABLE', 'PERMISSION_DENIED', 'STALE_APP', 'SESSION_LOCKED'])('propagates %s while selecting a window without attempting image or input fallback', async (code) => { + it.each(['WINDOW_UNAVAILABLE', 'PERMISSION_DENIED', 'STALE_APP', 'SESSION_LOCKED'])('propagates %s from coordinate input on a selected window without image or input fallback', async (code) => { const f = fixture() const app = await f.api.getApp('Editor') - await app.getAccessibilityTree({ emit: false }) + const window = await app.getWindow(23) const failure = new Error(code) f.dispatch.mockRejectedValueOnce(failure) const calls = f.dispatch.mock.calls.length - await expect(app.getWindow(23)).rejects.toBe(failure) - expect(f.dispatch.mock.calls.slice(calls).map(([operation]) => operation.method)).toEqual(['listWindows']) + await expect(window.click([1, 2])).rejects.toBe(failure) + expect(f.dispatch.mock.calls.slice(calls).map(([operation]) => operation.method)).toEqual(['click']) }) it('captures and clicks the explicitly selected window among several', async () => { @@ -629,15 +629,12 @@ describe('application and window observations', () => { }) describe('screenshots', () => { - it('rejects a missing or foreign window without reading AX or taking a screenshot', async () => { + it('selects windows without a native request and rejects invalid IDs locally', async () => { const f = fixture() const app = await f.api.getApp('Editor') - await expect(app.getWindow(24)).rejects.toMatchObject({ name: 'HarnessError', code: 'CUA_WINDOW_UNAVAILABLE' }) - f.setWindows([]) - await expect(app.getWindow(23)).rejects.toMatchObject({ name: 'HarnessError', code: 'CUA_WINDOW_UNAVAILABLE' }) - expect(f.dispatch.mock.calls.map(([request]) => request.method)).toEqual(['getCapabilities', 'getApp', 'listWindows', 'listWindows']) - await expect(app.getWindow(0)).rejects.toMatchObject({ name: 'HarnessError', code: 'INVALID_ARGS', message: 'getWindow windowId: Too small: expected number to be >0' }) - expect(f.dispatch).toHaveBeenCalledTimes(4) + expect((await app.getWindow(24)).windowId).toBe(24) + await expect(app.getWindow(0)).rejects.toThrow('getWindow windowId: Too small: expected number to be >0') + expect(f.dispatch.mock.calls.map(([request]) => request.method)).toEqual(['getCapabilities', 'getApp']) }) it.each(['APP_ACCESS_REVOKED', 'WINDOW_UNAVAILABLE', 'STALE_APP'])('preserves %s when capturing a previously selected window', async (code) => { @@ -989,10 +986,10 @@ describe('input actions', () => { const failure = new Error('Application was restarted; select it again') f.dispatch.mockRejectedValueOnce(failure) await expect(app.typeText('hello')).rejects.toBe(failure) - expect(f.dispatch).toHaveBeenCalledTimes(6) + expect(f.dispatch).toHaveBeenCalledTimes(5) f.dispatch.mockResolvedValueOnce({ dispatched: 'yes' }) await expect(window.click(element('0'))).rejects.toThrow() - expect(f.dispatch).toHaveBeenCalledTimes(7) + expect(f.dispatch).toHaveBeenCalledTimes(6) }) it('rejects invalid selection inputs before dispatch', async () => { diff --git a/packages/experimental/cua/tests/bridge.spec.ts b/packages/experimental/cua/tests/bridge.spec.ts index 7bba3c4e0b..be2edddb9f 100644 --- a/packages/experimental/cua/tests/bridge.spec.ts +++ b/packages/experimental/cua/tests/bridge.spec.ts @@ -27,7 +27,6 @@ it('uses the current tool binding when a retained app is called by another execu const first = vi.fn(async (operation: CuaOperation): Promise => { if (operation.method === 'getCapabilities') return capabilities if (operation.method === 'getApp') return selected - if (operation.method === 'listWindows') return snapshot.windows if (operation.method === 'getState') return { ...snapshot, windowId: operation.params.windowId ?? null } return snapshot }) @@ -41,7 +40,7 @@ it('uses the current tool binding when a retained app is called by another execu current = second await expect(window.click(element('0'))).resolves.toEqual({ dispatched: true }) - expect(first).toHaveBeenCalledTimes(4) + expect(first).toHaveBeenCalledTimes(3) expect(second).toHaveBeenCalledWith({ method: 'click', params: { appHandle: 'app-17', snapshotId: 'snapshot-1', elementId: '0', windowId: 23 } }) expect(text).not.toHaveBeenCalled() current = first @@ -71,7 +70,6 @@ it('returns screenshot bytes and metadata with one image-bearing tool call and n const tool = vi.fn(async (operation: CuaOperation): Promise => { if (operation.method === 'getCapabilities') return capabilities if (operation.method === 'getApp') return selected - if (operation.method === 'listWindows') return snapshot.windows if (operation.method === 'getState') return { ...snapshot, windowId: operation.params.windowId ?? null } return { mimeType: 'image/png', data: PNG, pixelWidth: 1, pixelHeight: 1, windowId: 23, bounds, coordinateSpace: 'screenshot-pixels', @@ -83,6 +81,6 @@ it('returns screenshot bytes and metadata with one image-bearing tool call and n text.mockClear() expect(await window.getScreenshot()).toEqual({ bytes: Uint8Array.from(Buffer.from(PNG, 'base64')), mimeType: 'image/png', pixelWidth: 1, pixelHeight: 1, windowId: 23 }) - expect(tool.mock.calls.map(([operation]) => operation.method)).toEqual(['getCapabilities', 'getApp', 'listWindows', 'screenshot']) + expect(tool.mock.calls.map(([operation]) => operation.method)).toEqual(['getCapabilities', 'getApp', 'screenshot']) expect(text).not.toHaveBeenCalled() }) diff --git a/packages/experimental/cua/tsconfig.host.json b/packages/experimental/cua/tsconfig.host.json index 4bf0b7ddda..2386f83640 100644 --- a/packages/experimental/cua/tsconfig.host.json +++ b/packages/experimental/cua/tsconfig.host.json @@ -74,9 +74,6 @@ }, { "path": "../../boot/config-editor" - }, - { - "path": "../../util/error" } ], "exclude": [ diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index f33fef7d42..d016fe8deb 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -9894,9 +9894,6 @@ importers: '@deepseek-ai/dsh-config-editor': specifier: workspace:* version: link:../../boot/config-editor - '@deepseek-ai/dsh-error': - specifier: workspace:* - version: link:../../util/error '@deepseek-ai/dsh-fs-local': specifier: workspace:* version: link:../../fs/fs-local diff --git a/snapshots/session/cua-native/system-prompt.expected.md b/snapshots/session/cua-native/system-prompt.expected.md index 04fb520f49..e8dfa822c9 100644 --- a/snapshots/session/cua-native/system-prompt.expected.md +++ b/snapshots/session/cua-native/system-prompt.expected.md @@ -48,7 +48,7 @@ Setup and discovery: - cua.getState(options?: { emit?: boolean }) -> { apps: { id: string, displayName: string, isRunning: true, pid: number }[], browsers: [] } — summarize running applications; this is not an accessibility observation. - cua.getApp(selector: string) -> App — select an application by name, bundle ID, absolute path, or running PID string, obtain consent, launch in the background if needed, and return without reading AX or activating the app. - app.getWindows(options?: { emit?: boolean }) -> { windowId: number, appName: string, title: string, bounds: { x: number, y: number, width: number, height: number }, isOnScreen: boolean, onCurrentSpace: boolean | null }[] — enumerate windows with global native bounds; null onCurrentSpace means unknown Space membership. -- app.getWindow(windowId: number) -> Window — select a window from the application’s current window list without reading AX; do this before any window screenshot or coordinate input. +- app.getWindow(windowId: number) -> Window — select a window by an ID from getWindows without reading AX; do this before any window screenshot or coordinate input, which rejects a closed or foreign window. - app.activate() -> ActionResult — request explicit foreground activation; use only when foreground access is required. - app.show() -> { dispatched: boolean, windowId: number | null, foregroundChanged: boolean | null } — unhide the app and restore its selected, focused, main, or sole window without activation. - window.show() -> { dispatched: boolean, windowId: number | null, foregroundChanged: boolean | null } — unhide and restore that explicit window without activation. @@ -80,6 +80,6 @@ Available at app or window level depending on the platform. - paste(text: string, options?: { format?: 'text' | 'html' }) -> ActionResult — deliver text or HTML through the app's Paste command, restoring prior clipboard contents unless another writer changes them; a Paste failure is surfaced without substituting typeText, and PASTE_NOT_CONSUMED means inspect before any retry. - pressKey(key: string, modifiers?: string[]) -> ActionResult — press and release one complete keystroke. For example pressKey('Tab', ['shift']) or pressKey('+'). Key and modifier names are platform-defined strings. Pass key and modifiers separately; do not pack them into a chord string. -Raw tools use { method, params }: include appHandle for app calls and windowId for window calls, and use snapshotId with elementId for element routes. The emit option is facade-only: emit is local output control, so raw callers must follow the raw operation schema for admitted params. getAccessibilityTree maps to getState; getWindow validates against listWindows; getWindows maps to listWindows; getScreenshot maps to screenshot; cua.getState maps to listApps; every other named operation uses the same method name. +Raw tools use { method, params }: include appHandle for app calls and windowId for window calls, and use snapshotId with elementId for element routes. The emit option is facade-only: emit is local output control, so raw callers must follow the raw operation schema for admitted params. getAccessibilityTree maps to getState; getWindow has no raw method; getWindows maps to listWindows; getScreenshot maps to screenshot; cua.getState maps to listApps; every other named operation uses the same method name. Maximum click holdMs: 5000 ms. diff --git a/snapshots/session/cua-ptc/session.v4.jsonl b/snapshots/session/cua-ptc/session.v4.jsonl index f9d25d914b..dd3c7c5ffa 100644 --- a/snapshots/session/cua-ptc/session.v4.jsonl +++ b/snapshots/session/cua-ptc/session.v4.jsonl @@ -12,7 +12,7 @@ {"type":"request/header","data":{"header":{"config":{"provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"tools":"{{tools}}"},"reason":"initial"}} {"type":"request/context","data":{"provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"}} {"type":"session/title","data":{"title":"Use tools.cua from Node PTC","messageSeqs":[8],"source":{"kind":"fallback"}}} -{"type":"assistant/message","data":{"turn":1,"step":1,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-ptc-native-result","name":"run_code","arguments":"{\"code\":\"const { Buffer } = await import('node:buffer');\\nconst app = await tools.cua({ method: 'getApp', params: { selector: 'test.fixture' } });\\nconst state = await tools.cua({ method: 'getState', params: { appHandle: app.appHandle, windowId: 7 } });\\nconst shot = await tools.cua({ method: 'screenshot', params: { appHandle: app.appHandle, windowId: 7 } });\\nconst bytes = Buffer.from(shot.data, 'base64');\\nreturn { appHandle: app.appHandle, hasIncrement: state.tree.includes('Increment'), pngBytes: bytes.length, signature: bytes.subarray(0, 8).toString('hex'), imageWidth: shot.attachment.width, imageHeight: shot.attachment.height };\",\"description\":\"Read native CUA values and receive one screenshot\"}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:4}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1794024973829,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1794024973829,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-ptc-native-result","name":"run_code","arguments":"{\"code\":\"const { Buffer } = await import('node:buffer');\\nconst app = await tools.cua({ method: 'getApp', params: { selector: 'test.fixture' } });\\nconst state = await tools.cua({ method: 'getState', params: { appHandle: app.appHandle, windowId: 7 } });\\nconst shot = await tools.cua({ method: 'screenshot', params: { appHandle: app.appHandle, windowId: 7 } });\\nconst bytes = Buffer.from(shot.data, 'base64');\\nreturn { appHandle: app.appHandle, hasIncrement: state.tree.includes('Increment'), pngBytes: bytes.length, signature: bytes.subarray(0, 8).toString('hex'), imageWidth: shot.attachment.width, imageHeight: shot.attachment.height };\",\"description\":\"Read native CUA values and receive one screenshot\"}"}}},{"type":"chunk","time":1794024973829,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1794024973829,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} +{"type":"assistant/message","data":{"turn":1,"step":1,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-ptc-native-result","name":"run_code","arguments":"{\"code\":\"const { Buffer } = await import('node:buffer');\\nconst app = await tools.cua({ method: 'getApp', params: { selector: 'test.fixture' } });\\nconst state = await tools.cua({ method: 'getState', params: { appHandle: app.appHandle, windowId: 7 } });\\nconst shot = await tools.cua({ method: 'screenshot', params: { appHandle: app.appHandle, windowId: 7 } });\\nconst bytes = Buffer.from(shot.data, 'base64');\\nreturn { appHandle: app.appHandle, hasIncrement: state.tree.includes('Increment'), pngBytes: bytes.length, signature: bytes.subarray(0, 8).toString('hex'), imageWidth: shot.attachment.width, imageHeight: shot.attachment.height };\",\"description\":\"Read native CUA values and receive one screenshot\"}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:4}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1794031895518,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1794031895518,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-ptc-native-result","name":"run_code","arguments":"{\"code\":\"const { Buffer } = await import('node:buffer');\\nconst app = await tools.cua({ method: 'getApp', params: { selector: 'test.fixture' } });\\nconst state = await tools.cua({ method: 'getState', params: { appHandle: app.appHandle, windowId: 7 } });\\nconst shot = await tools.cua({ method: 'screenshot', params: { appHandle: app.appHandle, windowId: 7 } });\\nconst bytes = Buffer.from(shot.data, 'base64');\\nreturn { appHandle: app.appHandle, hasIncrement: state.tree.includes('Increment'), pngBytes: bytes.length, signature: bytes.subarray(0, 8).toString('hex'), imageWidth: shot.attachment.width, imageHeight: shot.attachment.height };\",\"description\":\"Read native CUA values and receive one screenshot\"}"}}},{"type":"chunk","time":1794031895518,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1794031895518,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} {"type":"tool/call","data":{"turn":1,"step":1,"callId":"cua-ptc-native-result","name":"run_code","arguments":"{\"code\":\"const { Buffer } = await import('node:buffer');\\nconst app = await tools.cua({ method: 'getApp', params: { selector: 'test.fixture' } });\\nconst state = await tools.cua({ method: 'getState', params: { appHandle: app.appHandle, windowId: 7 } });\\nconst shot = await tools.cua({ method: 'screenshot', params: { appHandle: app.appHandle, windowId: 7 } });\\nconst bytes = Buffer.from(shot.data, 'base64');\\nreturn { appHandle: app.appHandle, hasIncrement: state.tree.includes('Increment'), pngBytes: bytes.length, signature: bytes.subarray(0, 8).toString('hex'), imageWidth: shot.attachment.width, imageHeight: shot.attachment.height };\",\"description\":\"Read native CUA values and receive one screenshot\"}"}} {"type":"tool/ptc-dispatch-start","data":{"rootCallId":"cua-ptc-native-result","parentCallId":"cua-ptc-native-result","subCallId":"cua-ptc-native-result:ptc:1","name":"cua","arguments":{"method":"getApp","params":{"selector":"test.fixture"}}}} {"type":"tool/ptc-dispatch","data":{"rootCallId":"cua-ptc-native-result","parentCallId":"cua-ptc-native-result","subCallId":"cua-ptc-native-result:ptc:1","name":"cua","arguments":{"method":"getApp","params":{"selector":"test.fixture"}},"isError":false,"content":[{"type":"text","text":"{\"appHandle\":\"fixture-app\",\"name\":\"Fixture\",\"bundleId\":\"test.fixture\",\"running\":true}"}]}} @@ -26,7 +26,7 @@ {"type":"agent/inbox/spliced","data":{"target":"next-step","start":0,"removedCount":1,"inserted":[]}} {"type":"step/start","data":{"turn":1,"step":2}} {"type":"user/message","data":{"content":[{"type":"text","text":"{\"pixelWidth\":1,\"pixelHeight\":1,\"windowId\":7,\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"coordinateSpace\":\"screenshot-pixels\"}"},{"type":"image","attachment":{"attachmentId":"sha256:999f1d1527ee7e79266f16add5430fff76b1225d742464a5b1ff1f02971bb8ee","mediaType":"image/png","width":1,"height":1,"bytes":90}}],"source":{"kind":"ptc-mode"},"role":"user","id":"{{message:6}}"},"surfaceOp":"append"} -{"type":"assistant/message","data":{"turn":1,"step":2,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-ptc-ax-failure","name":"run_code","arguments":"{\"code\":\"const { writeFile, unlink } = await import('node:fs/promises');\\nconst { resolve } = await import('node:path');\\nconst { pathToFileURL } = await import('node:url');\\nconst { connectCua } = await import(pathToFileURL(resolve('cua-bridge.mjs')).href);\\nawait writeFile('fixture-ax-failure', '');\\ntry {\\n const cua = connectCua(() => tools.cua);\\n const app = await cua.getApp('test.fixture');\\n const windows = await app.getWindows({ emit: false });\\n const window = await app.getWindow(windows[0].windowId);\\n const failures = [];\\n for (const surface of [app, window]) {\\n try { await surface.getAccessibilityTree(); throw new Error('Expected AX failure'); }\\n catch (error) { if (!error.message.includes('Fixture accessibility read failed')) throw error; failures.push(error.message); }\\n }\\n const shot = await window.getScreenshot();\\n let elementRefused = false;\\n try { await window.invoke('0'); }\\n catch (error) { if (!error.message.includes('Read getAccessibilityTree')) throw error; elementRefused = true; }\\n if (!elementRefused) throw new Error('Element input accepted without an AX snapshot');\\n return { axFailures: failures.length, screenshotWindow: shot.windowId, pngBytes: shot.bytes.length, elementRefused };\\n} finally { await unlink('fixture-ax-failure'); }\",\"description\":\"Capture through the object API despite application and window AX failures\"}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:7}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1794024974646,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1794024974646,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-ptc-ax-failure","name":"run_code","arguments":"{\"code\":\"const { writeFile, unlink } = await import('node:fs/promises');\\nconst { resolve } = await import('node:path');\\nconst { pathToFileURL } = await import('node:url');\\nconst { connectCua } = await import(pathToFileURL(resolve('cua-bridge.mjs')).href);\\nawait writeFile('fixture-ax-failure', '');\\ntry {\\n const cua = connectCua(() => tools.cua);\\n const app = await cua.getApp('test.fixture');\\n const windows = await app.getWindows({ emit: false });\\n const window = await app.getWindow(windows[0].windowId);\\n const failures = [];\\n for (const surface of [app, window]) {\\n try { await surface.getAccessibilityTree(); throw new Error('Expected AX failure'); }\\n catch (error) { if (!error.message.includes('Fixture accessibility read failed')) throw error; failures.push(error.message); }\\n }\\n const shot = await window.getScreenshot();\\n let elementRefused = false;\\n try { await window.invoke('0'); }\\n catch (error) { if (!error.message.includes('Read getAccessibilityTree')) throw error; elementRefused = true; }\\n if (!elementRefused) throw new Error('Element input accepted without an AX snapshot');\\n return { axFailures: failures.length, screenshotWindow: shot.windowId, pngBytes: shot.bytes.length, elementRefused };\\n} finally { await unlink('fixture-ax-failure'); }\",\"description\":\"Capture through the object API despite application and window AX failures\"}"}}},{"type":"chunk","time":1794024974646,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1794024974646,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} +{"type":"assistant/message","data":{"turn":1,"step":2,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-ptc-ax-failure","name":"run_code","arguments":"{\"code\":\"const { writeFile, unlink } = await import('node:fs/promises');\\nconst { resolve } = await import('node:path');\\nconst { pathToFileURL } = await import('node:url');\\nconst { connectCua } = await import(pathToFileURL(resolve('cua-bridge.mjs')).href);\\nawait writeFile('fixture-ax-failure', '');\\ntry {\\n const cua = connectCua(() => tools.cua);\\n const app = await cua.getApp('test.fixture');\\n const windows = await app.getWindows({ emit: false });\\n const window = await app.getWindow(windows[0].windowId);\\n const failures = [];\\n for (const surface of [app, window]) {\\n try { await surface.getAccessibilityTree(); throw new Error('Expected AX failure'); }\\n catch (error) { if (!error.message.includes('Fixture accessibility read failed')) throw error; failures.push(error.message); }\\n }\\n const shot = await window.getScreenshot();\\n let elementRefused = false;\\n try { await window.invoke('0'); }\\n catch (error) { if (!error.message.includes('Read getAccessibilityTree')) throw error; elementRefused = true; }\\n if (!elementRefused) throw new Error('Element input accepted without an AX snapshot');\\n return { axFailures: failures.length, screenshotWindow: shot.windowId, pngBytes: shot.bytes.length, elementRefused };\\n} finally { await unlink('fixture-ax-failure'); }\",\"description\":\"Capture through the object API despite application and window AX failures\"}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:7}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1794031896364,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1794031896364,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-ptc-ax-failure","name":"run_code","arguments":"{\"code\":\"const { writeFile, unlink } = await import('node:fs/promises');\\nconst { resolve } = await import('node:path');\\nconst { pathToFileURL } = await import('node:url');\\nconst { connectCua } = await import(pathToFileURL(resolve('cua-bridge.mjs')).href);\\nawait writeFile('fixture-ax-failure', '');\\ntry {\\n const cua = connectCua(() => tools.cua);\\n const app = await cua.getApp('test.fixture');\\n const windows = await app.getWindows({ emit: false });\\n const window = await app.getWindow(windows[0].windowId);\\n const failures = [];\\n for (const surface of [app, window]) {\\n try { await surface.getAccessibilityTree(); throw new Error('Expected AX failure'); }\\n catch (error) { if (!error.message.includes('Fixture accessibility read failed')) throw error; failures.push(error.message); }\\n }\\n const shot = await window.getScreenshot();\\n let elementRefused = false;\\n try { await window.invoke('0'); }\\n catch (error) { if (!error.message.includes('Read getAccessibilityTree')) throw error; elementRefused = true; }\\n if (!elementRefused) throw new Error('Element input accepted without an AX snapshot');\\n return { axFailures: failures.length, screenshotWindow: shot.windowId, pngBytes: shot.bytes.length, elementRefused };\\n} finally { await unlink('fixture-ax-failure'); }\",\"description\":\"Capture through the object API despite application and window AX failures\"}"}}},{"type":"chunk","time":1794031896364,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1794031896364,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} {"type":"tool/call","data":{"turn":1,"step":2,"callId":"cua-ptc-ax-failure","name":"run_code","arguments":"{\"code\":\"const { writeFile, unlink } = await import('node:fs/promises');\\nconst { resolve } = await import('node:path');\\nconst { pathToFileURL } = await import('node:url');\\nconst { connectCua } = await import(pathToFileURL(resolve('cua-bridge.mjs')).href);\\nawait writeFile('fixture-ax-failure', '');\\ntry {\\n const cua = connectCua(() => tools.cua);\\n const app = await cua.getApp('test.fixture');\\n const windows = await app.getWindows({ emit: false });\\n const window = await app.getWindow(windows[0].windowId);\\n const failures = [];\\n for (const surface of [app, window]) {\\n try { await surface.getAccessibilityTree(); throw new Error('Expected AX failure'); }\\n catch (error) { if (!error.message.includes('Fixture accessibility read failed')) throw error; failures.push(error.message); }\\n }\\n const shot = await window.getScreenshot();\\n let elementRefused = false;\\n try { await window.invoke('0'); }\\n catch (error) { if (!error.message.includes('Read getAccessibilityTree')) throw error; elementRefused = true; }\\n if (!elementRefused) throw new Error('Element input accepted without an AX snapshot');\\n return { axFailures: failures.length, screenshotWindow: shot.windowId, pngBytes: shot.bytes.length, elementRefused };\\n} finally { await unlink('fixture-ax-failure'); }\",\"description\":\"Capture through the object API despite application and window AX failures\"}"}} {"type":"tool/ptc-dispatch-start","data":{"rootCallId":"cua-ptc-ax-failure","parentCallId":"cua-ptc-ax-failure","subCallId":"cua-ptc-ax-failure:ptc:1","name":"cua","arguments":{"method":"getCapabilities","params":{}}}} {"type":"tool/ptc-dispatch","data":{"rootCallId":"cua-ptc-ax-failure","parentCallId":"cua-ptc-ax-failure","subCallId":"cua-ptc-ax-failure:ptc:1","name":"cua","arguments":{"method":"getCapabilities","params":{}},"isError":false,"content":[{"type":"text","text":"{\"sessionLocked\":false,\"permissions\":{\"accessibility\":true,\"screenRecording\":true},\"platform\":\"macos\",\"methods\":[\"activate\",\"click\",\"drag\",\"findApps\",\"getApp\",\"getCapabilities\",\"getState\",\"invoke\",\"listActions\",\"listApps\",\"listWindows\",\"mouseMove\",\"paste\",\"performAction\",\"pressKey\",\"screenshot\",\"scroll\",\"selectText\",\"setValue\",\"show\",\"typeText\"],\"appLevelAccessibility\":true,\"keyboard\":[\"app\"],\"browser\":false,\"coordinateSpace\":\"screenshot-pixels\"}"}]}} @@ -34,20 +34,18 @@ {"type":"tool/ptc-dispatch","data":{"rootCallId":"cua-ptc-ax-failure","parentCallId":"cua-ptc-ax-failure","subCallId":"cua-ptc-ax-failure:ptc:2","name":"cua","arguments":{"method":"getApp","params":{"selector":"test.fixture"}},"isError":false,"content":[{"type":"text","text":"{\"appHandle\":\"fixture-app\",\"name\":\"Fixture\",\"bundleId\":\"test.fixture\",\"running\":true}"}]}} {"type":"tool/ptc-dispatch-start","data":{"rootCallId":"cua-ptc-ax-failure","parentCallId":"cua-ptc-ax-failure","subCallId":"cua-ptc-ax-failure:ptc:3","name":"cua","arguments":{"method":"listWindows","params":{"appHandle":"fixture-app"}}}} {"type":"tool/ptc-dispatch","data":{"rootCallId":"cua-ptc-ax-failure","parentCallId":"cua-ptc-ax-failure","subCallId":"cua-ptc-ax-failure:ptc:3","name":"cua","arguments":{"method":"listWindows","params":{"appHandle":"fixture-app"}},"isError":false,"content":[{"type":"text","text":"[{\"windowId\":7,\"appName\":\"Fixture\",\"title\":\"CUA fixture\",\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"isOnScreen\":true,\"onCurrentSpace\":true},{\"windowId\":8,\"appName\":\"Fixture\",\"title\":\"AXless canvas\",\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"isOnScreen\":true,\"onCurrentSpace\":true}]"}]}} -{"type":"tool/ptc-dispatch-start","data":{"rootCallId":"cua-ptc-ax-failure","parentCallId":"cua-ptc-ax-failure","subCallId":"cua-ptc-ax-failure:ptc:4","name":"cua","arguments":{"method":"listWindows","params":{"appHandle":"fixture-app"}}}} -{"type":"tool/ptc-dispatch","data":{"rootCallId":"cua-ptc-ax-failure","parentCallId":"cua-ptc-ax-failure","subCallId":"cua-ptc-ax-failure:ptc:4","name":"cua","arguments":{"method":"listWindows","params":{"appHandle":"fixture-app"}},"isError":false,"content":[{"type":"text","text":"[{\"windowId\":7,\"appName\":\"Fixture\",\"title\":\"CUA fixture\",\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"isOnScreen\":true,\"onCurrentSpace\":true},{\"windowId\":8,\"appName\":\"Fixture\",\"title\":\"AXless canvas\",\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"isOnScreen\":true,\"onCurrentSpace\":true}]"}]}} -{"type":"tool/ptc-dispatch-start","data":{"rootCallId":"cua-ptc-ax-failure","parentCallId":"cua-ptc-ax-failure","subCallId":"cua-ptc-ax-failure:ptc:5","name":"cua","arguments":{"method":"getState","params":{"appHandle":"fixture-app"}}}} -{"type":"tool/ptc-dispatch","data":{"rootCallId":"cua-ptc-ax-failure","parentCallId":"cua-ptc-ax-failure","subCallId":"cua-ptc-ax-failure:ptc:5","name":"cua","arguments":{"method":"getState","params":{"appHandle":"fixture-app"}},"isError":true,"content":[{"type":"text","text":"Error: NATIVE_ERROR: Fixture accessibility read failed"}]}} -{"type":"tool/ptc-dispatch-start","data":{"rootCallId":"cua-ptc-ax-failure","parentCallId":"cua-ptc-ax-failure","subCallId":"cua-ptc-ax-failure:ptc:6","name":"cua","arguments":{"method":"getState","params":{"appHandle":"fixture-app","windowId":7}}}} -{"type":"tool/ptc-dispatch","data":{"rootCallId":"cua-ptc-ax-failure","parentCallId":"cua-ptc-ax-failure","subCallId":"cua-ptc-ax-failure:ptc:6","name":"cua","arguments":{"method":"getState","params":{"appHandle":"fixture-app","windowId":7}},"isError":true,"content":[{"type":"text","text":"Error: NATIVE_ERROR: Fixture accessibility read failed"}]}} -{"type":"tool/ptc-dispatch-start","data":{"rootCallId":"cua-ptc-ax-failure","parentCallId":"cua-ptc-ax-failure","subCallId":"cua-ptc-ax-failure:ptc:7","name":"cua","arguments":{"method":"screenshot","params":{"appHandle":"fixture-app","windowId":7}}}} -{"type":"tool/ptc-dispatch","data":{"rootCallId":"cua-ptc-ax-failure","parentCallId":"cua-ptc-ax-failure","subCallId":"cua-ptc-ax-failure:ptc:7","name":"cua","arguments":{"method":"screenshot","params":{"appHandle":"fixture-app","windowId":7}},"isError":false,"content":[{"type":"text","text":"{\"pixelWidth\":1,\"pixelHeight\":1,\"windowId\":7,\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"coordinateSpace\":\"screenshot-pixels\"}"},{"type":"image","attachment":{"attachmentId":"sha256:999f1d1527ee7e79266f16add5430fff76b1225d742464a5b1ff1f02971bb8ee","mediaType":"image/png","width":1,"height":1,"bytes":90}}]}} +{"type":"tool/ptc-dispatch-start","data":{"rootCallId":"cua-ptc-ax-failure","parentCallId":"cua-ptc-ax-failure","subCallId":"cua-ptc-ax-failure:ptc:4","name":"cua","arguments":{"method":"getState","params":{"appHandle":"fixture-app"}}}} +{"type":"tool/ptc-dispatch","data":{"rootCallId":"cua-ptc-ax-failure","parentCallId":"cua-ptc-ax-failure","subCallId":"cua-ptc-ax-failure:ptc:4","name":"cua","arguments":{"method":"getState","params":{"appHandle":"fixture-app"}},"isError":true,"content":[{"type":"text","text":"Error: NATIVE_ERROR: Fixture accessibility read failed"}]}} +{"type":"tool/ptc-dispatch-start","data":{"rootCallId":"cua-ptc-ax-failure","parentCallId":"cua-ptc-ax-failure","subCallId":"cua-ptc-ax-failure:ptc:5","name":"cua","arguments":{"method":"getState","params":{"appHandle":"fixture-app","windowId":7}}}} +{"type":"tool/ptc-dispatch","data":{"rootCallId":"cua-ptc-ax-failure","parentCallId":"cua-ptc-ax-failure","subCallId":"cua-ptc-ax-failure:ptc:5","name":"cua","arguments":{"method":"getState","params":{"appHandle":"fixture-app","windowId":7}},"isError":true,"content":[{"type":"text","text":"Error: NATIVE_ERROR: Fixture accessibility read failed"}]}} +{"type":"tool/ptc-dispatch-start","data":{"rootCallId":"cua-ptc-ax-failure","parentCallId":"cua-ptc-ax-failure","subCallId":"cua-ptc-ax-failure:ptc:6","name":"cua","arguments":{"method":"screenshot","params":{"appHandle":"fixture-app","windowId":7}}}} +{"type":"tool/ptc-dispatch","data":{"rootCallId":"cua-ptc-ax-failure","parentCallId":"cua-ptc-ax-failure","subCallId":"cua-ptc-ax-failure:ptc:6","name":"cua","arguments":{"method":"screenshot","params":{"appHandle":"fixture-app","windowId":7}},"isError":false,"content":[{"type":"text","text":"{\"pixelWidth\":1,\"pixelHeight\":1,\"windowId\":7,\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"coordinateSpace\":\"screenshot-pixels\"}"},{"type":"image","attachment":{"attachmentId":"sha256:999f1d1527ee7e79266f16add5430fff76b1225d742464a5b1ff1f02971bb8ee","mediaType":"image/png","width":1,"height":1,"bytes":90}}]}} {"type":"tool/result","data":{"turn":1,"step":2,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-ptc-ax-failure"},"toolCallId":"cua-ptc-ax-failure","content":[{"type":"text","text":"{\n \"axFailures\": 2,\n \"screenshotWindow\": 7,\n \"pngBytes\": 90,\n \"elementRefused\": true\n}"}],"isError":false,"id":"{{message:8}}"}},"sourceEventSeqs":[28],"surfaceOp":"append"} {"type":"agent/inbox/spliced","data":{"target":"next-step","start":0,"inserted":[{"content":[{"type":"text","text":"{\"pixelWidth\":1,\"pixelHeight\":1,\"windowId\":7,\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"coordinateSpace\":\"screenshot-pixels\"}"},{"type":"image","attachment":{"attachmentId":"sha256:999f1d1527ee7e79266f16add5430fff76b1225d742464a5b1ff1f02971bb8ee","mediaType":"image/png","width":1,"height":1,"bytes":90}}],"source":{"kind":"ptc-mode"},"role":"user","id":"{{message:9}}"}]}} {"type":"step/end","data":{"turn":1,"step":2}} {"type":"agent/inbox/spliced","data":{"target":"next-step","start":0,"removedCount":1,"inserted":[]}} {"type":"step/start","data":{"turn":1,"step":3}} {"type":"user/message","data":{"content":[{"type":"text","text":"{\"pixelWidth\":1,\"pixelHeight\":1,\"windowId\":7,\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"coordinateSpace\":\"screenshot-pixels\"}"},{"type":"image","attachment":{"attachmentId":"sha256:999f1d1527ee7e79266f16add5430fff76b1225d742464a5b1ff1f02971bb8ee","mediaType":"image/png","width":1,"height":1,"bytes":90}}],"source":{"kind":"ptc-mode"},"role":"user","id":"{{message:9}}"},"surfaceOp":"append"} -{"type":"assistant/message","data":{"turn":1,"step":3,"message":{"role":"assistant","content":[{"type":"text","text":"Received native CUA values and PNG screenshots through PTC. The object API captured a 90-byte PNG after two AX read failures and refused element input without an accessibility snapshot."}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:10}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1794024974851,"chunk":{"type":"block-start","index":0,"blockType":"text"}},{"type":"chunk","time":1794024974851,"chunk":{"type":"block-end","index":0,"block":{"type":"text","text":"Received native CUA values and PNG screenshots through PTC. The object API captured a 90-byte PNG after two AX read failures and refused element input without an accessibility snapshot."}}},{"type":"chunk","time":1794024974851,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1794024974851,"chunk":{"type":"finish","reason":{"kind":"stop"}}}]},"surfaceOp":"append"} +{"type":"assistant/message","data":{"turn":1,"step":3,"message":{"role":"assistant","content":[{"type":"text","text":"Received native CUA values and PNG screenshots through PTC. The object API captured a 90-byte PNG after two AX read failures and refused element input without an accessibility snapshot."}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:10}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1794031896552,"chunk":{"type":"block-start","index":0,"blockType":"text"}},{"type":"chunk","time":1794031896552,"chunk":{"type":"block-end","index":0,"block":{"type":"text","text":"Received native CUA values and PNG screenshots through PTC. The object API captured a 90-byte PNG after two AX read failures and refused element input without an accessibility snapshot."}}},{"type":"chunk","time":1794031896552,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1794031896552,"chunk":{"type":"finish","reason":{"kind":"stop"}}}]},"surfaceOp":"append"} {"type":"step/end","data":{"turn":1,"step":3}} {"type":"turn/end","data":{"turn":1,"reason":{"kind":"completed"}}} diff --git a/snapshots/session/cua-ptc/system-prompt.expected.md b/snapshots/session/cua-ptc/system-prompt.expected.md index be8ab5aed8..26186e4c92 100644 --- a/snapshots/session/cua-ptc/system-prompt.expected.md +++ b/snapshots/session/cua-ptc/system-prompt.expected.md @@ -50,7 +50,7 @@ Setup and discovery: - cua.getState(options?: { emit?: boolean }) -> { apps: { id: string, displayName: string, isRunning: true, pid: number }[], browsers: [] } — summarize running applications; this is not an accessibility observation. - cua.getApp(selector: string) -> App — select an application by name, bundle ID, absolute path, or running PID string, obtain consent, launch in the background if needed, and return without reading AX or activating the app. - app.getWindows(options?: { emit?: boolean }) -> { windowId: number, appName: string, title: string, bounds: { x: number, y: number, width: number, height: number }, isOnScreen: boolean, onCurrentSpace: boolean | null }[] — enumerate windows with global native bounds; null onCurrentSpace means unknown Space membership. -- app.getWindow(windowId: number) -> Window — select a window from the application’s current window list without reading AX; do this before any window screenshot or coordinate input. +- app.getWindow(windowId: number) -> Window — select a window by an ID from getWindows without reading AX; do this before any window screenshot or coordinate input, which rejects a closed or foreign window. - app.activate() -> ActionResult — request explicit foreground activation; use only when foreground access is required. - app.show() -> { dispatched: boolean, windowId: number | null, foregroundChanged: boolean | null } — unhide the app and restore its selected, focused, main, or sole window without activation. - window.show() -> { dispatched: boolean, windowId: number | null, foregroundChanged: boolean | null } — unhide and restore that explicit window without activation. @@ -82,7 +82,7 @@ Available at app or window level depending on the platform. - paste(text: string, options?: { format?: 'text' | 'html' }) -> ActionResult — deliver text or HTML through the app's Paste command, restoring prior clipboard contents unless another writer changes them; a Paste failure is surfaced without substituting typeText, and PASTE_NOT_CONSUMED means inspect before any retry. - pressKey(key: string, modifiers?: string[]) -> ActionResult — press and release one complete keystroke. For example pressKey('Tab', ['shift']) or pressKey('+'). Key and modifier names are platform-defined strings. Pass key and modifiers separately; do not pack them into a chord string. -Raw tools use { method, params }: include appHandle for app calls and windowId for window calls, and use snapshotId with elementId for element routes. The emit option is facade-only: emit is local output control, so raw callers must follow the raw operation schema for admitted params. getAccessibilityTree maps to getState; getWindow validates against listWindows; getWindows maps to listWindows; getScreenshot maps to screenshot; cua.getState maps to listApps; every other named operation uses the same method name. +Raw tools use { method, params }: include appHandle for app calls and windowId for window calls, and use snapshotId with elementId for element routes. The emit option is facade-only: emit is local output control, so raw callers must follow the raw operation schema for admitted params. getAccessibilityTree maps to getState; getWindow has no raw method; getWindows maps to listWindows; getScreenshot maps to screenshot; cua.getState maps to listApps; every other named operation uses the same method name. Maximum click holdMs: 5000 ms. diff --git a/snapshots/session/ptc-script-retained/system-prompt.expected.md b/snapshots/session/ptc-script-retained/system-prompt.expected.md index 63e91315c0..e61732cc68 100644 --- a/snapshots/session/ptc-script-retained/system-prompt.expected.md +++ b/snapshots/session/ptc-script-retained/system-prompt.expected.md @@ -50,7 +50,7 @@ Setup and discovery: - cua.getState(options?: { emit?: boolean }) -> { apps: { id: string, displayName: string, isRunning: true, pid: number }[], browsers: [] } — summarize running applications; this is not an accessibility observation. - cua.getApp(selector: string) -> App — select an application by name, bundle ID, absolute path, or running PID string, obtain consent, launch in the background if needed, and return without reading AX or activating the app. - app.getWindows(options?: { emit?: boolean }) -> { windowId: number, appName: string, title: string, bounds: { x: number, y: number, width: number, height: number }, isOnScreen: boolean, onCurrentSpace: boolean | null }[] — enumerate windows with global native bounds; null onCurrentSpace means unknown Space membership. -- app.getWindow(windowId: number) -> Window — select a window from the application’s current window list without reading AX; do this before any window screenshot or coordinate input. +- app.getWindow(windowId: number) -> Window — select a window by an ID from getWindows without reading AX; do this before any window screenshot or coordinate input, which rejects a closed or foreign window. - app.activate() -> ActionResult — request explicit foreground activation; use only when foreground access is required. - app.show() -> { dispatched: boolean, windowId: number | null, foregroundChanged: boolean | null } — unhide the app and restore its selected, focused, main, or sole window without activation. - window.show() -> { dispatched: boolean, windowId: number | null, foregroundChanged: boolean | null } — unhide and restore that explicit window without activation. @@ -82,7 +82,7 @@ Available at app or window level depending on the platform. - paste(text: string, options?: { format?: 'text' | 'html' }) -> ActionResult — deliver text or HTML through the app's Paste command, restoring prior clipboard contents unless another writer changes them; a Paste failure is surfaced without substituting typeText, and PASTE_NOT_CONSUMED means inspect before any retry. - pressKey(key: string, modifiers?: string[]) -> ActionResult — press and release one complete keystroke. For example pressKey('Tab', ['shift']) or pressKey('+'). Key and modifier names are platform-defined strings. Pass key and modifiers separately; do not pack them into a chord string. -Raw tools use { method, params }: include appHandle for app calls and windowId for window calls, and use snapshotId with elementId for element routes. The emit option is facade-only: emit is local output control, so raw callers must follow the raw operation schema for admitted params. getAccessibilityTree maps to getState; getWindow validates against listWindows; getWindows maps to listWindows; getScreenshot maps to screenshot; cua.getState maps to listApps; every other named operation uses the same method name. +Raw tools use { method, params }: include appHandle for app calls and windowId for window calls, and use snapshotId with elementId for element routes. The emit option is facade-only: emit is local output control, so raw callers must follow the raw operation schema for admitted params. getAccessibilityTree maps to getState; getWindow has no raw method; getWindows maps to listWindows; getScreenshot maps to screenshot; cua.getState maps to listApps; every other named operation uses the same method name. Maximum click holdMs: 5000 ms. From 9e802994cae8c181cf7b59b39257be7c7f6aece1 Mon Sep 17 00:00:00 2001 From: Dudu-0223 Date: Mon, 9 Nov 2026 16:47:12 +0800 Subject: [PATCH 5/9] fix(cua): preserve trees when writability discovery fails --- .../experimental/cua-macos/README.i18n.yaml | 4 +- packages/experimental/cua-macos/README.md | 2 +- packages/experimental/cua-macos/README.zh.md | 2 +- .../native/Sources/CuaNative/AXAccess.swift | 11 ++++- .../native/tests/swift/AXFailureTests.swift | 45 +++++++++++++++++++ .../cua-native/workspace/native-helper.mjs | 7 ++- snapshots/session/cua-ptc/consent-fixture.mjs | 2 +- .../session/cua-ptc/replay.override.json | 35 ++++++++++++++- snapshots/session/cua-ptc/session.v4.jsonl | 23 ++++++++-- 9 files changed, 119 insertions(+), 12 deletions(-) diff --git a/packages/experimental/cua-macos/README.i18n.yaml b/packages/experimental/cua-macos/README.i18n.yaml index 335f01b093..9235ea51c3 100644 --- a/packages/experimental/cua-macos/README.i18n.yaml +++ b/packages/experimental/cua-macos/README.i18n.yaml @@ -15,8 +15,8 @@ en: 1777e7bf6785f986 zh: e3daac1074ebb74d /deepseek-ai-dsh-experimental-cua-macos/use-this-package: - en: 8dd51156261bd414 - zh: 1e585b51193d891c + en: 98f739bfb1f2554b + zh: 4d4809f851df6b35 /deepseek-ai-dsh-experimental-cua-macos/use-this-package/connect-an-external-execution-environment: en: a08d37e868a0a860 zh: 5dc8a2cdd165cc7e diff --git a/packages/experimental/cua-macos/README.md b/packages/experimental/cua-macos/README.md index cdbc5dfc2e..bed625dc5a 100644 --- a/packages/experimental/cua-macos/README.md +++ b/packages/experimental/cua-macos/README.md @@ -91,7 +91,7 @@ Raw `getState` exposes validated state fields and the complete text for the requ AX observations show menu-bar entries by default. Use `app.getAccessibilityTree({ includeMenus: true })` to expand menu contents and `app.getAccessibilityTree({ details: true })` for identifiers, global-point bounds and complete action names. These options also apply to raw `getState`. Every read replaces the application's current snapshot; use element IDs from that read. Details do not bypass the configured node and depth limits. Truncation identifies the reached limit. -Values and placeholders are separate. Named, stateful or actionable groups and scroll areas retain their own rows. Repeated names, descriptions and help text on one node are shown once. Values, placeholders and URLs keep their distinct meanings even when their text matches; body text shared by distinct nodes remains intact. Writability is explicit on relevant controls, including `AXIncrementor`; `unknown` means the system did not report it. Missing optional state fields do not imply false. UTF-16 ranges outside the observed value are marked unavailable. Compact tree labels can omit AppKit target and selector details. Pass the exact name returned by `listActions(elementId)` to `performAction`. +Values and placeholders are separate. Named, stateful or actionable groups and scroll areas retain their own rows. Repeated names, descriptions and help text on one node are shown once. Values, placeholders and URLs keep their distinct meanings even when their text matches; body text shared by distinct nodes remains intact. Writability is explicit on relevant controls, including `AXIncrementor`; `unknown` means the system did not report it. Missing optional state fields do not imply false. During tree observation, an AXValue or AXSelectedTextRange writability probe returning -25200 leaves that field unknown and preserves the remaining nodes. Actual writes remain strict; permission, communication and invalid-element errors still propagate. UTF-16 ranges outside the observed value are marked unavailable. Compact tree labels can omit AppKit target and selector details. Pass the exact name returned by `listActions(elementId)` to `performAction`. Observations emit by default and also return their value. In persistent PTC, use `void await app.getAccessibilityTree()` for one emitted copy, or `emit:false` and one explicit print. Capturing a screenshot after an emitted tree can fail independently; retain the successful AX output. diff --git a/packages/experimental/cua-macos/README.zh.md b/packages/experimental/cua-macos/README.zh.md index 2dad1db12c..4367d116c3 100644 --- a/packages/experimental/cua-macos/README.zh.md +++ b/packages/experimental/cua-macos/README.zh.md @@ -91,7 +91,7 @@ CUA 工具在读取 AX、窗口标题、截图、激活和输入前检查应用 AX 观察默认只展示菜单栏入口。使用 `app.getAccessibilityTree({ includeMenus: true })` 展开菜单内容，使用 `app.getAccessibilityTree({ details: true })` 读取标识符、全局 point 坐标和完整动作名。原始 `getState` 也接受这两个选项。每次读取都会替换应用当前的快照；应使用该次读取返回的元素 ID。详情不会绕过配置的节点数和深度上限；截断提示会指明触发的上限。 -实际值与占位文字分别展示。有名称、状态或可用动作的分组和滚动区域保留独立行。同一节点中重复的名称、描述和帮助只显示一次。实际值、占位文字和 URL 即使文字相同也保留各自含义，不删除不同节点共有的正文。相关控件明确展示可写性，包括 `AXIncrementor`；`unknown` 表示系统未报告。缺少可选状态字段不代表 false。超出当前值长度的 UTF-16 选区标为不可用。树中的动作简称可能省略 AppKit 的 target 和 selector 信息。调用 `performAction` 时使用 `listActions(elementId)` 返回的完整名称。 +实际值与占位文字分别展示。有名称、状态或可用动作的分组和滚动区域保留独立行。同一节点中重复的名称、描述和帮助只显示一次。实际值、占位文字和 URL 即使文字相同也保留各自含义，不删除不同节点共有的正文。相关控件明确展示可写性，包括 `AXIncrementor`；`unknown` 表示系统未报告。缺少可选状态字段不代表 false。读树时，AXValue 或 AXSelectedTextRange 的可写性探测返回 -25200，会将该字段保留为未知并继续读取其他节点。实际写入仍严格校验；权限、通信和失效元素错误仍向上传播。超出当前值长度的 UTF-16 选区标为不可用。树中的动作简称可能省略 AppKit 的 target 和 selector 信息。调用 `performAction` 时使用 `listActions(elementId)` 返回的完整名称。 观察方法默认自动输出，同时返回结果。在 persistent PTC 中使用 `void await app.getAccessibilityTree()` 输出一次，或指定 `emit:false` 后自行打印一次。AX 输出后的截图可能单独失败，已成功取得的 AX 输出仍保留。 diff --git a/packages/experimental/cua-macos/native/Sources/CuaNative/AXAccess.swift b/packages/experimental/cua-macos/native/Sources/CuaNative/AXAccess.swift index 3b39ab1f02..53333a0b14 100644 --- a/packages/experimental/cua-macos/native/Sources/CuaNative/AXAccess.swift +++ b/packages/experimental/cua-macos/native/Sources/CuaNative/AXAccess.swift @@ -208,6 +208,12 @@ final class AXAccess { guard error == .success else { throw failure(error, "read writability of \(name)") } return value } + /// Observation keeps unsupported writability unknown; mutation callers use strict writable checks. + static func observationWritability(role: String, probe: (String, Bool) throws -> Bool?) rethrows -> (value: Bool?, selection: Bool?) { + let value = AXTextNode.valueRoles.contains(role) ? try probe("AXValue", true) : nil + let selection = AXTextNode.textRoles.contains(role) ? try probe("AXSelectedTextRange", true) : nil + return (value, selection) + } func requireEnabled(_ element: AXUIElement) throws { if try boolean(element, "AXEnabled") == false { throw NativeFailure(code: "NATIVE_ERROR", message: "Accessibility element is disabled") } } @@ -586,10 +592,11 @@ final class AXAccess { let placeholder = try stringValue(observed("AXPlaceholderValue"), attribute: "AXPlaceholderValue") let description = try stringValue(observed("AXDescription"), attribute: "AXDescription") let actionNames = try self.actions(element) - let valueWritable = AXTextNode.valueRoles.contains(role) ? try writable(element, "AXValue") : nil + let writability = try Self.observationWritability(role: role) { try writable(element, $0, forDiscovery: $1) } + let valueWritable = writability.value let enabled = try booleanValue(observed("AXEnabled"), attribute: "AXEnabled") let selected = try booleanValue(observed("AXSelected"), attribute: "AXSelected") - let selectionWritable = AXTextNode.textRoles.contains(role) ? try writable(element, "AXSelectedTextRange") : nil + let selectionWritable = writability.selection let selectedRange = AXTextNode.textRoles.contains(role) ? try textRange(observed("AXSelectedTextRange")) : nil let selectedText = AXTextNode.textRoles.contains(role) ? try stringValue(observed("AXSelectedText"), attribute: "AXSelectedText") : nil let identifier = try stringValue(observed("AXIdentifier"), attribute: "AXIdentifier") diff --git a/packages/experimental/cua-macos/native/tests/swift/AXFailureTests.swift b/packages/experimental/cua-macos/native/tests/swift/AXFailureTests.swift index 2c25c943e4..5ededcff6e 100644 --- a/packages/experimental/cua-macos/native/tests/swift/AXFailureTests.swift +++ b/packages/experimental/cua-macos/native/tests/swift/AXFailureTests.swift @@ -183,3 +183,48 @@ func consumedContainerMetadataPreservesStaleAndCommunicationErrors(role: String) try access.nodeObservation(attributes: ["AXRole": "AXGroup" as CFString, "AXChildren": try wrappedAXError(.illegalArgument)]) { _ in nil }.attribute("AXChildren") } } + +@Test(arguments: ["AXValue", "AXSelectedTextRange"]) +func observationRetainsUnknownWritabilityAndFollowingNodes(failedAttribute: String) throws { + let access = AXAccess(timeout: 1, cancellation: CancellationLatch()) + var lines: [String] = [] + for title in ["Affected control", "Following control"] { + let writable = try AXAccess.observationWritability(role: "AXTextField") { name, discovery in + try access.attributeWritability(true, error: title == "Affected control" && name == failedAttribute ? .failure : .success, + name: name, forDiscovery: discovery) + } + let node = AXTextNode(index: nil, role: "AXTextField", title: title, value: "Text", description: nil, + actions: [], enabled: true, valueWritable: writable.value, selectionWritable: writable.selection, + selected: nil, selectedRange: nil, selectedText: nil) + lines.append(AXTextFormatter.line(node, level: 0)) + } + let unknown = failedAttribute == "AXValue" ? "valueWritable" : "selectionWritable" + #expect(lines[0].contains("\(unknown)=unknown")) + #expect(!lines[0].contains("\(unknown)=true")) + #expect(lines[0].contains("Affected control")) + #expect(lines[1].contains("Following control")) + #expect(lines[1].contains("valueWritable=true selectionWritable=true")) + do { + _ = try access.attributeWritability(true, error: .failure, name: failedAttribute, forDiscovery: false) + Issue.record("Mutation admitted unknown writability") + } catch let error as NativeFailure { + #expect(error.accessibilityError == -25200) + #expect(error.code == "NATIVE_ERROR") + } +} + +@Test(arguments: ["AXValue", "AXSelectedTextRange"]) +func observationWritabilityPreservesPermissionCommunicationAndInvalidElementErrors(attribute: String) { + let access = AXAccess(timeout: 1, cancellation: CancellationLatch()) + for code in [AXError.apiDisabled, .cannotComplete, .invalidUIElement, .illegalArgument] { + do { + _ = try AXAccess.observationWritability(role: "AXTextField") { name, discovery in + try access.attributeWritability(true, error: name == attribute ? code : .success, name: name, forDiscovery: discovery) + } + Issue.record("Observation hid an actionable AX error") + } catch let error as NativeFailure { + #expect(error.accessibilityError == code.rawValue) + #expect(error.code == (code == .apiDisabled ? "PERMISSION_DENIED" : "NATIVE_ERROR")) + } catch { Issue.record("Unexpected error: \(error)") } + } +} diff --git a/snapshots/session/cua-native/workspace/native-helper.mjs b/snapshots/session/cua-native/workspace/native-helper.mjs index d5bdceb896..764093e559 100755 --- a/snapshots/session/cua-native/workspace/native-helper.mjs +++ b/snapshots/session/cua-native/workspace/native-helper.mjs @@ -160,6 +160,8 @@ function createApplicationState(socket, record, application, captureBroker, conn } if (method === 'listWindows') return [window, axlessWindow] if (params.windowId !== undefined && ![7, 8, 9].includes(params.windowId)) refuse('WINDOW_UNAVAILABLE', 'Window is closed or belongs to another application') + const writableFailureFile = new URL('./fixture-ax-writability-failure', import.meta.url) + const writableFailure = existsSync(writableFailureFile) ? readFileSync(writableFailureFile, 'utf8') : undefined if (method === 'getState') { snapshots.delete(params.windowId ?? null) if (params.windowId === undefined) observedWindows.clear() @@ -177,7 +179,7 @@ function createApplicationState(socket, record, application, captureBroker, conn if (axless) return { appHandle: app.appHandle, windowId: 8, snapshotId, tree: '', elements: [], windows: [axlessWindow], truncated: false } return { appHandle: app.appHandle, windowId: params.windowId ?? null, - snapshotId: `${scope}-snapshot-${generation}`, tree: `AX scope=${params.windowId === undefined ? 'application' : 'window'} offsets=UTF-16${params.details ? ' bounds=global-points' : ''}\n- [0] AXButton "Increment"${params.details ? ' id="increment" actions=["AXPress"] bounds=[60,80,100,20]' : ''}\n- [1] AXTextField valueWritable=true selectionWritable=true value=${JSON.stringify(application.value)} placeholder="Fixture text"\n- AXStaticText "Count: ${application.count}"${application.hovering ? '\n- AXStaticText "Hover information"' : ''}\n- [2] AXMenuBarItem "File"${params.includeMenus ? '\n - [3] AXMenuItem "Fixture command"' : '\nMenu contents omitted; request includeMenus=true to expand.'}`, + snapshotId: `${scope}-snapshot-${generation}`, tree: `AX scope=${params.windowId === undefined ? 'application' : 'window'} offsets=UTF-16${params.details ? ' bounds=global-points' : ''}\n- [0] AXButton "Increment"${params.details ? ' id="increment" actions=["AXPress"] bounds=[60,80,100,20]' : ''}\n- [1] AXTextField valueWritable=${writableFailure === 'AXValue' ? 'unknown' : 'true'} selectionWritable=${writableFailure === 'AXSelectedTextRange' ? 'unknown' : 'true'} value=${JSON.stringify(application.value)} placeholder="Fixture text"\n- AXStaticText "Count: ${application.count}"${application.hovering ? '\n- AXStaticText "Hover information"' : ''}\n- [2] AXMenuBarItem "File"${params.includeMenus ? '\n - [3] AXMenuItem "Fixture command"' : '\nMenu contents omitted; request includeMenus=true to expand.'}`, elements: [{ elementId: '0', role: 'AXButton', title: 'Increment', value: null, description: null, identifier: null, bounds: [60, 80, 100, 20], actions: ['AXPress'], enabled: true }], windows: [params.windowId === 9 ? transformedWindow : window], truncated: false } @@ -203,6 +205,9 @@ function createApplicationState(socket, record, application, captureBroker, conn } if (['click', 'mouseMove', 'scroll', 'drag'].includes(method) && params.elementId === undefined && !observedWindows.has(params.windowId)) refuse('STALE_SNAPSHOT', 'Window has no observation; read getAccessibilityTree or take a screenshot before coordinate input') + if ((method === 'setValue' && writableFailure === 'AXValue') || (method === 'selectText' && writableFailure === 'AXSelectedTextRange')) { + refuse('NATIVE_ERROR', `Accessibility read writability of ${writableFailure} failed with error -25200`) + } if (method === 'click' || method === 'setValue' || method === 'mouseMove') { const observed = params.snapshotId === undefined && params.elementId === undefined ? snapshots.get(params.windowId) ?? snapshots.get(null) diff --git a/snapshots/session/cua-ptc/consent-fixture.mjs b/snapshots/session/cua-ptc/consent-fixture.mjs index 2e958b6c3c..e93ba645f8 100644 --- a/snapshots/session/cua-ptc/consent-fixture.mjs +++ b/snapshots/session/cua-ptc/consent-fixture.mjs @@ -1,4 +1,4 @@ -/** Own the bridge module and private native substitute; the replay owns its AX-failure marker through finally. */ +/** Own the bridge module and private native substitute; the replay owns its AX-failure markers through finally. */ import { chmod, copyFile, unlink, writeFile } from 'node:fs/promises' import { join } from 'node:path' import { startFixtureHost } from '../cua-native/fixture-host.mjs' diff --git a/snapshots/session/cua-ptc/replay.override.json b/snapshots/session/cua-ptc/replay.override.json index 5098b66218..678896fb8d 100644 --- a/snapshots/session/cua-ptc/replay.override.json +++ b/snapshots/session/cua-ptc/replay.override.json @@ -65,6 +65,39 @@ } ] }, + { + "kind": "chunks", + "chunks": [ + { + "type": "block-start", + "index": 0, + "blockType": "tool-call" + }, + { + "type": "block-end", + "index": 0, + "block": { + "type": "tool-call", + "id": "cua-ptc-writability", + "name": "run_code", + "arguments": "{\"code\":\"const { writeFile, unlink } = await import('node:fs/promises');\\nconst { resolve } = await import('node:path');\\nconst { pathToFileURL } = await import('node:url');\\nconst { connectCua } = await import(pathToFileURL(resolve('cua-bridge.mjs')).href);\\nconst cua = connectCua(() => tools.cua);\\nconst app = await cua.getApp('test.fixture');\\nconst results = [];\\nfor (const attribute of ['AXValue', 'AXSelectedTextRange']) {\\n await writeFile('fixture-ax-writability-failure', attribute);\\n try {\\n const tree = await app.getAccessibilityTree({ emit: false });\\n const field = attribute === 'AXValue' ? 'valueWritable' : 'selectionWritable';\\n if (!tree.includes(field + '=unknown') || tree.includes(field + '=true')) throw new Error('Unknown writability was lost');\\n if (!tree.includes('Increment') || !tree.includes('Count:')) throw new Error('Other nodes were lost');\\n let writeRefused = false;\\n try {\\n if (attribute === 'AXValue') await app.setValue('1', 'must not be written');\\n else await app.selectText('1', 'Fixture');\\n } catch (error) {\\n if (!error.message.includes('writability of ' + attribute + ' failed with error -25200')) throw error;\\n writeRefused = true;\\n }\\n if (!writeRefused) throw new Error('Write accepted with unknown writability');\\n results.push({ attribute, unknown: true, otherNodesRetained: true, writeRefused });\\n } finally { await unlink('fixture-ax-writability-failure'); }\\n}\\nreturn results;\",\"description\":\"Retain AX trees with unknown writability and reject writes\"}" + } + }, + { + "type": "usage", + "usage": { + "inputTokens": 3, + "outputTokens": 3 + } + }, + { + "type": "finish", + "reason": { + "kind": "tool-calls" + } + } + ] + }, { "kind": "chunks", "chunks": [ @@ -78,7 +111,7 @@ "index": 0, "block": { "type": "text", - "text": "Received native CUA values and PNG screenshots through PTC. The object API captured a 90-byte PNG after two AX read failures and refused element input without an accessibility snapshot." + "text": "Received native CUA values and PNG screenshots through PTC. The object API captured a 90-byte PNG after two AX read failures and refused element input without an accessibility snapshot. Both writability probe failures left the tree readable with unknown fields and kept writes strict." } }, { diff --git a/snapshots/session/cua-ptc/session.v4.jsonl b/snapshots/session/cua-ptc/session.v4.jsonl index 734f20ae62..9e2e3d52d4 100644 --- a/snapshots/session/cua-ptc/session.v4.jsonl +++ b/snapshots/session/cua-ptc/session.v4.jsonl @@ -12,7 +12,7 @@ {"type":"request/header","data":{"header":{"config":{"provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"tools":"{{tools}}"},"reason":"initial"}} {"type":"request/context","data":{"provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"}} {"type":"session/title","data":{"title":"Use tools.cua from Node PTC","messageSeqs":[8],"source":{"kind":"fallback"}}} -{"type":"assistant/message","data":{"turn":1,"step":1,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-ptc-native-result","name":"run_code","arguments":"{\"code\":\"const { Buffer } = await import('node:buffer');\\nconst app = await tools.cua({ method: 'getApp', params: { selector: 'test.fixture' } });\\nconst state = await tools.cua({ method: 'getState', params: { appHandle: app.appHandle, windowId: 7 } });\\nconst shot = await tools.cua({ method: 'screenshot', params: { appHandle: app.appHandle, windowId: 7 } });\\nconst bytes = Buffer.from(shot.data, 'base64');\\nreturn { appHandle: app.appHandle, hasIncrement: state.tree.includes('Increment'), pngBytes: bytes.length, signature: bytes.subarray(0, 8).toString('hex'), imageWidth: shot.attachment.width, imageHeight: shot.attachment.height };\",\"description\":\"Read native CUA values and receive one screenshot\"}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:4}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1794212769549,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1794212769549,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-ptc-native-result","name":"run_code","arguments":"{\"code\":\"const { Buffer } = await import('node:buffer');\\nconst app = await tools.cua({ method: 'getApp', params: { selector: 'test.fixture' } });\\nconst state = await tools.cua({ method: 'getState', params: { appHandle: app.appHandle, windowId: 7 } });\\nconst shot = await tools.cua({ method: 'screenshot', params: { appHandle: app.appHandle, windowId: 7 } });\\nconst bytes = Buffer.from(shot.data, 'base64');\\nreturn { appHandle: app.appHandle, hasIncrement: state.tree.includes('Increment'), pngBytes: bytes.length, signature: bytes.subarray(0, 8).toString('hex'), imageWidth: shot.attachment.width, imageHeight: shot.attachment.height };\",\"description\":\"Read native CUA values and receive one screenshot\"}"}}},{"type":"chunk","time":1794212769549,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1794212769549,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} +{"type":"assistant/message","data":{"turn":1,"step":1,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-ptc-native-result","name":"run_code","arguments":"{\"code\":\"const { Buffer } = await import('node:buffer');\\nconst app = await tools.cua({ method: 'getApp', params: { selector: 'test.fixture' } });\\nconst state = await tools.cua({ method: 'getState', params: { appHandle: app.appHandle, windowId: 7 } });\\nconst shot = await tools.cua({ method: 'screenshot', params: { appHandle: app.appHandle, windowId: 7 } });\\nconst bytes = Buffer.from(shot.data, 'base64');\\nreturn { appHandle: app.appHandle, hasIncrement: state.tree.includes('Increment'), pngBytes: bytes.length, signature: bytes.subarray(0, 8).toString('hex'), imageWidth: shot.attachment.width, imageHeight: shot.attachment.height };\",\"description\":\"Read native CUA values and receive one screenshot\"}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:4}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1794213719496,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1794213719496,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-ptc-native-result","name":"run_code","arguments":"{\"code\":\"const { Buffer } = await import('node:buffer');\\nconst app = await tools.cua({ method: 'getApp', params: { selector: 'test.fixture' } });\\nconst state = await tools.cua({ method: 'getState', params: { appHandle: app.appHandle, windowId: 7 } });\\nconst shot = await tools.cua({ method: 'screenshot', params: { appHandle: app.appHandle, windowId: 7 } });\\nconst bytes = Buffer.from(shot.data, 'base64');\\nreturn { appHandle: app.appHandle, hasIncrement: state.tree.includes('Increment'), pngBytes: bytes.length, signature: bytes.subarray(0, 8).toString('hex'), imageWidth: shot.attachment.width, imageHeight: shot.attachment.height };\",\"description\":\"Read native CUA values and receive one screenshot\"}"}}},{"type":"chunk","time":1794213719496,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1794213719496,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} {"type":"tool/call","data":{"turn":1,"step":1,"callId":"cua-ptc-native-result","name":"run_code","arguments":"{\"code\":\"const { Buffer } = await import('node:buffer');\\nconst app = await tools.cua({ method: 'getApp', params: { selector: 'test.fixture' } });\\nconst state = await tools.cua({ method: 'getState', params: { appHandle: app.appHandle, windowId: 7 } });\\nconst shot = await tools.cua({ method: 'screenshot', params: { appHandle: app.appHandle, windowId: 7 } });\\nconst bytes = Buffer.from(shot.data, 'base64');\\nreturn { appHandle: app.appHandle, hasIncrement: state.tree.includes('Increment'), pngBytes: bytes.length, signature: bytes.subarray(0, 8).toString('hex'), imageWidth: shot.attachment.width, imageHeight: shot.attachment.height };\",\"description\":\"Read native CUA values and receive one screenshot\"}"}} {"type":"tool/ptc-dispatch-start","data":{"rootCallId":"cua-ptc-native-result","parentCallId":"cua-ptc-native-result","subCallId":"cua-ptc-native-result:ptc:1","name":"cua","arguments":{"method":"getApp","params":{"selector":"test.fixture"}}}} {"type":"tool/ptc-dispatch","data":{"rootCallId":"cua-ptc-native-result","parentCallId":"cua-ptc-native-result","subCallId":"cua-ptc-native-result:ptc:1","name":"cua","arguments":{"method":"getApp","params":{"selector":"test.fixture"}},"isError":false,"content":[{"type":"text","text":"{\"appHandle\":\"fixture-app\",\"name\":\"Fixture\",\"bundleId\":\"test.fixture\",\"running\":true}"}]}} @@ -26,7 +26,7 @@ {"type":"agent/inbox/spliced","data":{"target":"next-step","start":0,"removedCount":1,"inserted":[]}} {"type":"step/start","data":{"turn":1,"step":2}} {"type":"user/message","data":{"content":[{"type":"text","text":"{\"pixelWidth\":1,\"pixelHeight\":1,\"windowId\":7,\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"coordinateSpace\":\"screenshot-pixels\"}"},{"type":"image","attachment":{"attachmentId":"sha256:999f1d1527ee7e79266f16add5430fff76b1225d742464a5b1ff1f02971bb8ee","mediaType":"image/png","width":1,"height":1,"bytes":90}}],"source":{"kind":"ptc-mode"},"role":"user","id":"{{message:6}}"},"surfaceOp":"append"} -{"type":"assistant/message","data":{"turn":1,"step":2,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-ptc-ax-failure","name":"run_code","arguments":"{\"code\":\"const { writeFile, unlink } = await import('node:fs/promises');\\nconst { resolve } = await import('node:path');\\nconst { pathToFileURL } = await import('node:url');\\nconst { connectCua } = await import(pathToFileURL(resolve('cua-bridge.mjs')).href);\\nawait writeFile('fixture-ax-failure', '');\\ntry {\\n const cua = connectCua(() => tools.cua);\\n const app = await cua.getApp('test.fixture');\\n const windows = await app.getWindows({ emit: false });\\n const window = await app.getWindow(windows[0].windowId);\\n const failures = [];\\n for (const surface of [app, window]) {\\n try { await surface.getAccessibilityTree(); throw new Error('Expected AX failure'); }\\n catch (error) { if (!error.message.includes('Fixture accessibility read failed')) throw error; failures.push(error.message); }\\n }\\n let coordinateRefused = false;\\n try { await window.click([100, 100], { relative: true }); }\\n catch (error) { if (!error.message.includes('Window has no observation')) throw error; coordinateRefused = true; }\\n if (!coordinateRefused) throw new Error('Coordinate input accepted without observation');\\n const shot = await window.getScreenshot();\\n const coordinateDispatched = (await window.click([100, 100], { relative: true })).dispatched;\\n let elementRefused = false;\\n try { await window.invoke('0'); }\\n catch (error) { if (!error.message.includes('Read getAccessibilityTree')) throw error; elementRefused = true; }\\n if (!elementRefused) throw new Error('Element input accepted without an AX snapshot');\\n return { axFailures: failures.length, screenshotWindow: shot.windowId, pngBytes: shot.bytes.length, elementRefused, coordinateRefused, coordinateDispatched };\\n} finally { await unlink('fixture-ax-failure'); }\",\"description\":\"Capture through the object API despite application and window AX failures\"}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:7}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1794212770340,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1794212770340,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-ptc-ax-failure","name":"run_code","arguments":"{\"code\":\"const { writeFile, unlink } = await import('node:fs/promises');\\nconst { resolve } = await import('node:path');\\nconst { pathToFileURL } = await import('node:url');\\nconst { connectCua } = await import(pathToFileURL(resolve('cua-bridge.mjs')).href);\\nawait writeFile('fixture-ax-failure', '');\\ntry {\\n const cua = connectCua(() => tools.cua);\\n const app = await cua.getApp('test.fixture');\\n const windows = await app.getWindows({ emit: false });\\n const window = await app.getWindow(windows[0].windowId);\\n const failures = [];\\n for (const surface of [app, window]) {\\n try { await surface.getAccessibilityTree(); throw new Error('Expected AX failure'); }\\n catch (error) { if (!error.message.includes('Fixture accessibility read failed')) throw error; failures.push(error.message); }\\n }\\n let coordinateRefused = false;\\n try { await window.click([100, 100], { relative: true }); }\\n catch (error) { if (!error.message.includes('Window has no observation')) throw error; coordinateRefused = true; }\\n if (!coordinateRefused) throw new Error('Coordinate input accepted without observation');\\n const shot = await window.getScreenshot();\\n const coordinateDispatched = (await window.click([100, 100], { relative: true })).dispatched;\\n let elementRefused = false;\\n try { await window.invoke('0'); }\\n catch (error) { if (!error.message.includes('Read getAccessibilityTree')) throw error; elementRefused = true; }\\n if (!elementRefused) throw new Error('Element input accepted without an AX snapshot');\\n return { axFailures: failures.length, screenshotWindow: shot.windowId, pngBytes: shot.bytes.length, elementRefused, coordinateRefused, coordinateDispatched };\\n} finally { await unlink('fixture-ax-failure'); }\",\"description\":\"Capture through the object API despite application and window AX failures\"}"}}},{"type":"chunk","time":1794212770340,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1794212770340,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} +{"type":"assistant/message","data":{"turn":1,"step":2,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-ptc-ax-failure","name":"run_code","arguments":"{\"code\":\"const { writeFile, unlink } = await import('node:fs/promises');\\nconst { resolve } = await import('node:path');\\nconst { pathToFileURL } = await import('node:url');\\nconst { connectCua } = await import(pathToFileURL(resolve('cua-bridge.mjs')).href);\\nawait writeFile('fixture-ax-failure', '');\\ntry {\\n const cua = connectCua(() => tools.cua);\\n const app = await cua.getApp('test.fixture');\\n const windows = await app.getWindows({ emit: false });\\n const window = await app.getWindow(windows[0].windowId);\\n const failures = [];\\n for (const surface of [app, window]) {\\n try { await surface.getAccessibilityTree(); throw new Error('Expected AX failure'); }\\n catch (error) { if (!error.message.includes('Fixture accessibility read failed')) throw error; failures.push(error.message); }\\n }\\n let coordinateRefused = false;\\n try { await window.click([100, 100], { relative: true }); }\\n catch (error) { if (!error.message.includes('Window has no observation')) throw error; coordinateRefused = true; }\\n if (!coordinateRefused) throw new Error('Coordinate input accepted without observation');\\n const shot = await window.getScreenshot();\\n const coordinateDispatched = (await window.click([100, 100], { relative: true })).dispatched;\\n let elementRefused = false;\\n try { await window.invoke('0'); }\\n catch (error) { if (!error.message.includes('Read getAccessibilityTree')) throw error; elementRefused = true; }\\n if (!elementRefused) throw new Error('Element input accepted without an AX snapshot');\\n return { axFailures: failures.length, screenshotWindow: shot.windowId, pngBytes: shot.bytes.length, elementRefused, coordinateRefused, coordinateDispatched };\\n} finally { await unlink('fixture-ax-failure'); }\",\"description\":\"Capture through the object API despite application and window AX failures\"}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:7}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1794213720632,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1794213720632,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-ptc-ax-failure","name":"run_code","arguments":"{\"code\":\"const { writeFile, unlink } = await import('node:fs/promises');\\nconst { resolve } = await import('node:path');\\nconst { pathToFileURL } = await import('node:url');\\nconst { connectCua } = await import(pathToFileURL(resolve('cua-bridge.mjs')).href);\\nawait writeFile('fixture-ax-failure', '');\\ntry {\\n const cua = connectCua(() => tools.cua);\\n const app = await cua.getApp('test.fixture');\\n const windows = await app.getWindows({ emit: false });\\n const window = await app.getWindow(windows[0].windowId);\\n const failures = [];\\n for (const surface of [app, window]) {\\n try { await surface.getAccessibilityTree(); throw new Error('Expected AX failure'); }\\n catch (error) { if (!error.message.includes('Fixture accessibility read failed')) throw error; failures.push(error.message); }\\n }\\n let coordinateRefused = false;\\n try { await window.click([100, 100], { relative: true }); }\\n catch (error) { if (!error.message.includes('Window has no observation')) throw error; coordinateRefused = true; }\\n if (!coordinateRefused) throw new Error('Coordinate input accepted without observation');\\n const shot = await window.getScreenshot();\\n const coordinateDispatched = (await window.click([100, 100], { relative: true })).dispatched;\\n let elementRefused = false;\\n try { await window.invoke('0'); }\\n catch (error) { if (!error.message.includes('Read getAccessibilityTree')) throw error; elementRefused = true; }\\n if (!elementRefused) throw new Error('Element input accepted without an AX snapshot');\\n return { axFailures: failures.length, screenshotWindow: shot.windowId, pngBytes: shot.bytes.length, elementRefused, coordinateRefused, coordinateDispatched };\\n} finally { await unlink('fixture-ax-failure'); }\",\"description\":\"Capture through the object API despite application and window AX failures\"}"}}},{"type":"chunk","time":1794213720632,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1794213720632,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} {"type":"tool/call","data":{"turn":1,"step":2,"callId":"cua-ptc-ax-failure","name":"run_code","arguments":"{\"code\":\"const { writeFile, unlink } = await import('node:fs/promises');\\nconst { resolve } = await import('node:path');\\nconst { pathToFileURL } = await import('node:url');\\nconst { connectCua } = await import(pathToFileURL(resolve('cua-bridge.mjs')).href);\\nawait writeFile('fixture-ax-failure', '');\\ntry {\\n const cua = connectCua(() => tools.cua);\\n const app = await cua.getApp('test.fixture');\\n const windows = await app.getWindows({ emit: false });\\n const window = await app.getWindow(windows[0].windowId);\\n const failures = [];\\n for (const surface of [app, window]) {\\n try { await surface.getAccessibilityTree(); throw new Error('Expected AX failure'); }\\n catch (error) { if (!error.message.includes('Fixture accessibility read failed')) throw error; failures.push(error.message); }\\n }\\n let coordinateRefused = false;\\n try { await window.click([100, 100], { relative: true }); }\\n catch (error) { if (!error.message.includes('Window has no observation')) throw error; coordinateRefused = true; }\\n if (!coordinateRefused) throw new Error('Coordinate input accepted without observation');\\n const shot = await window.getScreenshot();\\n const coordinateDispatched = (await window.click([100, 100], { relative: true })).dispatched;\\n let elementRefused = false;\\n try { await window.invoke('0'); }\\n catch (error) { if (!error.message.includes('Read getAccessibilityTree')) throw error; elementRefused = true; }\\n if (!elementRefused) throw new Error('Element input accepted without an AX snapshot');\\n return { axFailures: failures.length, screenshotWindow: shot.windowId, pngBytes: shot.bytes.length, elementRefused, coordinateRefused, coordinateDispatched };\\n} finally { await unlink('fixture-ax-failure'); }\",\"description\":\"Capture through the object API despite application and window AX failures\"}"}} {"type":"tool/ptc-dispatch-start","data":{"rootCallId":"cua-ptc-ax-failure","parentCallId":"cua-ptc-ax-failure","subCallId":"cua-ptc-ax-failure:ptc:1","name":"cua","arguments":{"method":"getCapabilities","params":{}}}} {"type":"tool/ptc-dispatch","data":{"rootCallId":"cua-ptc-ax-failure","parentCallId":"cua-ptc-ax-failure","subCallId":"cua-ptc-ax-failure:ptc:1","name":"cua","arguments":{"method":"getCapabilities","params":{}},"isError":false,"content":[{"type":"text","text":"{\"sessionLocked\":false,\"permissions\":{\"accessibility\":true,\"screenRecording\":true},\"platform\":\"macos\",\"systemPointer\":true,\"methods\":[\"activate\",\"click\",\"drag\",\"findApps\",\"getApp\",\"getCapabilities\",\"getState\",\"invoke\",\"listActions\",\"listApps\",\"listWindows\",\"mouseMove\",\"paste\",\"performAction\",\"pressKey\",\"screenshot\",\"scroll\",\"selectText\",\"sendKeyEvent\",\"setValue\",\"show\",\"typeText\"],\"appLevelAccessibility\":true,\"keyboard\":[\"app\"],\"browser\":false,\"coordinateSpace\":\"screenshot-pixels\"}"}]}} @@ -50,6 +50,23 @@ {"type":"agent/inbox/spliced","data":{"target":"next-step","start":0,"removedCount":1,"inserted":[]}} {"type":"step/start","data":{"turn":1,"step":3}} {"type":"user/message","data":{"content":[{"type":"text","text":"{\"pixelWidth\":1,\"pixelHeight\":1,\"windowId\":7,\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"coordinateSpace\":\"screenshot-pixels\"}"},{"type":"image","attachment":{"attachmentId":"sha256:999f1d1527ee7e79266f16add5430fff76b1225d742464a5b1ff1f02971bb8ee","mediaType":"image/png","width":1,"height":1,"bytes":90}}],"source":{"kind":"ptc-mode"},"role":"user","id":"{{message:9}}"},"surfaceOp":"append"} -{"type":"assistant/message","data":{"turn":1,"step":3,"message":{"role":"assistant","content":[{"type":"text","text":"Received native CUA values and PNG screenshots through PTC. The object API captured a 90-byte PNG after two AX read failures and refused element input without an accessibility snapshot."}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:10}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1794212770534,"chunk":{"type":"block-start","index":0,"blockType":"text"}},{"type":"chunk","time":1794212770534,"chunk":{"type":"block-end","index":0,"block":{"type":"text","text":"Received native CUA values and PNG screenshots through PTC. The object API captured a 90-byte PNG after two AX read failures and refused element input without an accessibility snapshot."}}},{"type":"chunk","time":1794212770534,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1794212770534,"chunk":{"type":"finish","reason":{"kind":"stop"}}}]},"surfaceOp":"append"} +{"type":"assistant/message","data":{"turn":1,"step":3,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-ptc-writability","name":"run_code","arguments":"{\"code\":\"const { writeFile, unlink } = await import('node:fs/promises');\\nconst { resolve } = await import('node:path');\\nconst { pathToFileURL } = await import('node:url');\\nconst { connectCua } = await import(pathToFileURL(resolve('cua-bridge.mjs')).href);\\nconst cua = connectCua(() => tools.cua);\\nconst app = await cua.getApp('test.fixture');\\nconst results = [];\\nfor (const attribute of ['AXValue', 'AXSelectedTextRange']) {\\n await writeFile('fixture-ax-writability-failure', attribute);\\n try {\\n const tree = await app.getAccessibilityTree({ emit: false });\\n const field = attribute === 'AXValue' ? 'valueWritable' : 'selectionWritable';\\n if (!tree.includes(field + '=unknown') || tree.includes(field + '=true')) throw new Error('Unknown writability was lost');\\n if (!tree.includes('Increment') || !tree.includes('Count:')) throw new Error('Other nodes were lost');\\n let writeRefused = false;\\n try {\\n if (attribute === 'AXValue') await app.setValue('1', 'must not be written');\\n else await app.selectText('1', 'Fixture');\\n } catch (error) {\\n if (!error.message.includes('writability of ' + attribute + ' failed with error -25200')) throw error;\\n writeRefused = true;\\n }\\n if (!writeRefused) throw new Error('Write accepted with unknown writability');\\n results.push({ attribute, unknown: true, otherNodesRetained: true, writeRefused });\\n } finally { await unlink('fixture-ax-writability-failure'); }\\n}\\nreturn results;\",\"description\":\"Retain AX trees with unknown writability and reject writes\"}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:10}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1794213720832,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1794213720832,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-ptc-writability","name":"run_code","arguments":"{\"code\":\"const { writeFile, unlink } = await import('node:fs/promises');\\nconst { resolve } = await import('node:path');\\nconst { pathToFileURL } = await import('node:url');\\nconst { connectCua } = await import(pathToFileURL(resolve('cua-bridge.mjs')).href);\\nconst cua = connectCua(() => tools.cua);\\nconst app = await cua.getApp('test.fixture');\\nconst results = [];\\nfor (const attribute of ['AXValue', 'AXSelectedTextRange']) {\\n await writeFile('fixture-ax-writability-failure', attribute);\\n try {\\n const tree = await app.getAccessibilityTree({ emit: false });\\n const field = attribute === 'AXValue' ? 'valueWritable' : 'selectionWritable';\\n if (!tree.includes(field + '=unknown') || tree.includes(field + '=true')) throw new Error('Unknown writability was lost');\\n if (!tree.includes('Increment') || !tree.includes('Count:')) throw new Error('Other nodes were lost');\\n let writeRefused = false;\\n try {\\n if (attribute === 'AXValue') await app.setValue('1', 'must not be written');\\n else await app.selectText('1', 'Fixture');\\n } catch (error) {\\n if (!error.message.includes('writability of ' + attribute + ' failed with error -25200')) throw error;\\n writeRefused = true;\\n }\\n if (!writeRefused) throw new Error('Write accepted with unknown writability');\\n results.push({ attribute, unknown: true, otherNodesRetained: true, writeRefused });\\n } finally { await unlink('fixture-ax-writability-failure'); }\\n}\\nreturn results;\",\"description\":\"Retain AX trees with unknown writability and reject writes\"}"}}},{"type":"chunk","time":1794213720832,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1794213720832,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} +{"type":"tool/call","data":{"turn":1,"step":3,"callId":"cua-ptc-writability","name":"run_code","arguments":"{\"code\":\"const { writeFile, unlink } = await import('node:fs/promises');\\nconst { resolve } = await import('node:path');\\nconst { pathToFileURL } = await import('node:url');\\nconst { connectCua } = await import(pathToFileURL(resolve('cua-bridge.mjs')).href);\\nconst cua = connectCua(() => tools.cua);\\nconst app = await cua.getApp('test.fixture');\\nconst results = [];\\nfor (const attribute of ['AXValue', 'AXSelectedTextRange']) {\\n await writeFile('fixture-ax-writability-failure', attribute);\\n try {\\n const tree = await app.getAccessibilityTree({ emit: false });\\n const field = attribute === 'AXValue' ? 'valueWritable' : 'selectionWritable';\\n if (!tree.includes(field + '=unknown') || tree.includes(field + '=true')) throw new Error('Unknown writability was lost');\\n if (!tree.includes('Increment') || !tree.includes('Count:')) throw new Error('Other nodes were lost');\\n let writeRefused = false;\\n try {\\n if (attribute === 'AXValue') await app.setValue('1', 'must not be written');\\n else await app.selectText('1', 'Fixture');\\n } catch (error) {\\n if (!error.message.includes('writability of ' + attribute + ' failed with error -25200')) throw error;\\n writeRefused = true;\\n }\\n if (!writeRefused) throw new Error('Write accepted with unknown writability');\\n results.push({ attribute, unknown: true, otherNodesRetained: true, writeRefused });\\n } finally { await unlink('fixture-ax-writability-failure'); }\\n}\\nreturn results;\",\"description\":\"Retain AX trees with unknown writability and reject writes\"}"}} +{"type":"tool/ptc-dispatch-start","data":{"rootCallId":"cua-ptc-writability","parentCallId":"cua-ptc-writability","subCallId":"cua-ptc-writability:ptc:1","name":"cua","arguments":{"method":"getCapabilities","params":{}}}} +{"type":"tool/ptc-dispatch","data":{"rootCallId":"cua-ptc-writability","parentCallId":"cua-ptc-writability","subCallId":"cua-ptc-writability:ptc:1","name":"cua","arguments":{"method":"getCapabilities","params":{}},"isError":false,"content":[{"type":"text","text":"{\"sessionLocked\":false,\"permissions\":{\"accessibility\":true,\"screenRecording\":true},\"platform\":\"macos\",\"systemPointer\":true,\"methods\":[\"activate\",\"click\",\"drag\",\"findApps\",\"getApp\",\"getCapabilities\",\"getState\",\"invoke\",\"listActions\",\"listApps\",\"listWindows\",\"mouseMove\",\"paste\",\"performAction\",\"pressKey\",\"screenshot\",\"scroll\",\"selectText\",\"sendKeyEvent\",\"setValue\",\"show\",\"typeText\"],\"appLevelAccessibility\":true,\"keyboard\":[\"app\"],\"browser\":false,\"coordinateSpace\":\"screenshot-pixels\"}"}]}} +{"type":"tool/ptc-dispatch-start","data":{"rootCallId":"cua-ptc-writability","parentCallId":"cua-ptc-writability","subCallId":"cua-ptc-writability:ptc:2","name":"cua","arguments":{"method":"getApp","params":{"selector":"test.fixture"}}}} +{"type":"tool/ptc-dispatch","data":{"rootCallId":"cua-ptc-writability","parentCallId":"cua-ptc-writability","subCallId":"cua-ptc-writability:ptc:2","name":"cua","arguments":{"method":"getApp","params":{"selector":"test.fixture"}},"isError":false,"content":[{"type":"text","text":"{\"appHandle\":\"fixture-app\",\"name\":\"Fixture\",\"bundleId\":\"test.fixture\",\"running\":true}"}]}} +{"type":"tool/ptc-dispatch-start","data":{"rootCallId":"cua-ptc-writability","parentCallId":"cua-ptc-writability","subCallId":"cua-ptc-writability:ptc:3","name":"cua","arguments":{"method":"getState","params":{"appHandle":"fixture-app"}}}} +{"type":"tool/ptc-dispatch","data":{"rootCallId":"cua-ptc-writability","parentCallId":"cua-ptc-writability","subCallId":"cua-ptc-writability:ptc:3","name":"cua","arguments":{"method":"getState","params":{"appHandle":"fixture-app"}},"isError":false,"content":[{"type":"text","text":"{\"appHandle\":\"fixture-app\",\"windowId\":null,\"snapshotId\":\"fixture-snapshot-2\",\"tree\":\"AX scope=application offsets=UTF-16\\n- [0] AXButton \\\"Increment\\\"\\n- [1] AXTextField valueWritable=unknown selectionWritable=true value=\\\"\\\" placeholder=\\\"Fixture text\\\"\\n- AXStaticText \\\"Count: 1\\\"\\n- [2] AXMenuBarItem \\\"File\\\"\\nMenu contents omitted; request includeMenus=true to expand.\",\"windows\":[{\"windowId\":7,\"appName\":\"Fixture\",\"title\":\"CUA fixture\",\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"isOnScreen\":true,\"onCurrentSpace\":true}],\"truncated\":false}"}]}} +{"type":"tool/ptc-dispatch-start","data":{"rootCallId":"cua-ptc-writability","parentCallId":"cua-ptc-writability","subCallId":"cua-ptc-writability:ptc:4","name":"cua","arguments":{"method":"setValue","params":{"appHandle":"fixture-app","snapshotId":"fixture-snapshot-2","elementId":"1","value":"must not be written"}}}} +{"type":"tool/ptc-dispatch","data":{"rootCallId":"cua-ptc-writability","parentCallId":"cua-ptc-writability","subCallId":"cua-ptc-writability:ptc:4","name":"cua","arguments":{"method":"setValue","params":{"appHandle":"fixture-app","snapshotId":"fixture-snapshot-2","elementId":"1","value":"must not be written"}},"isError":true,"content":[{"type":"text","text":"Error: NATIVE_ERROR: Accessibility read writability of AXValue failed with error -25200"}]}} +{"type":"tool/ptc-dispatch-start","data":{"rootCallId":"cua-ptc-writability","parentCallId":"cua-ptc-writability","subCallId":"cua-ptc-writability:ptc:5","name":"cua","arguments":{"method":"getState","params":{"appHandle":"fixture-app"}}}} +{"type":"tool/ptc-dispatch","data":{"rootCallId":"cua-ptc-writability","parentCallId":"cua-ptc-writability","subCallId":"cua-ptc-writability:ptc:5","name":"cua","arguments":{"method":"getState","params":{"appHandle":"fixture-app"}},"isError":false,"content":[{"type":"text","text":"{\"appHandle\":\"fixture-app\",\"windowId\":null,\"snapshotId\":\"fixture-snapshot-3\",\"tree\":\"AX scope=application offsets=UTF-16\\n- [0] AXButton \\\"Increment\\\"\\n- [1] AXTextField valueWritable=true selectionWritable=unknown value=\\\"\\\" placeholder=\\\"Fixture text\\\"\\n- AXStaticText \\\"Count: 1\\\"\\n- [2] AXMenuBarItem \\\"File\\\"\\nMenu contents omitted; request includeMenus=true to expand.\",\"windows\":[{\"windowId\":7,\"appName\":\"Fixture\",\"title\":\"CUA fixture\",\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"isOnScreen\":true,\"onCurrentSpace\":true}],\"truncated\":false}"}]}} +{"type":"tool/ptc-dispatch-start","data":{"rootCallId":"cua-ptc-writability","parentCallId":"cua-ptc-writability","subCallId":"cua-ptc-writability:ptc:6","name":"cua","arguments":{"method":"selectText","params":{"appHandle":"fixture-app","snapshotId":"fixture-snapshot-3","elementId":"1","text":"Fixture"}}}} +{"type":"tool/ptc-dispatch","data":{"rootCallId":"cua-ptc-writability","parentCallId":"cua-ptc-writability","subCallId":"cua-ptc-writability:ptc:6","name":"cua","arguments":{"method":"selectText","params":{"appHandle":"fixture-app","snapshotId":"fixture-snapshot-3","elementId":"1","text":"Fixture"}},"isError":true,"content":[{"type":"text","text":"Error: NATIVE_ERROR: Accessibility read writability of AXSelectedTextRange failed with error -25200"}]}} +{"type":"tool/result","data":{"turn":1,"step":3,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-ptc-writability"},"toolCallId":"cua-ptc-writability","content":[{"type":"text","text":"[\n {\n \"attribute\": \"AXValue\",\n \"unknown\": true,\n \"otherNodesRetained\": true,\n \"writeRefused\": true\n },\n {\n \"attribute\": \"AXSelectedTextRange\",\n \"unknown\": true,\n \"otherNodesRetained\": true,\n \"writeRefused\": true\n }\n]"}],"isError":false,"id":"{{message:11}}"}},"sourceEventSeqs":[52],"surfaceOp":"append"} {"type":"step/end","data":{"turn":1,"step":3}} +{"type":"step/start","data":{"turn":1,"step":4}} +{"type":"assistant/message","data":{"turn":1,"step":4,"message":{"role":"assistant","content":[{"type":"text","text":"Received native CUA values and PNG screenshots through PTC. The object API captured a 90-byte PNG after two AX read failures and refused element input without an accessibility snapshot. Both writability probe failures left the tree readable with unknown fields and kept writes strict."}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:12}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1794213721002,"chunk":{"type":"block-start","index":0,"blockType":"text"}},{"type":"chunk","time":1794213721002,"chunk":{"type":"block-end","index":0,"block":{"type":"text","text":"Received native CUA values and PNG screenshots through PTC. The object API captured a 90-byte PNG after two AX read failures and refused element input without an accessibility snapshot. Both writability probe failures left the tree readable with unknown fields and kept writes strict."}}},{"type":"chunk","time":1794213721002,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1794213721002,"chunk":{"type":"finish","reason":{"kind":"stop"}}}]},"surfaceOp":"append"} +{"type":"step/end","data":{"turn":1,"step":4}} {"type":"turn/end","data":{"turn":1,"reason":{"kind":"completed"}}} From d1711894c844128bbced3d3482ccdee761f71a0d Mon Sep 17 00:00:00 2001 From: Yichen Jiang Date: Mon, 9 Nov 2026 16:54:57 +0800 Subject: [PATCH 6/9] fix(llm): classify pi-ai failures by the leading HTTP status pi-ai's SDK-backed providers render an HTTP failure as ` `. A gateway 500 that embeds its upstream's 401 in the body matched the AUTH rule before the 5xx rule, so Chat reported an invalid API key and the retry policy gave up. When a message leads with a 4xx/5xx status, the status rules now read only that status; other wordings still scan the whole message. Refs #8182 --- packages/llm/llm-pi-ai/src/stream.ts | 19 ++++++++++++++----- packages/llm/llm-pi-ai/tests/convert.spec.ts | 12 ++++++++++++ 2 files changed, 26 insertions(+), 5 deletions(-) diff --git a/packages/llm/llm-pi-ai/src/stream.ts b/packages/llm/llm-pi-ai/src/stream.ts index 58bd2e8e3e..59bfdce550 100644 --- a/packages/llm/llm-pi-ai/src/stream.ts +++ b/packages/llm/llm-pi-ai/src/stream.ts @@ -39,15 +39,24 @@ export function mapUsage(usage: PiUsage): TokenUsage { // wrapper a bare `terminated`, so we are left pattern-matching terse words here. // If pi-ai ever forwards the original Error (or a fetch/dispatcher hook that lets // us capture the cause ourselves), classify on `code`/`cause` instead of text. +// +// pi-ai's SDK-backed providers render an HTTP failure as ` ` +// (Anthropic and OpenAI SDK `APIError`, pi-messages ` : …`). +// The body can quote another hop's status, such as a gateway 500 that embeds its +// upstream's 401, so when a message leads with a status the status rules read +// only that status; other wordings scan the whole message. +const LEADING_HTTP_STATUS = /^[45]\d\d\b/ + function classifyPiAiError(message: string): BuiltinLlmFailureCode { - if (/\b(?:401|403)\b/.test(message)) return 'AUTH' + const statusSource = LEADING_HTTP_STATUS.exec(message)?.[0] ?? message + if (/\b(?:401|403)\b/.test(statusSource)) return 'AUTH' if (isQuotaExceededError(message)) return QUOTA_EXCEEDED_CODE - if (/\b429\b|rate.?limit/i.test(message)) return 'RATE_LIMIT' + if (/\b429\b/.test(statusSource) || /rate.?limit/i.test(message)) return 'RATE_LIMIT' // A rejected request body (gateway or provider size cap): resending the // same request cannot succeed, so it is invalid, not transient. - if (/\b413\b|failed to buffer the request body:\s*length limit exceeded|payload too large|request body too large/i.test(message)) return 'INVALID_REQUEST' - if (/\b400\b|invalid.?request/i.test(message)) return 'INVALID_REQUEST' - if (/\b5\d\d\b/.test(message)) return 'SERVER' + if (/\b413\b/.test(statusSource) || /failed to buffer the request body:\s*length limit exceeded|payload too large|request body too large/i.test(message)) return 'INVALID_REQUEST' + if (/\b400\b/.test(statusSource) || /invalid.?request/i.test(message)) return 'INVALID_REQUEST' + if (/\b5\d\d\b/.test(statusSource)) return 'SERVER' if (/\btime(?:d)?\s*out\b|timeout/i.test(message)) return 'TIMEOUT' // A stream truncated before the provider's terminal event: each pi-ai provider // throws its own wording when the wire closes mid-response without a terminal diff --git a/packages/llm/llm-pi-ai/tests/convert.spec.ts b/packages/llm/llm-pi-ai/tests/convert.spec.ts index fe6d72a2b4..ebdf64fdb0 100644 --- a/packages/llm/llm-pi-ai/tests/convert.spec.ts +++ b/packages/llm/llm-pi-ai/tests/convert.spec.ts @@ -939,6 +939,18 @@ describe('mapStopReason / mapUsage', () => { }))).toMatchObject({ kind: 'error', failure: { code: 'PI_AI_ERROR' } }) }) + it.each([ + // A gateway 500 that relays its upstream's 401 in the body, as the Anthropic SDK renders it. + ['500 {"error":{"origin":"upstream","message":"upstream returned an error response: User not found.","type":"upstream_error","code":"upstream_response_error","upstream":{"provider":"custom","status":401,"body":{"error":{"type":"","message":"User not found."},"type":"error"}}}}', 'SERVER'], + ['502 {"error":{"message":"upstream status 400"}}', 'SERVER'], + ['404 {"error":{"message":"upstream status 403"}}', 'PI_AI_ERROR'], + ['401 {"type":"error","error":{"type":"authentication_error","message":"invalid x-api-key"}}', 'AUTH'], + ['429 Too Many Requests: slow down', 'RATE_LIMIT'], + ] as const)('classifies %s by its leading response status', (errorMessage, code) => { + expect(mapStopReason(assistant({ stopReason: 'error', errorMessage }))) + .toMatchObject({ kind: 'error', failure: { code } }) + }) + it.each([ 'other side closed', 'HTTP2 request did not get a response', From c1001a4b3cd2271df0dd84eb9053af42b5f2b0c9 Mon Sep 17 00:00:00 2001 From: Dudu-0223 Date: Mon, 9 Nov 2026 17:10:45 +0800 Subject: [PATCH 7/9] test(plugin-manager): observe Git timeout exit status --- .../boot/plugin-manager/tests/github-connection.spec.ts | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/packages/boot/plugin-manager/tests/github-connection.spec.ts b/packages/boot/plugin-manager/tests/github-connection.spec.ts index 1d76e1eaf1..008830210f 100644 --- a/packages/boot/plugin-manager/tests/github-connection.spec.ts +++ b/packages/boot/plugin-manager/tests/github-connection.spec.ts @@ -118,8 +118,12 @@ it('ends a silent proxy connection and its Git child at the connection deadline' await transport.connected const result = await running expect(result?.kind).toBe('timeout') - // The result records the status the terminated Git is read as; only the streamed run reports no exit code. - expect(result).toMatchObject({ exitCode: 1, timedOut: true, argv: checkArgv('https://github.com/acme/plugin.git') }) + // Git's termination status varies by platform; preserve the observed status, or the fallback when absent. + const childResult = await vi.mocked(subprocess.execa).mock.results.at(-1)!.value + expect(childResult.timedOut).toBe(true) + expect(result).toMatchObject({ + exitCode: childResult.exitCode ?? 1, timedOut: true, argv: checkArgv('https://github.com/acme/plugin.git'), + }) expect(result!.output).toContain('connection to github.com timed out after 3000ms') await transport.disconnected expect(transport.sockets.size).toBe(0) From c8191be1e865ca1877e8f384450aa161c7394a1f Mon Sep 17 00:00:00 2001 From: Dudu-0223 Date: Mon, 9 Nov 2026 17:12:47 +0800 Subject: [PATCH 8/9] Revert "test(plugin-manager): observe Git timeout exit status" This reverts commit 3c904f6b3a78f92ae1f3b3eff42d354ffb58a952. --- .../boot/plugin-manager/tests/github-connection.spec.ts | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/packages/boot/plugin-manager/tests/github-connection.spec.ts b/packages/boot/plugin-manager/tests/github-connection.spec.ts index 008830210f..1d76e1eaf1 100644 --- a/packages/boot/plugin-manager/tests/github-connection.spec.ts +++ b/packages/boot/plugin-manager/tests/github-connection.spec.ts @@ -118,12 +118,8 @@ it('ends a silent proxy connection and its Git child at the connection deadline' await transport.connected const result = await running expect(result?.kind).toBe('timeout') - // Git's termination status varies by platform; preserve the observed status, or the fallback when absent. - const childResult = await vi.mocked(subprocess.execa).mock.results.at(-1)!.value - expect(childResult.timedOut).toBe(true) - expect(result).toMatchObject({ - exitCode: childResult.exitCode ?? 1, timedOut: true, argv: checkArgv('https://github.com/acme/plugin.git'), - }) + // The result records the status the terminated Git is read as; only the streamed run reports no exit code. + expect(result).toMatchObject({ exitCode: 1, timedOut: true, argv: checkArgv('https://github.com/acme/plugin.git') }) expect(result!.output).toContain('connection to github.com timed out after 3000ms') await transport.disconnected expect(transport.sockets.size).toBe(0) From 6a6195ac2899770cbd8b2f8ee7514ddbb6d62d58 Mon Sep 17 00:00:00 2001 From: Yichen Jiang Date: Mon, 9 Nov 2026 17:28:49 +0800 Subject: [PATCH 9/9] test(llm): pin pi-ai leading-status classification through the Anthropic route Add a composition test that sends a gateway 500 relaying an upstream 401 through the real Anthropic SDK, so a change in the SDK's rendering fails here instead of reaching the classifier unnoticed. Label the unit matrix, assert the preserved message, and pin a 400 that quotes a retryable 429. Document the remaining text-classification gaps in the README. Refs #8182 --- packages/llm/llm-pi-ai/README.i18n.yaml | 4 ++-- packages/llm/llm-pi-ai/README.md | 2 +- packages/llm/llm-pi-ai/README.zh.md | 2 +- packages/llm/llm-pi-ai/src/stream.ts | 15 ++++++------- packages/llm/llm-pi-ai/tests/adapter.spec.ts | 23 ++++++++++++++++++++ packages/llm/llm-pi-ai/tests/convert.spec.ts | 17 ++++++++------- 6 files changed, 43 insertions(+), 20 deletions(-) diff --git a/packages/llm/llm-pi-ai/README.i18n.yaml b/packages/llm/llm-pi-ai/README.i18n.yaml index f7c103d726..fd6a360e8b 100644 --- a/packages/llm/llm-pi-ai/README.i18n.yaml +++ b/packages/llm/llm-pi-ai/README.i18n.yaml @@ -93,8 +93,8 @@ en: b6bd570c0159ba08 zh: 7d2d91eea6a8d830 /deepseek-ai-dsh-llm-pi-ai/known-limitations-and-deferred-work: - en: 3a0cf0d369d25d37 - zh: 5974f328aea06aa8 + en: fdbe8ce850250b66 + zh: d8b9a06a996850c1 /deepseek-ai-dsh-llm-pi-ai/known-limitations-and-deferred-work/dev-note: en: 3678dd0f6642c2a5 zh: 75cbdd064cc64282 diff --git a/packages/llm/llm-pi-ai/README.md b/packages/llm/llm-pi-ai/README.md index c063137301..2bba84d774 100644 --- a/packages/llm/llm-pi-ai/README.md +++ b/packages/llm/llm-pi-ai/README.md @@ -271,7 +271,7 @@ These limits define where the adapter stops and future work begins. They are cur - **`GenerateOptions.stop` is unsupported** — pi-ai's common stream options cannot guarantee stop-sequence behavior across providers. - **Conversation update support comes only from the installed pi-ai catalog** — native system and tool updates require the catalog model’s mid-conversation compat flags, which profiles cannot set. Hand-declared routes and unflagged models receive later system text as user-role reminders and the complete current tool list at the start of each request. - **Images are accepted only in user and tool-result messages** — images in system, developer, or assistant history fail with `UNSUPPORTED_CONTENT` on both conversion paths. -- **Provider HTTP status is unavailable** — pi-ai error events do not expose a stable HTTP status across providers. +- **Provider HTTP status is unavailable** — pi-ai error events do not expose a stable HTTP status across providers, so failures are classified from the error text. When a message leads with a 4xx or 5xx status, as Anthropic and OpenAI SDK errors do, the status rules read only that status. Other formats, such as `Mistral API error (): `, still match a status quoted anywhere in the message, and the quota, rate-limit, and invalid-request wording rules read the whole message, so a gateway 5xx that relays an upstream `insufficient_quota` or `invalid_request_error` becomes a non-retryable `QUOTA` or `INVALID_REQUEST` failure. - **Retry policy is provider-owned, not an SDK retry** — pi-ai SDK retries stay disabled so durable agent steps and `llm/retry` events own every visible attempt, and direct `ctx.llm.stream()` calls remain single-attempt. - **Streamed JSON tool-call arguments avoid per-delta reparsing** — the installed pi-ai carries [`patches/@earendil-works__pi-ai@1.3.0.patch`](../../../patches/@earendil-works__pi-ai@1.3.0.patch), which removes the per-delta re-parse of the whole accumulated argument JSON in every stream adapter (upstream [earendil-works/pi#9265](https://github.com/earendil-works/pi/issues/9265)); unpatched, a multi-megabyte argument stream costs O(n²) CPU on the event loop and stalls every session in the process. Completion events supply parsed arguments; Responses may parse at both argument completion and output-item completion. This adapter reads only delta strings and finalized arguments. Re-apply or retire the patch on every pi-ai upgrade. - **Stale Anthropic thinking is omitted even where the endpoint would accept it** — prefix checks depend on the model, the account, and the request, which the adapter cannot observe, so every prefix change and every replay state without `prefixDigest` costs the model that reasoning, and Anthropic regenerates it as output tokens. The digest covers the pi-ai context objects, not the wire body: switching authentication or wire-changing compat settings mid-conversation is not detected, a pi-ai upgrade that changes Anthropic message serialization requires a new `DIGEST_SEED` in [`src/prefix-binding.ts`](src/prefix-binding.ts), and any change to how this package builds those objects drops recorded thinking once. diff --git a/packages/llm/llm-pi-ai/README.zh.md b/packages/llm/llm-pi-ai/README.zh.md index a8ee6f2eb6..9a964b8681 100644 --- a/packages/llm/llm-pi-ai/README.zh.md +++ b/packages/llm/llm-pi-ai/README.zh.md @@ -271,7 +271,7 @@ pi-ai 事件变成 harness 的推理、文本、工具调用、用量与 finish - **不支持 `GenerateOptions.stop`**——pi-ai 的通用流式选项无法跨提供方保证停止序列行为。 - **对话更新支持只来自已安装的 pi-ai 目录**——原生系统与工具更新需要目录模型的对话中 compat 标志，profile 不能设置这些标志。手工声明的路由与未标记的模型会以用户角色提醒接收后续系统文本，并在每次请求开头接收完整的当前工具列表。 - **图片只在 user 与工具结果消息中被接受**——system、developer 或 assistant 历史中的图片在两条转换路径上都会以 `UNSUPPORTED_CONTENT` 失败。 -- **提供方 HTTP 状态不可用**——pi-ai 错误事件不跨提供方暴露稳定 HTTP 状态。 +- **提供方 HTTP 状态不可用**——pi-ai 错误事件不跨提供方暴露稳定 HTTP 状态，失败按错误文本分类。信息以 4xx 或 5xx 状态码开头时（Anthropic 与 OpenAI SDK 的错误即如此），状态码规则只读取该状态码。其他格式（如 `Mistral API error (): `）仍会匹配信息中任意位置引用的状态码；配额、限流与无效请求的措辞规则读取整条信息，因此网关在 5xx 响应体中转述上游的 `insufficient_quota` 或 `invalid_request_error` 时，会分类为不重试的 `QUOTA` 或 `INVALID_REQUEST`。 - **重试策略由提供方自有，而非 SDK 重试**——pi-ai SDK 重试保持禁用，因此持久 agent（智能体）步骤与 `llm/retry` 事件拥有每个可见尝试，直接 `ctx.llm.stream()` 调用仍是单次尝试。 - **流式 JSON 工具调用参数避免逐 delta 重复解析**——安装的 pi-ai 带有 [`patches/@earendil-works__pi-ai@1.3.0.patch`](../../../patches/@earendil-works__pi-ai@1.3.0.patch)，它移除了每个流适配器中对整段累计参数 JSON 的逐 delta 重新解析（上游 [earendil-works/pi#9265](https://github.com/earendil-works/pi/issues/9265)）；未打补丁时，数 MB 的参数流会在事件循环上消耗 O(n²) CPU，并使进程内所有会话停滞。完成事件提供解析后的参数；Responses 可能在参数完成和输出项完成时分别解析。本适配器只读取 delta 字符串与最终参数。每次升级 pi-ai 时都要重新应用或撤销该补丁。 - **即使端点会接受，过期的 Anthropic thinking 也会被省略**——前缀检查取决于模型、账户和请求，而适配器无法观测这些因素，因此每次前缀变化以及每个不含 `prefixDigest` 的回放状态都会让模型失去这部分推理，Anthropic 会以输出 token 重新生成它。摘要覆盖 pi-ai 上下文对象，而不是线上请求体：对话中途切换认证或会改变线上请求的 compat 设置不会被检测到；改变 Anthropic 消息序列化的 pi-ai 升级需要在 [`src/prefix-binding.ts`](src/prefix-binding.ts) 中使用新的 `DIGEST_SEED`；本包构建这些对象的方式发生任何变化，都会使已记录的 thinking 被丢弃一次。 diff --git a/packages/llm/llm-pi-ai/src/stream.ts b/packages/llm/llm-pi-ai/src/stream.ts index 59bfdce550..c3bfc1cd3f 100644 --- a/packages/llm/llm-pi-ai/src/stream.ts +++ b/packages/llm/llm-pi-ai/src/stream.ts @@ -31,6 +31,13 @@ export function mapUsage(usage: PiUsage): TokenUsage { } } +// pi-ai's SDK-backed providers render an HTTP failure as ` ` +// (Anthropic and OpenAI SDK `APIError`, pi-messages ` : …`). +// The body can quote another hop's status, such as a gateway 500 that embeds its +// upstream's 401, so when a message leads with a status the status rules read +// only that status; other wordings scan the whole message. +const LEADING_HTTP_STATUS = /^[45]\d\d\b/ + // XXX(pi-ai upstream): pi-ai flattens the caught error to `error.message` // (api/anthropic-messages.js: `errorMessage = error instanceof Error ? // error.message : JSON.stringify(error)`), discarding the original Error and its @@ -39,14 +46,6 @@ export function mapUsage(usage: PiUsage): TokenUsage { // wrapper a bare `terminated`, so we are left pattern-matching terse words here. // If pi-ai ever forwards the original Error (or a fetch/dispatcher hook that lets // us capture the cause ourselves), classify on `code`/`cause` instead of text. -// -// pi-ai's SDK-backed providers render an HTTP failure as ` ` -// (Anthropic and OpenAI SDK `APIError`, pi-messages ` : …`). -// The body can quote another hop's status, such as a gateway 500 that embeds its -// upstream's 401, so when a message leads with a status the status rules read -// only that status; other wordings scan the whole message. -const LEADING_HTTP_STATUS = /^[45]\d\d\b/ - function classifyPiAiError(message: string): BuiltinLlmFailureCode { const statusSource = LEADING_HTTP_STATUS.exec(message)?.[0] ?? message if (/\b(?:401|403)\b/.test(statusSource)) return 'AUTH' diff --git a/packages/llm/llm-pi-ai/tests/adapter.spec.ts b/packages/llm/llm-pi-ai/tests/adapter.spec.ts index 4810a236c2..076f0fdf5f 100644 --- a/packages/llm/llm-pi-ai/tests/adapter.spec.ts +++ b/packages/llm/llm-pi-ai/tests/adapter.spec.ts @@ -474,6 +474,29 @@ describe('PiAiAdapter provider routing', () => { expect(server.paths).toEqual(['/chat/completions']) }) + it('classifies a gateway 500 that relays an upstream 401 as SERVER with the rendered message', async () => { + vi.stubEnv('PI_TEST_KEY', 'test-key') + // The Anthropic SDK renders the whole JSON body after the gateway's own status. + const body = JSON.stringify({ + error: { + origin: 'upstream', + message: 'upstream returned an error response: User not found.', + type: 'upstream_error', + upstream: { provider: 'custom', status: 401, body: { error: { message: 'User not found.' } } }, + }, + }) + const server = await mockServer([{ status: 500, body }]) + const ctx = new Context() + await ctx.plugin(LlmRuntime) + await ctx.plugin(LlmPiAi, { + providers: { + gateway: { apiKeyEnv: 'PI_TEST_KEY', api: 'anthropic-messages', baseURL: server.url, models: [{ id: 'gateway-model' }] }, + }, + }) + const result = await assemble(ctx, { provider: 'gateway', model: 'gateway-model', messages: [] }) + expect(result.finish).toEqual({ kind: 'error', failure: { message: `500 ${body}`, code: 'SERVER' } }) + }) + it('uses the resolved catalog context window for usage-based overflow detection', async () => { const model = getBuiltinModels('deepseek').find(candidate => candidate.id === 'deepseek-flash') if (model === undefined) throw new Error('deepseek-flash missing from pi-ai test catalog') diff --git a/packages/llm/llm-pi-ai/tests/convert.spec.ts b/packages/llm/llm-pi-ai/tests/convert.spec.ts index ebdf64fdb0..ff5a65870b 100644 --- a/packages/llm/llm-pi-ai/tests/convert.spec.ts +++ b/packages/llm/llm-pi-ai/tests/convert.spec.ts @@ -940,15 +940,16 @@ describe('mapStopReason / mapUsage', () => { }) it.each([ - // A gateway 500 that relays its upstream's 401 in the body, as the Anthropic SDK renders it. - ['500 {"error":{"origin":"upstream","message":"upstream returned an error response: User not found.","type":"upstream_error","code":"upstream_response_error","upstream":{"provider":"custom","status":401,"body":{"error":{"type":"","message":"User not found."},"type":"error"}}}}', 'SERVER'], - ['502 {"error":{"message":"upstream status 400"}}', 'SERVER'], - ['404 {"error":{"message":"upstream status 403"}}', 'PI_AI_ERROR'], - ['401 {"type":"error","error":{"type":"authentication_error","message":"invalid x-api-key"}}', 'AUTH'], - ['429 Too Many Requests: slow down', 'RATE_LIMIT'], - ] as const)('classifies %s by its leading response status', (errorMessage, code) => { + // The gateway body as the Anthropic SDK renders it: the gateway's own 500 leads. + ['a gateway 500 relaying an upstream 401', '500 {"error":{"origin":"upstream","message":"upstream returned an error response: User not found.","type":"upstream_error","code":"upstream_response_error","upstream":{"provider":"custom","status":401,"body":{"error":{"type":"","message":"User not found."},"type":"error"}}}}', 'SERVER'], + ['a 502 quoting a non-retryable 400', '502 {"error":{"message":"upstream status 400"}}', 'SERVER'], + ['a 400 quoting a retryable 429', '400 {"error":{"message":"upstream returned 429 Too Many Requests"}}', 'INVALID_REQUEST'], + ['a 404 quoting a 403', '404 {"error":{"message":"upstream status 403"}}', 'PI_AI_ERROR'], + ['a leading 401', '401 {"type":"error","error":{"type":"authentication_error","message":"invalid x-api-key"}}', 'AUTH'], + ['a leading 429', '429 Too Many Requests: slow down', 'RATE_LIMIT'], + ] as const)('classifies %s by its leading response status', (_label, errorMessage, code) => { expect(mapStopReason(assistant({ stopReason: 'error', errorMessage }))) - .toMatchObject({ kind: 'error', failure: { code } }) + .toEqual({ kind: 'error', failure: { message: errorMessage, code } }) }) it.each([