diff --git a/.agents/notes/implemented/architecture/2026-11-05-cua-scoped-surfaces-and-input.i18n.yaml b/.agents/notes/implemented/architecture/2026-11-05-cua-scoped-surfaces-and-input.i18n.yaml index f3a0d8f1d3..5f1ff64a4d 100644 --- a/.agents/notes/implemented/architecture/2026-11-05-cua-scoped-surfaces-and-input.i18n.yaml +++ b/.agents/notes/implemented/architecture/2026-11-05-cua-scoped-surfaces-and-input.i18n.yaml @@ -9,8 +9,8 @@ en: f72db0b79aa02fb5 zh: b91c1f177dc93ab4 /agent-note-cua-scoped-surfaces-and-separated-input-capabilities/decision: - en: ec115e11f96ffe60 - zh: c1fe9eb2866196d0 + en: 4fa61247fe33edb0 + zh: afa76f74841eb586 /agent-note-cua-scoped-surfaces-and-separated-input-capabilities/alternatives-considered: en: 32c970ae153e0f6c zh: 8431be0a10ffed15 diff --git a/.agents/notes/implemented/architecture/2026-11-05-cua-scoped-surfaces-and-input.md b/.agents/notes/implemented/architecture/2026-11-05-cua-scoped-surfaces-and-input.md index 609e5bf901..3888527013 100644 --- a/.agents/notes/implemented/architecture/2026-11-05-cua-scoped-surfaces-and-input.md +++ b/.agents/notes/implemented/architecture/2026-11-05-cua-scoped-surfaces-and-input.md @@ -16,7 +16,7 @@ Provider startup capabilities are static. Platform, supported methods, applicati Keyboard input keeps keys and modifiers separate: `pressKey(key, modifiers?)` presses and releases one key while holding the listed modifiers, and its keyboard modifier list is a plain string array defined by the provider. `sendKeyEvent(key, direction)` sends exactly one key-down or key-up without an implicit counterpart. A caller can therefore hold a modifier or split a gesture; neither method packs a chord string, and the shared schema does not enumerate platform key names. -Within one API instance, the object facade keeps one current accessibility snapshot per application handle, shared by every application and window object for that handle, including repeated selections of the same application. `getWindow` reads its initial tree silently; `getApp` does the same when the provider reports application-level accessibility. A later `getAccessibilityTree` read invalidates the previous snapshot before dispatch. The facade automatically attaches the latest snapshot ID to element operations; callers use element IDs from the latest tree and do not assume an older ID still identifies the same control. Element IDs are opaque provider strings. Only an explicit `CuaWindow` captures screenshots or sends pointer input. The native helper retains its own application and optional-window snapshots behind that facade; the native layer and the facade snapshot remain separate. The external runtime continues to own variable persistence and completion serialization; the facade adds no evaluator. The [native object decision](2026-09-17-experimental-cua-object-api.md) owns evaluator independence, and the [shared helper decision](2026-10-09-cua-shared-native-helper.md) owns connection lifetime. +Within one API instance, the object facade keeps one current accessibility snapshot per application handle, shared by every application and window object for that handle, including repeated selections of the same application. `getApp` and `getWindow` select objects without reading accessibility or replacing that snapshot; `getWindow` validates membership in the application’s current window list. An explicit `getAccessibilityTree` read invalidates the previous snapshot before dispatch. The facade automatically attaches the latest snapshot ID to element operations; callers use element IDs from the latest tree and do not assume an older ID still identifies the same control. Element IDs are opaque provider strings. Only an explicit `CuaWindow` captures screenshots or sends pointer input. The native helper retains its own application and optional-window snapshots behind that facade; the native layer and the facade snapshot remain separate. The external runtime continues to own variable persistence and completion serialization; the facade adds no evaluator. The [native object decision](2026-09-17-experimental-cua-object-api.md) owns evaluator independence, and the [shared helper decision](2026-10-09-cua-shared-native-helper.md) owns connection lifetime. ## Alternatives considered diff --git a/.agents/notes/implemented/architecture/2026-11-05-cua-scoped-surfaces-and-input.zh.md b/.agents/notes/implemented/architecture/2026-11-05-cua-scoped-surfaces-and-input.zh.md index 081f02986e..8a3f152e67 100644 --- a/.agents/notes/implemented/architecture/2026-11-05-cua-scoped-surfaces-and-input.zh.md +++ b/.agents/notes/implemented/architecture/2026-11-05-cua-scoped-surfaces-and-input.zh.md @@ -16,7 +16,7 @@ Provider 的启动能力是静态的。平台、支持的方法、应用级辅 键盘输入把按键与修饰键分开：`pressKey(key, modifiers?)` 在按住所列修饰键时按下并释放一个键，其键盘修饰键列表是 provider 定义的普通字符串数组。`sendKeyEvent(key, direction)` 只发送一次按下或抬起，不隐式发送相反事件。因此调用方可以保持修饰键或拆分手势；两个方法都不接受组合键字符串，共享 schema 也不枚举平台按键名。 -在单个 API 实例内，对象门面按应用句柄保存一份当前辅助功能快照，该句柄的所有应用对象和窗口对象共享这份快照，包括对同一应用的重复选择。`getWindow` 静默执行首次树读取；当 provider 报告应用级辅助功能时，`getApp` 也静默读取。之后的 `getAccessibilityTree` 读取会在分派前使上一份快照失效。门面会为元素操作自动附加最新快照 ID；调用方应使用最新树中的元素 ID，不应假定旧 ID 仍指向同一控件。元素 ID 是 provider 定义的不透明字符串。只有显式 `CuaWindow` 可以截图或发送指针输入。原生 helper 在该门面之后保留自己的应用快照和可选窗口快照；原生层与门面快照相互独立。外部运行时继续负责变量持久化和最终表达式序列化；门面不加入 evaluator。[原生对象决定](2026-09-17-experimental-cua-object-api.zh.md)负责 evaluator 独立性，[共享 helper 决定](2026-10-09-cua-shared-native-helper.zh.md)负责连接生命期。 +在单个 API 实例内，对象门面按应用句柄保存一份当前辅助功能快照，该句柄的所有应用对象和窗口对象共享这份快照，包括对同一应用的重复选择。`getApp` 和 `getWindow` 只选择对象，不读取辅助功能或替换该快照；`getWindow` 会核验窗口属于应用的当前窗口列表。显式 `getAccessibilityTree` 读取会在分派前使上一份快照失效。门面会为元素操作自动附加最新快照 ID；调用方应使用最新树中的元素 ID，不应假定旧 ID 仍指向同一控件。元素 ID 是 provider 定义的不透明字符串。只有显式 `CuaWindow` 可以截图或发送指针输入。原生 helper 在该门面之后保留自己的应用快照和可选窗口快照；原生层与门面快照相互独立。外部运行时继续负责变量持久化和最终表达式序列化；门面不加入 evaluator。[原生对象决定](2026-09-17-experimental-cua-object-api.zh.md)负责 evaluator 独立性，[共享 helper 决定](2026-10-09-cua-shared-native-helper.zh.md)负责连接生命期。 ## 考虑过的替代方案 diff --git a/docs/upgrade-guide/v0.2.2-rc.1/cua-native-platform/guide.i18n.yaml b/docs/upgrade-guide/v0.2.2-rc.1/cua-native-platform/guide.i18n.yaml index a76d4de320..c907006371 100644 --- a/docs/upgrade-guide/v0.2.2-rc.1/cua-native-platform/guide.i18n.yaml +++ b/docs/upgrade-guide/v0.2.2-rc.1/cua-native-platform/guide.i18n.yaml @@ -12,5 +12,5 @@ en: e5d1027faa55adba zh: 4b88ad6a70977fb3 /native-cua-application-and-window-api/migration: - en: b09e3dde32391786 - zh: f702e1e25b260b29 + en: b4d5bf5c30f42dfb + zh: 7ec59dc6a90165ef diff --git a/docs/upgrade-guide/v0.2.2-rc.1/cua-native-platform/guide.md b/docs/upgrade-guide/v0.2.2-rc.1/cua-native-platform/guide.md index 3179dae0d9..a49a251c37 100644 --- a/docs/upgrade-guide/v0.2.2-rc.1/cua-native-platform/guide.md +++ b/docs/upgrade-guide/v0.2.2-rc.1/cua-native-platform/guide.md @@ -16,7 +16,7 @@ Providers supply startup capabilities alongside their platform guidance. The nat ## Migration 1. After `cua.getApp(selector)`, select the target from `app.getWindows()` with `app.getWindow(windowId)`. Use `window.getScreenshot()`, `window.click()`, `window.scroll()`, and `window.drag()`. Replace `hover` with `window.mouseMove()`. On macOS, `window.click(id)` sends mouse events at the element center. Switch to `invoke(id)` for the default accessibility action, or `performAction(id, "AXShowMenu")` to request an advertised menu action. -2. Replace `getAXState()` with `getAccessibilityTree()`. Replace `getAXStateAndScreenshot()` with separate tree and screenshot reads. Selection reads the initial tree silently for a window, and for an app when `appLevelAccessibility` is true. Read the tree explicitly to display it. +2. Replace `getAXState()` with `getAccessibilityTree()`. Replace `getAXStateAndScreenshot()` with separate tree and screenshot reads. Application and window selection no longer read AX implicitly. Before using element IDs, call `getAccessibilityTree()` explicitly; use `{ emit: false }` to retain the snapshot without displaying it. A screenshot does not require a successful AX read. 3. Pass the opaque string ID from the latest tree wherever you passed a numeric element index. In raw requests use `elementId` with `snapshotId`. Replace `performSecondaryAction` with `listActions(id)` and `performAction(id, exactName, argument?)`, or use `invoke(id)` for the default action. `setValue` accepts the provider’s supported JSON values. On macOS, `performAction(id, 'AXRaise')` performs the advertised raise action. Call `app.activate()` first when the application must become active. 4. Replace a packed chord such as `pressKey('shift+Tab')` with `pressKey('Tab', ['shift'])`. Check `(await cua.getCapabilities()).methods` before using `sendKeyEvent`: macOS supports complete keystrokes and rejects directional events with `NOT_IMPLEMENTED`. 5. Read `dispatched` from action results, then observe the application to verify the outcome. `invoke` returns `dispatched: false` when the element has no default action. Remove reads of the old `verified` field and of `pid` on selected objects or window records. Use `listApps()` or running `findApps()` matches when you need a PID. diff --git a/docs/upgrade-guide/v0.2.2-rc.1/cua-native-platform/guide.zh.md b/docs/upgrade-guide/v0.2.2-rc.1/cua-native-platform/guide.zh.md index f0a4a905e0..fae92008f1 100644 --- a/docs/upgrade-guide/v0.2.2-rc.1/cua-native-platform/guide.zh.md +++ b/docs/upgrade-guide/v0.2.2-rc.1/cua-native-platform/guide.zh.md @@ -16,7 +16,7 @@ CUA 脚本在指定窗口上截图和操作鼠标。App 和 Window 对象共享 ## 迁移 1. 调用 `cua.getApp(selector)` 后，从 `app.getWindows()` 返回的列表选定目标，再调用 `app.getWindow(windowId)`。在窗口上调用 `getScreenshot()`、`click()`、`scroll()` 和 `drag()`，将原有的 `hover` 调用改为 `window.mouseMove()`。在 macOS 上，`window.click(id)` 在元素中心发送鼠标事件。需要执行默认辅助功能动作时，改用 `invoke(id)`。需要打开控件菜单且控件声明了 `AXShowMenu` 时，调用 `performAction(id, "AXShowMenu")`。 -2. 将 `getAXState()` 改为 `getAccessibilityTree()`，将 `getAXStateAndScreenshot()` 拆成读取树和截图两次调用。选择窗口时会静默读取初始树。选择应用时，只有 `appLevelAccessibility` 为真才读取。要显示树，请显式读取。 +2. 将 `getAXState()` 改为 `getAccessibilityTree()`，将 `getAXStateAndScreenshot()` 拆成读取树和截图两次调用。选择应用或窗口不再隐式读取 AX。使用元素 ID 前须显式调用 `getAccessibilityTree()`；使用 `{ emit: false }` 可保留快照但不显示树。截图不要求 AX 读取成功。 3. 将数字元素索引改为从最新树中取得的字符串 ID。直接发送请求时使用 `elementId` 和 `snapshotId`。原先调用 `performSecondaryAction` 的地方，先用 `listActions(id)` 取得完整动作名，再调用 `performAction(id, exactName, argument?)`。要执行默认动作，可调用 `invoke(id)`。`setValue` 接受提供方支持的 JSON 值。 在 macOS 上，`performAction(id, 'AXRaise')` 只执行元素声明的抬起窗口动作。需要把应用切到前台时先调用 `app.activate()`。 4. 将 `pressKey('shift+Tab')` 之类的组合键写法改为 `pressKey('Tab', ['shift'])`。调用 `sendKeyEvent` 前检查 `(await cua.getCapabilities()).methods`。macOS 支持完整按键，单独发送按下或松开事件会返回 `NOT_IMPLEMENTED`。 5. 从动作结果读取 `dispatched`，再观察应用确认效果。元素没有默认动作时，`invoke` 返回 `dispatched: false`。移除读取旧字段 `verified` 的代码，以及对已选对象和窗口记录中 `pid` 的读取。需要 PID 时，使用 `listApps()` 或 `findApps()` 返回的运行中应用记录。 diff --git a/packages/experimental/cua-macos/README.i18n.yaml b/packages/experimental/cua-macos/README.i18n.yaml index 0d83c6e538..7571c8004d 100644 --- a/packages/experimental/cua-macos/README.i18n.yaml +++ b/packages/experimental/cua-macos/README.i18n.yaml @@ -15,11 +15,11 @@ en: 1777e7bf6785f986 zh: e3daac1074ebb74d /deepseek-ai-dsh-experimental-cua-macos/use-this-package: - en: 533ed4dd11af5126 - zh: 03036e8c949d6aca + en: e9c78a8cbd68147c + zh: 98c64b09f858edf7 /deepseek-ai-dsh-experimental-cua-macos/use-this-package/connect-an-external-execution-environment: - en: b734a0907747580e - zh: 8110c5951b70e100 + en: 7c460511bb1cee4c + zh: 2eba4fc997427e18 /deepseek-ai-dsh-experimental-cua-macos/understand-the-implementation: en: 4e82b85a5aa72eee zh: 4e004b7a205d8688 diff --git a/packages/experimental/cua-macos/README.md b/packages/experimental/cua-macos/README.md index 46b035112e..d139d988a3 100644 --- a/packages/experimental/cua-macos/README.md +++ b/packages/experimental/cua-macos/README.md @@ -41,7 +41,7 @@ For repository Web use, enable [Computer Use](../cua-bundle/README.md) in Plugin pnpm dsh headless --patch packages/experimental/cua-macos/examples/macos.cordis.yml --dump-config ``` -The dump command checks profile composition without a model request. For a real task, replace `--dump-config` with a prompt and configure the profile’s model credentials. The overlay mounts the computer-use registry and this provider; remove an existing competing provider before using it. The helper needs Accessibility permission for AX/input and Screen Recording permission for capture; use the Computer Use permission page to open System Settings and add the helper. Source builds use the bundle filename `DSH Computer Use Dev.app` and declare `DSH Computer Use Dev` in Info.plist; select the development entry when granting access. `getCapabilities` returns the provider’s declared capabilities with observed permissions and session lock state. Locked or unavailable desktop sessions reject UI observations and input before dispatch; the user must unlock macOS before resuming. `listApps` and `cua.getState` list running applications. `findApps(query)` searches standard macOS application directories by name, bundle ID or absolute path. `getApp` selects a unique name, bundle ID, path or running PID, obtains application consent, and launches a stopped app in the background when needed. A stopped application’s bundle identifier and executable path are read from disk before approval and rechecked before and after launch; applications without a bundle identifier use their canonical executable path for reusable consent. Selecting a stopped installation requests a new instance of that copy. The helper waits for launch to complete before reporting the running state. A post-launch verification failure expires the pending handle but can leave the process running; inspect running applications before selecting a fresh handle. The returned object reads initial AX silently without activating the app. Launch cancellation may leave the application running. Model guidance starts with background observation and input, without activation during initialization or before each action. For hidden or minimized windows, try `app.show()` or `window.show()` and read fresh state. This requests unhide and unminimize without activation, raising or switching Spaces; app-level restoration selects the focused, main or sole AX window. The result names the selected `windowId`, or null when no window was selected, and reports `foregroundChanged` by comparing the front app and top window before and after the request. Null means foreground evidence was unavailable; later focus changes remain possible. If restoration fails or the window remains unusable, explain the needed foreground step, call `app.activate()` or an observed window’s advertised `AXRaise` action, then observe again. Necessary foreground access takes priority over preserving focus. Applications may change focus after input; background delivery does not guarantee an undisturbed user session. +The dump command checks profile composition without a model request. For a real task, replace `--dump-config` with a prompt and configure the profile’s model credentials. The overlay mounts the computer-use registry and this provider; remove an existing competing provider before using it. The helper needs Accessibility permission for AX/input and Screen Recording permission for capture; use the Computer Use permission page to open System Settings and add the helper. Source builds use the bundle filename `DSH Computer Use Dev.app` and declare `DSH Computer Use Dev` in Info.plist; select the development entry when granting access. `getCapabilities` returns the provider’s declared capabilities with observed permissions and session lock state. Locked or unavailable desktop sessions reject UI observations and input before dispatch; the user must unlock macOS before resuming. `listApps` and `cua.getState` list running applications. `findApps(query)` searches standard macOS application directories by name, bundle ID or absolute path. `getApp` selects a unique name, bundle ID, path or running PID, obtains application consent, and launches a stopped app in the background when needed. A stopped application’s bundle identifier and executable path are read from disk before approval and rechecked before and after launch; applications without a bundle identifier use their canonical executable path for reusable consent. Selecting a stopped installation requests a new instance of that copy. The helper waits for launch to complete before reporting the running state. A post-launch verification failure expires the pending handle but can leave the process running; inspect running applications before selecting a fresh handle. The returned object does not read AX or activate the app. Launch cancellation may leave the application running. Model guidance starts with background observation and input, without activation during initialization or before each action. For hidden or minimized windows, try `app.show()` or `window.show()` and read fresh state. This requests unhide and unminimize without activation, raising or switching Spaces; app-level restoration selects the focused, main or sole AX window. The result names the selected `windowId`, or null when no window was selected, and reports `foregroundChanged` by comparing the front app and top window before and after the request. Null means foreground evidence was unavailable; later focus changes remain possible. If restoration fails or the window remains unusable, explain the needed foreground step, call `app.activate()` or an observed window’s advertised `AXRaise` action, then observe again. Necessary foreground access takes priority over preserving focus. Applications may change focus after input; background delivery does not guarantee an undisturbed user session. Source builds default to `sourceSigning: ad-hoc`, without creating certificates or keychains. Changed binaries can require new permission grants. To reduce repeated grants during rebuilds and across worktrees, select an existing certificate and its absolute keychain path in the DSH home’s `cordis.patch.yml`; `cua` is the entry ID loading `@deepseek-ai/dsh-experimental-cua-macos`: @@ -111,9 +111,9 @@ const window = await app.getWindow(windows[0].windowId) void await window.getScreenshot() ``` -Use the `cua` API to discover and select the application (`listApps`, `getState`, `findApps`, `getApp`); use the selected app to activate it, enumerate its windows, and select one. Window selection is required for screenshots and pointer input, not only when several windows exist. `getApp` and `app.getWindow(windowId)` read their initial trees silently; choose the window by its observed title and bounds. +Use the `cua` API to discover and select the application (`listApps`, `getState`, `findApps`, `getApp`); use the selected app to activate it, enumerate its windows, and select one. Window selection is required for screenshots and pointer input, not only when several windows exist. `getApp` and `app.getWindow(windowId)` return without reading AX; choose the window from `getWindows()` by its title and bounds. -`getWindow` reads its initial tree silently. An owned window without an AX surface returns an empty snapshot and remains available for screenshots and coordinate input. Within one API instance, the facade keeps one current accessibility snapshot per application handle, shared by every app and window object for that handle, including repeated selections. Reading any window replaces that snapshot; element operations use the latest snapshot ID, so use IDs from the latest tree and do not assume an older ID still identifies the same control. A failed read publishes no partial snapshot; read again without repeating the preceding input. Window objects own screenshots and pointer input. Keyboard methods stay on `app` because input follows the app’s current focus. +`getWindow` validates the ID against the current application window list. Read `getAccessibilityTree()` explicitly before element operations. Screenshots and coordinate input do not require a successful AX read; absolute pixel coordinates require a delivered screenshot. An owned window without an AX surface returns an empty snapshot when explicitly observed. Within one API instance, the facade keeps one current accessibility snapshot per application handle, shared by every app and window object for that handle, including repeated selections. Reading any window replaces that snapshot; element operations use the latest snapshot ID, so use IDs from the latest tree and do not assume an older ID still identifies the same control. A failed read publishes no partial snapshot; read again without repeating the preceding input. Window objects own screenshots and pointer input. Keyboard methods stay on `app` because input follows the app’s current focus. `mouseMove(target)` defaults to AX hover. An enabled target or ancestor must advertise both `AXShowAlternateUI` and `AXShowDefaultUI`. Moving into its descendants keeps the UI open, and leaving restores it. Without an AX route the helper returns `HOVER_UNAVAILABLE` before sending input. Click a visible control or use `performAction` with an advertised action. For browser content without these AX actions, use browser tools. Enable `systemPointerHover` only when moving the user’s visible cursor is acceptable. That route requires a foreground, unobscured target and leaves the cursor at the destination. @@ -189,9 +189,9 @@ Setup and discovery: - cua.listApps(options?: { emit?: boolean }) -> { id: string, displayName: string, isRunning: true, pid: number }[] — enumerate running applications. - cua.findApps(query: string, options?: { emit?: boolean }) -> ({ id: string, displayName: string, path: string } & ({ isRunning: true, pid: number } | { isRunning: false }))[] — search installed applications, including stopped matches; only running matches carry pid. - cua.getState(options?: { emit?: boolean }) -> { apps: { id: string, displayName: string, isRunning: true, pid: number }[], browsers: [] } — summarize running applications; this is not an accessibility observation. -- cua.getApp(selector: string) -> App — select an application by name, bundle ID, absolute path, or running PID string, obtain consent, launch in the background if needed, and read initial AX state silently; it does not activate the app. +- cua.getApp(selector: string) -> App — select an application by name, bundle ID, absolute path, or running PID string, obtain consent, launch in the background if needed, and return without reading AX or activating the app. - app.getWindows(options?: { emit?: boolean }) -> { windowId: number, appName: string, title: string, bounds: { x: number, y: number, width: number, height: number }, isOnScreen: boolean, onCurrentSpace: boolean | null }[] — enumerate windows with global native bounds; null onCurrentSpace means unknown Space membership. -- app.getWindow(windowId: number) -> Window — select one window and read its initial AX state silently; do this before any window screenshot or coordinate input. +- app.getWindow(windowId: number) -> Window — select a window from the application’s current window list without reading AX; do this before any window screenshot or coordinate input. - app.activate() -> ActionResult — request explicit foreground activation; use only when foreground access is required. - app.show() -> { dispatched: boolean, windowId: number | null, foregroundChanged: boolean | null } — unhide the app and restore its selected, focused, main, or sole window without activation. - window.show() -> { dispatched: boolean, windowId: number | null, foregroundChanged: boolean | null } — unhide and restore that explicit window without activation. @@ -199,7 +199,7 @@ Setup and discovery: ## Accessibility tree Available at app or window level depending on the platform. -Element ids are opaque strings from the latest successful tree; the object facade keeps one current snapshot per app across its repeated app and window objects, a new read replaces it, and a failed read invalidates the previous one. Repeated strings can appear for different elements, so do not reuse an old id blindly, and element operations do not fall back to pointer input or typing. +Read getAccessibilityTree explicitly before using element ids. Selection and screenshots do not read or replace the current AX snapshot. Element ids are opaque strings from the latest successful tree; the object facade keeps one current snapshot per app across its repeated app and window objects, a new read replaces it, and a failed read invalidates the previous one. Repeated strings can appear for different elements, so do not reuse an old id blindly, and element operations do not fall back to pointer input or typing. - getAccessibilityTree(options?: { emit?: boolean, includeMenus?: boolean, details?: boolean }) -> string — read the full tree text; includeMenus expands menu contents and details adds identifiers, native bounds, and complete action names. - listActions(elementId: string) -> string[] — list the actions an element advertises. - invoke(elementId: string) -> ActionResult — invoke the element's default action; dispatched is false when it has none. @@ -223,7 +223,7 @@ Available at app or window level depending on the platform. - paste(text: string, options?: { format?: 'text' | 'html' }) -> ActionResult — deliver text or HTML through the app's Paste command, restoring prior clipboard contents unless another writer changes them; a Paste failure is surfaced without substituting typeText, and PASTE_NOT_CONSUMED means inspect before any retry. - pressKey(key: string, modifiers?: string[]) -> ActionResult — press and release one complete keystroke. For example pressKey('Tab', ['shift']) or pressKey('+'). Key and modifier names are platform-defined strings. Pass key and modifiers separately; do not pack them into a chord string. -Raw tools use { method, params }: include appHandle for app calls and windowId for window calls, and use snapshotId with elementId for element routes. The emit option is facade-only: emit is local output control, so raw callers must follow the raw operation schema for admitted params. getAccessibilityTree and getWindow read through raw getState; getWindows maps to listWindows; getScreenshot maps to screenshot; cua.getState maps to listApps; every other named operation uses the same method name. +Raw tools use { method, params }: include appHandle for app calls and windowId for window calls, and use snapshotId with elementId for element routes. The emit option is facade-only: emit is local output control, so raw callers must follow the raw operation schema for admitted params. getAccessibilityTree maps to getState; getWindow validates against listWindows; getWindows maps to listWindows; getScreenshot maps to screenshot; cua.getState maps to listApps; every other named operation uses the same method name. ``` #### Token effect diff --git a/packages/experimental/cua-macos/README.zh.md b/packages/experimental/cua-macos/README.zh.md index 341796b8c4..f8fed7b624 100644 --- a/packages/experimental/cua-macos/README.zh.md +++ b/packages/experimental/cua-macos/README.zh.md @@ -41,7 +41,7 @@ provider 按当前界面语言生成引导和恢复文案，不支持的语言 pnpm dsh headless --patch packages/experimental/cua-macos/examples/macos.cordis.yml --dump-config ``` -配置输出命令不发起模型请求。执行真实任务时，将 `--dump-config` 替换为任务提示，并配置该 profile 的模型凭据。配置挂载 computer-use 注册表和本 provider；使用前需移除已有的竞争 provider。helper 需要辅助功能权限来读取 AX 和输入，需要屏幕录制权限来截图；通过 Computer Use 权限页打开系统设置并添加 helper。源码构建的应用包名为 `DSH Computer Use Dev.app`，Info.plist 声明名称 `DSH Computer Use Dev`；授权时选择开发版条目。`getCapabilities` 返回提供方声明的平台能力，以及实际查询的权限和锁屏状态。桌面会话锁定或不可用时，界面观察与输入会在分派前拒绝，需由用户解锁 macOS 后继续。`listApps` 和 `cua.getState` 列出运行中的应用。`findApps(query)` 按名称、bundle ID 或绝对路径搜索 macOS 标准应用目录。`getApp` 按唯一名称、bundle ID、路径或运行中的 PID 选择应用，取得应用许可后，在需要时后台启动尚未运行的应用。授权前从磁盘读取应用的 Bundle ID 和可执行路径，在启动前后重新读取并核对；没有 Bundle ID 的应用使用规范化的可执行路径复用授权。选择尚未运行的安装副本时，会请求为该副本创建新实例。helper 等待应用完成启动后返回运行状态。启动后验证失败会使待启动句柄失效，但进程可能仍在运行；应先查看运行中的应用，再取得新句柄。返回的对象静默读取初始 AX，不自动激活应用。启动被取消时，应用可能已经运行。模型指引要求先在后台观察和输入，不在初始化或每次动作前激活应用。窗口隐藏或最小化时，先尝试 `app.show()` 或 `window.show()`，再重新观察。它请求取消隐藏和恢复最小化，不主动激活、抬起窗口或切换桌面空间；应用级恢复选择焦点窗口、主窗口或唯一 AX 窗口。结果通过 `windowId` 指明选中的窗口，未选中时为 null；`foregroundChanged` 比较操作前后的前台应用和最上层窗口，无法取得完整观察时为 null，返回后仍可能发生焦点变化。恢复失败或窗口仍不可用时，说明需要前台的步骤，再调用 `app.activate()` 或窗口声明的 `AXRaise` 动作并重新观察。必须使用前台时，以完成任务为先。应用收到输入后仍可能改变焦点，后台投递不保证用户操作始终不受干扰。 +配置输出命令不发起模型请求。执行真实任务时，将 `--dump-config` 替换为任务提示，并配置该 profile 的模型凭据。配置挂载 computer-use 注册表和本 provider；使用前需移除已有的竞争 provider。helper 需要辅助功能权限来读取 AX 和输入，需要屏幕录制权限来截图；通过 Computer Use 权限页打开系统设置并添加 helper。源码构建的应用包名为 `DSH Computer Use Dev.app`，Info.plist 声明名称 `DSH Computer Use Dev`；授权时选择开发版条目。`getCapabilities` 返回提供方声明的平台能力，以及实际查询的权限和锁屏状态。桌面会话锁定或不可用时，界面观察与输入会在分派前拒绝，需由用户解锁 macOS 后继续。`listApps` 和 `cua.getState` 列出运行中的应用。`findApps(query)` 按名称、bundle ID 或绝对路径搜索 macOS 标准应用目录。`getApp` 按唯一名称、bundle ID、路径或运行中的 PID 选择应用，取得应用许可后，在需要时后台启动尚未运行的应用。授权前从磁盘读取应用的 Bundle ID 和可执行路径，在启动前后重新读取并核对；没有 Bundle ID 的应用使用规范化的可执行路径复用授权。选择尚未运行的安装副本时，会请求为该副本创建新实例。helper 等待应用完成启动后返回运行状态。启动后验证失败会使待启动句柄失效，但进程可能仍在运行；应先查看运行中的应用，再取得新句柄。返回对象时不读取 AX，也不激活应用。启动被取消时，应用可能已经运行。模型指引要求先在后台观察和输入，不在初始化或每次动作前激活应用。窗口隐藏或最小化时，先尝试 `app.show()` 或 `window.show()`，再重新观察。它请求取消隐藏和恢复最小化，不主动激活、抬起窗口或切换桌面空间；应用级恢复选择焦点窗口、主窗口或唯一 AX 窗口。结果通过 `windowId` 指明选中的窗口，未选中时为 null；`foregroundChanged` 比较操作前后的前台应用和最上层窗口，无法取得完整观察时为 null，返回后仍可能发生焦点变化。恢复失败或窗口仍不可用时，说明需要前台的步骤，再调用 `app.activate()` 或窗口声明的 `AXRaise` 动作并重新观察。必须使用前台时，以完成任务为先。应用收到输入后仍可能改变焦点，后台投递不保证用户操作始终不受干扰。 源码构建默认使用 `sourceSigning: ad-hoc` 临时签名，不创建证书或钥匙串。二进制变化后可能需要重新授权。需要减少开发重建和跨 worktree 的授权次数时，在 DSH home 的 `cordis.patch.yml` 中选择已有证书及其钥匙串绝对路径；`cua` 是加载 `@deepseek-ai/dsh-experimental-cua-macos` 的节点 ID： @@ -111,9 +111,9 @@ const window = await app.getWindow(windows[0].windowId) void await window.getScreenshot() ``` -使用 `cua` API 发现并选择应用（`listApps`、`getState`、`findApps`、`getApp`）；使用选定的应用来激活它、枚举其窗口并选择窗口。截图和指针输入都要求选择窗口，不只在存在多个窗口时才需要。`getApp` 和 `app.getWindow(windowId)` 静默读取初始树；根据观察到的标题与位置选择窗口。 +使用 `cua` API 发现并选择应用（`listApps`、`getState`、`findApps`、`getApp`）；使用选定的应用来激活它、枚举其窗口并选择窗口。截图和指针输入都要求选择窗口，不只在存在多个窗口时才需要。`getApp` 和 `app.getWindow(windowId)` 返回时不读取 AX；根据 `getWindows()` 中的标题与位置选择窗口。 -`getWindow` 静默读取初始树。已确认归属但没有 AX 界面的窗口返回空快照，仍可截图和发送坐标输入。在单个 API 实例内，门面按应用句柄保存一份当前辅助功能快照，该句柄的所有应用对象和窗口对象共享它，包括重复选择。读取任一窗口都会替换该快照；元素操作使用最新快照 ID，因此应使用最新树中的元素 ID，不应假定旧 ID 仍指向同一控件。读取失败不会发布部分快照；可以重新读取，但不要重复此前的输入。窗口对象负责截图和指针输入。键盘方法保留在 `app` 上，因为输入跟随应用的当前焦点。 +`getWindow` 根据当前应用窗口清单验证 ID。元素操作前须显式读取 `getAccessibilityTree()`。截图和坐标输入不要求 AX 读取成功；绝对像素坐标要求已交付的截图。已确认归属但没有 AX 界面的窗口在显式观察时返回空快照。在单个 API 实例内，门面按应用句柄保存一份当前辅助功能快照，该句柄的所有应用对象和窗口对象共享它，包括重复选择。读取任一窗口都会替换该快照；元素操作使用最新快照 ID，因此应使用最新树中的元素 ID，不应假定旧 ID 仍指向同一控件。读取失败不会发布部分快照；可以重新读取，但不要重复此前的输入。窗口对象负责截图和指针输入。键盘方法保留在 `app` 上，因为输入跟随应用的当前焦点。 `mouseMove(target)` 默认通过 AX 悬停。目标或其祖先必须已启用，并同时声明 `AXShowAlternateUI` 和 `AXShowDefaultUI`。进入子控件会保持展开，离开后恢复默认界面。如果没有可用的 AX 路径，helper 返回 `HOVER_UNAVAILABLE`，不发送输入。可点击已经可见的控件，或用 `performAction` 执行控件声明的动作。浏览器内容缺少这些 AX 动作时，请使用浏览器工具。只有允许移动用户可见光标时才开启 `systemPointerHover`。这条路径要求目标在前台且未被遮挡，操作后光标停在目标位置。 @@ -189,9 +189,9 @@ Setup and discovery: - cua.listApps(options?: { emit?: boolean }) -> { id: string, displayName: string, isRunning: true, pid: number }[] — enumerate running applications. - cua.findApps(query: string, options?: { emit?: boolean }) -> ({ id: string, displayName: string, path: string } & ({ isRunning: true, pid: number } | { isRunning: false }))[] — search installed applications, including stopped matches; only running matches carry pid. - cua.getState(options?: { emit?: boolean }) -> { apps: { id: string, displayName: string, isRunning: true, pid: number }[], browsers: [] } — summarize running applications; this is not an accessibility observation. -- cua.getApp(selector: string) -> App — select an application by name, bundle ID, absolute path, or running PID string, obtain consent, launch in the background if needed, and read initial AX state silently; it does not activate the app. +- cua.getApp(selector: string) -> App — select an application by name, bundle ID, absolute path, or running PID string, obtain consent, launch in the background if needed, and return without reading AX or activating the app. - app.getWindows(options?: { emit?: boolean }) -> { windowId: number, appName: string, title: string, bounds: { x: number, y: number, width: number, height: number }, isOnScreen: boolean, onCurrentSpace: boolean | null }[] — enumerate windows with global native bounds; null onCurrentSpace means unknown Space membership. -- app.getWindow(windowId: number) -> Window — select one window and read its initial AX state silently; do this before any window screenshot or coordinate input. +- app.getWindow(windowId: number) -> Window — select a window from the application’s current window list without reading AX; do this before any window screenshot or coordinate input. - app.activate() -> ActionResult — request explicit foreground activation; use only when foreground access is required. - app.show() -> { dispatched: boolean, windowId: number | null, foregroundChanged: boolean | null } — unhide the app and restore its selected, focused, main, or sole window without activation. - window.show() -> { dispatched: boolean, windowId: number | null, foregroundChanged: boolean | null } — unhide and restore that explicit window without activation. @@ -199,7 +199,7 @@ Setup and discovery: ## Accessibility tree Available at app or window level depending on the platform. -Element ids are opaque strings from the latest successful tree; the object facade keeps one current snapshot per app across its repeated app and window objects, a new read replaces it, and a failed read invalidates the previous one. Repeated strings can appear for different elements, so do not reuse an old id blindly, and element operations do not fall back to pointer input or typing. +Read getAccessibilityTree explicitly before using element ids. Selection and screenshots do not read or replace the current AX snapshot. Element ids are opaque strings from the latest successful tree; the object facade keeps one current snapshot per app across its repeated app and window objects, a new read replaces it, and a failed read invalidates the previous one. Repeated strings can appear for different elements, so do not reuse an old id blindly, and element operations do not fall back to pointer input or typing. - getAccessibilityTree(options?: { emit?: boolean, includeMenus?: boolean, details?: boolean }) -> string — read the full tree text; includeMenus expands menu contents and details adds identifiers, native bounds, and complete action names. - listActions(elementId: string) -> string[] — list the actions an element advertises. - invoke(elementId: string) -> ActionResult — invoke the element's default action; dispatched is false when it has none. @@ -223,7 +223,7 @@ Available at app or window level depending on the platform. - paste(text: string, options?: { format?: 'text' | 'html' }) -> ActionResult — deliver text or HTML through the app's Paste command, restoring prior clipboard contents unless another writer changes them; a Paste failure is surfaced without substituting typeText, and PASTE_NOT_CONSUMED means inspect before any retry. - pressKey(key: string, modifiers?: string[]) -> ActionResult — press and release one complete keystroke. For example pressKey('Tab', ['shift']) or pressKey('+'). Key and modifier names are platform-defined strings. Pass key and modifiers separately; do not pack them into a chord string. -Raw tools use { method, params }: include appHandle for app calls and windowId for window calls, and use snapshotId with elementId for element routes. The emit option is facade-only: emit is local output control, so raw callers must follow the raw operation schema for admitted params. getAccessibilityTree and getWindow read through raw getState; getWindows maps to listWindows; getScreenshot maps to screenshot; cua.getState maps to listApps; every other named operation uses the same method name. +Raw tools use { method, params }: include appHandle for app calls and windowId for window calls, and use snapshotId with elementId for element routes. The emit option is facade-only: emit is local output control, so raw callers must follow the raw operation schema for admitted params. getAccessibilityTree maps to getState; getWindow validates against listWindows; getWindows maps to listWindows; getScreenshot maps to screenshot; cua.getState maps to listApps; every other named operation uses the same method name. ``` #### Token 影响 diff --git a/packages/experimental/cua-macos/tests/loader-composition.spec.ts b/packages/experimental/cua-macos/tests/loader-composition.spec.ts index e92f138d87..e1395145d4 100644 --- a/packages/experimental/cua-macos/tests/loader-composition.spec.ts +++ b/packages/experimental/cua-macos/tests/loader-composition.spec.ts @@ -181,7 +181,7 @@ it.skipIf(process.platform === 'win32')('shares one application between retained const screenshot = await peerWindow.getScreenshot({ emit: false }) expect(screenshot).toMatchObject({ mimeType: 'image/png', pixelWidth: 1, pixelHeight: 1, windowId: 7 }) expect(Buffer.from(screenshot.bytes).toString('base64')).toBe(png) - await peerApp.getWindow(8) + await (await peerApp.getWindow(8)).getAccessibilityTree({ emit: false }) const recovery = 'Snapshot belongs to window 8; read getAccessibilityTree on window 7' await expect(peerWindow.click(element('0'))).rejects.toThrow(recovery) await expect(peerWindow.mouseMove(element('0'))).rejects.toThrow(recovery) diff --git a/packages/experimental/cua-macos/tests/persistent-bridge.built.e2e.ts b/packages/experimental/cua-macos/tests/persistent-bridge.built.e2e.ts index b37b79b6c1..61fe3b3bdb 100644 --- a/packages/experimental/cua-macos/tests/persistent-bridge.built.e2e.ts +++ b/packages/experimental/cua-macos/tests/persistent-bridge.built.e2e.ts @@ -74,7 +74,7 @@ it.skipIf(process.platform === 'win32' || !existsSync(bridge) || !existsSync(boo const section = (await ctx.systemPrompt.assemble()).sections.find(value => value.name === 'computer-use:dsh-cua') const initialization = section?.text.match(/Persistent PTC initialization:\n\n```ts\n([\s\S]*?)\n```/u)?.[1] expect(initialization).toContain('/lib/bridge.js') - const first = await run(`${initialization}\nvar app = await cua.getApp("test.fixture"); var win = await app.getWindow(7); await win.click("0"); var shot = await win.getScreenshot(); ({ mimeType: shot.mimeType, pixelWidth: shot.pixelWidth, pixelHeight: shot.pixelHeight, windowId: shot.windowId, byteLength: shot.bytes.length });`, 'first') + const first = await run(`${initialization}\nvar app = await cua.getApp("test.fixture"); var win = await app.getWindow(7); await win.getAccessibilityTree({emit:false}); await win.click("0"); var shot = await win.getScreenshot(); ({ mimeType: shot.mimeType, pixelWidth: shot.pixelWidth, pixelHeight: shot.pixelHeight, windowId: shot.windowId, byteLength: shot.bytes.length });`, 'first') expect(first.error).toBeUndefined() expect(first.value).toEqual({ mimeType: 'image/png', pixelWidth: 1, pixelHeight: 1, windowId: 7, byteLength: 90 }) expect(images).toBe(1) diff --git a/packages/experimental/cua/README.i18n.yaml b/packages/experimental/cua/README.i18n.yaml index f4abb13c43..fffdfaa5c6 100644 --- a/packages/experimental/cua/README.i18n.yaml +++ b/packages/experimental/cua/README.i18n.yaml @@ -15,14 +15,14 @@ en: 5db4d4fea25493b9 zh: 65ab64fc24eb0d09 /deepseek-ai-dsh-cua/use-this-package: - en: ba1f8e6ae51f69d7 - zh: 8f4fde9857bae319 + en: 77dbaaa91d5b92c8 + zh: 94c3a758a91476bd /deepseek-ai-dsh-cua/use-this-package/permission-controls: en: d92d5948a7941e8d zh: d43491ae6521b87e /deepseek-ai-dsh-cua/understand-the-implementation: - en: 2ab74a1ff120459e - zh: 39b3ef198989f133 + en: fbd52f5ae2ca44f8 + zh: 386fb22fa7f7703d /deepseek-ai-dsh-cua/further-exploration: en: c174fb5a00ea3681 zh: 039340042676881d diff --git a/packages/experimental/cua/README.md b/packages/experimental/cua/README.md index c95e4ae8ec..0b3d39fdd1 100644 --- a/packages/experimental/cua/README.md +++ b/packages/experimental/cua/README.md @@ -31,7 +31,7 @@ Application consent supplies English, Chinese, and Japanese display text from `s Mount the [macOS provider](../cua-macos/README.md) to enable native operations on macOS. External PTC hosts import `connectCua` from `@deepseek-ai/dsh-cua/bridge` and supply the current `tools.cua` binding. Source compositions and native preparation belong to the provider. -Use `cua.getApp(selector)` to select an application; it obtains consent, optionally launches a stopped application in the background, and reads its initial accessibility tree silently when the provider supports application-level accessibility. `cua.listApps` and `cua.getState` list running applications; `cua.findApps(query)` searches installed applications, including stopped ones. `app.activate()`, `app.getWindows()`, and `app.getWindow(windowId)` operate on the selected application; `getWindow` returns the object for screenshots and pointer input. Select an application and one of its windows before any capture or pointer input. Call `app.getAccessibilityTree()` to emit the tree. Within one API instance, the facade keeps one current accessibility snapshot per application handle across app and window objects, including repeated selections; each new read invalidates the previous snapshot. Element operations use the latest snapshot ID, so callers use IDs from the latest tree and do not assume an older ID still identifies the same control. Element IDs are opaque strings. +Use `cua.getApp(selector)` to select an application; it obtains consent and optionally launches a stopped application in the background, without reading its accessibility tree. `cua.listApps` and `cua.getState` list running applications; `cua.findApps(query)` searches installed applications, including stopped ones. `app.activate()`, `app.getWindows()`, and `app.getWindow(windowId)` operate on the selected application; `getWindow` validates the ID against the current application window list and returns the object without reading AX. Select an application and one of its windows before any capture or pointer input. Read `app.getAccessibilityTree()` or `window.getAccessibilityTree()` explicitly before element operations. Screenshots remain available when AX reading fails. Within one API instance, the facade keeps one current accessibility snapshot per application handle across app and window objects, including repeated selections; each new read invalidates the previous snapshot. Element operations use the latest snapshot ID, so callers use IDs from the latest tree and do not assume an older ID still identifies the same control. Element IDs are opaque strings. Keyboard methods are available in the scopes a provider declares; the provider owns how input is delivered within each scope. A lost connection invalidates its handles and coordinates. Failed cleanup is reported before that connection is retired; select the application again and inspect fresh state, because earlier input may have partially executed. Screenshot image errors retain the failing stage and attachment error code. Invalid request arguments reject before dispatch with a readable method, parameter and requirement. @@ -87,7 +87,7 @@ The provider calls [installCua](src/index.ts) with its prepared helper, connecti [The operation schema](src/protocol.ts) is the authority for model parameters and separately declared Host-only fields. Native implementations enforce target identity, window geometry, resource limits, and operating-system access at execution. The permission store owns durable consent separately from the native protocol. -One object facade ([src/api.ts](src/api.ts)) builds a shared `CuaSurface` and derives `CuaApp` and `CuaWindow` from it. The derived factories inject scope parameters and the accessibility and keyboard checks for their scope; the shared base does not branch on application-only capabilities. Within one API instance, the facade keeps one current accessibility snapshot per `appHandle` across every app and window object for that handle. `getWindow` reads its initial tree silently; `getApp` does the same when the provider reports application-level accessibility. Each later `getAccessibilityTree()` invalidates the previous snapshot before dispatch, and element operations use the latest snapshot ID. Only a `CuaWindow` captures screenshots or sends pointer input. +One object facade ([src/api.ts](src/api.ts)) builds a shared `CuaSurface` and derives `CuaApp` and `CuaWindow` from it. The derived factories inject scope parameters and the accessibility and keyboard checks for their scope; the shared base does not branch on application-only capabilities. Within one API instance, the facade keeps one current accessibility snapshot per `appHandle` across every app and window object for that handle. Selection leaves the current accessibility snapshot unchanged. Each explicit `getAccessibilityTree()` invalidates the previous snapshot before dispatch, and element operations use the latest snapshot ID. Only a `CuaWindow` captures screenshots or sends pointer input. Provider startup capabilities are fixed at installation: platform, methods, app-level accessibility, keyboard scopes, browser support, and coordinate space. The native `getCapabilities` RPC reports runtime permissions and session lock only; `installCua` merges both layers for the model-visible result. The raw RPC remains a common operation representation rather than a capability inventory, so provider guidance states the actions its platform supports. diff --git a/packages/experimental/cua/README.zh.md b/packages/experimental/cua/README.zh.md index c1c5ed863c..b79cc6fdb5 100644 --- a/packages/experimental/cua/README.zh.md +++ b/packages/experimental/cua/README.zh.md @@ -31,7 +31,7 @@ kind: "package-reference" 挂载 [macOS 提供方](../cua-macos/README.zh.md)以在 macOS 上启用原生操作。外部 PTC 宿主从 `@deepseek-ai/dsh-cua/bridge` 导入 `connectCua`，并提供当前 `tools.cua` 绑定。源码组合与原生组件准备由提供方负责。 -使用 `cua.getApp(selector)` 选择应用；它会获取授权、按需在后台启动已停止的应用，并在 provider 支持应用级辅助功能时静默读取初始辅助功能树。`cua.listApps` 和 `cua.getState` 列出运行中的应用；`cua.findApps(query)` 搜索已安装应用，包括未运行的应用。`app.activate()`、`app.getWindows()` 和 `app.getWindow(windowId)` 操作选定的应用；`getWindow` 返回用于截图和指针输入的对象。进行任何截图或指针输入前，必须选择应用及其一个窗口。调用 `app.getAccessibilityTree()` 输出树。在单个 API 实例内，门面按应用句柄保存一份当前辅助功能快照，该句柄的所有应用对象和窗口对象共享它，包括重复选择；每次新读取都会使上一份快照失效。元素操作使用最新快照 ID，因此调用方应使用最新树中的元素 ID，不应假定旧 ID 仍指向同一控件。元素 ID 是不透明字符串。 +使用 `cua.getApp(selector)` 选择应用；它会获取授权、按需在后台启动已停止的应用，不读取辅助功能树。`cua.listApps` 和 `cua.getState` 列出运行中的应用；`cua.findApps(query)` 搜索已安装应用，包括未运行的应用。`app.activate()`、`app.getWindows()` 和 `app.getWindow(windowId)` 操作选定的应用；`getWindow` 根据当前应用窗口清单验证 ID，然后返回对象，不读取 AX。进行任何截图或指针输入前，必须选择应用及其一个窗口。元素操作前须显式读取 `app.getAccessibilityTree()` 或 `window.getAccessibilityTree()`。AX 读取失败时仍可截图。在单个 API 实例内，门面按应用句柄保存一份当前辅助功能快照，该句柄的所有应用对象和窗口对象共享它，包括重复选择；每次新读取都会使上一份快照失效。元素操作使用最新快照 ID，因此调用方应使用最新树中的元素 ID，不应假定旧 ID 仍指向同一控件。元素 ID 是不透明字符串。 键盘方法可在 provider 声明的作用域中使用；每个作用域内的输入投递由 provider 负责。连接丢失会使其句柄和坐标失效。清理失败会先报告，再释放该连接；应重新选择应用并读取新状态，因为先前的输入可能已部分执行。截图图像错误会保留失败阶段和附件错误码。请求参数校验失败时，在分派前返回包含方法、参数和要求的可读消息。 @@ -87,7 +87,7 @@ Computer Use 详情页的权限卡片、组件列表和来源信息使用 l3 边 [操作 schema](src/protocol.ts) 定义模型参数，并单独声明 Host 专用字段。原生实现在执行时检查目标身份、窗口几何、资源上限和操作系统访问条件。授权存储独立于原生协议保存持久许可。 -对象门面（[src/api.ts](src/api.ts)）构建共享的 `CuaSurface`，并从它派生 `CuaApp` 和 `CuaWindow`。派生工厂注入作用域参数以及该作用域的辅助功能和键盘检查；共享基类不为仅属于应用的能力分支。在单个 API 实例内，门面按 `appHandle` 保存一份当前辅助功能快照，该句柄的所有应用对象和窗口对象共享它。`getWindow` 静默读取初始树；当 provider 报告应用级辅助功能时，`getApp` 也静默读取。之后每次 `getAccessibilityTree()` 在分派前使上一份快照失效，元素操作使用最新快照 ID。只有 `CuaWindow` 可以截图或发送指针输入。 +对象门面（[src/api.ts](src/api.ts)）构建共享的 `CuaSurface`，并从它派生 `CuaApp` 和 `CuaWindow`。派生工厂注入作用域参数以及该作用域的辅助功能和键盘检查；共享基类不为仅属于应用的能力分支。在单个 API 实例内，门面按 `appHandle` 保存一份当前辅助功能快照，该句柄的所有应用对象和窗口对象共享它。选择对象不改变当前辅助功能快照。每次显式调用 `getAccessibilityTree()` 在分派前使上一份快照失效，元素操作使用最新快照 ID。只有 `CuaWindow` 可以截图或发送指针输入。 提供方启动能力在安装时固定：平台、方法、应用级辅助功能、键盘作用域、浏览器支持和坐标空间。原生 `getCapabilities` RPC 只报告运行时权限和会话锁定状态；`installCua` 为模型可见结果合并两层。原生 RPC 仍是通用操作表示，而不是能力清单，因此 provider 指引说明其平台支持的动作。 diff --git a/packages/experimental/cua/package.json b/packages/experimental/cua/package.json index 440f321935..0d62908a8a 100644 --- a/packages/experimental/cua/package.json +++ b/packages/experimental/cua/package.json @@ -107,7 +107,8 @@ "@deepseek-ai/dsh-atomic-write": "workspace:*", "@deepseek-ai/dsh-typert-protocol": "workspace:*", "@deepseek-ai/dsh-computer-use": "workspace:*", - "@deepseek-ai/dsh-storage-domain": "workspace:*" + "@deepseek-ai/dsh-storage-domain": "workspace:*", + "@deepseek-ai/dsh-error": "workspace:*" }, "devDependencies": { "@deepseek-ai/dsh-working-directory": "workspace:*", diff --git a/packages/experimental/cua/src/api.ts b/packages/experimental/cua/src/api.ts index 22039c6591..509b9b8676 100644 --- a/packages/experimental/cua/src/api.ts +++ b/packages/experimental/cua/src/api.ts @@ -1,5 +1,6 @@ /** Object facade over logged native operations; its caller owns evaluator persistence. */ import { Buffer } from 'node:buffer' +import { HarnessError } from '@deepseek-ai/dsh-error' import type { z } from 'zod' import type { JsonValue } from '@deepseek-ai/dsh-util-values' import { ActionNamesResult, ActionResult, AppResult, AppsResult, CapabilitiesResult, FoundAppsResult, HoverResult, Request, ScreenshotResult, ShowResult, StateResult, WindowsResult } from './protocol.ts' @@ -79,9 +80,7 @@ export function createCuaApi(dispatch: CuaDispatch, sink?: CuaObservationSink): if (!request.success) throw new Error(`getApp ${requestIssueMessage(request.error.issues)}`) const capabilities = await currentCapabilities() const { appHandle } = AppResult.parse(await call('getApp', { selector })) - const app = objects.createCuaApp(appHandle, capabilities) - if (capabilities.appLevelAccessibility) await app.getAccessibilityTree({ emit: false }) - return app + return objects.createCuaApp(appHandle, capabilities) }, } return api @@ -259,9 +258,13 @@ function createCuaObjects(call: ReturnType, sink: CuaObser return windows }, async getWindow(windowId): Promise { - const window = createCuaWindow(appHandle, windowId, capabilities) - await window.getAccessibilityTree({ emit: false }) - return window + const parsed = WindowsResult.element.shape.windowId.safeParse(windowId) + if (!parsed.success) throw new HarnessError(`getWindow ${requestIssueMessage(parsed.error.issues, ['params', 'windowId'])}`, 'INVALID_ARGS') + const windows = WindowsResult.parse(await base.call('listWindows', {})) + if (!windows.some(window => window.windowId === windowId)) { + throw new HarnessError('Selected window is unavailable for this application; read getWindows again', 'CUA_WINDOW_UNAVAILABLE') + } + return createCuaWindow(appHandle, windowId, capabilities) }, } } diff --git a/packages/experimental/cua/src/guidance.ts b/packages/experimental/cua/src/guidance.ts index 745ccfe9be..89a32bd21a 100644 --- a/packages/experimental/cua/src/guidance.ts +++ b/packages/experimental/cua/src/guidance.ts @@ -11,9 +11,9 @@ Setup and discovery: - cua.listApps(options?: { emit?: boolean }) -> { id: string, displayName: string, isRunning: true, pid: number }[] — enumerate running applications. - cua.findApps(query: string, options?: { emit?: boolean }) -> ({ id: string, displayName: string, path: string } & ({ isRunning: true, pid: number } | { isRunning: false }))[] — search installed applications, including stopped matches; only running matches carry pid. - cua.getState(options?: { emit?: boolean }) -> { apps: { id: string, displayName: string, isRunning: true, pid: number }[], browsers: [] } — summarize running applications; this is not an accessibility observation. -- cua.getApp(selector: string) -> App — select an application by name, bundle ID, absolute path, or running PID string, obtain consent, launch in the background if needed, and read initial AX state silently; it does not activate the app. +- cua.getApp(selector: string) -> App — select an application by name, bundle ID, absolute path, or running PID string, obtain consent, launch in the background if needed, and return without reading AX or activating the app. - app.getWindows(options?: { emit?: boolean }) -> { windowId: number, appName: string, title: string, bounds: { x: number, y: number, width: number, height: number }, isOnScreen: boolean, onCurrentSpace: boolean | null }[] — enumerate windows with global native bounds; null onCurrentSpace means unknown Space membership. -- app.getWindow(windowId: number) -> Window — select one window and read its initial AX state silently; do this before any window screenshot or coordinate input. +- app.getWindow(windowId: number) -> Window — select a window from the application’s current window list without reading AX; do this before any window screenshot or coordinate input. - app.activate() -> ActionResult — request explicit foreground activation; use only when foreground access is required. - app.show() -> { dispatched: boolean, windowId: number | null, foregroundChanged: boolean | null } — unhide the app and restore its selected, focused, main, or sole window without activation. - window.show() -> { dispatched: boolean, windowId: number | null, foregroundChanged: boolean | null } — unhide and restore that explicit window without activation. @@ -21,7 +21,7 @@ Setup and discovery: ## Accessibility tree Available at app or window level depending on the platform. -Element ids are opaque strings from the latest successful tree; the object facade keeps one current snapshot per app across its repeated app and window objects, a new read replaces it, and a failed read invalidates the previous one. Repeated strings can appear for different elements, so do not reuse an old id blindly, and element operations do not fall back to pointer input or typing. +Read getAccessibilityTree explicitly before using element ids. Selection and screenshots do not read or replace the current AX snapshot. Element ids are opaque strings from the latest successful tree; the object facade keeps one current snapshot per app across its repeated app and window objects, a new read replaces it, and a failed read invalidates the previous one. Repeated strings can appear for different elements, so do not reuse an old id blindly, and element operations do not fall back to pointer input or typing. - getAccessibilityTree(options?: { emit?: boolean, includeMenus?: boolean, details?: boolean }) -> string — read the full tree text; includeMenus expands menu contents and details adds identifiers, native bounds, and complete action names. - listActions(elementId: string) -> string[] — list the actions an element advertises. - invoke(elementId: string) -> ActionResult — invoke the element's default action; dispatched is false when it has none. @@ -45,4 +45,4 @@ Available at app or window level depending on the platform. - paste(text: string, options?: { format?: 'text' | 'html' }) -> ActionResult — deliver text or HTML through the app's Paste command, restoring prior clipboard contents unless another writer changes them; a Paste failure is surfaced without substituting typeText, and PASTE_NOT_CONSUMED means inspect before any retry. - pressKey(key: string, modifiers?: string[]) -> ActionResult — press and release one complete keystroke. For example pressKey('Tab', ['shift']) or pressKey('+'). Key and modifier names are platform-defined strings. Pass key and modifiers separately; do not pack them into a chord string. -Raw tools use { method, params }: include appHandle for app calls and windowId for window calls, and use snapshotId with elementId for element routes. The emit option is facade-only: emit is local output control, so raw callers must follow the raw operation schema for admitted params. getAccessibilityTree and getWindow read through raw getState; getWindows maps to listWindows; getScreenshot maps to screenshot; cua.getState maps to listApps; every other named operation uses the same method name.` +Raw tools use { method, params }: include appHandle for app calls and windowId for window calls, and use snapshotId with elementId for element routes. The emit option is facade-only: emit is local output control, so raw callers must follow the raw operation schema for admitted params. getAccessibilityTree maps to getState; getWindow validates against listWindows; getWindows maps to listWindows; getScreenshot maps to screenshot; cua.getState maps to listApps; every other named operation uses the same method name.` diff --git a/packages/experimental/cua/src/types.ts b/packages/experimental/cua/src/types.ts index b84663f9e2..0e8624b6ea 100644 --- a/packages/experimental/cua/src/types.ts +++ b/packages/experimental/cua/src/types.ts @@ -238,9 +238,9 @@ export interface CuaApp extends CuaSurface { */ getWindows(options?: CuaObservationOptions): Promise /** - * Observe a specific window. + * Select a specific window. * @param windowId - ID obtained from getWindows. - * @returns A window object with its initial accessibility tree read silently; this updates the application's current snapshot. + * @returns A window object from the app’s current window list, without reading or replacing its accessibility snapshot. */ getWindow(windowId: number): Promise } @@ -272,9 +272,9 @@ export interface CuaApi { */ getState(options?: CuaObservationOptions): Promise<{ apps: CuaAppInfo[]; browsers: [] }> /** - * Select an app, obtain consent, launch it in the background if necessary, and silently retain initial AX state. + * Select an app, obtain consent, and launch it in the background if necessary, without reading AX. * @param selector - Exact name, bundle ID, absolute application path or running PID string; ambiguous names reject. - * @returns An application handle tied to this process lifetime; its initial tree read becomes the current snapshot. + * @returns An application handle tied to this process lifetime; call getAccessibilityTree explicitly before element operations. */ getApp(selector: string): Promise } diff --git a/packages/experimental/cua/tests/api.spec.ts b/packages/experimental/cua/tests/api.spec.ts index cbaa419951..fb2bf5a204 100644 --- a/packages/experimental/cua/tests/api.spec.ts +++ b/packages/experimental/cua/tests/api.spec.ts @@ -83,7 +83,7 @@ describe('application discovery', () => { const window = await app.getWindow(23) expect(await window.getScreenshot()).toEqual({ bytes: Uint8Array.from(Buffer.from(PNG, 'base64')), mimeType: 'image/png', pixelWidth: 1, pixelHeight: 1, windowId: 23 }) expect(f.dispatch.mock.calls.map(([operation]) => operation.method)).toEqual([ - 'getCapabilities', 'listApps', 'getApp', 'getState', 'getState', 'screenshot', + 'getCapabilities', 'listApps', 'getApp', 'listWindows', 'screenshot', ]) expect(await app.getWindows()).toEqual([windowInfo(23)]) expect(f.sink.text).not.toHaveBeenCalled() @@ -124,8 +124,11 @@ describe('application discovery', () => { describe('background window restoration', () => { it('shows an app or selected window without dispatching activation or AXRaise', async () => { const f = fixture() + f.setWindows([23, 24]) const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) const window = await app.getWindow(24) + await window.getAccessibilityTree({ emit: false }) f.dispatch.mockClear() expect(await app.show()).toEqual({ ...delivered, windowId: 23, foregroundChanged: false }) expect(await window.show()).toEqual({ ...delivered, windowId: 24, foregroundChanged: false }) @@ -138,6 +141,7 @@ describe('background window restoration', () => { it.each([true, null])('preserves foreground change evidence %j without switching focus back', async (foregroundChanged) => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) f.dispatch.mockClear() f.dispatch.mockResolvedValueOnce({ ...delivered, windowId: null, foregroundChanged }) expect(await app.show()).toEqual({ ...delivered, windowId: null, foregroundChanged }) @@ -147,6 +151,7 @@ describe('background window restoration', () => { it('leaves foreground fallback to the caller after restoration fails', async () => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) f.dispatch.mockClear() f.dispatch.mockRejectedValueOnce(new Error('UNSUPPORTED: Window cannot be restored')) await expect(app.show()).rejects.toThrow('UNSUPPORTED') @@ -157,8 +162,11 @@ describe('background window restoration', () => { describe('mouseMove across observations', () => { it('moves to an AX element then into a popup without inserting clicks or restoring a point', async () => { const f = fixture() + f.setWindows([23, 24]) const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) const window = await app.getWindow(23) + await window.getAccessibilityTree({ emit: false }) f.dispatch.mockClear() expect(await window.mouseMove(element('0'))).toEqual({ ...delivered, hoverRoute: 'accessibility', systemPointerMoved: false }) await window.getAccessibilityTree({ emit: false }) @@ -171,13 +179,14 @@ describe('mouseMove across observations', () => { { method: 'mouseMove', params: { appHandle: 'app-17', snapshotId: 'snapshot-3', windowId: 23, x: 30, y: 40 } }, ]) const other = await app.getWindow(24) + await other.getAccessibilityTree({ emit: false }) await other.mouseMove([5, 6]) expect(f.dispatch).toHaveBeenLastCalledWith({ method: 'mouseMove', params: { appHandle: 'app-17', snapshotId: 'snapshot-4', windowId: 24, x: 5, y: 6, } }) f.dispatch.mockRejectedValueOnce(new Error('WINDOW_CHANGED')) await expect(other.mouseMove([5, 6])).rejects.toThrow('WINDOW_CHANGED') - expect(f.dispatch).toHaveBeenCalledTimes(7) + expect(f.dispatch).toHaveBeenCalledTimes(8) }) }) @@ -185,7 +194,9 @@ describe('pointer buttons and modifiers', () => { it.each([0, 800])('preserves %d ms for window clicks at elements and coordinates', async (holdMs) => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) const window = await app.getWindow(23) + await window.getAccessibilityTree({ emit: false }) for (const target of [element('0'), [10, 20] as const]) { await window.click(target, { holdMs, mouseButton: 'right' }) expect(f.dispatch.mock.calls.at(-1)?.[0]).toMatchObject({ method: 'click', params: { holdMs, mouseButton: 'right' } }) @@ -198,7 +209,9 @@ describe('pointer buttons and modifiers', () => { it('retains middle-button and modifier intent through click and drag dispatch', async () => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) const window = await app.getWindow(23) + await window.getAccessibilityTree({ emit: false }) await window.click(element('0'), { mouseButton: 'middle', modifiers: ['shift'] }) expect(f.dispatch).toHaveBeenLastCalledWith({ method: 'click', params: { appHandle: 'app-17', snapshotId: 'snapshot-2', elementId: '0', windowId: 23, mouseButton: 'middle', modifiers: ['shift'], @@ -223,6 +236,7 @@ describe('text selections', () => { it('binds literal text and optional disambiguation to the latest observed control', async () => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) await app.selectText(element('1'), 'target') expect(f.dispatch).toHaveBeenLastCalledWith({ method: 'selectText', params: { appHandle: 'app-17', snapshotId: 'snapshot-1', elementId: '1', text: 'target', @@ -242,6 +256,7 @@ describe('element action queries', () => { it('returns raw provider action names including attribute-setting operations without reading new state', async () => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) f.dispatch.mockClear() expect(await app.listActions(element('42.12345'))).toEqual(['AXPress', 'AXShowMenu', 'AXTextArea.AXValue.Set']) expect(f.dispatch.mock.calls.map(([operation]) => operation)).toEqual([ @@ -257,6 +272,7 @@ describe('element actions', () => { it('dispatches advertised actions with omitted, null or direct values without wrapping the value', async () => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) f.dispatch.mockClear() expect(await app.performAction(element('42.12345'), 'AXPress')).toEqual(delivered) expect(await app.performAction(element('42.12345'), 'AXPress', null)).toEqual(delivered) @@ -274,6 +290,7 @@ describe('element actions', () => { it('rejects an empty action before dispatch', async () => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) f.dispatch.mockClear() await expect(app.performAction(element('0'), '')).rejects.toThrow('performAction action') expect(f.dispatch).not.toHaveBeenCalled() @@ -284,6 +301,7 @@ describe('element default invocation', () => { it('uses the latest snapshot and opaque element id without reading fresh state', async () => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) f.dispatch.mockClear() expect(await app.invoke(element('42.12345'))).toEqual(delivered) expect(f.dispatch.mock.calls.map(([operation]) => operation)).toEqual([ @@ -295,6 +313,7 @@ describe('element default invocation', () => { it('returns confirmed absence of a default action as un-dispatched evidence', async () => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) f.dispatch.mockResolvedValueOnce({ dispatched: false }) await expect(app.invoke(element('9'))).resolves.toEqual({ dispatched: false }) }) @@ -302,6 +321,7 @@ describe('element default invocation', () => { it('rejects provider failures and malformed dispatch evidence', async () => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) const failure = new Error('PERMISSION_DENIED') f.dispatch.mockRejectedValueOnce(failure) await expect(app.invoke(element('0'))).rejects.toBe(failure) @@ -316,9 +336,12 @@ describe('application and window observations', () => { const f = fixture() f.setWindows([23, 24]) const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) expect(await app.getWindows({ emit: false })).toEqual([windowInfo(23), windowInfo(24)]) const first = await app.getWindow(23) + await first.getAccessibilityTree({ emit: false }) const second = await app.getWindow(24) + await second.getAccessibilityTree({ emit: false }) await app.setValue(element('1'), 'First document') await first.click(element('2')) await second.click(element('3')) @@ -342,9 +365,11 @@ describe('application and window observations', () => { it('keeps separate snapshots for different application handles', async () => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) f.dispatch.mockResolvedValueOnce({ appHandle: 'app-18', name: 'Utility', running: true }) f.dispatch.mockResolvedValueOnce(state({ appHandle: 'app-18', snapshotId: 'other', windows: [] })) const other = await f.api.getApp('Utility') + await other.getAccessibilityTree({ emit: false }) await app.setValue(element('0'), 'Editor') await other.setValue(element('0'), 'Utility') expect(f.dispatch.mock.calls.slice(-2).map(([operation]) => operation)).toEqual([ @@ -356,7 +381,15 @@ describe('application and window observations', () => { it('shares one current snapshot between repeated handles for the same application', async () => { const f = fixture() const first = await f.api.getApp('Editor') + await first.getAccessibilityTree({ emit: false }) const second = await f.api.getApp('Editor') + const window = await second.getWindow(23) + await window.getScreenshot({ emit: false }) + await first.setValue(element('1'), 'Preserved') + expect(f.dispatch).toHaveBeenLastCalledWith({ method: 'setValue', params: { + appHandle: 'app-17', snapshotId: 'snapshot-1', elementId: '1', value: 'Preserved', + } }) + await second.getAccessibilityTree({ emit: false }) expect(second.appHandle).toBe(first.appHandle) await first.setValue(element('1'), 'Shared') expect(f.dispatch).toHaveBeenLastCalledWith({ method: 'setValue', params: { @@ -367,9 +400,11 @@ describe('application and window observations', () => { it('exposes screenshots and pointer input only on explicit windows', async () => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) expect(app).toHaveProperty('getAccessibilityTree') for (const method of ['getScreenshot', 'click', 'mouseMove', 'scroll', 'drag']) expect(app).not.toHaveProperty(method) const window = await app.getWindow(23) + await window.getAccessibilityTree({ emit: false }) for (const method of ['getScreenshot', 'click', 'mouseMove', 'scroll', 'drag']) expect(window).toHaveProperty(method) expect(window).not.toHaveProperty('hover') expect(window).toHaveProperty('mouseMove') @@ -377,13 +412,17 @@ describe('application and window observations', () => { it('attaches the shared snapshot to coordinates only within its window scope', async () => { const f = fixture() + f.setWindows([23, 24]) const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) const first = await app.getWindow(23) + await first.getAccessibilityTree({ emit: false }) await first.click([10, 20]) expect(f.dispatch).toHaveBeenLastCalledWith({ method: 'click', params: { appHandle: 'app-17', windowId: 23, snapshotId: 'snapshot-2', x: 10, y: 20, mouseButton: 'left', clickCount: 1, } }) const second = await app.getWindow(24) + await second.getAccessibilityTree({ emit: false }) await first.click([10, 20]) expect(f.dispatch).toHaveBeenLastCalledWith({ method: 'click', params: { appHandle: 'app-17', windowId: 23, x: 10, y: 20, mouseButton: 'left', clickCount: 1, @@ -398,8 +437,10 @@ describe('application and window observations', () => { it('keeps a validated AXless window usable for screenshots and coordinate input', async () => { const f = fixture() const app = await f.api.getApp('Editor') - f.dispatch.mockResolvedValueOnce(state({ windowId: 23, snapshotId: 'axless', tree: '', elements: [] })) + await app.getAccessibilityTree({ emit: false }) const window = await app.getWindow(23) + f.dispatch.mockResolvedValueOnce(state({ windowId: 23, snapshotId: 'axless', tree: '', elements: [] })) + await window.getAccessibilityTree({ emit: false }) expect(f.sink.text).not.toHaveBeenCalled() await expect(window.getScreenshot()).resolves.toEqual({ bytes: Uint8Array.from(Buffer.from(PNG, 'base64')), mimeType: 'image/png', pixelWidth: 1, pixelHeight: 1, windowId: 23 }) await window.click([10, 20]) @@ -414,11 +455,12 @@ describe('application and window observations', () => { it.each(['WINDOW_UNAVAILABLE', 'PERMISSION_DENIED', 'STALE_APP', 'SESSION_LOCKED'])('propagates %s while selecting a window without attempting image or input fallback', async (code) => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) const failure = new Error(code) f.dispatch.mockRejectedValueOnce(failure) const calls = f.dispatch.mock.calls.length await expect(app.getWindow(23)).rejects.toBe(failure) - expect(f.dispatch.mock.calls.slice(calls).map(([operation]) => operation.method)).toEqual(['getState']) + expect(f.dispatch.mock.calls.slice(calls).map(([operation]) => operation.method)).toEqual(['listWindows']) }) it('captures and clicks the explicitly selected window among several', async () => { @@ -431,8 +473,10 @@ describe('application and window observations', () => { ] f.setWindowRecords(windows) const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) expect(await app.getWindows({ emit: false })).toEqual(windows) const window = await app.getWindow(23) + await window.getAccessibilityTree({ emit: false }) await window.getScreenshot() expect(f.dispatch).toHaveBeenLastCalledWith({ method: 'screenshot', params: { appHandle: 'app-17', windowId: 23 } }) await window.click([10, 20]) @@ -440,6 +484,7 @@ describe('application and window observations', () => { appHandle: 'app-17', windowId: 23, snapshotId: 'snapshot-2', x: 10, y: 20, mouseButton: 'left', clickCount: 1, } }) const hidden = await app.getWindow(25) + await hidden.getAccessibilityTree({ emit: false }) await hidden.click([10, 20]) expect(f.dispatch).toHaveBeenLastCalledWith({ method: 'click', params: { appHandle: 'app-17', windowId: 25, snapshotId: 'snapshot-3', x: 10, y: 20, mouseButton: 'left', clickCount: 1, @@ -471,6 +516,7 @@ describe('application and window observations', () => { it('returns provider window records without process filtering', async () => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) const observed = windowInfo(24) f.dispatch.mockResolvedValueOnce(state({ windows: [observed] })) await expect(app.getAccessibilityTree()).resolves.toBe('Changed tree') @@ -496,11 +542,11 @@ describe('application and window observations', () => { { method: 'typeText', params: { appHandle: 'app-17', text: 'keyboard without accessibility' } }, ]) const window = await app.getWindow(23) - await expect(window.getAccessibilityTree({ emit: false })).resolves.toBe('Tree 2') + await expect(window.getAccessibilityTree({ emit: false })).resolves.toBe('Tree 1') await expect(app.getAccessibilityTree()).rejects.toThrow('Application-level accessibility is not supported on current platform; observe a window instead.') await expect(window.invoke(element('0'))).resolves.toEqual(delivered) expect(f.dispatch).toHaveBeenLastCalledWith({ method: 'invoke', params: { - appHandle: 'app-17', windowId: 23, snapshotId: 'snapshot-2', elementId: '0', + appHandle: 'app-17', windowId: 23, snapshotId: 'snapshot-1', elementId: '0', } }) }) @@ -508,7 +554,9 @@ describe('application and window observations', () => { const f = fixture() f.setCapabilities({ ...capabilities, keyboard: ['app', 'window'] }) const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) const window = await app.getWindow(23) + await window.getAccessibilityTree({ emit: false }) f.dispatch.mockClear() await window.typeText('hello') await window.paste('bold', { format: 'html' }) @@ -530,7 +578,9 @@ describe('application and window observations', () => { const f = fixture() f.setCapabilities({ ...capabilities, keyboard: ['window'] }) const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) const window = await app.getWindow(23) + await window.getAccessibilityTree({ emit: false }) f.dispatch.mockClear() await expect(app.typeText('app text')).rejects.toThrow('App-scoped keyboard input is not supported on current platform.') await expect(app.paste('app text')).rejects.toThrow('App-scoped keyboard input is not supported on current platform.') @@ -544,7 +594,9 @@ describe('application and window observations', () => { it('accepts coordinate input without a current snapshot', async () => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) const window = await app.getWindow(23) + await window.getAccessibilityTree({ emit: false }) f.dispatch.mockRejectedValueOnce(new Error('Native app is busy')) await expect(window.getAccessibilityTree()).rejects.toThrow('Native app is busy') await window.click([10, 20]) @@ -556,7 +608,9 @@ describe('application and window observations', () => { it('clears a failed refresh and requires a new read before element operations', async () => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) const window = await app.getWindow(23) + await window.getAccessibilityTree({ emit: false }) const failure = new Error('Native app is busy') f.dispatch.mockRejectedValueOnce(failure) await expect(app.getAccessibilityTree()).rejects.toBe(failure) @@ -575,11 +629,54 @@ describe('application and window observations', () => { }) describe('screenshots', () => { + it('rejects a missing or foreign window without reading AX or taking a screenshot', async () => { + const f = fixture() + const app = await f.api.getApp('Editor') + await expect(app.getWindow(24)).rejects.toMatchObject({ name: 'HarnessError', code: 'CUA_WINDOW_UNAVAILABLE' }) + f.setWindows([]) + await expect(app.getWindow(23)).rejects.toMatchObject({ name: 'HarnessError', code: 'CUA_WINDOW_UNAVAILABLE' }) + expect(f.dispatch.mock.calls.map(([request]) => request.method)).toEqual(['getCapabilities', 'getApp', 'listWindows', 'listWindows']) + await expect(app.getWindow(0)).rejects.toMatchObject({ name: 'HarnessError', code: 'INVALID_ARGS', message: 'getWindow windowId: Too small: expected number to be >0' }) + expect(f.dispatch).toHaveBeenCalledTimes(4) + }) + + it.each(['APP_ACCESS_REVOKED', 'WINDOW_UNAVAILABLE', 'STALE_APP'])('preserves %s when capturing a previously selected window', async (code) => { + const f = fixture() + const app = await f.api.getApp('Editor') + const window = await app.getWindow(23) + f.dispatch.mockRejectedValueOnce(new Error(code)) + await expect(window.getScreenshot()).rejects.toThrow(code) + expect(f.sink.image).not.toHaveBeenCalled() + }) + + it.each(['application', 'window'])('captures the first screenshot when %s AX reads fail', async (scope) => { + const f = fixture() + const dispatch = f.dispatch.getMockImplementation()! + const failure = new Error('AX read failed') + f.dispatch.mockImplementation(async (request) => { + if (request.method === 'getState' && (scope === 'application' ? request.params.windowId === undefined : request.params.windowId !== undefined)) throw failure + return dispatch(request) + }) + const app = await f.api.getApp('Editor') + expect(await app.getWindows({ emit: false })).toEqual([windowInfo(23)]) + const window = await app.getWindow(23) + const target = scope === 'application' ? app : window + await expect(target.getAccessibilityTree()).rejects.toBe(failure) + await expect(window.getScreenshot()).resolves.toMatchObject({ windowId: 23, pixelWidth: 1 }) + await expect(window.invoke(element('0'))).rejects.toThrow('Read getAccessibilityTree') + await window.click([0, 0]) + expect(f.dispatch).toHaveBeenLastCalledWith({ method: 'click', params: { + appHandle: 'app-17', windowId: 23, x: 0, y: 0, mouseButton: 'left', clickCount: 1, + } }) + }) + it('returns screenshot metadata independently of input point bounds and supports selective output', async () => { const f = fixture() f.setWindows([23, 24]) const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) const window = await app.getWindow(24) + await window.getAccessibilityTree({ emit: false }) f.sink.text.mockClear() const tree = await window.getAccessibilityTree() const image = await window.getScreenshot() @@ -597,7 +694,9 @@ describe('screenshots', () => { it('rejects a screenshot from another window before emitting its image', async () => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) const window = await app.getWindow(23) + await window.getAccessibilityTree({ emit: false }) f.dispatch.mockResolvedValueOnce(screenshot({ windowId: 24 })) await expect(window.getScreenshot()).rejects.toThrow('belongs to another window') expect(f.sink.image).not.toHaveBeenCalled() @@ -606,7 +705,9 @@ describe('screenshots', () => { it.each(['image/jpeg', 'image/webp', 'image/gif'])('preserves Host-admitted %s bytes and metadata in the return value and image sink', async (mimeType) => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) const window = await app.getWindow(23) + await window.getAccessibilityTree({ emit: false }) const data = Buffer.from('admitted request image').toString('base64') f.dispatch.mockResolvedValueOnce(screenshot({ mimeType, data, pixelWidth: 320, pixelHeight: 240 })) const image = await window.getScreenshot() @@ -620,7 +721,9 @@ describe('input actions', () => { it('passes explicit relative mode to every coordinate gesture and rejects it on element ids', async () => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) const window = await app.getWindow(23) + await window.getAccessibilityTree({ emit: false }) await window.click([500, 250], { relative: true }) await window.mouseMove([500, 250], { relative: true }) await window.drag([0, 0], [999, 999], { relative: true }) @@ -638,7 +741,9 @@ describe('input actions', () => { it('keeps the selected window when scroll options include another target identity', async () => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) const window = await app.getWindow(23) + await window.getAccessibilityTree({ emit: false }) const options = { x: 10, y: 20, deltaLinesX: 0, deltaLinesY: 1, appHandle: 'another-app', snapshotId: 'another-snapshot', windowId: 24, @@ -654,7 +759,9 @@ describe('input actions', () => { it('dispatches window-point gestures and reports delivery without claiming the action succeeded', async () => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) const window = await app.getWindow(23) + await window.getAccessibilityTree({ emit: false }) expect(await window.click([10, 20], { mouseButton: 'right' })).toEqual(delivered) expect(await window.click([11, 21], { clickCount: 2 })).toEqual(delivered) expect(await window.scroll({ x: 10, y: 20, deltaLinesX: -2, deltaLinesY: 3 })).toEqual(delivered) @@ -685,7 +792,6 @@ describe('input actions', () => { ['empty key', ({ app }) => app.pressKey('')], ] it.each<[string, (surfaces: Surfaces) => Promise]>([ - ['getState windowId: Too small: expected number to be >0', ({ app }) => app.getWindow(0)], ['selectText text: Too small: expected string to have >=1 characters', ({ app }) => app.selectText(element('0'), '')], ['mouseMove x: Invalid input: expected number, received NaN', ({ window }) => window.mouseMove([NaN, 0])], ['mouseMove x: Too big: expected number to be <=999', ({ window }) => window.mouseMove([1000, 0], { relative: true })], @@ -693,7 +799,9 @@ describe('input actions', () => { ])('reports a readable validation failure: %s', async (message, run) => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) const window = await app.getWindow(23) + await window.getAccessibilityTree({ emit: false }) f.dispatch.mockClear() await expect(run({ app, window })).rejects.toMatchObject({ name: 'Error', message }) expect(f.dispatch).not.toHaveBeenCalled() @@ -702,7 +810,9 @@ describe('input actions', () => { it.each(invalidActions)('rejects %s without invoking native input', async (_label, run) => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) const window = await app.getWindow(23) + await window.getAccessibilityTree({ emit: false }) const calls = f.dispatch.mock.calls.length await expect(run({ app, window })).rejects.toThrow() expect(f.dispatch).toHaveBeenCalledTimes(calls) @@ -716,6 +826,7 @@ describe('input actions', () => { ])('forwards literal key %s with modifiers %j without splitting', async (key, modifiers, expected) => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) if (modifiers === undefined) await app.pressKey(key) else await app.pressKey(key, modifiers) expect(f.dispatch).toHaveBeenLastCalledWith({ method: 'pressKey', params: { appHandle: 'app-17', key, modifiers: expected } }) @@ -724,6 +835,7 @@ describe('input actions', () => { it('forwards the literal key unsplit in one RPC without mutating modifiers', async () => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) const modifiers: string[] = ['control', 'shift'] f.dispatch.mockClear() await app.pressKey('+', modifiers) @@ -737,6 +849,7 @@ describe('input actions', () => { it('forwards arbitrary provider-defined key and modifier names in one RPC', async () => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) f.dispatch.mockClear() await app.pressKey('PlatformKey', ['platform-modifier']) expect(f.dispatch).toHaveBeenCalledTimes(1) @@ -748,6 +861,7 @@ describe('input actions', () => { it('passes literal text and keys to the selected app identity', async () => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) await app.typeText('中文\nline two') await app.paste('粘贴\nplain text') await app.paste('bold', { format: 'html' }) @@ -765,7 +879,9 @@ describe('input actions', () => { it('rejects keyboard input on a window surface without dispatching an app-level fallback', async () => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) const window = await app.getWindow(23) + await window.getAccessibilityTree({ emit: false }) f.dispatch.mockClear() for (const run of [ () => window.typeText('hello'), @@ -787,6 +903,7 @@ describe('input actions', () => { const f = fixture() f.setCapabilities({ ...capabilities, keyboard: ['app', 'window'] }) const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) const surface = scope === 'window' ? await app.getWindow(23) : app f.dispatch.mockClear() f.dispatch.mockResolvedValueOnce(delivered) @@ -800,6 +917,7 @@ describe('input actions', () => { it('returns an un-dispatched key-event result from one RPC without retrying', async () => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) f.dispatch.mockClear() f.dispatch.mockResolvedValueOnce({ dispatched: false }) await expect(app.sendKeyEvent('Command', 'down')).resolves.toEqual({ dispatched: false }) @@ -829,7 +947,9 @@ describe('input actions', () => { it('retains the snapshot identity for element right and double clicks', async () => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) const window = await app.getWindow(23) + await window.getAccessibilityTree({ emit: false }) await window.click(element('12'), { mouseButton: 'right' }) await window.click(element('13'), { clickCount: 2 }) expect(f.dispatch.mock.calls.slice(-2).map(([operation]) => operation)).toEqual([ @@ -841,7 +961,9 @@ describe('input actions', () => { it('dispatches an opaque provider element id unchanged', async () => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) const window = await app.getWindow(23) + await window.getAccessibilityTree({ emit: false }) await window.click(element('42.12345')) await window.mouseMove(element('42.12345')) expect(f.dispatch.mock.calls.slice(-2).map(([operation]) => operation)).toEqual([ @@ -853,7 +975,7 @@ describe('input actions', () => { it('does not activate on selection and dispatches only an explicit activation request', async () => { const f = fixture() const app = await f.api.getApp('Editor') - expect(f.dispatch.mock.calls.map(([call]) => call.method)).toEqual(['getCapabilities', 'getApp', 'getState']) + expect(f.dispatch.mock.calls.map(([call]) => call.method)).toEqual(['getCapabilities', 'getApp']) await app.activate() expect(f.dispatch).toHaveBeenLastCalledWith({ method: 'activate', params: { appHandle: 'app-17' } }) }) @@ -861,14 +983,16 @@ describe('input actions', () => { it('propagates native failures once and rejects malformed delivery claims', async () => { const f = fixture() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) const window = await app.getWindow(23) + await window.getAccessibilityTree({ emit: false }) const failure = new Error('Application was restarted; select it again') f.dispatch.mockRejectedValueOnce(failure) await expect(app.typeText('hello')).rejects.toBe(failure) - expect(f.dispatch).toHaveBeenCalledTimes(5) + expect(f.dispatch).toHaveBeenCalledTimes(6) f.dispatch.mockResolvedValueOnce({ dispatched: 'yes' }) await expect(window.click(element('0'))).rejects.toThrow() - expect(f.dispatch).toHaveBeenCalledTimes(6) + expect(f.dispatch).toHaveBeenCalledTimes(7) }) it('rejects invalid selection inputs before dispatch', async () => { @@ -876,6 +1000,7 @@ describe('input actions', () => { await expect(f.api.getApp('')).rejects.toThrow() expect(f.dispatch).not.toHaveBeenCalled() const app = await f.api.getApp('Editor') + await app.getAccessibilityTree({ emit: false }) const calls = f.dispatch.mock.calls.length await expect(app.getWindow(0)).rejects.toThrow() expect(f.dispatch).toHaveBeenCalledTimes(calls) diff --git a/packages/experimental/cua/tests/bridge.spec.ts b/packages/experimental/cua/tests/bridge.spec.ts index 14a36baaba..7bba3c4e0b 100644 --- a/packages/experimental/cua/tests/bridge.spec.ts +++ b/packages/experimental/cua/tests/bridge.spec.ts @@ -27,6 +27,7 @@ it('uses the current tool binding when a retained app is called by another execu const first = vi.fn(async (operation: CuaOperation): Promise => { if (operation.method === 'getCapabilities') return capabilities if (operation.method === 'getApp') return selected + if (operation.method === 'listWindows') return snapshot.windows if (operation.method === 'getState') return { ...snapshot, windowId: operation.params.windowId ?? null } return snapshot }) @@ -35,6 +36,7 @@ it('uses the current tool binding when a retained app is called by another execu const cua = connectCua(() => current, text) const app = await cua.getApp('Editor') const window = await app.getWindow(23) + await window.getAccessibilityTree({ emit: false }) const second = vi.fn(async (): Promise => ({ dispatched: true })) current = second @@ -69,6 +71,7 @@ it('returns screenshot bytes and metadata with one image-bearing tool call and n const tool = vi.fn(async (operation: CuaOperation): Promise => { if (operation.method === 'getCapabilities') return capabilities if (operation.method === 'getApp') return selected + if (operation.method === 'listWindows') return snapshot.windows if (operation.method === 'getState') return { ...snapshot, windowId: operation.params.windowId ?? null } return { mimeType: 'image/png', data: PNG, pixelWidth: 1, pixelHeight: 1, windowId: 23, bounds, coordinateSpace: 'screenshot-pixels', @@ -80,6 +83,6 @@ it('returns screenshot bytes and metadata with one image-bearing tool call and n text.mockClear() expect(await window.getScreenshot()).toEqual({ bytes: Uint8Array.from(Buffer.from(PNG, 'base64')), mimeType: 'image/png', pixelWidth: 1, pixelHeight: 1, windowId: 23 }) - expect(tool.mock.calls.map(([operation]) => operation.method)).toEqual(['getCapabilities', 'getApp', 'getState', 'getState', 'screenshot']) + expect(tool.mock.calls.map(([operation]) => operation.method)).toEqual(['getCapabilities', 'getApp', 'listWindows', 'screenshot']) expect(text).not.toHaveBeenCalled() }) diff --git a/packages/experimental/cua/tsconfig.host.json b/packages/experimental/cua/tsconfig.host.json index 2386f83640..4bf0b7ddda 100644 --- a/packages/experimental/cua/tsconfig.host.json +++ b/packages/experimental/cua/tsconfig.host.json @@ -74,6 +74,9 @@ }, { "path": "../../boot/config-editor" + }, + { + "path": "../../util/error" } ], "exclude": [ diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index d016fe8deb..83d9bd763d 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -9800,6 +9800,9 @@ importers: '@deepseek-ai/dsh-computer-use': specifier: workspace:* version: link:../../computer-use/computer-use + '@deepseek-ai/dsh-error': + specifier: workspace:* + version: link:../../util/error '@deepseek-ai/dsh-storage-domain': specifier: workspace:* version: link:../../storage/storage-domain diff --git a/snapshots/session/cua-native/system-prompt.expected.md b/snapshots/session/cua-native/system-prompt.expected.md index 0b9c1b29f6..04fb520f49 100644 --- a/snapshots/session/cua-native/system-prompt.expected.md +++ b/snapshots/session/cua-native/system-prompt.expected.md @@ -46,9 +46,9 @@ Setup and discovery: - cua.listApps(options?: { emit?: boolean }) -> { id: string, displayName: string, isRunning: true, pid: number }[] — enumerate running applications. - cua.findApps(query: string, options?: { emit?: boolean }) -> ({ id: string, displayName: string, path: string } & ({ isRunning: true, pid: number } | { isRunning: false }))[] — search installed applications, including stopped matches; only running matches carry pid. - cua.getState(options?: { emit?: boolean }) -> { apps: { id: string, displayName: string, isRunning: true, pid: number }[], browsers: [] } — summarize running applications; this is not an accessibility observation. -- cua.getApp(selector: string) -> App — select an application by name, bundle ID, absolute path, or running PID string, obtain consent, launch in the background if needed, and read initial AX state silently; it does not activate the app. +- cua.getApp(selector: string) -> App — select an application by name, bundle ID, absolute path, or running PID string, obtain consent, launch in the background if needed, and return without reading AX or activating the app. - app.getWindows(options?: { emit?: boolean }) -> { windowId: number, appName: string, title: string, bounds: { x: number, y: number, width: number, height: number }, isOnScreen: boolean, onCurrentSpace: boolean | null }[] — enumerate windows with global native bounds; null onCurrentSpace means unknown Space membership. -- app.getWindow(windowId: number) -> Window — select one window and read its initial AX state silently; do this before any window screenshot or coordinate input. +- app.getWindow(windowId: number) -> Window — select a window from the application’s current window list without reading AX; do this before any window screenshot or coordinate input. - app.activate() -> ActionResult — request explicit foreground activation; use only when foreground access is required. - app.show() -> { dispatched: boolean, windowId: number | null, foregroundChanged: boolean | null } — unhide the app and restore its selected, focused, main, or sole window without activation. - window.show() -> { dispatched: boolean, windowId: number | null, foregroundChanged: boolean | null } — unhide and restore that explicit window without activation. @@ -56,7 +56,7 @@ Setup and discovery: ## Accessibility tree Available at app or window level depending on the platform. -Element ids are opaque strings from the latest successful tree; the object facade keeps one current snapshot per app across its repeated app and window objects, a new read replaces it, and a failed read invalidates the previous one. Repeated strings can appear for different elements, so do not reuse an old id blindly, and element operations do not fall back to pointer input or typing. +Read getAccessibilityTree explicitly before using element ids. Selection and screenshots do not read or replace the current AX snapshot. Element ids are opaque strings from the latest successful tree; the object facade keeps one current snapshot per app across its repeated app and window objects, a new read replaces it, and a failed read invalidates the previous one. Repeated strings can appear for different elements, so do not reuse an old id blindly, and element operations do not fall back to pointer input or typing. - getAccessibilityTree(options?: { emit?: boolean, includeMenus?: boolean, details?: boolean }) -> string — read the full tree text; includeMenus expands menu contents and details adds identifiers, native bounds, and complete action names. - listActions(elementId: string) -> string[] — list the actions an element advertises. - invoke(elementId: string) -> ActionResult — invoke the element's default action; dispatched is false when it has none. @@ -80,6 +80,6 @@ Available at app or window level depending on the platform. - paste(text: string, options?: { format?: 'text' | 'html' }) -> ActionResult — deliver text or HTML through the app's Paste command, restoring prior clipboard contents unless another writer changes them; a Paste failure is surfaced without substituting typeText, and PASTE_NOT_CONSUMED means inspect before any retry. - pressKey(key: string, modifiers?: string[]) -> ActionResult — press and release one complete keystroke. For example pressKey('Tab', ['shift']) or pressKey('+'). Key and modifier names are platform-defined strings. Pass key and modifiers separately; do not pack them into a chord string. -Raw tools use { method, params }: include appHandle for app calls and windowId for window calls, and use snapshotId with elementId for element routes. The emit option is facade-only: emit is local output control, so raw callers must follow the raw operation schema for admitted params. getAccessibilityTree and getWindow read through raw getState; getWindows maps to listWindows; getScreenshot maps to screenshot; cua.getState maps to listApps; every other named operation uses the same method name. +Raw tools use { method, params }: include appHandle for app calls and windowId for window calls, and use snapshotId with elementId for element routes. The emit option is facade-only: emit is local output control, so raw callers must follow the raw operation schema for admitted params. getAccessibilityTree maps to getState; getWindow validates against listWindows; getWindows maps to listWindows; getScreenshot maps to screenshot; cua.getState maps to listApps; every other named operation uses the same method name. Maximum click holdMs: 5000 ms. diff --git a/snapshots/session/cua-native/workspace/native-helper.mjs b/snapshots/session/cua-native/workspace/native-helper.mjs index 1e148d7ab5..c39f62f769 100755 --- a/snapshots/session/cua-native/workspace/native-helper.mjs +++ b/snapshots/session/cua-native/workspace/native-helper.mjs @@ -162,6 +162,7 @@ function createApplicationState(socket, record, application, captureBroker, conn if (method === 'listWindows') return [window, axlessWindow] if (method === 'getState') { snapshots.delete(params.windowId ?? null) + if (existsSync(new URL('./fixture-ax-failure', import.meta.url))) refuse('NATIVE_ERROR', 'Fixture accessibility read failed') if (observationChanged) { observationChanged = false refuse('STALE_SNAPSHOT', 'Accessibility tree changed during observation; read fresh getAccessibilityTree') diff --git a/snapshots/session/cua-ptc/consent-fixture.mjs b/snapshots/session/cua-ptc/consent-fixture.mjs index 85e83bcf69..6fdbdbbfc2 100644 --- a/snapshots/session/cua-ptc/consent-fixture.mjs +++ b/snapshots/session/cua-ptc/consent-fixture.mjs @@ -1,5 +1,5 @@ /** Own a private native substitute for the PTC result replay. */ -import { chmod, copyFile, unlink } from 'node:fs/promises' +import { chmod, copyFile, unlink, writeFile } from 'node:fs/promises' import { join } from 'node:path' import { startFixtureHost } from '../cua-native/fixture-host.mjs' export const name = 'cua-ptc-fixture' @@ -7,6 +7,10 @@ export const inject = ['userQuestions'] /** Prepare the helper and retire its service before removing the private executable. */ export async function apply(ctx) { + const bridge = join(process.cwd(), 'cua-bridge.mjs') + const face = process.env.DSH_EXAMPLE_MODE === 'src' ? 'src/bridge.ts' : 'lib/bridge.js' + await writeFile(bridge, `export { connectCua } from ${JSON.stringify(new URL('../../../packages/experimental/cua/' + face, import.meta.url).href)}\n`) + ctx.effect(() => () => unlink(bridge)) const helper = join(process.cwd(), 'native-helper.mjs') await copyFile(new URL('../cua-native/workspace/native-helper.mjs', import.meta.url), helper) await chmod(helper, 0o700) diff --git a/snapshots/session/cua-ptc/replay.override.json b/snapshots/session/cua-ptc/replay.override.json index 63e4009990..5ea62fee03 100644 --- a/snapshots/session/cua-ptc/replay.override.json +++ b/snapshots/session/cua-ptc/replay.override.json @@ -32,6 +32,39 @@ } ] }, + { + "kind": "chunks", + "chunks": [ + { + "type": "block-start", + "index": 0, + "blockType": "tool-call" + }, + { + "type": "block-end", + "index": 0, + "block": { + "type": "tool-call", + "id": "cua-ptc-ax-failure", + "name": "run_code", + "arguments": "{\"code\":\"const { writeFile, unlink } = await import('node:fs/promises');\\nconst { resolve } = await import('node:path');\\nconst { pathToFileURL } = await import('node:url');\\nconst { connectCua } = await import(pathToFileURL(resolve('cua-bridge.mjs')).href);\\nawait writeFile('fixture-ax-failure', '');\\ntry {\\n const cua = connectCua(() => tools.cua);\\n const app = await cua.getApp('test.fixture');\\n const windows = await app.getWindows({ emit: false });\\n const window = await app.getWindow(windows[0].windowId);\\n const failures = [];\\n for (const surface of [app, window]) {\\n try { await surface.getAccessibilityTree(); throw new Error('Expected AX failure'); }\\n catch (error) { if (!error.message.includes('Fixture accessibility read failed')) throw error; failures.push(error.message); }\\n }\\n const shot = await window.getScreenshot();\\n let elementRefused = false;\\n try { await window.invoke('0'); }\\n catch (error) { if (!error.message.includes('Read getAccessibilityTree')) throw error; elementRefused = true; }\\n if (!elementRefused) throw new Error('Element input accepted without an AX snapshot');\\n return { axFailures: failures.length, screenshotWindow: shot.windowId, pngBytes: shot.bytes.length, elementRefused };\\n} finally { await unlink('fixture-ax-failure'); }\",\"description\":\"Capture through the object API despite application and window AX failures\"}" + } + }, + { + "type": "usage", + "usage": { + "inputTokens": 3, + "outputTokens": 3 + } + }, + { + "type": "finish", + "reason": { + "kind": "tool-calls" + } + } + ] + }, { "kind": "chunks", "chunks": [ @@ -45,7 +78,7 @@ "index": 0, "block": { "type": "text", - "text": "Received native CUA values and one PNG screenshot through PTC. The PNG is 90 bytes with signature 89504e470d0a1a0a." + "text": "Received native CUA values and PNG screenshots through PTC. The object API captured a 90-byte PNG after two AX read failures and refused element input without an accessibility snapshot." } }, { diff --git a/snapshots/session/cua-ptc/session.v4.jsonl b/snapshots/session/cua-ptc/session.v4.jsonl index 87791413b2..f9d25d914b 100644 --- a/snapshots/session/cua-ptc/session.v4.jsonl +++ b/snapshots/session/cua-ptc/session.v4.jsonl @@ -12,7 +12,7 @@ {"type":"request/header","data":{"header":{"config":{"provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"tools":"{{tools}}"},"reason":"initial"}} {"type":"request/context","data":{"provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"}} {"type":"session/title","data":{"title":"Use tools.cua from Node PTC","messageSeqs":[8],"source":{"kind":"fallback"}}} -{"type":"assistant/message","data":{"turn":1,"step":1,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-ptc-native-result","name":"run_code","arguments":"{\"code\":\"const { Buffer } = await import('node:buffer');\\nconst app = await tools.cua({ method: 'getApp', params: { selector: 'test.fixture' } });\\nconst state = await tools.cua({ method: 'getState', params: { appHandle: app.appHandle, windowId: 7 } });\\nconst shot = await tools.cua({ method: 'screenshot', params: { appHandle: app.appHandle, windowId: 7 } });\\nconst bytes = Buffer.from(shot.data, 'base64');\\nreturn { appHandle: app.appHandle, hasIncrement: state.tree.includes('Increment'), pngBytes: bytes.length, signature: bytes.subarray(0, 8).toString('hex'), imageWidth: shot.attachment.width, imageHeight: shot.attachment.height };\",\"description\":\"Read native CUA values and receive one screenshot\"}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:4}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791864698727,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791864698727,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-ptc-native-result","name":"run_code","arguments":"{\"code\":\"const { Buffer } = await import('node:buffer');\\nconst app = await tools.cua({ method: 'getApp', params: { selector: 'test.fixture' } });\\nconst state = await tools.cua({ method: 'getState', params: { appHandle: app.appHandle, windowId: 7 } });\\nconst shot = await tools.cua({ method: 'screenshot', params: { appHandle: app.appHandle, windowId: 7 } });\\nconst bytes = Buffer.from(shot.data, 'base64');\\nreturn { appHandle: app.appHandle, hasIncrement: state.tree.includes('Increment'), pngBytes: bytes.length, signature: bytes.subarray(0, 8).toString('hex'), imageWidth: shot.attachment.width, imageHeight: shot.attachment.height };\",\"description\":\"Read native CUA values and receive one screenshot\"}"}}},{"type":"chunk","time":1791864698727,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791864698727,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} +{"type":"assistant/message","data":{"turn":1,"step":1,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-ptc-native-result","name":"run_code","arguments":"{\"code\":\"const { Buffer } = await import('node:buffer');\\nconst app = await tools.cua({ method: 'getApp', params: { selector: 'test.fixture' } });\\nconst state = await tools.cua({ method: 'getState', params: { appHandle: app.appHandle, windowId: 7 } });\\nconst shot = await tools.cua({ method: 'screenshot', params: { appHandle: app.appHandle, windowId: 7 } });\\nconst bytes = Buffer.from(shot.data, 'base64');\\nreturn { appHandle: app.appHandle, hasIncrement: state.tree.includes('Increment'), pngBytes: bytes.length, signature: bytes.subarray(0, 8).toString('hex'), imageWidth: shot.attachment.width, imageHeight: shot.attachment.height };\",\"description\":\"Read native CUA values and receive one screenshot\"}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:4}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1794024973829,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1794024973829,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-ptc-native-result","name":"run_code","arguments":"{\"code\":\"const { Buffer } = await import('node:buffer');\\nconst app = await tools.cua({ method: 'getApp', params: { selector: 'test.fixture' } });\\nconst state = await tools.cua({ method: 'getState', params: { appHandle: app.appHandle, windowId: 7 } });\\nconst shot = await tools.cua({ method: 'screenshot', params: { appHandle: app.appHandle, windowId: 7 } });\\nconst bytes = Buffer.from(shot.data, 'base64');\\nreturn { appHandle: app.appHandle, hasIncrement: state.tree.includes('Increment'), pngBytes: bytes.length, signature: bytes.subarray(0, 8).toString('hex'), imageWidth: shot.attachment.width, imageHeight: shot.attachment.height };\",\"description\":\"Read native CUA values and receive one screenshot\"}"}}},{"type":"chunk","time":1794024973829,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1794024973829,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} {"type":"tool/call","data":{"turn":1,"step":1,"callId":"cua-ptc-native-result","name":"run_code","arguments":"{\"code\":\"const { Buffer } = await import('node:buffer');\\nconst app = await tools.cua({ method: 'getApp', params: { selector: 'test.fixture' } });\\nconst state = await tools.cua({ method: 'getState', params: { appHandle: app.appHandle, windowId: 7 } });\\nconst shot = await tools.cua({ method: 'screenshot', params: { appHandle: app.appHandle, windowId: 7 } });\\nconst bytes = Buffer.from(shot.data, 'base64');\\nreturn { appHandle: app.appHandle, hasIncrement: state.tree.includes('Increment'), pngBytes: bytes.length, signature: bytes.subarray(0, 8).toString('hex'), imageWidth: shot.attachment.width, imageHeight: shot.attachment.height };\",\"description\":\"Read native CUA values and receive one screenshot\"}"}} {"type":"tool/ptc-dispatch-start","data":{"rootCallId":"cua-ptc-native-result","parentCallId":"cua-ptc-native-result","subCallId":"cua-ptc-native-result:ptc:1","name":"cua","arguments":{"method":"getApp","params":{"selector":"test.fixture"}}}} {"type":"tool/ptc-dispatch","data":{"rootCallId":"cua-ptc-native-result","parentCallId":"cua-ptc-native-result","subCallId":"cua-ptc-native-result:ptc:1","name":"cua","arguments":{"method":"getApp","params":{"selector":"test.fixture"}},"isError":false,"content":[{"type":"text","text":"{\"appHandle\":\"fixture-app\",\"name\":\"Fixture\",\"bundleId\":\"test.fixture\",\"running\":true}"}]}} @@ -26,6 +26,28 @@ {"type":"agent/inbox/spliced","data":{"target":"next-step","start":0,"removedCount":1,"inserted":[]}} {"type":"step/start","data":{"turn":1,"step":2}} {"type":"user/message","data":{"content":[{"type":"text","text":"{\"pixelWidth\":1,\"pixelHeight\":1,\"windowId\":7,\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"coordinateSpace\":\"screenshot-pixels\"}"},{"type":"image","attachment":{"attachmentId":"sha256:999f1d1527ee7e79266f16add5430fff76b1225d742464a5b1ff1f02971bb8ee","mediaType":"image/png","width":1,"height":1,"bytes":90}}],"source":{"kind":"ptc-mode"},"role":"user","id":"{{message:6}}"},"surfaceOp":"append"} -{"type":"assistant/message","data":{"turn":1,"step":2,"message":{"role":"assistant","content":[{"type":"text","text":"Received native CUA values and one PNG screenshot through PTC. The PNG is 90 bytes with signature 89504e470d0a1a0a."}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:7}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791864699463,"chunk":{"type":"block-start","index":0,"blockType":"text"}},{"type":"chunk","time":1791864699463,"chunk":{"type":"block-end","index":0,"block":{"type":"text","text":"Received native CUA values and one PNG screenshot through PTC. The PNG is 90 bytes with signature 89504e470d0a1a0a."}}},{"type":"chunk","time":1791864699463,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791864699463,"chunk":{"type":"finish","reason":{"kind":"stop"}}}]},"surfaceOp":"append"} +{"type":"assistant/message","data":{"turn":1,"step":2,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-ptc-ax-failure","name":"run_code","arguments":"{\"code\":\"const { writeFile, unlink } = await import('node:fs/promises');\\nconst { resolve } = await import('node:path');\\nconst { pathToFileURL } = await import('node:url');\\nconst { connectCua } = await import(pathToFileURL(resolve('cua-bridge.mjs')).href);\\nawait writeFile('fixture-ax-failure', '');\\ntry {\\n const cua = connectCua(() => tools.cua);\\n const app = await cua.getApp('test.fixture');\\n const windows = await app.getWindows({ emit: false });\\n const window = await app.getWindow(windows[0].windowId);\\n const failures = [];\\n for (const surface of [app, window]) {\\n try { await surface.getAccessibilityTree(); throw new Error('Expected AX failure'); }\\n catch (error) { if (!error.message.includes('Fixture accessibility read failed')) throw error; failures.push(error.message); }\\n }\\n const shot = await window.getScreenshot();\\n let elementRefused = false;\\n try { await window.invoke('0'); }\\n catch (error) { if (!error.message.includes('Read getAccessibilityTree')) throw error; elementRefused = true; }\\n if (!elementRefused) throw new Error('Element input accepted without an AX snapshot');\\n return { axFailures: failures.length, screenshotWindow: shot.windowId, pngBytes: shot.bytes.length, elementRefused };\\n} finally { await unlink('fixture-ax-failure'); }\",\"description\":\"Capture through the object API despite application and window AX failures\"}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:7}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1794024974646,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1794024974646,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-ptc-ax-failure","name":"run_code","arguments":"{\"code\":\"const { writeFile, unlink } = await import('node:fs/promises');\\nconst { resolve } = await import('node:path');\\nconst { pathToFileURL } = await import('node:url');\\nconst { connectCua } = await import(pathToFileURL(resolve('cua-bridge.mjs')).href);\\nawait writeFile('fixture-ax-failure', '');\\ntry {\\n const cua = connectCua(() => tools.cua);\\n const app = await cua.getApp('test.fixture');\\n const windows = await app.getWindows({ emit: false });\\n const window = await app.getWindow(windows[0].windowId);\\n const failures = [];\\n for (const surface of [app, window]) {\\n try { await surface.getAccessibilityTree(); throw new Error('Expected AX failure'); }\\n catch (error) { if (!error.message.includes('Fixture accessibility read failed')) throw error; failures.push(error.message); }\\n }\\n const shot = await window.getScreenshot();\\n let elementRefused = false;\\n try { await window.invoke('0'); }\\n catch (error) { if (!error.message.includes('Read getAccessibilityTree')) throw error; elementRefused = true; }\\n if (!elementRefused) throw new Error('Element input accepted without an AX snapshot');\\n return { axFailures: failures.length, screenshotWindow: shot.windowId, pngBytes: shot.bytes.length, elementRefused };\\n} finally { await unlink('fixture-ax-failure'); }\",\"description\":\"Capture through the object API despite application and window AX failures\"}"}}},{"type":"chunk","time":1794024974646,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1794024974646,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} +{"type":"tool/call","data":{"turn":1,"step":2,"callId":"cua-ptc-ax-failure","name":"run_code","arguments":"{\"code\":\"const { writeFile, unlink } = await import('node:fs/promises');\\nconst { resolve } = await import('node:path');\\nconst { pathToFileURL } = await import('node:url');\\nconst { connectCua } = await import(pathToFileURL(resolve('cua-bridge.mjs')).href);\\nawait writeFile('fixture-ax-failure', '');\\ntry {\\n const cua = connectCua(() => tools.cua);\\n const app = await cua.getApp('test.fixture');\\n const windows = await app.getWindows({ emit: false });\\n const window = await app.getWindow(windows[0].windowId);\\n const failures = [];\\n for (const surface of [app, window]) {\\n try { await surface.getAccessibilityTree(); throw new Error('Expected AX failure'); }\\n catch (error) { if (!error.message.includes('Fixture accessibility read failed')) throw error; failures.push(error.message); }\\n }\\n const shot = await window.getScreenshot();\\n let elementRefused = false;\\n try { await window.invoke('0'); }\\n catch (error) { if (!error.message.includes('Read getAccessibilityTree')) throw error; elementRefused = true; }\\n if (!elementRefused) throw new Error('Element input accepted without an AX snapshot');\\n return { axFailures: failures.length, screenshotWindow: shot.windowId, pngBytes: shot.bytes.length, elementRefused };\\n} finally { await unlink('fixture-ax-failure'); }\",\"description\":\"Capture through the object API despite application and window AX failures\"}"}} +{"type":"tool/ptc-dispatch-start","data":{"rootCallId":"cua-ptc-ax-failure","parentCallId":"cua-ptc-ax-failure","subCallId":"cua-ptc-ax-failure:ptc:1","name":"cua","arguments":{"method":"getCapabilities","params":{}}}} +{"type":"tool/ptc-dispatch","data":{"rootCallId":"cua-ptc-ax-failure","parentCallId":"cua-ptc-ax-failure","subCallId":"cua-ptc-ax-failure:ptc:1","name":"cua","arguments":{"method":"getCapabilities","params":{}},"isError":false,"content":[{"type":"text","text":"{\"sessionLocked\":false,\"permissions\":{\"accessibility\":true,\"screenRecording\":true},\"platform\":\"macos\",\"methods\":[\"activate\",\"click\",\"drag\",\"findApps\",\"getApp\",\"getCapabilities\",\"getState\",\"invoke\",\"listActions\",\"listApps\",\"listWindows\",\"mouseMove\",\"paste\",\"performAction\",\"pressKey\",\"screenshot\",\"scroll\",\"selectText\",\"setValue\",\"show\",\"typeText\"],\"appLevelAccessibility\":true,\"keyboard\":[\"app\"],\"browser\":false,\"coordinateSpace\":\"screenshot-pixels\"}"}]}} +{"type":"tool/ptc-dispatch-start","data":{"rootCallId":"cua-ptc-ax-failure","parentCallId":"cua-ptc-ax-failure","subCallId":"cua-ptc-ax-failure:ptc:2","name":"cua","arguments":{"method":"getApp","params":{"selector":"test.fixture"}}}} +{"type":"tool/ptc-dispatch","data":{"rootCallId":"cua-ptc-ax-failure","parentCallId":"cua-ptc-ax-failure","subCallId":"cua-ptc-ax-failure:ptc:2","name":"cua","arguments":{"method":"getApp","params":{"selector":"test.fixture"}},"isError":false,"content":[{"type":"text","text":"{\"appHandle\":\"fixture-app\",\"name\":\"Fixture\",\"bundleId\":\"test.fixture\",\"running\":true}"}]}} +{"type":"tool/ptc-dispatch-start","data":{"rootCallId":"cua-ptc-ax-failure","parentCallId":"cua-ptc-ax-failure","subCallId":"cua-ptc-ax-failure:ptc:3","name":"cua","arguments":{"method":"listWindows","params":{"appHandle":"fixture-app"}}}} +{"type":"tool/ptc-dispatch","data":{"rootCallId":"cua-ptc-ax-failure","parentCallId":"cua-ptc-ax-failure","subCallId":"cua-ptc-ax-failure:ptc:3","name":"cua","arguments":{"method":"listWindows","params":{"appHandle":"fixture-app"}},"isError":false,"content":[{"type":"text","text":"[{\"windowId\":7,\"appName\":\"Fixture\",\"title\":\"CUA fixture\",\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"isOnScreen\":true,\"onCurrentSpace\":true},{\"windowId\":8,\"appName\":\"Fixture\",\"title\":\"AXless canvas\",\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"isOnScreen\":true,\"onCurrentSpace\":true}]"}]}} +{"type":"tool/ptc-dispatch-start","data":{"rootCallId":"cua-ptc-ax-failure","parentCallId":"cua-ptc-ax-failure","subCallId":"cua-ptc-ax-failure:ptc:4","name":"cua","arguments":{"method":"listWindows","params":{"appHandle":"fixture-app"}}}} +{"type":"tool/ptc-dispatch","data":{"rootCallId":"cua-ptc-ax-failure","parentCallId":"cua-ptc-ax-failure","subCallId":"cua-ptc-ax-failure:ptc:4","name":"cua","arguments":{"method":"listWindows","params":{"appHandle":"fixture-app"}},"isError":false,"content":[{"type":"text","text":"[{\"windowId\":7,\"appName\":\"Fixture\",\"title\":\"CUA fixture\",\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"isOnScreen\":true,\"onCurrentSpace\":true},{\"windowId\":8,\"appName\":\"Fixture\",\"title\":\"AXless canvas\",\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"isOnScreen\":true,\"onCurrentSpace\":true}]"}]}} +{"type":"tool/ptc-dispatch-start","data":{"rootCallId":"cua-ptc-ax-failure","parentCallId":"cua-ptc-ax-failure","subCallId":"cua-ptc-ax-failure:ptc:5","name":"cua","arguments":{"method":"getState","params":{"appHandle":"fixture-app"}}}} +{"type":"tool/ptc-dispatch","data":{"rootCallId":"cua-ptc-ax-failure","parentCallId":"cua-ptc-ax-failure","subCallId":"cua-ptc-ax-failure:ptc:5","name":"cua","arguments":{"method":"getState","params":{"appHandle":"fixture-app"}},"isError":true,"content":[{"type":"text","text":"Error: NATIVE_ERROR: Fixture accessibility read failed"}]}} +{"type":"tool/ptc-dispatch-start","data":{"rootCallId":"cua-ptc-ax-failure","parentCallId":"cua-ptc-ax-failure","subCallId":"cua-ptc-ax-failure:ptc:6","name":"cua","arguments":{"method":"getState","params":{"appHandle":"fixture-app","windowId":7}}}} +{"type":"tool/ptc-dispatch","data":{"rootCallId":"cua-ptc-ax-failure","parentCallId":"cua-ptc-ax-failure","subCallId":"cua-ptc-ax-failure:ptc:6","name":"cua","arguments":{"method":"getState","params":{"appHandle":"fixture-app","windowId":7}},"isError":true,"content":[{"type":"text","text":"Error: NATIVE_ERROR: Fixture accessibility read failed"}]}} +{"type":"tool/ptc-dispatch-start","data":{"rootCallId":"cua-ptc-ax-failure","parentCallId":"cua-ptc-ax-failure","subCallId":"cua-ptc-ax-failure:ptc:7","name":"cua","arguments":{"method":"screenshot","params":{"appHandle":"fixture-app","windowId":7}}}} +{"type":"tool/ptc-dispatch","data":{"rootCallId":"cua-ptc-ax-failure","parentCallId":"cua-ptc-ax-failure","subCallId":"cua-ptc-ax-failure:ptc:7","name":"cua","arguments":{"method":"screenshot","params":{"appHandle":"fixture-app","windowId":7}},"isError":false,"content":[{"type":"text","text":"{\"pixelWidth\":1,\"pixelHeight\":1,\"windowId\":7,\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"coordinateSpace\":\"screenshot-pixels\"}"},{"type":"image","attachment":{"attachmentId":"sha256:999f1d1527ee7e79266f16add5430fff76b1225d742464a5b1ff1f02971bb8ee","mediaType":"image/png","width":1,"height":1,"bytes":90}}]}} +{"type":"tool/result","data":{"turn":1,"step":2,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-ptc-ax-failure"},"toolCallId":"cua-ptc-ax-failure","content":[{"type":"text","text":"{\n \"axFailures\": 2,\n \"screenshotWindow\": 7,\n \"pngBytes\": 90,\n \"elementRefused\": true\n}"}],"isError":false,"id":"{{message:8}}"}},"sourceEventSeqs":[28],"surfaceOp":"append"} +{"type":"agent/inbox/spliced","data":{"target":"next-step","start":0,"inserted":[{"content":[{"type":"text","text":"{\"pixelWidth\":1,\"pixelHeight\":1,\"windowId\":7,\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"coordinateSpace\":\"screenshot-pixels\"}"},{"type":"image","attachment":{"attachmentId":"sha256:999f1d1527ee7e79266f16add5430fff76b1225d742464a5b1ff1f02971bb8ee","mediaType":"image/png","width":1,"height":1,"bytes":90}}],"source":{"kind":"ptc-mode"},"role":"user","id":"{{message:9}}"}]}} {"type":"step/end","data":{"turn":1,"step":2}} +{"type":"agent/inbox/spliced","data":{"target":"next-step","start":0,"removedCount":1,"inserted":[]}} +{"type":"step/start","data":{"turn":1,"step":3}} +{"type":"user/message","data":{"content":[{"type":"text","text":"{\"pixelWidth\":1,\"pixelHeight\":1,\"windowId\":7,\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"coordinateSpace\":\"screenshot-pixels\"}"},{"type":"image","attachment":{"attachmentId":"sha256:999f1d1527ee7e79266f16add5430fff76b1225d742464a5b1ff1f02971bb8ee","mediaType":"image/png","width":1,"height":1,"bytes":90}}],"source":{"kind":"ptc-mode"},"role":"user","id":"{{message:9}}"},"surfaceOp":"append"} +{"type":"assistant/message","data":{"turn":1,"step":3,"message":{"role":"assistant","content":[{"type":"text","text":"Received native CUA values and PNG screenshots through PTC. The object API captured a 90-byte PNG after two AX read failures and refused element input without an accessibility snapshot."}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:10}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1794024974851,"chunk":{"type":"block-start","index":0,"blockType":"text"}},{"type":"chunk","time":1794024974851,"chunk":{"type":"block-end","index":0,"block":{"type":"text","text":"Received native CUA values and PNG screenshots through PTC. The object API captured a 90-byte PNG after two AX read failures and refused element input without an accessibility snapshot."}}},{"type":"chunk","time":1794024974851,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1794024974851,"chunk":{"type":"finish","reason":{"kind":"stop"}}}]},"surfaceOp":"append"} +{"type":"step/end","data":{"turn":1,"step":3}} {"type":"turn/end","data":{"turn":1,"reason":{"kind":"completed"}}} diff --git a/snapshots/session/cua-ptc/system-prompt.expected.md b/snapshots/session/cua-ptc/system-prompt.expected.md index 587436227a..be8ab5aed8 100644 --- a/snapshots/session/cua-ptc/system-prompt.expected.md +++ b/snapshots/session/cua-ptc/system-prompt.expected.md @@ -48,9 +48,9 @@ Setup and discovery: - cua.listApps(options?: { emit?: boolean }) -> { id: string, displayName: string, isRunning: true, pid: number }[] — enumerate running applications. - cua.findApps(query: string, options?: { emit?: boolean }) -> ({ id: string, displayName: string, path: string } & ({ isRunning: true, pid: number } | { isRunning: false }))[] — search installed applications, including stopped matches; only running matches carry pid. - cua.getState(options?: { emit?: boolean }) -> { apps: { id: string, displayName: string, isRunning: true, pid: number }[], browsers: [] } — summarize running applications; this is not an accessibility observation. -- cua.getApp(selector: string) -> App — select an application by name, bundle ID, absolute path, or running PID string, obtain consent, launch in the background if needed, and read initial AX state silently; it does not activate the app. +- cua.getApp(selector: string) -> App — select an application by name, bundle ID, absolute path, or running PID string, obtain consent, launch in the background if needed, and return without reading AX or activating the app. - app.getWindows(options?: { emit?: boolean }) -> { windowId: number, appName: string, title: string, bounds: { x: number, y: number, width: number, height: number }, isOnScreen: boolean, onCurrentSpace: boolean | null }[] — enumerate windows with global native bounds; null onCurrentSpace means unknown Space membership. -- app.getWindow(windowId: number) -> Window — select one window and read its initial AX state silently; do this before any window screenshot or coordinate input. +- app.getWindow(windowId: number) -> Window — select a window from the application’s current window list without reading AX; do this before any window screenshot or coordinate input. - app.activate() -> ActionResult — request explicit foreground activation; use only when foreground access is required. - app.show() -> { dispatched: boolean, windowId: number | null, foregroundChanged: boolean | null } — unhide the app and restore its selected, focused, main, or sole window without activation. - window.show() -> { dispatched: boolean, windowId: number | null, foregroundChanged: boolean | null } — unhide and restore that explicit window without activation. @@ -58,7 +58,7 @@ Setup and discovery: ## Accessibility tree Available at app or window level depending on the platform. -Element ids are opaque strings from the latest successful tree; the object facade keeps one current snapshot per app across its repeated app and window objects, a new read replaces it, and a failed read invalidates the previous one. Repeated strings can appear for different elements, so do not reuse an old id blindly, and element operations do not fall back to pointer input or typing. +Read getAccessibilityTree explicitly before using element ids. Selection and screenshots do not read or replace the current AX snapshot. Element ids are opaque strings from the latest successful tree; the object facade keeps one current snapshot per app across its repeated app and window objects, a new read replaces it, and a failed read invalidates the previous one. Repeated strings can appear for different elements, so do not reuse an old id blindly, and element operations do not fall back to pointer input or typing. - getAccessibilityTree(options?: { emit?: boolean, includeMenus?: boolean, details?: boolean }) -> string — read the full tree text; includeMenus expands menu contents and details adds identifiers, native bounds, and complete action names. - listActions(elementId: string) -> string[] — list the actions an element advertises. - invoke(elementId: string) -> ActionResult — invoke the element's default action; dispatched is false when it has none. @@ -82,7 +82,7 @@ Available at app or window level depending on the platform. - paste(text: string, options?: { format?: 'text' | 'html' }) -> ActionResult — deliver text or HTML through the app's Paste command, restoring prior clipboard contents unless another writer changes them; a Paste failure is surfaced without substituting typeText, and PASTE_NOT_CONSUMED means inspect before any retry. - pressKey(key: string, modifiers?: string[]) -> ActionResult — press and release one complete keystroke. For example pressKey('Tab', ['shift']) or pressKey('+'). Key and modifier names are platform-defined strings. Pass key and modifiers separately; do not pack them into a chord string. -Raw tools use { method, params }: include appHandle for app calls and windowId for window calls, and use snapshotId with elementId for element routes. The emit option is facade-only: emit is local output control, so raw callers must follow the raw operation schema for admitted params. getAccessibilityTree and getWindow read through raw getState; getWindows maps to listWindows; getScreenshot maps to screenshot; cua.getState maps to listApps; every other named operation uses the same method name. +Raw tools use { method, params }: include appHandle for app calls and windowId for window calls, and use snapshotId with elementId for element routes. The emit option is facade-only: emit is local output control, so raw callers must follow the raw operation schema for admitted params. getAccessibilityTree maps to getState; getWindow validates against listWindows; getWindows maps to listWindows; getScreenshot maps to screenshot; cua.getState maps to listApps; every other named operation uses the same method name. Maximum click holdMs: 5000 ms. diff --git a/snapshots/session/ptc-script-retained/system-prompt.expected.md b/snapshots/session/ptc-script-retained/system-prompt.expected.md index dd88fa8d9e..63e91315c0 100644 --- a/snapshots/session/ptc-script-retained/system-prompt.expected.md +++ b/snapshots/session/ptc-script-retained/system-prompt.expected.md @@ -48,9 +48,9 @@ Setup and discovery: - cua.listApps(options?: { emit?: boolean }) -> { id: string, displayName: string, isRunning: true, pid: number }[] — enumerate running applications. - cua.findApps(query: string, options?: { emit?: boolean }) -> ({ id: string, displayName: string, path: string } & ({ isRunning: true, pid: number } | { isRunning: false }))[] — search installed applications, including stopped matches; only running matches carry pid. - cua.getState(options?: { emit?: boolean }) -> { apps: { id: string, displayName: string, isRunning: true, pid: number }[], browsers: [] } — summarize running applications; this is not an accessibility observation. -- cua.getApp(selector: string) -> App — select an application by name, bundle ID, absolute path, or running PID string, obtain consent, launch in the background if needed, and read initial AX state silently; it does not activate the app. +- cua.getApp(selector: string) -> App — select an application by name, bundle ID, absolute path, or running PID string, obtain consent, launch in the background if needed, and return without reading AX or activating the app. - app.getWindows(options?: { emit?: boolean }) -> { windowId: number, appName: string, title: string, bounds: { x: number, y: number, width: number, height: number }, isOnScreen: boolean, onCurrentSpace: boolean | null }[] — enumerate windows with global native bounds; null onCurrentSpace means unknown Space membership. -- app.getWindow(windowId: number) -> Window — select one window and read its initial AX state silently; do this before any window screenshot or coordinate input. +- app.getWindow(windowId: number) -> Window — select a window from the application’s current window list without reading AX; do this before any window screenshot or coordinate input. - app.activate() -> ActionResult — request explicit foreground activation; use only when foreground access is required. - app.show() -> { dispatched: boolean, windowId: number | null, foregroundChanged: boolean | null } — unhide the app and restore its selected, focused, main, or sole window without activation. - window.show() -> { dispatched: boolean, windowId: number | null, foregroundChanged: boolean | null } — unhide and restore that explicit window without activation. @@ -58,7 +58,7 @@ Setup and discovery: ## Accessibility tree Available at app or window level depending on the platform. -Element ids are opaque strings from the latest successful tree; the object facade keeps one current snapshot per app across its repeated app and window objects, a new read replaces it, and a failed read invalidates the previous one. Repeated strings can appear for different elements, so do not reuse an old id blindly, and element operations do not fall back to pointer input or typing. +Read getAccessibilityTree explicitly before using element ids. Selection and screenshots do not read or replace the current AX snapshot. Element ids are opaque strings from the latest successful tree; the object facade keeps one current snapshot per app across its repeated app and window objects, a new read replaces it, and a failed read invalidates the previous one. Repeated strings can appear for different elements, so do not reuse an old id blindly, and element operations do not fall back to pointer input or typing. - getAccessibilityTree(options?: { emit?: boolean, includeMenus?: boolean, details?: boolean }) -> string — read the full tree text; includeMenus expands menu contents and details adds identifiers, native bounds, and complete action names. - listActions(elementId: string) -> string[] — list the actions an element advertises. - invoke(elementId: string) -> ActionResult — invoke the element's default action; dispatched is false when it has none. @@ -82,7 +82,7 @@ Available at app or window level depending on the platform. - paste(text: string, options?: { format?: 'text' | 'html' }) -> ActionResult — deliver text or HTML through the app's Paste command, restoring prior clipboard contents unless another writer changes them; a Paste failure is surfaced without substituting typeText, and PASTE_NOT_CONSUMED means inspect before any retry. - pressKey(key: string, modifiers?: string[]) -> ActionResult — press and release one complete keystroke. For example pressKey('Tab', ['shift']) or pressKey('+'). Key and modifier names are platform-defined strings. Pass key and modifiers separately; do not pack them into a chord string. -Raw tools use { method, params }: include appHandle for app calls and windowId for window calls, and use snapshotId with elementId for element routes. The emit option is facade-only: emit is local output control, so raw callers must follow the raw operation schema for admitted params. getAccessibilityTree and getWindow read through raw getState; getWindows maps to listWindows; getScreenshot maps to screenshot; cua.getState maps to listApps; every other named operation uses the same method name. +Raw tools use { method, params }: include appHandle for app calls and windowId for window calls, and use snapshotId with elementId for element routes. The emit option is facade-only: emit is local output control, so raw callers must follow the raw operation schema for admitted params. getAccessibilityTree maps to getState; getWindow validates against listWindows; getWindows maps to listWindows; getScreenshot maps to screenshot; cua.getState maps to listApps; every other named operation uses the same method name. Maximum click holdMs: 5000 ms.