feat(cua): request system pointer access on the target mouse move

This commit is contained in:
creatixchu committed 2026-11-07 16:31:24 +08:00
1 parent ed5d4d23de
commit c51ecb2bff
33 files changed
+263 -193

No files matched your search

@@ -9,8 +9,8 @@
en: f96bf1ebc0719fb0
zh: e9a1144f0e76572d
/agent-note-background-hover-and-screenshot-coordinates/decision:
en: 93631d45c7ecfb4c
zh: 77a08aafb1115ba1
en: 42f40e78c2fff8a8
zh: b7c5fa5fe538ed77
/agent-note-background-hover-and-screenshot-coordinates/alternatives-considered:
en: 5fc4beaf79761c50
zh: 4948333b0020be28
@@ -10,7 +10,7 @@ Hover-only controls can sometimes be revealed without moving the user's pointer.
## Decision
By default, native hover uses a same-window element or ancestor advertising both `AXShowAlternateUI` and `AXShowDefaultUI`. Entering its revealed descendants preserves the parent UI; leaving restores it. An unavailable route returns `HOVER_UNAVAILABLE` by default. After the user approves `cua.requestMouseControl()`, each `mouseMove` in that session uses the system pointer until DSH restarts. This lets a caller reach pointer-tracking controls even when the same target also advertises AX alternate UI. The Host passes the session decision with each move. Physical movement leaves the visibility of previously revealed controls to the application, so switching routes cannot dismiss a target before dispatch. An AX action failure propagates because received effects may be uncertain. The result records the route and whether the system pointer moved. System movement requires the selected foreground app and AppKit's mouse hit test, which excludes input-transparent overlays. No route grants [cross-call ownership](2026-10-13-cua-continuous-hover.md).
By default, native hover uses a same-window element or ancestor advertising both `AXShowAlternateUI` and `AXShowDefaultUI`. Entering its revealed descendants preserves the parent UI; leaving restores it. An unavailable route returns `HOVER_UNAVAILABLE` by default. With `requireSystemPointerApproval` enabled, the first explicit `mouseMove(target, { useSystemPointer: true })` asks the user. Approval applies to this move and later ordinary moves in the live session. Closing the session or restarting DSH clears it. Without the approval requirement only an explicit request selects real input. This lets a caller reach pointer-tracking controls even when the same target also advertises AX alternate UI. The Host passes the session decision with each move. Physical movement leaves the visibility of previously revealed controls to the application, so switching routes cannot dismiss a target before dispatch. An AX action failure propagates because received effects may be uncertain. The result records the route and whether the system pointer moved. System movement requires the selected foreground app and AppKit's mouse hit test, which excludes input-transparent overlays. No route grants [cross-call ownership](2026-10-13-cua-continuous-hover.md).
Model-facing coordinate input defaults to pixels in the returned screenshot. `relative: true` accepts integer thousandths from 0 to 999 for `click`, `mouseMove`, `scroll` and `drag`; it does not apply to opaque element IDs. The Host owns the mapping to native window points. An application-wide observation refreshes its inventory; a scoped observation preserves other windows' image mappings. Window resizing invalidates the old image, while movement preserves local coordinates. Relative endpoints remain inside the window and do not depend on screenshot resolution.
@@ -10,7 +10,7 @@ Status: implemented
## 决策
默认情况下,原生悬停会寻找同一窗口内同时声明 `AXShowAlternateUI` 和 `AXShowDefaultUI` 的元素或祖先。进入其展开区域的子控件时保持父控件展开,离开后恢复。没有可用路径时默认返回 `HOVER_UNAVAILABLE`。用户同意 `cua.requestMouseControl()` 后,本会话的每次 `mouseMove` 都使用系统光标,重启 DSH 后授权失效。有些控件既支持 AX 备用界面,又跟踪真实鼠标。Host 在每次悬停请求中传入本会话的授权结果,让调用方能够触发后者。切换到系统鼠标后,由应用决定是否收起先前展开的控件,以免派发前目标消失。AX 动作失败直接返回,因为已产生的效果可能不确定。结果记录实际路径及是否移动系统鼠标。系统移动要求所选应用处于前台,并使用 AppKit 的鼠标命中查询排除不接收输入的覆盖窗口。所有路径都遵守[跨调用不分配所有权](2026-10-13-cua-continuous-hover.zh.md)的决定。
默认情况下,原生悬停会寻找同一窗口内同时声明 `AXShowAlternateUI` 和 `AXShowDefaultUI` 的元素或祖先。进入其展开区域的子控件时保持父控件展开,离开后恢复。没有可用路径时默认返回 `HOVER_UNAVAILABLE`。启用 `requireSystemPointerApproval` 时,首次显式调用 `mouseMove(target, { useSystemPointer: true })` 会询问用户。同意后,本次移动和本会话后续普通悬停使用系统光标。关闭会话或重启 DSH 后授权失效。免询问配置下,只有显式请求才会移动系统光标。有些控件既支持 AX 备用界面,又跟踪真实鼠标。Host 在每次悬停请求中传入本会话的授权结果,让调用方能够触发后者。切换到系统鼠标后,由应用决定是否收起先前展开的控件,以免派发前目标消失。AX 动作失败直接返回,因为已产生的效果可能不确定。结果记录实际路径及是否移动系统鼠标。系统移动要求所选应用处于前台,并使用 AppKit 的鼠标命中查询排除不接收输入的覆盖窗口。所有路径都遵守[跨调用不分配所有权](2026-10-13-cua-continuous-hover.zh.md)的决定。
面向模型的坐标默认使用返回截图中的像素。`click`、`mouseMove`、`scroll` 和 `drag` 传入 `relative: true` 时接受 0~999 的千分比整数,仅用于坐标目标。Host 负责换算原生窗口 point。应用级观察刷新窗口清单,单窗口观察保留其他窗口的图片映射。窗口尺寸变化使旧图片失效,移动窗口保留局部坐标。相对坐标的端点保持在窗口内,不随截图分辨率改变。
+2
View File
@@ -1163,6 +1163,8 @@ export interface Config {
export interface Config extends CuaConfig {
/** Interval in milliseconds between permission and position checks while the native guide is open. */
permissionPollIntervalMs: number
/** Ask before a main task may move the system pointer. When false only explicitly requested moves use it. */
requireSystemPointerApproval: boolean
/**
* Source-build signing: `ad-hoc` (default) binds grants to the code hash;
* an object selects an existing certificate in an explicitly configured keychain.
+2
View File
@@ -1165,6 +1165,8 @@ export interface Config {
export interface Config extends CuaConfig {
/** Interval in milliseconds between permission and position checks while the native guide is open. */
permissionPollIntervalMs: number
/** Ask before a main task may move the system pointer. When false only explicitly requested moves use it. */
requireSystemPointerApproval: boolean
/**
* Source-build signing: `ad-hoc` (default) binds grants to the code hash;
* an object selects an existing certificate in an explicitly configured keychain.
@@ -9,8 +9,8 @@
en: 6833a7851486f10d
zh: 22983f326fb725ce
/macos-hover-keeps-the-system-pointer-in-place-by-default/change:
en: 2369a7daf049311a
zh: 2ad1bfb579be4426
en: e6d2e56ccb50e46a
zh: a788fc19e0771a46
/macos-hover-keeps-the-system-pointer-in-place-by-default/migration:
en: 7c89c67c9acefd25
zh: 54a0064c30af043c
en: b1070d660b90318c
zh: 2f2ad1b8897dc044
@@ -9,10 +9,10 @@ English | [中文](guide.zh.md)
## Change
On macOS, `win.mouseMove(target)` uses an enabled element or ancestor that advertises both `AXShowAlternateUI` and `AXShowDefaultUI`. Without that route, previous versions moved the visible system cursor. The default now returns `HOVER_UNAVAILABLE` before sending input. Moving the pointer requires the user’s approval for the current session.
On macOS, `win.mouseMove(target)` uses an enabled element or ancestor that advertises both `AXShowAlternateUI` and `AXShowDefaultUI`. Without that route, previous versions moved the visible system cursor. The default now returns `HOVER_UNAVAILABLE` before sending input. By default, explicitly requesting the system pointer asks the user for this session.
## Migration
1. For an available control, use its advertised action or click it directly. Use browser tools for web content that requires page hover.
2. If your profile sets `helperPath`, rebuild the custom helper to support the per-request `useSystemPointer` field and restart the profile.
3. When the task requires a physical hover, call `cua.requestMouseControl()`. After the user approves, `win.mouseMove` on an unobscured foreground target returns `hoverRoute: "system-pointer"` and `systemPointerMoved: true`. This grant expires when DSH restarts.
3. When the task requires physical movement, call `win.mouseMove(target, { useSystemPointer: true })` on an unobscured foreground target. After approval this move reports `systemPointerMoved: true`. Closing the session or restarting DSH clears approval.
@@ -9,10 +9,10 @@ description: "macOS 的 mouseMove 默认要求控件支持 AX 备用界面,否
## 变更
在 macOS 上,`win.mouseMove(target)` 会查找已启用且同时声明 `AXShowAlternateUI` 和 `AXShowDefaultUI` 的控件或祖先。找不到时,旧版会移动系统光标,新版默认在发送输入前返回 `HOVER_UNAVAILABLE`。移动系统光标需要用户单独授权当前会话。
在 macOS 上,`win.mouseMove(target)` 会查找已启用且同时声明 `AXShowAlternateUI` 和 `AXShowDefaultUI` 的控件或祖先。找不到时,旧版会移动系统光标,新版默认在发送输入前返回 `HOVER_UNAVAILABLE`。默认配置下,显式请求移动系统光标时会询问用户是否授权当前会话。
## 迁移
1. 对已经可用的控件,可执行它声明的动作或直接点击。需要网页悬停时可用浏览器工具。
2. 如果 profile 设置了 `helperPath`,请重新构建该 helper,使其支持按请求读取 `useSystemPointer` 字段,然后重启 profile。
3. 任务确实需要移动鼠标时,调用 `cua.requestMouseControl()`。用户同意后,对前台未受遮挡的目标调用 `win.mouseMove`,结果为 `hoverRoute: "system-pointer"` 和 `systemPointerMoved: true`。重启 DSH 后授权失效。
3. 任务确实需要移动鼠标时,在前台未受遮挡的目标上调用 `win.mouseMove(target, { useSystemPointer: true })`。用户同意后,本次移动返回 `systemPointerMoved: true`。关闭会话或重启 DSH 后需要重新授权。
@@ -15,11 +15,11 @@
en: 1777e7bf6785f986
zh: e3daac1074ebb74d
/deepseek-ai-dsh-experimental-cua-macos/use-this-package:
en: c81e5fbf8ff7e110
zh: f70adab1257ed26d
en: e51a6c3625873430
zh: 0a6998897532a05f
/deepseek-ai-dsh-experimental-cua-macos/use-this-package/connect-an-external-execution-environment:
en: 2391a460e295cfca
zh: 1889a903efb6142d
en: b87eede533256337
zh: 15c6b1323a60c5b6
/deepseek-ai-dsh-experimental-cua-macos/understand-the-implementation:
en: 4e82b85a5aa72eee
zh: 4e004b7a205d8688
+3 -3
View File
@@ -60,6 +60,7 @@ Source builds default to `sourceSigning: ad-hoc`, without creating certificates
| `agentPreset` | `ptc` | Register tools and guidance only for Agents bound to this preset; `false` registers globally. |
| `allowedApps` | `[]` | Legacy path/signature grants, imported once for removal; they do not authorize new bundle-ID identities. |
| `helperPath` | package-owned executable | Absolute executable inside the signed helper `.app`; a bare binary fails with `NATIVE_APPLICATION_REQUIRED`. |
| `requireSystemPointerApproval` | `true` | Ask on the first explicit request to move the system pointer. With `false`, only explicitly requested moves use it. |
| `sourceSigning` | `ad-hoc` | Source signing: `ad-hoc`, or `{ identity, keychain }` for an existing certificate and keychain path. Ignored for installed packages and `helperPath`. |
| `buildTimeoutMs` | `300000` | Maximum native preparation time. |
| `timeoutMs` | `30000` | Per-operation deadline; cancellation can leave partial input. |
@@ -116,7 +117,7 @@ Use the `cua` API to discover and select the application (`listApps`, `getState`
`getWindow` reads its initial tree silently. An owned window without an AX surface returns an empty snapshot and remains available for screenshots and coordinate input. Within one API instance, the facade keeps one current accessibility snapshot per application handle, shared by every app and window object for that handle, including repeated selections. Reading any window replaces that snapshot; element operations use the latest snapshot ID, so use IDs from the latest tree and do not assume an older ID still identifies the same control. A failed read publishes no partial snapshot; read again without repeating the preceding input. Window objects own screenshots and pointer input. Keyboard methods stay on `app` because input follows the app’s current focus.
`mouseMove(target)` defaults to AX hover. An enabled target or ancestor must advertise both `AXShowAlternateUI` and `AXShowDefaultUI`. Moving into its descendants keeps the UI open, and leaving restores it. Without an AX route the helper returns `HOVER_UNAVAILABLE` before sending input. Click a visible control or use `performAction` with an advertised action. For browser content without these AX actions, use browser tools. If these alternatives cannot complete the task, `cua.requestMouseControl()` asks for this session until DSH restarts. On approval, every subsequent hover moves the system pointer to the foreground, unobscured target and leaves it there.
Ordinary `mouseMove(target)` defaults to AX hover. Without both `AXShowAlternateUI` and `AXShowDefaultUI` on an enabled target or ancestor, it returns `HOVER_UNAVAILABLE` without prompting. Only when the task requires physical movement should the main task call `win.mouseMove(target, { useSystemPointer: true })`. On approval the Host rechecks the window and point, dispatches this move, and uses the visible pointer for later ordinary moves. Refusal sends no input and suppresses further questions. Closing the session or restarting DSH clears the decision. Explicit `useSystemPointer: false` selects AX for one call.
`paste(text, { format: "text" | "html" })` temporarily publishes the requested clipboard format and invokes the application’s enabled standard Paste menu command. It keeps the temporary format until a pasteboard reader requests its data, then restores every captured original format while it still owns the clipboard, preserving a later copy. Paste operations share a cross-process lock and cancellation joins helper cleanup. A forced process exit can prevent restoration. Unsupported Paste commands fail without typing fallback; `typeText` remains separate key-event input.
@@ -187,7 +188,6 @@ Signatures show resolved return values after await. ActionResult = { dispatched:
Setup and discovery:
- cua.getCapabilities(options?: { emit?: boolean }) -> { platform: string, permissions: { accessibility: boolean, screenRecording: boolean }, appLevelAccessibility: boolean, keyboard: ('app' | 'window')[], sessionLocked: boolean | null, methods: string[], coordinateSpace: 'screenshot-pixels', browser: false } — inspect reported permissions, sessionLocked, methods, and keyboard scopes without requesting permissions.
- cua.requestMouseControl() -> boolean — ask the user to allow mouseMove to move their system pointer for this session until DSH restarts. Only the main task may ask, and only when a physical hover is necessary. A false result is a refusal for this session until restart. If the user dismisses the request, continue without mouse control unless they ask to reconsider. Await the outstanding request before calling again.
- cua.listApps(options?: { emit?: boolean }) -> { id: string, displayName: string, isRunning: true, pid: number }[] — enumerate running applications.
- cua.findApps(query: string, options?: { emit?: boolean }) -> ({ id: string, displayName: string, path: string } & ({ isRunning: true, pid: number } | { isRunning: false }))[] — search installed applications, including stopped matches; only running matches carry pid.
- cua.getState(options?: { emit?: boolean }) -> { apps: { id: string, displayName: string, isRunning: true, pid: number }[], browsers: [] } — summarize running applications; this is not an accessibility observation.
@@ -212,7 +212,7 @@ Element ids are opaque strings from the latest successful tree; the object facad
## Simulated mouse
Use an explicit window for screenshots and pointer input.
- getScreenshot(options?: { emit?: boolean }) -> { bytes: Uint8Array, mimeType: 'image/png' | 'image/jpeg' | 'image/webp' | 'image/gif', pixelWidth: number, pixelHeight: number, windowId: number } — capture the selected window as one encoded image; bytes is a JavaScript byte array, and mimeType identifies its encoding, so do not assume PNG.
- mouseMove(target: string | [number, number], options?: { relative?: boolean }) -> { dispatched: boolean, foregroundChanged?: boolean | null, hoverRoute: 'accessibility' | 'system-pointer', systemPointerMoved: boolean } — request a mouse move to the target without pressing a button; inspect hoverRoute and systemPointerMoved, and expect no automatic pointer restoration. Default hover requires both AXShowAlternateUI and AXShowDefaultUI on an enabled element or ancestor. Without an AX route it returns HOVER_UNAVAILABLE before sending input. Prefer click on a visible control or performAction with an advertised action. For web content without an AX hover action, use browser tools. Only request mouse control if the task cannot be completed with those alternatives. The question interrupts the user, and moving their pointer can disrupt their work. After approval, every mouseMove requires the target app to be in front and the point unobscured, including targets that previously supported background AX hover. If mouseMove returns FOREGROUND_REQUIRED, explain the necessary foreground step before activating the app, then observe the target again.
- mouseMove(target: string | [number, number], options?: { relative?: boolean, useSystemPointer?: boolean }) -> { dispatched: boolean, foregroundChanged?: boolean | null, hoverRoute: 'accessibility' | 'system-pointer', systemPointerMoved: boolean } — request a mouse move to the target without pressing a button; inspect hoverRoute and systemPointerMoved, and expect no automatic pointer restoration. Default hover requires both AXShowAlternateUI and AXShowDefaultUI on an enabled element or ancestor. HOVER_UNAVAILABLE sends no input and never opens a question. Use an advertised accessibility action, click a visible control, or use browser tools for web content whenever possible. Only if the task cannot be completed without moving the user’s pointer, call await win.mouseMove(target, { useSystemPointer: true }). The question interrupts the user; approval moves their pointer to this target immediately and makes subsequent ordinary moves use it until the session closes or DSH restarts. A refusal sends no input; continue with pointer-free actions. If the question is dismissed, ask again only when the user requests it. Await an outstanding request before making another. Without an approval requirement only explicitly requested moves use the system pointer. Subagents cannot request physical moves. System movement requires an unobscured foreground target; explain any necessary activation and observe the target again.
- click(target: string | [number, number], options?: { mouseButton?: 'left' | 'right' | 'middle', clickCount?: 1 | 2, modifiers?: ('command' | 'control' | 'option' | 'shift' | 'fn')[], holdMs?: number, relative?: boolean }) -> ActionResult — click an element id or point, or perform a left-button double-click; holdMs controls the press duration of each press. click is a Window method: select a window with const win = await app.getWindow(windowId), then call win.click(target). For a background control’s default action, use app.invoke(id). For a mouse click on a background app, call app.activate() first and re-read the target. Both target forms send mouse-down and mouse-up events. click(id) uses the center of the full element bounds. If that center is outside the window, take a fresh win.getScreenshot() and use click([x, y]) on a visible part of the control in screenshot pixels. A background app can ignore the events even when dispatched is true. Verify the expected effect, such as a changed selection, before typing or retrying.
- scroll(options: { x: number, y: number, deltaLinesX: number, deltaLinesY: number, relative?: boolean }) -> ActionResult — scroll at a point; deltas are integers from -10 to 10 with at least one nonzero, and positive values reveal content above or left.
- drag(from: [number, number], to: [number, number], options?: { mouseButton?: 'left' | 'right' | 'middle', modifiers?: ('command' | 'control' | 'option' | 'shift' | 'fn')[], relative?: boolean }) -> ActionResult — drag between two points of one window.
+3 -3
View File
@@ -60,6 +60,7 @@ pnpm dsh headless --patch packages/experimental/cua-macos/examples/macos.cordis.
| `agentPreset` | `ptc` | 仅为绑定该 preset 的 Agent 注册工具和指引;`false` 表示全局注册。 |
| `allowedApps` | `[]` | 旧路径和签名许可,只导入 profile 存储一次供撤销,不授予新 Bundle ID 身份访问权限。 |
| `helperPath` | 包内可执行文件 | 已签名 helper `.app` 内可执行文件的绝对路径;裸二进制会触发 `NATIVE_APPLICATION_REQUIRED`。 |
| `requireSystemPointerApproval` | `true` | 主任务首次显式请求移动系统指针时询问用户。设为 `false` 时仅本次显式移动免询问。 |
| `sourceSigning` | `ad-hoc` | 源码签名:`ad-hoc` 临时签名,或 `{ identity, keychain }` 指定已有证书及钥匙串路径。安装版和 `helperPath` 忽略此项。 |
| `buildTimeoutMs` | `300000` | 原生组件准备的最长时间。 |
| `timeoutMs` | `30000` | 单次操作截止时间,取消后可能已发生部分输入。 |
@@ -116,7 +117,7 @@ void await window.getScreenshot()
`getWindow` 静默读取初始树。已确认归属但没有 AX 界面的窗口返回空快照,仍可截图和发送坐标输入。在单个 API 实例内,门面按应用句柄保存一份当前辅助功能快照,该句柄的所有应用对象和窗口对象共享它,包括重复选择。读取任一窗口都会替换该快照;元素操作使用最新快照 ID,因此应使用最新树中的元素 ID,不应假定旧 ID 仍指向同一控件。读取失败不会发布部分快照;可以重新读取,但不要重复此前的输入。窗口对象负责截图和指针输入。键盘方法保留在 `app` 上,因为输入跟随应用的当前焦点。
`mouseMove(target)` 默认通过 AX 悬停。目标或其祖先必须已启用,并同时声明 `AXShowAlternateUI` 和 `AXShowDefaultUI`。进入子控件会保持展开,离开后恢复默认界面。如果没有可用的 AX 路径,helper 返回 `HOVER_UNAVAILABLE`,不发送输入。可点击已经可见的控件,或用 `performAction` 执行控件声明的动作。浏览器内容缺少这些 AX 动作时,请使用浏览器工具。只有其他操作都无法完成任务时,才调用 `cua.requestMouseControl()` 询问用户。用户同意后,本会话后续悬停会把系统光标移到前台未受遮挡的目标并停在那里,重启 DSH 后授权失效。
普通 `mouseMove(target)` 默认通过辅助功能悬停。如果目标或其祖先没有同时声明 `AXShowAlternateUI` 和 `AXShowDefaultUI`,则返回 `HOVER_UNAVAILABLE`,不会弹窗。只有任务确实需要移动用户的鼠标时,主任务才调用 `win.mouseMove(target, { useSystemPointer: true })`。用户允许后,Host 重新检查窗口和目标位置,执行这一次移动,此后本会话的普通悬停也使用系统指针。拒绝后不移动,也不再提问。关闭会话或重启 DSH 后决定失效。传入 `useSystemPointer: false` 可在本次调用中使用辅助功能。
`paste(text, { format: "text" | "html" })` 临时提供指定格式的剪贴板内容,并调用应用已启用的标准 Paste 菜单命令。临时格式保留到剪贴板读取方请求其数据,然后只在仍持有剪贴板时恢复保存的全部原格式,保留期间产生的新复制内容。粘贴操作共用跨进程锁,取消会等待辅助进程清理。强制终止进程可能阻止恢复。应用不支持 Paste 命令时明确失败,不自动改成逐字输入;`typeText` 仍单独提供按键事件输入。
@@ -187,7 +188,6 @@ Signatures show resolved return values after await. ActionResult = { dispatched:
Setup and discovery:
- cua.getCapabilities(options?: { emit?: boolean }) -> { platform: string, permissions: { accessibility: boolean, screenRecording: boolean }, appLevelAccessibility: boolean, keyboard: ('app' | 'window')[], sessionLocked: boolean | null, methods: string[], coordinateSpace: 'screenshot-pixels', browser: false } — inspect reported permissions, sessionLocked, methods, and keyboard scopes without requesting permissions.
- cua.requestMouseControl() -> boolean — ask the user to allow mouseMove to move their system pointer for this session until DSH restarts. Only the main task may ask, and only when a physical hover is necessary. A false result is a refusal for this session until restart. If the user dismisses the request, continue without mouse control unless they ask to reconsider. Await the outstanding request before calling again.
- cua.listApps(options?: { emit?: boolean }) -> { id: string, displayName: string, isRunning: true, pid: number }[] — enumerate running applications.
- cua.findApps(query: string, options?: { emit?: boolean }) -> ({ id: string, displayName: string, path: string } & ({ isRunning: true, pid: number } | { isRunning: false }))[] — search installed applications, including stopped matches; only running matches carry pid.
- cua.getState(options?: { emit?: boolean }) -> { apps: { id: string, displayName: string, isRunning: true, pid: number }[], browsers: [] } — summarize running applications; this is not an accessibility observation.
@@ -212,7 +212,7 @@ Element ids are opaque strings from the latest successful tree; the object facad
## Simulated mouse
Use an explicit window for screenshots and pointer input.
- getScreenshot(options?: { emit?: boolean }) -> { bytes: Uint8Array, mimeType: 'image/png' | 'image/jpeg' | 'image/webp' | 'image/gif', pixelWidth: number, pixelHeight: number, windowId: number } — capture the selected window as one encoded image; bytes is a JavaScript byte array, and mimeType identifies its encoding, so do not assume PNG.
- mouseMove(target: string | [number, number], options?: { relative?: boolean }) -> { dispatched: boolean, foregroundChanged?: boolean | null, hoverRoute: 'accessibility' | 'system-pointer', systemPointerMoved: boolean } — request a mouse move to the target without pressing a button; inspect hoverRoute and systemPointerMoved, and expect no automatic pointer restoration. Default hover requires both AXShowAlternateUI and AXShowDefaultUI on an enabled element or ancestor. Without an AX route it returns HOVER_UNAVAILABLE before sending input. Prefer click on a visible control or performAction with an advertised action. For web content without an AX hover action, use browser tools. Only request mouse control if the task cannot be completed with those alternatives. The question interrupts the user, and moving their pointer can disrupt their work. After approval, every mouseMove requires the target app to be in front and the point unobscured, including targets that previously supported background AX hover. If mouseMove returns FOREGROUND_REQUIRED, explain the necessary foreground step before activating the app, then observe the target again.
- mouseMove(target: string | [number, number], options?: { relative?: boolean, useSystemPointer?: boolean }) -> { dispatched: boolean, foregroundChanged?: boolean | null, hoverRoute: 'accessibility' | 'system-pointer', systemPointerMoved: boolean } — request a mouse move to the target without pressing a button; inspect hoverRoute and systemPointerMoved, and expect no automatic pointer restoration. Default hover requires both AXShowAlternateUI and AXShowDefaultUI on an enabled element or ancestor. HOVER_UNAVAILABLE sends no input and never opens a question. Use an advertised accessibility action, click a visible control, or use browser tools for web content whenever possible. Only if the task cannot be completed without moving the user’s pointer, call await win.mouseMove(target, { useSystemPointer: true }). The question interrupts the user; approval moves their pointer to this target immediately and makes subsequent ordinary moves use it until the session closes or DSH restarts. A refusal sends no input; continue with pointer-free actions. If the question is dismissed, ask again only when the user requests it. Await an outstanding request before making another. Without an approval requirement only explicitly requested moves use the system pointer. Subagents cannot request physical moves. System movement requires an unobscured foreground target; explain any necessary activation and observe the target again.
- click(target: string | [number, number], options?: { mouseButton?: 'left' | 'right' | 'middle', clickCount?: 1 | 2, modifiers?: ('command' | 'control' | 'option' | 'shift' | 'fn')[], holdMs?: number, relative?: boolean }) -> ActionResult — click an element id or point, or perform a left-button double-click; holdMs controls the press duration of each press. click is a Window method: select a window with const win = await app.getWindow(windowId), then call win.click(target). For a background control’s default action, use app.invoke(id). For a mouse click on a background app, call app.activate() first and re-read the target. Both target forms send mouse-down and mouse-up events. click(id) uses the center of the full element bounds. If that center is outside the window, take a fresh win.getScreenshot() and use click([x, y]) on a visible part of the control in screenshot pixels. A background app can ignore the events even when dispatched is true. Verify the expected effect, such as a changed selection, before typing or retrying.
- scroll(options: { x: number, y: number, deltaLinesX: number, deltaLinesY: number, relative?: boolean }) -> ActionResult — scroll at a point; deltas are integers from -10 to 10 with at least one nonzero, and positive values reveal content above or left.
- drag(from: [number, number], to: [number, number], options?: { mouseButton?: 'left' | 'right' | 'middle', modifiers?: ('command' | 'control' | 'option' | 'shift' | 'fn')[], relative?: boolean }) -> ActionResult — drag between two points of one window.
@@ -21,8 +21,8 @@
en: 2a69d89b7fc95998
zh: 084575bb31fb40dd
/dsh-cua-native-helper/observation-and-input:
en: 88ad57b61e873f32
zh: ff8a89697a6c0cc5
en: 7e63469ddd8c4e72
zh: dbc87b8d8953c7e7
/dsh-cua-native-helper/verification:
en: c87945aeec587bf5
zh: e5d69e4927e2fed3
@@ -123,7 +123,7 @@ Tree observation and action discovery treat a generic AX failure reading one ele
`paste` publishes one plain-text or HTML clipboard format, dispatches the application's enabled standard Paste menu command, and waits for a clipboard data request. It then restores the preserved formats only while its ownership marker and change count still match. The Host serializes its pastes across helpers with a file lock. maxClipboardBytes bounds both input and preserved contents; unsupported menus and oversized data fail before dispatch. A later copy is preserved. Cancellation restores owned clipboard data when possible; forced termination cannot guarantee restoration, and a data request does not prove the target completed editing. After dispatch, `pasteReadTimeoutMs` bounds waiting for clipboard consumption. Expiry returns `PASTE_NOT_CONSUMED`, closes the promised data provider, and restores the original clipboard only while still owned. The failure does not prove the app will never process its queued Paste command; inspect the target before deciding whether to retry.
`mouseMove` reveals or restores alternate UI through an enabled element or ancestor advertising both `AXShowAlternateUI` and `AXShowDefaultUI`. Without an AX route it returns `HOVER_UNAVAILABLE` before sending input. After session approval, the Host sends `useSystemPointer: true` on each `mouseMove` to select the system pointer directly.
`mouseMove` reveals or restores alternate UI through an enabled element or ancestor advertising both `AXShowAlternateUI` and `AXShowDefaultUI`. Without an AX route it returns `HOVER_UNAVAILABLE` before sending input. The Host resolves the requested route against session approval or the configured exemption before sending `useSystemPointer` to the helper.
The opt-in system pointer route returns `FOREGROUND_REQUIRED` if the target is obscured or belongs to a background app. The result reports the route and whether the visible cursor moved in `hoverRoute` and `systemPointerMoved`. It never presses a button or restores the cursor after a read. Later input may interrupt the hover. Clicks send their button events at the target without an off-window priming click.
@@ -123,7 +123,7 @@ AppKit 在主事件循环上运行。阻塞式 socket 接收与握手使用后
`paste` 提供一种纯文本或 HTML 剪贴板格式,调用应用已启用的标准 Paste 菜单命令,并等待剪贴板数据请求。随后仅在所有权标记和修改计数仍匹配时恢复原有格式。Host 通过文件锁串行化不同 helper 的粘贴。maxClipboardBytes 同时限制输入和保留内容,不支持的菜单和过大的数据在分派前失败。后续复制会被保留。取消会尽可能恢复仍持有的剪贴板数据,强制结束不保证恢复,数据请求也不证明目标已完成编辑。 分派完成后,`pasteReadTimeoutMs` 限制等待剪贴板读取的时间。到期返回 `PASTE_NOT_CONSUMED`,关闭延迟数据提供方,并仅在仍持有剪贴板时恢复原内容。该失败不能证明应用以后不会处理已排队的 Paste 命令;决定是否重试前先检查目标内容。
默认情况下,`mouseMove` 通过同一窗口内启用的控件或祖先声明的 `AXShowAlternateUI` 和 `AXShowDefaultUI` 展开或恢复界面。缺少这条路径时返回 `HOVER_UNAVAILABLE`,不发送输入。用户授权后,Host 在本会话的每次 `mouseMove` 请求中传入 `useSystemPointer: true`,直接使用系统光标。
默认情况下,`mouseMove` 通过同一窗口内启用的控件或祖先声明的 `AXShowAlternateUI` 和 `AXShowDefaultUI` 展开或恢复界面。缺少这条路径时返回 `HOVER_UNAVAILABLE`,不发送输入。Host 根据本会话的授权或免询问配置决定本次悬停的路径,再向 helper 传入 `useSystemPointer`。
开启后,系统指针路线会检查前台应用和坐标命中的窗口,不满足条件时返回 `FOREGROUND_REQUIRED`。返回值 `hoverRoute` 和 `systemPointerMoved` 说明走了哪条路线以及是否移动了可见光标。悬停不会按下鼠标,也不会在读取后还原光标。后续输入可能中断悬停。点击只在目标点发送按键事件,不先在窗口外点击。
@@ -6,7 +6,7 @@ export const PLATFORM_GUIDANCE = 'macOS: accessibility tree operations are avail
const methodDetails = new Map<string, string>([
['getAccessibilityTree', 'If the tree reports maxElements or maxDepth truncation, select a narrower window or ask the user to raise the named macOS provider setting and read again. In persistent PTC, capture a large tree with const tree = await win.getAccessibilityTree({ emit: false }). Filter its lines in memory, or save it with await tools.write({ file_path: path, content: tree }) to a new workspace file. Search its contents with tools.grep({ path, pattern }) and use tools.read around a matching line. Print only the matches you need. Use element ids from the file only while that observation is current.'],
['mouseMove', 'Default hover requires both AXShowAlternateUI and AXShowDefaultUI on an enabled element or ancestor. Without an AX route it returns HOVER_UNAVAILABLE before sending input. Prefer click on a visible control or performAction with an advertised action. For web content without an AX hover action, use browser tools. Only request mouse control if the task cannot be completed with those alternatives. The question interrupts the user, and moving their pointer can disrupt their work. After approval, every mouseMove requires the target app to be in front and the point unobscured, including targets that previously supported background AX hover. If mouseMove returns FOREGROUND_REQUIRED, explain the necessary foreground step before activating the app, then observe the target again.'],
['mouseMove', 'Default hover requires both AXShowAlternateUI and AXShowDefaultUI on an enabled element or ancestor. HOVER_UNAVAILABLE sends no input and never opens a question. Use an advertised accessibility action, click a visible control, or use browser tools for web content whenever possible. Only if the task cannot be completed without moving the user’s pointer, call await win.mouseMove(target, { useSystemPointer: true }). The question interrupts the user; approval moves their pointer to this target immediately and makes subsequent ordinary moves use it until the session closes or DSH restarts. A refusal sends no input; continue with pointer-free actions. If the question is dismissed, ask again only when the user requests it. Await an outstanding request before making another. Without an approval requirement only explicitly requested moves use the system pointer. Subagents cannot request physical moves. System movement requires an unobscured foreground target; explain any necessary activation and observe the target again.'],
['click', 'click is a Window method: select a window with const win = await app.getWindow(windowId), then call win.click(target). For a background control’s default action, use app.invoke(id). For a mouse click on a background app, call app.activate() first and re-read the target. Both target forms send mouse-down and mouse-up events. click(id) uses the center of the full element bounds. If that center is outside the window, take a fresh win.getScreenshot() and use click([x, y]) on a visible part of the control in screenshot pixels. A background app can ignore the events even when dispatched is true. Verify the expected effect, such as a changed selection, before typing or retrying.'],
])
+5 -2
View File
@@ -16,7 +16,7 @@ const CAPABILITIES: CuaStartupCapabilities = {
platform: 'macos',
methods: [
'activate', 'click', 'drag', 'findApps', 'getApp', 'getCapabilities', 'getState', 'invoke',
'listActions', 'listApps', 'listWindows', 'mouseMove', 'paste', 'performAction', 'pressKey', 'requestMouseControl', 'screenshot', 'scroll',
'listActions', 'listApps', 'listWindows', 'mouseMove', 'paste', 'performAction', 'pressKey', 'screenshot', 'scroll',
'selectText', 'sendKeyEvent', 'setValue', 'show', 'typeText',
],
appLevelAccessibility: true,
@@ -33,6 +33,8 @@ export const inject = ['computerUse', 'tools', 'systemPrompt', 'subprocess', 'ag
export interface Config extends CuaConfig {
/** Interval in milliseconds between permission and position checks while the native guide is open. */
permissionPollIntervalMs: number
/** Ask before a main task may move the system pointer. When false only explicitly requested moves use it. */
requireSystemPointerApproval: boolean
/**
* Source-build signing: `ad-hoc` (default) binds grants to the code hash;
* an object selects an existing certificate in an explicitly configured keychain.
@@ -43,6 +45,7 @@ export interface Config extends CuaConfig {
/** Validate shared operation bounds and macOS source signing before preparing the helper. */
export const Config = Schema.object({
...CuaConfigFields,
requireSystemPointerApproval: Schema.boolean().default(true),
permissionPollIntervalMs: Schema.number().step(1).min(100).max(2 ** 31 - 1).default(1000),
sourceSigning: Schema.union([
Schema.const('ad-hoc'),
@@ -62,7 +65,7 @@ export const Config = Schema.object({
*/
export async function apply(ctx: Context, config: Config): Promise<void> {
const backend = { createChannel: createSharedNativeChannel, sendKeyEvent: installHeldKeys(ctx),
guidance: GUIDANCE, capabilities: CAPABILITIES,
guidance: GUIDANCE, capabilities: CAPABILITIES, requireSystemPointerApproval: config.requireSystemPointerApproval,
permissionParameters: createPermissionParameters(config.permissionPollIntervalMs) }
if (config.helperPath !== undefined) {
if (!isAbsolute(config.helperPath)) throw new Error('dsh-cua helperPath must be absolute')
@@ -4,7 +4,7 @@ import type { CuaApi, CuaApp, CuaSurface, CuaWindow } from '@deepseek-ai/dsh-cua
import { COMMON_GUIDANCE } from '@deepseek-ai/dsh-cua/guidance'
import { GUIDANCE, PLATFORM_GUIDANCE } from '../src/guidance.ts'
const apiNames: Array<keyof CuaApi> = ['getCapabilities', 'requestMouseControl', 'listApps', 'findApps', 'getState', 'getApp']
const apiNames: Array<keyof CuaApi> = ['getCapabilities', 'listApps', 'findApps', 'getState', 'getApp']
const accessibilityNames: Array<Exclude<keyof CuaSurface, 'sendKeyEvent'>> = [
'getAccessibilityTree', 'listActions', 'invoke', 'performAction', 'setValue', 'selectText',
]
@@ -56,8 +56,8 @@ it('enumerates every available method once in its catalog entry', () => {
for (const catalog of [COMMON_GUIDANCE, GUIDANCE]) {
for (const signature of signatures) expect(catalog.split('\n').filter(line => line.startsWith(`- ${signature}`))).toHaveLength(1)
}
expect(signatures).toHaveLength(25)
expect(new Set(signatures.map(signature => signature.slice(signature.indexOf('.') + 1, -1))).size).toBe(24)
expect(signatures).toHaveLength(24)
expect(new Set(signatures.map(signature => signature.slice(signature.indexOf('.') + 1, -1))).size).toBe(23)
expect(COMMON_GUIDANCE).toContain('app.appHandle')
expect(COMMON_GUIDANCE).toContain('window.windowId')
})
@@ -33,7 +33,6 @@ describe('native protocol generation', () => {
{ method: 'drag', params: { appHandle: 'app', windowId: 1, fromX: 1, fromY: 1, toX: 2, toY: 2, steps: 1 } },
{ method: 'paste', params: { appHandle: 'app', text: 'text', maxClipboardBytes: 1024 } },
{ method: 'paste', params: { appHandle: 'app', text: 'text', pasteReadTimeoutMs: 1000 } },
{ method: 'mouseMove', params: { appHandle: 'app', windowId: 1, x: 0, y: 0, useSystemPointer: true } },
{ method: 'findApps', params: { query: 'Editor', maxAppMatches: 1 } },
{ method: 'releaseHeldKeys', params: { runId: 'execution' } },
{ method: 'launchApp', params: { appHandle: 'pending' } },
@@ -67,7 +67,7 @@ vi.mock('@deepseek-ai/dsh-cua/native-client', async importOriginal => ({
}))
const config = {
permissionPollIntervalMs: 1000,
permissionPollIntervalMs: 1000, requireSystemPointerApproval: true,
agentPreset: false,
helperPath: resolve('fixture-helper'), sourceSigning: 'ad-hoc', buildTimeoutMs: 300_000, timeoutMs: 10_000, connectTimeoutMs: 5000, screenshotTimeoutMs: 5000, axTimeoutMs: 3000, maxRequestBytes: 1024, maxResponseBytes: 4096, maxClipboardBytes: 8192, pasteReadTimeoutMs: 1000, maxAppMatches: 20,
graceMs: 10, textEventIntervalMs: 8, maxClickHoldMs: 7000, maxElements: 64, maxDepth: 8, dragDurationMs: 200, dragSteps: 10,
@@ -77,6 +77,7 @@ const { helperPath: _helperPath, ...automaticConfig } = config
it('defaults to ad-hoc signing and requires a keychain path for certificate signing', () => {
expect(NativeProvider.Config({}).sourceSigning).toBe('ad-hoc')
expect(NativeProvider.Config({}).requireSystemPointerApproval).toBe(true)
expect(NativeProvider.Config({ sourceSigning: 'ad-hoc' }).sourceSigning).toBe('ad-hoc')
for (const sourceSigning of [{ identity: 'Apple Development: Example', keychain: '/keychains/custom.keychain-db' }, { identity: 'ABC123', keychain: '/keychains/custom.keychain-db' }]) {
expect(NativeProvider.Config({ sourceSigning }).sourceSigning).toEqual(sourceSigning)
@@ -290,7 +291,7 @@ it('merges startup capabilities into runtime native fields', async () => {
expect(result.value).toEqual({ ...runtime, platform: 'macos', appLevelAccessibility: true, keyboard: ['app'], browser: false,
coordinateSpace: 'screenshot-pixels', methods: [
'activate', 'click', 'drag', 'findApps', 'getApp', 'getCapabilities', 'getState', 'invoke',
'listActions', 'listApps', 'listWindows', 'mouseMove', 'paste', 'performAction', 'pressKey', 'requestMouseControl', 'screenshot', 'scroll',
'listActions', 'listApps', 'listWindows', 'mouseMove', 'paste', 'performAction', 'pressKey', 'screenshot', 'scroll',
'selectText', 'sendKeyEvent', 'setValue', 'show', 'typeText',
] })
expect(native.request).toHaveBeenCalledExactlyOnceWith('getCapabilities', {}, expect.any(AbortSignal), ToolCallId('lifecycle-1'))
@@ -970,111 +971,149 @@ it('routes permission actions through the macOS provider with the Client languag
}, expect.any(AbortSignal))
})
it('moves only for the consenting session, without granting application access or retaining consent after reload', async () => {
const hoverTarget = { appHandle: 'app-one', snapshotId: 'snapshot', elementId: '0' }
const physicalHover = { ...hoverTarget, useSystemPointer: true }
async function hoverFixture(requireApproval = true) {
const f = await setup()
await f.ctx.plugin(UserQuestions)
const ask = vi.fn((id: string) => ({ answers: [{ id, selected: ['Allow for this session'] }] }))
f.ctx.on('user-questions/request', async ({ questions }) => ask(questions[0]!.id))
const target = { appHandle: 'app-one', snapshotId: 'snapshot', elementId: '0' }
const application = { appHandle: 'app-one', name: 'Fixture', running: true,
authorization: { id: 'a'.repeat(64), name: 'Fixture', persistable: false } }
native.request.mockResolvedValueOnce(application)
expect((await f.execute('getApp', { selector: 'Fixture' })).isError).toBe(false)
const move = async (useSystemPointer: boolean) => {
expect((await f.execute('mouseMove', target)).isError).toBe(false)
expect(native.request).toHaveBeenLastCalledWith('mouseMove', { ...target, useSystemPointer }, expect.any(AbortSignal), expect.any(String))
if (!requireApproval) {
await f.provider.dispose()
await f.ctx.plugin(NativeProvider, { ...config, requireSystemPointerApproval: false })
}
await move(false)
expect((await f.execute('mouseMove', { ...target, useSystemPointer: true })).isError).toBe(true)
for (let i = 0; i < 2; i++) expect((await f.execute('requestMouseControl')).value).toBe(true)
await move(true)
const other = await f.ctx.agents.create({ sessionId: SessionId('other-mouse-session') })
const otherCall = (method: string, params: Record<string, unknown>) => executeToolFixture(f.ctx, {
agent: other.agent, signal: new AbortController().signal, callId: ToolCallId(method), name: 'cua', arguments: { method, params },
const questions = await f.ctx.plugin(UserQuestions)
const questionId = 'cua-mouse-control'
const approved = { answers: [{ id: questionId, selected: ['Allow for this session'] }] }
const ask = vi.fn(async () => approved)
f.ctx.on('user-questions/request', async ({ questions }) => questions[0]!.id === questionId ? ask()
: { answers: [{ id: 'cua-app-access', selected: ['Allow for this session'] }] })
let index = 0
const call = (agent: typeof f.handle.agent, method: string, params: Record<string, unknown>) => executeToolFixture(f.ctx, {
agent, signal: new AbortController().signal, callId: ToolCallId(`hover-${++index}`), name: 'cua', arguments: { method, params },
})
expect(JSON.stringify((await otherCall('mouseMove', target)).content)).toContain('APP_ACCESS_REQUIRED')
native.request.mockResolvedValueOnce(application)
expect((await otherCall('getApp', { selector: 'Fixture' })).isError).toBe(false)
ask.mockImplementationOnce(id => ({ answers: [{ id, selected: ['Deny'] }] }))
expect((await otherCall('requestMouseControl', {})).value).toBe(false)
expect((await otherCall('mouseMove', target)).isError).toBe(false)
expect(native.request).toHaveBeenLastCalledWith('mouseMove', { ...target, useSystemPointer: false }, expect.any(AbortSignal), expect.any(String))
await move(true)
const select = async (agent = f.handle.agent) => {
native.request.mockResolvedValueOnce({ appHandle: 'app-one', name: 'Fixture', running: true,
authorization: { id: 'a'.repeat(64), name: 'Fixture', persistable: false } })
expect((await call(agent, 'getApp', { selector: 'Fixture' })).isError).toBe(false)
}
await select()
native.request.mockClear()
return { ...f, ask, approved, call, select, questions }
}
it('asks only on an explicit move and dispatches that same move after approval', async () => {
const f = await hoverFixture()
native.request.mockRejectedValueOnce(new Error('HOVER_UNAVAILABLE'))
expect(JSON.stringify((await f.execute('mouseMove', hoverTarget)).content)).toContain('HOVER_UNAVAILABLE')
expect(f.ask).not.toHaveBeenCalled()
const entered: PromiseWithResolvers<void> = Promise.withResolvers()
const answer = Promise.withResolvers<typeof f.approved>()
f.ask.mockImplementationOnce(() => { entered.resolve(); return answer.promise })
onTestFinished(() => { answer.resolve(f.approved) })
const pending = f.execute('mouseMove', physicalHover)
await entered.promise
expect(native.request).toHaveBeenCalledTimes(1)
expect(JSON.stringify((await f.execute('mouseMove', physicalHover)).content)).toContain('MOUSE_CONTROL_PENDING')
answer.resolve(f.approved)
expect((await pending).isError).toBe(false)
for (const [params, physical] of [
[hoverTarget, true], [{ ...hoverTarget, useSystemPointer: false }, false], [physicalHover, true],
] as const) {
expect((await f.execute('mouseMove', params)).isError).toBe(false)
expect(native.request).toHaveBeenLastCalledWith('mouseMove', { ...hoverTarget, useSystemPointer: physical }, expect.any(AbortSignal), expect.any(String))
}
expect(native.request.mock.calls[1]?.[1]).toEqual(physicalHover)
expect(f.ask).toHaveBeenCalledTimes(1)
})
it('keeps refusals and grants separate and forgets a closed session on reopen', async () => {
const f = await hoverFixture()
expect((await f.execute('mouseMove', physicalHover)).isError).toBe(false)
const { agent } = await f.ctx.agents.create({ sessionId: SessionId('other-mouse-session') })
expect(JSON.stringify((await f.call(agent, 'mouseMove', physicalHover)).content)).toContain('APP_ACCESS_REQUIRED')
await f.select(agent)
f.ask.mockResolvedValueOnce({ answers: [{ id: 'cua-mouse-control', selected: ['Deny'] }] })
native.request.mockClear()
for (let i = 0; i < 2; i++) expect(JSON.stringify((await f.call(agent, 'mouseMove', physicalHover)).content)).toContain('MOUSE_CONTROL_DENIED')
expect(native.request).not.toHaveBeenCalled()
expect((await f.call(agent, 'mouseMove', hoverTarget)).isError).toBe(false)
expect(native.request).toHaveBeenLastCalledWith('mouseMove', { ...hoverTarget, useSystemPointer: false }, expect.any(AbortSignal), expect.any(String))
expect((await f.execute('mouseMove', hoverTarget)).isError).toBe(false)
expect(native.request).toHaveBeenLastCalledWith('mouseMove', physicalHover, expect.any(AbortSignal), expect.any(String))
const sessionId = f.handle.agent.id
await f.handle.dispose()
const reopened = await f.ctx.agents.create({ sessionId })
await f.select(reopened.agent)
expect((await f.call(reopened.agent, 'mouseMove', hoverTarget)).isError).toBe(false)
expect(native.request).toHaveBeenLastCalledWith('mouseMove', { ...hoverTarget, useSystemPointer: false }, expect.any(AbortSignal), expect.any(String))
expect((await f.call(reopened.agent, 'mouseMove', physicalHover)).isError).toBe(false)
expect(f.ask).toHaveBeenCalledTimes(3)
})
it('rechecks the selection after a pending approval and does not dispatch revoked or stale input', async () => {
const f = await hoverFixture()
f.ask.mockImplementationOnce(async () => {
await f.ctx.cuaAccess.revoke('a'.repeat(64) as Parameters<typeof f.ctx.cuaAccess.revoke>[0])
return f.approved
})
expect((await f.execute('mouseMove', physicalHover)).isError).toBe(true)
expect(native.request).not.toHaveBeenCalled()
})
it('revalidates window coordinates after the user answers', async () => {
const f = await hoverFixture()
native.request.mockResolvedValueOnce([{ windowId: 7, appName: 'Fixture', title: 'Window', bounds: { x: 0, y: 0, width: 100, height: 100 },
isOnScreen: true, onCurrentSpace: true }])
await f.execute('listWindows', { appHandle: 'app-one' })
f.ask.mockImplementationOnce(async () => {
native.request.mockResolvedValueOnce([])
await f.execute('listWindows', { appHandle: 'app-one' })
return f.approved
})
const result = await f.execute('mouseMove', { appHandle: 'app-one', windowId: 7, x: 500, y: 500, relative: true, useSystemPointer: true })
expect(JSON.stringify(result.content)).toContain('STALE_SNAPSHOT')
expect(native.request.mock.calls.map(([name]) => name)).toEqual(['listWindows', 'listWindows'])
})
it('drops dismissal, aborted answers and missing human answerers without allowing input', async () => {
const f = await hoverFixture()
f.ask.mockRejectedValueOnce(new UserQuestionError('dismissed', 'ASK_CANCELLED'))
expect(JSON.stringify((await f.execute('mouseMove', physicalHover)).content)).toContain('MOUSE_APPROVAL_CANCELLED')
const controller = new AbortController()
f.ask.mockImplementationOnce(async () => { controller.abort(); return f.approved })
await Promise.allSettled([f.execute('mouseMove', physicalHover, controller.signal)])
expect(native.request).not.toHaveBeenCalled()
expect((await f.execute('mouseMove', physicalHover)).isError).toBe(false)
expect(f.ask).toHaveBeenCalledTimes(3)
await f.provider.dispose()
await f.ctx.plugin(NativeProvider, config)
expect((await f.execute('requestMouseControl')).value).toBe(true)
expect(ask.mock.calls.map(([id]) => id)).toEqual(['cua-app-access', 'cua-mouse-control', 'cua-app-access', 'cua-mouse-control', 'cua-mouse-control'])
await f.select()
await f.questions.dispose()
expect(JSON.stringify((await f.execute('mouseMove', physicalHover)).content)).toContain('MOUSE_APPROVAL_UNAVAILABLE')
})
it.each(['Deny', 'custom'] as const)('remembers a %s answer without moving the pointer or asking again', async (answer) => {
const f = await setup()
await f.ctx.plugin(UserQuestions)
const ask = vi.fn(() => {
return { answers: [{ id: 'cua-mouse-control', selected: [answer === 'custom' ? 'Allow for this session' : answer],
...(answer === 'custom' ? { custom: 'maybe later' } : {}) }] }
})
f.ctx.on('user-questions/request', async () => ask())
expect((await f.execute('requestMouseControl')).value).toBe(false)
expect((await f.execute('requestMouseControl')).value).toBe(false)
expect(ask).toHaveBeenCalledTimes(1)
expect(native.request).not.toHaveBeenCalled()
it('allows only the explicitly requested physical move when approval is disabled', async () => {
const f = await hoverFixture(false)
for (const [target, expected] of [[hoverTarget, false], [physicalHover, true], [hoverTarget, false]] as const) {
expect((await f.execute('mouseMove', target)).isError).toBe(false)
expect(native.request).toHaveBeenLastCalledWith('mouseMove', { ...hoverTarget, useSystemPointer: expected }, expect.any(AbortSignal), expect.any(String))
}
expect(f.ask).not.toHaveBeenCalled()
})
it('allows a fresh request after dismissal without authorizing the first one', async () => {
const f = await setup()
await f.ctx.plugin(UserQuestions)
const ask = vi.spyOn(f.ctx.userQuestions, 'ask')
.mockRejectedValueOnce(new UserQuestionError('User dismissed the request', 'ASK_CANCELLED'))
.mockResolvedValueOnce({ answers: [{ id: 'cua-mouse-control', selected: ['Allow for this session'] }] })
const dismissed = await f.execute('requestMouseControl')
expect(dismissed.isError).toBe(true)
expect(JSON.stringify(dismissed.content)).toContain('MOUSE_APPROVAL_CANCELLED')
expect((await f.execute('requestMouseControl')).value).toBe(true)
expect(ask).toHaveBeenCalledTimes(2)
expect(native.request).not.toHaveBeenCalled()
})
it('refuses delegated mouse control before asking the user even when the parent has approval', async () => {
const f = await setup()
await f.ctx.plugin(UserQuestions)
const ask = vi.spyOn(f.ctx.userQuestions, 'ask').mockResolvedValue({
answers: [{ id: 'cua-mouse-control', selected: ['Allow for this session'] }],
})
expect((await f.execute('requestMouseControl')).value).toBe(true)
ask.mockClear()
it.each([true, false])('rejects a delegated pointer move before asking or dispatching with approval=%s', async (approval) => {
const f = await hoverFixture(approval)
const { agent } = await f.ctx.agents.create({ sessionId: SessionId('delegated-mouse-session'),
meta: { origin: 'subagent', parentSession: f.handle.agent.id } })
const refused = await executeToolFixture(f.ctx, { agent, signal: new AbortController().signal,
callId: ToolCallId('delegated-mouse-control'), name: 'cua', arguments: { method: 'requestMouseControl', params: {} } })
expect(refused.isError).toBe(true)
// This case starts with application access already granted to the child.
const selectAnswer = vi.spyOn(f.ctx.userQuestions, 'ask')
.mockResolvedValueOnce({ answers: [{ id: 'cua-app-access', selected: ['Allow for this session'] }] })
await f.select(agent)
selectAnswer.mockClear()
native.request.mockClear()
const refused = await f.call(agent, 'mouseMove', physicalHover)
expect(JSON.stringify(refused.content)).toContain('MOUSE_CONTROL_UNAVAILABLE')
expect(JSON.stringify(refused.content)).toContain('report that step in your final result for the main task')
expect(ask).not.toHaveBeenCalled()
expect(native.request).not.toHaveBeenCalled()
})
it('drops late approval after cancellation and admits only one pending question', async () => {
const f = await setup()
expect((await f.execute('requestMouseControl')).isError).toBe(true)
await f.ctx.plugin(UserQuestions)
const entered = Promise.withResolvers<undefined>()
let count = 0
const release = Promise.withResolvers<undefined>()
f.ctx.on('user-questions/request', async () => {
count++
entered.resolve(undefined)
await release.promise
return { answers: [{ id: 'cua-mouse-control', selected: ['Allow for this session'] }] }
})
const controller = new AbortController()
const first = f.execute('requestMouseControl', {}, controller.signal)
await entered.promise
expect(JSON.stringify((await f.execute('requestMouseControl')).content)).toContain('MOUSE_CONTROL_PENDING')
controller.abort()
release.resolve(undefined)
await Promise.allSettled([first])
expect((await f.execute('requestMouseControl')).value).toBe(true)
expect(count).toBe(2)
expect(selectAnswer).not.toHaveBeenCalled()
expect(f.ask).not.toHaveBeenCalled()
expect(native.request).not.toHaveBeenCalled()
})
@@ -1089,8 +1128,8 @@ it('keeps provider-specific pointer flags and questions off providers without mo
keyboard: ['app'], coordinateSpace: 'screenshot-pixels', browser: false },
})
} })
expect(JSON.stringify((await f.execute('requestMouseControl')).content)).toContain('UNSUPPORTED_OPERATION')
await selectApplication(f)
expect(JSON.stringify((await f.execute('mouseMove', physicalHover)).content)).toContain('UNSUPPORTED_OPERATION')
const target = { appHandle: 'app-one', snapshotId: 'snapshot', elementId: '0' }
expect((await f.execute('mouseMove', target)).isError).toBe(false)
expect(native.request).toHaveBeenLastCalledWith('mouseMove', target, expect.any(AbortSignal), expect.any(String))
+2 -2
View File
@@ -15,8 +15,8 @@
en: 5db4d4fea25493b9
zh: 65ab64fc24eb0d09
/deepseek-ai-dsh-cua/use-this-package:
en: a55ba012aedca751
zh: fcddea780c0ec58e
en: 5e95c138cb107385
zh: b126c56fb7d35538
/deepseek-ai-dsh-cua/use-this-package/permission-controls:
en: d92d5948a7941e8d
zh: d43491ae6521b87e
+1 -1
View File
@@ -41,7 +41,7 @@ Observation methods emit and return their result. Use `void await app.getAccessi
Application consent precedes launch, observation, capture, and input. Session access is stored by conversation and application identity and survives DSH restart and Computer Use disablement. Other conversations require their own consent. Saved grants require a platform-provided reusable application identity. Refusal and revocation close affected Agent connections, so their old objects require fresh selection. No input is replayed after cancellation, timeout, or reconnect.
Use `cua.requestMouseControl()` only when the task requires moving the user’s pointer. Approval applies to `mouseMove` in this session until DSH restarts. Actions return a `dispatched` boolean; `mouseMove` also reports `hoverRoute` and `systemPointerMoved`, and `show()` adds `windowId` and `foregroundChanged`. Read fresh state to verify the result. Screenshots and pointer actions are available only on explicit window objects. Coordinate actions default to pixels in the latest returned screenshot; `relative: true` accepts integers from 0 to 999 after a successful window observation. Values are divided by 1000 and scaled to the window dimensions, then clamped inside the window. The Host resolves attachment normalization and the selected model route’s image projection before returning the screenshot, then converts input to the provider’s native window coordinates. Resizing invalidates the screenshot mapping; `SCREENSHOT_REQUIRED` requires a new screenshot for absolute coordinates. Resizing rejects old coordinates with `WINDOW_CHANGED`; movement alone preserves window-local coordinates.
On supported providers the main task can request a physical move with `win.mouseMove(target, { useSystemPointer: true })`. Actions return a `dispatched` boolean; `mouseMove` also reports `hoverRoute` and `systemPointerMoved`, and `show()` adds `windowId` and `foregroundChanged`. Read fresh state to verify the result. Screenshots and pointer actions are available only on explicit window objects. Coordinate actions default to pixels in the latest returned screenshot; `relative: true` accepts integers from 0 to 999 after a successful window observation. Values are divided by 1000 and scaled to the window dimensions, then clamped inside the window. The Host resolves attachment normalization and the selected model route’s image projection before returning the screenshot, then converts input to the provider’s native window coordinates. Resizing invalidates the screenshot mapping; `SCREENSHOT_REQUIRED` requires a new screenshot for absolute coordinates. Resizing rejects old coordinates with `WINDOW_CHANGED`; movement alone preserves window-local coordinates.
`click(target, { holdMs: 800 })` holds the mouse button for 800 milliseconds before release. `holdMs` applies only to `click`; it accepts nonnegative integers and applies to each press of a double-click; zero releases without an additional hold. The provider config `maxClickHoldMs` limits explicit holds and defaults to 5000 ms. Explicit holds use pointer input; omission preserves ordinary click behavior. The operation deadline includes the hold, and cancellation or timeout releases the button. Received input cannot be undone; forced helper termination cannot guarantee release.
+1 -1
View File
@@ -41,7 +41,7 @@ kind: "package-reference"
应用授权发生在启动、观察、截图和输入之前。会话许可按对话和应用身份持久化,重启 DSH 或关闭 Computer Use 后仍可复用,其他对话需要单独授权。长期许可要求平台提供可复用的应用身份。拒绝或撤销会关闭受影响的 Agent 连接,因此旧对象必须重新选择目标。取消、超时或重新连接后均不重放输入。
只有任务确实需要移动用户的鼠标时,才调用 `cua.requestMouseControl()` 询问用户。授权仅适用于当前会话的 `mouseMove`,重启 DSH 后失效。动作返回 `dispatched` 布尔值;`mouseMove` 还会返回 `hoverRoute` 和 `systemPointerMoved`,`show()` 还会返回 `windowId` 和 `foregroundChanged`。请读取新状态以验证结果。截图和指针操作只能通过显式窗口对象使用。坐标操作默认使用最近返回截图中的像素;成功观察窗口后,传入 `relative: true` 可使用 0~999 的整数坐标。数值除以 1000 后按窗口尺寸缩放,并限制在窗口内。Host 在返回截图前解析附件归一化和当前模型的图片缩放规则,再将输入换算为提供方的原生窗口坐标。窗口尺寸变化会使截图映射失效,绝对坐标遇到 `SCREENSHOT_REQUIRED` 时需要重新截图。窗口缩放后以 `WINDOW_CHANGED` 拒绝旧坐标;单纯移动窗口不改变窗口内坐标。
在支持系统鼠标的提供方上,主任务可通过 `win.mouseMove(target, { useSystemPointer: true })` 请求真实移动。动作返回 `dispatched` 布尔值;`mouseMove` 还会返回 `hoverRoute` 和 `systemPointerMoved`,`show()` 还会返回 `windowId` 和 `foregroundChanged`。请读取新状态以验证结果。截图和指针操作只能通过显式窗口对象使用。坐标操作默认使用最近返回截图中的像素;成功观察窗口后,传入 `relative: true` 可使用 0~999 的整数坐标。数值除以 1000 后按窗口尺寸缩放,并限制在窗口内。Host 在返回截图前解析附件归一化和当前模型的图片缩放规则,再将输入换算为提供方的原生窗口坐标。窗口尺寸变化会使截图映射失效,绝对坐标遇到 `SCREENSHOT_REQUIRED` 时需要重新截图。窗口缩放后以 `WINDOW_CHANGED` 拒绝旧坐标;单纯移动窗口不改变窗口内坐标。
`click(target, { holdMs: 800 })` 按住鼠标按钮 800 毫秒后松开。`holdMs` 仅适用于 `click`;它接受非负整数,双击时应用于每次按下;0 表示按下后立即松开,不额外等待。提供方配置 `maxClickHoldMs` 限制显式按住时长,默认 5000 毫秒。指定时长后使用鼠标事件;省略时保留普通点击行为。按住时间计入操作超时,取消或超时会松开按钮。已接收的输入无法撤销;辅助进程被强制终止时无法保证松开。
+7 -7
View File
@@ -1,6 +1,6 @@
/** Object facade over logged native operations; its caller owns evaluator persistence. */
import { Buffer } from 'node:buffer'
import { z } from 'zod'
import type { z } from 'zod'
import type { JsonValue } from '@deepseek-ai/dsh-util-values'
import { ActionNamesResult, ActionResult, AppResult, AppsResult, CapabilitiesResult, FoundAppsResult, HoverResult, Request, ScreenshotResult, ShowResult, StateResult, WindowsResult } from './protocol.ts'
import type { CuaApi, CuaApp, CuaAppHandle, CuaCapabilities, CuaDispatch, CuaObservationOptions, CuaObservationSink, CuaOperation, CuaScreenshot, CuaSurface, CuaWindow } from './types.ts'
@@ -51,9 +51,6 @@ export function createCuaApi(dispatch: CuaDispatch, sink?: CuaObservationSink):
const objects = createCuaObjects(call, sink)
const api: CuaApi = {
async requestMouseControl() {
return z.boolean().parse(await call('requestMouseControl', {}))
},
async getCapabilities(options) {
const result = CapabilitiesResult.parse(await call('getCapabilities', {}))
capabilities = result
@@ -203,12 +200,15 @@ function createCuaObjects(call: ReturnType<typeof validatedCall>, sink: CuaObser
return screenshot
},
async mouseMove(targetPoint, options) {
const relative = options?.relative === undefined ? {} : { relative: options.relative }
const movement = {
...options?.relative === undefined ? {} : { relative: options.relative },
...options?.useSystemPointer === undefined ? {} : { useSystemPointer: options.useSystemPointer },
}
if (typeof targetPoint === 'string') {
return HoverResult.parse(await base.call('mouseMove', { ...base.elementParams(), elementId: targetPoint, ...relative }))
return HoverResult.parse(await base.call('mouseMove', { ...base.elementParams(), elementId: targetPoint, ...movement }))
}
const [x, y] = targetPoint
return HoverResult.parse(await base.call('mouseMove', { ...coordinateParams(appHandle, windowId), x, y, ...relative }))
return HoverResult.parse(await base.call('mouseMove', { ...coordinateParams(appHandle, windowId), x, y, ...movement }))
},
async click(targetPoint, options) {
const mouseButton = options?.mouseButton ?? 'left'
@@ -231,16 +231,17 @@ export class ApplicationAccess {
* @param handle Selected application.
* @param signal Calling tool lifetime.
* @param dispatch One native operation, with revocation cancellation.
* @param prepare Optional interactive preparation outside the application lock. Dispatch rechecks access after it settles.
* @returns The native result, with no implicit retry after consent or cancellation.
*/
async run<T>(client: NativeClient, handle: CuaAppHandle, signal: AbortSignal,
dispatch: (signal: AbortSignal) => Promise<T>): Promise<T> {
dispatch: (signal: AbortSignal) => Promise<T>, prepare?: (signal: AbortSignal) => Promise<void>): Promise<T> {
const state = this.clients.get(client)
const selected = state?.handles.get(handle)
if (!state || !selected) throw new NativeError('APP_ACCESS_REQUIRED', 'Select the application with getApp and obtain user consent before reading or controlling it')
if (selected.lifetime.signal.aborted) throw new NativeError('APP_ACCESS_REVOKED', 'Application access was revoked; stop using this application')
const lifetime = AbortSignal.any([signal, this.lifetime.signal, selected.lifetime.signal])
return this.options.store.withApplication(selected.identity.id, lifetime, async () => {
const requireAccess = async () => {
const permitted = await this.options.store.withState(lifetime, ({ state: permissions }) =>
this.permitted(selected, permissions.applications[selected.identity.id] ?? this.empty()))
if (!permitted) {
@@ -248,6 +249,14 @@ export class ApplicationAccess {
throw new NativeError('APP_ACCESS_REVOKED', 'Application access was revoked; stop using this application')
}
if (client.disposed) throw new NativeError('NATIVE_CLOSED', 'Helper is closed; acquire fresh application handles')
}
if (prepare) {
await requireAccess()
await prepare(lifetime)
lifetime.throwIfAborted()
}
return this.options.store.withApplication(selected.identity.id, lifetime, async () => {
await requireAccess()
return this.withOperation(state.agent, client, selected.identity, lifetime, dispatch)
})
}
+1 -2
View File
@@ -8,7 +8,6 @@ Signatures show resolved return values after await. ActionResult = { dispatched:
Setup and discovery:
- cua.getCapabilities(options?: { emit?: boolean }) -> { platform: string, permissions: { accessibility: boolean, screenRecording: boolean }, appLevelAccessibility: boolean, keyboard: ('app' | 'window')[], sessionLocked: boolean | null, methods: string[], coordinateSpace: 'screenshot-pixels', browser: false } — inspect reported permissions, sessionLocked, methods, and keyboard scopes without requesting permissions.
- cua.requestMouseControl() -> boolean — ask the user to allow mouseMove to move their system pointer for this session until DSH restarts. Only the main task may ask, and only when a physical hover is necessary. A false result is a refusal for this session until restart. If the user dismisses the request, continue without mouse control unless they ask to reconsider. Await the outstanding request before calling again.
- cua.listApps(options?: { emit?: boolean }) -> { id: string, displayName: string, isRunning: true, pid: number }[] — enumerate running applications.
- cua.findApps(query: string, options?: { emit?: boolean }) -> ({ id: string, displayName: string, path: string } & ({ isRunning: true, pid: number } | { isRunning: false }))[] — search installed applications, including stopped matches; only running matches carry pid.
- cua.getState(options?: { emit?: boolean }) -> { apps: { id: string, displayName: string, isRunning: true, pid: number }[], browsers: [] } — summarize running applications; this is not an accessibility observation.
@@ -33,7 +32,7 @@ Element ids are opaque strings from the latest successful tree; the object facad
## Simulated mouse
Use an explicit window for screenshots and pointer input.
- getScreenshot(options?: { emit?: boolean }) -> { bytes: Uint8Array, mimeType: 'image/png' | 'image/jpeg' | 'image/webp' | 'image/gif', pixelWidth: number, pixelHeight: number, windowId: number } — capture the selected window as one encoded image; bytes is a JavaScript byte array, and mimeType identifies its encoding, so do not assume PNG.
- mouseMove(target: string | [number, number], options?: { relative?: boolean }) -> { dispatched: boolean, foregroundChanged?: boolean | null, hoverRoute: 'accessibility' | 'system-pointer', systemPointerMoved: boolean } — request a mouse move to the target without pressing a button; inspect hoverRoute and systemPointerMoved, and expect no automatic pointer restoration.
- mouseMove(target: string | [number, number], options?: { relative?: boolean, useSystemPointer?: boolean }) -> { dispatched: boolean, foregroundChanged?: boolean | null, hoverRoute: 'accessibility' | 'system-pointer', systemPointerMoved: boolean } — request a mouse move to the target without pressing a button; inspect hoverRoute and systemPointerMoved, and expect no automatic pointer restoration.
- click(target: string | [number, number], options?: { mouseButton?: 'left' | 'right' | 'middle', clickCount?: 1 | 2, modifiers?: ('command' | 'control' | 'option' | 'shift' | 'fn')[], holdMs?: number, relative?: boolean }) -> ActionResult — click an element id or point, or perform a left-button double-click; holdMs controls the press duration of each press.
- scroll(options: { x: number, y: number, deltaLinesX: number, deltaLinesY: number, relative?: boolean }) -> ActionResult — scroll at a point; deltas are integers from -10 to 10 with at least one nonzero, and positive values reveal content above or left.
- drag(from: [number, number], to: [number, number], options?: { mouseButton?: 'left' | 'right' | 'middle', modifiers?: ('command' | 'control' | 'option' | 'shift' | 'fn')[], relative?: boolean }) -> ActionResult — drag between two points of one window.
+25 -9
View File
@@ -127,6 +127,8 @@ export interface CuaBackend {
createChannel: NativeChannelFactory
guidance: string
capabilities: CuaStartupCapabilities
/** Whether explicit system-pointer moves require consent. Omit for providers without this route. */
requireSystemPointerApproval?: boolean
/** Dispatch a directional key through the provider's execution-scoped input owner, after application authorization. */
sendKeyEvent?: (client: NativeClient, params: Record<string, JsonValue>,
execution: Readonly<ToolExecution>) => Promise<JsonValue>
@@ -160,7 +162,8 @@ export function installCua(ctx: Context, config: CuaConfig, backend: CuaBackend)
permissionParameters: backend.permissionParameters,
store: new CuaPermissionStore(ctx, config.allowedApps.map(grant => ({ ...grant, id: grant.id as CuaApplicationId })), config.timeoutMs),
})
const mouseControl = backend.capabilities.methods.includes('requestMouseControl') ? new MouseControl(ctx) : undefined
const mouseControl = backend.requireSystemPointerApproval === undefined
? undefined : new MouseControl(ctx, backend.requireSystemPointerApproval)
const clients = new Map<Agent, NativeClient>()
const lifetime = new AbortController()
const pending = new Map<Promise<unknown>, Agent>()
@@ -209,7 +212,7 @@ export function installCua(ctx: Context, config: CuaConfig, backend: CuaBackend)
for (const result of [...await permissions, ...results, ...releases]) if (result.status === 'rejected') errors.push(result.reason)
if (errors.length > 0) throw new AggregateError(errors, 'dsh-cua helper cleanup failed')
}
const callNative = async (request: CuaOperation, execution: Readonly<CuaExecution>) => {
const callNative = async (request: CuaOperation, execution: Readonly<CuaExecution>, pointerRoute?: boolean) => {
const { method, params: parsedParams } = request
const params = execution.coordinates.parameters(request)
const agent = execution.agent
@@ -226,7 +229,10 @@ export function installCua(ctx: Context, config: CuaConfig, backend: CuaBackend)
? { ...params, maxElements: config.maxElements, maxDepth: config.maxDepth }
: method === 'drag' ? { ...params, durationMs: config.dragDurationMs, steps: config.dragSteps }
: params
if (method === 'mouseMove' && mouseControl) nativeParams.useSystemPointer = mouseControl.allowed(agent)
if (method === 'mouseMove') {
delete nativeParams.useSystemPointer
if (mouseControl) nativeParams.useSystemPointer = pointerRoute ?? mouseControl.allowed(agent)
}
const requestNative = () => client.request(method, nativeParams, signal, execution.callId)
if (method === 'screenshot') {
const { attachments, target } = await screenshotStore(ctx, execution)
@@ -313,10 +319,6 @@ export function installCua(ctx: Context, config: CuaConfig, backend: CuaBackend)
const signal = AbortSignal.any([execution.signal, lifetime.signal, owner.signal])
signal.throwIfAborted()
const operation = (async () => {
if (request.method === 'requestMouseControl') {
if (!mouseControl) throw new NativeError('UNSUPPORTED_OPERATION', 'This provider does not support system mouse control')
return mouseControl.request(agent, signal)
}
const previous = clients.get(agent)
if (previous?.disposed) {
try { await previous.dispose() } catch (error) {
@@ -342,14 +344,28 @@ export function installCua(ctx: Context, config: CuaConfig, backend: CuaBackend)
owner.coordinates = new CuaCoordinates()
}
const nativeClient = client
let pointerRoute: boolean | undefined
let prepare: ((signal: AbortSignal) => Promise<void>) | undefined
if (request.method === 'mouseMove') {
if (request.params.useSystemPointer === true) {
prepare = async (signal) => {
if (!mouseControl) throw new NativeError('UNSUPPORTED_OPERATION', 'This provider does not support system mouse control')
owner.coordinates.parameters(request)
if (!await mouseControl.request(agent, signal)) {
throw new NativeError('MOUSE_CONTROL_DENIED', 'The user declined system-pointer movement for this session. No input was sent. Use pointer-free actions.')
}
pointerRoute = true
}
} else pointerRoute = request.params.useSystemPointer
}
const dispatch = async (signal: AbortSignal) => {
const scoped = { ...execution, agent, signal, nativeClient, coordinates: owner.coordinates }
const result = await callNative(request, scoped)
const result = await callNative(request, scoped, pointerRoute)
signal.throwIfAborted()
return result
}
return 'appHandle' in request.params
? access.run(nativeClient, request.params.appHandle as CuaAppHandle, signal, dispatch)
? access.run(nativeClient, request.params.appHandle as CuaAppHandle, signal, dispatch, prepare)
: dispatch(signal)
})()
pending.set(operation, agent)
+8 -8
View File
@@ -117,14 +117,14 @@ export const mouseControlDisplay = {
pt: 'Permitir que o agente mova o ponteiro do mouse?',
},
detail: {
en: 'Hovering will move your pointer and leave it at the target, even while you are using the mouse. This permission applies only to this session until DSH restarts. If you decline, this session will not ask again before restart.',
zh: '智能体悬停时会移动你的鼠标指针,并将它留在目标位置,即使你正在使用鼠标。授权仅限当前会话,重启 DSH 后失效。如果拒绝,重启前本会话不会再次询问。',
ja: 'ホバーすると、マウスを使用中でもポインタが移動し、移動先に残ります。許可はこのセッションに限り、DSH を再起動すると失効します。拒否すると、再起動までこのセッションでは再度確認しません。',
es: 'Al pasar sobre un elemento, el puntero se moverá y quedará allí aunque estés usando el ratón. El permiso solo se aplica a esta sesión y caduca al reiniciar DSH. Si lo deniegas, esta sesión no volverá a preguntar hasta que reinicies DSH.',
fr: 'Lors d’un survol, le pointeur se déplace et reste sur la cible, même si vous utilisez la souris. Cette autorisation vaut pour cette session et expire au redémarrage de DSH. Si vous refusez, cette session ne redemandera pas avant le redémarrage.',
de: 'Beim Überfahren eines Ziels bewegt sich der Zeiger und bleibt dort, auch wenn Sie die Maus verwenden. Die Erlaubnis gilt nur für diese Sitzung und erlischt beim Neustart von DSH. Wenn Sie ablehnen, fragt diese Sitzung bis zum Neustart nicht erneut.',
ru: 'При наведении указатель переместится и останется на цели, даже если вы пользуетесь мышью. Разрешение действует только в этом сеансе до перезапуска DSH. Если вы откажете, запрос в этом сеансе не повторится до перезапуска.',
pt: 'Ao passar sobre um elemento, o ponteiro se moverá e ficará no destino, mesmo enquanto você usa o mouse. A permissão vale apenas para esta sessão e expira ao reiniciar o DSH. Se você recusar, esta sessão não perguntará novamente até a reinicialização.',
en: 'Allowing moves your pointer to this target now. Later hovers can also move it. If you decline, this session will not ask again. The decision expires when you close the session or restart DSH.',
zh: '允许后,你的鼠标指针会立即移到目标位置,本会话后续悬停也会移动指针。如果拒绝,本会话不会再询问。关闭会话或重启 DSH 后,这个决定失效。',
ja: '許可するとポインタが今すぐ移動し、以後のホバーでも移動します。拒否すると、このセッションでは再度確認しません。セッションを閉じるか DSH を再起動すると、この選択は失効します。',
es: 'Si aceptas, el puntero se moverá ahora al destino y también podrá moverse al pasar sobre otros elementos. Si rechazas, no se volverá a preguntar. La decisión caduca al cerrar la sesión o reiniciar DSH.',
fr: 'Si vous acceptez, le pointeur se déplace immédiatement sur la cible, puis lors des survols suivants. Si vous refusez, la question ne sera plus posée. Ce choix expire à la fermeture de la session ou au redémarrage de DSH.',
de: 'Bei Zustimmung bewegt sich der Zeiger sofort zum Ziel und auch bei späteren Mausbewegungen. Bei Ablehnung wird nicht erneut gefragt. Die Entscheidung gilt, bis Sie die Sitzung schließen oder DSH neu starten.',
ru: 'При разрешении указатель сразу переместится на цель и сможет перемещаться при следующих наведениях. При отказе вопрос не повторится. Решение действует до закрытия сеанса или перезапуска DSH.',
pt: 'Se permitir, o ponteiro irá para o alvo agora e poderá se mover nas próximas ações. Se recusar, a pergunta não será repetida. A decisão expira ao encerrar a sessão ou reiniciar o DSH.',
},
approve: applicationAccessDisplay.approve,
}
@@ -7,9 +7,9 @@ import { NativeError } from './native-client.ts'
/** Mouse control applies only to mouseMove and never grants access to an application. */
export class MouseControl {
private readonly decisions = new Map<Agent['id'], boolean | 'asking'>()
private readonly decisions = new WeakMap<Agent, boolean | 'asking'>()
constructor(private readonly ctx: Context) {}
constructor(private readonly ctx: Context, private readonly requireApproval: boolean) {}
/**
* Select the native route without prompting the user.
@@ -17,7 +17,7 @@ export class MouseControl {
* @returns Whether this Session explicitly approved moving the system pointer.
*/
allowed(agent: Agent): boolean {
return this.decisions.get(agent.id) === true
return this.decisions.get(agent) === true
}
/**
@@ -30,12 +30,13 @@ export class MouseControl {
if (agent.session.header.origin === 'subagent') {
throw new NativeError('MOUSE_CONTROL_UNAVAILABLE', "Delegated tasks cannot move the user's pointer. Use AX hover, click, or performAction. If the task requires a physical hover, report that step in your final result for the main task.")
}
const decision = this.decisions.get(agent.id)
if (!this.requireApproval) return true
const decision = this.decisions.get(agent)
if (decision === 'asking') throw new NativeError('MOUSE_CONTROL_PENDING', 'Wait for the pending mouse control request in this session before continuing')
if (decision !== undefined) return decision
const questions = this.ctx.get('userQuestions')
if (!questions) throw new NativeError('MOUSE_APPROVAL_UNAVAILABLE', 'No human answerer is available for mouse control')
this.decisions.set(agent.id, 'asking')
this.decisions.set(agent, 'asking')
const id = 'cua-mouse-control'
const approve = mouseControlDisplay.approve.en
const reject = 'Deny'
@@ -49,10 +50,10 @@ export class MouseControl {
const [answer] = result.answers
const granted = result.answers.length === 1 && answer?.id === id && !answer.custom?.trim()
&& answer.selected.length === 1 && answer.selected[0] === approve
this.decisions.set(agent.id, granted)
this.decisions.set(agent, granted)
return granted
} catch (error) {
this.decisions.delete(agent.id)
this.decisions.delete(agent)
if (error instanceof UserQuestionError && error.code === 'ASK_CANCELLED') {
throw new NativeError('MOUSE_APPROVAL_CANCELLED', 'The mouse control request was dismissed. Continue without mouse control unless the user asks to reconsider', { cause: error })
}
@@ -1,7 +1,7 @@
/** Build-time native admission metadata derived from the shared operation definitions. */
import { z } from 'zod'
import type { JSONSchema } from 'zod/v4/core'
import { NativeOnlyOperations, NativeResourceParameters, NativeToolOperations, type Request } from './protocol.ts'
import { NativeOnlyOperations, NativeResourceParameters, Request } from './protocol.ts'
type Schema = JSONSchema.JSONSchema
@@ -26,7 +26,7 @@ export function nativeProtocol(
json: string
} {
const resources = new Map(Object.entries({ ...NativeResourceParameters, ...platformFields }))
const schemas = [...NativeToolOperations, ...NativeOnlyOperations, ...privateOperations]
const schemas = [...Request.options, ...NativeOnlyOperations, ...privateOperations]
.map(schema => z.toJSONSchema(schema))
const fields = new Map<string, Set<string>>()
const enums = new Map<string, Map<string, Set<string>>>()
+7 -7
View File
@@ -34,8 +34,10 @@ const imageBase64 = z.string().min(1).refine(value => Buffer.from(value, 'base64
message: 'Expected canonical base64 image data',
})
/** Operations shared by the model and native helper, before Host parameters are added. */
export const NativeToolOperations = [
const mouseMove = { useSystemPointer: z.boolean().optional().describe('Request use of the system pointer. The Host checks permission before dispatch.') }
/** Model-submittable operations, before the Host applies permission decisions and resource limits. */
export const Request = z.discriminatedUnion('method', [
operation('getCapabilities', {}), operation('listApps', {}),
operation('findApps', { query: z.string().min(1, 'Expected a non-empty search string').regex(/\S/, 'Expected a non-empty search string') }),
operation('getApp', { selector: z.string().min(1) }),
@@ -50,7 +52,8 @@ export const NativeToolOperations = [
operation('performAction', { ...snapshot, action: z.string().min(1), argument: jsonValue.optional() }),
operation('invoke', snapshot),
z.object({ method: z.literal('mouseMove'), params: z.union([
z.object(snapshot).strict(), z.object(absoluteCoordinate).strict(), z.object(relativeCoordinate).strict(),
z.object({ ...snapshot, ...mouseMove }).strict(), z.object({ ...absoluteCoordinate, ...mouseMove }).strict(),
z.object({ ...relativeCoordinate, ...mouseMove }).strict(),
]) }).strict(),
z.object({ method: z.literal('click'), params: z.union([
z.object({ ...snapshot, ...click }).strict(),
@@ -77,10 +80,7 @@ export const NativeToolOperations = [
fromX: relativePoint, fromY: relativePoint, toX: relativePoint, toY: relativePoint }).strict(),
]) }).strict(),
operation('screenshot', window),
] as const
/** Model-submittable operations. The Host answers mouse-control requests without dispatching native input. */
export const Request = z.discriminatedUnion('method', [...NativeToolOperations, operation('requestMouseControl', {})])
])
/** Resource parameters supplied by the Host, never accepted from model tool arguments. */
export const NativeResourceParameters = {
+3 -8
View File
@@ -196,10 +196,11 @@ export interface CuaWindow extends CuaSurface {
* Returning and subsequent observations do not restore the previous pointer position.
* Other input and application changes can interrupt the interaction; observe before continuing.
* @param target - current element id or screenshot-local [x,y].
* @param options - coordinate mode; applies only to [x,y].
* @param options - Coordinate mode for points and optional system-pointer request. False explicitly selects pointer-free hover.
* @returns dispatch evidence and the selected route; observe the resulting UI after any UI delay.
*/
mouseMove(target: CuaElementId | readonly [number, number], options?: CuaCoordinateOptions): Promise<CuaHoverResult>
mouseMove(target: CuaElementId | readonly [number, number],
options?: CuaCoordinateOptions & { useSystemPointer?: boolean }): Promise<CuaHoverResult>
/**
* Click a current element id or screenshot-local point.
* Cancellation or timeout releases the held button; received input cannot be undone. The operation deadline includes holdMs.
@@ -246,12 +247,6 @@ export interface CuaApp extends CuaSurface {
}
/** Native app API; its caller owns global bindings and variable persistence. */
export interface CuaApi {
/**
* Ask to move the user's pointer with mouseMove for this session until restart. Only the main task can ask.
* @returns True on approval, false on refusal. Both decisions last until restart for this session.
* @throws If the user dismisses the question without deciding or another request is pending.
*/
requestMouseControl(): Promise<boolean>
/**
* Inspect permissions without requesting them.
* @param options - output options.
+11 -6
View File
@@ -891,10 +891,15 @@ it('expands menu contents and details only for the requested observation', async
expect(f.dispatch).toHaveBeenLastCalledWith({ method: 'getState', params: { appHandle: selected.appHandle, includeMenus: false, details: false } })
})
it('requests mouse control without dispatching a move and returns the decision', async () => {
const dispatch = vi.fn().mockResolvedValueOnce(false).mockResolvedValueOnce(true)
const cua = createCuaApi(dispatch)
expect(await cua.requestMouseControl()).toBe(false)
expect(await cua.requestMouseControl()).toBe(true)
expect(dispatch.mock.calls).toEqual(Array(2).fill([{ method: 'requestMouseControl', params: {} }]))
it('passes explicit system-pointer intent for both mouseMove target forms', async () => {
const f = fixture()
const win = await (await f.api.getApp('Editor')).getWindow(23)
await win.mouseMove(element('0'), { useSystemPointer: true })
expect(f.dispatch).toHaveBeenLastCalledWith({ method: 'mouseMove', params: {
appHandle: 'app-17', windowId: 23, snapshotId: 'snapshot-2', elementId: '0', useSystemPointer: true,
} })
await win.mouseMove([250, 750], { relative: true, useSystemPointer: false })
expect(f.dispatch).toHaveBeenLastCalledWith({ method: 'mouseMove', params: {
appHandle: 'app-17', windowId: 23, snapshotId: 'snapshot-2', x: 250, y: 750, relative: true, useSystemPointer: false,
} })
})
@@ -1431,9 +1431,9 @@
"index": 0,
"block": {
"type": "tool-call",
"id": "cua-pointer-request",
"id": "cua-explicit-pointer-move",
"name": "cua",
"arguments": "{\"method\":\"requestMouseControl\",\"params\":{}}"
"arguments": "{\"method\":\"mouseMove\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7,\"x\":100,\"y\":200,\"relative\":true,\"useSystemPointer\":true}}"
}
},
{