Verify and document mouse clicks separately from AX actions
This commit is contained in:
creatixchu committed
1 parent
97ec654f44
commit
ecd56a18d3
16 files changed
+49
-40
No files matched your search
@@ -12,5 +12,5 @@
|
||||
en: e5d1027faa55adba
|
||||
zh: 4b88ad6a70977fb3
|
||||
/native-cua-application-and-window-api/migration:
|
||||
en: 6cabae699a340845
|
||||
zh: f2256751518c5e36
|
||||
en: b09e3dde32391786
|
||||
zh: f702e1e25b260b29
|
||||
@@ -15,7 +15,7 @@ Providers supply startup capabilities alongside their platform guidance. The nat
|
||||
|
||||
## Migration
|
||||
|
||||
1. After `cua.getApp(selector)`, select the target from `app.getWindows()` with `app.getWindow(windowId)`. Use `window.getScreenshot()`, `window.click()`, `window.scroll()`, and `window.drag()`. Replace `hover` with `window.mouseMove()`.
|
||||
1. After `cua.getApp(selector)`, select the target from `app.getWindows()` with `app.getWindow(windowId)`. Use `window.getScreenshot()`, `window.click()`, `window.scroll()`, and `window.drag()`. Replace `hover` with `window.mouseMove()`. On macOS, `window.click(id)` sends mouse events at the element center. Switch to `invoke(id)` for the default accessibility action, or `performAction(id, "AXShowMenu")` to request an advertised menu action.
|
||||
2. Replace `getAXState()` with `getAccessibilityTree()`. Replace `getAXStateAndScreenshot()` with separate tree and screenshot reads. Selection reads the initial tree silently for a window, and for an app when `appLevelAccessibility` is true. Read the tree explicitly to display it.
|
||||
3. Pass the opaque string ID from the latest tree wherever you passed a numeric element index. In raw requests use `elementId` with `snapshotId`. Replace `performSecondaryAction` with `listActions(id)` and `performAction(id, exactName, argument?)`, or use `invoke(id)` for the default action. `setValue` accepts the provider’s supported JSON values. On macOS, `performAction(id, 'AXRaise')` performs the advertised raise action. Call `app.activate()` first when the application must become active.
|
||||
4. Replace a packed chord such as `pressKey('shift+Tab')` with `pressKey('Tab', ['shift'])`. Check `(await cua.getCapabilities()).methods` before using `sendKeyEvent`: macOS supports complete keystrokes and rejects directional events with `NOT_IMPLEMENTED`.
|
||||
|
||||
@@ -15,7 +15,7 @@ CUA 脚本在指定窗口上截图和操作鼠标。App 和 Window 对象共享
|
||||
|
||||
## 迁移
|
||||
|
||||
1. 调用 `cua.getApp(selector)` 后，从 `app.getWindows()` 返回的列表选定目标，再调用 `app.getWindow(windowId)`。在窗口上调用 `getScreenshot()`、`click()`、`scroll()` 和 `drag()`，将原有的 `hover` 调用改为 `window.mouseMove()`。
|
||||
1. 调用 `cua.getApp(selector)` 后，从 `app.getWindows()` 返回的列表选定目标，再调用 `app.getWindow(windowId)`。在窗口上调用 `getScreenshot()`、`click()`、`scroll()` 和 `drag()`，将原有的 `hover` 调用改为 `window.mouseMove()`。在 macOS 上，`window.click(id)` 在元素中心发送鼠标事件。需要执行默认辅助功能动作时，改用 `invoke(id)`。需要打开控件菜单且控件声明了 `AXShowMenu` 时，调用 `performAction(id, "AXShowMenu")`。
|
||||
2. 将 `getAXState()` 改为 `getAccessibilityTree()`，将 `getAXStateAndScreenshot()` 拆成读取树和截图两次调用。选择窗口时会静默读取初始树。选择应用时，只有 `appLevelAccessibility` 为真才读取。要显示树，请显式读取。
|
||||
3. 将数字元素索引改为从最新树中取得的字符串 ID。直接发送请求时使用 `elementId` 和 `snapshotId`。原先调用 `performSecondaryAction` 的地方，先用 `listActions(id)` 取得完整动作名，再调用 `performAction(id, exactName, argument?)`。要执行默认动作，可调用 `invoke(id)`。`setValue` 接受提供方支持的 JSON 值。 在 macOS 上，`performAction(id, 'AXRaise')` 只执行元素声明的抬起窗口动作。需要把应用切到前台时先调用 `app.activate()`。
|
||||
4. 将 `pressKey('shift+Tab')` 之类的组合键写法改为 `pressKey('Tab', ['shift'])`。调用 `sendKeyEvent` 前检查 `(await cua.getCapabilities()).methods`。macOS 支持完整按键，单独发送按下或松开事件会返回 `NOT_IMPLEMENTED`。
|
||||
|
||||
@@ -18,8 +18,8 @@
|
||||
en: 7fdb7c0462275417
|
||||
zh: 0a4a9562f4a82945
|
||||
/deepseek-ai-dsh-experimental-cua-macos/use-this-package/connect-an-external-execution-environment:
|
||||
en: d80ab0362946144c
|
||||
zh: 2cb9fc12bb473300
|
||||
en: 6054576016409f5d
|
||||
zh: 5dc25764c92b8db2
|
||||
/deepseek-ai-dsh-experimental-cua-macos/understand-the-implementation:
|
||||
en: 4e82b85a5aa72eee
|
||||
zh: 4e004b7a205d8688
|
||||
|
||||
@@ -122,7 +122,7 @@ Use the `cua` API to discover and select the application (`listApps`, `getState`
|
||||
|
||||
Screenshot capture requires an attachment store and a resolved model route declaring image input. The shared attachment admission validates canonical Base64, the raster and its configured capacity, then stores the image once. Native data remains available to programmatic callers; model content carries coordinate metadata and the durable image reference. Response byte limits apply before parsing.
|
||||
|
||||
Window clicks and mouse moves accept an element ID from the latest tree. The element must belong to the selected window. To trigger the default action of a menu item outside any window, use `app.invoke(id)`.
|
||||
Use `win.click(id)` to send mouse events at the center of an element, or `win.click([x, y])` to click a point. For both `click(id)` and `mouseMove(id)`, the element must belong to the selected window. To trigger the default action of a menu item outside any window, use `app.invoke(id)`.
|
||||
|
||||
Absolute coordinates for `mouseMove`, `click`, and `drag` use `[x, y]` tuples; `scroll` uses named `x` and `y`. All use pixels in the latest returned screenshot. Pass `relative: true` for integer thousandths of window width and height, from 0 to 999; omission or `false` uses pixels. Relative input can use a successful AX read or screenshot, while absolute input requires a screenshot. The Host maps the delivered image to native points, including attachment and model-request resizing. A value of 999 remains inside the window edge. Coordinate options do not apply to element IDs. A width or height change since the latest observation returns `WINDOW_CHANGED` before input, including raw coordinate calls without a snapshot ID; read fresh AX or take a new screenshot before proceeding. Window movement alone preserves window-local coordinates. Capture and pointer input reject known disagreement between the selected window’s accessibility frame and displayed bounds with `WINDOW_CHANGED`, before posting pointer events. This includes Stage Manager transformations when AX retains the full window frame. Read fresh state; any activation or window raise remains explicit. Advertised AX actions remain available. Missing AX geometry retains the AXless path and does not establish that a transformed window can be operated.
|
||||
|
||||
@@ -130,7 +130,7 @@ Retained objects resolve the current tool binding on every call. The external ru
|
||||
|
||||
AppKit sheets are selectable by their window IDs and remain visible in their document window’s tree. Keyboard input follows a system Open/Save panel only when the application’s focused sheet identifies one Apple panel-service window and that service still focuses the same panel. Unrelated or ambiguous remote recipients fail before dispatch.
|
||||
|
||||
Element clicks use opaque element IDs from the current snapshot. Left clicks support a single or double click; right and middle clicks support one. Click and drag options accept a button and modifier names; a modified click uses pointer input. Drags hold that button and carry its modifiers through every movement and release. AX text marks disabled controls, `valueWritable=true`, and available `selected` state. The helper uses an advertised accessibility action for unmodified clicks when available; pointer delivery resolves the element’s current bounds and verifies its owning window. Actions report dispatch, not application success. A visual pointer or click marker can be rendered independently of input; AX actions do not require moving the system cursor. Window-directed mouse input can still change application focus, selection or dialogs. Read state after input. `STALE_APP` requires acquiring the application again; `STALE_SNAPSHOT` requires a fresh AX read. Request timeout/cancellation waits for native acknowledgement and preserves usable handles. Missing cleanup acknowledgement closes that Agent’s connection. A screenshot RPC still awaiting its system callback occupies the single capture slot; further screenshots return `CAPTURE_BUSY`, while AX and input remain available. A late capture result is discarded, and its real completion releases that slot. Only an unresponsive operation or failed cleanup can invalidate the shared process and all its handles. Never retry uncertain input automatically. Closing an Agent releases its connection and AX references, without undoing changes to applications.
|
||||
Element clicks use opaque element IDs from the current snapshot. Left clicks support a single or double click; right and middle clicks support one. Click and drag options accept a button and modifier names. Drags hold that button and carry its modifiers through every movement and release. AX text marks disabled controls, `valueWritable=true`, and available `selected` state. Actions report dispatch, not application success. A visual pointer or click marker can be rendered independently of input; AX actions do not require moving the system cursor. Window-directed mouse input can still change application focus, selection or dialogs. Read state after input. `STALE_APP` requires acquiring the application again; `STALE_SNAPSHOT` requires a fresh AX read. Request timeout/cancellation waits for native acknowledgement and preserves usable handles. Missing cleanup acknowledgement closes that Agent’s connection. A screenshot RPC still awaiting its system callback occupies the single capture slot; further screenshots return `CAPTURE_BUSY`, while AX and input remain available. A late capture result is discarded, and its real completion releases that slot. Only an unresponsive operation or failed cleanup can invalidate the shared process and all its handles. Never retry uncertain input automatically. Closing an Agent releases its connection and AX references, without undoing changes to applications.
|
||||
|
||||
-----
|
||||
|
||||
@@ -175,7 +175,7 @@ The plugin contributes one `cua` operation tool with method-specific parameters.
|
||||
##### Operation guidance (verbatim)
|
||||
|
||||
```markdown
|
||||
macOS: accessibility tree operations are available at app or window level. Simulated keyboard input is available at app level. Simulated mouse input is available at window level. For click and mouseMove, use window coordinates or an element ID from the latest tree that belongs to the selected window. getApp also accepts bundle IDs and .app paths, and findApps searches /Applications, /System/Applications, and ~/Applications. Use the Computer Use permission page to grant the helper application Accessibility and Screen Recording permissions in System Settings. Native window and AX bounds use macOS points, while delivered screenshots carry their own pixel dimensions. show does not switch Spaces, and Stage Manager can present reduced window geometry; explain any required foreground step before calling app.activate() or performing an advertised AXRaise action.
|
||||
macOS: accessibility tree operations are available at app or window level. Simulated keyboard input is available at app level. Simulated mouse input is available at window level. click sends mouse events at window coordinates or at the center of an element from the latest tree in that window. Use invoke for the default accessibility action, including on menu items. mouseMove accepts either target. getApp also accepts bundle IDs and .app paths, and findApps searches /Applications, /System/Applications, and ~/Applications. Use the Computer Use permission page to grant the helper application Accessibility and Screen Recording permissions in System Settings. Native window and AX bounds use macOS points, while delivered screenshots carry their own pixel dimensions. show does not switch Spaces, and Stage Manager can present reduced window geometry; explain any required foreground step before calling app.activate() or performing an advertised AXRaise action.
|
||||
|
||||
Use cua to observe and operate native applications. Always use cua.getCapabilities() to confirm whether a specific API is available on the current platform; its reported capabilities take precedence over availability statements in these instructions. All methods are async; observation options default to { emit: true }, and emit: false suppresses sink output only, leaving returned values unchanged. Actions report dispatch evidence, not verified completion. If sessionLocked is true or an operation reports SESSION_LOCKED, stop UI actions and ask the user to unlock the desktop session; null or unknown means verify a usable unlocked desktop session first. cua.getApp requests user consent; if access is denied or revoked, stop using that application until the user explicitly permits another request, and never bypass a refusal or switch tools to avoid it. Never replay uncertain input without a fresh observation.
|
||||
|
||||
|
||||
@@ -122,7 +122,7 @@ void await window.getScreenshot()
|
||||
|
||||
截图要求附件服务可用，并确认当前模型支持图片输入。共享附件接入校验规范 Base64、图像数据及配置容量，然后保存一次图片。程序调用方仍可取得原生数据，模型内容包含坐标元数据和持久图片引用。响应字节上限在解析前执行。
|
||||
|
||||
窗口的 `click` 和 `mouseMove` 接受最新树中的元素 ID。元素必须属于所选窗口。菜单项如果不属于任何窗口，可用 `app.invoke(id)` 触发默认动作。
|
||||
`win.click(id)` 向元素中心发送鼠标事件，`win.click([x, y])` 向给定位置发送鼠标事件。`click(id)` 和 `mouseMove(id)` 使用的元素须属于所选窗口。菜单项如果不属于任何窗口，可用 `app.invoke(id)` 触发默认动作。
|
||||
|
||||
`mouseMove`、`click` 和 `drag` 的绝对坐标使用 `[x, y]` 元组；`scroll` 使用命名的 `x` 和 `y`。它们都使用最近返回截图中的像素。传入 `relative: true` 时使用 0～999 的千分比整数；省略或传入 `false` 时使用像素。相对坐标可基于成功的 AX 读取或截图，绝对坐标必须先截图。Host 根据实际送入模型的图片尺寸换算原生 point，包含附件归一化和模型请求阶段的缩放。999 映射到窗口边缘以内。坐标选项不适用于元素 ID。窗口宽高与最近一次观察不同时，在输入前返回 `WINDOW_CHANGED`，即使原始坐标调用不传快照 ID 也会检查；重新读取 AX 或截图后再操作。单纯移动窗口不使窗口内坐标失效。截图和鼠标输入会核对所选窗口的辅助功能框与显示边界；已知两者不一致时，在发送任何鼠标事件前返回 `WINDOW_CHANGED`。当 AX 保留完整窗口框时，这也会拦住 Stage Manager 变换后的窗口。需要重新读取状态；激活应用或抬起窗口仍须显式调用。元素声明的 AX 动作保持可用。缺少 AX 几何时保留无 AX 路径，不代表变换后的窗口一定可以操作。
|
||||
|
||||
@@ -130,7 +130,7 @@ void await window.getScreenshot()
|
||||
|
||||
AppKit sheet 可以通过其窗口 ID 单独选择，也会出现在所属文档窗口的控件树中。只有应用当前获得焦点的 sheet 指向唯一的 Apple 面板服务窗口，且该服务仍聚焦于同一面板时，键盘输入才转交系统打开或保存面板。无法确认或存在歧义的远端接收方会在输入前报错。
|
||||
|
||||
元素点击使用当前快照中的不透明元素 ID。左键支持单击或双击；右键和中键支持单击。点击和拖动选项可指定按键与修饰键名称；带修饰键的点击使用指针输入。拖动期间保持指定按键，并在移动和释放事件中携带修饰键。AX 文本标明禁用控件、`valueWritable=true` 和可读取的 `selected` 状态。不带修饰键的点击优先使用元素声明的辅助功能动作；需要发送鼠标事件时，会重新读取元素位置并核实所属窗口。动作返回只表示输入已分派。可视指针或点击标记可以独立绘制；AX 动作无需移动系统指针。窗口定向鼠标输入仍可能改变应用焦点、选区或对话框。输入后需要读取状态确认结果。`STALE_APP` 需要重新获取应用，`STALE_SNAPSHOT` 需要重新读取 AX。请求超时或取消会等待原生停止确认，保留仍可用的句柄；无法确认清理完成时关闭该 Agent 的连接。未返回系统回调的截图 RPC 占用唯一捕获槽位，新截图立即返回 `CAPTURE_BUSY`，AX 和输入仍可使用。迟到图片被丢弃，真实回调结束后释放槽位。只有操作无法停止或清理失败时，才可能使共享进程及所有旧句柄失效。不能自动重试结果不确定的输入。关闭 Agent 会释放其连接和 AX 引用，但不会撤销应用中的修改。
|
||||
元素点击使用当前快照中的不透明元素 ID。左键支持单击或双击；右键和中键支持单击。点击和拖动选项可指定按键与修饰键名称。拖动期间保持指定按键，并在移动和释放事件中携带修饰键。AX 文本标明禁用控件、`valueWritable=true` 和可读取的 `selected` 状态。动作返回只表示输入已分派。可视指针或点击标记可以独立绘制；AX 动作无需移动系统指针。窗口定向鼠标输入仍可能改变应用焦点、选区或对话框。输入后需要读取状态确认结果。`STALE_APP` 需要重新获取应用，`STALE_SNAPSHOT` 需要重新读取 AX。请求超时或取消会等待原生停止确认，保留仍可用的句柄；无法确认清理完成时关闭该 Agent 的连接。未返回系统回调的截图 RPC 占用唯一捕获槽位，新截图立即返回 `CAPTURE_BUSY`，AX 和输入仍可使用。迟到图片被丢弃，真实回调结束后释放槽位。只有操作无法停止或清理失败时，才可能使共享进程及所有旧句柄失效。不能自动重试结果不确定的输入。关闭 Agent 会释放其连接和 AX 引用，但不会撤销应用中的修改。
|
||||
|
||||
-----
|
||||
|
||||
@@ -175,7 +175,7 @@ AppKit sheet 可以通过其窗口 ID 单独选择，也会出现在所属文档
|
||||
##### 操作指导文本（原文）
|
||||
|
||||
```markdown
|
||||
macOS: accessibility tree operations are available at app or window level. Simulated keyboard input is available at app level. Simulated mouse input is available at window level. For click and mouseMove, use window coordinates or an element ID from the latest tree that belongs to the selected window. getApp also accepts bundle IDs and .app paths, and findApps searches /Applications, /System/Applications, and ~/Applications. Use the Computer Use permission page to grant the helper application Accessibility and Screen Recording permissions in System Settings. Native window and AX bounds use macOS points, while delivered screenshots carry their own pixel dimensions. show does not switch Spaces, and Stage Manager can present reduced window geometry; explain any required foreground step before calling app.activate() or performing an advertised AXRaise action.
|
||||
macOS: accessibility tree operations are available at app or window level. Simulated keyboard input is available at app level. Simulated mouse input is available at window level. click sends mouse events at window coordinates or at the center of an element from the latest tree in that window. Use invoke for the default accessibility action, including on menu items. mouseMove accepts either target. getApp also accepts bundle IDs and .app paths, and findApps searches /Applications, /System/Applications, and ~/Applications. Use the Computer Use permission page to grant the helper application Accessibility and Screen Recording permissions in System Settings. Native window and AX bounds use macOS points, while delivered screenshots carry their own pixel dimensions. show does not switch Spaces, and Stage Manager can present reduced window geometry; explain any required foreground step before calling app.activate() or performing an advertised AXRaise action.
|
||||
|
||||
Use cua to observe and operate native applications. Always use cua.getCapabilities() to confirm whether a specific API is available on the current platform; its reported capabilities take precedence over availability statements in these instructions. All methods are async; observation options default to { emit: true }, and emit: false suppresses sink output only, leaving returned values unchanged. Actions report dispatch evidence, not verified completion. If sessionLocked is true or an operation reports SESSION_LOCKED, stop UI actions and ask the user to unlock the desktop session; null or unknown means verify a usable unlocked desktop session first. cua.getApp requests user consent; if access is denied or revoked, stop using that application until the user explicitly permits another request, and never bypass a refusal or switch tools to avoid it. Never replay uncertain input without a fresh observation.
|
||||
|
||||
|
||||
@@ -21,8 +21,8 @@
|
||||
en: 20be674c0592d66d
|
||||
zh: e26112b908187acc
|
||||
/dsh-cua-native-helper/observation-and-input:
|
||||
en: bf1fa9dbe1cb6dfe
|
||||
zh: 14097def08a240a4
|
||||
en: 8c8ec56eeb88dccf
|
||||
zh: 4a11a9682e4caec2
|
||||
/dsh-cua-native-helper/verification:
|
||||
en: 77b266136d5e3631
|
||||
zh: a5f135e1b12f7a52
|
||||
|
||||
@@ -97,9 +97,9 @@ Protocol declarations are generated from [the shared protocol](../../cua/src/pro
|
||||
|
||||
`getState` optionally scopes AX traversal to an owned window or nested sheet `windowId`. A new or failed refresh invalidates the previous snapshot for that app/window scope, while other window scopes remain usable. When an owned CG window has no matching AX surface, getState returns a new snapshot with empty tree/elements and no truncation; its coordinates and screenshots remain available. It rechecks process lifetime, window ownership, the unlocked session and Accessibility permission after traversal. Window-discovery APIDisabled and communication errors remain failures; unsupported or absent optional AX attributes can produce an empty observation. Missing or foreign-owned windows never produce an empty success. An AX element invalidated during traversal rejects the whole read with `STALE_SNAPSHOT`; no partial snapshot is published. Callers may read again after the application changes, without repeating input. Raw element targets require `appHandle`, `snapshotId`, and an opaque string `elementId`; coordinate actions require `windowId` and may include a snapshot ID from that window or an app-wide observation. Native references never cross JSON. Window-list refresh releases snapshots for closed windows. Native process identity, snapshot generation, and window ownership checks do not lock the desktop against simultaneous user or application changes.
|
||||
|
||||
Element clicks without holdMs prefer advertised AX actions and otherwise use pointer delivery. Coordinate click, scroll and drag validate window ownership and bounds, then use background PID/window-directed primitives. Each event uses SkyLight when available, otherwise the public PID-targeted API. A posted event is never repeated through the other API. Coordinates are macOS points from the top-left of the entire window, including its title bar. Positive `deltaLinesY` reveals content above; negative reveals below. Positive `deltaLinesX` reveals content left; negative reveals right. Each axis accepts an integer from -10 to 10, and at least one must be nonzero. The helper sends line-unit scroll events directly. Click and drag accept left/right/middle buttons and command/control/option/shift/fn modifiers. Only left clicks support double-click. Modified element clicks use pointer delivery. Drag uses one button press/move/release in one window; callers explicitly supply durationMs (1..5000) and steps (1..240). The movement delay budget is divided with fractional precision across the segments; fixed press/release settling and scheduler latency are additional. No implicit fallback sends global desktop input.
|
||||
Clicks send mouse-down and mouse-up events at the element center or the supplied coordinates. Use `invoke` for the default AX action or `performAction` for a named action. Click, scroll and drag validate window ownership and bounds, then use background PID/window-directed primitives. Each event uses SkyLight when available, otherwise the public PID-targeted API. A posted event is never repeated through the other API. Coordinates are macOS points from the top-left of the entire window, including its title bar. Positive `deltaLinesY` reveals content above; negative reveals below. Positive `deltaLinesX` reveals content left; negative reveals right. Each axis accepts an integer from -10 to 10, and at least one must be nonzero. The helper sends line-unit scroll events directly. Click and drag accept left/right/middle buttons and command/control/option/shift/fn modifiers. Only left clicks support double-click. Drag uses one button press/move/release in one window; callers explicitly supply durationMs (1..5000) and steps (1..240). The movement delay budget is divided with fractional precision across the segments; fixed press/release settling and scheduler latency are additional. No implicit fallback sends global desktop input.
|
||||
|
||||
`click.holdMs` is an optional nonnegative integer in milliseconds per press; zero adds no hold delay. Each connection supplies `maxClickHoldMs` to bound explicit holds. The diagnostic helper option `--max-click-hold-ms` defaults to 5000. It bypasses AX actions and uses the element center or supplied coordinates. Omission retains ordinary click timing. The complete gesture holds the request scheduler slot; cancellation, timeout and disconnect interrupt the hold and post the pending mouse-up before releasing that slot. Received events remain effective, and forced process termination cannot guarantee cleanup.
|
||||
`click.holdMs` is an optional nonnegative integer in milliseconds per press; zero adds no hold delay. Each connection supplies `maxClickHoldMs` to bound explicit holds. The diagnostic helper option `--max-click-hold-ms` defaults to 5000. Omission retains ordinary click timing. The complete gesture holds the request scheduler slot; cancellation, timeout and disconnect interrupt the hold and post the pending mouse-up before releasing that slot. Received events remain effective, and forced process termination cannot guarantee cleanup.
|
||||
|
||||
Capture and pointer delivery read CG bounds before and after an exact-window AX frame query. Movement during the query returns WINDOW_CHANGED. Valid AX/CG frames must have equal outward-rounded integral point rectangles; a known mismatch returns WINDOW_CHANGED before mouse movement, priming or button delivery. The helper preserves both coordinate sources rather than substituting AX geometry for CG geometry. Missing or unusable AX geometry remains unknown, while AX communication, permission and invalid-element errors propagate. Capture without Accessibility permission retains its CG/SCK checks. Advertised AX actions bypass this pointer preflight; an AX action error does not trigger pointer input. These reads do not lock the window against later movement, and no geometry check activates or raises it.
|
||||
|
||||
@@ -111,7 +111,7 @@ Host-only permission controls are `permissionStatus {}`, `requestPermission` and
|
||||
|
||||
Observation reads each non-layout node’s descriptive properties and children through `AXUIElementCopyMultipleAttributeValues`. Per-attribute errors retain the individual-read rules; unsupported batch APIs use individual reads. Values are retained only during that observation, and input reads live attributes. Application state can change between nodes; a batch does not make the whole tree atomic.
|
||||
|
||||
System file-panel keyboard routing resolves the remote recipient through the selected app’s focused AX sheet. Only Apple’s Open and Save Panel Service is accepted, and its focused ancestry must still contain the identified remote window. Element pointer fallbacks resolve an app-owned window or sheet through AX parents when AXWindow is unavailable. Ordinary clicks retain AX action priority over pointer input; visual click markers need no system-cursor movement.
|
||||
System file-panel keyboard routing resolves the remote recipient through the selected app’s focused AX sheet. Only Apple’s Open and Save Panel Service is accepted, and its focused ancestry must still contain the identified remote window. Element-targeted pointer input follows AX parents to find the enclosing window or sheet.
|
||||
|
||||
Keyboard modifier names are plain strings. The shared object API passes the names through unchanged; the macOS helper accepts `command`/`cmd`, `shift`, `option`/`alt`, `control`/`ctrl`, and `fn`, and rejects an unknown name with `INVALID_ARGUMENT`. It does not parse chord strings.
|
||||
|
||||
|
||||
@@ -97,9 +97,9 @@ AppKit 在主事件循环上运行。阻塞式 socket 接收与握手使用后
|
||||
|
||||
`getState` 可将 AX 遍历限定到应用所属窗口或嵌套 sheet 的 `windowId`。新的刷新或失败的刷新使同一应用/窗口范围的旧快照失效，其他窗口范围仍可使用。应用所属 CG 窗口没有匹配 AX 界面时，getState 返回空 tree/elements、truncated 为 false 的新快照，仍可截图和坐标输入。遍历后重新检查进程生命期、窗口归属、会话未锁定及辅助功能权限。窗口发现中的 APIDisabled 和通信错误仍会失败；不支持或缺失的可选 AX 属性可以产生空观察。不存在或属于其他应用的窗口不会返回空成功结果。遍历中 AX 元素失效会以 `STALE_SNAPSHOT` 拒绝整次读取，不发布部分树。应用变化后可以重新观察，但不重复输入。原始元素目标要求 `appHandle`、`snapshotId` 和不透明字符串 `elementId`；坐标动作要求 `windowId`，并可携带属于该窗口或应用级观察的快照 ID。原生引用不跨 JSON 传递。刷新窗口列表会释放已关闭窗口的快照。进程身份、快照代次和窗口归属检查不会锁住桌面，用户或应用仍可同时改变状态。
|
||||
|
||||
未指定 holdMs 的元素点击优先使用声明的 AX 动作，否则通过鼠标事件发送。坐标点击、滚动和拖动核实窗口归属和边界后，使用后台 PID/窗口定向输入。事件优先使用 SkyLight，不可用时使用公开的 PID 定向 API；已发送的事件不通过另一 API 重发。坐标是整个窗口左上角起算的 macOS point，包含标题栏。`deltaLinesY` 为正时显示上方内容，为负时显示下方内容；`deltaLinesX` 为正时显示左侧内容，为负时显示右侧内容。每个轴接受 -10 到 10 的整数，至少一个轴非零。helper 直接发送按行计量的滚轮事件。点击和拖动支持左、右、中键及 command/control/option/shift/fn 修饰键，只有左键支持双击。带修饰键的元素点击使用鼠标输入。拖动在一个窗口中完成一次按下、移动、抬起，调用方显式提供 durationMs（1..5000）和 steps（1..240）。移动延迟按小数精度分配到各段，按下和抬起的固定等待及调度延迟另计。不存在隐式改用全局桌面输入的后备路径。
|
||||
点击在元素中心或给定坐标发送鼠标按下和抬起事件。执行默认 AX 动作用 `invoke`，执行指定动作用 `performAction`。点击、滚动和拖动核实窗口归属和边界后，使用后台 PID/窗口定向输入。事件优先使用 SkyLight，不可用时使用公开的 PID 定向 API；已发送的事件不通过另一 API 重发。坐标是整个窗口左上角起算的 macOS point，包含标题栏。`deltaLinesY` 为正时显示上方内容，为负时显示下方内容；`deltaLinesX` 为正时显示左侧内容，为负时显示右侧内容。每个轴接受 -10 到 10 的整数，至少一个轴非零。helper 直接发送按行计量的滚轮事件。点击和拖动支持左、右、中键及 command/control/option/shift/fn 修饰键，只有左键支持双击。拖动在一个窗口中完成一次按下、移动、抬起，调用方显式提供 durationMs（1..5000）和 steps（1..240）。移动延迟按小数精度分配到各段，按下和抬起的固定等待及调度延迟另计。不存在隐式改用全局桌面输入的后备路径。
|
||||
|
||||
`click.holdMs` 是可选的每次按住时长，单位为毫秒，接受非负整数；0 表示不额外等待。每个连接通过 `maxClickHoldMs` 限制显式按住时长。诊断 helper 的 `--max-click-hold-ms` 选项默认 5000。指定后使用元素中心或给定坐标发送鼠标事件，跳过 AX 动作。省略时保留普通点击时长。整个手势占用请求调度槽位；取消、超时和断开连接会中断等待，在释放槽位前发送对应的松开事件。已接收的事件仍然生效，进程被强制终止时无法保证清理。
|
||||
`click.holdMs` 是可选的每次按住时长，单位为毫秒，接受非负整数；0 表示不额外等待。每个连接通过 `maxClickHoldMs` 限制显式按住时长。诊断 helper 的 `--max-click-hold-ms` 选项默认 5000。省略时保留普通点击时长。整个手势占用请求调度槽位；取消、超时和断开连接会中断等待，在释放槽位前发送对应的松开事件。已接收的事件仍然生效，进程被强制终止时无法保证清理。
|
||||
|
||||
截图和鼠标输入在精确窗口的 AX 边界查询前后读取 CG 边界，查询期间移动会返回 WINDOW_CHANGED。有效的 AX/CG 边界向外取整后必须一致；已知不一致会在移动、预备事件或按键发送前返回 WINDOW_CHANGED。helper 保留两个坐标来源，不用 AX 几何替换 CG 几何。缺失或无效的 AX 几何仍表示未知，AX 通信、权限和无效元素错误继续传递。无辅助功能权限的截图保留 CG/SCK 检查。声明的 AX 动作不经过鼠标预检，AX 动作失败也不改用鼠标输入。这些读取不会锁住窗口以阻止后续移动，几何检查不激活或抬起窗口。
|
||||
|
||||
@@ -111,7 +111,7 @@ AppKit 在主事件循环上运行。阻塞式 socket 接收与握手使用后
|
||||
|
||||
观察通过 `AXUIElementCopyMultipleAttributeValues` 读取每个非布局节点的描述属性和子节点。逐属性错误保留单独读取的规则；不支持批量 API 时使用单独读取。值只在本次观察期间保留，输入读取即时属性。应用状态可能在节点之间变化，批量读取不使整棵树具备原子性。
|
||||
|
||||
系统文件面板的键盘路由通过所选应用的 AX 焦点 sheet 解析远端接收方。只接受 Apple 的 Open and Save Panel Service，且其焦点祖先链仍须包含已识别的远端窗口。AXWindow 不可用时，元素鼠标后备路径沿 AX 父节点解析应用所属窗口或 sheet。普通点击保持 AX 动作优先，可视点击标记不需要系统鼠标移动。
|
||||
系统文件面板的键盘路由通过所选应用的 AX 焦点 sheet 解析远端接收方。只接受 Apple 的 Open and Save Panel Service，且其焦点祖先链仍须包含已识别的远端窗口。按元素发送鼠标事件时，helper 沿 AX 父节点查找所属窗口或 sheet。
|
||||
|
||||
键盘修饰键名称是普通字符串。共享对象 API 原样传递名称；macOS helper 接受 `command`/`cmd`、`shift`、`option`/`alt`、`control`/`ctrl` 和 `fn`，未知名称返回 `INVALID_ARGUMENT`。它不解析组合键字符串。
|
||||
|
||||
|
||||
@@ -4,9 +4,11 @@ import AppKit
|
||||
final class CapabilityPad: NSView {
|
||||
var received: ((NSEvent) -> Void)?
|
||||
var pressed: (() -> Void)?
|
||||
var showMenu: (() -> Void)?
|
||||
override func isAccessibilityElement() -> Bool { true }
|
||||
override func accessibilityRole() -> NSAccessibility.Role? { .button }
|
||||
override func accessibilityPerformPress() -> Bool { pressed?(); return true }
|
||||
override func accessibilityPerformShowMenu() -> Bool { showMenu?(); return true }
|
||||
override var isFlipped: Bool { true }
|
||||
override var acceptsFirstResponder: Bool { true }
|
||||
override func acceptsFirstMouse(for event: NSEvent?) -> Bool { true }
|
||||
@@ -46,6 +48,7 @@ final class CapabilitiesFixture: NSObject, NSApplicationDelegate, NSWindowDelega
|
||||
var pad: CapabilityPad!
|
||||
var events: [[String: Any]] = []
|
||||
var axPresses = 0
|
||||
var axMenus = 0
|
||||
init(statePath: String) { self.statePath = statePath }
|
||||
func applicationDidFinishLaunching(_ notification: Notification) {
|
||||
let bar = NSMenu()
|
||||
@@ -77,6 +80,7 @@ final class CapabilitiesFixture: NSObject, NSApplicationDelegate, NSWindowDelega
|
||||
pad = CapabilityPad(frame: NSRect(x: 20, y: 25, width: 600, height: 170))
|
||||
pad.setAccessibilityTitle("Input receiver")
|
||||
pad.pressed = { [weak self] in self?.axPresses += 1; self?.persist() }
|
||||
pad.showMenu = { [weak self] in self?.axMenus += 1; self?.persist() }
|
||||
pad.received = { [weak self] event in
|
||||
guard let self else { return }
|
||||
var value: [String: Any] = ["type": event.type.rawValue, "timestamp": event.timestamp, "flags": event.modifierFlags.rawValue,
|
||||
@@ -119,7 +123,7 @@ final class CapabilitiesFixture: NSObject, NSApplicationDelegate, NSWindowDelega
|
||||
let result: [String: Any] = ["pid": ProcessInfo.processInfo.processIdentifier, "windowId": window.windowNumber,
|
||||
"active": NSApplication.shared.isActive, "keyWindow": window.isKeyWindow,
|
||||
"textFocused": window.firstResponder === text, "pasteAttempts": pasteAttempts,
|
||||
"text": text.string, "selection": [selection.location, selection.length], "runs": runs, "events": events, "axPresses": axPresses,
|
||||
"text": text.string, "selection": [selection.location, selection.length], "runs": runs, "events": events, "axPresses": axPresses, "axMenus": axMenus,
|
||||
"pad": [frame.minX, window.frame.height - frame.maxY, frame.width, frame.height]]
|
||||
try! JSONSerialization.data(withJSONObject: result).write(to: URL(fileURLWithPath: statePath), options: .atomic)
|
||||
}
|
||||
|
||||
@@ -44,7 +44,7 @@ def main():
|
||||
owner = {"appHandle": app["appHandle"], "windowId": state["windowId"]}
|
||||
observed = helper.call("getState", {**owner, "maxElements": 2000, "maxDepth": 25})
|
||||
pad = next(item for item in observed["elements"] if item["role"] == "AXButton" and item["title"] == "Input receiver")
|
||||
helper.call("click", {"appHandle": app["appHandle"], "snapshotId": observed["snapshotId"], "elementId": pad["elementId"]})
|
||||
helper.call("invoke", {**owner, "snapshotId": observed["snapshotId"], "elementId": pad["elementId"]})
|
||||
wait_for_state(state_path, lambda value: value["axPresses"] == state["axPresses"] + 1)
|
||||
captured = helper.call("screenshot", owner)
|
||||
assert base64.b64decode(captured["data"], validate=True).startswith(b"\x89PNG\r\n\x1a\n")
|
||||
@@ -184,7 +184,7 @@ def main():
|
||||
os.kill(pid, 0)
|
||||
print(f"Deleted executable: bundle_id={bundle_id is not None}, proc_pidpath_bytes={copied}, errno={path_error}")
|
||||
print("PASS: changed bundles cannot launch; cold/running and direct-bundle identities agree; copies share consent; other IDs stay separate; missing IDs and standalone GUIs use persistent paths across restarts"
|
||||
+ (f"; {checked_windows} GUI instances expose controls, receive an AX click and produce a PNG" if args.desktop else ""))
|
||||
+ (f"; {checked_windows} GUI instances expose controls, receive a default AX action and produce a PNG" if args.desktop else ""))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
@@ -46,19 +46,24 @@ def stop_application(pid):
|
||||
raise AssertionError("fixture did not exit after SIGTERM")
|
||||
|
||||
|
||||
def check_click_holds(helper, owner, window, point, state_file):
|
||||
observed = helper.call("getState", {**owner, "maxElements": 2000, "maxDepth": 25})
|
||||
def check_clicks(helper, window, point, state_file):
|
||||
observed = helper.call("getState", {**window, "maxElements": 2000, "maxDepth": 25})
|
||||
pad = next(item for item in observed["elements"] if item["role"] == "AXButton" and item["title"] == "Input receiver")
|
||||
target = {**owner, "snapshotId": observed["snapshotId"], "elementId": pad["elementId"]}
|
||||
target = {**window, "snapshotId": observed["snapshotId"], "elementId": pad["elementId"]}
|
||||
state = json.loads(state_file.read_text())
|
||||
presses = state["axPresses"]
|
||||
helper.call("click", target)
|
||||
state = wait_for_state(state_file, lambda value: value["axPresses"] == presses + 1)
|
||||
count = len(state["events"])
|
||||
presses, menus = state["axPresses"], state["axMenus"]
|
||||
assert {"AXPress", "AXShowMenu"}.issubset(helper.call("listActions", target))
|
||||
helper.call("invoke", target)
|
||||
state = wait_for_state(state_file, lambda value: value["axPresses"] == presses + 1)
|
||||
helper.call("performAction", {**target, "action": "AXShowMenu"})
|
||||
state = wait_for_state(state_file, lambda value: value["axMenus"] == menus + 1)
|
||||
assert len(state["events"]) == count, state
|
||||
for invalid in [-1, 1.5, 5001, "800", None]:
|
||||
helper.call("click", {**target, "holdMs": invalid}, "INVALID_ARGUMENT")
|
||||
assert json.loads(state_file.read_text())["axPresses"] == presses + 1
|
||||
for params, types in [({**target, "holdMs": 0}, [1, 2]),
|
||||
for params, types in [(target, [1, 2]), ({**target, "mouseButton": "right"}, [3, 4]),
|
||||
({**window, **point}, [1, 2]),
|
||||
({**target, "holdMs": 0}, [1, 2]),
|
||||
({**target, "holdMs": 150}, [1, 2]),
|
||||
({**window, **point, "holdMs": 150, "mouseButton": "right"}, [3, 4]),
|
||||
({**window, **point, "holdMs": 150, "mouseButton": "middle"}, [25, 26]),
|
||||
@@ -68,8 +73,8 @@ def check_click_holds(helper, owner, window, point, state_file):
|
||||
events = state["events"][count:]
|
||||
assert [event["type"] for event in events] == types, events
|
||||
for index in range(0, len(events), 2):
|
||||
assert events[index + 1]["timestamp"] - events[index]["timestamp"] >= params["holdMs"] / 1000, events
|
||||
assert state["axPresses"] == presses + 1, state
|
||||
assert events[index + 1]["timestamp"] - events[index]["timestamp"] >= params.get("holdMs", 0) / 1000, events
|
||||
assert (state["axPresses"], state["axMenus"]) == (presses + 1, menus + 1), state
|
||||
count += len(types)
|
||||
# Production cancellation travels directly over the socket; the diagnostic stdio proxy exchanges whole requests.
|
||||
key = hashlib.sha256(Path(helper.process.args[0]).read_bytes()).hexdigest()
|
||||
@@ -77,7 +82,7 @@ def check_click_holds(helper, owner, window, point, state_file):
|
||||
for ending in ["cancel", "timeout", "disconnect"]:
|
||||
connection = Connection(socket_path, key, maxClickHoldMs=7000, timeoutMs=2000 if ending == "timeout" else 30000)
|
||||
try:
|
||||
app = connection.call("getApp", {"selector": str(observed["pid"])})
|
||||
app = connection.call("getApp", {"selector": str(state["pid"])})
|
||||
connection.call("getState", {"appHandle": app["appHandle"], "windowId": window["windowId"],
|
||||
"maxElements": 2000, "maxDepth": 25})
|
||||
params = {**window, **point, "appHandle": app["appHandle"], "holdMs": 6000}
|
||||
@@ -104,7 +109,7 @@ def check_click_holds(helper, owner, window, point, state_file):
|
||||
connection.close()
|
||||
connection = Connection(socket_path, key, maxClickHoldMs=0)
|
||||
try:
|
||||
app = connection.call("getApp", {"selector": str(observed["pid"])})
|
||||
app = connection.call("getApp", {"selector": str(state["pid"])})
|
||||
connection.call("getState", {"appHandle": app["appHandle"], "windowId": window["windowId"],
|
||||
"maxElements": 2000, "maxDepth": 25})
|
||||
count = len(state["events"])
|
||||
@@ -189,7 +194,7 @@ def main():
|
||||
call("pressKey", {**owner, "key": "KP_0"})
|
||||
call("pressKey", {**owner, "key": "KP_Enter"})
|
||||
state = wait_for_state(state_file, lambda value: {82, 76}.issubset({event.get("keyCode") for event in value["events"]}))
|
||||
state = check_click_holds(helper, owner, window, point, state_file)
|
||||
state = check_clicks(helper, window, point, state_file)
|
||||
if args.artifacts:
|
||||
args.artifacts.mkdir(parents=True, exist_ok=True)
|
||||
shot = call("screenshot", window)
|
||||
|
||||
@@ -99,7 +99,7 @@ def main():
|
||||
|
||||
window = next(window for window in state['windows'] if window['windowId'] == window_id)
|
||||
if not args.hover_only:
|
||||
call('click', {**owner, 'snapshotId': state['snapshotId'], 'elementId': buttons[0]['elementId']})
|
||||
call('invoke', {**owner, 'snapshotId': state['snapshotId'], 'elementId': buttons[0]['elementId']})
|
||||
wait_state(lambda actual: actual['clicks'] == 1 and actual['lastClickTrusted'] is True,
|
||||
'Native AX press did not produce one trusted DOM click')
|
||||
state = observe()
|
||||
|
||||
@@ -33,7 +33,7 @@ Start independent delegations with `subagent` or `subagent_fork` together in one
|
||||
|
||||
Bash, formatters, code generators, and scripts can overwrite concurrent edits without conflict detection; coordinate their changes explicitly. The agent integrating the work must review the final diff and run relevant tests.
|
||||
|
||||
macOS: accessibility tree operations are available at app or window level. Simulated keyboard input is available at app level. Simulated mouse input is available at window level. For click and mouseMove, use window coordinates or an element ID from the latest tree that belongs to the selected window. getApp also accepts bundle IDs and .app paths, and findApps searches /Applications, /System/Applications, and ~/Applications. Use the Computer Use permission page to grant the helper application Accessibility and Screen Recording permissions in System Settings. Native window and AX bounds use macOS points, while delivered screenshots carry their own pixel dimensions. show does not switch Spaces, and Stage Manager can present reduced window geometry; explain any required foreground step before calling app.activate() or performing an advertised AXRaise action.
|
||||
macOS: accessibility tree operations are available at app or window level. Simulated keyboard input is available at app level. Simulated mouse input is available at window level. click sends mouse events at window coordinates or at the center of an element from the latest tree in that window. Use invoke for the default accessibility action, including on menu items. mouseMove accepts either target. getApp also accepts bundle IDs and .app paths, and findApps searches /Applications, /System/Applications, and ~/Applications. Use the Computer Use permission page to grant the helper application Accessibility and Screen Recording permissions in System Settings. Native window and AX bounds use macOS points, while delivered screenshots carry their own pixel dimensions. show does not switch Spaces, and Stage Manager can present reduced window geometry; explain any required foreground step before calling app.activate() or performing an advertised AXRaise action.
|
||||
|
||||
Use cua to observe and operate native applications. Always use cua.getCapabilities() to confirm whether a specific API is available on the current platform; its reported capabilities take precedence over availability statements in these instructions. All methods are async; observation options default to { emit: true }, and emit: false suppresses sink output only, leaving returned values unchanged. Actions report dispatch evidence, not verified completion. If sessionLocked is true or an operation reports SESSION_LOCKED, stop UI actions and ask the user to unlock the desktop session; null or unknown means verify a usable unlocked desktop session first. cua.getApp requests user consent; if access is denied or revoked, stop using that application until the user explicitly permits another request, and never bypass a refusal or switch tools to avoid it. Never replay uncertain input without a fresh observation.
|
||||
|
||||
|
||||
@@ -35,7 +35,7 @@ Start independent delegations with `subagent` or `subagent_fork` together in one
|
||||
|
||||
Bash, formatters, code generators, and scripts can overwrite concurrent edits without conflict detection; coordinate their changes explicitly. The agent integrating the work must review the final diff and run relevant tests.
|
||||
|
||||
macOS: accessibility tree operations are available at app or window level. Simulated keyboard input is available at app level. Simulated mouse input is available at window level. For click and mouseMove, use window coordinates or an element ID from the latest tree that belongs to the selected window. getApp also accepts bundle IDs and .app paths, and findApps searches /Applications, /System/Applications, and ~/Applications. Use the Computer Use permission page to grant the helper application Accessibility and Screen Recording permissions in System Settings. Native window and AX bounds use macOS points, while delivered screenshots carry their own pixel dimensions. show does not switch Spaces, and Stage Manager can present reduced window geometry; explain any required foreground step before calling app.activate() or performing an advertised AXRaise action.
|
||||
macOS: accessibility tree operations are available at app or window level. Simulated keyboard input is available at app level. Simulated mouse input is available at window level. click sends mouse events at window coordinates or at the center of an element from the latest tree in that window. Use invoke for the default accessibility action, including on menu items. mouseMove accepts either target. getApp also accepts bundle IDs and .app paths, and findApps searches /Applications, /System/Applications, and ~/Applications. Use the Computer Use permission page to grant the helper application Accessibility and Screen Recording permissions in System Settings. Native window and AX bounds use macOS points, while delivered screenshots carry their own pixel dimensions. show does not switch Spaces, and Stage Manager can present reduced window geometry; explain any required foreground step before calling app.activate() or performing an advertised AXRaise action.
|
||||
|
||||
Use cua to observe and operate native applications. Always use cua.getCapabilities() to confirm whether a specific API is available on the current platform; its reported capabilities take precedence over availability statements in these instructions. All methods are async; observation options default to { emit: true }, and emit: false suppresses sink output only, leaving returned values unchanged. Actions report dispatch evidence, not verified completion. If sessionLocked is true or an operation reports SESSION_LOCKED, stop UI actions and ask the user to unlock the desktop session; null or unknown means verify a usable unlocked desktop session first. cua.getApp requests user consent; if access is denied or revoked, stop using that application until the user explicitly permits another request, and never bypass a refusal or switch tools to avoid it. Never replay uncertain input without a fresh observation.
|
||||
|
||||
|
||||
@@ -35,7 +35,7 @@ Start independent delegations with `subagent` or `subagent_fork` together in one
|
||||
|
||||
Bash, formatters, code generators, and scripts can overwrite concurrent edits without conflict detection; coordinate their changes explicitly. The agent integrating the work must review the final diff and run relevant tests.
|
||||
|
||||
macOS: accessibility tree operations are available at app or window level. Simulated keyboard input is available at app level. Simulated mouse input is available at window level. For click and mouseMove, use window coordinates or an element ID from the latest tree that belongs to the selected window. getApp also accepts bundle IDs and .app paths, and findApps searches /Applications, /System/Applications, and ~/Applications. Use the Computer Use permission page to grant the helper application Accessibility and Screen Recording permissions in System Settings. Native window and AX bounds use macOS points, while delivered screenshots carry their own pixel dimensions. show does not switch Spaces, and Stage Manager can present reduced window geometry; explain any required foreground step before calling app.activate() or performing an advertised AXRaise action.
|
||||
macOS: accessibility tree operations are available at app or window level. Simulated keyboard input is available at app level. Simulated mouse input is available at window level. click sends mouse events at window coordinates or at the center of an element from the latest tree in that window. Use invoke for the default accessibility action, including on menu items. mouseMove accepts either target. getApp also accepts bundle IDs and .app paths, and findApps searches /Applications, /System/Applications, and ~/Applications. Use the Computer Use permission page to grant the helper application Accessibility and Screen Recording permissions in System Settings. Native window and AX bounds use macOS points, while delivered screenshots carry their own pixel dimensions. show does not switch Spaces, and Stage Manager can present reduced window geometry; explain any required foreground step before calling app.activate() or performing an advertised AXRaise action.
|
||||
|
||||
Use cua to observe and operate native applications. Always use cua.getCapabilities() to confirm whether a specific API is available on the current platform; its reported capabilities take precedence over availability statements in these instructions. All methods are async; observation options default to { emit: true }, and emit: false suppresses sink output only, leaving returned values unchanged. Actions report dispatch evidence, not verified completion. If sessionLocked is true or an operation reports SESSION_LOCKED, stop UI actions and ask the user to unlock the desktop session; null or unknown means verify a usable unlocked desktop session first. cua.getApp requests user consent; if access is denied or revoked, stop using that application until the user explicitly permits another request, and never bypass a refusal or switch tools to avoid it. Never replay uncertain input without a fresh observation.
|
||||
|
||||
|
||||
Reference in new issue
Block a user