fix(cua): make hover use persistent system-pointer movement
This commit is contained in:
creatixchu committed
1 parent
8304d5dee0
commit
8a7f941112
18 files changed
+51
-71
No files matched your search
@@ -9,11 +9,11 @@
|
||||
en: cdc04139c8d69415
|
||||
zh: a3fe3e78ada08d95
|
||||
/agent-note-continuous-cua-hover-without-cross-call-ownership/decision:
|
||||
en: 2b7f851ba01445e6
|
||||
zh: c4401d3ebbd0b353
|
||||
en: 546ca32488de6dd7
|
||||
zh: 646717cf294498f0
|
||||
/agent-note-continuous-cua-hover-without-cross-call-ownership/alternatives-considered:
|
||||
en: 1b1a6ad5f6862fdf
|
||||
zh: 91675194375a1ad3
|
||||
en: 0a873be31e432506
|
||||
zh: 86eb043a33321587
|
||||
/agent-note-continuous-cua-hover-without-cross-call-ownership/consequences:
|
||||
en: 714789d0a472a308
|
||||
zh: e324c2eb537a5bf9
|
||||
@@ -10,7 +10,7 @@ Hover interactions span movement, delayed content, observation and input inside
|
||||
|
||||
## Decision
|
||||
|
||||
App and Window expose `hover` for an observed AX element or window-local point. It sends unpressed movement, including over disabled controls. Completion, AX reads and screenshots do not restore a previous pointer position. Successive calls can visit intermediate points before entering a popup. Clicks deliver button events at their target without an off-window priming click. Background movement reaches Chromium hover. Native AppKit tracking can require the explicit `useSystemPointer` option, which moves the visible cursor only after checking the foreground app and unobscured target window. It never activates or raises the app implicitly. Other input or window changes can race that check.
|
||||
App and Window expose `hover` for an observed AX element or window-local point. It sends unpressed movement, including over disabled controls. Completion, AX reads and screenshots do not restore a previous pointer position. Successive calls can visit intermediate points before entering a popup. Clicks deliver button events at their target without an off-window priming click. Hover moves the visible system cursor after checking the foreground app and unobscured target window. It never activates or raises the app implicitly. Other input or window changes can race that check.
|
||||
|
||||
The [shared helper](2026-10-09-cua-shared-native-helper.md) serializes complete individual operations. It grants no application ownership across calls. Earlier arrival, a retained app object and Session lifetime do not establish task priority. Waiting for the model does not reserve the application. Explicit task coordination determines any required execution order.
|
||||
|
||||
@@ -20,6 +20,8 @@ The [shared helper](2026-10-09-cua-shared-native-helper.md) serializes complete
|
||||
|
||||
**Click or drag as hover** introduces button input that can activate a control, select text or dismiss a popup. Hover requires a separate unpressed movement operation.
|
||||
|
||||
**PID-directed virtual movement** can display Chromium hover content after prior interaction, but a fresh window can ignore it, and later pointer checks can dismiss the content. Native AppKit tracking also requires real pointer movement in the measured fixture. A single system-pointer operation avoids reporting this unreliable route as continuous hover.
|
||||
|
||||
## Consequences
|
||||
|
||||
Other input and application-driven changes can interrupt the interaction; callers observe fresh state before continuing. Dispatch does not verify that a hover delay has elapsed or a popup has appeared. The Electron fixture verifies delayed reveal, AX and screenshot preservation, entry into the popup, exactly one trusted click and dismissal on exit. The recorded Session covers hover dispatch and observed popup content; it does not substitute for native event verification.
|
||||
@@ -10,7 +10,7 @@ Status: implemented
|
||||
|
||||
## 决策
|
||||
|
||||
App 和 Window 提供 `hover`，接受已观察的 AX 元素或窗口内坐标。操作只发送未按键的移动，也可悬停禁用控件。调用结束、AX 读取和截图不恢复先前的鼠标位置。进入浮层前可连续调用并经过中间位置。点击只在目标位置发送按键事件，不先在窗口外点击。 后台移动可触发 Chromium 悬停。原生 AppKit 跟踪可能需要显式指定 `useSystemPointer`，该选项检查前台应用和未遮挡的目标窗口后移动可见鼠标，不隐式激活或抬起应用。其他输入或窗口变化仍可能与检查交错。
|
||||
App 和 Window 提供 `hover`，接受已观察的 AX 元素或窗口内坐标。操作只发送未按键的移动，也可悬停禁用控件。调用结束、AX 读取和截图不恢复先前的鼠标位置。进入浮层前可连续调用并经过中间位置。点击只在目标位置发送按键事件，不先在窗口外点击。 悬停检查前台应用和未遮挡的目标窗口后移动可见的系统鼠标，不隐式激活或抬起应用。其他输入或窗口变化仍可能与检查交错。
|
||||
|
||||
[共享 helper](2026-10-09-cua-shared-native-helper.zh.md)串行执行完整的单次操作，不分配跨调用的应用所有权。先发起请求、持有 app 对象和 Session 寿命均不代表任务优先级。等待模型期间不占用应用。需要确定执行顺序时，由明确的任务协调决定。
|
||||
|
||||
@@ -20,6 +20,8 @@ App 和 Window 提供 `hover`，接受已观察的 AX 元素或窗口内坐标
|
||||
|
||||
**用点击或拖动代替悬停**会引入按键输入，可能触发控件、选中文字或关闭浮层。悬停需要独立的不按键移动操作。
|
||||
|
||||
**向 PID 定向发送虚拟移动**可能在已有交互的 Chromium 窗口中显示悬停内容，但新窗口可能忽略它，后续鼠标位置检查也可能关闭内容。实测 AppKit 跟踪区域同样需要真实鼠标移动。统一使用系统鼠标操作，避免把这条不可靠的路径当作连续悬停。
|
||||
|
||||
## 影响
|
||||
|
||||
其他输入和应用自身变化可能中断交互，调用方需要重新观察再继续。分派成功不代表悬停延迟已经结束或浮层已经出现。Electron fixture 验证延迟出现、AX 和截图保留悬停、进入浮层、恰好一次可信点击以及移出后关闭。Session 回放覆盖悬停分派和浮层内容观察，不代替原生事件验证。
|
||||
@@ -18,8 +18,8 @@
|
||||
en: b80fbe57355aefad
|
||||
zh: 6ae84f80ad3f7829
|
||||
/deepseek-ai-dsh-experimental-cua-macos/use-this-package/connect-an-external-execution-environment:
|
||||
en: 43d9fbb666d5f081
|
||||
zh: 15125c94448f173b
|
||||
en: 6143c68497c74147
|
||||
zh: b65acc877da5ac9d
|
||||
/deepseek-ai-dsh-experimental-cua-macos/understand-the-implementation:
|
||||
en: 8b9ec208ca057a4a
|
||||
zh: e8dbb45891ad15b8
|
||||
|
||||
@@ -87,7 +87,7 @@ Use app methods directly for single-window tasks. `getApp` reads initial AX stat
|
||||
|
||||
`getWindow` emits its initial AX tree. An owned window without an AX surface returns an empty snapshot and remains available for screenshots and coordinate input. Window objects retain separate snapshots, so reading one window does not invalidate another. Re-reading the same scope invalidates old indices. An AX node removed during observation rejects the read with `STALE_SNAPSHOT` and publishes no partial tree; read again without repeating the preceding input. App-scoped screenshots and coordinates reuse the latest AX observation: its focused on-screen, nonzero-size window, otherwise the sole such window. They do not query current focus again; read `app.getAXState()` to follow a focus change. Explicit window objects retain their selected target. Keyboard methods stay on `app` because input follows the app’s current focus.
|
||||
|
||||
`hover(target)` moves to an AX element center or window-local point without pressing a button, including over disabled controls. Call completion, AX reads and screenshots leave the hover position unchanged. Wait for the application’s hover delay, observe the revealed content, then hover into it and click; successive calls can visit intermediate points. Other input or application changes can interrupt the interaction. [Continuous-interaction rationale](../../../.agents/notes/implemented/architecture/2026-10-13-cua-continuous-hover.md). Applications can ignore background movement, including native AppKit tracking and freshly opened Chromium windows. These can require `{ useSystemPointer: true }`. This explicitly moves the visible system cursor, requires the selected app to be foreground and its target point unobscured, and rejects with `FOREGROUND_REQUIRED` otherwise. It never activates or raises a window automatically.
|
||||
`hover(target)` moves the visible system cursor to an AX element center or window-local point without pressing a button, including over disabled controls. It requires the selected app to be foreground and the target point unobscured; otherwise `FOREGROUND_REQUIRED` requires explicit activation or raising and a fresh observation. Completion, AX reads and screenshots do not restore the previous point. Wait for the application’s hover delay, observe the revealed content, then hover into it and click; successive calls can visit intermediate points. Other input or application changes can interrupt the interaction. [Continuous-interaction rationale](../../../.agents/notes/implemented/architecture/2026-10-13-cua-continuous-hover.md).
|
||||
|
||||
`paste(text, { format: "text" | "html" })` temporarily publishes the requested clipboard format and invokes the application’s enabled standard Paste menu command. It keeps the temporary format until a pasteboard reader requests its data, then restores every captured original format while it still owns the clipboard, preserving a later copy. Paste operations share a cross-process lock and cancellation joins helper cleanup. A forced process exit can prevent restoration. Unsupported Paste commands fail without typing fallback; `typeText` remains separate key-event input.
|
||||
|
||||
@@ -171,7 +171,7 @@ App and Window methods:
|
||||
- getAXState(options?: { emit?: boolean, disableDiffing?: boolean, includeMenus?: boolean, details?: boolean }) -> string
|
||||
- getScreenshot(options?: { emit?: boolean }) -> Uint8Array
|
||||
- getAXStateAndScreenshot(options?) -> { state: string, screenshot: Uint8Array }
|
||||
- hover(target: number | [x, y], options?: { useSystemPointer?: boolean }) -> action result
|
||||
- hover(target: number | [x, y]) -> action result
|
||||
- click(target: number | [x, y], options?: { mouseButton?: "left" | "right" | "middle", clickCount?: 1 | 2, modifiers?: Array<"command" | "control" | "option" | "shift" | "fn"> }) -> action result
|
||||
- setValue(elementIndex: number, value: string) -> action result
|
||||
- selectText(elementIndex: number, text: string, options?: { prefix?: string, suffix?: string, selectionType?: "text" | "cursor_before" | "cursor_after" }) -> action result
|
||||
@@ -183,7 +183,7 @@ Raw operation tools use { method, params }, including params: {} for parameterle
|
||||
|
||||
Await operations sequentially. After input, read fresh state to verify the effect. A successful action means dispatched, not verified completion. Cancellation may leave partial effects; never retry uncertain input without observing. STALE_APP requires getApp again; STALE_SNAPSHOT requires getState again. Coordinate hovering, clicks, scrolling and dragging require a successful AX observation or screenshot of that window. Window resizing since the latest observation rejects coordinate input; movement alone preserves window-local coordinates. WINDOW_CHANGED requires fresh state before any retry; use an advertised AX action or explicitly restore the window before screenshot or coordinate input, then observe again. PERMISSION_DENIED requires the user to grant macOS Accessibility or Screen Recording access in System Settings. After granting access, observe fresh state before continuing; do not automatically replay prior input.
|
||||
|
||||
Hover defaults to background window-directed movement. If fresh observation shows no hover response, use hover(target, { useSystemPointer: true }), which moves the visible system cursor and requires the target point to be unobscured in the foreground app. FOREGROUND_REQUIRED requires explicit activation or raising and a fresh observation before retrying. Hover never presses a button; returning, AX reads and screenshots do not restore the previous point. Wait for the application's hover delay, observe, then hover into the revealed controls before clicking. Use successive hover calls for intermediate points. Other input or application changes can interrupt the interaction; app objects do not reserve applications between calls.
|
||||
Hover moves the visible system cursor without pressing a button. It requires the target point to be unobscured in the foreground app. FOREGROUND_REQUIRED requires explicit activation or raising and a fresh observation before retrying. Returning, AX reads and screenshots do not restore the previous point. Wait for the application's hover delay, observe, then hover into the revealed controls before clicking. Use successive hover calls for intermediate points. Other input or application changes can interrupt the interaction; app objects do not reserve applications between calls.
|
||||
|
||||
Use selectText only on controls advertising selectionWritable. It selects a unique literal occurrence, or places the cursor before/after it; prefix and suffix must be immediately adjacent. AMBIGUOUS_TEXT requires more context; TEXT_NOT_FOUND or SELECTION_CHANGED requires a fresh observation before deciding on input. Left clicks support single/double clicks; right and middle clicks support single clicks. Modified clicks use pointer input. Keypad names such as KP_0 and KP_Enter retain their physical key identity. Window objects expose observation and pointer/AX actions; keyboard input remains app-scoped and follows current focus. Scroll deltas are integer lines from -10 to 10; positive values reveal content above or left. Paste delivers plain text or HTML through the application's enabled standard Paste menu command. Original clipboard formats are restored if the clipboard still belongs to this paste; a later copy is preserved. Unsupported Paste commands fail without typing fallback. CLIPBOARD_CHANGED means another writer intervened; observe before deciding on further input. Browser applications can be controlled with getApp like other native apps; browser tabs are unavailable. Installed-app search covers standard macOS application directories; use an absolute application path for another location. Launch does not restore a window or document, and cancellation may leave an application already started. The execution host must install the object API and own variable persistence; this plugin does not preload a cua global. In persistent PTC, use the host-provided initialization snippet below once. Its installation-owned file URL works independently of the project directory. Retain app objects between calls. Observation methods emit by default and also return their value. Use void await app.getAXState() to avoid a duplicate REPL completion, or pass emit:false and print the returned text once. End screenshot calls with void await app.getScreenshot() or void await app.getAXStateAndScreenshot() so image bytes are not serialized as the program completion value.
|
||||
```
|
||||
|
||||
@@ -87,7 +87,7 @@ void await app.getScreenshot()
|
||||
|
||||
`getWindow` 会输出初始 AX 树。已确认归属但没有 AX 界面的窗口返回空快照，仍可截图和发送坐标输入。不同窗口分别保留快照，读取一个窗口不会使另一个窗口失效；重新读取同一范围会使旧索引失效。观察期间节点被移除时，该次读取返回 `STALE_SNAPSHOT`，不发布部分树；可以重新读取，但不要重复此前的输入。应用级截图和坐标操作复用最近一次 AX 观察：优先使用其中在屏幕上且尺寸非零的焦点窗口，否则要求只有一个符合条件的窗口。它们不会重新查询即时焦点；需要跟随焦点变化时，先读取 `app.getAXState()`。显式窗口对象保持选定的目标。键盘方法保留在 `app` 上，因为输入跟随应用的当前焦点。
|
||||
|
||||
`hover(target)` 将鼠标移到 AX 元素中心或窗口内坐标，不按鼠标按钮，也可用于禁用控件。调用结束、AX 读取和截图保持悬停位置。等待应用的悬停延迟后，观察出现的内容，再移入并点击；连续调用可经过中间位置。其他输入或应用自身变化可能中断交互。[连续交互的设计理由](../../../.agents/notes/implemented/architecture/2026-10-13-cua-continuous-hover.zh.md)。 应用可能忽略后台移动，已观察到的情况包括原生 AppKit 跟踪和刚打开的 Chromium 窗口。这些情况可能需要 `{ useSystemPointer: true }`。该选项显式移动可见的系统鼠标，要求所选应用在前台且目标点未被遮挡，否则返回 `FOREGROUND_REQUIRED`，不会自动激活或抬起窗口。
|
||||
`hover(target)` 将可见的系统鼠标移到 AX 元素中心或窗口内坐标，不按鼠标按钮，也可用于禁用控件。所选应用必须在前台，且目标点未被遮挡，否则返回 `FOREGROUND_REQUIRED`，调用方需显式激活或抬起窗口并重新观察。调用结束、AX 读取和截图不恢复先前的位置。等待应用的悬停延迟后，观察出现的内容，再移入并点击；连续调用可经过中间位置。其他输入或应用自身变化可能中断交互。[连续交互的设计理由](../../../.agents/notes/implemented/architecture/2026-10-13-cua-continuous-hover.zh.md)。
|
||||
|
||||
`paste(text, { format: "text" | "html" })` 临时提供指定格式的剪贴板内容，并调用应用已启用的标准 Paste 菜单命令。临时格式保留到剪贴板读取方请求其数据，然后只在仍持有剪贴板时恢复保存的全部原格式，保留期间产生的新复制内容。粘贴操作共用跨进程锁，取消会等待辅助进程清理。强制终止进程可能阻止恢复。应用不支持 Paste 命令时明确失败，不自动改成逐字输入；`typeText` 仍单独提供按键事件输入。
|
||||
|
||||
@@ -171,7 +171,7 @@ App and Window methods:
|
||||
- getAXState(options?: { emit?: boolean, disableDiffing?: boolean, includeMenus?: boolean, details?: boolean }) -> string
|
||||
- getScreenshot(options?: { emit?: boolean }) -> Uint8Array
|
||||
- getAXStateAndScreenshot(options?) -> { state: string, screenshot: Uint8Array }
|
||||
- hover(target: number | [x, y], options?: { useSystemPointer?: boolean }) -> action result
|
||||
- hover(target: number | [x, y]) -> action result
|
||||
- click(target: number | [x, y], options?: { mouseButton?: "left" | "right" | "middle", clickCount?: 1 | 2, modifiers?: Array<"command" | "control" | "option" | "shift" | "fn"> }) -> action result
|
||||
- setValue(elementIndex: number, value: string) -> action result
|
||||
- selectText(elementIndex: number, text: string, options?: { prefix?: string, suffix?: string, selectionType?: "text" | "cursor_before" | "cursor_after" }) -> action result
|
||||
@@ -183,7 +183,7 @@ Raw operation tools use { method, params }, including params: {} for parameterle
|
||||
|
||||
Await operations sequentially. After input, read fresh state to verify the effect. A successful action means dispatched, not verified completion. Cancellation may leave partial effects; never retry uncertain input without observing. STALE_APP requires getApp again; STALE_SNAPSHOT requires getState again. Coordinate hovering, clicks, scrolling and dragging require a successful AX observation or screenshot of that window. Window resizing since the latest observation rejects coordinate input; movement alone preserves window-local coordinates. WINDOW_CHANGED requires fresh state before any retry; use an advertised AX action or explicitly restore the window before screenshot or coordinate input, then observe again. PERMISSION_DENIED requires the user to grant macOS Accessibility or Screen Recording access in System Settings. After granting access, observe fresh state before continuing; do not automatically replay prior input.
|
||||
|
||||
Hover defaults to background window-directed movement. If fresh observation shows no hover response, use hover(target, { useSystemPointer: true }), which moves the visible system cursor and requires the target point to be unobscured in the foreground app. FOREGROUND_REQUIRED requires explicit activation or raising and a fresh observation before retrying. Hover never presses a button; returning, AX reads and screenshots do not restore the previous point. Wait for the application's hover delay, observe, then hover into the revealed controls before clicking. Use successive hover calls for intermediate points. Other input or application changes can interrupt the interaction; app objects do not reserve applications between calls.
|
||||
Hover moves the visible system cursor without pressing a button. It requires the target point to be unobscured in the foreground app. FOREGROUND_REQUIRED requires explicit activation or raising and a fresh observation before retrying. Returning, AX reads and screenshots do not restore the previous point. Wait for the application's hover delay, observe, then hover into the revealed controls before clicking. Use successive hover calls for intermediate points. Other input or application changes can interrupt the interaction; app objects do not reserve applications between calls.
|
||||
|
||||
Use selectText only on controls advertising selectionWritable. It selects a unique literal occurrence, or places the cursor before/after it; prefix and suffix must be immediately adjacent. AMBIGUOUS_TEXT requires more context; TEXT_NOT_FOUND or SELECTION_CHANGED requires a fresh observation before deciding on input. Left clicks support single/double clicks; right and middle clicks support single clicks. Modified clicks use pointer input. Keypad names such as KP_0 and KP_Enter retain their physical key identity. Window objects expose observation and pointer/AX actions; keyboard input remains app-scoped and follows current focus. Scroll deltas are integer lines from -10 to 10; positive values reveal content above or left. Paste delivers plain text or HTML through the application's enabled standard Paste menu command. Original clipboard formats are restored if the clipboard still belongs to this paste; a later copy is preserved. Unsupported Paste commands fail without typing fallback. CLIPBOARD_CHANGED means another writer intervened; observe before deciding on further input. Browser applications can be controlled with getApp like other native apps; browser tabs are unavailable. Installed-app search covers standard macOS application directories; use an absolute application path for another location. Launch does not restore a window or document, and cancellation may leave an application already started. The execution host must install the object API and own variable persistence; this plugin does not preload a cua global. In persistent PTC, use the host-provided initialization snippet below once. Its installation-owned file URL works independently of the project directory. Retain app objects between calls. Observation methods emit by default and also return their value. Use void await app.getAXState() to avoid a duplicate REPL completion, or pass emit:false and print the returned text once. End screenshot calls with void await app.getScreenshot() or void await app.getAXStateAndScreenshot() so image bytes are not serialized as the program completion value.
|
||||
```
|
||||
|
||||
@@ -30,7 +30,7 @@ Every app-targeted operation requires `appHandle` from `getApp` in the same help
|
||||
|
||||
`getState` optionally scopes AX traversal to an owned window or nested sheet `windowId`. A new or failed refresh invalidates the previous snapshot for that app/window scope, while other window scopes remain usable. When an owned CG window has no matching AX surface, getState returns a new snapshot with empty tree/elements and no truncation; its coordinates and screenshots remain available. It rechecks process lifetime, window ownership, the unlocked session and Accessibility permission after traversal. Window-discovery APIDisabled and communication errors remain failures; unsupported or absent optional AX attributes can produce an empty observation. Missing or foreign-owned windows never produce an empty success. An AX element invalidated during traversal rejects the whole read with `STALE_SNAPSHOT`; no partial snapshot is published. Callers may read again after the application changes, without repeating input. Element actions require the exact snapshot ID. Native references never cross JSON. An optional snapshot ID on coordinate actions must belong to that window or an app-wide observation. Window-list refresh releases snapshots for closed windows. Native process identity, snapshot generation, and window ownership checks do not lock the desktop against simultaneous user or application changes.
|
||||
|
||||
Element clicks prefer advertised AX actions and otherwise use pointer delivery. Coordinate hover, click, scroll and drag validate window ownership and bounds, then use background PID/window-directed primitives. Each event uses SkyLight when available, otherwise the public PID-targeted API. A posted event is never repeated through the other API. Coordinates are macOS points from the top-left of the entire window, including its title bar. Positive `deltaLinesY` reveals content above; negative reveals below. Positive `deltaLinesX` reveals content left; negative reveals right. Each axis accepts an integer from -10 to 10, and at least one must be nonzero. The helper sends line-unit scroll events directly. Click and drag accept left/right/middle buttons and command/control/option/shift/fn modifiers. Only left clicks support double-click. Modified element clicks use pointer delivery. Drag uses one button press/move/release in one window; callers explicitly supply durationMs (1..5000) and steps (1..240). The movement delay budget is divided with fractional precision across the segments; fixed press/release settling and scheduler latency are additional. No implicit fallback sends global desktop input.
|
||||
Element clicks prefer advertised AX actions and otherwise use pointer delivery. Coordinate click, scroll and drag validate window ownership and bounds, then use background PID/window-directed primitives. Each event uses SkyLight when available, otherwise the public PID-targeted API. A posted event is never repeated through the other API. Coordinates are macOS points from the top-left of the entire window, including its title bar. Positive `deltaLinesY` reveals content above; negative reveals below. Positive `deltaLinesX` reveals content left; negative reveals right. Each axis accepts an integer from -10 to 10, and at least one must be nonzero. The helper sends line-unit scroll events directly. Click and drag accept left/right/middle buttons and command/control/option/shift/fn modifiers. Only left clicks support double-click. Modified element clicks use pointer delivery. Drag uses one button press/move/release in one window; callers explicitly supply durationMs (1..5000) and steps (1..240). The movement delay budget is divided with fractional precision across the segments; fixed press/release settling and scheduler latency are additional. No implicit fallback sends global desktop input.
|
||||
|
||||
Capture and pointer delivery read CG bounds before and after an exact-window AX frame query. Movement during the query returns WINDOW_CHANGED. Valid AX/CG frames must have equal outward-rounded integral point rectangles; a known mismatch returns WINDOW_CHANGED before mouse movement, priming or button delivery. The helper preserves both coordinate sources rather than substituting AX geometry for CG geometry. Missing or unusable AX geometry remains unknown, while AX communication, permission and invalid-element errors propagate. Capture without Accessibility permission retains its CG/SCK checks. Advertised AX actions bypass this pointer preflight; an AX action error does not trigger pointer input. These reads do not lock the window against later movement, and no geometry check activates or raises it.
|
||||
|
||||
@@ -76,4 +76,4 @@ Run `python3 packages/experimental/cua/native/tests/capabilities_e2e.py --helper
|
||||
|
||||
`hover` accepts an observed element index or a window-local point and sends only unpressed movement. Disabled controls may be hovered. No request completion, AX read or screenshot restores the previous point; subsequent hover calls can enter revealed content. Click sends its button events at the target without an off-window priming click. Individual requests are serialized, but no application is reserved between calls. Other input or application changes can interrupt the interaction.
|
||||
|
||||
PID-directed movement does not trigger every AppKit tracking area. Explicit `useSystemPointer: true` posts unpressed movement through the system input queue after checking the foreground PID and first on-screen window at the point. A failed check returns `FOREGROUND_REQUIRED` without moving the cursor. This option does not activate, raise or restore focus. Other input or window changes can race the check and dispatch.
|
||||
`hover` moves the visible system cursor through the system input queue after checking the foreground PID and first on-screen window at the point. A failed check returns `FOREGROUND_REQUIRED` without moving the cursor. Hover does not activate, raise or restore focus. Other input or window changes can race the check and dispatch.
|
||||
@@ -368,7 +368,7 @@ actor AXWorker {
|
||||
point = try params.point()
|
||||
}
|
||||
let route = WindowRoute(owner: identity, windowID: window.windowID, bounds: window.bounds)
|
||||
if method == "hover" { try inputs().1.hover(in: route, at: point, useSystemPointer: params.optionalBoolean("useSystemPointer") ?? false) }
|
||||
if method == "hover" { try inputs().1.hover(in: route, at: point) }
|
||||
else { try inputs().1.click(in: route, at: point, button: button, count: count, modifiers: modifiers) }
|
||||
case "scroll", "drag":
|
||||
let id = try params.windowID()
|
||||
|
||||
+1
-1
@@ -9,7 +9,7 @@ enum NativeProtocol {
|
||||
"getApp": ["selector"],
|
||||
"getCapabilities": [],
|
||||
"getState": ["appHandle", "details", "includeMenus", "maxDepth", "maxElements", "windowId"],
|
||||
"hover": ["appHandle", "elementIndex", "snapshotId", "useSystemPointer", "windowId", "x", "y"],
|
||||
"hover": ["appHandle", "elementIndex", "snapshotId", "windowId", "x", "y"],
|
||||
"launchApp": ["appHandle"],
|
||||
"listApps": [],
|
||||
"listWindows": ["appHandle"],
|
||||
|
||||
@@ -28,27 +28,18 @@ final class WindowPointerInput {
|
||||
Double(durationMs) / Double(steps) / 1000
|
||||
}
|
||||
|
||||
/// Send unpressed movement without restoring the prior point or reserving the application.
|
||||
func hover(in route: WindowRoute, at point: WindowPoint, useSystemPointer: Bool = false) throws {
|
||||
/// Move the visible cursor without restoring the prior point or reserving the application.
|
||||
func hover(in route: WindowRoute, at point: WindowPoint) throws {
|
||||
try checkCancellation()
|
||||
try Self.validate(point, in: route)
|
||||
if useSystemPointer {
|
||||
guard let event = CGEvent(mouseEventSource: nil, mouseType: .mouseMoved,
|
||||
mouseCursorPosition: Self.screenPoint(point, in: route), mouseButton: .left) else {
|
||||
throw NativeFailure(code: "NATIVE_ERROR", message: "Could not create system pointer movement")
|
||||
}
|
||||
event.flags = []
|
||||
try Self.requireSystemPointerTarget(route, at: point, frontPID: NativePlatform.frontPID(), windows: NativePlatform.windows(allLayers: true))
|
||||
try checkCancellation()
|
||||
event.post(tap: .cghidEventTap)
|
||||
} else {
|
||||
let event = try mouseEvent(
|
||||
source: Self.source(), route: route, screen: Self.screenPoint(point, in: route), local: point,
|
||||
type: .mouseMoved, button: .left, clickState: 0, subtype: 3, group: Self.clickGroup(), number: 2
|
||||
)
|
||||
try checkCancellation()
|
||||
poster.post(event, to: route.owner.pid)
|
||||
guard let event = CGEvent(mouseEventSource: nil, mouseType: .mouseMoved,
|
||||
mouseCursorPosition: Self.screenPoint(point, in: route), mouseButton: .left) else {
|
||||
throw NativeFailure(code: "NATIVE_ERROR", message: "Could not create system pointer movement")
|
||||
}
|
||||
event.flags = []
|
||||
try Self.requireSystemPointerTarget(route, at: point, frontPID: NativePlatform.frontPID(), windows: NativePlatform.windows(allLayers: true))
|
||||
try checkCancellation()
|
||||
event.post(tap: .cghidEventTap)
|
||||
try checkCancellation()
|
||||
}
|
||||
|
||||
|
||||
@@ -300,10 +300,6 @@
|
||||
"type": "integer",
|
||||
"minimum": 0,
|
||||
"maximum": 9007199254740991
|
||||
},
|
||||
"useSystemPointer": {
|
||||
"description": "Move the visible system cursor for native hover tracking; requires the target to be unobscured in the foreground app. Default false sends window-directed movement.",
|
||||
"type": "boolean"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
@@ -336,10 +332,6 @@
|
||||
"y": {
|
||||
"type": "number",
|
||||
"minimum": 0
|
||||
},
|
||||
"useSystemPointer": {
|
||||
"description": "Move the visible system cursor for native hover tracking; requires the target to be unobscured in the foreground app. Default false sends window-directed movement.",
|
||||
"type": "boolean"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
|
||||
@@ -182,16 +182,16 @@ def main():
|
||||
def check_hover():
|
||||
call("activate", owner)
|
||||
x, y, width, height = json.loads(state_path.read_text())["dragBounds"]
|
||||
call("hover", {**owner, "windowId": primary["windowId"], "x": x + 20, "y": y + 20, "useSystemPointer": True})
|
||||
call("hover", {**owner, "windowId": primary["windowId"], "x": x + 20, "y": y + 20})
|
||||
wait_state(lambda actual: actual["hoverVisible"], "AppKit tracking area did not reveal hover controls")
|
||||
hover_state = call("getState", primary_params)
|
||||
capture(primary, "02-native-hover.png")
|
||||
assert json.loads(state_path.read_text())["hoverVisible"], "Observation dismissed native hover controls"
|
||||
hover_target = button_target(hover_state, "Use native hover details")
|
||||
call("hover", {**hover_target, "useSystemPointer": True})
|
||||
call("hover", {**hover_target})
|
||||
call("click", hover_target)
|
||||
wait_state(lambda actual: actual["hoverClicks"] == 1, "Native hover control did not activate")
|
||||
call("hover", {**owner, "windowId": primary["windowId"], "x": 10, "y": 40, "useSystemPointer": True})
|
||||
call("hover", {**owner, "windowId": primary["windowId"], "x": 10, "y": 40})
|
||||
wait_state(lambda actual: not actual["hoverVisible"], "AppKit tracking area did not exit")
|
||||
|
||||
if args.hover_only:
|
||||
|
||||
@@ -53,8 +53,7 @@ def main():
|
||||
assert permission == {'accessibility': True, 'screenRecording': True}, permission
|
||||
app = call('getApp', {'selector': str(fixture.pid)})
|
||||
owner = {'appHandle': app['appHandle']}
|
||||
if args.hover_only:
|
||||
call('activate', owner)
|
||||
call('activate', owner)
|
||||
deadline = time.monotonic() + 15
|
||||
while True:
|
||||
windows = call('listWindows', owner)
|
||||
@@ -124,18 +123,17 @@ def main():
|
||||
if args.artifacts:
|
||||
(args.artifacts / 'electron-input.png').write_bytes(png)
|
||||
(args.artifacts / 'fixture-state.json').write_text(state_path.read_text())
|
||||
def hover_sequence(use_system_pointer):
|
||||
def hover_sequence():
|
||||
state = observe()
|
||||
trigger = next(element for element in state['elements'] if element['role'] == 'AXButton' and
|
||||
'Reveal hover details' in (element.get('title') or element.get('description') or ''))
|
||||
before = json.loads(state_path.read_text())
|
||||
baseline = before['pointerDowns']
|
||||
expected_clicks = before['popupClicks'] + 1
|
||||
hover_options = {'useSystemPointer': use_system_pointer}
|
||||
call('hover', {**owner, 'windowId': window_id, 'x': -1, 'y': 0}, 'INVALID_ARGUMENT')
|
||||
call('hover', {**owner, 'windowId': window_id, 'x': 100000, 'y': 0}, 'INVALID_ARGUMENT')
|
||||
call('hover', {**owner, 'snapshotId': state['snapshotId'], 'elementIndex': trigger['index'], 'x': 0}, 'INVALID_ARGUMENT')
|
||||
call('hover', {**owner, 'snapshotId': state['snapshotId'], 'elementIndex': trigger['index'], **hover_options})
|
||||
call('hover', {**owner, 'snapshotId': state['snapshotId'], 'elementIndex': trigger['index']})
|
||||
actual = wait_state(lambda actual: actual['popupOpen'], 'Hover did not reveal delayed information')
|
||||
assert actual['pointerDowns'] == baseline and actual['lastMove'] == {'buttons': 0, 'trusted': True}, actual
|
||||
state = observe()
|
||||
@@ -150,17 +148,15 @@ def main():
|
||||
target = {**owner, 'windowId': window_id,
|
||||
'x': bounds[0] + bounds[2] / 2 - window['bounds']['x'],
|
||||
'y': bounds[1] + bounds[3] / 2 - window['bounds']['y']}
|
||||
call('hover', {**target, **hover_options})
|
||||
call('hover', target)
|
||||
state = observe()
|
||||
assert 'Hover information' in state['tree'], 'Moving into the popup dismissed it'
|
||||
call('click', target)
|
||||
actual = wait_state(lambda actual: actual['popupClicks'] == expected_clicks, 'Popup action did not receive its click')
|
||||
assert actual['popupOpen'] and actual['pointerDowns'] == baseline + 1 and actual['popupClickTrusted'], actual
|
||||
call('hover', {**owner, 'windowId': window_id, 'x': 10, 'y': 40, **hover_options})
|
||||
call('hover', {**owner, 'windowId': window_id, 'x': 10, 'y': 40})
|
||||
wait_state(lambda actual: not actual['popupOpen'], 'Leaving the hover region did not close it')
|
||||
if args.hover_only:
|
||||
hover_sequence(True)
|
||||
hover_sequence(False)
|
||||
hover_sequence()
|
||||
helper.eof()
|
||||
print(f"PASS: Electron {runtime['electron']} delayed hover across observations, popup entry/click/exit, invalid targets"
|
||||
+ (" (text input omitted)" if args.hover_only else ", trusted DOM click, Unicode input, Command+A replacement, PNG capture"))
|
||||
|
||||
@@ -25,7 +25,7 @@ App and Window methods:
|
||||
- getAXState(options?: { emit?: boolean, disableDiffing?: boolean, includeMenus?: boolean, details?: boolean }) -> string
|
||||
- getScreenshot(options?: { emit?: boolean }) -> Uint8Array
|
||||
- getAXStateAndScreenshot(options?) -> { state: string, screenshot: Uint8Array }
|
||||
- hover(target: number | [x, y], options?: { useSystemPointer?: boolean }) -> action result
|
||||
- hover(target: number | [x, y]) -> action result
|
||||
- click(target: number | [x, y], options?: { mouseButton?: "left" | "right" | "middle", clickCount?: 1 | 2, modifiers?: Array<"command" | "control" | "option" | "shift" | "fn"> }) -> action result
|
||||
- setValue(elementIndex: number, value: string) -> action result
|
||||
- selectText(elementIndex: number, text: string, options?: { prefix?: string, suffix?: string, selectionType?: "text" | "cursor_before" | "cursor_after" }) -> action result
|
||||
@@ -37,6 +37,6 @@ Raw operation tools use { method, params }, including params: {} for parameterle
|
||||
|
||||
Await operations sequentially. After input, read fresh state to verify the effect. A successful action means dispatched, not verified completion. Cancellation may leave partial effects; never retry uncertain input without observing. STALE_APP requires getApp again; STALE_SNAPSHOT requires getState again. Coordinate hovering, clicks, scrolling and dragging require a successful AX observation or screenshot of that window. Window resizing since the latest observation rejects coordinate input; movement alone preserves window-local coordinates. WINDOW_CHANGED requires fresh state before any retry; use an advertised AX action or explicitly restore the window before screenshot or coordinate input, then observe again. PERMISSION_DENIED requires the user to grant macOS Accessibility or Screen Recording access in System Settings. After granting access, observe fresh state before continuing; do not automatically replay prior input.
|
||||
|
||||
Hover defaults to background window-directed movement. If fresh observation shows no hover response, use hover(target, { useSystemPointer: true }), which moves the visible system cursor and requires the target point to be unobscured in the foreground app. FOREGROUND_REQUIRED requires explicit activation or raising and a fresh observation before retrying. Hover never presses a button; returning, AX reads and screenshots do not restore the previous point. Wait for the application's hover delay, observe, then hover into the revealed controls before clicking. Use successive hover calls for intermediate points. Other input or application changes can interrupt the interaction; app objects do not reserve applications between calls.
|
||||
Hover moves the visible system cursor without pressing a button. It requires the target point to be unobscured in the foreground app. FOREGROUND_REQUIRED requires explicit activation or raising and a fresh observation before retrying. Returning, AX reads and screenshots do not restore the previous point. Wait for the application's hover delay, observe, then hover into the revealed controls before clicking. Use successive hover calls for intermediate points. Other input or application changes can interrupt the interaction; app objects do not reserve applications between calls.
|
||||
|
||||
Use selectText only on controls advertising selectionWritable. It selects a unique literal occurrence, or places the cursor before/after it; prefix and suffix must be immediately adjacent. AMBIGUOUS_TEXT requires more context; TEXT_NOT_FOUND or SELECTION_CHANGED requires a fresh observation before deciding on input. Left clicks support single/double clicks; right and middle clicks support single clicks. Modified clicks use pointer input. Keypad names such as KP_0 and KP_Enter retain their physical key identity. Window objects expose observation and pointer/AX actions; keyboard input remains app-scoped and follows current focus. Scroll deltas are integer lines from -10 to 10; positive values reveal content above or left. Paste delivers plain text or HTML through the application's enabled standard Paste menu command. Original clipboard formats are restored if the clipboard still belongs to this paste; a later copy is preserved. Unsupported Paste commands fail without typing fallback. CLIPBOARD_CHANGED means another writer intervened; observe before deciding on further input. Browser applications can be controlled with getApp like other native apps; browser tabs are unavailable. Installed-app search covers standard macOS application directories; use an absolute application path for another location. Launch does not restore a window or document, and cancellation may leave an application already started. The execution host must install the object API and own variable persistence; this plugin does not preload a cua global. In persistent PTC, use the host-provided initialization snippet below once. Its installation-owned file URL works independently of the project directory. Retain app objects between calls. Observation methods emit by default and also return their value. Use void await app.getAXState() to avoid a duplicate REPL completion, or pass emit:false and print the returned text once. End screenshot calls with void await app.getScreenshot() or void await app.getAXStateAndScreenshot() so image bytes are not serialized as the program completion value.`
|
||||
@@ -93,11 +93,10 @@ export function createCuaApi(dispatch: CuaDispatch, sink?: CuaObservationSink):
|
||||
const state = await result.getAXState(options)
|
||||
return { state, screenshot: await result.getScreenshot(options) }
|
||||
},
|
||||
async hover(target, options) {
|
||||
const pointer = options?.useSystemPointer === undefined ? {} : { useSystemPointer: options.useSystemPointer }
|
||||
if (typeof target === 'number') return action('hover', { ...snapshotParams(), ...pointer, elementIndex: target })
|
||||
async hover(target) {
|
||||
if (typeof target === 'number') return action('hover', { ...snapshotParams(), elementIndex: target })
|
||||
const [x, y] = target
|
||||
return action('hover', { ...snapshotParams(), ...pointer, windowId: windowId(), x, y })
|
||||
return action('hover', { ...snapshotParams(), windowId: windowId(), x, y })
|
||||
},
|
||||
async click(target, options) {
|
||||
const mouseButton = options?.mouseButton ?? 'left'
|
||||
|
||||
@@ -9,7 +9,6 @@ const point = z.number().nonnegative()
|
||||
const app = { appHandle: z.string().min(1) }
|
||||
const pointer = { mouseButton: z.enum(['left', 'right', 'middle']).optional(),
|
||||
modifiers: z.array(z.enum(['command', 'control', 'option', 'shift', 'fn'])).optional() }
|
||||
const hover = { useSystemPointer: z.boolean().optional().describe('Move the visible system cursor for native hover tracking; requires the target to be unobscured in the foreground app. Default false sends window-directed movement.') }
|
||||
const window = { ...app, windowId: id }
|
||||
const snapshot = { ...app, snapshotId: z.string().min(1), elementIndex: z.number().int().nonnegative() }
|
||||
const coordinate = { ...window, snapshotId: z.string().min(1).optional(), x: point, y: point }
|
||||
@@ -37,8 +36,7 @@ export const Request = z.discriminatedUnion('method', [
|
||||
details: z.boolean().optional().describe('Include identifiers, global-point bounds and complete action names.'),
|
||||
}),
|
||||
z.object({ method: z.literal('hover'), params: z.union([
|
||||
z.object({ ...snapshot, ...hover }).strict(),
|
||||
z.object({ ...coordinate, ...hover }).strict(),
|
||||
z.object(snapshot).strict(), z.object(coordinate).strict(),
|
||||
]) }).strict(),
|
||||
z.object({ method: z.literal('click'), params: z.union([
|
||||
z.object({ ...snapshot, ...pointer, clickCount: z.union([z.literal(1), z.literal(2)]).optional() }).strict(),
|
||||
|
||||
@@ -98,14 +98,14 @@ export interface CuaSurface {
|
||||
*/
|
||||
getAXStateAndScreenshot(options?: CuaAxOptions): Promise<CuaObservation>
|
||||
/**
|
||||
* Move to an observed element's center or window point without pressing a button.
|
||||
* Move the visible system cursor to an observed element center or window point without pressing a button.
|
||||
* Requires the selected app to be foreground and the target point unobscured; never activates or raises it.
|
||||
* Returning and subsequent observations do not restore the previous pointer position.
|
||||
* Other input and application changes can interrupt hover; observe before continuing.
|
||||
* @param target - Current AX index, including disabled controls, or window-local [x,y].
|
||||
* @param options - useSystemPointer moves the visible cursor for native tracking; requires an unobscured foreground target.
|
||||
* @returns Dispatch evidence; the application's hover delay and resulting UI are not verified.
|
||||
*/
|
||||
hover(target: number | readonly [number, number], options?: { useSystemPointer?: boolean }): Promise<CuaActionResult>
|
||||
hover(target: number | readonly [number, number]): Promise<CuaActionResult>
|
||||
/**
|
||||
* Click a current AX index or window point.
|
||||
* @param target - index or [x,y].
|
||||
|
||||
@@ -153,9 +153,9 @@ describe('hover across observations', () => {
|
||||
{ method: 'hover', params: { appHandle: 'app-17', snapshotId: 'snapshot-2', windowId: 23, x: 30, y: 40 } },
|
||||
])
|
||||
const window = await app.getWindow(24)
|
||||
await window.hover([5, 6], { useSystemPointer: true })
|
||||
await window.hover([5, 6])
|
||||
expect(f.dispatch).toHaveBeenLastCalledWith({ method: 'hover', params: {
|
||||
appHandle: 'app-17', snapshotId: 'snapshot-3', windowId: 24, x: 5, y: 6, useSystemPointer: true,
|
||||
appHandle: 'app-17', snapshotId: 'snapshot-3', windowId: 24, x: 5, y: 6,
|
||||
} })
|
||||
f.dispatch.mockRejectedValueOnce(new Error('WINDOW_CHANGED'))
|
||||
await expect(window.hover([5, 6])).rejects.toThrow('WINDOW_CHANGED')
|
||||
|
||||
Reference in new issue
Block a user