test(cua): cover fresh-window hover with explicit system input

packages/experimental/cua-macos/README.i18n.yaml

@@ -18,8 +18,8 @@
   en: b80fbe57355aefad
   zh: 6ae84f80ad3f7829
 /deepseek-ai-dsh-experimental-cua-macos/use-this-package/connect-an-external-execution-environment:
-  en: fb04a1a5e6c51a5f
-  zh: 2362a5989d982f46
+  en: 43d9fbb666d5f081
+  zh: 15125c94448f173b
 /deepseek-ai-dsh-experimental-cua-macos/understand-the-implementation:
   en: 8b9ec208ca057a4a
   zh: e8dbb45891ad15b8

packages/experimental/cua-macos/README.md

@@ -87,7 +87,7 @@ Use app methods directly for single-window tasks. `getApp` reads initial AX stat
 
 `getWindow` emits its initial AX tree. An owned window without an AX surface returns an empty snapshot and remains available for screenshots and coordinate input. Window objects retain separate snapshots, so reading one window does not invalidate another. Re-reading the same scope invalidates old indices. An AX node removed during observation rejects the read with `STALE_SNAPSHOT` and publishes no partial tree; read again without repeating the preceding input. App-scoped screenshots and coordinates reuse the latest AX observation: its focused on-screen, nonzero-size window, otherwise the sole such window. They do not query current focus again; read `app.getAXState()` to follow a focus change. Explicit window objects retain their selected target. Keyboard methods stay on `app` because input follows the app’s current focus.
 
-`hover(target)` moves to an AX element center or window-local point without pressing a button, including over disabled controls. Call completion, AX reads and screenshots leave the hover position unchanged. Wait for the application’s hover delay, observe the revealed content, then hover into it and click; successive calls can visit intermediate points. Other input or application changes can interrupt the interaction. [Continuous-interaction rationale](../../../.agents/notes/implemented/architecture/2026-10-13-cua-continuous-hover.md). Native AppKit tracking areas and tooltips can require `{ useSystemPointer: true }`. This explicitly moves the visible system cursor, requires the selected app to be foreground and its target point unobscured, and rejects with `FOREGROUND_REQUIRED` otherwise. It never activates or raises a window automatically.
+`hover(target)` moves to an AX element center or window-local point without pressing a button, including over disabled controls. Call completion, AX reads and screenshots leave the hover position unchanged. Wait for the application’s hover delay, observe the revealed content, then hover into it and click; successive calls can visit intermediate points. Other input or application changes can interrupt the interaction. [Continuous-interaction rationale](../../../.agents/notes/implemented/architecture/2026-10-13-cua-continuous-hover.md). Applications can ignore background movement, including native AppKit tracking and freshly opened Chromium windows. These can require `{ useSystemPointer: true }`. This explicitly moves the visible system cursor, requires the selected app to be foreground and its target point unobscured, and rejects with `FOREGROUND_REQUIRED` otherwise. It never activates or raises a window automatically.
 
 `paste(text, { format: "text" | "html" })` temporarily publishes the requested clipboard format and invokes the application’s enabled standard Paste menu command. It keeps the temporary format until a pasteboard reader requests its data, then restores every captured original format while it still owns the clipboard, preserving a later copy. Paste operations share a cross-process lock and cancellation joins helper cleanup. A forced process exit can prevent restoration. Unsupported Paste commands fail without typing fallback; `typeText` remains separate key-event input.
 
@@ -183,7 +183,7 @@ Raw operation tools use { method, params }, including params: {} for parameterle
 
 Await operations sequentially. After input, read fresh state to verify the effect. A successful action means dispatched, not verified completion. Cancellation may leave partial effects; never retry uncertain input without observing. STALE_APP requires getApp again; STALE_SNAPSHOT requires getState again. Coordinate hovering, clicks, scrolling and dragging require a successful AX observation or screenshot of that window. Window resizing since the latest observation rejects coordinate input; movement alone preserves window-local coordinates. WINDOW_CHANGED requires fresh state before any retry; use an advertised AX action or explicitly restore the window before screenshot or coordinate input, then observe again. PERMISSION_DENIED requires the user to grant macOS Accessibility or Screen Recording access in System Settings. After granting access, observe fresh state before continuing; do not automatically replay prior input.
 
-Hover defaults to background window-directed movement. Native AppKit tracking and tooltips may require hover(target, { useSystemPointer: true }), which moves the visible system cursor and requires the target point to be unobscured in the foreground app. FOREGROUND_REQUIRED requires explicit activation or raising and a fresh observation before retrying. Hover never presses a button; returning, AX reads and screenshots do not restore the previous point. Wait for the application's hover delay, observe, then hover into the revealed controls before clicking. Use successive hover calls for intermediate points. Other input or application changes can interrupt the interaction; app objects do not reserve applications between calls.
+Hover defaults to background window-directed movement. If fresh observation shows no hover response, use hover(target, { useSystemPointer: true }), which moves the visible system cursor and requires the target point to be unobscured in the foreground app. FOREGROUND_REQUIRED requires explicit activation or raising and a fresh observation before retrying. Hover never presses a button; returning, AX reads and screenshots do not restore the previous point. Wait for the application's hover delay, observe, then hover into the revealed controls before clicking. Use successive hover calls for intermediate points. Other input or application changes can interrupt the interaction; app objects do not reserve applications between calls.
 
 Use selectText only on controls advertising selectionWritable. It selects a unique literal occurrence, or places the cursor before/after it; prefix and suffix must be immediately adjacent. AMBIGUOUS_TEXT requires more context; TEXT_NOT_FOUND or SELECTION_CHANGED requires a fresh observation before deciding on input. Left clicks support single/double clicks; right and middle clicks support single clicks. Modified clicks use pointer input. Keypad names such as KP_0 and KP_Enter retain their physical key identity. Window objects expose observation and pointer/AX actions; keyboard input remains app-scoped and follows current focus. Scroll deltas are integer lines from -10 to 10; positive values reveal content above or left. Paste delivers plain text or HTML through the application's enabled standard Paste menu command. Original clipboard formats are restored if the clipboard still belongs to this paste; a later copy is preserved. Unsupported Paste commands fail without typing fallback. CLIPBOARD_CHANGED means another writer intervened; observe before deciding on further input. Browser applications can be controlled with getApp like other native apps; browser tabs are unavailable. Installed-app search covers standard macOS application directories; use an absolute application path for another location. Launch does not restore a window or document, and cancellation may leave an application already started. The execution host must install the object API and own variable persistence; this plugin does not preload a cua global. In persistent PTC, use the host-provided initialization snippet below once. Its installation-owned file URL works independently of the project directory. Retain app objects between calls. Observation methods emit by default and also return their value. Use void await app.getAXState() to avoid a duplicate REPL completion, or pass emit:false and print the returned text once. End screenshot calls with void await app.getScreenshot() or void await app.getAXStateAndScreenshot() so image bytes are not serialized as the program completion value.
 ```

packages/experimental/cua-macos/README.zh.md

@@ -87,7 +87,7 @@ void await app.getScreenshot()
 
 `getWindow` 会输出初始 AX 树。已确认归属但没有 AX 界面的窗口返回空快照，仍可截图和发送坐标输入。不同窗口分别保留快照，读取一个窗口不会使另一个窗口失效；重新读取同一范围会使旧索引失效。观察期间节点被移除时，该次读取返回 `STALE_SNAPSHOT`，不发布部分树；可以重新读取，但不要重复此前的输入。应用级截图和坐标操作复用最近一次 AX 观察：优先使用其中在屏幕上且尺寸非零的焦点窗口，否则要求只有一个符合条件的窗口。它们不会重新查询即时焦点；需要跟随焦点变化时，先读取 `app.getAXState()`。显式窗口对象保持选定的目标。键盘方法保留在 `app` 上，因为输入跟随应用的当前焦点。
 
-`hover(target)` 将鼠标移到 AX 元素中心或窗口内坐标，不按鼠标按钮，也可用于禁用控件。调用结束、AX 读取和截图保持悬停位置。等待应用的悬停延迟后，观察出现的内容，再移入并点击；连续调用可经过中间位置。其他输入或应用自身变化可能中断交互。[连续交互的设计理由](../../../.agents/notes/implemented/architecture/2026-10-13-cua-continuous-hover.zh.md)。 原生 AppKit 跟踪区域和工具提示可能需要 `{ useSystemPointer: true }`。该选项显式移动可见的系统鼠标，要求所选应用在前台且目标点未被遮挡，否则返回 `FOREGROUND_REQUIRED`，不会自动激活或抬起窗口。
+`hover(target)` 将鼠标移到 AX 元素中心或窗口内坐标，不按鼠标按钮，也可用于禁用控件。调用结束、AX 读取和截图保持悬停位置。等待应用的悬停延迟后，观察出现的内容，再移入并点击；连续调用可经过中间位置。其他输入或应用自身变化可能中断交互。[连续交互的设计理由](../../../.agents/notes/implemented/architecture/2026-10-13-cua-continuous-hover.zh.md)。 应用可能忽略后台移动，已观察到的情况包括原生 AppKit 跟踪和刚打开的 Chromium 窗口。这些情况可能需要 `{ useSystemPointer: true }`。该选项显式移动可见的系统鼠标，要求所选应用在前台且目标点未被遮挡，否则返回 `FOREGROUND_REQUIRED`，不会自动激活或抬起窗口。
 
 `paste(text, { format: "text" | "html" })` 临时提供指定格式的剪贴板内容，并调用应用已启用的标准 Paste 菜单命令。临时格式保留到剪贴板读取方请求其数据，然后只在仍持有剪贴板时恢复保存的全部原格式，保留期间产生的新复制内容。粘贴操作共用跨进程锁，取消会等待辅助进程清理。强制终止进程可能阻止恢复。应用不支持 Paste 命令时明确失败，不自动改成逐字输入；`typeText` 仍单独提供按键事件输入。
 
@@ -183,7 +183,7 @@ Raw operation tools use { method, params }, including params: {} for parameterle
 
 Await operations sequentially. After input, read fresh state to verify the effect. A successful action means dispatched, not verified completion. Cancellation may leave partial effects; never retry uncertain input without observing. STALE_APP requires getApp again; STALE_SNAPSHOT requires getState again. Coordinate hovering, clicks, scrolling and dragging require a successful AX observation or screenshot of that window. Window resizing since the latest observation rejects coordinate input; movement alone preserves window-local coordinates. WINDOW_CHANGED requires fresh state before any retry; use an advertised AX action or explicitly restore the window before screenshot or coordinate input, then observe again. PERMISSION_DENIED requires the user to grant macOS Accessibility or Screen Recording access in System Settings. After granting access, observe fresh state before continuing; do not automatically replay prior input.
 
-Hover defaults to background window-directed movement. Native AppKit tracking and tooltips may require hover(target, { useSystemPointer: true }), which moves the visible system cursor and requires the target point to be unobscured in the foreground app. FOREGROUND_REQUIRED requires explicit activation or raising and a fresh observation before retrying. Hover never presses a button; returning, AX reads and screenshots do not restore the previous point. Wait for the application's hover delay, observe, then hover into the revealed controls before clicking. Use successive hover calls for intermediate points. Other input or application changes can interrupt the interaction; app objects do not reserve applications between calls.
+Hover defaults to background window-directed movement. If fresh observation shows no hover response, use hover(target, { useSystemPointer: true }), which moves the visible system cursor and requires the target point to be unobscured in the foreground app. FOREGROUND_REQUIRED requires explicit activation or raising and a fresh observation before retrying. Hover never presses a button; returning, AX reads and screenshots do not restore the previous point. Wait for the application's hover delay, observe, then hover into the revealed controls before clicking. Use successive hover calls for intermediate points. Other input or application changes can interrupt the interaction; app objects do not reserve applications between calls.
 
 Use selectText only on controls advertising selectionWritable. It selects a unique literal occurrence, or places the cursor before/after it; prefix and suffix must be immediately adjacent. AMBIGUOUS_TEXT requires more context; TEXT_NOT_FOUND or SELECTION_CHANGED requires a fresh observation before deciding on input. Left clicks support single/double clicks; right and middle clicks support single clicks. Modified clicks use pointer input. Keypad names such as KP_0 and KP_Enter retain their physical key identity. Window objects expose observation and pointer/AX actions; keyboard input remains app-scoped and follows current focus. Scroll deltas are integer lines from -10 to 10; positive values reveal content above or left. Paste delivers plain text or HTML through the application's enabled standard Paste menu command. Original clipboard formats are restored if the clipboard still belongs to this paste; a later copy is preserved. Unsupported Paste commands fail without typing fallback. CLIPBOARD_CHANGED means another writer intervened; observe before deciding on further input. Browser applications can be controlled with getApp like other native apps; browser tabs are unavailable. Installed-app search covers standard macOS application directories; use an absolute application path for another location. Launch does not restore a window or document, and cancellation may leave an application already started. The execution host must install the object API and own variable persistence; this plugin does not preload a cua global. In persistent PTC, use the host-provided initialization snippet below once. Its installation-owned file URL works independently of the project directory. Retain app objects between calls. Observation methods emit by default and also return their value. Use void await app.getAXState() to avoid a duplicate REPL completion, or pass emit:false and print the returned text once. End screenshot calls with void await app.getScreenshot() or void await app.getAXStateAndScreenshot() so image bytes are not serialized as the program completion value.
 ```

packages/experimental/cua-macos/native/tests/electron_e2e.py

@@ -122,36 +122,44 @@ def main():
                 if args.artifacts:
                     (args.artifacts / 'electron-input.png').write_bytes(png)
                     (args.artifacts / 'fixture-state.json').write_text(state_path.read_text())
-                state = observe()
-                trigger = next(element for element in state['elements'] if element['role'] == 'AXButton' and
-                               'Reveal hover details' in (element.get('title') or element.get('description') or ''))
-                baseline = json.loads(state_path.read_text())['pointerDowns']
-                call('hover', {**owner, 'windowId': window_id, 'x': -1, 'y': 0}, 'INVALID_ARGUMENT')
-                call('hover', {**owner, 'windowId': window_id, 'x': 100000, 'y': 0}, 'INVALID_ARGUMENT')
-                call('hover', {**owner, 'snapshotId': state['snapshotId'], 'elementIndex': trigger['index'], 'x': 0}, 'INVALID_ARGUMENT')
-                call('hover', {**owner, 'snapshotId': state['snapshotId'], 'elementIndex': trigger['index']})
-                actual = wait_state(lambda actual: actual['popupOpen'], 'Hover did not reveal delayed information')
-                assert actual['pointerDowns'] == baseline and actual['lastMove'] == {'buttons': 0, 'trusted': True}, actual
-                state = observe()
-                assert 'Hover information' in state['tree'], state['tree']
-                image = call('screenshot', {**owner, 'windowId': window_id})
-                assert json.loads(state_path.read_text())['popupOpen'], 'Observation dismissed the popup'
-                if args.artifacts:
-                    (args.artifacts / 'electron-hover.png').write_bytes(base64.b64decode(image['data']))
-                action = next(element for element in state['elements'] if element['role'] == 'AXButton' and
-                              'Use hover details' in (element.get('title') or element.get('description') or ''))
-                bounds = action['bounds']
-                target = {**owner, 'windowId': window_id,
-                          'x': bounds[0] + bounds[2] / 2 - window['bounds']['x'],
-                          'y': bounds[1] + bounds[3] / 2 - window['bounds']['y']}
-                call('hover', target)
-                state = observe()
-                assert 'Hover information' in state['tree'], 'Moving into the popup dismissed it'
-                call('click', target)
-                actual = wait_state(lambda actual: actual['popupClicks'] == 1, 'Popup action did not receive its click')
-                assert actual['popupOpen'] and actual['pointerDowns'] == baseline + 1 and actual['popupClickTrusted'], actual
-                call('hover', {**owner, 'windowId': window_id, 'x': 10, 'y': 40})
-                wait_state(lambda actual: not actual['popupOpen'], 'Leaving the hover region did not close it')
+                def hover_sequence(use_system_pointer):
+                    state = observe()
+                    trigger = next(element for element in state['elements'] if element['role'] == 'AXButton' and
+                                   'Reveal hover details' in (element.get('title') or element.get('description') or ''))
+                    before = json.loads(state_path.read_text())
+                    baseline = before['pointerDowns']
+                    expected_clicks = before['popupClicks'] + 1
+                    hover_options = {'useSystemPointer': use_system_pointer}
+                    call('hover', {**owner, 'windowId': window_id, 'x': -1, 'y': 0}, 'INVALID_ARGUMENT')
+                    call('hover', {**owner, 'windowId': window_id, 'x': 100000, 'y': 0}, 'INVALID_ARGUMENT')
+                    call('hover', {**owner, 'snapshotId': state['snapshotId'], 'elementIndex': trigger['index'], 'x': 0}, 'INVALID_ARGUMENT')
+                    call('hover', {**owner, 'snapshotId': state['snapshotId'], 'elementIndex': trigger['index'], **hover_options})
+                    actual = wait_state(lambda actual: actual['popupOpen'], 'Hover did not reveal delayed information')
+                    assert actual['pointerDowns'] == baseline and actual['lastMove'] == {'buttons': 0, 'trusted': True}, actual
+                    state = observe()
+                    assert 'Hover information' in state['tree'], state['tree']
+                    image = call('screenshot', {**owner, 'windowId': window_id})
+                    assert json.loads(state_path.read_text())['popupOpen'], 'Observation dismissed the popup'
+                    if args.artifacts:
+                        (args.artifacts / 'electron-hover.png').write_bytes(base64.b64decode(image['data']))
+                    action = next(element for element in state['elements'] if element['role'] == 'AXButton' and
+                                  'Use hover details' in (element.get('title') or element.get('description') or ''))
+                    bounds = action['bounds']
+                    target = {**owner, 'windowId': window_id,
+                              'x': bounds[0] + bounds[2] / 2 - window['bounds']['x'],
+                              'y': bounds[1] + bounds[3] / 2 - window['bounds']['y']}
+                    call('hover', {**target, **hover_options})
+                    state = observe()
+                    assert 'Hover information' in state['tree'], 'Moving into the popup dismissed it'
+                    call('click', target)
+                    actual = wait_state(lambda actual: actual['popupClicks'] == expected_clicks, 'Popup action did not receive its click')
+                    assert actual['popupOpen'] and actual['pointerDowns'] == baseline + 1 and actual['popupClickTrusted'], actual
+                    call('hover', {**owner, 'windowId': window_id, 'x': 10, 'y': 40, **hover_options})
+                    wait_state(lambda actual: not actual['popupOpen'], 'Leaving the hover region did not close it')
+                if args.hover_only:
+                    call('activate', owner)
+                    hover_sequence(True)
+                hover_sequence(False)
                 helper.eof()
                 print(f"PASS: Electron {runtime['electron']} delayed hover across observations, popup entry/click/exit, invalid targets"
                       + (" (text input omitted)" if args.hover_only else ", trusted DOM click, Unicode input, Command+A replacement, PNG capture"))

packages/experimental/cua-macos/src/guidance.ts

@@ -37,6 +37,6 @@ Raw operation tools use { method, params }, including params: {} for parameterle
 
 Await operations sequentially. After input, read fresh state to verify the effect. A successful action means dispatched, not verified completion. Cancellation may leave partial effects; never retry uncertain input without observing. STALE_APP requires getApp again; STALE_SNAPSHOT requires getState again. Coordinate hovering, clicks, scrolling and dragging require a successful AX observation or screenshot of that window. Window resizing since the latest observation rejects coordinate input; movement alone preserves window-local coordinates. WINDOW_CHANGED requires fresh state before any retry; use an advertised AX action or explicitly restore the window before screenshot or coordinate input, then observe again. PERMISSION_DENIED requires the user to grant macOS Accessibility or Screen Recording access in System Settings. After granting access, observe fresh state before continuing; do not automatically replay prior input.
 
-Hover defaults to background window-directed movement. Native AppKit tracking and tooltips may require hover(target, { useSystemPointer: true }), which moves the visible system cursor and requires the target point to be unobscured in the foreground app. FOREGROUND_REQUIRED requires explicit activation or raising and a fresh observation before retrying. Hover never presses a button; returning, AX reads and screenshots do not restore the previous point. Wait for the application's hover delay, observe, then hover into the revealed controls before clicking. Use successive hover calls for intermediate points. Other input or application changes can interrupt the interaction; app objects do not reserve applications between calls.
+Hover defaults to background window-directed movement. If fresh observation shows no hover response, use hover(target, { useSystemPointer: true }), which moves the visible system cursor and requires the target point to be unobscured in the foreground app. FOREGROUND_REQUIRED requires explicit activation or raising and a fresh observation before retrying. Hover never presses a button; returning, AX reads and screenshots do not restore the previous point. Wait for the application's hover delay, observe, then hover into the revealed controls before clicking. Use successive hover calls for intermediate points. Other input or application changes can interrupt the interaction; app objects do not reserve applications between calls.
 
 Use selectText only on controls advertising selectionWritable. It selects a unique literal occurrence, or places the cursor before/after it; prefix and suffix must be immediately adjacent. AMBIGUOUS_TEXT requires more context; TEXT_NOT_FOUND or SELECTION_CHANGED requires a fresh observation before deciding on input. Left clicks support single/double clicks; right and middle clicks support single clicks. Modified clicks use pointer input. Keypad names such as KP_0 and KP_Enter retain their physical key identity. Window objects expose observation and pointer/AX actions; keyboard input remains app-scoped and follows current focus. Scroll deltas are integer lines from -10 to 10; positive values reveal content above or left. Paste delivers plain text or HTML through the application's enabled standard Paste menu command. Original clipboard formats are restored if the clipboard still belongs to this paste; a later copy is preserved. Unsupported Paste commands fail without typing fallback. CLIPBOARD_CHANGED means another writer intervened; observe before deciding on further input. Browser applications can be controlled with getApp like other native apps; browser tabs are unavailable. Installed-app search covers standard macOS application directories; use an absolute application path for another location. Launch does not restore a window or document, and cancellation may leave an application already started. The execution host must install the object API and own variable persistence; this plugin does not preload a cua global. In persistent PTC, use the host-provided initialization snippet below once. Its installation-owned file URL works independently of the project directory. Retain app objects between calls. Observation methods emit by default and also return their value. Use void await app.getAXState() to avoid a duplicate REPL completion, or pass emit:false and print the returned text once. End screenshot calls with void await app.getScreenshot() or void await app.getAXStateAndScreenshot() so image bytes are not serialized as the program completion value.`