diff --git a/packages/experimental/cua-macos/README.md b/packages/experimental/cua-macos/README.md index 742e218285..6abe3c08de 100644 --- a/packages/experimental/cua-macos/README.md +++ b/packages/experimental/cua-macos/README.md @@ -176,7 +176,7 @@ The plugin contributes one `cua` operation tool with method-specific parameters. ##### Operation guidance (verbatim) ```markdown -macOS: accessibility tree operations are available at app or window level. Simulated keyboard input is available at app level. Simulated mouse input is available at window level. Default hover uses AX alternate UI and keeps the system cursor in place. For pointer input on a sheet, select its window ID with app.getWindow(id). For menu items outside a window, use app.invoke(id). getApp also accepts bundle IDs and .app paths, and findApps searches /Applications, /System/Applications, and ~/Applications. Use the Computer Use permission page to grant the helper application Accessibility and Screen Recording permissions in System Settings. Native window and AX bounds use macOS points, while delivered screenshots carry their own pixel dimensions. show does not switch Spaces, and Stage Manager can present reduced window geometry; explain any required foreground step before calling app.activate() or performing an advertised AXRaise action. +macOS: accessibility tree operations are available at app or window level. Simulated keyboard input is available at app level. Simulated mouse input is available at window level. For pointer input on a sheet, select its window ID with app.getWindow(id). For menu items outside a window, use app.invoke(id). Use the Computer Use permission page to grant the helper application Accessibility and Screen Recording permissions in System Settings. Native window and AX bounds use macOS points, while delivered screenshots carry their own pixel dimensions. Stage Manager can present reduced window geometry; explain any required foreground step before calling app.activate() or performing an advertised AXRaise action. Use cua to observe and operate native applications. Always use cua.getCapabilities() to confirm whether a specific API is available on the current platform; its reported capabilities take precedence over availability statements in these instructions. All methods are async; observation options default to { emit: true }, and emit: false suppresses sink output only, leaving returned values unchanged. Actions report dispatch evidence, not verified completion. If sessionLocked is true or an operation reports SESSION_LOCKED, stop UI actions and ask the user to unlock the desktop session; null or unknown means verify a usable unlocked desktop session first. cua.getApp requests user consent; if access is denied or revoked, stop using that application until the user explicitly permits another request, and never bypass a refusal or switch tools to avoid it. Never replay uncertain input without a fresh observation. @@ -187,20 +187,20 @@ Signatures show resolved return values after await. ActionResult = { dispatched: Setup and discovery: - cua.getCapabilities(options?: { emit?: boolean }) -> { platform: string, permissions: { accessibility: boolean, screenRecording: boolean }, appLevelAccessibility: boolean, keyboard: ('app' | 'window')[], sessionLocked: boolean | null, methods: string[], coordinateSpace: 'screenshot-pixels', browser: false } — inspect reported permissions, sessionLocked, methods, and keyboard scopes without requesting permissions. - cua.listApps(options?: { emit?: boolean }) -> { id: string, displayName: string, isRunning: true, pid: number }[] — enumerate running applications. -- cua.findApps(query: string, options?: { emit?: boolean }) -> ({ id: string, displayName: string, path: string } & ({ isRunning: true, pid: number } | { isRunning: false }))[] — search installed applications, including stopped matches; only running matches carry pid. +- cua.findApps(query: string, options?: { emit?: boolean }) -> ({ id: string, displayName: string, path: string } & ({ isRunning: true, pid: number } | { isRunning: false }))[] — search for applications that match the query. Searches /Applications, /System/Applications, and ~/Applications. - cua.getState(options?: { emit?: boolean }) -> { apps: { id: string, displayName: string, isRunning: true, pid: number }[], browsers: [] } — summarize running applications; this is not an accessibility observation. -- cua.getApp(selector: string) -> App — select an application by name, bundle ID, absolute path, or running PID string, obtain consent, launch in the background if needed, and read initial AX state silently; it does not activate the app. -- app.getWindows(options?: { emit?: boolean }) -> { windowId: number, appName: string, title: string, bounds: { x: number, y: number, width: number, height: number }, isOnScreen: boolean, onCurrentSpace: boolean | null }[] — enumerate windows with global native bounds; null onCurrentSpace means unknown Space membership. -- app.getWindow(windowId: number) -> Window — select one window and read its initial AX state silently; do this before any window screenshot or coordinate input. +- cua.getApp(selector: string) -> App — select an application using a provider-supported selector and obtain consent. When app-level accessibility is available, selection reads the initial tree silently. Accepts a name, bundle ID, .app path, or running PID string. A stopped app launches in the background. +- app.getWindows(options?: { emit?: boolean }) -> { windowId: number, appName: string, title: string, bounds: { x: number, y: number, width: number, height: number }, isOnScreen: boolean, onCurrentSpace: boolean | null }[] — enumerate windows with global native bounds. A null onCurrentSpace means desktop membership is unknown. +- app.getWindow(windowId: number) -> Window — select one window and silently read its initial accessibility tree. Select a window before screenshots or pointer input. - app.activate() -> ActionResult — request explicit foreground activation; use only when foreground access is required. -- app.show() -> { dispatched: boolean, windowId: number | null, foregroundChanged: boolean | null } — unhide the app and restore its selected, focused, main, or sole window without activation. -- window.show() -> { dispatched: boolean, windowId: number | null, foregroundChanged: boolean | null } — unhide and restore that explicit window without activation. +- app.show() -> { dispatched: boolean, windowId: number | null, foregroundChanged: boolean | null } — request that the application be shown. Unhides the app and restores its focused, main, or sole window without activating it. +- window.show() -> { dispatched: boolean, windowId: number | null, foregroundChanged: boolean | null } — request that the selected window be shown. Unhides the app and restores this window without activating it. ## Accessibility tree Available at app or window level depending on the platform. -Element ids are opaque strings from the latest successful tree; the object facade keeps one current snapshot per app across its repeated app and window objects, a new read replaces it, and a failed read invalidates the previous one. Repeated strings can appear for different elements, so do not reuse an old id blindly, and element operations do not fall back to pointer input or typing. -- getAccessibilityTree(options?: { emit?: boolean, includeMenus?: boolean, details?: boolean }) -> string — read the full tree text; includeMenus expands menu contents and details adds identifiers, native bounds, and complete action names. +Element ids are opaque strings from the latest successful tree; the object facade keeps one current snapshot per app across its repeated app and window objects, a new read replaces it, and a failed read invalidates the previous one. Repeated strings can appear for different elements, so use IDs from the most recent observation. +- getAccessibilityTree(options?: { emit?: boolean, includeMenus?: boolean, details?: boolean }) -> string — read the full tree text. includeMenus requests menu contents and details requests additional element information. - listActions(elementId: string) -> string[] — list the actions an element advertises. - invoke(elementId: string) -> ActionResult — invoke the element's default action; dispatched is false when it has none. - performAction(elementId: string, action: string, argument?: JsonValue) -> ActionResult — perform an observed action with an optional direct JSON argument. @@ -210,17 +210,17 @@ Element ids are opaque strings from the latest successful tree; the object facad ## Simulated mouse Use an explicit window for screenshots and pointer input. - getScreenshot(options?: { emit?: boolean }) -> { bytes: Uint8Array, mimeType: 'image/png' | 'image/jpeg' | 'image/webp' | 'image/gif', pixelWidth: number, pixelHeight: number, windowId: number } — capture the selected window as one encoded image; bytes is a JavaScript byte array, and mimeType identifies its encoding, so do not assume PNG. -- mouseMove(target: string | [number, number], options?: { relative?: boolean }) -> { dispatched: boolean, foregroundChanged?: boolean | null, hoverRoute: 'accessibility' | 'system-pointer', systemPointerMoved: boolean } — on macOS, prefer acting on an available control. Default hover requires both AXShowAlternateUI and AXShowDefaultUI on the element or an enabled ancestor. Without an AX route it returns HOVER_UNAVAILABLE and sends no input. For browser content without those actions, use browser tools. Use click for a visible target or performAction for an advertised AXShowAlternateUI, AXShowMenu or AXPress. Only systemPointerHover: true permits moving the visible cursor; this requires a foreground, unobscured target. -- click(target: string | [number, number], options?: { mouseButton?: 'left' | 'right' | 'middle', clickCount?: 1 | 2, modifiers?: ('command' | 'control' | 'option' | 'shift' | 'fn')[], holdMs?: number, relative?: boolean }) -> ActionResult — on macOS, send mouse-down and mouse-up events at the center of an element in this window (id) or at the given point ([x, y]). Prefer invoke(id) for a background control’s default action. When a mouse click is needed, call app.activate() first and observe the target again: a background app can ignore a dispatched click. Set clickCount to 2 to perform a left-button double-click. holdMs controls each press duration. +- mouseMove(target: string | [number, number], options?: { relative?: boolean }) -> { dispatched: boolean, foregroundChanged?: boolean | null, hoverRoute: 'accessibility' | 'system-pointer', systemPointerMoved: boolean } — request hover at an observed element or window point. Inspect hoverRoute and systemPointerMoved to see how the provider dispatched it. Default hover requires both AXShowAlternateUI and AXShowDefaultUI on an enabled element or ancestor. Without an AX route it returns HOVER_UNAVAILABLE before sending input. Prefer click on a visible control or performAction with an advertised action. For web content without an AX hover action, use browser tools. Only systemPointerHover: true permits moving the visible cursor. If this returns FOREGROUND_REQUIRED, activate the app and check that the point is unobscured. +- click(target: string | [number, number], options?: { mouseButton?: 'left' | 'right' | 'middle', clickCount?: 1 | 2, modifiers?: ('command' | 'control' | 'option' | 'shift' | 'fn')[], holdMs?: number, relative?: boolean }) -> ActionResult — click an observed element or a window point. Set clickCount to 2 to perform a left-button double-click. holdMs controls each press duration. Both target forms send mouse-down and mouse-up events: an ID targets the element center and coordinates target the given point. Prefer invoke(id) for a background control’s default action. For a mouse click, call app.activate() first and observe the target again. A background app can ignore a dispatched click. - scroll(options: { x: number, y: number, deltaLinesX: number, deltaLinesY: number, relative?: boolean }) -> ActionResult — scroll at a point; deltas are integers from -10 to 10 with at least one nonzero, and positive values reveal content above or left. - drag(from: [number, number], to: [number, number], options?: { mouseButton?: 'left' | 'right' | 'middle', modifiers?: ('command' | 'control' | 'option' | 'shift' | 'fn')[], relative?: boolean }) -> ActionResult — drag between two points of one window. -A string target is an element id; coordinate targets are [x, y] and are pixels in the latest returned screenshot by default. With relative: true they are integer thousandths of the window width and height from 0 to 999, based on the current AX observation or screenshot. Left clicks support one or two presses; right and middle clicks support one. For hover-only content, prefer an available click or advertised accessibility action. With system-pointer hover enabled, FOREGROUND_REQUIRED means activate the app explicitly and check that the point is unobscured. Read fresh state after actions, and treat WINDOW_CHANGED, STALE_SNAPSHOT, and SCREENSHOT_REQUIRED as requiring the stated fresh observation or capture before any retry. +A string target is an element id; coordinate targets are [x, y] and are pixels in the latest returned screenshot by default. With relative: true they are integer thousandths of the window width and height from 0 to 999, based on the current window observation or screenshot. Left clicks support one or two presses; right and middle clicks support one. Read fresh state after actions, and treat WINDOW_CHANGED, STALE_SNAPSHOT, and SCREENSHOT_REQUIRED as requiring the stated fresh observation or capture before any retry. ## Simulated keyboard Available at app or window level depending on the platform. -- typeText(text: string) -> ActionResult — type literal text at the app's current focus; line endings send Return and tabs send Tab. -- paste(text: string, options?: { format?: 'text' | 'html' }) -> ActionResult — deliver text or HTML through the app's Paste command, restoring prior clipboard contents unless another writer changes them; a Paste failure is surfaced without substituting typeText, and PASTE_NOT_CONSUMED means inspect before any retry. +- typeText(text: string) -> ActionResult — type text at the current input focus. Line endings send Return and tabs send Tab. +- paste(text: string, options?: { format?: 'text' | 'html' }) -> ActionResult — paste text or HTML at the current input focus. Uses the app’s Paste command and restores the clipboard unless another writer changes it. For PASTE_NOT_CONSUMED, inspect the target before retrying. - pressKey(key: string, modifiers?: string[]) -> ActionResult — press and release one complete keystroke. For example pressKey('Tab', ['shift']) or pressKey('+'). Key and modifier names are platform-defined strings. Pass key and modifiers separately; do not pack them into a chord string. Raw tools use { method, params }: include appHandle for app calls and windowId for window calls, and use snapshotId with elementId for element routes. The emit option is facade-only: emit is local output control, so raw callers must follow the raw operation schema for admitted params. getAccessibilityTree and getWindow read through raw getState; getWindows maps to listWindows; getScreenshot maps to screenshot; cua.getState maps to listApps; every other named operation uses the same method name. diff --git a/packages/experimental/cua-macos/README.zh.md b/packages/experimental/cua-macos/README.zh.md index 276baf2131..13f4fa1d06 100644 --- a/packages/experimental/cua-macos/README.zh.md +++ b/packages/experimental/cua-macos/README.zh.md @@ -176,7 +176,7 @@ AppKit 的 sheet 可以出现在文档窗口的树里,并有独立的窗口 ID ##### 操作指导文本(原文) ```markdown -macOS: accessibility tree operations are available at app or window level. Simulated keyboard input is available at app level. Simulated mouse input is available at window level. Default hover uses AX alternate UI and keeps the system cursor in place. For pointer input on a sheet, select its window ID with app.getWindow(id). For menu items outside a window, use app.invoke(id). getApp also accepts bundle IDs and .app paths, and findApps searches /Applications, /System/Applications, and ~/Applications. Use the Computer Use permission page to grant the helper application Accessibility and Screen Recording permissions in System Settings. Native window and AX bounds use macOS points, while delivered screenshots carry their own pixel dimensions. show does not switch Spaces, and Stage Manager can present reduced window geometry; explain any required foreground step before calling app.activate() or performing an advertised AXRaise action. +macOS: accessibility tree operations are available at app or window level. Simulated keyboard input is available at app level. Simulated mouse input is available at window level. For pointer input on a sheet, select its window ID with app.getWindow(id). For menu items outside a window, use app.invoke(id). Use the Computer Use permission page to grant the helper application Accessibility and Screen Recording permissions in System Settings. Native window and AX bounds use macOS points, while delivered screenshots carry their own pixel dimensions. Stage Manager can present reduced window geometry; explain any required foreground step before calling app.activate() or performing an advertised AXRaise action. Use cua to observe and operate native applications. Always use cua.getCapabilities() to confirm whether a specific API is available on the current platform; its reported capabilities take precedence over availability statements in these instructions. All methods are async; observation options default to { emit: true }, and emit: false suppresses sink output only, leaving returned values unchanged. Actions report dispatch evidence, not verified completion. If sessionLocked is true or an operation reports SESSION_LOCKED, stop UI actions and ask the user to unlock the desktop session; null or unknown means verify a usable unlocked desktop session first. cua.getApp requests user consent; if access is denied or revoked, stop using that application until the user explicitly permits another request, and never bypass a refusal or switch tools to avoid it. Never replay uncertain input without a fresh observation. @@ -187,20 +187,20 @@ Signatures show resolved return values after await. ActionResult = { dispatched: Setup and discovery: - cua.getCapabilities(options?: { emit?: boolean }) -> { platform: string, permissions: { accessibility: boolean, screenRecording: boolean }, appLevelAccessibility: boolean, keyboard: ('app' | 'window')[], sessionLocked: boolean | null, methods: string[], coordinateSpace: 'screenshot-pixels', browser: false } — inspect reported permissions, sessionLocked, methods, and keyboard scopes without requesting permissions. - cua.listApps(options?: { emit?: boolean }) -> { id: string, displayName: string, isRunning: true, pid: number }[] — enumerate running applications. -- cua.findApps(query: string, options?: { emit?: boolean }) -> ({ id: string, displayName: string, path: string } & ({ isRunning: true, pid: number } | { isRunning: false }))[] — search installed applications, including stopped matches; only running matches carry pid. +- cua.findApps(query: string, options?: { emit?: boolean }) -> ({ id: string, displayName: string, path: string } & ({ isRunning: true, pid: number } | { isRunning: false }))[] — search for applications that match the query. Searches /Applications, /System/Applications, and ~/Applications. - cua.getState(options?: { emit?: boolean }) -> { apps: { id: string, displayName: string, isRunning: true, pid: number }[], browsers: [] } — summarize running applications; this is not an accessibility observation. -- cua.getApp(selector: string) -> App — select an application by name, bundle ID, absolute path, or running PID string, obtain consent, launch in the background if needed, and read initial AX state silently; it does not activate the app. -- app.getWindows(options?: { emit?: boolean }) -> { windowId: number, appName: string, title: string, bounds: { x: number, y: number, width: number, height: number }, isOnScreen: boolean, onCurrentSpace: boolean | null }[] — enumerate windows with global native bounds; null onCurrentSpace means unknown Space membership. -- app.getWindow(windowId: number) -> Window — select one window and read its initial AX state silently; do this before any window screenshot or coordinate input. +- cua.getApp(selector: string) -> App — select an application using a provider-supported selector and obtain consent. When app-level accessibility is available, selection reads the initial tree silently. Accepts a name, bundle ID, .app path, or running PID string. A stopped app launches in the background. +- app.getWindows(options?: { emit?: boolean }) -> { windowId: number, appName: string, title: string, bounds: { x: number, y: number, width: number, height: number }, isOnScreen: boolean, onCurrentSpace: boolean | null }[] — enumerate windows with global native bounds. A null onCurrentSpace means desktop membership is unknown. +- app.getWindow(windowId: number) -> Window — select one window and silently read its initial accessibility tree. Select a window before screenshots or pointer input. - app.activate() -> ActionResult — request explicit foreground activation; use only when foreground access is required. -- app.show() -> { dispatched: boolean, windowId: number | null, foregroundChanged: boolean | null } — unhide the app and restore its selected, focused, main, or sole window without activation. -- window.show() -> { dispatched: boolean, windowId: number | null, foregroundChanged: boolean | null } — unhide and restore that explicit window without activation. +- app.show() -> { dispatched: boolean, windowId: number | null, foregroundChanged: boolean | null } — request that the application be shown. Unhides the app and restores its focused, main, or sole window without activating it. +- window.show() -> { dispatched: boolean, windowId: number | null, foregroundChanged: boolean | null } — request that the selected window be shown. Unhides the app and restores this window without activating it. ## Accessibility tree Available at app or window level depending on the platform. -Element ids are opaque strings from the latest successful tree; the object facade keeps one current snapshot per app across its repeated app and window objects, a new read replaces it, and a failed read invalidates the previous one. Repeated strings can appear for different elements, so do not reuse an old id blindly, and element operations do not fall back to pointer input or typing. -- getAccessibilityTree(options?: { emit?: boolean, includeMenus?: boolean, details?: boolean }) -> string — read the full tree text; includeMenus expands menu contents and details adds identifiers, native bounds, and complete action names. +Element ids are opaque strings from the latest successful tree; the object facade keeps one current snapshot per app across its repeated app and window objects, a new read replaces it, and a failed read invalidates the previous one. Repeated strings can appear for different elements, so use IDs from the most recent observation. +- getAccessibilityTree(options?: { emit?: boolean, includeMenus?: boolean, details?: boolean }) -> string — read the full tree text. includeMenus requests menu contents and details requests additional element information. - listActions(elementId: string) -> string[] — list the actions an element advertises. - invoke(elementId: string) -> ActionResult — invoke the element's default action; dispatched is false when it has none. - performAction(elementId: string, action: string, argument?: JsonValue) -> ActionResult — perform an observed action with an optional direct JSON argument. @@ -210,17 +210,17 @@ Element ids are opaque strings from the latest successful tree; the object facad ## Simulated mouse Use an explicit window for screenshots and pointer input. - getScreenshot(options?: { emit?: boolean }) -> { bytes: Uint8Array, mimeType: 'image/png' | 'image/jpeg' | 'image/webp' | 'image/gif', pixelWidth: number, pixelHeight: number, windowId: number } — capture the selected window as one encoded image; bytes is a JavaScript byte array, and mimeType identifies its encoding, so do not assume PNG. -- mouseMove(target: string | [number, number], options?: { relative?: boolean }) -> { dispatched: boolean, foregroundChanged?: boolean | null, hoverRoute: 'accessibility' | 'system-pointer', systemPointerMoved: boolean } — on macOS, prefer acting on an available control. Default hover requires both AXShowAlternateUI and AXShowDefaultUI on the element or an enabled ancestor. Without an AX route it returns HOVER_UNAVAILABLE and sends no input. For browser content without those actions, use browser tools. Use click for a visible target or performAction for an advertised AXShowAlternateUI, AXShowMenu or AXPress. Only systemPointerHover: true permits moving the visible cursor; this requires a foreground, unobscured target. -- click(target: string | [number, number], options?: { mouseButton?: 'left' | 'right' | 'middle', clickCount?: 1 | 2, modifiers?: ('command' | 'control' | 'option' | 'shift' | 'fn')[], holdMs?: number, relative?: boolean }) -> ActionResult — on macOS, send mouse-down and mouse-up events at the center of an element in this window (id) or at the given point ([x, y]). Prefer invoke(id) for a background control’s default action. When a mouse click is needed, call app.activate() first and observe the target again: a background app can ignore a dispatched click. Set clickCount to 2 to perform a left-button double-click. holdMs controls each press duration. +- mouseMove(target: string | [number, number], options?: { relative?: boolean }) -> { dispatched: boolean, foregroundChanged?: boolean | null, hoverRoute: 'accessibility' | 'system-pointer', systemPointerMoved: boolean } — request hover at an observed element or window point. Inspect hoverRoute and systemPointerMoved to see how the provider dispatched it. Default hover requires both AXShowAlternateUI and AXShowDefaultUI on an enabled element or ancestor. Without an AX route it returns HOVER_UNAVAILABLE before sending input. Prefer click on a visible control or performAction with an advertised action. For web content without an AX hover action, use browser tools. Only systemPointerHover: true permits moving the visible cursor. If this returns FOREGROUND_REQUIRED, activate the app and check that the point is unobscured. +- click(target: string | [number, number], options?: { mouseButton?: 'left' | 'right' | 'middle', clickCount?: 1 | 2, modifiers?: ('command' | 'control' | 'option' | 'shift' | 'fn')[], holdMs?: number, relative?: boolean }) -> ActionResult — click an observed element or a window point. Set clickCount to 2 to perform a left-button double-click. holdMs controls each press duration. Both target forms send mouse-down and mouse-up events: an ID targets the element center and coordinates target the given point. Prefer invoke(id) for a background control’s default action. For a mouse click, call app.activate() first and observe the target again. A background app can ignore a dispatched click. - scroll(options: { x: number, y: number, deltaLinesX: number, deltaLinesY: number, relative?: boolean }) -> ActionResult — scroll at a point; deltas are integers from -10 to 10 with at least one nonzero, and positive values reveal content above or left. - drag(from: [number, number], to: [number, number], options?: { mouseButton?: 'left' | 'right' | 'middle', modifiers?: ('command' | 'control' | 'option' | 'shift' | 'fn')[], relative?: boolean }) -> ActionResult — drag between two points of one window. -A string target is an element id; coordinate targets are [x, y] and are pixels in the latest returned screenshot by default. With relative: true they are integer thousandths of the window width and height from 0 to 999, based on the current AX observation or screenshot. Left clicks support one or two presses; right and middle clicks support one. For hover-only content, prefer an available click or advertised accessibility action. With system-pointer hover enabled, FOREGROUND_REQUIRED means activate the app explicitly and check that the point is unobscured. Read fresh state after actions, and treat WINDOW_CHANGED, STALE_SNAPSHOT, and SCREENSHOT_REQUIRED as requiring the stated fresh observation or capture before any retry. +A string target is an element id; coordinate targets are [x, y] and are pixels in the latest returned screenshot by default. With relative: true they are integer thousandths of the window width and height from 0 to 999, based on the current window observation or screenshot. Left clicks support one or two presses; right and middle clicks support one. Read fresh state after actions, and treat WINDOW_CHANGED, STALE_SNAPSHOT, and SCREENSHOT_REQUIRED as requiring the stated fresh observation or capture before any retry. ## Simulated keyboard Available at app or window level depending on the platform. -- typeText(text: string) -> ActionResult — type literal text at the app's current focus; line endings send Return and tabs send Tab. -- paste(text: string, options?: { format?: 'text' | 'html' }) -> ActionResult — deliver text or HTML through the app's Paste command, restoring prior clipboard contents unless another writer changes them; a Paste failure is surfaced without substituting typeText, and PASTE_NOT_CONSUMED means inspect before any retry. +- typeText(text: string) -> ActionResult — type text at the current input focus. Line endings send Return and tabs send Tab. +- paste(text: string, options?: { format?: 'text' | 'html' }) -> ActionResult — paste text or HTML at the current input focus. Uses the app’s Paste command and restores the clipboard unless another writer changes it. For PASTE_NOT_CONSUMED, inspect the target before retrying. - pressKey(key: string, modifiers?: string[]) -> ActionResult — press and release one complete keystroke. For example pressKey('Tab', ['shift']) or pressKey('+'). Key and modifier names are platform-defined strings. Pass key and modifiers separately; do not pack them into a chord string. Raw tools use { method, params }: include appHandle for app calls and windowId for window calls, and use snapshotId with elementId for element routes. The emit option is facade-only: emit is local output control, so raw callers must follow the raw operation schema for admitted params. getAccessibilityTree and getWindow read through raw getState; getWindows maps to listWindows; getScreenshot maps to screenshot; cua.getState maps to listApps; every other named operation uses the same method name. diff --git a/packages/experimental/cua-macos/native/README.i18n.yaml b/packages/experimental/cua-macos/native/README.i18n.yaml index 25d5fb8952..8d43fbd6b8 100644 --- a/packages/experimental/cua-macos/native/README.i18n.yaml +++ b/packages/experimental/cua-macos/native/README.i18n.yaml @@ -21,8 +21,8 @@ en: 88a59904ca4ebedc zh: 14f2b666d7382145 /dsh-cua-native-helper/observation-and-input: - en: da6a1c45fd166818 - zh: 7f378d5409a6085d + en: 2d380e942a4057e4 + zh: fe7c9339ec63a6e5 /dsh-cua-native-helper/verification: en: 49f4cce2a4cc3114 zh: e9d65c92ef35fe24 diff --git a/packages/experimental/cua-macos/native/README.md b/packages/experimental/cua-macos/native/README.md index bdb844e806..bc80110951 100644 --- a/packages/experimental/cua-macos/native/README.md +++ b/packages/experimental/cua-macos/native/README.md @@ -121,7 +121,7 @@ Tree observation and action discovery treat a generic AX failure reading one ele `paste` publishes one plain-text or HTML clipboard format, dispatches the application's enabled standard Paste menu command, and waits for a clipboard data request. It then restores the preserved formats only while its ownership marker and change count still match. The Host serializes its pastes across helpers with a file lock. maxClipboardBytes bounds both input and preserved contents; unsupported menus and oversized data fail before dispatch. A later copy is preserved. Cancellation restores owned clipboard data when possible; forced termination cannot guarantee restoration, and a data request does not prove the target completed editing. After dispatch, `pasteReadTimeoutMs` bounds waiting for clipboard consumption. Expiry returns `PASTE_NOT_CONSUMED`, closes the promised data provider, and restores the original clipboard only while still owned. The failure does not prove the app will never process its queued Paste command; inspect the target before deciding whether to retry. -`mouseMove` reveals or restores alternate UI through an enabled element or ancestor advertising both `AXShowAlternateUI` and `AXShowDefaultUI`. Without an AX route it returns `HOVER_UNAVAILABLE` before sending input. The Host supplies the native boolean `allowSystemPointer` from its `systemPointerHover` setting. Omission means false. Only true permits posting a system pointer move after checking the foreground process and window hit. +`mouseMove` reveals or restores alternate UI through an enabled element or ancestor advertising both `AXShowAlternateUI` and `AXShowDefaultUI`. Without an AX route it returns `HOVER_UNAVAILABLE` before sending input. The macOS provider sets `systemPointerHover` in the connection handshake. Diagnostic clients that omit this setting may opt in per request with `allowSystemPointer: true`. The default is false. Only true permits posting a system pointer move after checking the foreground process and window hit. The opt-in system pointer route returns `FOREGROUND_REQUIRED` if the target is obscured or belongs to a background app. The result reports the route and whether the visible cursor moved in `hoverRoute` and `systemPointerMoved`. It never presses a button or restores the cursor after a read. Later input may interrupt the hover. Clicks send their button events at the target without an off-window priming click. diff --git a/packages/experimental/cua-macos/native/README.zh.md b/packages/experimental/cua-macos/native/README.zh.md index 51d6c13c29..0927a82bce 100644 --- a/packages/experimental/cua-macos/native/README.zh.md +++ b/packages/experimental/cua-macos/native/README.zh.md @@ -121,7 +121,7 @@ AppKit 在主事件循环上运行。阻塞式 socket 接收与握手使用后 `paste` 提供一种纯文本或 HTML 剪贴板格式,调用应用已启用的标准 Paste 菜单命令,并等待剪贴板数据请求。随后仅在所有权标记和修改计数仍匹配时恢复原有格式。Host 通过文件锁串行化不同 helper 的粘贴。maxClipboardBytes 同时限制输入和保留内容,不支持的菜单和过大的数据在分派前失败。后续复制会被保留。取消会尽可能恢复仍持有的剪贴板数据,强制结束不保证恢复,数据请求也不证明目标已完成编辑。 分派完成后,`pasteReadTimeoutMs` 限制等待剪贴板读取的时间。到期返回 `PASTE_NOT_CONSUMED`,关闭延迟数据提供方,并仅在仍持有剪贴板时恢复原内容。该失败不能证明应用以后不会处理已排队的 Paste 命令;决定是否重试前先检查目标内容。 -默认情况下,`mouseMove` 通过同一窗口内启用的控件或祖先声明的 `AXShowAlternateUI` 和 `AXShowDefaultUI` 展开或恢复界面。缺少这条路径时返回 `HOVER_UNAVAILABLE`,不发送输入。Host 根据部署配置 `systemPointerHover` 传入原生布尔参数 `allowSystemPointer`,省略时为 false。只有值为 true 才允许移动全局光标,且须检查前台进程和鼠标命中的窗口。 +默认情况下,`mouseMove` 通过同一窗口内启用的控件或祖先声明的 `AXShowAlternateUI` 和 `AXShowDefaultUI` 展开或恢复界面。缺少这条路径时返回 `HOVER_UNAVAILABLE`,不发送输入。macOS 提供方在连接握手中设置 `systemPointerHover`。未在握手中指定该配置的诊断客户端,可以在单次请求中传入 `allowSystemPointer: true`。默认值为 false。只有值为 true 才允许移动全局光标,且须检查前台进程和鼠标命中的窗口。 开启后,系统指针路线会检查前台应用和坐标命中的窗口,不满足条件时返回 `FOREGROUND_REQUIRED`。返回值 `hoverRoute` 和 `systemPointerMoved` 说明走了哪条路线以及是否移动了可见光标。悬停不会按下鼠标,也不会在读取后还原光标。后续输入可能中断悬停。点击只在目标点发送按键事件,不先在窗口外点击。 diff --git a/packages/experimental/cua-macos/native/Sources/CuaNative/AXWorker.swift b/packages/experimental/cua-macos/native/Sources/CuaNative/AXWorker.swift index 73e8e78fd9..96a6054525 100644 --- a/packages/experimental/cua-macos/native/Sources/CuaNative/AXWorker.swift +++ b/packages/experimental/cua-macos/native/Sources/CuaNative/AXWorker.swift @@ -78,6 +78,7 @@ actor AXWorker { private let axTimeout: Float private let textEventInterval: Double private let maxClickHoldMs: Int + private let systemPointerHover: Bool? private let sessionState: @Sendable () -> Bool? private var access: AXAccess? private var keyboard: KeyboardInput? @@ -85,12 +86,13 @@ actor AXWorker { private var hovers: [String: AXHover] = [:] private var closed = false - init(cancellation: CancellationLatch, axTimeout: Float, textEventInterval: Double, maxClickHoldMs: Int, + init(cancellation: CancellationLatch, axTimeout: Float, textEventInterval: Double, maxClickHoldMs: Int, systemPointerHover: Bool? = nil, sessionState: @escaping @Sendable () -> Bool? = NativePlatform.sessionLocked) { self.cancellation = cancellation self.axTimeout = axTimeout self.textEventInterval = textEventInterval self.maxClickHoldMs = maxClickHoldMs + self.systemPointerHover = systemPointerHover self.sessionState = sessionState } private func identifier(_ kind: String) throws -> String { @@ -105,7 +107,9 @@ actor AXWorker { return created } private func moveUsingAX(_ element: AXUIElement?, windowID: UInt32, handle: String, params: RequestParameters) throws -> Bool { - let allowSystemPointer = try params.optionalBoolean("allowSystemPointer") ?? false + let requestSetting = try params.optionalBoolean("allowSystemPointer") + // Diagnostic JSONL clients without a deployment setting may opt in per request. + let allowSystemPointer = systemPointerHover ?? requestSetting ?? false let hover = hovers[handle] ?? AXHover() hovers[handle] = hover if try hover.move(to: element, in: windowID, access: ax()) { return true } diff --git a/packages/experimental/cua-macos/native/Sources/CuaNative/RequestCoordinator.swift b/packages/experimental/cua-macos/native/Sources/CuaNative/RequestCoordinator.swift index 3a778cb8ae..000a69e848 100644 --- a/packages/experimental/cua-macos/native/Sources/CuaNative/RequestCoordinator.swift +++ b/packages/experimental/cua-macos/native/Sources/CuaNative/RequestCoordinator.swift @@ -42,7 +42,8 @@ import Darwin self.onClose = onClose self.onFatal = onFatal worker = AXWorker(cancellation: cancellation, axTimeout: configuration.axTimeoutSeconds, - textEventInterval: configuration.textEventIntervalSeconds, maxClickHoldMs: configuration.maxClickHoldMs) + textEventInterval: configuration.textEventIntervalSeconds, maxClickHoldMs: configuration.maxClickHoldMs, + systemPointerHover: configuration.systemPointerHover) input = JSONLInput(limit: configuration.maxRequestBytes, cancellation: cancellation, channel: channel) self.writer = writer } diff --git a/packages/experimental/cua-macos/native/Sources/CuaNative/Runtime.swift b/packages/experimental/cua-macos/native/Sources/CuaNative/Runtime.swift index d61f0803ab..c31a07ebef 100644 --- a/packages/experimental/cua-macos/native/Sources/CuaNative/Runtime.swift +++ b/packages/experimental/cua-macos/native/Sources/CuaNative/Runtime.swift @@ -21,6 +21,7 @@ struct HelperConfiguration: Sendable { var shutdownGraceSeconds: Double = 3 var textEventIntervalSeconds: Double = 0.00025 var maxClickHoldMs = 5000 + var systemPointerHover: Bool? var timeoutSeconds: Double = 30 var screenshotTimeoutSeconds: Double = 5 init(arguments: [String]) throws { @@ -64,7 +65,7 @@ struct HelperConfiguration: Sendable { guard let fields = decoded.object, Set(fields.keys) == ["type", "protocol", "runtimeKey", "config"], fields["type"] == .string("hello"), fields["protocol"] == .number(1), fields["runtimeKey"] == .string(runtimeKey), let config = fields["config"]?.object, - Set(config.keys) == ["maxRequestBytes", "maxResponseBytes", "axTimeoutMs", "textEventIntervalMs", "maxClickHoldMs", "screenshotTimeoutMs", "timeoutMs", "graceMs"] else { + Set(config.keys).subtracting(["systemPointerHover"]) == ["maxRequestBytes", "maxResponseBytes", "axTimeoutMs", "textEventIntervalMs", "maxClickHoldMs", "screenshotTimeoutMs", "timeoutMs", "graceMs"] else { throw NativeFailure(code: "NATIVE_PROTOCOL", message: "Expected shared helper protocol 1 hello for this runtime") } func number(_ name: String) throws -> String { @@ -82,6 +83,10 @@ struct HelperConfiguration: Sendable { "--screenshot-timeout-ms", number("screenshotTimeoutMs"), "--timeout-ms", number("timeoutMs"), "--shutdown-grace-ms", number("graceMs"), ]) + if let setting = config["systemPointerHover"] { + guard case .bool(let allowed) = setting else { throw RequestParameters.invalid("systemPointerHover must be a boolean") } + systemPointerHover = allowed + } } } diff --git a/packages/experimental/cua-macos/native/tests/swift/SharedHelperTests.swift b/packages/experimental/cua-macos/native/tests/swift/SharedHelperTests.swift index 9b43bc330e..52f7f0f224 100644 --- a/packages/experimental/cua-macos/native/tests/swift/SharedHelperTests.swift +++ b/packages/experimental/cua-macos/native/tests/swift/SharedHelperTests.swift @@ -33,6 +33,14 @@ private func hello(overrides: [String: JSONValue] = [:]) throws -> Data { } } +@Test func readsSystemPointerHoverFromEachConnection() throws { + #expect(try HelperConfiguration(hello: hello(), runtimeKey: runtimeKey).systemPointerHover == nil) + for allowed in [true, false] { + let configuration = try HelperConfiguration(hello: hello(overrides: ["systemPointerHover": .bool(allowed)]), runtimeKey: runtimeKey) + #expect(configuration.systemPointerHover == allowed) + } +} + @Test func resolvesClickHoldLimitFromHelperArguments() throws { #expect(try HelperConfiguration(arguments: []).maxClickHoldMs == 5000) for limit in [0, 7000] { @@ -48,7 +56,7 @@ private func hello(overrides: [String: JSONValue] = [:]) throws -> Data { ("axTimeoutMs", .number(0)), ("textEventIntervalMs", .number(-1)), ("screenshotTimeoutMs", .number(0)), ("timeoutMs", .string("8000")), ("graceMs", .number(-1)), ("maxClickHoldMs", .number(-1)), ("maxClickHoldMs", .number(0.5)), - ("maxClickHoldMs", .number(2147483648)), ("unexpected", .bool(true)), + ("maxClickHoldMs", .number(2147483648)), ("unexpected", .bool(true)), ("systemPointerHover", .string("true")), ]) func refusesInvalidSharedConfiguration(_ name: String, _ value: JSONValue) throws { #expect(throws: NativeFailure.self) { try HelperConfiguration(hello: hello(overrides: [name: value]), runtimeKey: runtimeKey) } diff --git a/packages/experimental/cua-macos/src/guidance.ts b/packages/experimental/cua-macos/src/guidance.ts index 9da4e5f85a..0379d91120 100644 --- a/packages/experimental/cua-macos/src/guidance.ts +++ b/packages/experimental/cua-macos/src/guidance.ts @@ -2,9 +2,23 @@ import { COMMON_GUIDANCE } from '@deepseek-ai/dsh-cua/guidance' /** macOS operation levels, permissions, selectors, and native measurement units. */ -export const PLATFORM_GUIDANCE = 'macOS: accessibility tree operations are available at app or window level. Simulated keyboard input is available at app level. Simulated mouse input is available at window level. Default hover uses AX alternate UI and keeps the system cursor in place. For pointer input on a sheet, select its window ID with app.getWindow(id). For menu items outside a window, use app.invoke(id). getApp also accepts bundle IDs and .app paths, and findApps searches /Applications, /System/Applications, and ~/Applications. Use the Computer Use permission page to grant the helper application Accessibility and Screen Recording permissions in System Settings. Native window and AX bounds use macOS points, while delivered screenshots carry their own pixel dimensions. show does not switch Spaces, and Stage Manager can present reduced window geometry; explain any required foreground step before calling app.activate() or performing an advertised AXRaise action.' +export const PLATFORM_GUIDANCE = 'macOS: accessibility tree operations are available at app or window level. Simulated keyboard input is available at app level. Simulated mouse input is available at window level. For pointer input on a sheet, select its window ID with app.getWindow(id). For menu items outside a window, use app.invoke(id). Use the Computer Use permission page to grant the helper application Accessibility and Screen Recording permissions in System Settings. Native window and AX bounds use macOS points, while delivered screenshots carry their own pixel dimensions. Stage Manager can present reduced window geometry; explain any required foreground step before calling app.activate() or performing an advertised AXRaise action.' + +const methodDetails = new Map([ + ['cua.getApp', 'Accepts a name, bundle ID, .app path, or running PID string. A stopped app launches in the background.'], + ['cua.findApps', 'Searches /Applications, /System/Applications, and ~/Applications.'], + ['app.show', 'Unhides the app and restores its focused, main, or sole window without activating it.'], + ['window.show', 'Unhides the app and restores this window without activating it.'], + ['mouseMove', 'Default hover requires both AXShowAlternateUI and AXShowDefaultUI on an enabled element or ancestor. Without an AX route it returns HOVER_UNAVAILABLE before sending input. Prefer click on a visible control or performAction with an advertised action. For web content without an AX hover action, use browser tools. Only systemPointerHover: true permits moving the visible cursor. If this returns FOREGROUND_REQUIRED, activate the app and check that the point is unobscured.'], + ['click', 'Both target forms send mouse-down and mouse-up events: an ID targets the element center and coordinates target the given point. Prefer invoke(id) for a background control’s default action. For a mouse click, call app.activate() first and observe the target again. A background app can ignore a dispatched click.'], + ['typeText', 'Line endings send Return and tabs send Tab.'], + ['paste', 'Uses the app’s Paste command and restores the clipboard unless another writer changes it. For PASTE_NOT_CONSUMED, inspect the target before retrying.'], +]) /** Complete model instructions for this provider. */ export const GUIDANCE = `${PLATFORM_GUIDANCE} -${COMMON_GUIDANCE}` +${COMMON_GUIDANCE.replace(/^- ([\w.]+)\(.*$/gm, (entry, method: string) => { + const detail = methodDetails.get(method) + return detail ? `${entry} ${detail}` : entry +})}` diff --git a/packages/experimental/cua-macos/src/index.ts b/packages/experimental/cua-macos/src/index.ts index 52788c0752..4b07e9c8b7 100644 --- a/packages/experimental/cua-macos/src/index.ts +++ b/packages/experimental/cua-macos/src/index.ts @@ -4,6 +4,7 @@ import { fileURLToPath } from 'node:url' import type { Context } from '@deepseek-ai/cordis' import Schema from '@deepseek-ai/schemastery' import { installCua, type CuaConfig, type CuaStartupCapabilities } from '@deepseek-ai/dsh-cua' +import type { NativeChannelFactory } from '@deepseek-ai/dsh-cua/native-client' import { CuaConfigFields } from '@deepseek-ai/dsh-cua/config' import { createPermissionParameters } from './permission-guide.ts' import { nativeHostArchitecture, prepareNativeExecutable, type SourceSigning } from './native-build.ts' @@ -63,8 +64,8 @@ export const Config = Schema.object({ * @returns Completion after native preparation and tool registration. */ export async function apply(ctx: Context, config: Config): Promise { - const backend = { createChannel: createSharedNativeChannel, guidance: GUIDANCE, capabilities: CAPABILITIES, - systemPointerHover: config.systemPointerHover, + const createChannel: NativeChannelFactory = (owner, native) => createSharedNativeChannel(owner, native, config.systemPointerHover) + const backend = { createChannel, guidance: GUIDANCE, capabilities: CAPABILITIES, permissionParameters: createPermissionParameters(config.permissionPollIntervalMs) } if (config.helperPath !== undefined) { if (!isAbsolute(config.helperPath)) throw new Error('dsh-cua helperPath must be absolute') diff --git a/packages/experimental/cua-macos/src/shared-native.ts b/packages/experimental/cua-macos/src/shared-native.ts index 0fe2cf6436..7be638a96b 100644 --- a/packages/experimental/cua-macos/src/shared-native.ts +++ b/packages/experimental/cua-macos/src/shared-native.ts @@ -56,7 +56,8 @@ class SharedNativeChannel implements NativeChannel { private cleanupTimer: ReturnType | undefined private cleanupFailure: Error | undefined - constructor(private readonly owner: { subprocess: Pick }, private readonly config: NativeClientConfig) { + constructor(private readonly owner: { subprocess: Pick }, private readonly config: NativeClientConfig, + private readonly systemPointerHover: boolean) { this.stdin = new Writable({ write: (chunk: Buffer, encoding, callback) => { void this.ready.promise.then((socket) => { @@ -214,7 +215,8 @@ class SharedNativeChannel implements NativeChannel { const { maxRequestBytes, maxResponseBytes, axTimeoutMs, textEventIntervalMs, maxClickHoldMs, screenshotTimeoutMs, timeoutMs, graceMs } = this.config socket.write(JSON.stringify({ type: 'hello', protocol: 1, runtimeKey, - config: { maxRequestBytes, maxResponseBytes, axTimeoutMs, textEventIntervalMs, maxClickHoldMs, screenshotTimeoutMs, timeoutMs, graceMs } }) + '\n') + config: { maxRequestBytes, maxResponseBytes, axTimeoutMs, textEventIntervalMs, maxClickHoldMs, + screenshotTimeoutMs, timeoutMs, graceMs, systemPointerHover: this.systemPointerHover } }) + '\n') }) const receive = (chunk: Buffer): void => { const newline = chunk.indexOf(10) @@ -286,8 +288,9 @@ class SharedNativeChannel implements NativeChannel { * Prepare or join the native process; writes wait for identity-checked readiness. * @param owner - Provider owning the short-lived native startup command. * @param config - Native runtime identity, operation limits and connection deadlines. + * @param systemPointerHover - Whether this connection may move the visible pointer when AX hover is unavailable. * @returns One independently disposable Agent connection, without operation replay. */ -export function createSharedNativeChannel(owner: { subprocess: Pick }, config: NativeClientConfig): NativeChannel { - return new SharedNativeChannel(owner, config) +export function createSharedNativeChannel(owner: { subprocess: Pick }, config: NativeClientConfig, systemPointerHover: boolean): NativeChannel { + return new SharedNativeChannel(owner, config, systemPointerHover) } diff --git a/packages/experimental/cua-macos/tests/guidance.spec.ts b/packages/experimental/cua-macos/tests/guidance.spec.ts index 7afb53b84c..c00f600a79 100644 --- a/packages/experimental/cua-macos/tests/guidance.spec.ts +++ b/packages/experimental/cua-macos/tests/guidance.spec.ts @@ -19,7 +19,7 @@ it('keeps exactly three ordered operation headings', () => { }) it('prefixes concise platform guidance before the shared catalog', () => { - expect(GUIDANCE).toBe(`${PLATFORM_GUIDANCE}\n\n${COMMON_GUIDANCE}`) + expect(GUIDANCE.startsWith(`${PLATFORM_GUIDANCE}\n\nUse cua`)).toBe(true) expect(PLATFORM_GUIDANCE).not.toContain('##') }) @@ -27,9 +27,12 @@ it('states the macOS accessibility, keyboard, and mouse levels', () => { expect(PLATFORM_GUIDANCE).toContain('accessibility tree operations are available at app or window level') expect(PLATFORM_GUIDANCE).toContain('Simulated keyboard input is available at app level') expect(PLATFORM_GUIDANCE).toContain('Simulated mouse input is available at window level') - const hover = COMMON_GUIDANCE.split('\n').find(line => line.startsWith('- mouseMove(')) + const hover = GUIDANCE.split('\n').find(line => line.startsWith('- mouseMove(')) for (const term of ['AXShowAlternateUI', 'AXShowDefaultUI', 'HOVER_UNAVAILABLE', 'systemPointerHover']) expect(hover).toContain(term) - expect(COMMON_GUIDANCE.split('\n').find(line => line.startsWith('- click('))).toContain('call app.activate() first') + expect(GUIDANCE.split('\n').find(line => line.startsWith('- click('))).toContain('call app.activate() first') + for (const platformDetail of ['macOS', 'AXShowAlternateUI', 'AXRaise', 'HOVER_UNAVAILABLE', 'systemPointerHover', 'mouse-down', 'bundle ID', 'Paste command']) { + expect(COMMON_GUIDANCE).not.toContain(platformDetail) + } }) it('enumerates every available method once in its catalog entry', () => { @@ -41,7 +44,9 @@ it('enumerates every available method once in its catalog entry', () => { ...windowNames.filter(name => name !== 'windowId').map(name => `${name}(`), ...['typeText', 'paste', 'pressKey'].map(name => `${name}(`), ] - for (const signature of signatures) expect(COMMON_GUIDANCE.split('\n').filter(line => line.startsWith(`- ${signature}`))).toHaveLength(1) + for (const catalog of [COMMON_GUIDANCE, GUIDANCE]) { + for (const signature of signatures) expect(catalog.split('\n').filter(line => line.startsWith(`- ${signature}`))).toHaveLength(1) + } expect(signatures).toHaveLength(24) expect(new Set(signatures.map(signature => signature.slice(signature.indexOf('.') + 1, -1))).size).toBe(23) expect(COMMON_GUIDANCE).toContain('app.appHandle') diff --git a/packages/experimental/cua-macos/tests/provider-lifecycle.spec.ts b/packages/experimental/cua-macos/tests/provider-lifecycle.spec.ts index 38e632e6ae..017a605ce1 100644 --- a/packages/experimental/cua-macos/tests/provider-lifecycle.spec.ts +++ b/packages/experimental/cua-macos/tests/provider-lifecycle.spec.ts @@ -71,6 +71,7 @@ const { helperPath: _helperPath, ...automaticConfig } = config it('defaults to ad-hoc signing and requires a keychain path for certificate signing', () => { expect(NativeProvider.Config({}).sourceSigning).toBe('ad-hoc') + expect(NativeProvider.Config({}).systemPointerHover).toBe(false) expect(NativeProvider.Config({ sourceSigning: 'ad-hoc' }).sourceSigning).toBe('ad-hoc') for (const sourceSigning of [{ identity: 'Apple Development: Example', keychain: '/keychains/custom.keychain-db' }, { identity: 'ABC123', keychain: '/keychains/custom.keychain-db' }]) { expect(NativeProvider.Config({ sourceSigning }).sourceSigning).toEqual(sourceSigning) @@ -109,7 +110,7 @@ class ImageAdapter extends LlmAdapter { const imageData = 'iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAIAAACQd1PeAAAACXBIWXMAAAPoAAAD6AG1e1JrAAAADElEQVQImWNgZGIGAAAOAAeCcsnOAAAAAElFTkSuQmCC' const imageMetadata = { pixelWidth: 1, pixelHeight: 1, bounds: { x: 2, y: 3, width: 1, height: 1 }, coordinateSpace: 'window-points', windowId: 7 } -async function setup(model?: { provider: string; model: string }, systemPointerHover = false) { +async function setup(model?: { provider: string; model: string }) { native.request.mockReset().mockResolvedValue([]) native.dispose.mockReset() native.instances.length = 0 @@ -119,7 +120,7 @@ async function setup(model?: { provider: string; model: string }, systemPointerH await mountAgentLoopTestDependencies(ctx) await ctx.plugin(ComputerUseRegistry) await ctx.plugin(Subprocess, []) - const provider = await ctx.plugin(NativeProvider, { ...config, systemPointerHover }) + const provider = await ctx.plugin(NativeProvider, config) await mountAgentLoopTestHarness(ctx) const handle = await ctx.agents.create({ sessionId: SessionId('native-lifecycle'), ...model ? { agentOptions: model } : {} }) let index = 0 @@ -386,26 +387,6 @@ it('rejects malformed and unowned operations before acquiring a helper', async ( expect(native.request).not.toHaveBeenCalled() }) -it.each([false, true])('rejects an ended Agent with prior helper acquisition %s', async (acquired) => { - const f = await setup() - if (acquired) await f.execute() - await f.handle.dispose() - const requests = native.request.mock.calls.length - const result = await f.execute() - expect(result.isError).toBe(true) - expect(JSON.stringify(result.content)).toContain('agent has been disposed') - expect(native.instances).toHaveLength(acquired ? 1 : 0) - expect(native.request).toHaveBeenCalledTimes(requests) -}) - -it.each([false, true])('passes the deployment hover policy (%s) to native input', async (allowed) => { - const f = await setup(undefined, allowed) - await selectApplication(f) - const params = { appHandle: 'app-one', windowId: 7, snapshotId: 's', elementId: '0' } - await f.execute('mouseMove', params) - expect(native.request).toHaveBeenLastCalledWith('mouseMove', { ...params, allowSystemPointer: allowed }, expect.any(AbortSignal), expect.any(String)) -}) - it('supplies configured resource bounds and clears registrations on unload', async () => { const f = await setup() await selectApplication(f) diff --git a/packages/experimental/cua-macos/tests/shared-native.spec.ts b/packages/experimental/cua-macos/tests/shared-native.spec.ts index cef346ab9b..871475fb53 100644 --- a/packages/experimental/cua-macos/tests/shared-native.spec.ts +++ b/packages/experimental/cua-macos/tests/shared-native.spec.ts @@ -133,8 +133,8 @@ async function fixture(options: { timeoutMs: 30_000, graceMs: options.graceMs ?? 1000, axTimeoutMs: 3000, maxRequestBytes: 65536, maxResponseBytes: 1048576, textEventIntervalMs: 0.5, maxClickHoldMs: 7000, } - function open(): NativeChannel { - const channel = createSharedNativeChannel({ subprocess: { spawn } }, config) + function open(systemPointerHover = false): NativeChannel { + const channel = createSharedNativeChannel({ subprocess: { spawn } }, config, systemPointerHover) channels.push(channel) return channel } @@ -172,7 +172,7 @@ describe.skipIf(process.platform === 'win32')('shared native Unix connection', ( it('keeps concurrent Agent scopes independent and leaves other sockets alive when one closes', async () => { const f = await fixture() const first = f.open() - const second = f.open() + const second = f.open(true) const firstReply = nextBytes(first) const secondReply = nextBytes(second) await Promise.all([write(first, 'one\n'), write(second, 'two\n')]) @@ -186,10 +186,14 @@ describe.skipIf(process.platform === 'win32')('shared native Unix connection', ( expect(f.spawn).toHaveBeenCalledTimes(2) // oxlint-disable-next-line typescript/unbound-method -- Fixture termination methods are spies inspected without invocation. for (const handle of f.handles) expect(handle.terminate).not.toHaveBeenCalled() - expect(f.hellos).toEqual([expect.objectContaining({ type: 'hello', protocol: 1, config: { + const limits = { axTimeoutMs: 3000, timeoutMs: 30000, screenshotTimeoutMs: 5000, graceMs: 1000, maxRequestBytes: 65536, maxResponseBytes: 1048576, textEventIntervalMs: 0.5, maxClickHoldMs: 7000, - } }), expect.anything()]) + } + expect(f.hellos).toHaveLength(2) + for (const systemPointerHover of [true, false]) { + expect(f.hellos).toContainEqual(expect.objectContaining({ type: 'hello', protocol: 1, config: { ...limits, systemPointerHover } })) + } }) it('waits for both launcher outcome and managed range release before sending any bytes', async () => { diff --git a/packages/experimental/cua/README.i18n.yaml b/packages/experimental/cua/README.i18n.yaml index dd3c6d217a..f4abb13c43 100644 --- a/packages/experimental/cua/README.i18n.yaml +++ b/packages/experimental/cua/README.i18n.yaml @@ -15,8 +15,8 @@ en: 5db4d4fea25493b9 zh: 65ab64fc24eb0d09 /deepseek-ai-dsh-cua/use-this-package: - en: d0088d2ec1ab8cf2 - zh: d927289a63d7a564 + en: ba1f8e6ae51f69d7 + zh: 8f4fde9857bae319 /deepseek-ai-dsh-cua/use-this-package/permission-controls: en: d92d5948a7941e8d zh: d43491ae6521b87e diff --git a/packages/experimental/cua/README.md b/packages/experimental/cua/README.md index 365c549650..c95e4ae8ec 100644 --- a/packages/experimental/cua/README.md +++ b/packages/experimental/cua/README.md @@ -43,7 +43,7 @@ Application consent precedes launch, observation, capture, and input. Session ac Actions return a `dispatched` boolean; `mouseMove` also reports `hoverRoute` and `systemPointerMoved`, and `show()` adds `windowId` and `foregroundChanged`. Read fresh state to verify the result. Screenshots and pointer actions are available only on explicit window objects. Coordinate actions default to pixels in the latest returned screenshot; `relative: true` accepts integers from 0 to 999 after a successful window observation. Values are divided by 1000 and scaled to the window dimensions, then clamped inside the window. The Host resolves attachment normalization and the selected model route’s image projection before returning the screenshot, then converts input to the provider’s native window coordinates. Resizing invalidates the screenshot mapping; `SCREENSHOT_REQUIRED` requires a new screenshot for absolute coordinates. Resizing rejects old coordinates with `WINDOW_CHANGED`; movement alone preserves window-local coordinates. -`click(target, { holdMs: 800 })` holds the mouse button for 800 milliseconds before release. `holdMs` applies only to `click`; it accepts nonnegative integers and applies to each press of a double-click; zero releases without an additional hold. The provider config `maxClickHoldMs` limits explicit holds and defaults to 5000 ms. On macOS, both ordinary and held clicks send mouse events at the element center or the supplied point. Use `invoke(id)` for the default accessibility action. The operation deadline includes the hold, and cancellation or timeout releases the button. Received input cannot be undone; forced helper termination cannot guarantee release. +`click(target, { holdMs: 800 })` holds the mouse button for 800 milliseconds before release. `holdMs` applies only to `click`; it accepts nonnegative integers and applies to each press of a double-click; zero releases without an additional hold. The provider config `maxClickHoldMs` limits explicit holds and defaults to 5000 ms. Explicit holds use pointer input; omission preserves ordinary click behavior. The operation deadline includes the hold, and cancellation or timeout releases the button. Received input cannot be undone; forced helper termination cannot guarantee release. ### Permission controls diff --git a/packages/experimental/cua/README.zh.md b/packages/experimental/cua/README.zh.md index b64c445c44..c1c5ed863c 100644 --- a/packages/experimental/cua/README.zh.md +++ b/packages/experimental/cua/README.zh.md @@ -43,7 +43,7 @@ kind: "package-reference" 动作返回 `dispatched` 布尔值;`mouseMove` 还会返回 `hoverRoute` 和 `systemPointerMoved`,`show()` 还会返回 `windowId` 和 `foregroundChanged`。请读取新状态以验证结果。截图和指针操作只能通过显式窗口对象使用。坐标操作默认使用最近返回截图中的像素;成功观察窗口后,传入 `relative: true` 可使用 0~999 的整数坐标。数值除以 1000 后按窗口尺寸缩放,并限制在窗口内。Host 在返回截图前解析附件归一化和当前模型的图片缩放规则,再将输入换算为提供方的原生窗口坐标。窗口尺寸变化会使截图映射失效,绝对坐标遇到 `SCREENSHOT_REQUIRED` 时需要重新截图。窗口缩放后以 `WINDOW_CHANGED` 拒绝旧坐标;单纯移动窗口不改变窗口内坐标。 -`click(target, { holdMs: 800 })` 按住鼠标按钮 800 毫秒后松开。`holdMs` 仅适用于 `click`;它接受非负整数,双击时应用于每次按下;0 表示按下后立即松开,不额外等待。提供方配置 `maxClickHoldMs` 限制显式按住时长,默认 5000 毫秒。在 macOS 上,无论是否指定按住时长,点击都会向元素中心或给定坐标发送鼠标事件。要执行控件的默认辅助功能动作,请用 `invoke(id)`。按住时间计入操作超时,取消或超时会松开按钮。已接收的输入无法撤销;辅助进程被强制终止时无法保证松开。 +`click(target, { holdMs: 800 })` 按住鼠标按钮 800 毫秒后松开。`holdMs` 仅适用于 `click`;它接受非负整数,双击时应用于每次按下;0 表示按下后立即松开,不额外等待。提供方配置 `maxClickHoldMs` 限制显式按住时长,默认 5000 毫秒。指定时长后使用鼠标事件;省略时保留普通点击行为。按住时间计入操作超时,取消或超时会松开按钮。已接收的输入无法撤销;辅助进程被强制终止时无法保证松开。 ### 权限控件 diff --git a/packages/experimental/cua/src/guidance.ts b/packages/experimental/cua/src/guidance.ts index 1d6d0266d1..27be40b4c7 100644 --- a/packages/experimental/cua/src/guidance.ts +++ b/packages/experimental/cua/src/guidance.ts @@ -1,5 +1,4 @@ /** Shared model instructions for native application and window operations. */ -/** Async method catalog with consent and observation rules, grouped by observation, mouse, and keyboard. */ export const COMMON_GUIDANCE = `Use cua to observe and operate native applications. Always use cua.getCapabilities() to confirm whether a specific API is available on the current platform; its reported capabilities take precedence over availability statements in these instructions. All methods are async; observation options default to { emit: true }, and emit: false suppresses sink output only, leaving returned values unchanged. Actions report dispatch evidence, not verified completion. If sessionLocked is true or an operation reports SESSION_LOCKED, stop UI actions and ask the user to unlock the desktop session; null or unknown means verify a usable unlocked desktop session first. cua.getApp requests user consent; if access is denied or revoked, stop using that application until the user explicitly permits another request, and never bypass a refusal or switch tools to avoid it. Never replay uncertain input without a fresh observation. In persistent PTC, use the host-provided bridge initialization once and retain the returned objects; cua comes from that bridge and is not a preinstalled global. Use void await for observations that emit, and never stringify screenshot image bytes. @@ -9,20 +8,20 @@ Signatures show resolved return values after await. ActionResult = { dispatched: Setup and discovery: - cua.getCapabilities(options?: { emit?: boolean }) -> { platform: string, permissions: { accessibility: boolean, screenRecording: boolean }, appLevelAccessibility: boolean, keyboard: ('app' | 'window')[], sessionLocked: boolean | null, methods: string[], coordinateSpace: 'screenshot-pixels', browser: false } — inspect reported permissions, sessionLocked, methods, and keyboard scopes without requesting permissions. - cua.listApps(options?: { emit?: boolean }) -> { id: string, displayName: string, isRunning: true, pid: number }[] — enumerate running applications. -- cua.findApps(query: string, options?: { emit?: boolean }) -> ({ id: string, displayName: string, path: string } & ({ isRunning: true, pid: number } | { isRunning: false }))[] — search installed applications, including stopped matches; only running matches carry pid. +- cua.findApps(query: string, options?: { emit?: boolean }) -> ({ id: string, displayName: string, path: string } & ({ isRunning: true, pid: number } | { isRunning: false }))[] — search for applications that match the query. - cua.getState(options?: { emit?: boolean }) -> { apps: { id: string, displayName: string, isRunning: true, pid: number }[], browsers: [] } — summarize running applications; this is not an accessibility observation. -- cua.getApp(selector: string) -> App — select an application by name, bundle ID, absolute path, or running PID string, obtain consent, launch in the background if needed, and read initial AX state silently; it does not activate the app. -- app.getWindows(options?: { emit?: boolean }) -> { windowId: number, appName: string, title: string, bounds: { x: number, y: number, width: number, height: number }, isOnScreen: boolean, onCurrentSpace: boolean | null }[] — enumerate windows with global native bounds; null onCurrentSpace means unknown Space membership. -- app.getWindow(windowId: number) -> Window — select one window and read its initial AX state silently; do this before any window screenshot or coordinate input. +- cua.getApp(selector: string) -> App — select an application using a provider-supported selector and obtain consent. When app-level accessibility is available, selection reads the initial tree silently. +- app.getWindows(options?: { emit?: boolean }) -> { windowId: number, appName: string, title: string, bounds: { x: number, y: number, width: number, height: number }, isOnScreen: boolean, onCurrentSpace: boolean | null }[] — enumerate windows with global native bounds. A null onCurrentSpace means desktop membership is unknown. +- app.getWindow(windowId: number) -> Window — select one window and silently read its initial accessibility tree. Select a window before screenshots or pointer input. - app.activate() -> ActionResult — request explicit foreground activation; use only when foreground access is required. -- app.show() -> { dispatched: boolean, windowId: number | null, foregroundChanged: boolean | null } — unhide the app and restore its selected, focused, main, or sole window without activation. -- window.show() -> { dispatched: boolean, windowId: number | null, foregroundChanged: boolean | null } — unhide and restore that explicit window without activation. +- app.show() -> { dispatched: boolean, windowId: number | null, foregroundChanged: boolean | null } — request that the application be shown. +- window.show() -> { dispatched: boolean, windowId: number | null, foregroundChanged: boolean | null } — request that the selected window be shown. ## Accessibility tree Available at app or window level depending on the platform. -Element ids are opaque strings from the latest successful tree; the object facade keeps one current snapshot per app across its repeated app and window objects, a new read replaces it, and a failed read invalidates the previous one. Repeated strings can appear for different elements, so do not reuse an old id blindly, and element operations do not fall back to pointer input or typing. -- getAccessibilityTree(options?: { emit?: boolean, includeMenus?: boolean, details?: boolean }) -> string — read the full tree text; includeMenus expands menu contents and details adds identifiers, native bounds, and complete action names. +Element ids are opaque strings from the latest successful tree; the object facade keeps one current snapshot per app across its repeated app and window objects, a new read replaces it, and a failed read invalidates the previous one. Repeated strings can appear for different elements, so use IDs from the most recent observation. +- getAccessibilityTree(options?: { emit?: boolean, includeMenus?: boolean, details?: boolean }) -> string — read the full tree text. includeMenus requests menu contents and details requests additional element information. - listActions(elementId: string) -> string[] — list the actions an element advertises. - invoke(elementId: string) -> ActionResult — invoke the element's default action; dispatched is false when it has none. - performAction(elementId: string, action: string, argument?: JsonValue) -> ActionResult — perform an observed action with an optional direct JSON argument. @@ -32,17 +31,17 @@ Element ids are opaque strings from the latest successful tree; the object facad ## Simulated mouse Use an explicit window for screenshots and pointer input. - getScreenshot(options?: { emit?: boolean }) -> { bytes: Uint8Array, mimeType: 'image/png' | 'image/jpeg' | 'image/webp' | 'image/gif', pixelWidth: number, pixelHeight: number, windowId: number } — capture the selected window as one encoded image; bytes is a JavaScript byte array, and mimeType identifies its encoding, so do not assume PNG. -- mouseMove(target: string | [number, number], options?: { relative?: boolean }) -> { dispatched: boolean, foregroundChanged?: boolean | null, hoverRoute: 'accessibility' | 'system-pointer', systemPointerMoved: boolean } — on macOS, prefer acting on an available control. Default hover requires both AXShowAlternateUI and AXShowDefaultUI on the element or an enabled ancestor. Without an AX route it returns HOVER_UNAVAILABLE and sends no input. For browser content without those actions, use browser tools. Use click for a visible target or performAction for an advertised AXShowAlternateUI, AXShowMenu or AXPress. Only systemPointerHover: true permits moving the visible cursor; this requires a foreground, unobscured target. -- click(target: string | [number, number], options?: { mouseButton?: 'left' | 'right' | 'middle', clickCount?: 1 | 2, modifiers?: ('command' | 'control' | 'option' | 'shift' | 'fn')[], holdMs?: number, relative?: boolean }) -> ActionResult — on macOS, send mouse-down and mouse-up events at the center of an element in this window (id) or at the given point ([x, y]). Prefer invoke(id) for a background control’s default action. When a mouse click is needed, call app.activate() first and observe the target again: a background app can ignore a dispatched click. Set clickCount to 2 to perform a left-button double-click. holdMs controls each press duration. +- mouseMove(target: string | [number, number], options?: { relative?: boolean }) -> { dispatched: boolean, foregroundChanged?: boolean | null, hoverRoute: 'accessibility' | 'system-pointer', systemPointerMoved: boolean } — request hover at an observed element or window point. Inspect hoverRoute and systemPointerMoved to see how the provider dispatched it. +- click(target: string | [number, number], options?: { mouseButton?: 'left' | 'right' | 'middle', clickCount?: 1 | 2, modifiers?: ('command' | 'control' | 'option' | 'shift' | 'fn')[], holdMs?: number, relative?: boolean }) -> ActionResult — click an observed element or a window point. Set clickCount to 2 to perform a left-button double-click. holdMs controls each press duration. - scroll(options: { x: number, y: number, deltaLinesX: number, deltaLinesY: number, relative?: boolean }) -> ActionResult — scroll at a point; deltas are integers from -10 to 10 with at least one nonzero, and positive values reveal content above or left. - drag(from: [number, number], to: [number, number], options?: { mouseButton?: 'left' | 'right' | 'middle', modifiers?: ('command' | 'control' | 'option' | 'shift' | 'fn')[], relative?: boolean }) -> ActionResult — drag between two points of one window. -A string target is an element id; coordinate targets are [x, y] and are pixels in the latest returned screenshot by default. With relative: true they are integer thousandths of the window width and height from 0 to 999, based on the current AX observation or screenshot. Left clicks support one or two presses; right and middle clicks support one. For hover-only content, prefer an available click or advertised accessibility action. With system-pointer hover enabled, FOREGROUND_REQUIRED means activate the app explicitly and check that the point is unobscured. Read fresh state after actions, and treat WINDOW_CHANGED, STALE_SNAPSHOT, and SCREENSHOT_REQUIRED as requiring the stated fresh observation or capture before any retry. +A string target is an element id; coordinate targets are [x, y] and are pixels in the latest returned screenshot by default. With relative: true they are integer thousandths of the window width and height from 0 to 999, based on the current window observation or screenshot. Left clicks support one or two presses; right and middle clicks support one. Read fresh state after actions, and treat WINDOW_CHANGED, STALE_SNAPSHOT, and SCREENSHOT_REQUIRED as requiring the stated fresh observation or capture before any retry. ## Simulated keyboard Available at app or window level depending on the platform. -- typeText(text: string) -> ActionResult — type literal text at the app's current focus; line endings send Return and tabs send Tab. -- paste(text: string, options?: { format?: 'text' | 'html' }) -> ActionResult — deliver text or HTML through the app's Paste command, restoring prior clipboard contents unless another writer changes them; a Paste failure is surfaced without substituting typeText, and PASTE_NOT_CONSUMED means inspect before any retry. +- typeText(text: string) -> ActionResult — type text at the current input focus. +- paste(text: string, options?: { format?: 'text' | 'html' }) -> ActionResult — paste text or HTML at the current input focus. - pressKey(key: string, modifiers?: string[]) -> ActionResult — press and release one complete keystroke. For example pressKey('Tab', ['shift']) or pressKey('+'). Key and modifier names are platform-defined strings. Pass key and modifiers separately; do not pack them into a chord string. Raw tools use { method, params }: include appHandle for app calls and windowId for window calls, and use snapshotId with elementId for element routes. The emit option is facade-only: emit is local output control, so raw callers must follow the raw operation schema for admitted params. getAccessibilityTree and getWindow read through raw getState; getWindows maps to listWindows; getScreenshot maps to screenshot; cua.getState maps to listApps; every other named operation uses the same method name.` diff --git a/packages/experimental/cua/src/index.ts b/packages/experimental/cua/src/index.ts index 7cb5177f71..d6c75f46bc 100644 --- a/packages/experimental/cua/src/index.ts +++ b/packages/experimental/cua/src/index.ts @@ -126,8 +126,6 @@ export interface CuaBackend { createChannel: NativeChannelFactory guidance: string capabilities: CuaStartupCapabilities - /** macOS deployment policy sent only to the native mouseMove operation. */ - systemPointerHover?: boolean /** * Build native permission parameters for the selected action and Client language. * @param method Native permission action. @@ -213,7 +211,6 @@ export function installCua(ctx: Context, config: CuaConfig, backend: CuaBackend) const signal = execution.signal signal.throwIfAborted() const client = execution.nativeClient - if (method === 'mouseMove' && backend.systemPointerHover !== undefined) params.allowSystemPointer = backend.systemPointerHover const nativeParams = method === 'findApps' ? { ...params, maxAppMatches: config.maxAppMatches } : method === 'paste' diff --git a/packages/experimental/cua/src/native-schema.ts b/packages/experimental/cua/src/native-schema.ts index bbd52e2591..a6122ed71e 100644 --- a/packages/experimental/cua/src/native-schema.ts +++ b/packages/experimental/cua/src/native-schema.ts @@ -13,10 +13,10 @@ function variants(schema: Schema): Schema[] { return schema.anyOf ?? schema.oneO /** * Derive native request schemas and admission fields from the shared operation definitions. - * @param platformFields Platform-specific parameters added by the Host before native dispatch. + * @param platformFields Additional native protocol fields accepted by the provider. * @returns Native field and enum tables and the complete wire schema. */ -export function nativeProtocol(platformFields: Partial> = {}): { +export function nativeProtocol(platformFields: Partial['method'] | 'requestPermission' | 'openPermissionSettings', z.ZodObject>> = {}): { fields: Map> enums: Map>> json: string diff --git a/packages/experimental/cua/src/types.ts b/packages/experimental/cua/src/types.ts index 4e5f1ef0d9..4c6b27b78e 100644 --- a/packages/experimental/cua/src/types.ts +++ b/packages/experimental/cua/src/types.ts @@ -79,6 +79,7 @@ export type CuaMouseButton = NonNullable['mouseButton']> /** * Left clicks support one or two presses; right and middle clicks support one. * holdMs is a nonnegative integer in milliseconds, limited by maxClickHoldMs; zero releases immediately. + * Explicit holds use pointer input; omission preserves ordinary clicks. */ export type CuaClickOptions = Pick, 'mouseButton' | 'clickCount' | 'modifiers' | 'holdMs'> & CuaCoordinateOptions /** One same-window drag; omitted button is left and omitted modifiers are empty. */ @@ -95,7 +96,7 @@ export type CuaScrollOptions = Pick, 'x' | 'y' | 'deltaL */ export interface CuaSurface { /** - * Unhide the app and restore a minimized selected, focused, main or sole window without requesting activation. + * Request that the provider show the selected application or window. * @returns Dispatch evidence, the selected window when available, and sampled foreground change; read fresh state before input. */ show(): Promise @@ -190,22 +191,17 @@ export interface CuaWindow extends CuaSurface { */ getScreenshot(options?: CuaObservationOptions): Promise /** - * On macOS, prefer an available control's click or advertised accessibility action over hover. - * By default, mouseMove reveals or restores alternate UI on an enabled target or ancestor that advertises - * both AXShowAlternateUI and AXShowDefaultUI. Without an AX route it throws HOVER_UNAVAILABLE before sending input. - * Only the macOS provider's systemPointerHover setting permits moving the visible cursor as a fallback. - * That route requires a foreground app and an unobscured target and leaves the cursor at the destination. + * Request hover at an element or a point in this window. The provider reports the route used and whether + * the visible system pointer moved. Observe the resulting UI after its hover delay. * @param target - current element id or screenshot-local [x,y]. * @param options - coordinate mode; applies only to [x,y]. * @returns dispatch evidence and the selected route; observe the resulting UI after its hover delay. */ mouseMove(target: CuaElementId | readonly [number, number], options?: CuaCoordinateOptions): Promise /** - * Click an element or a point in this window. On macOS both forms send mouse-down and mouse-up events. - * For a background control, prefer {@link CuaSurface.invoke} for its default action. When a mouse click is needed, - * call app.activate() first and observe the target again. Background apps can ignore dispatched mouse events. + * Click a current element id or screenshot-local point. * Cancellation or timeout releases the held button; received input cannot be undone. The operation deadline includes holdMs. - * @param target - On macOS, an id from the latest tree targets the element center. A coordinate pair targets the given point. + * @param target - element id or [x,y]. * @param options - button, click count, modifiers, optional hold duration and coordinate mode. * @returns dispatch evidence. */ @@ -274,9 +270,9 @@ export interface CuaApi { */ getState(options?: CuaObservationOptions): Promise<{ apps: CuaAppInfo[]; browsers: [] }> /** - * Select an app, obtain consent, launch it in the background if necessary, and silently retain initial AX state. - * @param selector - Exact name, bundle ID, absolute application path or running PID string; ambiguous names reject. - * @returns An application handle tied to this process lifetime; its initial tree read becomes the current snapshot. + * Select an app and obtain consent. Providers with app-level accessibility silently read the initial tree. + * @param selector - A selector accepted by the provider; ambiguous matches reject. + * @returns An application handle tied to this process lifetime. */ getApp(selector: string): Promise } diff --git a/snapshots/session/cua-native/system-prompt.expected.md b/snapshots/session/cua-native/system-prompt.expected.md index 1738793701..5a793421cf 100644 --- a/snapshots/session/cua-native/system-prompt.expected.md +++ b/snapshots/session/cua-native/system-prompt.expected.md @@ -33,7 +33,7 @@ Start independent delegations with `subagent` or `subagent_fork` together in one Bash, formatters, code generators, and scripts can overwrite concurrent edits without conflict detection; coordinate their changes explicitly. The agent integrating the work must review the final diff and run relevant tests. -macOS: accessibility tree operations are available at app or window level. Simulated keyboard input is available at app level. Simulated mouse input is available at window level. Default hover uses AX alternate UI and keeps the system cursor in place. For pointer input on a sheet, select its window ID with app.getWindow(id). For menu items outside a window, use app.invoke(id). getApp also accepts bundle IDs and .app paths, and findApps searches /Applications, /System/Applications, and ~/Applications. Use the Computer Use permission page to grant the helper application Accessibility and Screen Recording permissions in System Settings. Native window and AX bounds use macOS points, while delivered screenshots carry their own pixel dimensions. show does not switch Spaces, and Stage Manager can present reduced window geometry; explain any required foreground step before calling app.activate() or performing an advertised AXRaise action. +macOS: accessibility tree operations are available at app or window level. Simulated keyboard input is available at app level. Simulated mouse input is available at window level. For pointer input on a sheet, select its window ID with app.getWindow(id). For menu items outside a window, use app.invoke(id). Use the Computer Use permission page to grant the helper application Accessibility and Screen Recording permissions in System Settings. Native window and AX bounds use macOS points, while delivered screenshots carry their own pixel dimensions. Stage Manager can present reduced window geometry; explain any required foreground step before calling app.activate() or performing an advertised AXRaise action. Use cua to observe and operate native applications. Always use cua.getCapabilities() to confirm whether a specific API is available on the current platform; its reported capabilities take precedence over availability statements in these instructions. All methods are async; observation options default to { emit: true }, and emit: false suppresses sink output only, leaving returned values unchanged. Actions report dispatch evidence, not verified completion. If sessionLocked is true or an operation reports SESSION_LOCKED, stop UI actions and ask the user to unlock the desktop session; null or unknown means verify a usable unlocked desktop session first. cua.getApp requests user consent; if access is denied or revoked, stop using that application until the user explicitly permits another request, and never bypass a refusal or switch tools to avoid it. Never replay uncertain input without a fresh observation. @@ -44,20 +44,20 @@ Signatures show resolved return values after await. ActionResult = { dispatched: Setup and discovery: - cua.getCapabilities(options?: { emit?: boolean }) -> { platform: string, permissions: { accessibility: boolean, screenRecording: boolean }, appLevelAccessibility: boolean, keyboard: ('app' | 'window')[], sessionLocked: boolean | null, methods: string[], coordinateSpace: 'screenshot-pixels', browser: false } — inspect reported permissions, sessionLocked, methods, and keyboard scopes without requesting permissions. - cua.listApps(options?: { emit?: boolean }) -> { id: string, displayName: string, isRunning: true, pid: number }[] — enumerate running applications. -- cua.findApps(query: string, options?: { emit?: boolean }) -> ({ id: string, displayName: string, path: string } & ({ isRunning: true, pid: number } | { isRunning: false }))[] — search installed applications, including stopped matches; only running matches carry pid. +- cua.findApps(query: string, options?: { emit?: boolean }) -> ({ id: string, displayName: string, path: string } & ({ isRunning: true, pid: number } | { isRunning: false }))[] — search for applications that match the query. Searches /Applications, /System/Applications, and ~/Applications. - cua.getState(options?: { emit?: boolean }) -> { apps: { id: string, displayName: string, isRunning: true, pid: number }[], browsers: [] } — summarize running applications; this is not an accessibility observation. -- cua.getApp(selector: string) -> App — select an application by name, bundle ID, absolute path, or running PID string, obtain consent, launch in the background if needed, and read initial AX state silently; it does not activate the app. -- app.getWindows(options?: { emit?: boolean }) -> { windowId: number, appName: string, title: string, bounds: { x: number, y: number, width: number, height: number }, isOnScreen: boolean, onCurrentSpace: boolean | null }[] — enumerate windows with global native bounds; null onCurrentSpace means unknown Space membership. -- app.getWindow(windowId: number) -> Window — select one window and read its initial AX state silently; do this before any window screenshot or coordinate input. +- cua.getApp(selector: string) -> App — select an application using a provider-supported selector and obtain consent. When app-level accessibility is available, selection reads the initial tree silently. Accepts a name, bundle ID, .app path, or running PID string. A stopped app launches in the background. +- app.getWindows(options?: { emit?: boolean }) -> { windowId: number, appName: string, title: string, bounds: { x: number, y: number, width: number, height: number }, isOnScreen: boolean, onCurrentSpace: boolean | null }[] — enumerate windows with global native bounds. A null onCurrentSpace means desktop membership is unknown. +- app.getWindow(windowId: number) -> Window — select one window and silently read its initial accessibility tree. Select a window before screenshots or pointer input. - app.activate() -> ActionResult — request explicit foreground activation; use only when foreground access is required. -- app.show() -> { dispatched: boolean, windowId: number | null, foregroundChanged: boolean | null } — unhide the app and restore its selected, focused, main, or sole window without activation. -- window.show() -> { dispatched: boolean, windowId: number | null, foregroundChanged: boolean | null } — unhide and restore that explicit window without activation. +- app.show() -> { dispatched: boolean, windowId: number | null, foregroundChanged: boolean | null } — request that the application be shown. Unhides the app and restores its focused, main, or sole window without activating it. +- window.show() -> { dispatched: boolean, windowId: number | null, foregroundChanged: boolean | null } — request that the selected window be shown. Unhides the app and restores this window without activating it. ## Accessibility tree Available at app or window level depending on the platform. -Element ids are opaque strings from the latest successful tree; the object facade keeps one current snapshot per app across its repeated app and window objects, a new read replaces it, and a failed read invalidates the previous one. Repeated strings can appear for different elements, so do not reuse an old id blindly, and element operations do not fall back to pointer input or typing. -- getAccessibilityTree(options?: { emit?: boolean, includeMenus?: boolean, details?: boolean }) -> string — read the full tree text; includeMenus expands menu contents and details adds identifiers, native bounds, and complete action names. +Element ids are opaque strings from the latest successful tree; the object facade keeps one current snapshot per app across its repeated app and window objects, a new read replaces it, and a failed read invalidates the previous one. Repeated strings can appear for different elements, so use IDs from the most recent observation. +- getAccessibilityTree(options?: { emit?: boolean, includeMenus?: boolean, details?: boolean }) -> string — read the full tree text. includeMenus requests menu contents and details requests additional element information. - listActions(elementId: string) -> string[] — list the actions an element advertises. - invoke(elementId: string) -> ActionResult — invoke the element's default action; dispatched is false when it has none. - performAction(elementId: string, action: string, argument?: JsonValue) -> ActionResult — perform an observed action with an optional direct JSON argument. @@ -67,17 +67,17 @@ Element ids are opaque strings from the latest successful tree; the object facad ## Simulated mouse Use an explicit window for screenshots and pointer input. - getScreenshot(options?: { emit?: boolean }) -> { bytes: Uint8Array, mimeType: 'image/png' | 'image/jpeg' | 'image/webp' | 'image/gif', pixelWidth: number, pixelHeight: number, windowId: number } — capture the selected window as one encoded image; bytes is a JavaScript byte array, and mimeType identifies its encoding, so do not assume PNG. -- mouseMove(target: string | [number, number], options?: { relative?: boolean }) -> { dispatched: boolean, foregroundChanged?: boolean | null, hoverRoute: 'accessibility' | 'system-pointer', systemPointerMoved: boolean } — on macOS, prefer acting on an available control. Default hover requires both AXShowAlternateUI and AXShowDefaultUI on the element or an enabled ancestor. Without an AX route it returns HOVER_UNAVAILABLE and sends no input. For browser content without those actions, use browser tools. Use click for a visible target or performAction for an advertised AXShowAlternateUI, AXShowMenu or AXPress. Only systemPointerHover: true permits moving the visible cursor; this requires a foreground, unobscured target. -- click(target: string | [number, number], options?: { mouseButton?: 'left' | 'right' | 'middle', clickCount?: 1 | 2, modifiers?: ('command' | 'control' | 'option' | 'shift' | 'fn')[], holdMs?: number, relative?: boolean }) -> ActionResult — on macOS, send mouse-down and mouse-up events at the center of an element in this window (id) or at the given point ([x, y]). Prefer invoke(id) for a background control’s default action. When a mouse click is needed, call app.activate() first and observe the target again: a background app can ignore a dispatched click. Set clickCount to 2 to perform a left-button double-click. holdMs controls each press duration. +- mouseMove(target: string | [number, number], options?: { relative?: boolean }) -> { dispatched: boolean, foregroundChanged?: boolean | null, hoverRoute: 'accessibility' | 'system-pointer', systemPointerMoved: boolean } — request hover at an observed element or window point. Inspect hoverRoute and systemPointerMoved to see how the provider dispatched it. Default hover requires both AXShowAlternateUI and AXShowDefaultUI on an enabled element or ancestor. Without an AX route it returns HOVER_UNAVAILABLE before sending input. Prefer click on a visible control or performAction with an advertised action. For web content without an AX hover action, use browser tools. Only systemPointerHover: true permits moving the visible cursor. If this returns FOREGROUND_REQUIRED, activate the app and check that the point is unobscured. +- click(target: string | [number, number], options?: { mouseButton?: 'left' | 'right' | 'middle', clickCount?: 1 | 2, modifiers?: ('command' | 'control' | 'option' | 'shift' | 'fn')[], holdMs?: number, relative?: boolean }) -> ActionResult — click an observed element or a window point. Set clickCount to 2 to perform a left-button double-click. holdMs controls each press duration. Both target forms send mouse-down and mouse-up events: an ID targets the element center and coordinates target the given point. Prefer invoke(id) for a background control’s default action. For a mouse click, call app.activate() first and observe the target again. A background app can ignore a dispatched click. - scroll(options: { x: number, y: number, deltaLinesX: number, deltaLinesY: number, relative?: boolean }) -> ActionResult — scroll at a point; deltas are integers from -10 to 10 with at least one nonzero, and positive values reveal content above or left. - drag(from: [number, number], to: [number, number], options?: { mouseButton?: 'left' | 'right' | 'middle', modifiers?: ('command' | 'control' | 'option' | 'shift' | 'fn')[], relative?: boolean }) -> ActionResult — drag between two points of one window. -A string target is an element id; coordinate targets are [x, y] and are pixels in the latest returned screenshot by default. With relative: true they are integer thousandths of the window width and height from 0 to 999, based on the current AX observation or screenshot. Left clicks support one or two presses; right and middle clicks support one. For hover-only content, prefer an available click or advertised accessibility action. With system-pointer hover enabled, FOREGROUND_REQUIRED means activate the app explicitly and check that the point is unobscured. Read fresh state after actions, and treat WINDOW_CHANGED, STALE_SNAPSHOT, and SCREENSHOT_REQUIRED as requiring the stated fresh observation or capture before any retry. +A string target is an element id; coordinate targets are [x, y] and are pixels in the latest returned screenshot by default. With relative: true they are integer thousandths of the window width and height from 0 to 999, based on the current window observation or screenshot. Left clicks support one or two presses; right and middle clicks support one. Read fresh state after actions, and treat WINDOW_CHANGED, STALE_SNAPSHOT, and SCREENSHOT_REQUIRED as requiring the stated fresh observation or capture before any retry. ## Simulated keyboard Available at app or window level depending on the platform. -- typeText(text: string) -> ActionResult — type literal text at the app's current focus; line endings send Return and tabs send Tab. -- paste(text: string, options?: { format?: 'text' | 'html' }) -> ActionResult — deliver text or HTML through the app's Paste command, restoring prior clipboard contents unless another writer changes them; a Paste failure is surfaced without substituting typeText, and PASTE_NOT_CONSUMED means inspect before any retry. +- typeText(text: string) -> ActionResult — type text at the current input focus. Line endings send Return and tabs send Tab. +- paste(text: string, options?: { format?: 'text' | 'html' }) -> ActionResult — paste text or HTML at the current input focus. Uses the app’s Paste command and restores the clipboard unless another writer changes it. For PASTE_NOT_CONSUMED, inspect the target before retrying. - pressKey(key: string, modifiers?: string[]) -> ActionResult — press and release one complete keystroke. For example pressKey('Tab', ['shift']) or pressKey('+'). Key and modifier names are platform-defined strings. Pass key and modifiers separately; do not pack them into a chord string. Raw tools use { method, params }: include appHandle for app calls and windowId for window calls, and use snapshotId with elementId for element routes. The emit option is facade-only: emit is local output control, so raw callers must follow the raw operation schema for admitted params. getAccessibilityTree and getWindow read through raw getState; getWindows maps to listWindows; getScreenshot maps to screenshot; cua.getState maps to listApps; every other named operation uses the same method name. diff --git a/snapshots/session/cua-native/workspace/native-helper.mjs b/snapshots/session/cua-native/workspace/native-helper.mjs index 0786eeefe6..1e148d7ab5 100755 --- a/snapshots/session/cua-native/workspace/native-helper.mjs +++ b/snapshots/session/cua-native/workspace/native-helper.mjs @@ -47,6 +47,7 @@ function serve() { const server = createServer({ allowHalfOpen: true }, socket => { sockets.add(socket) let connected = false + let systemPointerHover const connectionId = ++connection const input = createInterface({ input: socket }) const scopedRecord = (event, data = {}) => record(event, { connection: connectionId, ...data }) @@ -58,6 +59,7 @@ function serve() { socket.destroy() return } + systemPointerHover = request.config.systemPointerHover connected = true hadClient = true clearTimeout(startupDeadline) @@ -71,7 +73,7 @@ function serve() { } let response try { - const result = call(request.id, request.method, request.params) + const result = call(request.id, request.method, request.params, systemPointerHover) if (result === undefined) return response = { id: request.id, result } } catch (error) { @@ -122,7 +124,7 @@ function createApplicationState(socket, record, application, captureBroker, conn let observationChanged = false let malformedScreenshot = false function refuse(code, message) { throw Object.assign(new Error(message), { code }) } - function call(requestId, method, params) { + function call(requestId, method, params, systemPointerHover) { record('call', { requestId, method, params }) if (method === 'applicationIcons') return Object.fromEntries(params.paths.map(path => [path, `data:image/png;base64,${png}`])) if (method === 'openPermissionSettings') return { opened: true } @@ -203,7 +205,7 @@ function createApplicationState(socket, record, application, captureBroker, conn if (method === 'mouseMove') { const alternate = params.elementId === '0' || (params.x >= 20 && params.x <= 40) if (!alternate && !application.hovering) { - if (!params.allowSystemPointer) refuse('HOVER_UNAVAILABLE', 'No AX alternate UI at this target. No input was sent and the pointer did not move.') + if (!(systemPointerHover ?? params.allowSystemPointer)) refuse('HOVER_UNAVAILABLE', 'No AX alternate UI at this target. No input was sent and the pointer did not move.') systemPointerMoved = true } application.hovering = alternate diff --git a/snapshots/session/cua-ptc/system-prompt.expected.md b/snapshots/session/cua-ptc/system-prompt.expected.md index ff29eddb7f..8e3daa7253 100644 --- a/snapshots/session/cua-ptc/system-prompt.expected.md +++ b/snapshots/session/cua-ptc/system-prompt.expected.md @@ -35,7 +35,7 @@ Start independent delegations with `subagent` or `subagent_fork` together in one Bash, formatters, code generators, and scripts can overwrite concurrent edits without conflict detection; coordinate their changes explicitly. The agent integrating the work must review the final diff and run relevant tests. -macOS: accessibility tree operations are available at app or window level. Simulated keyboard input is available at app level. Simulated mouse input is available at window level. Default hover uses AX alternate UI and keeps the system cursor in place. For pointer input on a sheet, select its window ID with app.getWindow(id). For menu items outside a window, use app.invoke(id). getApp also accepts bundle IDs and .app paths, and findApps searches /Applications, /System/Applications, and ~/Applications. Use the Computer Use permission page to grant the helper application Accessibility and Screen Recording permissions in System Settings. Native window and AX bounds use macOS points, while delivered screenshots carry their own pixel dimensions. show does not switch Spaces, and Stage Manager can present reduced window geometry; explain any required foreground step before calling app.activate() or performing an advertised AXRaise action. +macOS: accessibility tree operations are available at app or window level. Simulated keyboard input is available at app level. Simulated mouse input is available at window level. For pointer input on a sheet, select its window ID with app.getWindow(id). For menu items outside a window, use app.invoke(id). Use the Computer Use permission page to grant the helper application Accessibility and Screen Recording permissions in System Settings. Native window and AX bounds use macOS points, while delivered screenshots carry their own pixel dimensions. Stage Manager can present reduced window geometry; explain any required foreground step before calling app.activate() or performing an advertised AXRaise action. Use cua to observe and operate native applications. Always use cua.getCapabilities() to confirm whether a specific API is available on the current platform; its reported capabilities take precedence over availability statements in these instructions. All methods are async; observation options default to { emit: true }, and emit: false suppresses sink output only, leaving returned values unchanged. Actions report dispatch evidence, not verified completion. If sessionLocked is true or an operation reports SESSION_LOCKED, stop UI actions and ask the user to unlock the desktop session; null or unknown means verify a usable unlocked desktop session first. cua.getApp requests user consent; if access is denied or revoked, stop using that application until the user explicitly permits another request, and never bypass a refusal or switch tools to avoid it. Never replay uncertain input without a fresh observation. @@ -46,20 +46,20 @@ Signatures show resolved return values after await. ActionResult = { dispatched: Setup and discovery: - cua.getCapabilities(options?: { emit?: boolean }) -> { platform: string, permissions: { accessibility: boolean, screenRecording: boolean }, appLevelAccessibility: boolean, keyboard: ('app' | 'window')[], sessionLocked: boolean | null, methods: string[], coordinateSpace: 'screenshot-pixels', browser: false } — inspect reported permissions, sessionLocked, methods, and keyboard scopes without requesting permissions. - cua.listApps(options?: { emit?: boolean }) -> { id: string, displayName: string, isRunning: true, pid: number }[] — enumerate running applications. -- cua.findApps(query: string, options?: { emit?: boolean }) -> ({ id: string, displayName: string, path: string } & ({ isRunning: true, pid: number } | { isRunning: false }))[] — search installed applications, including stopped matches; only running matches carry pid. +- cua.findApps(query: string, options?: { emit?: boolean }) -> ({ id: string, displayName: string, path: string } & ({ isRunning: true, pid: number } | { isRunning: false }))[] — search for applications that match the query. Searches /Applications, /System/Applications, and ~/Applications. - cua.getState(options?: { emit?: boolean }) -> { apps: { id: string, displayName: string, isRunning: true, pid: number }[], browsers: [] } — summarize running applications; this is not an accessibility observation. -- cua.getApp(selector: string) -> App — select an application by name, bundle ID, absolute path, or running PID string, obtain consent, launch in the background if needed, and read initial AX state silently; it does not activate the app. -- app.getWindows(options?: { emit?: boolean }) -> { windowId: number, appName: string, title: string, bounds: { x: number, y: number, width: number, height: number }, isOnScreen: boolean, onCurrentSpace: boolean | null }[] — enumerate windows with global native bounds; null onCurrentSpace means unknown Space membership. -- app.getWindow(windowId: number) -> Window — select one window and read its initial AX state silently; do this before any window screenshot or coordinate input. +- cua.getApp(selector: string) -> App — select an application using a provider-supported selector and obtain consent. When app-level accessibility is available, selection reads the initial tree silently. Accepts a name, bundle ID, .app path, or running PID string. A stopped app launches in the background. +- app.getWindows(options?: { emit?: boolean }) -> { windowId: number, appName: string, title: string, bounds: { x: number, y: number, width: number, height: number }, isOnScreen: boolean, onCurrentSpace: boolean | null }[] — enumerate windows with global native bounds. A null onCurrentSpace means desktop membership is unknown. +- app.getWindow(windowId: number) -> Window — select one window and silently read its initial accessibility tree. Select a window before screenshots or pointer input. - app.activate() -> ActionResult — request explicit foreground activation; use only when foreground access is required. -- app.show() -> { dispatched: boolean, windowId: number | null, foregroundChanged: boolean | null } — unhide the app and restore its selected, focused, main, or sole window without activation. -- window.show() -> { dispatched: boolean, windowId: number | null, foregroundChanged: boolean | null } — unhide and restore that explicit window without activation. +- app.show() -> { dispatched: boolean, windowId: number | null, foregroundChanged: boolean | null } — request that the application be shown. Unhides the app and restores its focused, main, or sole window without activating it. +- window.show() -> { dispatched: boolean, windowId: number | null, foregroundChanged: boolean | null } — request that the selected window be shown. Unhides the app and restores this window without activating it. ## Accessibility tree Available at app or window level depending on the platform. -Element ids are opaque strings from the latest successful tree; the object facade keeps one current snapshot per app across its repeated app and window objects, a new read replaces it, and a failed read invalidates the previous one. Repeated strings can appear for different elements, so do not reuse an old id blindly, and element operations do not fall back to pointer input or typing. -- getAccessibilityTree(options?: { emit?: boolean, includeMenus?: boolean, details?: boolean }) -> string — read the full tree text; includeMenus expands menu contents and details adds identifiers, native bounds, and complete action names. +Element ids are opaque strings from the latest successful tree; the object facade keeps one current snapshot per app across its repeated app and window objects, a new read replaces it, and a failed read invalidates the previous one. Repeated strings can appear for different elements, so use IDs from the most recent observation. +- getAccessibilityTree(options?: { emit?: boolean, includeMenus?: boolean, details?: boolean }) -> string — read the full tree text. includeMenus requests menu contents and details requests additional element information. - listActions(elementId: string) -> string[] — list the actions an element advertises. - invoke(elementId: string) -> ActionResult — invoke the element's default action; dispatched is false when it has none. - performAction(elementId: string, action: string, argument?: JsonValue) -> ActionResult — perform an observed action with an optional direct JSON argument. @@ -69,17 +69,17 @@ Element ids are opaque strings from the latest successful tree; the object facad ## Simulated mouse Use an explicit window for screenshots and pointer input. - getScreenshot(options?: { emit?: boolean }) -> { bytes: Uint8Array, mimeType: 'image/png' | 'image/jpeg' | 'image/webp' | 'image/gif', pixelWidth: number, pixelHeight: number, windowId: number } — capture the selected window as one encoded image; bytes is a JavaScript byte array, and mimeType identifies its encoding, so do not assume PNG. -- mouseMove(target: string | [number, number], options?: { relative?: boolean }) -> { dispatched: boolean, foregroundChanged?: boolean | null, hoverRoute: 'accessibility' | 'system-pointer', systemPointerMoved: boolean } — on macOS, prefer acting on an available control. Default hover requires both AXShowAlternateUI and AXShowDefaultUI on the element or an enabled ancestor. Without an AX route it returns HOVER_UNAVAILABLE and sends no input. For browser content without those actions, use browser tools. Use click for a visible target or performAction for an advertised AXShowAlternateUI, AXShowMenu or AXPress. Only systemPointerHover: true permits moving the visible cursor; this requires a foreground, unobscured target. -- click(target: string | [number, number], options?: { mouseButton?: 'left' | 'right' | 'middle', clickCount?: 1 | 2, modifiers?: ('command' | 'control' | 'option' | 'shift' | 'fn')[], holdMs?: number, relative?: boolean }) -> ActionResult — on macOS, send mouse-down and mouse-up events at the center of an element in this window (id) or at the given point ([x, y]). Prefer invoke(id) for a background control’s default action. When a mouse click is needed, call app.activate() first and observe the target again: a background app can ignore a dispatched click. Set clickCount to 2 to perform a left-button double-click. holdMs controls each press duration. +- mouseMove(target: string | [number, number], options?: { relative?: boolean }) -> { dispatched: boolean, foregroundChanged?: boolean | null, hoverRoute: 'accessibility' | 'system-pointer', systemPointerMoved: boolean } — request hover at an observed element or window point. Inspect hoverRoute and systemPointerMoved to see how the provider dispatched it. Default hover requires both AXShowAlternateUI and AXShowDefaultUI on an enabled element or ancestor. Without an AX route it returns HOVER_UNAVAILABLE before sending input. Prefer click on a visible control or performAction with an advertised action. For web content without an AX hover action, use browser tools. Only systemPointerHover: true permits moving the visible cursor. If this returns FOREGROUND_REQUIRED, activate the app and check that the point is unobscured. +- click(target: string | [number, number], options?: { mouseButton?: 'left' | 'right' | 'middle', clickCount?: 1 | 2, modifiers?: ('command' | 'control' | 'option' | 'shift' | 'fn')[], holdMs?: number, relative?: boolean }) -> ActionResult — click an observed element or a window point. Set clickCount to 2 to perform a left-button double-click. holdMs controls each press duration. Both target forms send mouse-down and mouse-up events: an ID targets the element center and coordinates target the given point. Prefer invoke(id) for a background control’s default action. For a mouse click, call app.activate() first and observe the target again. A background app can ignore a dispatched click. - scroll(options: { x: number, y: number, deltaLinesX: number, deltaLinesY: number, relative?: boolean }) -> ActionResult — scroll at a point; deltas are integers from -10 to 10 with at least one nonzero, and positive values reveal content above or left. - drag(from: [number, number], to: [number, number], options?: { mouseButton?: 'left' | 'right' | 'middle', modifiers?: ('command' | 'control' | 'option' | 'shift' | 'fn')[], relative?: boolean }) -> ActionResult — drag between two points of one window. -A string target is an element id; coordinate targets are [x, y] and are pixels in the latest returned screenshot by default. With relative: true they are integer thousandths of the window width and height from 0 to 999, based on the current AX observation or screenshot. Left clicks support one or two presses; right and middle clicks support one. For hover-only content, prefer an available click or advertised accessibility action. With system-pointer hover enabled, FOREGROUND_REQUIRED means activate the app explicitly and check that the point is unobscured. Read fresh state after actions, and treat WINDOW_CHANGED, STALE_SNAPSHOT, and SCREENSHOT_REQUIRED as requiring the stated fresh observation or capture before any retry. +A string target is an element id; coordinate targets are [x, y] and are pixels in the latest returned screenshot by default. With relative: true they are integer thousandths of the window width and height from 0 to 999, based on the current window observation or screenshot. Left clicks support one or two presses; right and middle clicks support one. Read fresh state after actions, and treat WINDOW_CHANGED, STALE_SNAPSHOT, and SCREENSHOT_REQUIRED as requiring the stated fresh observation or capture before any retry. ## Simulated keyboard Available at app or window level depending on the platform. -- typeText(text: string) -> ActionResult — type literal text at the app's current focus; line endings send Return and tabs send Tab. -- paste(text: string, options?: { format?: 'text' | 'html' }) -> ActionResult — deliver text or HTML through the app's Paste command, restoring prior clipboard contents unless another writer changes them; a Paste failure is surfaced without substituting typeText, and PASTE_NOT_CONSUMED means inspect before any retry. +- typeText(text: string) -> ActionResult — type text at the current input focus. Line endings send Return and tabs send Tab. +- paste(text: string, options?: { format?: 'text' | 'html' }) -> ActionResult — paste text or HTML at the current input focus. Uses the app’s Paste command and restores the clipboard unless another writer changes it. For PASTE_NOT_CONSUMED, inspect the target before retrying. - pressKey(key: string, modifiers?: string[]) -> ActionResult — press and release one complete keystroke. For example pressKey('Tab', ['shift']) or pressKey('+'). Key and modifier names are platform-defined strings. Pass key and modifiers separately; do not pack them into a chord string. Raw tools use { method, params }: include appHandle for app calls and windowId for window calls, and use snapshotId with elementId for element routes. The emit option is facade-only: emit is local output control, so raw callers must follow the raw operation schema for admitted params. getAccessibilityTree and getWindow read through raw getState; getWindows maps to listWindows; getScreenshot maps to screenshot; cua.getState maps to listApps; every other named operation uses the same method name. diff --git a/snapshots/session/ptc-script-retained/system-prompt.expected.md b/snapshots/session/ptc-script-retained/system-prompt.expected.md index 434e92e3f7..fcd315e68c 100644 --- a/snapshots/session/ptc-script-retained/system-prompt.expected.md +++ b/snapshots/session/ptc-script-retained/system-prompt.expected.md @@ -35,7 +35,7 @@ Start independent delegations with `subagent` or `subagent_fork` together in one Bash, formatters, code generators, and scripts can overwrite concurrent edits without conflict detection; coordinate their changes explicitly. The agent integrating the work must review the final diff and run relevant tests. -macOS: accessibility tree operations are available at app or window level. Simulated keyboard input is available at app level. Simulated mouse input is available at window level. Default hover uses AX alternate UI and keeps the system cursor in place. For pointer input on a sheet, select its window ID with app.getWindow(id). For menu items outside a window, use app.invoke(id). getApp also accepts bundle IDs and .app paths, and findApps searches /Applications, /System/Applications, and ~/Applications. Use the Computer Use permission page to grant the helper application Accessibility and Screen Recording permissions in System Settings. Native window and AX bounds use macOS points, while delivered screenshots carry their own pixel dimensions. show does not switch Spaces, and Stage Manager can present reduced window geometry; explain any required foreground step before calling app.activate() or performing an advertised AXRaise action. +macOS: accessibility tree operations are available at app or window level. Simulated keyboard input is available at app level. Simulated mouse input is available at window level. For pointer input on a sheet, select its window ID with app.getWindow(id). For menu items outside a window, use app.invoke(id). Use the Computer Use permission page to grant the helper application Accessibility and Screen Recording permissions in System Settings. Native window and AX bounds use macOS points, while delivered screenshots carry their own pixel dimensions. Stage Manager can present reduced window geometry; explain any required foreground step before calling app.activate() or performing an advertised AXRaise action. Use cua to observe and operate native applications. Always use cua.getCapabilities() to confirm whether a specific API is available on the current platform; its reported capabilities take precedence over availability statements in these instructions. All methods are async; observation options default to { emit: true }, and emit: false suppresses sink output only, leaving returned values unchanged. Actions report dispatch evidence, not verified completion. If sessionLocked is true or an operation reports SESSION_LOCKED, stop UI actions and ask the user to unlock the desktop session; null or unknown means verify a usable unlocked desktop session first. cua.getApp requests user consent; if access is denied or revoked, stop using that application until the user explicitly permits another request, and never bypass a refusal or switch tools to avoid it. Never replay uncertain input without a fresh observation. @@ -46,20 +46,20 @@ Signatures show resolved return values after await. ActionResult = { dispatched: Setup and discovery: - cua.getCapabilities(options?: { emit?: boolean }) -> { platform: string, permissions: { accessibility: boolean, screenRecording: boolean }, appLevelAccessibility: boolean, keyboard: ('app' | 'window')[], sessionLocked: boolean | null, methods: string[], coordinateSpace: 'screenshot-pixels', browser: false } — inspect reported permissions, sessionLocked, methods, and keyboard scopes without requesting permissions. - cua.listApps(options?: { emit?: boolean }) -> { id: string, displayName: string, isRunning: true, pid: number }[] — enumerate running applications. -- cua.findApps(query: string, options?: { emit?: boolean }) -> ({ id: string, displayName: string, path: string } & ({ isRunning: true, pid: number } | { isRunning: false }))[] — search installed applications, including stopped matches; only running matches carry pid. +- cua.findApps(query: string, options?: { emit?: boolean }) -> ({ id: string, displayName: string, path: string } & ({ isRunning: true, pid: number } | { isRunning: false }))[] — search for applications that match the query. Searches /Applications, /System/Applications, and ~/Applications. - cua.getState(options?: { emit?: boolean }) -> { apps: { id: string, displayName: string, isRunning: true, pid: number }[], browsers: [] } — summarize running applications; this is not an accessibility observation. -- cua.getApp(selector: string) -> App — select an application by name, bundle ID, absolute path, or running PID string, obtain consent, launch in the background if needed, and read initial AX state silently; it does not activate the app. -- app.getWindows(options?: { emit?: boolean }) -> { windowId: number, appName: string, title: string, bounds: { x: number, y: number, width: number, height: number }, isOnScreen: boolean, onCurrentSpace: boolean | null }[] — enumerate windows with global native bounds; null onCurrentSpace means unknown Space membership. -- app.getWindow(windowId: number) -> Window — select one window and read its initial AX state silently; do this before any window screenshot or coordinate input. +- cua.getApp(selector: string) -> App — select an application using a provider-supported selector and obtain consent. When app-level accessibility is available, selection reads the initial tree silently. Accepts a name, bundle ID, .app path, or running PID string. A stopped app launches in the background. +- app.getWindows(options?: { emit?: boolean }) -> { windowId: number, appName: string, title: string, bounds: { x: number, y: number, width: number, height: number }, isOnScreen: boolean, onCurrentSpace: boolean | null }[] — enumerate windows with global native bounds. A null onCurrentSpace means desktop membership is unknown. +- app.getWindow(windowId: number) -> Window — select one window and silently read its initial accessibility tree. Select a window before screenshots or pointer input. - app.activate() -> ActionResult — request explicit foreground activation; use only when foreground access is required. -- app.show() -> { dispatched: boolean, windowId: number | null, foregroundChanged: boolean | null } — unhide the app and restore its selected, focused, main, or sole window without activation. -- window.show() -> { dispatched: boolean, windowId: number | null, foregroundChanged: boolean | null } — unhide and restore that explicit window without activation. +- app.show() -> { dispatched: boolean, windowId: number | null, foregroundChanged: boolean | null } — request that the application be shown. Unhides the app and restores its focused, main, or sole window without activating it. +- window.show() -> { dispatched: boolean, windowId: number | null, foregroundChanged: boolean | null } — request that the selected window be shown. Unhides the app and restores this window without activating it. ## Accessibility tree Available at app or window level depending on the platform. -Element ids are opaque strings from the latest successful tree; the object facade keeps one current snapshot per app across its repeated app and window objects, a new read replaces it, and a failed read invalidates the previous one. Repeated strings can appear for different elements, so do not reuse an old id blindly, and element operations do not fall back to pointer input or typing. -- getAccessibilityTree(options?: { emit?: boolean, includeMenus?: boolean, details?: boolean }) -> string — read the full tree text; includeMenus expands menu contents and details adds identifiers, native bounds, and complete action names. +Element ids are opaque strings from the latest successful tree; the object facade keeps one current snapshot per app across its repeated app and window objects, a new read replaces it, and a failed read invalidates the previous one. Repeated strings can appear for different elements, so use IDs from the most recent observation. +- getAccessibilityTree(options?: { emit?: boolean, includeMenus?: boolean, details?: boolean }) -> string — read the full tree text. includeMenus requests menu contents and details requests additional element information. - listActions(elementId: string) -> string[] — list the actions an element advertises. - invoke(elementId: string) -> ActionResult — invoke the element's default action; dispatched is false when it has none. - performAction(elementId: string, action: string, argument?: JsonValue) -> ActionResult — perform an observed action with an optional direct JSON argument. @@ -69,17 +69,17 @@ Element ids are opaque strings from the latest successful tree; the object facad ## Simulated mouse Use an explicit window for screenshots and pointer input. - getScreenshot(options?: { emit?: boolean }) -> { bytes: Uint8Array, mimeType: 'image/png' | 'image/jpeg' | 'image/webp' | 'image/gif', pixelWidth: number, pixelHeight: number, windowId: number } — capture the selected window as one encoded image; bytes is a JavaScript byte array, and mimeType identifies its encoding, so do not assume PNG. -- mouseMove(target: string | [number, number], options?: { relative?: boolean }) -> { dispatched: boolean, foregroundChanged?: boolean | null, hoverRoute: 'accessibility' | 'system-pointer', systemPointerMoved: boolean } — on macOS, prefer acting on an available control. Default hover requires both AXShowAlternateUI and AXShowDefaultUI on the element or an enabled ancestor. Without an AX route it returns HOVER_UNAVAILABLE and sends no input. For browser content without those actions, use browser tools. Use click for a visible target or performAction for an advertised AXShowAlternateUI, AXShowMenu or AXPress. Only systemPointerHover: true permits moving the visible cursor; this requires a foreground, unobscured target. -- click(target: string | [number, number], options?: { mouseButton?: 'left' | 'right' | 'middle', clickCount?: 1 | 2, modifiers?: ('command' | 'control' | 'option' | 'shift' | 'fn')[], holdMs?: number, relative?: boolean }) -> ActionResult — on macOS, send mouse-down and mouse-up events at the center of an element in this window (id) or at the given point ([x, y]). Prefer invoke(id) for a background control’s default action. When a mouse click is needed, call app.activate() first and observe the target again: a background app can ignore a dispatched click. Set clickCount to 2 to perform a left-button double-click. holdMs controls each press duration. +- mouseMove(target: string | [number, number], options?: { relative?: boolean }) -> { dispatched: boolean, foregroundChanged?: boolean | null, hoverRoute: 'accessibility' | 'system-pointer', systemPointerMoved: boolean } — request hover at an observed element or window point. Inspect hoverRoute and systemPointerMoved to see how the provider dispatched it. Default hover requires both AXShowAlternateUI and AXShowDefaultUI on an enabled element or ancestor. Without an AX route it returns HOVER_UNAVAILABLE before sending input. Prefer click on a visible control or performAction with an advertised action. For web content without an AX hover action, use browser tools. Only systemPointerHover: true permits moving the visible cursor. If this returns FOREGROUND_REQUIRED, activate the app and check that the point is unobscured. +- click(target: string | [number, number], options?: { mouseButton?: 'left' | 'right' | 'middle', clickCount?: 1 | 2, modifiers?: ('command' | 'control' | 'option' | 'shift' | 'fn')[], holdMs?: number, relative?: boolean }) -> ActionResult — click an observed element or a window point. Set clickCount to 2 to perform a left-button double-click. holdMs controls each press duration. Both target forms send mouse-down and mouse-up events: an ID targets the element center and coordinates target the given point. Prefer invoke(id) for a background control’s default action. For a mouse click, call app.activate() first and observe the target again. A background app can ignore a dispatched click. - scroll(options: { x: number, y: number, deltaLinesX: number, deltaLinesY: number, relative?: boolean }) -> ActionResult — scroll at a point; deltas are integers from -10 to 10 with at least one nonzero, and positive values reveal content above or left. - drag(from: [number, number], to: [number, number], options?: { mouseButton?: 'left' | 'right' | 'middle', modifiers?: ('command' | 'control' | 'option' | 'shift' | 'fn')[], relative?: boolean }) -> ActionResult — drag between two points of one window. -A string target is an element id; coordinate targets are [x, y] and are pixels in the latest returned screenshot by default. With relative: true they are integer thousandths of the window width and height from 0 to 999, based on the current AX observation or screenshot. Left clicks support one or two presses; right and middle clicks support one. For hover-only content, prefer an available click or advertised accessibility action. With system-pointer hover enabled, FOREGROUND_REQUIRED means activate the app explicitly and check that the point is unobscured. Read fresh state after actions, and treat WINDOW_CHANGED, STALE_SNAPSHOT, and SCREENSHOT_REQUIRED as requiring the stated fresh observation or capture before any retry. +A string target is an element id; coordinate targets are [x, y] and are pixels in the latest returned screenshot by default. With relative: true they are integer thousandths of the window width and height from 0 to 999, based on the current window observation or screenshot. Left clicks support one or two presses; right and middle clicks support one. Read fresh state after actions, and treat WINDOW_CHANGED, STALE_SNAPSHOT, and SCREENSHOT_REQUIRED as requiring the stated fresh observation or capture before any retry. ## Simulated keyboard Available at app or window level depending on the platform. -- typeText(text: string) -> ActionResult — type literal text at the app's current focus; line endings send Return and tabs send Tab. -- paste(text: string, options?: { format?: 'text' | 'html' }) -> ActionResult — deliver text or HTML through the app's Paste command, restoring prior clipboard contents unless another writer changes them; a Paste failure is surfaced without substituting typeText, and PASTE_NOT_CONSUMED means inspect before any retry. +- typeText(text: string) -> ActionResult — type text at the current input focus. Line endings send Return and tabs send Tab. +- paste(text: string, options?: { format?: 'text' | 'html' }) -> ActionResult — paste text or HTML at the current input focus. Uses the app’s Paste command and restores the clipboard unless another writer changes it. For PASTE_NOT_CONSUMED, inspect the target before retrying. - pressKey(key: string, modifiers?: string[]) -> ActionResult — press and release one complete keystroke. For example pressKey('Tab', ['shift']) or pressKey('+'). Key and modifier names are platform-defined strings. Pass key and modifiers separately; do not pack them into a chord string. Raw tools use { method, params }: include appHandle for app calls and windowId for window calls, and use snapshotId with elementId for element routes. The emit option is facade-only: emit is local output control, so raw callers must follow the raw operation schema for admitted params. getAccessibilityTree and getWindow read through raw getState; getWindows maps to listWindows; getScreenshot maps to screenshot; cua.getState maps to listApps; every other named operation uses the same method name.