diff --git a/packages/experimental/cua/README.i18n.yaml b/packages/experimental/cua/README.i18n.yaml index bdb5dc4b87..1e33abe9e8 100644 --- a/packages/experimental/cua/README.i18n.yaml +++ b/packages/experimental/cua/README.i18n.yaml @@ -2,5 +2,5 @@ # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: # pnpm run verify-translation-pairing --write packages/experimental/cua/README.md -README.md: e21ff31fc5f69743094efeb05653857e63ac5560 -README.zh.md: c73d39298b808dafae9771d2cf69669c086695be +README.md: b4d75d3dd9c9cb3a81760d1be74fc4d8b95c6b86 +README.zh.md: 5d6c9c842aef33395cc9dfc87a398deae4db1a32 diff --git a/packages/experimental/cua/README.md b/packages/experimental/cua/README.md index e21ff31fc5..b4d75d3dd9 100644 --- a/packages/experimental/cua/README.md +++ b/packages/experimental/cua/README.md @@ -34,7 +34,7 @@ export DSH_CUA_HELPER="$cua_native_root/target/release/dsh-cua-native" pnpm dsh headless --patch packages/experimental/cua/examples/macos.cordis.yml --dump-config ``` -The dump command checks profile composition without a model request. For a real task, replace `--dump-config` with a prompt and configure the profile’s model credentials. The overlay mounts the computer-use registry and this provider; remove an existing competing provider before using it. The responsible host/helper needs Accessibility permission for AX/input and Screen Recording permission for capture. `getCapabilities` reports permission state without opening a system dialog. Start the target application yourself. Select it by name, bundle ID, absolute application path, or PID. `getApp` activates the selected application before reading its initial state. +The dump command checks profile composition without a model request. For a real task, replace `--dump-config` with a prompt and configure the profile’s model credentials. The overlay mounts the computer-use registry and this provider; remove an existing competing provider before using it. The responsible host/helper needs Accessibility permission for AX/input and Screen Recording permission for capture. `getCapabilities` reports permission state without opening a system dialog. Start the target application yourself. Select it by name, bundle ID, absolute application path, or PID. `getApp` reads the selected application without activating it. Use `performSecondaryAction(index, "raise")` on an observed window to activate it explicitly. Applications may change focus after input; background delivery does not guarantee an undisturbed user session. | Field | Default | Meaning | | --- | --- | --- | @@ -104,9 +104,11 @@ The plugin contributes one `cua` operation tool with method-specific parameters. ```markdown Use cua with method getCapabilities to inspect permissions, then listApps and getApp to select an already running application. Keep the returned appHandle; every app operation requires it. listWindows returns window IDs and point bounds. getState returns a complete AX tree and snapshotId. Use its element indices only with that snapshotId. Select a windowId explicitly for screenshots and coordinate actions. Coordinates are window-local macOS points, which may differ from screenshot pixels. +getApp selects without activating the app. Prefer background actions so the user can keep working elsewhere; native apps may still change focus in response to input. Invoke performSecondaryAction with an action advertised in the AX tree, such as raise, only when that visible action is needed. Raising a window explicitly activates its app. + Await operations sequentially. After input, read fresh state to verify the effect. A successful action means dispatched, not verified completion. Cancellation may leave partial effects; never retry uncertain input without observing. STALE_APP requires getApp again; STALE_SNAPSHOT requires getState again. PERMISSION_DENIED requires the user to grant macOS Accessibility or Screen Recording permission. -The optional object API exposes getCapabilities, listApps, getState and getApp. App objects expose getWindows, getWindow(windowId), getAXState, getScreenshot, getAXStateAndScreenshot, click, setValue, scroll, drag, typeText, paste and pressKey. Window objects expose observation and pointer/AX actions; keyboard input remains app-scoped and follows current focus. Scroll deltas are integer lines from -10 to 10; positive values reveal content above or left. Paste inserts plain text using native Unicode input without changing the clipboard; rich-text formats are unavailable. Browser applications can be controlled with getApp like other native apps; browser tabs and app launching are unavailable. The execution host must install the object API and own variable persistence; this plugin does not preload a cua global. In persistent PTC, initialize once with const { connectCua } = await import("@deepseek-ai/dsh-experimental-cua/bridge"); const cua = connectCua(() => tools.cua, text => console.log(text)). Retain app objects between calls. End screenshot calls with void await app.getScreenshot() or void await app.getAXStateAndScreenshot() so image bytes are not serialized as the program completion value. +The optional object API exposes getCapabilities, listApps, getState and getApp. App objects expose getWindows, getWindow(windowId), getAXState, getScreenshot, getAXStateAndScreenshot, click, setValue, performSecondaryAction, scroll, drag, typeText, paste and pressKey. Window objects expose observation and pointer/AX actions; keyboard input remains app-scoped and follows current focus. Scroll deltas are integer lines from -10 to 10; positive values reveal content above or left. Paste inserts plain text using native Unicode input without changing the clipboard; rich-text formats are unavailable. Browser applications can be controlled with getApp like other native apps; browser tabs and app launching are unavailable. The execution host must install the object API and own variable persistence; this plugin does not preload a cua global. In persistent PTC, initialize once with const { connectCua } = await import("@deepseek-ai/dsh-experimental-cua/bridge"); const cua = connectCua(() => tools.cua, text => console.log(text)). Retain app objects between calls. End screenshot calls with void await app.getScreenshot() or void await app.getAXStateAndScreenshot() so image bytes are not serialized as the program completion value. ``` #### Token effect diff --git a/packages/experimental/cua/README.zh.md b/packages/experimental/cua/README.zh.md index c73d39298b..5d6c9c842a 100644 --- a/packages/experimental/cua/README.zh.md +++ b/packages/experimental/cua/README.zh.md @@ -34,7 +34,7 @@ export DSH_CUA_HELPER="$cua_native_root/target/release/dsh-cua-native" pnpm dsh headless --patch packages/experimental/cua/examples/macos.cordis.yml --dump-config ``` -配置输出命令不发起模型请求。执行真实任务时，将 `--dump-config` 替换为任务提示，并配置该 profile 的模型凭据。配置挂载 computer-use 注册表和本 provider；使用前需移除已有的竞争 provider。负责执行的宿主或辅助进程需要辅助功能权限来读取 AX 和输入，需要屏幕录制权限来截图。`getCapabilities` 只查询权限，不弹出系统授权窗口。目标应用需提前启动，可按名称、bundle ID、应用绝对路径或 PID 选择。`getApp` 激活选中的应用，再读取初始状态。 +配置输出命令不发起模型请求。执行真实任务时，将 `--dump-config` 替换为任务提示，并配置该 profile 的模型凭据。配置挂载 computer-use 注册表和本 provider；使用前需移除已有的竞争 provider。负责执行的宿主或辅助进程需要辅助功能权限来读取 AX 和输入，需要屏幕录制权限来截图。`getCapabilities` 只查询权限，不弹出系统授权窗口。目标应用需提前启动，可按名称、bundle ID、应用绝对路径或 PID 选择。`getApp` 读取选中的应用，不自动激活它。需要前台操作时，对观察到的窗口调用 `performSecondaryAction(index, "raise")`。应用收到输入后仍可能改变焦点，后台投递不保证用户操作始终不受干扰。 | 字段 | 默认值 | 含义 | | --- | --- | --- | @@ -104,9 +104,11 @@ await window.getScreenshot() ```markdown Use cua with method getCapabilities to inspect permissions, then listApps and getApp to select an already running application. Keep the returned appHandle; every app operation requires it. listWindows returns window IDs and point bounds. getState returns a complete AX tree and snapshotId. Use its element indices only with that snapshotId. Select a windowId explicitly for screenshots and coordinate actions. Coordinates are window-local macOS points, which may differ from screenshot pixels. +getApp selects without activating the app. Prefer background actions so the user can keep working elsewhere; native apps may still change focus in response to input. Invoke performSecondaryAction with an action advertised in the AX tree, such as raise, only when that visible action is needed. Raising a window explicitly activates its app. + Await operations sequentially. After input, read fresh state to verify the effect. A successful action means dispatched, not verified completion. Cancellation may leave partial effects; never retry uncertain input without observing. STALE_APP requires getApp again; STALE_SNAPSHOT requires getState again. PERMISSION_DENIED requires the user to grant macOS Accessibility or Screen Recording permission. -The optional object API exposes getCapabilities, listApps, getState and getApp. App objects expose getWindows, getWindow(windowId), getAXState, getScreenshot, getAXStateAndScreenshot, click, setValue, scroll, drag, typeText, paste and pressKey. Window objects expose observation and pointer/AX actions; keyboard input remains app-scoped and follows current focus. Scroll deltas are integer lines from -10 to 10; positive values reveal content above or left. Paste inserts plain text using native Unicode input without changing the clipboard; rich-text formats are unavailable. Browser applications can be controlled with getApp like other native apps; browser tabs and app launching are unavailable. The execution host must install the object API and own variable persistence; this plugin does not preload a cua global. In persistent PTC, initialize once with const { connectCua } = await import("@deepseek-ai/dsh-experimental-cua/bridge"); const cua = connectCua(() => tools.cua, text => console.log(text)). Retain app objects between calls. End screenshot calls with void await app.getScreenshot() or void await app.getAXStateAndScreenshot() so image bytes are not serialized as the program completion value. +The optional object API exposes getCapabilities, listApps, getState and getApp. App objects expose getWindows, getWindow(windowId), getAXState, getScreenshot, getAXStateAndScreenshot, click, setValue, performSecondaryAction, scroll, drag, typeText, paste and pressKey. Window objects expose observation and pointer/AX actions; keyboard input remains app-scoped and follows current focus. Scroll deltas are integer lines from -10 to 10; positive values reveal content above or left. Paste inserts plain text using native Unicode input without changing the clipboard; rich-text formats are unavailable. Browser applications can be controlled with getApp like other native apps; browser tabs and app launching are unavailable. The execution host must install the object API and own variable persistence; this plugin does not preload a cua global. In persistent PTC, initialize once with const { connectCua } = await import("@deepseek-ai/dsh-experimental-cua/bridge"); const cua = connectCua(() => tools.cua, text => console.log(text)). Retain app objects between calls. End screenshot calls with void await app.getScreenshot() or void await app.getAXStateAndScreenshot() so image bytes are not serialized as the program completion value. ``` #### Token 影响 diff --git a/packages/experimental/cua/native/protocol.schema.json b/packages/experimental/cua/native/protocol.schema.json index af13a24749..fefa81b9e0 100644 --- a/packages/experimental/cua/native/protocol.schema.json +++ b/packages/experimental/cua/native/protocol.schema.json @@ -566,6 +566,48 @@ "method", "params" ] + }, + { + "type": "object", + "additionalProperties": false, + "properties": { + "id": { + "type": "string" + }, + "method": { + "const": "performSecondaryAction" + }, + "params": { + "type": "object", + "additionalProperties": false, + "properties": { + "appHandle": { + "type": "string" + }, + "snapshotId": { + "type": "string" + }, + "elementIndex": { + "type": "integer", + "minimum": 0 + }, + "action": { + "type": "string" + } + }, + "required": [ + "appHandle", + "snapshotId", + "elementIndex", + "action" + ] + } + }, + "required": [ + "id", + "method", + "params" + ] } ] } diff --git a/packages/experimental/cua/native/src/apps.rs b/packages/experimental/cua/native/src/apps.rs index 6e99182db4..451f687872 100644 --- a/packages/experimental/cua/native/src/apps.rs +++ b/packages/experimental/cua/native/src/apps.rs @@ -59,9 +59,7 @@ pub fn activate(pid: i32) -> anyhow::Result<()> { .ok_or_else(|| anyhow::anyhow!("Application exited before activation"))?; app.unhide(); anyhow::ensure!( - app.activateWithOptions( - NSApplicationActivationOptions::NSApplicationActivateIgnoringOtherApps - ), + app.activateWithOptions(NSApplicationActivationOptions::empty()), "macOS refused application activation" ); } diff --git a/packages/experimental/cua/native/src/element_input.rs b/packages/experimental/cua/native/src/element_input.rs index fbbebbc550..82d4dd258d 100644 --- a/packages/experimental/cua/native/src/element_input.rs +++ b/packages/experimental/cua/native/src/element_input.rs @@ -3,6 +3,35 @@ use crate::{ax::bindings::*, input, owned_window}; use anyhow::{ensure, Context, Result}; use core_foundation::base::{CFRelease, CFTypeRef}; +fn action_key(value: &str) -> String { + value + .strip_prefix("AX") + .unwrap_or(value) + .chars() + .filter(|character| !character.is_whitespace()) + .flat_map(char::to_lowercase) + .collect() +} + +pub fn secondary_action(pid: i32, pointer: usize, requested: &str) -> Result<()> { + let element = pointer as AXUIElementRef; + let actions = unsafe { copy_action_names(element) }; + let action = actions + .iter() + .find(|action| action_key(action) == action_key(requested)) + .context("Element does not advertise this action; refresh getAXState")?; + input::ax_actions::ensure_ax_action_enabled(pointer, action)?; + if action == "AXRaise" { + crate::apps::activate(pid)?; + } + let error = unsafe { perform_action(element, action) }; + ensure!( + error == kAXErrorSuccess, + "Accessibility action {requested} failed with error {error}" + ); + Ok(()) +} + pub fn click(pid: i32, pointer: usize, button: &str, count: usize) -> Result<()> { ensure!( matches!((button, count), ("left", 1 | 2) | ("right", 1)), @@ -63,3 +92,15 @@ pub fn click(pid: i32, pointer: usize, button: &str, count: usize) -> Result<()> ) } } + +#[cfg(test)] +mod tests { + use super::action_key; + + #[test] + fn observed_native_and_display_action_names_match() { + assert_eq!(action_key("AXRaise"), action_key("Raise")); + assert_eq!(action_key("AXZoomWindow"), action_key("zoom window")); + assert_ne!(action_key("AXRaise"), action_key("AXPress")); + } +} diff --git a/packages/experimental/cua/native/src/main.rs b/packages/experimental/cua/native/src/main.rs index 94915586b7..2936abacc1 100644 --- a/packages/experimental/cua/native/src/main.rs +++ b/packages/experimental/cua/native/src/main.rs @@ -25,6 +25,7 @@ const METHODS: &[&str] = &[ "getState", "click", "setValue", + "performSecondaryAction", "typeText", "pressKey", "scroll", @@ -331,6 +332,7 @@ impl Driver { "clickCount", ], "setValue" => &["appHandle", "snapshotId", "elementIndex", "value"], + "performSecondaryAction" => &["appHandle", "snapshotId", "elementIndex", "action"], "typeText" => &["appHandle", "text"], "pressKey" => &["appHandle", "key", "modifiers"], "scroll" => &[ @@ -397,7 +399,6 @@ impl Driver { "Selector matches multiple running apps; select an exact PID from listApps", )?; let app = &matches[0]; - apps::activate(app.pid)?; let handle = self.remember_app(app.pid, process_started(app.pid)?); let mut result = json!({"pid":app.pid,"appHandle":handle,"name":app.name}); if let Some(bundle_id) = &app.bundle_id { @@ -582,6 +583,14 @@ impl Driver { )?; Ok(json!({"dispatched":true,"verified":false})) } + "performSecondaryAction" => { + element_input::secondary_action( + pid, + self.element(&handle, params)?, + text(params, "action")?, + )?; + Ok(json!({"dispatched":true,"verified":false})) + } "typeText" => { input::keyboard::type_text(pid, text(params, "text")?)?; Ok(json!({"dispatched":true,"verified":false})) diff --git a/packages/experimental/cua/src/api.ts b/packages/experimental/cua/src/api.ts index 1365ab2950..cb6b9b338c 100644 --- a/packages/experimental/cua/src/api.ts +++ b/packages/experimental/cua/src/api.ts @@ -96,6 +96,7 @@ export function createCuaApi(dispatch: CuaDispatch, sink?: CuaObservationSink): return action('click', { ...snapshotParams(), windowId: windowId(), x, y, mouseButton, clickCount }) }, async setValue(index, value) { return action('setValue', { ...snapshotParams(), elementIndex: index, value }) }, + async performSecondaryAction(index, name) { return action('performSecondaryAction', { ...snapshotParams(), elementIndex: index, action: name }) }, async scroll(options) { if (options.deltaLinesX === 0 && options.deltaLinesY === 0) throw new Error('At least one scroll delta must be nonzero') return action('scroll', { diff --git a/packages/experimental/cua/src/index.ts b/packages/experimental/cua/src/index.ts index 7d792fb628..19c662f8c1 100644 --- a/packages/experimental/cua/src/index.ts +++ b/packages/experimental/cua/src/index.ts @@ -56,9 +56,11 @@ export const Config: Schema = Schema.object({ /** Instructions for raw operation calls and optional object-API bindings. */ export const GUIDANCE = `Use cua with method getCapabilities to inspect permissions, then listApps and getApp to select an already running application. Keep the returned appHandle; every app operation requires it. listWindows returns window IDs and point bounds. getState returns a complete AX tree and snapshotId. Use its element indices only with that snapshotId. Select a windowId explicitly for screenshots and coordinate actions. Coordinates are window-local macOS points, which may differ from screenshot pixels. +getApp selects without activating the app. Prefer background actions so the user can keep working elsewhere; native apps may still change focus in response to input. Invoke performSecondaryAction with an action advertised in the AX tree, such as raise, only when that visible action is needed. Raising a window explicitly activates its app. + Await operations sequentially. After input, read fresh state to verify the effect. A successful action means dispatched, not verified completion. Cancellation may leave partial effects; never retry uncertain input without observing. STALE_APP requires getApp again; STALE_SNAPSHOT requires getState again. PERMISSION_DENIED requires the user to grant macOS Accessibility or Screen Recording permission. -The optional object API exposes getCapabilities, listApps, getState and getApp. App objects expose getWindows, getWindow(windowId), getAXState, getScreenshot, getAXStateAndScreenshot, click, setValue, scroll, drag, typeText, paste and pressKey. Window objects expose observation and pointer/AX actions; keyboard input remains app-scoped and follows current focus. Scroll deltas are integer lines from -10 to 10; positive values reveal content above or left. Paste inserts plain text using native Unicode input without changing the clipboard; rich-text formats are unavailable. Browser applications can be controlled with getApp like other native apps; browser tabs and app launching are unavailable. The execution host must install the object API and own variable persistence; this plugin does not preload a cua global. In persistent PTC, initialize once with const { connectCua } = await import("@deepseek-ai/dsh-experimental-cua/bridge"); const cua = connectCua(() => tools.cua, text => console.log(text)). Retain app objects between calls. End screenshot calls with void await app.getScreenshot() or void await app.getAXStateAndScreenshot() so image bytes are not serialized as the program completion value.` +The optional object API exposes getCapabilities, listApps, getState and getApp. App objects expose getWindows, getWindow(windowId), getAXState, getScreenshot, getAXStateAndScreenshot, click, setValue, performSecondaryAction, scroll, drag, typeText, paste and pressKey. Window objects expose observation and pointer/AX actions; keyboard input remains app-scoped and follows current focus. Scroll deltas are integer lines from -10 to 10; positive values reveal content above or left. Paste inserts plain text using native Unicode input without changing the clipboard; rich-text formats are unavailable. Browser applications can be controlled with getApp like other native apps; browser tabs and app launching are unavailable. The execution host must install the object API and own variable persistence; this plugin does not preload a cua global. In persistent PTC, initialize once with const { connectCua } = await import("@deepseek-ai/dsh-experimental-cua/bridge"); const cua = connectCua(() => tools.cua, text => console.log(text)). Retain app objects between calls. End screenshot calls with void await app.getScreenshot() or void await app.getAXStateAndScreenshot() so image bytes are not serialized as the program completion value.` /** * Mount logged native operations with a separate helper per agent. diff --git a/packages/experimental/cua/src/protocol.ts b/packages/experimental/cua/src/protocol.ts index 8d9dd7adb3..adc2619c9b 100644 --- a/packages/experimental/cua/src/protocol.ts +++ b/packages/experimental/cua/src/protocol.ts @@ -26,6 +26,7 @@ export const Request = z.discriminatedUnion('method', [ z.object({ ...coordinate, mouseButton: z.enum(['left', 'right']).optional(), clickCount: z.union([z.literal(1), z.literal(2)]).optional() }).strict(), ]) }).strict(), operation('setValue', { ...snapshot, value: z.string() }), + operation('performSecondaryAction', { ...snapshot, action: z.string().min(1) }), operation('typeText', { ...app, text: z.string() }), operation('pressKey', { ...app, key: z.string().min(1), modifiers: z.array(z.enum(['command', 'control', 'option', 'shift', 'fn'])).optional() }), operation('scroll', { ...coordinate, deltaLinesX: z.number().int().min(-10).max(10), deltaLinesY: z.number().int().min(-10).max(10) }), diff --git a/packages/experimental/cua/src/types.ts b/packages/experimental/cua/src/types.ts index d0671d1595..4f265b3c57 100644 --- a/packages/experimental/cua/src/types.ts +++ b/packages/experimental/cua/src/types.ts @@ -91,6 +91,13 @@ export interface CuaSurface { * @returns dispatch evidence. */ setValue(index: number, value: string): Promise + /** + * Invoke an action advertised by an observed element; raising a window activates its app. + * @param index - Index from this surface's latest AX tree. + * @param action - Advertised action name, such as raise or zoomwindow. + * @returns Dispatch evidence; menu and window actions can change focus. + */ + performSecondaryAction(index: number, action: string): Promise /** * Scroll at a window point. * @param options - point and line deltas; at least one delta must be nonzero. @@ -163,7 +170,7 @@ export interface CuaApi { */ getState(options?: CuaObservationOptions): Promise<{ apps: CuaAppInfo[]; browsers: [] }> /** - * Activate an existing app and emit initial AX state. + * Select an existing app and emit initial AX state without activating it. * @param selector - exact name, bundle ID, absolute application path or PID string. * @returns a handle tied to this process lifetime. */ diff --git a/packages/experimental/cua/tests/api.spec.ts b/packages/experimental/cua/tests/api.spec.ts index 97048e6a21..c2f4a2e1c1 100644 --- a/packages/experimental/cua/tests/api.spec.ts +++ b/packages/experimental/cua/tests/api.spec.ts @@ -336,6 +336,17 @@ describe('input actions', () => { ]) }) + it('routes an explicit secondary action through the selected window snapshot', async () => { + const f = fixture() + const app = await f.api.getApp('Editor') + const window = await app.getWindow(23) + await window.performSecondaryAction(2, 'raise') + expect(f.dispatch).toHaveBeenLastCalledWith({ method: 'performSecondaryAction', params: { + appHandle: 'app-17', snapshotId: 'snapshot-2', elementIndex: 2, action: 'raise', + } }) + await expect(window.performSecondaryAction(2, '')).rejects.toThrow() + }) + it('propagates native failures once and rejects malformed delivery claims', async () => { const f = fixture() const app = await f.api.getApp('Editor')