diff --git a/packages/experimental/cua/README.i18n.yaml b/packages/experimental/cua/README.i18n.yaml index 1e33abe9e8..216829b5b6 100644 --- a/packages/experimental/cua/README.i18n.yaml +++ b/packages/experimental/cua/README.i18n.yaml @@ -2,5 +2,5 @@ # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: # pnpm run verify-translation-pairing --write packages/experimental/cua/README.md -README.md: b4d75d3dd9c9cb3a81760d1be74fc4d8b95c6b86 -README.zh.md: 5d6c9c842aef33395cc9dfc87a398deae4db1a32 +README.md: 5e3f2875de3da1a3ba27811e9077640eb0f6f4c6 +README.zh.md: 836decc2d67fd0987e9d4932a5af81e6eb024e5e diff --git a/packages/experimental/cua/README.md b/packages/experimental/cua/README.md index b4d75d3dd9..5e3f2875de 100644 --- a/packages/experimental/cua/README.md +++ b/packages/experimental/cua/README.md @@ -34,7 +34,7 @@ export DSH_CUA_HELPER="$cua_native_root/target/release/dsh-cua-native" pnpm dsh headless --patch packages/experimental/cua/examples/macos.cordis.yml --dump-config ``` -The dump command checks profile composition without a model request. For a real task, replace `--dump-config` with a prompt and configure the profile’s model credentials. The overlay mounts the computer-use registry and this provider; remove an existing competing provider before using it. The responsible host/helper needs Accessibility permission for AX/input and Screen Recording permission for capture. `getCapabilities` reports permission state without opening a system dialog. Start the target application yourself. Select it by name, bundle ID, absolute application path, or PID. `getApp` reads the selected application without activating it. Use `performSecondaryAction(index, "raise")` on an observed window to activate it explicitly. Applications may change focus after input; background delivery does not guarantee an undisturbed user session. +The dump command checks profile composition without a model request. For a real task, replace `--dump-config` with a prompt and configure the profile’s model credentials. The overlay mounts the computer-use registry and this provider; remove an existing competing provider before using it. The responsible host/helper needs Accessibility permission for AX/input and Screen Recording permission for capture. `getCapabilities` reports permission state without opening a system dialog. Start the target application yourself. Select it by name, bundle ID, absolute application path, or PID. `getApp` reads the selected application without activating it. Use `performSecondaryAction(index, "raise")` on an observed window to activate it explicitly. If a hidden window or another Space prevents observation, `app.activate()` requests foreground recovery and may switch the focused app and Space; read fresh state afterward. Applications may change focus after input; background delivery does not guarantee an undisturbed user session. | Field | Default | Meaning | | --- | --- | --- | @@ -104,11 +104,11 @@ The plugin contributes one `cua` operation tool with method-specific parameters. ```markdown Use cua with method getCapabilities to inspect permissions, then listApps and getApp to select an already running application. Keep the returned appHandle; every app operation requires it. listWindows returns window IDs and point bounds. getState returns a complete AX tree and snapshotId. Use its element indices only with that snapshotId. Select a windowId explicitly for screenshots and coordinate actions. Coordinates are window-local macOS points, which may differ from screenshot pixels. -getApp selects without activating the app. Prefer background actions so the user can keep working elsewhere; native apps may still change focus in response to input. Invoke performSecondaryAction with an action advertised in the AX tree, such as raise, only when that visible action is needed. Raising a window explicitly activates its app. +getApp selects without activating the app. Prefer background actions so the user can keep working elsewhere; native apps may still change focus in response to input. Invoke performSecondaryAction with an action advertised in the AX tree, such as raise, only when that visible action is needed. Raising a window explicitly activates its app. When a hidden window or a window on another Space cannot be observed, app.activate() is an explicit foreground recovery that may switch the focused app and Space; read fresh AX state afterward. Await operations sequentially. After input, read fresh state to verify the effect. A successful action means dispatched, not verified completion. Cancellation may leave partial effects; never retry uncertain input without observing. STALE_APP requires getApp again; STALE_SNAPSHOT requires getState again. PERMISSION_DENIED requires the user to grant macOS Accessibility or Screen Recording permission. -The optional object API exposes getCapabilities, listApps, getState and getApp. App objects expose getWindows, getWindow(windowId), getAXState, getScreenshot, getAXStateAndScreenshot, click, setValue, performSecondaryAction, scroll, drag, typeText, paste and pressKey. Window objects expose observation and pointer/AX actions; keyboard input remains app-scoped and follows current focus. Scroll deltas are integer lines from -10 to 10; positive values reveal content above or left. Paste inserts plain text using native Unicode input without changing the clipboard; rich-text formats are unavailable. Browser applications can be controlled with getApp like other native apps; browser tabs and app launching are unavailable. The execution host must install the object API and own variable persistence; this plugin does not preload a cua global. In persistent PTC, initialize once with const { connectCua } = await import("@deepseek-ai/dsh-experimental-cua/bridge"); const cua = connectCua(() => tools.cua, text => console.log(text)). Retain app objects between calls. End screenshot calls with void await app.getScreenshot() or void await app.getAXStateAndScreenshot() so image bytes are not serialized as the program completion value. +The optional object API exposes getCapabilities, listApps, getState and getApp. App objects expose activate, getWindows, getWindow(windowId), getAXState, getScreenshot, getAXStateAndScreenshot, click, setValue, performSecondaryAction, scroll, drag, typeText, paste and pressKey. Window objects expose observation and pointer/AX actions; keyboard input remains app-scoped and follows current focus. Scroll deltas are integer lines from -10 to 10; positive values reveal content above or left. Paste inserts plain text using native Unicode input without changing the clipboard; rich-text formats are unavailable. Browser applications can be controlled with getApp like other native apps; browser tabs and app launching are unavailable. The execution host must install the object API and own variable persistence; this plugin does not preload a cua global. In persistent PTC, initialize once with const { connectCua } = await import("@deepseek-ai/dsh-experimental-cua/bridge"); const cua = connectCua(() => tools.cua, text => console.log(text)). Retain app objects between calls. End screenshot calls with void await app.getScreenshot() or void await app.getAXStateAndScreenshot() so image bytes are not serialized as the program completion value. ``` #### Token effect diff --git a/packages/experimental/cua/README.zh.md b/packages/experimental/cua/README.zh.md index 5d6c9c842a..836decc2d6 100644 --- a/packages/experimental/cua/README.zh.md +++ b/packages/experimental/cua/README.zh.md @@ -34,7 +34,7 @@ export DSH_CUA_HELPER="$cua_native_root/target/release/dsh-cua-native" pnpm dsh headless --patch packages/experimental/cua/examples/macos.cordis.yml --dump-config ``` -配置输出命令不发起模型请求。执行真实任务时，将 `--dump-config` 替换为任务提示，并配置该 profile 的模型凭据。配置挂载 computer-use 注册表和本 provider；使用前需移除已有的竞争 provider。负责执行的宿主或辅助进程需要辅助功能权限来读取 AX 和输入，需要屏幕录制权限来截图。`getCapabilities` 只查询权限，不弹出系统授权窗口。目标应用需提前启动，可按名称、bundle ID、应用绝对路径或 PID 选择。`getApp` 读取选中的应用，不自动激活它。需要前台操作时，对观察到的窗口调用 `performSecondaryAction(index, "raise")`。应用收到输入后仍可能改变焦点，后台投递不保证用户操作始终不受干扰。 +配置输出命令不发起模型请求。执行真实任务时，将 `--dump-config` 替换为任务提示，并配置该 profile 的模型凭据。配置挂载 computer-use 注册表和本 provider；使用前需移除已有的竞争 provider。负责执行的宿主或辅助进程需要辅助功能权限来读取 AX 和输入，需要屏幕录制权限来截图。`getCapabilities` 只查询权限，不弹出系统授权窗口。目标应用需提前启动，可按名称、bundle ID、应用绝对路径或 PID 选择。`getApp` 读取选中的应用，不自动激活它。需要前台操作时，对观察到的窗口调用 `performSecondaryAction(index, "raise")`。窗口隐藏或位于其他桌面空间而无法观察时，可显式调用 `app.activate()`，这可能切换焦点应用和桌面空间，之后需要重新读取状态。应用收到输入后仍可能改变焦点，后台投递不保证用户操作始终不受干扰。 | 字段 | 默认值 | 含义 | | --- | --- | --- | @@ -104,11 +104,11 @@ await window.getScreenshot() ```markdown Use cua with method getCapabilities to inspect permissions, then listApps and getApp to select an already running application. Keep the returned appHandle; every app operation requires it. listWindows returns window IDs and point bounds. getState returns a complete AX tree and snapshotId. Use its element indices only with that snapshotId. Select a windowId explicitly for screenshots and coordinate actions. Coordinates are window-local macOS points, which may differ from screenshot pixels. -getApp selects without activating the app. Prefer background actions so the user can keep working elsewhere; native apps may still change focus in response to input. Invoke performSecondaryAction with an action advertised in the AX tree, such as raise, only when that visible action is needed. Raising a window explicitly activates its app. +getApp selects without activating the app. Prefer background actions so the user can keep working elsewhere; native apps may still change focus in response to input. Invoke performSecondaryAction with an action advertised in the AX tree, such as raise, only when that visible action is needed. Raising a window explicitly activates its app. When a hidden window or a window on another Space cannot be observed, app.activate() is an explicit foreground recovery that may switch the focused app and Space; read fresh AX state afterward. Await operations sequentially. After input, read fresh state to verify the effect. A successful action means dispatched, not verified completion. Cancellation may leave partial effects; never retry uncertain input without observing. STALE_APP requires getApp again; STALE_SNAPSHOT requires getState again. PERMISSION_DENIED requires the user to grant macOS Accessibility or Screen Recording permission. -The optional object API exposes getCapabilities, listApps, getState and getApp. App objects expose getWindows, getWindow(windowId), getAXState, getScreenshot, getAXStateAndScreenshot, click, setValue, performSecondaryAction, scroll, drag, typeText, paste and pressKey. Window objects expose observation and pointer/AX actions; keyboard input remains app-scoped and follows current focus. Scroll deltas are integer lines from -10 to 10; positive values reveal content above or left. Paste inserts plain text using native Unicode input without changing the clipboard; rich-text formats are unavailable. Browser applications can be controlled with getApp like other native apps; browser tabs and app launching are unavailable. The execution host must install the object API and own variable persistence; this plugin does not preload a cua global. In persistent PTC, initialize once with const { connectCua } = await import("@deepseek-ai/dsh-experimental-cua/bridge"); const cua = connectCua(() => tools.cua, text => console.log(text)). Retain app objects between calls. End screenshot calls with void await app.getScreenshot() or void await app.getAXStateAndScreenshot() so image bytes are not serialized as the program completion value. +The optional object API exposes getCapabilities, listApps, getState and getApp. App objects expose activate, getWindows, getWindow(windowId), getAXState, getScreenshot, getAXStateAndScreenshot, click, setValue, performSecondaryAction, scroll, drag, typeText, paste and pressKey. Window objects expose observation and pointer/AX actions; keyboard input remains app-scoped and follows current focus. Scroll deltas are integer lines from -10 to 10; positive values reveal content above or left. Paste inserts plain text using native Unicode input without changing the clipboard; rich-text formats are unavailable. Browser applications can be controlled with getApp like other native apps; browser tabs and app launching are unavailable. The execution host must install the object API and own variable persistence; this plugin does not preload a cua global. In persistent PTC, initialize once with const { connectCua } = await import("@deepseek-ai/dsh-experimental-cua/bridge"); const cua = connectCua(() => tools.cua, text => console.log(text)). Retain app objects between calls. End screenshot calls with void await app.getScreenshot() or void await app.getAXStateAndScreenshot() so image bytes are not serialized as the program completion value. ``` #### Token 影响 diff --git a/packages/experimental/cua/native/protocol.schema.json b/packages/experimental/cua/native/protocol.schema.json index fefa81b9e0..303f952baf 100644 --- a/packages/experimental/cua/native/protocol.schema.json +++ b/packages/experimental/cua/native/protocol.schema.json @@ -608,6 +608,35 @@ "method", "params" ] + }, + { + "type": "object", + "additionalProperties": false, + "properties": { + "id": { + "type": "string" + }, + "method": { + "const": "activate" + }, + "params": { + "type": "object", + "additionalProperties": false, + "properties": { + "appHandle": { + "type": "string" + } + }, + "required": [ + "appHandle" + ] + } + }, + "required": [ + "id", + "method", + "params" + ] } ] } diff --git a/packages/experimental/cua/native/src/main.rs b/packages/experimental/cua/native/src/main.rs index 2936abacc1..69b3037b09 100644 --- a/packages/experimental/cua/native/src/main.rs +++ b/packages/experimental/cua/native/src/main.rs @@ -21,6 +21,7 @@ const METHODS: &[&str] = &[ "getCapabilities", "listApps", "getApp", + "activate", "listWindows", "getState", "click", @@ -319,7 +320,7 @@ impl Driver { let allowed: &[&str] = match method { "getCapabilities" | "listApps" => &[], "getApp" => &["selector"], - "listWindows" => &["appHandle"], + "listWindows" | "activate" => &["appHandle"], "getState" => &["appHandle", "windowId", "maxElements", "maxDepth"], "click" => &[ "appHandle", @@ -436,6 +437,10 @@ impl Driver { require(permissions::accessibility(), "PERMISSION_DENIED", "Accessibility permission is required for the responsible DSH host/helper; enable it in macOS Privacy & Security")?; } match method { + "activate" => { + apps::activate(pid)?; + Ok(json!({"dispatched":true,"verified":false})) + } "getState" => { let window = if params.get("windowId").is_some() { Some(window_id(params)?) @@ -481,7 +486,7 @@ impl Driver { require( scope.is_matched(), "WINDOW_UNAVAILABLE", - "Selected window has no matching accessibility surface", + "Selected window has no matching accessibility surface. If it is hidden or on another Space, activate the app explicitly before observing again", )?; } let elements: Vec<_> = tree.nodes.iter().filter_map(|node| node.element_index.map(|index| json!({ diff --git a/packages/experimental/cua/src/api.ts b/packages/experimental/cua/src/api.ts index cb6b9b338c..d7e774086e 100644 --- a/packages/experimental/cua/src/api.ts +++ b/packages/experimental/cua/src/api.ts @@ -133,6 +133,7 @@ export function createCuaApi(dispatch: CuaDispatch, sink?: CuaObservationSink): const { appHandle, pid } = AppResult.parse(await call('getApp', { selector })) const app: CuaApp = { ...surface(appHandle, pid), pid, appHandle, + async activate() { return action('activate', { appHandle }) }, async getWindows(options) { const windows = WindowsResult.parse(await call('listWindows', { appHandle })) if (windows.some(window => window.pid !== pid)) throw new Error('Native windows belong to another application') diff --git a/packages/experimental/cua/src/index.ts b/packages/experimental/cua/src/index.ts index 19c662f8c1..6fa35afbfc 100644 --- a/packages/experimental/cua/src/index.ts +++ b/packages/experimental/cua/src/index.ts @@ -56,11 +56,11 @@ export const Config: Schema = Schema.object({ /** Instructions for raw operation calls and optional object-API bindings. */ export const GUIDANCE = `Use cua with method getCapabilities to inspect permissions, then listApps and getApp to select an already running application. Keep the returned appHandle; every app operation requires it. listWindows returns window IDs and point bounds. getState returns a complete AX tree and snapshotId. Use its element indices only with that snapshotId. Select a windowId explicitly for screenshots and coordinate actions. Coordinates are window-local macOS points, which may differ from screenshot pixels. -getApp selects without activating the app. Prefer background actions so the user can keep working elsewhere; native apps may still change focus in response to input. Invoke performSecondaryAction with an action advertised in the AX tree, such as raise, only when that visible action is needed. Raising a window explicitly activates its app. +getApp selects without activating the app. Prefer background actions so the user can keep working elsewhere; native apps may still change focus in response to input. Invoke performSecondaryAction with an action advertised in the AX tree, such as raise, only when that visible action is needed. Raising a window explicitly activates its app. When a hidden window or a window on another Space cannot be observed, app.activate() is an explicit foreground recovery that may switch the focused app and Space; read fresh AX state afterward. Await operations sequentially. After input, read fresh state to verify the effect. A successful action means dispatched, not verified completion. Cancellation may leave partial effects; never retry uncertain input without observing. STALE_APP requires getApp again; STALE_SNAPSHOT requires getState again. PERMISSION_DENIED requires the user to grant macOS Accessibility or Screen Recording permission. -The optional object API exposes getCapabilities, listApps, getState and getApp. App objects expose getWindows, getWindow(windowId), getAXState, getScreenshot, getAXStateAndScreenshot, click, setValue, performSecondaryAction, scroll, drag, typeText, paste and pressKey. Window objects expose observation and pointer/AX actions; keyboard input remains app-scoped and follows current focus. Scroll deltas are integer lines from -10 to 10; positive values reveal content above or left. Paste inserts plain text using native Unicode input without changing the clipboard; rich-text formats are unavailable. Browser applications can be controlled with getApp like other native apps; browser tabs and app launching are unavailable. The execution host must install the object API and own variable persistence; this plugin does not preload a cua global. In persistent PTC, initialize once with const { connectCua } = await import("@deepseek-ai/dsh-experimental-cua/bridge"); const cua = connectCua(() => tools.cua, text => console.log(text)). Retain app objects between calls. End screenshot calls with void await app.getScreenshot() or void await app.getAXStateAndScreenshot() so image bytes are not serialized as the program completion value.` +The optional object API exposes getCapabilities, listApps, getState and getApp. App objects expose activate, getWindows, getWindow(windowId), getAXState, getScreenshot, getAXStateAndScreenshot, click, setValue, performSecondaryAction, scroll, drag, typeText, paste and pressKey. Window objects expose observation and pointer/AX actions; keyboard input remains app-scoped and follows current focus. Scroll deltas are integer lines from -10 to 10; positive values reveal content above or left. Paste inserts plain text using native Unicode input without changing the clipboard; rich-text formats are unavailable. Browser applications can be controlled with getApp like other native apps; browser tabs and app launching are unavailable. The execution host must install the object API and own variable persistence; this plugin does not preload a cua global. In persistent PTC, initialize once with const { connectCua } = await import("@deepseek-ai/dsh-experimental-cua/bridge"); const cua = connectCua(() => tools.cua, text => console.log(text)). Retain app objects between calls. End screenshot calls with void await app.getScreenshot() or void await app.getAXStateAndScreenshot() so image bytes are not serialized as the program completion value.` /** * Mount logged native operations with a separate helper per agent. diff --git a/packages/experimental/cua/src/protocol.ts b/packages/experimental/cua/src/protocol.ts index adc2619c9b..f5902b9aa0 100644 --- a/packages/experimental/cua/src/protocol.ts +++ b/packages/experimental/cua/src/protocol.ts @@ -19,6 +19,7 @@ const operation = (method: M, params: export const Request = z.discriminatedUnion('method', [ operation('getCapabilities', {}), operation('listApps', {}), operation('getApp', { selector: z.string().min(1) }), + operation('activate', app), operation('listWindows', app), operation('getState', { ...app, windowId: id.optional() }), z.object({ method: z.literal('click'), params: z.union([ diff --git a/packages/experimental/cua/src/types.ts b/packages/experimental/cua/src/types.ts index 4f265b3c57..f943534b28 100644 --- a/packages/experimental/cua/src/types.ts +++ b/packages/experimental/cua/src/types.ts @@ -118,6 +118,11 @@ export interface CuaWindow extends CuaSurface { readonly windowId: number } export interface CuaApp extends CuaSurface { readonly pid: number readonly appHandle: CuaAppHandle + /** + * Activate this application explicitly, potentially changing the user's focused app and Space. + * @returns Dispatch evidence; read fresh state before further interaction. + */ + activate(): Promise /** * Enumerate current windows. * @param options - output options. diff --git a/packages/experimental/cua/tests/api.spec.ts b/packages/experimental/cua/tests/api.spec.ts index c2f4a2e1c1..19f46dd9d1 100644 --- a/packages/experimental/cua/tests/api.spec.ts +++ b/packages/experimental/cua/tests/api.spec.ts @@ -347,6 +347,14 @@ describe('input actions', () => { await expect(window.performSecondaryAction(2, '')).rejects.toThrow() }) + it('does not activate on selection and dispatches only an explicit activation request', async () => { + const f = fixture() + const app = await f.api.getApp('Editor') + expect(f.dispatch.mock.calls.map(([call]) => call.method)).toEqual(['getApp', 'getState']) + await app.activate() + expect(f.dispatch).toHaveBeenLastCalledWith({ method: 'activate', params: { appHandle: 'app-17' } }) + }) + it('propagates native failures once and rejects malformed delivery claims', async () => { const f = fixture() const app = await f.api.getApp('Editor')