fix(cua): prefer direct app operations for single-window tasks
This commit is contained in:
creatixchu committed
1 parent
22a5a41d0c
commit
9c33b13e5d
6 files changed
+32
-18
No files matched your search
@@ -18,8 +18,8 @@
|
||||
en: 7cb6848d7ccd7132
|
||||
zh: 48900d1231f97bb6
|
||||
/deepseek-ai-dsh-cua/use-this-package/connect-an-external-execution-environment:
|
||||
en: e2d25d61e40dc88a
|
||||
zh: 1b25e636684e14ca
|
||||
en: 2cdaaca90c5c78f7
|
||||
zh: bcdbf54da0353d9a
|
||||
/deepseek-ai-dsh-cua/understand-the-implementation:
|
||||
en: ee9fd0ddf93d496d
|
||||
zh: 38d8d5f8b58e6f50
|
||||
|
||||
@@ -68,12 +68,12 @@ To run through persistent PTC, add `--patch packages/computer-use/cua/examples/p
|
||||
import { connectCua } from '@deepseek-ai/dsh-cua/bridge'
|
||||
const cua = connectCua(() => tools.cua, text => console.log(text))
|
||||
const app = await cua.getApp("com.apple.TextEdit")
|
||||
const windows = await app.getWindows()
|
||||
const window = await app.getWindow(windows[0].windowId)
|
||||
await window.getScreenshot()
|
||||
void await app.getScreenshot()
|
||||
```
|
||||
|
||||
Choose the intended window from its title and bounds. `getWindow` emits its initial AX tree. An owned window without an AX surface returns an empty snapshot and remains available for screenshots and coordinate input. Window objects retain separate snapshots, so reading one window does not invalidate another. Re-reading the same scope invalidates old indices. An AX node removed during observation rejects the read with `STALE_SNAPSHOT` and publishes no partial tree; read again without repeating the preceding input. App-scoped screenshots and coordinates use the focused window when it appears in the latest observation, otherwise require exactly one visible, nonzero-size window. Explicit window objects select a particular target. Keyboard methods stay on `app` because input follows the app’s current focus.
|
||||
Use app methods directly for single-window tasks. `getApp` reads initial AX state, so a screenshot needs no window enumeration or additional AX read. Use `app.getWindows()` and `app.getWindow(windowId)` when the task needs a particular window or default selection is ambiguous; choose by the observed title and bounds.
|
||||
|
||||
`getWindow` emits its initial AX tree. An owned window without an AX surface returns an empty snapshot and remains available for screenshots and coordinate input. Window objects retain separate snapshots, so reading one window does not invalidate another. Re-reading the same scope invalidates old indices. An AX node removed during observation rejects the read with `STALE_SNAPSHOT` and publishes no partial tree; read again without repeating the preceding input. App-scoped screenshots and coordinates reuse the latest AX observation: its focused on-screen, nonzero-size window, otherwise the sole such window. They do not query current focus again; read `app.getAXState()` to follow a focus change. Explicit window objects retain their selected target. Keyboard methods stay on `app` because input follows the app’s current focus.
|
||||
|
||||
Screenshot data must use canonical Base64. Response byte limits apply before parsing, and attachment admission validates the raster and its configured capacity.
|
||||
|
||||
@@ -124,9 +124,11 @@ The plugin contributes one `cua` operation tool with method-specific parameters.
|
||||
##### Operation guidance (verbatim)
|
||||
|
||||
```markdown
|
||||
Use cua with method getCapabilities and params {} to inspect permissions and sessionLocked. If sessionLocked is true or an operation reports SESSION_LOCKED, stop UI attempts and ask the user to unlock macOS. A null sessionLocked means no usable console session was verified. After unlocking, read fresh state; screenshots from a locked desktop are not current observations. Use listApps and getApp to select an already running application. getApp requests user consent before application observations or input. If access is denied or revoked, stop using that application until the user explicitly grants access again. Never switch tools to bypass a refusal. Keep the returned appHandle; every app operation requires it. listWindows returns window IDs and point bounds. getState returns a complete AX tree and snapshotId. Use its element indices only with that snapshotId. Select a windowId explicitly for screenshots and coordinate actions. Coordinates are window-local macOS points, which may differ from screenshot pixels. A verified window can expose no AX elements. Its empty snapshot permits screenshots and coordinate input; element-index actions require an element from a fresh AX observation.
|
||||
Use cua with method getCapabilities and params {} to inspect permissions and sessionLocked. If sessionLocked is true or an operation reports SESSION_LOCKED, stop UI attempts and ask the user to unlock macOS. A null sessionLocked means no usable console session was verified. After unlocking, read fresh state; screenshots from a locked desktop are not current observations. Use listApps and getApp to select an already running application. getApp requests user consent before application observations or input. If access is denied or revoked, stop using that application until the user explicitly grants access again. Never switch tools to bypass a refusal. Keep the returned appHandle; every app operation requires it. listWindows returns window IDs and point bounds. getState returns a complete AX tree and snapshotId. Use its element indices only with that snapshotId. Raw screenshot and coordinate operations require a windowId. Coordinates are window-local macOS points, which may differ from screenshot pixels. A verified window can expose no AX elements. Its empty snapshot permits screenshots and coordinate input; element-index actions require an element from a fresh AX observation.
|
||||
|
||||
Initialize the object API once in persistent PTC. Reuse its variables in later calls; put temporary declarations inside { ... } blocks to avoid redeclaration errors. All object methods are async. Await getWindow before using the returned window object. getAXState returns text, not a structured state object.
|
||||
Initialize the object API once in persistent PTC. Reuse its variables in later calls; put temporary declarations inside { ... } blocks to avoid redeclaration errors. All object methods are async. getAXState returns text, not a structured state object.
|
||||
|
||||
Use app methods directly for single-window tasks; getApp already reads initial AX state. Call app.getWindows() and await app.getWindow(windowId) only to target a particular window or resolve ambiguous window selection. App screenshots and coordinate actions use the latest AX observation: its focused on-screen, nonzero-size window, otherwise the sole such window. They do not refresh focus automatically. Read app.getAXState() to follow changed focus.
|
||||
|
||||
Object API signatures (options default to { emit: true }):
|
||||
|
||||
|
||||
@@ -68,12 +68,12 @@ CUA 工具在读取 AX、窗口标题、截图、激活和输入之前检查应
|
||||
import { connectCua } from '@deepseek-ai/dsh-cua/bridge'
|
||||
const cua = connectCua(() => tools.cua, text => console.log(text))
|
||||
const app = await cua.getApp("com.apple.TextEdit")
|
||||
const windows = await app.getWindows()
|
||||
const window = await app.getWindow(windows[0].windowId)
|
||||
await window.getScreenshot()
|
||||
void await app.getScreenshot()
|
||||
```
|
||||
|
||||
根据标题和位置选择目标窗口。`getWindow` 会输出初始 AX 树。已确认归属但没有 AX 界面的窗口返回空快照，仍可截图和发送坐标输入。不同窗口分别保留快照，读取一个窗口不会使另一个窗口失效；重新读取同一范围会使旧索引失效。观察期间节点被移除时，该次读取返回 `STALE_SNAPSHOT`，不发布部分树；可以重新读取，但不要重复此前的输入。应用级截图和坐标操作优先使用最新观察中的焦点窗口，否则要求只有一个可见且非零尺寸窗口。显式窗口对象可以选择具体目标。键盘方法保留在 `app` 上，因为输入跟随应用的当前焦点。
|
||||
单窗口任务直接使用 app 方法。`getApp` 已读取初始 AX，截图不需要枚举窗口或额外读取 AX。任务需要指定窗口或默认选窗存在歧义时，再调用 `app.getWindows()` 和 `app.getWindow(windowId)`，根据观察到的标题与位置选择目标。
|
||||
|
||||
`getWindow` 会输出初始 AX 树。已确认归属但没有 AX 界面的窗口返回空快照，仍可截图和发送坐标输入。不同窗口分别保留快照，读取一个窗口不会使另一个窗口失效；重新读取同一范围会使旧索引失效。观察期间节点被移除时，该次读取返回 `STALE_SNAPSHOT`，不发布部分树；可以重新读取，但不要重复此前的输入。应用级截图和坐标操作复用最近一次 AX 观察：优先使用其中在屏幕上且尺寸非零的焦点窗口，否则要求只有一个符合条件的窗口。它们不会重新查询即时焦点；需要跟随焦点变化时，先读取 `app.getAXState()`。显式窗口对象保持选定的目标。键盘方法保留在 `app` 上，因为输入跟随应用的当前焦点。
|
||||
|
||||
截图数据必须使用规范 Base64。响应字节上限在解析前执行，附件接入校验图像数据及配置的容量。
|
||||
|
||||
@@ -124,9 +124,11 @@ AppKit sheet 可以通过其窗口 ID 单独选择，也会出现在所属文档
|
||||
##### 操作指导文本（原文）
|
||||
|
||||
```markdown
|
||||
Use cua with method getCapabilities and params {} to inspect permissions and sessionLocked. If sessionLocked is true or an operation reports SESSION_LOCKED, stop UI attempts and ask the user to unlock macOS. A null sessionLocked means no usable console session was verified. After unlocking, read fresh state; screenshots from a locked desktop are not current observations. Use listApps and getApp to select an already running application. getApp requests user consent before application observations or input. If access is denied or revoked, stop using that application until the user explicitly grants access again. Never switch tools to bypass a refusal. Keep the returned appHandle; every app operation requires it. listWindows returns window IDs and point bounds. getState returns a complete AX tree and snapshotId. Use its element indices only with that snapshotId. Select a windowId explicitly for screenshots and coordinate actions. Coordinates are window-local macOS points, which may differ from screenshot pixels. A verified window can expose no AX elements. Its empty snapshot permits screenshots and coordinate input; element-index actions require an element from a fresh AX observation.
|
||||
Use cua with method getCapabilities and params {} to inspect permissions and sessionLocked. If sessionLocked is true or an operation reports SESSION_LOCKED, stop UI attempts and ask the user to unlock macOS. A null sessionLocked means no usable console session was verified. After unlocking, read fresh state; screenshots from a locked desktop are not current observations. Use listApps and getApp to select an already running application. getApp requests user consent before application observations or input. If access is denied or revoked, stop using that application until the user explicitly grants access again. Never switch tools to bypass a refusal. Keep the returned appHandle; every app operation requires it. listWindows returns window IDs and point bounds. getState returns a complete AX tree and snapshotId. Use its element indices only with that snapshotId. Raw screenshot and coordinate operations require a windowId. Coordinates are window-local macOS points, which may differ from screenshot pixels. A verified window can expose no AX elements. Its empty snapshot permits screenshots and coordinate input; element-index actions require an element from a fresh AX observation.
|
||||
|
||||
Initialize the object API once in persistent PTC. Reuse its variables in later calls; put temporary declarations inside { ... } blocks to avoid redeclaration errors. All object methods are async. Await getWindow before using the returned window object. getAXState returns text, not a structured state object.
|
||||
Initialize the object API once in persistent PTC. Reuse its variables in later calls; put temporary declarations inside { ... } blocks to avoid redeclaration errors. All object methods are async. getAXState returns text, not a structured state object.
|
||||
|
||||
Use app methods directly for single-window tasks; getApp already reads initial AX state. Call app.getWindows() and await app.getWindow(windowId) only to target a particular window or resolve ambiguous window selection. App screenshots and coordinate actions use the latest AX observation: its focused on-screen, nonzero-size window, otherwise the sole such window. They do not refresh focus automatically. Read app.getAXState() to follow changed focus.
|
||||
|
||||
Object API signatures (options default to { emit: true }):
|
||||
|
||||
|
||||
@@ -80,9 +80,11 @@ export const Config = Schema.object({
|
||||
}) satisfies Schema<Record<string, unknown>, Config>
|
||||
|
||||
/** Instructions for raw operation calls and optional object-API bindings. */
|
||||
export const GUIDANCE = `Use cua with method getCapabilities and params {} to inspect permissions and sessionLocked. If sessionLocked is true or an operation reports SESSION_LOCKED, stop UI attempts and ask the user to unlock macOS. A null sessionLocked means no usable console session was verified. After unlocking, read fresh state; screenshots from a locked desktop are not current observations. Use listApps and getApp to select an already running application. getApp requests user consent before application observations or input. If access is denied or revoked, stop using that application until the user explicitly grants access again. Never switch tools to bypass a refusal. Keep the returned appHandle; every app operation requires it. listWindows returns window IDs and point bounds. getState returns a complete AX tree and snapshotId. Use its element indices only with that snapshotId. Select a windowId explicitly for screenshots and coordinate actions. Coordinates are window-local macOS points, which may differ from screenshot pixels. A verified window can expose no AX elements. Its empty snapshot permits screenshots and coordinate input; element-index actions require an element from a fresh AX observation.
|
||||
export const GUIDANCE = `Use cua with method getCapabilities and params {} to inspect permissions and sessionLocked. If sessionLocked is true or an operation reports SESSION_LOCKED, stop UI attempts and ask the user to unlock macOS. A null sessionLocked means no usable console session was verified. After unlocking, read fresh state; screenshots from a locked desktop are not current observations. Use listApps and getApp to select an already running application. getApp requests user consent before application observations or input. If access is denied or revoked, stop using that application until the user explicitly grants access again. Never switch tools to bypass a refusal. Keep the returned appHandle; every app operation requires it. listWindows returns window IDs and point bounds. getState returns a complete AX tree and snapshotId. Use its element indices only with that snapshotId. Raw screenshot and coordinate operations require a windowId. Coordinates are window-local macOS points, which may differ from screenshot pixels. A verified window can expose no AX elements. Its empty snapshot permits screenshots and coordinate input; element-index actions require an element from a fresh AX observation.
|
||||
|
||||
Initialize the object API once in persistent PTC. Reuse its variables in later calls; put temporary declarations inside { ... } blocks to avoid redeclaration errors. All object methods are async. Await getWindow before using the returned window object. getAXState returns text, not a structured state object.
|
||||
Initialize the object API once in persistent PTC. Reuse its variables in later calls; put temporary declarations inside { ... } blocks to avoid redeclaration errors. All object methods are async. getAXState returns text, not a structured state object.
|
||||
|
||||
Use app methods directly for single-window tasks; getApp already reads initial AX state. Call app.getWindows() and await app.getWindow(windowId) only to target a particular window or resolve ambiguous window selection. App screenshots and coordinate actions use the latest AX observation: its focused on-screen, nonzero-size window, otherwise the sole such window. They do not refresh focus automatically. Read app.getAXState() to follow changed focus.
|
||||
|
||||
Object API signatures (options default to { emit: true }):
|
||||
|
||||
|
||||
@@ -120,7 +120,12 @@ export interface CuaSurface {
|
||||
}
|
||||
/** One explicit window; keyboard operations remain app-scoped because they follow focus. */
|
||||
export interface CuaWindow extends CuaSurface { readonly windowId: number }
|
||||
/** Native app handle; implicit coordinates use the focused window or the sole visible window. */
|
||||
/**
|
||||
* Native app handle supporting direct observations and actions without window enumeration.
|
||||
* Screenshots and coordinates reuse the latest AX observation's focused on-screen,
|
||||
* nonzero-size window, or require exactly one such window. Read fresh AX to follow
|
||||
* focus changes; explicit window objects retain their selected target.
|
||||
*/
|
||||
export interface CuaApp extends CuaSurface {
|
||||
readonly pid: number
|
||||
readonly appHandle: CuaAppHandle
|
||||
|
||||
@@ -70,8 +70,11 @@ describe('application discovery', () => {
|
||||
const app = await f.api.getApp('Editor')
|
||||
expect(app.pid).toBe(17)
|
||||
expect(app.appHandle).toBe('app-17')
|
||||
expect(await app.getWindows()).toEqual([windowInfo(23)])
|
||||
expect(await app.getScreenshot()).toEqual(Uint8Array.from(Buffer.from(PNG, 'base64')))
|
||||
expect(f.dispatch.mock.calls.map(([operation]) => operation.method)).toEqual([
|
||||
'getCapabilities', 'listApps', 'getApp', 'getState', 'screenshot',
|
||||
])
|
||||
expect(await app.getWindows()).toEqual([windowInfo(23)])
|
||||
expect(f.sink.text).not.toHaveBeenCalled()
|
||||
expect(f.sink.image).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
Reference in new issue
Block a user