Merge remote-tracking branch 'origin/feat/cua-basic-onboarding' into feat/cua-permission-onboarding
# Conflicts: # .agents/notes/implemented/architecture/2026-09-21-experimental-capabilities-as-optional-bundles.i18n.yaml # .agents/notes/implemented/architecture/2026-09-21-experimental-capabilities-as-optional-bundles.md # .agents/notes/implemented/architecture/2026-09-21-experimental-capabilities-as-optional-bundles.zh.md # packages/experimental/client-ui-cua/src/client/PermissionPanel.tsx
This commit is contained in:
creatixchu committed
commit
e3b7123da1
22 files changed
+58
-55
No files matched your search
+2
-2
@@ -9,8 +9,8 @@
|
||||
en: 2ba6c43b20a3d347
|
||||
zh: 5647179ddaa38ad9
|
||||
/agent-note-ship-experimental-capabilities-as-optional-bundles/decision:
|
||||
en: 2d40970162451a14
|
||||
zh: 4e7801c3b5e0d7a3
|
||||
en: 8bfcb14ebbc4fd0e
|
||||
zh: af028b545fac806a
|
||||
/agent-note-ship-experimental-capabilities-as-optional-bundles/alternatives-considered:
|
||||
en: d6f40bf02df37ab7
|
||||
zh: e5b85ce9d41e8754
|
||||
|
||||
+2
@@ -20,6 +20,8 @@ Claude Code and Codex install on demand through the separate `ON_DEMAND_BUNDLES`
|
||||
|
||||
Computer Use is a shipped optional bundle in the experimental group. Its repository-owned native component is prepared only for enabled profiles, without an external Cua Driver runtime. It declares `dsh.bundle.experimental: true` for the experimental card label; existing experimental package names retain their labels.
|
||||
|
||||
The CUA bundle and client plugin use the explicit naming exceptions `@deepseek-ai/dsh-cua-bundle` and `@deepseek-ai/dsh-client-ui-cua`, alongside `@deepseek-ai/dsh-cua`. These names preserve profile references when the capability moves package groups; the experimental classification still controls isolation and promotion.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
**Ship every provider as an optional bundle.** Composes and displays correctly, but grows every install by provider runtimes for a capability most installations never switch on; voice input's `sherpa-onnx-node` is the one accepted precedent.
|
||||
|
||||
+2
@@ -20,6 +20,8 @@ Claude Code 与 Codex 通过独立的 `ON_DEMAND_BUNDLES` 目录按需安装，
|
||||
|
||||
Computer Use 是 experimental 包组内随安装提供的可选组合包。其仓库自有原生组件只为已启用的配置档准备，不引入外部 Cua Driver 运行时。它通过 `dsh.bundle.experimental: true` 显示实验性卡片标签；已有实验包名保留对应标签。
|
||||
|
||||
CUA 组合包和前端插件与 `@deepseek-ai/dsh-cua` 一起采用显式命名例外 `@deepseek-ai/dsh-cua-bundle`、`@deepseek-ai/dsh-client-ui-cua`，在能力移动分组时保留配置引用。依赖隔离和晋升仍由实验性分类约束。
|
||||
|
||||
## 考虑过的替代方案
|
||||
|
||||
**把所有提供方都做成可选 bundle。** 组合与展示都正确，但会为多数安装从不启用的能力把提供方运行时塞进每次安装；语音输入的 `sherpa-onnx-node` 是唯一被接受的先例。
|
||||
|
||||
+3
-3
@@ -1,13 +1,13 @@
|
||||
# Bilingual-pair consistency record for guide.md (docs/i18n/README.md): per heading
|
||||
# section, a hash of its English and Chinese blocks outside code blocks and generated regions.
|
||||
# After editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write docs/upgrade-guide/v0.2.1-alpha.2/cua-preset-scope/guide.md
|
||||
# pnpm run verify-translation-pairing --write docs/upgrade-guide/v0.2.1-rc.2/cua-preset-scope/guide.md
|
||||
/:
|
||||
en: 36d6b79b48c08dd5
|
||||
zh: 814caece159e9b0e
|
||||
/cua-defaults-to-ptc-session-scopes:
|
||||
en: 1d6eee8020ebed3b
|
||||
zh: d14279f527983c3c
|
||||
en: a53b6884785bdf56
|
||||
zh: 7d27fc9b029e0e3e
|
||||
/cua-defaults-to-ptc-session-scopes/change:
|
||||
en: dac96a29e3751249
|
||||
zh: d48cb030f43c07e4
|
||||
File renamed without changes.
File renamed without changes.
@@ -80,15 +80,16 @@ function PermissionDescription({ application, permission, loading, t }: Pick<Per
|
||||
loading: boolean
|
||||
}) {
|
||||
if (!loading && application === undefined) return <p>{t(permission === 'accessibility' ? 'accessibilityUse' : 'screenRecordingUse')}</p>
|
||||
const [before, after] = t(permission === 'accessibility' ? 'accessibilityPermission' : 'screenRecordingPermission').split('{application}')
|
||||
return <p>
|
||||
{t('permissionPrefix')}{' '}
|
||||
{before}
|
||||
<span className={css.applicationBadge}>
|
||||
{application === undefined ? <span className={css.appPlaceholder} aria-hidden /> : <>
|
||||
{application.icon !== undefined && <img src={application.icon} alt="" width={20} height={20} draggable={false} />}
|
||||
<span>{application.name}</span>
|
||||
</>}
|
||||
</span>{' '}
|
||||
{t(permission === 'accessibility' ? 'accessibilityAction' : 'screenRecordingAction')}
|
||||
</span>
|
||||
{after}
|
||||
</p>
|
||||
}
|
||||
|
||||
|
||||
@@ -17,9 +17,8 @@ export const en = {
|
||||
openSetup: 'Open settings', later: 'Later', close: 'Close',
|
||||
accessibility: 'Accessibility', screenRecording: 'Screen Recording',
|
||||
accessibilityUse: 'Read interface controls, click, and type.', screenRecordingUse: 'Capture the selected application window.',
|
||||
permissionPrefix: 'In System Settings, allow',
|
||||
accessibilityAction: 'to read interface controls, click, and type.',
|
||||
screenRecordingAction: 'to capture application windows.',
|
||||
accessibilityPermission: 'In System Settings, allow {application} to read interface controls, click, and type.',
|
||||
screenRecordingPermission: 'In System Settings, allow {application} to capture application windows.',
|
||||
permissionHint: 'Enable the application in System Settings. Follow the macOS prompt if it names another application.',
|
||||
permissionHintUnknown: 'Enable these permissions for the application named in the macOS prompt.',
|
||||
granted: 'Allowed', missing: 'Not allowed', unknown: 'Unknown', request: 'Request permission', settings: 'Open System Settings', retry: 'Retry',
|
||||
@@ -48,9 +47,8 @@ export const zh: Record<CuaPermissionLocaleKey, string> = {
|
||||
openSetup: '前往设置', later: '稍后', close: '关闭',
|
||||
accessibility: '辅助功能', screenRecording: '屏幕录制',
|
||||
accessibilityUse: '读取界面控件、点击和输入文字', screenRecordingUse: '截取所选应用的窗口',
|
||||
permissionPrefix: '在系统设置中允许',
|
||||
accessibilityAction: '读取界面、点击和输入',
|
||||
screenRecordingAction: '截取应用窗口',
|
||||
accessibilityPermission: '在系统设置中允许 {application} 读取界面、点击和输入。',
|
||||
screenRecordingPermission: '在系统设置中允许 {application} 截取应用窗口。',
|
||||
permissionHint: '在系统设置中开启对应应用；若 macOS 提示其他应用，以系统提示为准',
|
||||
permissionHintUnknown: '在系统设置中，为 macOS 提示的应用开启对应权限',
|
||||
granted: '已允许', missing: '未允许', unknown: '未确认', request: '申请权限', settings: '打开系统设置', retry: '重试',
|
||||
|
||||
@@ -229,13 +229,12 @@ it.each([en, zh])('shows three requirements with the target application beside e
|
||||
expect(within(requirements).getAllByRole('group')).toHaveLength(3)
|
||||
expect(screen.getByRole('region', { name: dictionary.applicationsTitle })).toBeTruthy()
|
||||
const accessibility = within(requirements).getByRole('group', { name: dictionary.accessibility })
|
||||
expect(accessibility.textContent).toContain(dictionary.permissionPrefix)
|
||||
expect(within(accessibility).getByText(application.name)).toBeTruthy()
|
||||
expect(accessibility.textContent).toContain(dictionary.accessibilityAction)
|
||||
expect(accessibility.textContent).toContain(dictionary.accessibilityPermission.replace('{application}', application.name))
|
||||
expect(accessibility.querySelector('img')?.getAttribute('src')).toBe(application.icon)
|
||||
const recording = within(requirements).getByRole('group', { name: dictionary.screenRecording })
|
||||
expect(within(recording).getByText(application.name)).toBeTruthy()
|
||||
expect(recording.textContent).toContain(dictionary.screenRecordingAction)
|
||||
expect(recording.textContent).toContain(dictionary.screenRecordingPermission.replace('{application}', application.name))
|
||||
expect(within(requirements).getByText(dictionary.permissionHint)).toBeTruthy()
|
||||
f.publish({ system: { ...f.state.getSnapshot().system, error: 'offline' } })
|
||||
expect(screen.queryByText(application.name)).toBeNull()
|
||||
|
||||
@@ -15,8 +15,8 @@
|
||||
en: 867d9ec544719acf
|
||||
zh: 92348599582dc33a
|
||||
/deepseek-ai-dsh-experimental-cua-macos/use-this-package:
|
||||
en: 5f070508770a1a48
|
||||
zh: 07d6e582c01b0ea3
|
||||
en: 107cc4d4e72e248d
|
||||
zh: 1aec124d4018d2a4
|
||||
/deepseek-ai-dsh-experimental-cua-macos/use-this-package/connect-an-external-execution-environment:
|
||||
en: e7909c9b43e44fbc
|
||||
zh: 98949877d4ea3a86
|
||||
|
||||
@@ -42,6 +42,7 @@ The dump command checks profile composition without a model request. For a real
|
||||
|
||||
| Field | Default | Meaning |
|
||||
| --- | --- | --- |
|
||||
| `agentPreset` | `ptc` | Register tools and guidance only for Agents bound to this preset; `false` registers globally. |
|
||||
| `allowedApps` | `[]` | Live profile-local application consent, keyed by executable path and verified signing requirement. |
|
||||
| `helperPath` | package-owned executable | Optional absolute native executable override. |
|
||||
| `buildTimeoutMs` | `300000` | Maximum native preparation time. |
|
||||
@@ -152,7 +153,7 @@ The plugin contributes one `cua` operation tool with method-specific parameters.
|
||||
##### Operation guidance (verbatim)
|
||||
|
||||
```markdown
|
||||
Use cua with method getCapabilities and params {} to inspect permissions and sessionLocked. If sessionLocked is true or an operation reports SESSION_LOCKED, stop UI attempts and ask the user to unlock macOS. A null sessionLocked means no usable console session was verified. After unlocking, read fresh state; screenshots from a locked desktop are not current observations. Use listApps or cua.getState to discover running applications, and findApps(query) to search installed applications by name, bundle ID or absolute path. getApp requests user consent before launching a stopped application in the background, or before observations and input on a running one. If access is denied or revoked, stop using that application until the user allows another request on the permission page. Never switch tools to bypass a refusal. Keep the returned appHandle; every app operation requires it. listWindows returns window IDs and point bounds. Raw getState returns AX text and snapshotId. Menu-bar entries are shown by default; includeMenus=true expands their contents. details=true adds identifiers, global-point bounds and complete action names. Both options also apply to getAXState. Each read replaces the indices for that app/window scope. Use its element indices only with that snapshotId. Raw screenshot and coordinate operations require a windowId. Coordinates are pixels in the latest returned screenshot, measured from its top-left. Set relative:true for integer thousandths from 0 to 1000 instead. The Host converts both modes to native window points; AX detail bounds remain global macOS points. A verified window can expose no AX elements. Its empty snapshot permits screenshots and coordinate input; element-index actions require an element from a fresh AX observation.
|
||||
Use cua with method getCapabilities and params {} to inspect permissions and sessionLocked. If sessionLocked is true or an operation reports SESSION_LOCKED, stop UI attempts and ask the user to unlock macOS. A null sessionLocked means no usable console session was verified. After unlocking, read fresh state; screenshots from a locked desktop are not current observations. Use listApps or cua.getState to discover running applications, and findApps(query) to search installed applications by name, bundle ID or absolute path. getApp requests user consent before launching a stopped application in the background, or before observations and input on a running one. If access is denied or revoked, stop using that application until the user explicitly permits another authorization request. Never switch tools to bypass a refusal. Keep the returned appHandle; every app operation requires it. listWindows returns window IDs and point bounds. Raw getState returns AX text and snapshotId. Menu-bar entries are shown by default; includeMenus=true expands their contents. details=true adds identifiers, global-point bounds and complete action names. Both options also apply to getAXState. Each read replaces the indices for that app/window scope. Use its element indices only with that snapshotId. Raw screenshot and coordinate operations require a windowId. Coordinates are pixels in the latest returned screenshot, measured from its top-left. Set relative:true for integer thousandths from 0 to 1000 instead. The Host converts both modes to native window points; AX detail bounds remain global macOS points. A verified window can expose no AX elements. Its empty snapshot permits screenshots and coordinate input; element-index actions require an element from a fresh AX observation.
|
||||
|
||||
Initialize the object API once in persistent PTC. Reuse its variables in later calls; put temporary declarations inside { ... } blocks to avoid redeclaration errors. All object methods are async. getAXState returns text, not a structured state object. Its text separates values and placeholders, and reports subroles, help, URLs, disabled/focused/selected/expanded state and UTF-16 selection ranges when available. Writability is true, false or unknown on relevant controls; omitted state fields do not assert false. selection=unavailable means the reported range exceeds the observed text. Compact output omits ordinary Press/Pick (click), ShowMenu (right click) and ScrollToVisible actions; details=true lists every available action. Use the displayed action name with performSecondaryAction; ambiguous aliases require the complete name from details=true. Truncation reports the reached provider limit. Narrow the window or increase that provider limit before reading again.
|
||||
|
||||
@@ -189,7 +190,7 @@ App and Window methods:
|
||||
|
||||
Raw operation tools use { method, params }, including params: {} for parameterless methods. getApp selects without activating the app. Keep applications in the background for observation and routine input. Do not activate or raise an app during initialization, before each action, or merely because another app is foreground. Prefer advertised AX actions. If a window is hidden or minimized, try app.show() or window.show() and read fresh state. show requests unhide and unminimize without activation, raising or switching Spaces; it chooses the explicit, focused, main or sole AX window. A null windowId means no window was selected for restoration. foregroundChanged compares the front app and top window before and after the request; null means unavailable, and delayed focus changes remain possible. If restoration is unsupported or the window remains unusable, including another Space or Stage Manager, explain the needed foreground step, use app.activate() or an advertised raise action, and read fresh state. Completing the task takes priority when foreground access is necessary. Avoid repeated activation when the user switches away; continue background-capable steps and report what requires foreground access. Applications may still change focus in response to input.
|
||||
|
||||
Await operations sequentially. After input, read fresh state to verify the effect. A successful action means dispatched, not verified completion. Cancellation may leave partial effects; never retry uncertain input without observing. STALE_APP requires getApp again; STALE_SNAPSHOT requires getState again. Absolute coordinate input requires a fresh screenshot of that window. Relative input can use a successful AX observation or screenshot. SCREENSHOT_REQUIRED means capture a fresh screenshot before using pixel coordinates. For click and hover, coordinate targets must be [x, y], for example hover([650, 400], {relative:true}). A bare number such as hover(650) is always an AX element index. The relative option applies only to coordinate targets, not AX indices. Window resizing since the latest observation rejects coordinate input; movement alone preserves window-local coordinates. WINDOW_CHANGED requires fresh state before any retry; use an advertised AX action or explicitly restore the window before screenshot or coordinate input, then observe again. PERMISSION_DENIED requires the user to complete the Computer Use permission setup for macOS Accessibility or Screen Recording. After granting access, observe fresh state before continuing; do not automatically replay prior input.
|
||||
Await operations sequentially. After input, read fresh state to verify the effect. A successful action means dispatched, not verified completion. Cancellation may leave partial effects; never retry uncertain input without observing. STALE_APP requires getApp again; STALE_SNAPSHOT requires getState again. Absolute coordinate input requires a fresh screenshot of that window. Relative input can use a successful AX observation or screenshot. SCREENSHOT_REQUIRED means capture a fresh screenshot before using pixel coordinates. For click and hover, coordinate targets must be [x, y], for example hover([650, 400], {relative:true}). A bare number such as hover(650) is always an AX element index. The relative option applies only to coordinate targets, not AX indices. Window resizing since the latest observation rejects coordinate input; movement alone preserves window-local coordinates. WINDOW_CHANGED requires fresh state before any retry; use an advertised AX action or explicitly restore the window before screenshot or coordinate input, then observe again. PERMISSION_DENIED requires the user to grant the required macOS system permissions for macOS Accessibility or Screen Recording. After granting access, observe fresh state before continuing; do not automatically replay prior input.
|
||||
|
||||
Hover first uses advertised AX alternate UI without moving the system cursor, preserving revealed descendants until the pointer target leaves their region. When unavailable, it moves the visible system cursor without pressing a button; that route requires the target point to be unobscured in the foreground app. FOREGROUND_REQUIRED requires explicit activation or raising and a fresh observation before retrying. Returning, AX reads and screenshots do not restore the previous point. Wait for the application's hover delay, observe, then hover into the revealed controls before clicking. Use successive hover calls for intermediate points. The result reports hoverRoute and systemPointerMoved. Other input or application changes can interrupt the interaction; app objects do not reserve applications between calls.
|
||||
|
||||
|
||||
@@ -42,6 +42,7 @@ pnpm dsh headless --patch packages/experimental/cua-macos/examples/macos.cordis.
|
||||
|
||||
| 字段 | 默认值 | 含义 |
|
||||
| --- | --- | --- |
|
||||
| `agentPreset` | `ptc` | 仅为绑定该 preset 的 Agent 注册工具和指引；`false` 表示全局注册。 |
|
||||
| `allowedApps` | `[]` | 当前配置档的动态应用授权，绑定可执行文件路径和已验证的代码签名要求。 |
|
||||
| `helperPath` | 包内可执行文件 | 可选的原生可执行文件绝对路径覆盖。 |
|
||||
| `buildTimeoutMs` | `300000` | 原生组件准备的最长时间。 |
|
||||
@@ -152,7 +153,7 @@ AppKit sheet 可以通过其窗口 ID 单独选择，也会出现在所属文档
|
||||
##### 操作指导文本（原文）
|
||||
|
||||
```markdown
|
||||
Use cua with method getCapabilities and params {} to inspect permissions and sessionLocked. If sessionLocked is true or an operation reports SESSION_LOCKED, stop UI attempts and ask the user to unlock macOS. A null sessionLocked means no usable console session was verified. After unlocking, read fresh state; screenshots from a locked desktop are not current observations. Use listApps or cua.getState to discover running applications, and findApps(query) to search installed applications by name, bundle ID or absolute path. getApp requests user consent before launching a stopped application in the background, or before observations and input on a running one. If access is denied or revoked, stop using that application until the user allows another request on the permission page. Never switch tools to bypass a refusal. Keep the returned appHandle; every app operation requires it. listWindows returns window IDs and point bounds. Raw getState returns AX text and snapshotId. Menu-bar entries are shown by default; includeMenus=true expands their contents. details=true adds identifiers, global-point bounds and complete action names. Both options also apply to getAXState. Each read replaces the indices for that app/window scope. Use its element indices only with that snapshotId. Raw screenshot and coordinate operations require a windowId. Coordinates are pixels in the latest returned screenshot, measured from its top-left. Set relative:true for integer thousandths from 0 to 1000 instead. The Host converts both modes to native window points; AX detail bounds remain global macOS points. A verified window can expose no AX elements. Its empty snapshot permits screenshots and coordinate input; element-index actions require an element from a fresh AX observation.
|
||||
Use cua with method getCapabilities and params {} to inspect permissions and sessionLocked. If sessionLocked is true or an operation reports SESSION_LOCKED, stop UI attempts and ask the user to unlock macOS. A null sessionLocked means no usable console session was verified. After unlocking, read fresh state; screenshots from a locked desktop are not current observations. Use listApps or cua.getState to discover running applications, and findApps(query) to search installed applications by name, bundle ID or absolute path. getApp requests user consent before launching a stopped application in the background, or before observations and input on a running one. If access is denied or revoked, stop using that application until the user explicitly permits another authorization request. Never switch tools to bypass a refusal. Keep the returned appHandle; every app operation requires it. listWindows returns window IDs and point bounds. Raw getState returns AX text and snapshotId. Menu-bar entries are shown by default; includeMenus=true expands their contents. details=true adds identifiers, global-point bounds and complete action names. Both options also apply to getAXState. Each read replaces the indices for that app/window scope. Use its element indices only with that snapshotId. Raw screenshot and coordinate operations require a windowId. Coordinates are pixels in the latest returned screenshot, measured from its top-left. Set relative:true for integer thousandths from 0 to 1000 instead. The Host converts both modes to native window points; AX detail bounds remain global macOS points. A verified window can expose no AX elements. Its empty snapshot permits screenshots and coordinate input; element-index actions require an element from a fresh AX observation.
|
||||
|
||||
Initialize the object API once in persistent PTC. Reuse its variables in later calls; put temporary declarations inside { ... } blocks to avoid redeclaration errors. All object methods are async. getAXState returns text, not a structured state object. Its text separates values and placeholders, and reports subroles, help, URLs, disabled/focused/selected/expanded state and UTF-16 selection ranges when available. Writability is true, false or unknown on relevant controls; omitted state fields do not assert false. selection=unavailable means the reported range exceeds the observed text. Compact output omits ordinary Press/Pick (click), ShowMenu (right click) and ScrollToVisible actions; details=true lists every available action. Use the displayed action name with performSecondaryAction; ambiguous aliases require the complete name from details=true. Truncation reports the reached provider limit. Narrow the window or increase that provider limit before reading again.
|
||||
|
||||
@@ -189,7 +190,7 @@ App and Window methods:
|
||||
|
||||
Raw operation tools use { method, params }, including params: {} for parameterless methods. getApp selects without activating the app. Keep applications in the background for observation and routine input. Do not activate or raise an app during initialization, before each action, or merely because another app is foreground. Prefer advertised AX actions. If a window is hidden or minimized, try app.show() or window.show() and read fresh state. show requests unhide and unminimize without activation, raising or switching Spaces; it chooses the explicit, focused, main or sole AX window. A null windowId means no window was selected for restoration. foregroundChanged compares the front app and top window before and after the request; null means unavailable, and delayed focus changes remain possible. If restoration is unsupported or the window remains unusable, including another Space or Stage Manager, explain the needed foreground step, use app.activate() or an advertised raise action, and read fresh state. Completing the task takes priority when foreground access is necessary. Avoid repeated activation when the user switches away; continue background-capable steps and report what requires foreground access. Applications may still change focus in response to input.
|
||||
|
||||
Await operations sequentially. After input, read fresh state to verify the effect. A successful action means dispatched, not verified completion. Cancellation may leave partial effects; never retry uncertain input without observing. STALE_APP requires getApp again; STALE_SNAPSHOT requires getState again. Absolute coordinate input requires a fresh screenshot of that window. Relative input can use a successful AX observation or screenshot. SCREENSHOT_REQUIRED means capture a fresh screenshot before using pixel coordinates. For click and hover, coordinate targets must be [x, y], for example hover([650, 400], {relative:true}). A bare number such as hover(650) is always an AX element index. The relative option applies only to coordinate targets, not AX indices. Window resizing since the latest observation rejects coordinate input; movement alone preserves window-local coordinates. WINDOW_CHANGED requires fresh state before any retry; use an advertised AX action or explicitly restore the window before screenshot or coordinate input, then observe again. PERMISSION_DENIED requires the user to complete the Computer Use permission setup for macOS Accessibility or Screen Recording. After granting access, observe fresh state before continuing; do not automatically replay prior input.
|
||||
Await operations sequentially. After input, read fresh state to verify the effect. A successful action means dispatched, not verified completion. Cancellation may leave partial effects; never retry uncertain input without observing. STALE_APP requires getApp again; STALE_SNAPSHOT requires getState again. Absolute coordinate input requires a fresh screenshot of that window. Relative input can use a successful AX observation or screenshot. SCREENSHOT_REQUIRED means capture a fresh screenshot before using pixel coordinates. For click and hover, coordinate targets must be [x, y], for example hover([650, 400], {relative:true}). A bare number such as hover(650) is always an AX element index. The relative option applies only to coordinate targets, not AX indices. Window resizing since the latest observation rejects coordinate input; movement alone preserves window-local coordinates. WINDOW_CHANGED requires fresh state before any retry; use an advertised AX action or explicitly restore the window before screenshot or coordinate input, then observe again. PERMISSION_DENIED requires the user to grant the required macOS system permissions for macOS Accessibility or Screen Recording. After granting access, observe fresh state before continuing; do not automatically replay prior input.
|
||||
|
||||
Hover first uses advertised AX alternate UI without moving the system cursor, preserving revealed descendants until the pointer target leaves their region. When unavailable, it moves the visible system cursor without pressing a button; that route requires the target point to be unobscured in the foreground app. FOREGROUND_REQUIRED requires explicit activation or raising and a fresh observation before retrying. Returning, AX reads and screenshots do not restore the previous point. Wait for the application's hover delay, observe, then hover into the revealed controls before clicking. Use successive hover calls for intermediate points. The result reports hoverRoute and systemPointerMoved. Other input or application changes can interrupt the interaction; app objects do not reserve applications between calls.
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
/** macOS operation guidance supplied with the shared CUA object API. */
|
||||
/** Instructions for raw operation calls and optional object-API bindings. */
|
||||
export const GUIDANCE = `Use cua with method getCapabilities and params {} to inspect permissions and sessionLocked. If sessionLocked is true or an operation reports SESSION_LOCKED, stop UI attempts and ask the user to unlock macOS. A null sessionLocked means no usable console session was verified. After unlocking, read fresh state; screenshots from a locked desktop are not current observations. Use listApps or cua.getState to discover running applications, and findApps(query) to search installed applications by name, bundle ID or absolute path. getApp requests user consent before launching a stopped application in the background, or before observations and input on a running one. If access is denied or revoked, stop using that application until the user allows another request on the permission page. Never switch tools to bypass a refusal. Keep the returned appHandle; every app operation requires it. listWindows returns window IDs and point bounds. Raw getState returns AX text and snapshotId. Menu-bar entries are shown by default; includeMenus=true expands their contents. details=true adds identifiers, global-point bounds and complete action names. Both options also apply to getAXState. Each read replaces the indices for that app/window scope. Use its element indices only with that snapshotId. Raw screenshot and coordinate operations require a windowId. Coordinates are pixels in the latest returned screenshot, measured from its top-left. Set relative:true for integer thousandths from 0 to 1000 instead. The Host converts both modes to native window points; AX detail bounds remain global macOS points. A verified window can expose no AX elements. Its empty snapshot permits screenshots and coordinate input; element-index actions require an element from a fresh AX observation.
|
||||
export const GUIDANCE = `Use cua with method getCapabilities and params {} to inspect permissions and sessionLocked. If sessionLocked is true or an operation reports SESSION_LOCKED, stop UI attempts and ask the user to unlock macOS. A null sessionLocked means no usable console session was verified. After unlocking, read fresh state; screenshots from a locked desktop are not current observations. Use listApps or cua.getState to discover running applications, and findApps(query) to search installed applications by name, bundle ID or absolute path. getApp requests user consent before launching a stopped application in the background, or before observations and input on a running one. If access is denied or revoked, stop using that application until the user explicitly permits another authorization request. Never switch tools to bypass a refusal. Keep the returned appHandle; every app operation requires it. listWindows returns window IDs and point bounds. Raw getState returns AX text and snapshotId. Menu-bar entries are shown by default; includeMenus=true expands their contents. details=true adds identifiers, global-point bounds and complete action names. Both options also apply to getAXState. Each read replaces the indices for that app/window scope. Use its element indices only with that snapshotId. Raw screenshot and coordinate operations require a windowId. Coordinates are pixels in the latest returned screenshot, measured from its top-left. Set relative:true for integer thousandths from 0 to 1000 instead. The Host converts both modes to native window points; AX detail bounds remain global macOS points. A verified window can expose no AX elements. Its empty snapshot permits screenshots and coordinate input; element-index actions require an element from a fresh AX observation.
|
||||
|
||||
Initialize the object API once in persistent PTC. Reuse its variables in later calls; put temporary declarations inside { ... } blocks to avoid redeclaration errors. All object methods are async. getAXState returns text, not a structured state object. Its text separates values and placeholders, and reports subroles, help, URLs, disabled/focused/selected/expanded state and UTF-16 selection ranges when available. Writability is true, false or unknown on relevant controls; omitted state fields do not assert false. selection=unavailable means the reported range exceeds the observed text. Compact output omits ordinary Press/Pick (click), ShowMenu (right click) and ScrollToVisible actions; details=true lists every available action. Use the displayed action name with performSecondaryAction; ambiguous aliases require the complete name from details=true. Truncation reports the reached provider limit. Narrow the window or increase that provider limit before reading again.
|
||||
|
||||
@@ -37,7 +37,7 @@ App and Window methods:
|
||||
|
||||
Raw operation tools use { method, params }, including params: {} for parameterless methods. getApp selects without activating the app. Keep applications in the background for observation and routine input. Do not activate or raise an app during initialization, before each action, or merely because another app is foreground. Prefer advertised AX actions. If a window is hidden or minimized, try app.show() or window.show() and read fresh state. show requests unhide and unminimize without activation, raising or switching Spaces; it chooses the explicit, focused, main or sole AX window. A null windowId means no window was selected for restoration. foregroundChanged compares the front app and top window before and after the request; null means unavailable, and delayed focus changes remain possible. If restoration is unsupported or the window remains unusable, including another Space or Stage Manager, explain the needed foreground step, use app.activate() or an advertised raise action, and read fresh state. Completing the task takes priority when foreground access is necessary. Avoid repeated activation when the user switches away; continue background-capable steps and report what requires foreground access. Applications may still change focus in response to input.
|
||||
|
||||
Await operations sequentially. After input, read fresh state to verify the effect. A successful action means dispatched, not verified completion. Cancellation may leave partial effects; never retry uncertain input without observing. STALE_APP requires getApp again; STALE_SNAPSHOT requires getState again. Absolute coordinate input requires a fresh screenshot of that window. Relative input can use a successful AX observation or screenshot. SCREENSHOT_REQUIRED means capture a fresh screenshot before using pixel coordinates. For click and hover, coordinate targets must be [x, y], for example hover([650, 400], {relative:true}). A bare number such as hover(650) is always an AX element index. The relative option applies only to coordinate targets, not AX indices. Window resizing since the latest observation rejects coordinate input; movement alone preserves window-local coordinates. WINDOW_CHANGED requires fresh state before any retry; use an advertised AX action or explicitly restore the window before screenshot or coordinate input, then observe again. PERMISSION_DENIED requires the user to complete the Computer Use permission setup for macOS Accessibility or Screen Recording. After granting access, observe fresh state before continuing; do not automatically replay prior input.
|
||||
Await operations sequentially. After input, read fresh state to verify the effect. A successful action means dispatched, not verified completion. Cancellation may leave partial effects; never retry uncertain input without observing. STALE_APP requires getApp again; STALE_SNAPSHOT requires getState again. Absolute coordinate input requires a fresh screenshot of that window. Relative input can use a successful AX observation or screenshot. SCREENSHOT_REQUIRED means capture a fresh screenshot before using pixel coordinates. For click and hover, coordinate targets must be [x, y], for example hover([650, 400], {relative:true}). A bare number such as hover(650) is always an AX element index. The relative option applies only to coordinate targets, not AX indices. Window resizing since the latest observation rejects coordinate input; movement alone preserves window-local coordinates. WINDOW_CHANGED requires fresh state before any retry; use an advertised AX action or explicitly restore the window before screenshot or coordinate input, then observe again. PERMISSION_DENIED requires the user to grant the required macOS system permissions for macOS Accessibility or Screen Recording. After granting access, observe fresh state before continuing; do not automatically replay prior input.
|
||||
|
||||
Hover first uses advertised AX alternate UI without moving the system cursor, preserving revealed descendants until the pointer target leaves their region. When unavailable, it moves the visible system cursor without pressing a button; that route requires the target point to be unobscured in the foreground app. FOREGROUND_REQUIRED requires explicit activation or raising and a fresh observation before retrying. Returning, AX reads and screenshots do not restore the previous point. Wait for the application's hover delay, observe, then hover into the revealed controls before clicking. Use successive hover calls for intermediate points. The result reports hoverRoute and systemPointerMoved. Other input or application changes can interrupt the interaction; app objects do not reserve applications between calls.
|
||||
|
||||
|
||||
@@ -18,12 +18,7 @@ describe('native protocol generation', () => {
|
||||
{ method: 'paste', params: { appHandle: 'app', text: 'text', pasteReadTimeoutMs: 1000 } },
|
||||
{ method: 'findApps', params: { query: 'Editor', maxAppMatches: 1 } },
|
||||
{ method: 'launchApp', params: { appHandle: 'pending' } },
|
||||
{ method: 'permissionStatus', params: {} },
|
||||
{ method: 'requestPermission', params: { permission: 'accessibility' } },
|
||||
{ method: 'openPermissionSettings', params: { permission: 'screenRecording' } },
|
||||
{ method: 'permissionStatus', params: {} },
|
||||
{ method: 'requestPermission', params: { permission: 'accessibility' } },
|
||||
{ method: 'openPermissionSettings', params: { permission: 'screenRecording' } },
|
||||
{ method: 'applicationIcons', params: { paths: ['/Applications/Fixture.app'] } },
|
||||
{ method: 'permissionStatus', params: {} },
|
||||
{ method: 'requestPermission', params: { permission: 'accessibility' } },
|
||||
{ method: 'openPermissionSettings', params: { permission: 'screenRecording' } },
|
||||
|
||||
@@ -18,8 +18,8 @@
|
||||
en: 577e1b18dd220c73
|
||||
zh: bde0692c6695d749
|
||||
/deepseek-ai-dsh-cua/understand-the-implementation:
|
||||
en: 751fe580f79dfaae
|
||||
zh: 3811db65f634ef5a
|
||||
en: 5f8f14fda4d69173
|
||||
zh: 5493e3ba74ffb7e0
|
||||
/deepseek-ai-dsh-cua/further-exploration:
|
||||
en: b8b0aece886fadaa
|
||||
zh: ecd8c88d601f2179
|
||||
@@ -39,8 +39,8 @@
|
||||
en: 2d5df52dafa1d940
|
||||
zh: 06e815759b9150c3
|
||||
/deepseek-ai-dsh-cua/known-limitations-and-deferred-work:
|
||||
en: da6aefcb279aea83
|
||||
zh: b45b7bb300b9f792
|
||||
en: ae98f1b2718b3b94
|
||||
zh: e6bf4e96ae08e669
|
||||
/deepseek-ai-dsh-cua/known-limitations-and-deferred-work/dev-note:
|
||||
en: 0a1253c6f7b5e8b9
|
||||
zh: 9f56528562c7c914
|
||||
@@ -51,7 +51,7 @@ Actions return `dispatched:true, verified:false`; read fresh state to verify the
|
||||
|
||||
The provider calls [installCua](src/index.ts) with its prepared helper, connection factory, and guidance. [NativeClient](src/native-client.ts) owns request correlation, byte limits, cancellation acknowledgement, and failure settlement. The platform connector owns startup and connection cleanup. The shared executor retains one connection per Agent and joins in-flight image writes during disposal.
|
||||
|
||||
`agentPreset` defaults to `ptc`, restricting tools and guidance to Agents bound to that preset. The provider uses the existing `agent/created` and `agent-preset/selected` events and Agent-owned scopes, including Agents already present when it loads. Provider unload and Agent disposal release those registrations. Set `agentPreset: false` for global registration in standalone compositions without presets. The Host continues to own permissions, native connections and cancellation; preset definitions remain unchanged.
|
||||
`agentPreset` defaults to `ptc`, restricting tools and guidance to Agents bound to that preset. The provider uses the existing `agent/created` and `agent-preset/selected` events and Agent-owned scopes, including Agents already present when it loads. Provider unload and Agent disposal release those registrations. Preset definitions may register after the provider; a preset name filters bound Agents and supplies no tools until a matching Agent exists. Set `agentPreset: false` for global registration in standalone compositions without presets. The Host continues to own permissions, native connections and cancellation; preset definitions remain unchanged.
|
||||
|
||||
[The operation schema](src/protocol.ts) is the authority for model parameters and separately declared Host-only fields. Native implementations enforce target identity, window geometry, resource limits, and operating-system access at execution. No invariant companion is published because this library has no separately projected registry or durable state.
|
||||
|
||||
@@ -88,6 +88,7 @@ Results append to history. Retaining objects does not rewrite previous messages;
|
||||
|
||||
<a id="known-limitations-and-deferred-work"></a>
|
||||
|
||||
- In-memory session decisions have no capacity limit or eviction. They remain until the provider unloads, so a long-running Host retains entries for every session/application pair that requested access.
|
||||
- Native execution currently uses the macOS provider. Windows support is planned as a separate provider package with a C# helper, retaining the shared app/window API, authorization, snapshot validity, and action-result semantics. Its implementation must document system-specific limitations and distinguish cross-compilation evidence from Windows desktop validation.
|
||||
- Persistent variables do not reserve an application across a sequence of operations. Other Agents and the user can change its state. Hover and observations do not restore the previous pointer position; [continuous interaction](../../../.agents/notes/implemented/architecture/2026-10-13-cua-continuous-hover.md) uses no cross-call ownership.
|
||||
- The package stays in `experimental` until Windows support and platform validation permit promotion to the `computer-use` group.
|
||||
|
||||
@@ -51,7 +51,7 @@ kind: "package-library"
|
||||
|
||||
提供方将准备好的 helper、连接工厂和指引传给 [installCua](src/index.ts)。[NativeClient](src/native-client.ts) 负责请求关联、字节上限、取消确认和失败处理。平台连接器负责启动和连接清理。共享执行器为每个 Agent 保留一个连接，并在销毁时等待进行中的图像写入。
|
||||
|
||||
`agentPreset` 默认为 `ptc`，将工具和指引限制到绑定该 preset 的 Agent。提供方使用现有 `agent/created`、`agent-preset/selected` 事件及 Agent 拥有的作用域，也处理启用前已存在的 Agent。提供方卸载和 Agent 销毁都会释放这些注册。不使用 preset 的独立组合可设置 `agentPreset: false` 来全局注册。Host 仍拥有权限、本机连接和取消操作，preset 定义保持不变。
|
||||
`agentPreset` 默认为 `ptc`，将工具和指引限制到绑定该 preset 的 Agent。提供方使用现有 `agent/created`、`agent-preset/selected` 事件及 Agent 拥有的作用域，也处理启用前已存在的 Agent。提供方卸载和 Agent 销毁都会释放这些注册。preset 定义可以晚于提供方注册；名称用于筛选已绑定的 Agent，在匹配的 Agent 出现前不提供工具。不使用 preset 的独立组合可设置 `agentPreset: false` 来全局注册。Host 仍拥有权限、本机连接和取消操作，preset 定义保持不变。
|
||||
|
||||
[操作 schema](src/protocol.ts) 定义模型参数，并单独声明 Host 专用字段。原生实现在执行时检查目标身份、窗口几何、资源上限和操作系统访问条件。本库没有独立投影的注册表或持久状态，因此不发布 invariant 模块。
|
||||
|
||||
@@ -88,6 +88,7 @@ kind: "package-library"
|
||||
|
||||
<a id="known-limitations-and-deferred-work"></a>
|
||||
|
||||
- 会话授权决策保存在内存中，没有容量上限或淘汰机制，直到提供方卸载才清空。长期运行的 Host 会保留每个曾请求访问的会话与应用组合。
|
||||
- 当前原生执行使用 macOS 提供方。Windows 支持计划使用独立的提供方包和 C# helper，保留共享应用与窗口 API、授权、快照有效性和动作结果语义。实现须说明系统特有的限制，并区分交叉编译证据和 Windows 桌面验证。
|
||||
- 持久变量不会在一组连续操作期间独占应用。其他 Agent 和用户可能改变其状态。 悬停和观察不恢复先前的鼠标位置；[连续交互](../../../.agents/notes/implemented/architecture/2026-10-13-cua-continuous-hover.zh.md)不分配跨调用所有权。
|
||||
- 本包保留在 `experimental`，Windows 支持和平台验证完成后可迁至 `computer-use` 组。
|
||||
|
||||
@@ -23,7 +23,7 @@ const provider = {
|
||||
},
|
||||
}
|
||||
|
||||
async function setup() {
|
||||
async function setup(presetIds = ['standard', 'ptc']) {
|
||||
const ctx = new Context()
|
||||
onTestFinished(() => ctx.fiber.dispose())
|
||||
ctx.baseUrl = import.meta.url
|
||||
@@ -33,7 +33,7 @@ async function setup() {
|
||||
await ctx.plugin(ComputerUseRegistry)
|
||||
await ctx.plugin(Subprocess, [])
|
||||
await ctx.plugin(AgentPresets, { default: 'standard' })
|
||||
for (const id of ['standard', 'ptc']) {
|
||||
for (const id of presetIds) {
|
||||
await ctx.plugin({ inject: ['agentPresets'], async* apply(owner: Context) {
|
||||
yield await owner.agentPresets.register({ id, plugins: [] })
|
||||
} })
|
||||
@@ -93,3 +93,14 @@ it('ignores unloaded Session selections and reports a failed live scope update',
|
||||
await ctx.agentPresets.select(handle.agent, 'standard')
|
||||
await expect.poll(() => ctx.tools.get('cua', handle.agent)).toBeUndefined()
|
||||
})
|
||||
|
||||
it('accepts a matching preset registered after the CUA provider', async () => {
|
||||
const { ctx, create } = await setup([])
|
||||
await ctx.plugin(provider, {})
|
||||
expect(ctx.tools.get('cua')).toBeUndefined()
|
||||
await ctx.plugin({ inject: ['agentPresets'], async* apply(owner: Context) {
|
||||
yield await owner.agentPresets.register({ id: 'ptc', plugins: [] })
|
||||
} })
|
||||
const handle = await create('late-preset', 'ptc')
|
||||
expect(ctx.tools.get('cua', handle.agent)).toBeDefined()
|
||||
})
|
||||
@@ -1,13 +1,13 @@
|
||||
/** Interpreted retired content must fail before migration publication or recoverable native-tail suppression. */
|
||||
import { Context } from '@deepseek-ai/cordis'
|
||||
import { SESSION_FORMAT_VERSION, SessionId } from '@deepseek-ai/dsh-session'
|
||||
import { SessionId } from '@deepseek-ai/dsh-session'
|
||||
import { SessionFormatUnsupportedError, SessionPersistenceCorruptionError } from '@deepseek-ai/dsh-session-persistence'
|
||||
import * as JsonlSessionPersistence from '@deepseek-ai/dsh-session-persistence-jsonl'
|
||||
import { mkdir, mkdtemp, readFile, readdir, rm, writeFile } from 'node:fs/promises'
|
||||
import { tmpdir } from 'node:os'
|
||||
import { basename, dirname, join } from 'node:path'
|
||||
import { describe, expect, it, onTestFinished } from 'vitest'
|
||||
import { generationLogPath, SessionLogScanner } from '../src/format.ts'
|
||||
import { generationLogPath } from '../src/format.ts'
|
||||
import { compressZstdFrame } from '../src/zstd.ts'
|
||||
|
||||
const id = SessionId('retired-content-admission')
|
||||
@@ -55,12 +55,3 @@ describe.each(modes)('retired content in JSONL ($compression, $access)', ({ comp
|
||||
expect(await readFile(path)).toEqual(bytes)
|
||||
})
|
||||
})
|
||||
|
||||
it.each(['strict', 'recoverable'] as const)('classifies retired native PTC tags as unsupported during %s scanning', (recovery) => {
|
||||
const header = { type: 'session', version: SESSION_FORMAT_VERSION, id, createdAt: 1, isSeeded: false, delegationDepth: 0 }
|
||||
for (const type of ['tool/code-dispatch-start', 'tool/code-dispatch']) {
|
||||
const scanner = new SessionLogScanner(Buffer.from(JSON.stringify(header) + '\n'), recovery)
|
||||
const row = { type, seq: 0, time: 1, data: {} }
|
||||
expect(() => { scanner.write(Buffer.from(JSON.stringify(row) + '\n')) }).toThrow(SessionFormatUnsupportedError)
|
||||
}
|
||||
})
|
||||
@@ -1,6 +1,6 @@
|
||||
/** Experimental package identities, publication policy, and retained npm names. */
|
||||
|
||||
/** Existing npm names retained by capabilities now housed under experimental/. */
|
||||
/** Explicit npm identities retained across experimental classification and package-role moves. */
|
||||
export const EXPERIMENTAL_PACKAGE_NAME_EXCEPTIONS: Readonly<Record<string, string>> = {
|
||||
'packages/experimental/cua': '@deepseek-ai/dsh-cua',
|
||||
'packages/experimental/cua-bundle': '@deepseek-ai/dsh-cua-bundle',
|
||||
|
||||
@@ -29,7 +29,7 @@ Use the workflow tool ONLY when the user explicitly asks for a workflow or for l
|
||||
|
||||
Start independent delegations with `subagent` or `subagent_fork` together in one assistant message and continue useful work while they run.
|
||||
|
||||
Use cua with method getCapabilities and params {} to inspect permissions and sessionLocked. If sessionLocked is true or an operation reports SESSION_LOCKED, stop UI attempts and ask the user to unlock macOS. A null sessionLocked means no usable console session was verified. After unlocking, read fresh state; screenshots from a locked desktop are not current observations. Use listApps or cua.getState to discover running applications, and findApps(query) to search installed applications by name, bundle ID or absolute path. getApp requests user consent before launching a stopped application in the background, or before observations and input on a running one. If access is denied or revoked, stop using that application until the user allows another request on the permission page. Never switch tools to bypass a refusal. Keep the returned appHandle; every app operation requires it. listWindows returns window IDs and point bounds. Raw getState returns AX text and snapshotId. Menu-bar entries are shown by default; includeMenus=true expands their contents. details=true adds identifiers, global-point bounds and complete action names. Both options also apply to getAXState. Each read replaces the indices for that app/window scope. Use its element indices only with that snapshotId. Raw screenshot and coordinate operations require a windowId. Coordinates are pixels in the latest returned screenshot, measured from its top-left. Set relative:true for integer thousandths from 0 to 1000 instead. The Host converts both modes to native window points; AX detail bounds remain global macOS points. A verified window can expose no AX elements. Its empty snapshot permits screenshots and coordinate input; element-index actions require an element from a fresh AX observation.
|
||||
Use cua with method getCapabilities and params {} to inspect permissions and sessionLocked. If sessionLocked is true or an operation reports SESSION_LOCKED, stop UI attempts and ask the user to unlock macOS. A null sessionLocked means no usable console session was verified. After unlocking, read fresh state; screenshots from a locked desktop are not current observations. Use listApps or cua.getState to discover running applications, and findApps(query) to search installed applications by name, bundle ID or absolute path. getApp requests user consent before launching a stopped application in the background, or before observations and input on a running one. If access is denied or revoked, stop using that application until the user explicitly permits another authorization request. Never switch tools to bypass a refusal. Keep the returned appHandle; every app operation requires it. listWindows returns window IDs and point bounds. Raw getState returns AX text and snapshotId. Menu-bar entries are shown by default; includeMenus=true expands their contents. details=true adds identifiers, global-point bounds and complete action names. Both options also apply to getAXState. Each read replaces the indices for that app/window scope. Use its element indices only with that snapshotId. Raw screenshot and coordinate operations require a windowId. Coordinates are pixels in the latest returned screenshot, measured from its top-left. Set relative:true for integer thousandths from 0 to 1000 instead. The Host converts both modes to native window points; AX detail bounds remain global macOS points. A verified window can expose no AX elements. Its empty snapshot permits screenshots and coordinate input; element-index actions require an element from a fresh AX observation.
|
||||
|
||||
Initialize the object API once in persistent PTC. Reuse its variables in later calls; put temporary declarations inside { ... } blocks to avoid redeclaration errors. All object methods are async. getAXState returns text, not a structured state object. Its text separates values and placeholders, and reports subroles, help, URLs, disabled/focused/selected/expanded state and UTF-16 selection ranges when available. Writability is true, false or unknown on relevant controls; omitted state fields do not assert false. selection=unavailable means the reported range exceeds the observed text. Compact output omits ordinary Press/Pick (click), ShowMenu (right click) and ScrollToVisible actions; details=true lists every available action. Use the displayed action name with performSecondaryAction; ambiguous aliases require the complete name from details=true. Truncation reports the reached provider limit. Narrow the window or increase that provider limit before reading again.
|
||||
|
||||
@@ -66,7 +66,7 @@ App and Window methods:
|
||||
|
||||
Raw operation tools use { method, params }, including params: {} for parameterless methods. getApp selects without activating the app. Keep applications in the background for observation and routine input. Do not activate or raise an app during initialization, before each action, or merely because another app is foreground. Prefer advertised AX actions. If a window is hidden or minimized, try app.show() or window.show() and read fresh state. show requests unhide and unminimize without activation, raising or switching Spaces; it chooses the explicit, focused, main or sole AX window. A null windowId means no window was selected for restoration. foregroundChanged compares the front app and top window before and after the request; null means unavailable, and delayed focus changes remain possible. If restoration is unsupported or the window remains unusable, including another Space or Stage Manager, explain the needed foreground step, use app.activate() or an advertised raise action, and read fresh state. Completing the task takes priority when foreground access is necessary. Avoid repeated activation when the user switches away; continue background-capable steps and report what requires foreground access. Applications may still change focus in response to input.
|
||||
|
||||
Await operations sequentially. After input, read fresh state to verify the effect. A successful action means dispatched, not verified completion. Cancellation may leave partial effects; never retry uncertain input without observing. STALE_APP requires getApp again; STALE_SNAPSHOT requires getState again. Absolute coordinate input requires a fresh screenshot of that window. Relative input can use a successful AX observation or screenshot. SCREENSHOT_REQUIRED means capture a fresh screenshot before using pixel coordinates. For click and hover, coordinate targets must be [x, y], for example hover([650, 400], {relative:true}). A bare number such as hover(650) is always an AX element index. The relative option applies only to coordinate targets, not AX indices. Window resizing since the latest observation rejects coordinate input; movement alone preserves window-local coordinates. WINDOW_CHANGED requires fresh state before any retry; use an advertised AX action or explicitly restore the window before screenshot or coordinate input, then observe again. PERMISSION_DENIED requires the user to complete the Computer Use permission setup for macOS Accessibility or Screen Recording. After granting access, observe fresh state before continuing; do not automatically replay prior input.
|
||||
Await operations sequentially. After input, read fresh state to verify the effect. A successful action means dispatched, not verified completion. Cancellation may leave partial effects; never retry uncertain input without observing. STALE_APP requires getApp again; STALE_SNAPSHOT requires getState again. Absolute coordinate input requires a fresh screenshot of that window. Relative input can use a successful AX observation or screenshot. SCREENSHOT_REQUIRED means capture a fresh screenshot before using pixel coordinates. For click and hover, coordinate targets must be [x, y], for example hover([650, 400], {relative:true}). A bare number such as hover(650) is always an AX element index. The relative option applies only to coordinate targets, not AX indices. Window resizing since the latest observation rejects coordinate input; movement alone preserves window-local coordinates. WINDOW_CHANGED requires fresh state before any retry; use an advertised AX action or explicitly restore the window before screenshot or coordinate input, then observe again. PERMISSION_DENIED requires the user to grant the required macOS system permissions for macOS Accessibility or Screen Recording. After granting access, observe fresh state before continuing; do not automatically replay prior input.
|
||||
|
||||
Hover first uses advertised AX alternate UI without moving the system cursor, preserving revealed descendants until the pointer target leaves their region. When unavailable, it moves the visible system cursor without pressing a button; that route requires the target point to be unobscured in the foreground app. FOREGROUND_REQUIRED requires explicit activation or raising and a fresh observation before retrying. Returning, AX reads and screenshots do not restore the previous point. Wait for the application's hover delay, observe, then hover into the revealed controls before clicking. Use successive hover calls for intermediate points. The result reports hoverRoute and systemPointerMoved. Other input or application changes can interrupt the interaction; app objects do not reserve applications between calls.
|
||||
|
||||
|
||||
@@ -31,7 +31,7 @@ Use the workflow tool ONLY when the user explicitly asks for a workflow or for l
|
||||
|
||||
Start independent delegations with `subagent` or `subagent_fork` together in one assistant message and continue useful work while they run.
|
||||
|
||||
Use cua with method getCapabilities and params {} to inspect permissions and sessionLocked. If sessionLocked is true or an operation reports SESSION_LOCKED, stop UI attempts and ask the user to unlock macOS. A null sessionLocked means no usable console session was verified. After unlocking, read fresh state; screenshots from a locked desktop are not current observations. Use listApps or cua.getState to discover running applications, and findApps(query) to search installed applications by name, bundle ID or absolute path. getApp requests user consent before launching a stopped application in the background, or before observations and input on a running one. If access is denied or revoked, stop using that application until the user allows another request on the permission page. Never switch tools to bypass a refusal. Keep the returned appHandle; every app operation requires it. listWindows returns window IDs and point bounds. Raw getState returns AX text and snapshotId. Menu-bar entries are shown by default; includeMenus=true expands their contents. details=true adds identifiers, global-point bounds and complete action names. Both options also apply to getAXState. Each read replaces the indices for that app/window scope. Use its element indices only with that snapshotId. Raw screenshot and coordinate operations require a windowId. Coordinates are pixels in the latest returned screenshot, measured from its top-left. Set relative:true for integer thousandths from 0 to 1000 instead. The Host converts both modes to native window points; AX detail bounds remain global macOS points. A verified window can expose no AX elements. Its empty snapshot permits screenshots and coordinate input; element-index actions require an element from a fresh AX observation.
|
||||
Use cua with method getCapabilities and params {} to inspect permissions and sessionLocked. If sessionLocked is true or an operation reports SESSION_LOCKED, stop UI attempts and ask the user to unlock macOS. A null sessionLocked means no usable console session was verified. After unlocking, read fresh state; screenshots from a locked desktop are not current observations. Use listApps or cua.getState to discover running applications, and findApps(query) to search installed applications by name, bundle ID or absolute path. getApp requests user consent before launching a stopped application in the background, or before observations and input on a running one. If access is denied or revoked, stop using that application until the user explicitly permits another authorization request. Never switch tools to bypass a refusal. Keep the returned appHandle; every app operation requires it. listWindows returns window IDs and point bounds. Raw getState returns AX text and snapshotId. Menu-bar entries are shown by default; includeMenus=true expands their contents. details=true adds identifiers, global-point bounds and complete action names. Both options also apply to getAXState. Each read replaces the indices for that app/window scope. Use its element indices only with that snapshotId. Raw screenshot and coordinate operations require a windowId. Coordinates are pixels in the latest returned screenshot, measured from its top-left. Set relative:true for integer thousandths from 0 to 1000 instead. The Host converts both modes to native window points; AX detail bounds remain global macOS points. A verified window can expose no AX elements. Its empty snapshot permits screenshots and coordinate input; element-index actions require an element from a fresh AX observation.
|
||||
|
||||
Initialize the object API once in persistent PTC. Reuse its variables in later calls; put temporary declarations inside { ... } blocks to avoid redeclaration errors. All object methods are async. getAXState returns text, not a structured state object. Its text separates values and placeholders, and reports subroles, help, URLs, disabled/focused/selected/expanded state and UTF-16 selection ranges when available. Writability is true, false or unknown on relevant controls; omitted state fields do not assert false. selection=unavailable means the reported range exceeds the observed text. Compact output omits ordinary Press/Pick (click), ShowMenu (right click) and ScrollToVisible actions; details=true lists every available action. Use the displayed action name with performSecondaryAction; ambiguous aliases require the complete name from details=true. Truncation reports the reached provider limit. Narrow the window or increase that provider limit before reading again.
|
||||
|
||||
@@ -68,7 +68,7 @@ App and Window methods:
|
||||
|
||||
Raw operation tools use { method, params }, including params: {} for parameterless methods. getApp selects without activating the app. Keep applications in the background for observation and routine input. Do not activate or raise an app during initialization, before each action, or merely because another app is foreground. Prefer advertised AX actions. If a window is hidden or minimized, try app.show() or window.show() and read fresh state. show requests unhide and unminimize without activation, raising or switching Spaces; it chooses the explicit, focused, main or sole AX window. A null windowId means no window was selected for restoration. foregroundChanged compares the front app and top window before and after the request; null means unavailable, and delayed focus changes remain possible. If restoration is unsupported or the window remains unusable, including another Space or Stage Manager, explain the needed foreground step, use app.activate() or an advertised raise action, and read fresh state. Completing the task takes priority when foreground access is necessary. Avoid repeated activation when the user switches away; continue background-capable steps and report what requires foreground access. Applications may still change focus in response to input.
|
||||
|
||||
Await operations sequentially. After input, read fresh state to verify the effect. A successful action means dispatched, not verified completion. Cancellation may leave partial effects; never retry uncertain input without observing. STALE_APP requires getApp again; STALE_SNAPSHOT requires getState again. Absolute coordinate input requires a fresh screenshot of that window. Relative input can use a successful AX observation or screenshot. SCREENSHOT_REQUIRED means capture a fresh screenshot before using pixel coordinates. For click and hover, coordinate targets must be [x, y], for example hover([650, 400], {relative:true}). A bare number such as hover(650) is always an AX element index. The relative option applies only to coordinate targets, not AX indices. Window resizing since the latest observation rejects coordinate input; movement alone preserves window-local coordinates. WINDOW_CHANGED requires fresh state before any retry; use an advertised AX action or explicitly restore the window before screenshot or coordinate input, then observe again. PERMISSION_DENIED requires the user to complete the Computer Use permission setup for macOS Accessibility or Screen Recording. After granting access, observe fresh state before continuing; do not automatically replay prior input.
|
||||
Await operations sequentially. After input, read fresh state to verify the effect. A successful action means dispatched, not verified completion. Cancellation may leave partial effects; never retry uncertain input without observing. STALE_APP requires getApp again; STALE_SNAPSHOT requires getState again. Absolute coordinate input requires a fresh screenshot of that window. Relative input can use a successful AX observation or screenshot. SCREENSHOT_REQUIRED means capture a fresh screenshot before using pixel coordinates. For click and hover, coordinate targets must be [x, y], for example hover([650, 400], {relative:true}). A bare number such as hover(650) is always an AX element index. The relative option applies only to coordinate targets, not AX indices. Window resizing since the latest observation rejects coordinate input; movement alone preserves window-local coordinates. WINDOW_CHANGED requires fresh state before any retry; use an advertised AX action or explicitly restore the window before screenshot or coordinate input, then observe again. PERMISSION_DENIED requires the user to grant the required macOS system permissions for macOS Accessibility or Screen Recording. After granting access, observe fresh state before continuing; do not automatically replay prior input.
|
||||
|
||||
Hover first uses advertised AX alternate UI without moving the system cursor, preserving revealed descendants until the pointer target leaves their region. When unavailable, it moves the visible system cursor without pressing a button; that route requires the target point to be unobscured in the foreground app. FOREGROUND_REQUIRED requires explicit activation or raising and a fresh observation before retrying. Returning, AX reads and screenshots do not restore the previous point. Wait for the application's hover delay, observe, then hover into the revealed controls before clicking. Use successive hover calls for intermediate points. The result reports hoverRoute and systemPointerMoved. Other input or application changes can interrupt the interaction; app objects do not reserve applications between calls.
|
||||
|
||||
|
||||
Reference in new issue
Block a user