mward4/deepseek-harness
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 29 Wiki Activity

Merge pull request #6509 from deepseek-harness/feat/cua-basic-onboarding

Browse Source 
feat(cua): 增加 macOS 插件入口与权限管理
This commit is contained in:
CreatixChu authored and GitHub committed 2026-10-19 22:32:22 +08:00
2 parents 854043a205 33381c86c4
commit a5773881db
157 files changed
+5294 -519
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

.agents/notes/implemented/architecture/2026-09-21-experimental-capabilities-as-optional-bundles.i18n.yaml
+4 -4
View File
@@ -9,11 +9,11 @@
en: 2ba6c43b20a3d347
zh: 5647179ddaa38ad9
/agent-note-ship-experimental-capabilities-as-optional-bundles/decision:
en: 6c8583bdefc5e36c
zh: 0928b4f2876bbfe4
en: d6328ba2b218f2ac
zh: 3731561313b1ad5e
/agent-note-ship-experimental-capabilities-as-optional-bundles/alternatives-considered:
en: d6f40bf02df37ab7
zh: e5b85ce9d41e8754
/agent-note-ship-experimental-capabilities-as-optional-bundles/consequences:
en: 38d27583533cd028
zh: 9bf11f5489c1aa43
en: 3064da835e892958
zh: bfcf967b6acc6f85
.agents/notes/implemented/architecture/2026-09-21-experimental-capabilities-as-optional-bundles.md
+5 -1
View File
@@ -12,12 +12,14 @@ The Web plugin page offered two optional bundles, Agent Teams and voice input. A
`OPTIONAL_BUNDLES` owns the installation's default-off selections, including maintained optional capabilities and experimental capabilities. Every bundle declares `icon` and exports `./locale/*.json` with `meta.title` and `meta.description`; the Official group displays that metadata. A bundle's presentation does not determine its [experimental status](../../../../packages/experimental/README.md#status). The isolation check validates admission and still rejects experimental dependencies of nonexperimental bundles.
An optional bundle's dependencies are downloaded with every `dsh` installation. [Reasoning translation](2026-10-05-anonymous-reasoning-translation.md) uses three experimental packages without adding third-party runtime dependencies. Admission therefore considers installation cost and whether a switch supplies a usable composition without an additional configuration form. Browser and computer providers remain explicit compositions because they add substantial runtimes or require an external executable. Stagehand requires model credentials; Python PTC replaces a core provider and conflicts with TypeScript workflow consumers. Demonstrative hooks, webhooks, and mods stay outside the list. [Session Inspector](../feature/2026-09-24-session-inspector.md) owns the Inspector's distinct activation behavior.
An optional bundle's dependencies are downloaded with every `dsh` installation. [Reasoning translation](2026-10-05-anonymous-reasoning-translation.md) uses three experimental packages without adding third-party runtime dependencies. Admission therefore considers installation cost and whether a switch supplies a usable composition without an additional configuration form. External browser and Cua Driver providers remain explicit compositions because they add substantial runtimes or require an external executable. Stagehand requires model credentials; Python PTC replaces a core provider and conflicts with TypeScript workflow consumers. Demonstrative hooks, webhooks, and mods stay outside the list. [Session Inspector](../feature/2026-09-24-session-inspector.md) owns the Inspector's distinct activation behavior.
Bundles insert ordinary Host rows; optional model tools register in the global tool layer, so every preset, including minimal, receives them. Plugin management, profile composition, and reload need no bundle-specific behavior. Search, Ralph, and terminals supply their own isolated services in a Host group. Search opens one in-memory index on demand; the Host search policy stays unchanged. Badge skills and title providers act on the Host. Switching a bundle registers or removes its tools for live Agents, and later user patch layers retain precedence. A replacement title provider may register once the previous provider's disposal starts; the previous provider's late results never commit.
Claude Code and Codex install on demand through the separate `ON_DEMAND_BUNDLES` catalog; the [on-demand native-bundle decision](2026-10-05-official-on-demand-bundles.md) owns the cost, version, and offline-discovery trade-offs.
Computer Use is a shipped optional bundle in the experimental group. Its bundle uses the explicit naming exception `@deepseek-ai/dsh-cua-bundle`, alongside `@deepseek-ai/dsh-cua`. These names preserve profile references when the capability moves package groups; the experimental classification still controls isolation and promotion. Its repository-owned native component is prepared when the provider loads, without an external Cua Driver runtime.
## Alternatives considered
**Ship every provider as an optional bundle.** Composes and displays correctly, but grows every install by provider runtimes for a capability most installations never switch on; voice input's `sherpa-onnx-node` is the one accepted precedent.
@@ -31,3 +33,5 @@ Claude Code and Codex install on demand through the separate `ON_DEMAND_BUNDLES`
## Consequences
Official discovery does not imply experimental status or a different plugin lifecycle. Switching a bundle off keeps its installed files; failed activation retains the existing saved-selection behavior. Bundle rows are ordinary Host rows with their usual switches. Individual and combined composition tests cover the Host rows and every preset's tool catalog; live switching and recorded Sessions cover activation and model-visible behavior.
Computer Use keeps its permission service and native provider on the Host and registers tools in Agent scopes whose bound preset is `ptc`. Existing Agent events and `createScope()` maintain current, new and resumed Agents. Preset declarations remain unchanged. Only the explicit detail-page action changes the new-session default through the existing settings form; the activation switch does not configure PTC execution.
.agents/notes/implemented/architecture/2026-09-21-experimental-capabilities-as-optional-bundles.zh.md
+5 -1
View File
@@ -12,12 +12,14 @@ Web 插件页只提供两个可选 bundle：Agent Teams 与语音输入。Auto r
`OPTIONAL_BUNDLES` 管理安装随附且默认关闭的选项，包括维护中的可选能力与实验性能力。每个 bundle 声明 `icon`，并导出带 `meta.title` 与 `meta.description` 的 `./locale/*.json`；官方分组显示这些元数据。Bundle 的展示方式不决定其[实验性状态](../../../../packages/experimental/README.zh.md#status)。隔离检查校验准入，并继续拒绝非实验 bundle 的实验性依赖。
可选 bundle 的依赖随每次 `dsh` 安装下载。[思考过程翻译](2026-10-05-anonymous-reasoning-translation.zh.md)使用三个实验包，不增加第三方运行时依赖。因此准入需要考虑安装成本，以及开关能否在不增加配置表单的情况下提供可用组合。浏览器与电脑提供方仍使用显式组合，因为它们会增加较大的运行时或要求外部可执行文件。Stagehand 需要模型凭据；Python PTC 替换核心提供方，并与 TypeScript workflow 消费方冲突。演示性钩子、webhook 与模组不进入名单。[Session Inspector](../feature/2026-09-24-session-inspector.zh.md) 管理 Inspector 特有的激活行为。
可选 bundle 的依赖随每次 `dsh` 安装下载。[思考过程翻译](2026-10-05-anonymous-reasoning-translation.zh.md)使用三个实验包，不增加第三方运行时依赖。因此准入需要考虑安装成本，以及开关能否在不增加配置表单的情况下提供可用组合。外部浏览器与 Cua Driver 提供方仍使用显式组合，因为它们会增加较大的运行时或要求外部可执行文件。Stagehand 需要模型凭据；Python PTC 替换核心提供方，并与 TypeScript workflow 消费方冲突。演示性钩子、webhook 与模组不进入名单。[Session Inspector](../feature/2026-09-24-session-inspector.zh.md) 管理 Inspector 特有的激活行为。
Bundle 插入普通 Host 行；可选模型工具注册到全局工具层，因此包括 minimal 在内的所有预设都会获得它们。插件管理、profile 组合与重载无需针对 bundle 的特殊行为。搜索、Ralph 与终端在 Host 组内提供自己的隔离服务。搜索按需打开一个内存索引；Host 搜索策略保持不变。徽章技能与标题提供方作用于 Host。切换 bundle 会为运行中的 Agent 注册或移除其工具，后续用户 patch 层继续优先。前一个标题提供方开始清理后，替代提供方即可注册；前一个提供方的迟到结果不会提交。
Claude Code 与 Codex 通过独立的 `ON_DEMAND_BUNDLES` 目录按需安装，其成本、版本与离线发现取舍由[按需原生 bundle 决策](2026-10-05-official-on-demand-bundles.zh.md)管理。
Computer Use 是 experimental 包组内随安装提供的可选组合包。组合包与 `@deepseek-ai/dsh-cua` 一起采用显式命名例外 `@deepseek-ai/dsh-cua-bundle`，在能力移动分组时保留配置引用。依赖隔离和晋升仍由实验性分类约束。提供方加载时准备仓库自有的本机组件，无需外部 Cua Driver 运行时。
## 考虑过的替代方案
**把所有提供方都做成可选 bundle。** 组合与展示都正确，但会为多数安装从不启用的能力把提供方运行时塞进每次安装；语音输入的 `sherpa-onnx-node` 是唯一被接受的先例。
@@ -31,3 +33,5 @@ Claude Code 与 Codex 通过独立的 `ON_DEMAND_BUNDLES` 目录按需安装，
## 后果
官方发现不表示实验性状态，也不引入不同的插件生命周期。关闭 bundle 保留已安装文件；激活失败保留现有的已保存选择行为。Bundle 行是普通 Host 行，使用常规开关。独立与组合测试覆盖 Host 行及每个预设的工具目录；实时切换与已记录 Session 覆盖激活和模型可见行为。
Computer Use 将权限服务和原生提供方保留在 Host，在已绑定 `ptc` preset 的 Agent 作用域中注册工具。现有 Agent 事件和 `createScope()` 维护当前、新建和恢复的 Agent，preset 声明保持不变。只有详情页的显式操作通过现有设置表单修改新会话默认模式，启用开关不配置 PTC 执行方式。
.agents/notes/implemented/architecture/2026-09-26-cua-application-consent.i18n.yaml
+6 -6
View File
@@ -9,11 +9,11 @@
en: b32a7ce6483b9b48
zh: ec0938ad032b99d8
/agent-note-application-consent-for-native-computer-use/decision:
en: 92247ff599c1103f
zh: 26845420a9711c83
en: 36c560c24c376f12
zh: 5e4ff72f66574e38
/agent-note-application-consent-for-native-computer-use/alternatives-considered:
en: 4693dcd217c4fedb
zh: 1dc9bf9067314006
en: 80cd5fd93a42d937
zh: 2a0e241b1b8c2e1d
/agent-note-application-consent-for-native-computer-use/consequences:
en: 0146a69ab2086edc
zh: 4257bc1ab4432c3f
en: 4bda3366af65250a
zh: 7c83561b4db07284
.agents/notes/implemented/architecture/2026-09-26-cua-application-consent.md
+16 -7
View File
@@ -10,29 +10,38 @@ macOS privileges authorize an execution host, not a user's choice of target appl
## Decision
The CUA executor owns application consent for every object-API and native-tool dispatch. It uses the existing User Questions service for human decisions and the provider's existing profile configuration for saved grants. General tool approval and operating-system privileges remain separate requirements. Arbitrary persistent PTC code and separately authorized shell tools retain their own execution authority.
The CUA executor owns application consent for every object-API and native-tool dispatch. It uses the existing User Questions service for human decisions and the provider's existing profile configuration for saved grants. The existing question UI presents the offered labels; the executor rejects custom text or answers that do not select exactly one offered label. Permission-page icons and names are presentation metadata, excluded from saved identities and model results. General tool approval and operating-system privileges remain separate requirements. Arbitrary persistent PTC code and separately authorized shell tools retain their own execution authority.
A saved identity combines the canonical executable path with a verified designated signing requirement, represented by a SHA-256 digest. The path prevents authorization from following a copied application, and the signing requirement distinguishes replacements while allowing matching signed updates. Unverifiable and ad-hoc applications receive task access only, tied to a process instance and Agent connection lifetime. Labels do not establish identity.
System-permission buttons call the native helper on the DSH host and remain available to remote browsers. The [usage documentation](../../../../packages/experimental/cua/README.md#use-this-package) explains where to complete authorization; the page has no persistent remote-access notice.
A saved identity combines the canonical executable path with a verified designated signing requirement, represented by a SHA-256 digest. The path prevents authorization from following a copied application, and the signing requirement distinguishes replacements while allowing matching signed updates. Unverifiable and ad-hoc applications receive session access only, tied to a process instance. Labels do not establish identity.
Cold application selection prepares an executable identity and handle without launching. The same consent decision guards the private launch continuation, and the helper checks the executable again before launch and binds the resulting process afterward. Signed selections retain the normal designated-requirement identity; an unverifiable executable uses a content identity for this pending selection and cannot create a saved grant. Preparing metadata before asking avoids starting an application the user refuses.
Selections of the same application within one task are serialized, so a pending question does not produce competing decisions. Saved-grant additions read the current configuration inside one mutation queue. Refusals prevent another prompt within the task; an explicitly restored saved grant takes precedence at dispatch.
Session decisions are kept in memory by Session ID and application identity. Agent disposal and helper replacement release native handles without discarding these decisions; provider disposal or DSH restart clears them. This scope covers follow-up turns without relying on the model to decide when a user intent has ended.
Removing a saved grant invalidates Agent connections retaining the application's handles and cancels active operations; Agent/provider disposal awaits native connection cleanup. Reauthorization uses a fresh connection, so old app and window snapshots cannot become valid again. The [native object decision](2026-09-17-experimental-cua-object-api.md) continues to own process identity, coordinate semantics, and separation from the evaluator.
Selections of the same application within one session are serialized, so a pending question does not produce competing decisions. Saved-grant additions and revocations share one mutation queue and read the current grant list inside it. Provider shutdown cancels writes still waiting for HMR admission; admitted writes settle before disposal completes. Refusals prevent another prompt until the user permits asking again; a restored saved grant takes precedence in both dispatch and the permission-page projection.
Revocation invalidates the Agent connections that retain the application's handles, then awaits native cleanup. Reauthorization uses a fresh connection, so old app and window snapshots cannot become valid again. The [native object decision](2026-09-17-experimental-cua-object-api.md) continues to own process identity, coordinate semantics, and separation from the evaluator.
## Alternatives considered
**Trust model instructions or hide controls.** Retained objects and direct tool calls can still reach native operations. The executor must enforce the decision at dispatch.
**Persist application names or unsigned paths.** A replacement can reuse both. Durable grants require a verifiable signing identity; task-only access supports unsigned development applications without that promise.
**Persist application names or unsigned paths.** A replacement can reuse both. Durable grants require a verifiable signing identity; session access supports unsigned development applications without that promise.
**Add an approval database or Session control stream.** Profile settings already own user configuration, and User Questions already owns interactive answers.
**Add an approval database or Session control stream.** Profile settings already own user configuration, and User Questions already owns interactive answers. The permission UI observes the provider and its owning settings namespace without another authorization store or event channel.
**Disable permission buttons based on the browser address.** A user may operate the DSH host through screen sharing and complete authorization there. Port forwarding and proxies make the address an unreliable indication of whether the browser and DSH run on the same computer.
**Use the nearest GUI ancestor as the permission target.** A source-built Desktop app can sit between Terminal and the helper while macOS still attributes Accessibility and Screen Recording to Terminal. Process responsibility supplies the system’s attribution; ancestry alone does not. The optional SPI adds compatibility risk, so lookup failure omits the application instead of guessing.
**Keep the native context after revocation.** Its application lookup can return an earlier handle and retain window snapshots. Closing the Agent connection discards its native context through existing transport cleanup; preserving unrelated handles would require a native per-application retirement protocol.
## Consequences
Revoking one application also invalidates other application handles on the same Agent connection. Those applications require fresh selection; saved grants remain reusable, while task grants expire. Other tasks with independent connections remain unaffected unless they also hold access to the revoked application.
Refusing or revoking one application also invalidates other application handles on the same Agent connection. Those applications require fresh selection; session and saved grants remain reusable. Other sessions with independent connections remain unaffected unless they also hold access to the revoked application.
Explicit enablement may start the shared helper for a read-only permission probe. The helper captures the application identified by macOS process responsibility before readiness releases the launcher, retaining its name and icon after reparenting. The optional native query is dynamically resolved; unavailable attribution uses generic system-prompt guidance. The macOS prompt remains authoritative. Enabling does not request macOS authorization or inspect application content.
Application decisions and failure results are exercised through policy regressions and the recorded CUA scenario. Desktop binary distribution, signing, and packaged-host permissions remain outside the source-launch workflow.
.agents/notes/implemented/architecture/2026-09-26-cua-application-consent.zh.md
+16 -7
View File
@@ -10,29 +10,38 @@ macOS 权限授权执行宿主，不能表达用户允许操作哪些目标应
## 决策
CUA 执行器在每次对象 API 和原生工具调用时检查应用授权。人工决定复用 User Questions 服务，长期许可保存在 provider 的现有 profile 配置中。通用工具审批和操作系统权限仍是独立要求。任意 persistent PTC 代码和另行授权的 shell 工具保留各自的执行权限。
CUA 执行器在每次对象 API 和原生工具调用时检查应用授权。人工决定复用 User Questions 服务，长期许可保存在 provider 的现有 profile 配置中。通用工具审批和操作系统权限仍是独立要求。任意 persistent PTC 代码和另行授权的 shell 工具保留各自的执行权限。现有问答界面展示授权选项；执行器拒绝自由文本或未准确选中一个所给选项的回答。权限页图标和名称仅用于展示，不进入保存的身份信息或模型结果。
长期身份由规范化可执行文件路径和经过验证的签名指定要求组成，以 SHA-256 摘要表示。路径阻止许可跟随应用副本，签名要求区分替换程序，并允许符合相同签名要求的更新。签名不可验证或使用 ad-hoc 签名的应用只能获得任务授权，许可绑定进程实例和 Agent 连接生命周期。显示名称不用于确定身份。
系统权限按钮调用运行 DSH 的电脑上的原生 helper，远程访问时仍可使用。授权地点写在[使用说明](../../../../packages/experimental/cua/README.zh.md#use-this-package)中，页面不常驻远程访问提示。
长期身份由规范化可执行文件路径和经过验证的签名指定要求组成，以 SHA-256 摘要表示。路径阻止许可跟随应用副本，签名要求区分替换程序，并允许符合相同签名要求的更新。签名不可验证或使用 ad-hoc 签名的应用只能获得会话授权，许可绑定进程实例。显示名称不用于确定身份。
选择尚未运行的应用时，先准备可执行文件身份和句柄，不立即启动。同一授权决定保护内部启动续接，helper 在启动前再次核对可执行文件，并在启动后绑定实际进程。已签名应用沿用签名指定要求身份；签名不可验证的程序使用本次待启动选择的内容身份，不能生成长期许可。先准备元数据再询问，可以避免启动用户拒绝的应用。
同一任务对同一应用的选择串行执行，避免未完成的问题产生相互冲突的决定。长期许可的新增在同一写入队列内读取当前配置。拒绝会阻止任务内的再次询问；用户显式恢复的长期许可在执行检查时优先于旧拒绝记录。
会话决定按 Session ID 和应用身份保存在内存中。Agent 销毁或 helper 重建会释放原生句柄，但保留授权决定；提供方卸载或 DSH 重启后清除。会话范围可以覆盖用户的后续指令，无须由模型判断一次用户意图何时结束。
删除长期许可会使保留目标应用句柄的 Agent 连接失效，并取消活动操作；Agent 或提供方销毁时等待对应的原生连接清理。重新授权使用新连接，因此旧应用对象和窗口快照不能重新生效。[原生对象决策](2026-09-17-experimental-cua-object-api.zh.md)继续负责进程身份、坐标语义和与执行器分离的设计。
同一会话对同一应用的选择串行执行，避免未完成的问题产生相互冲突的决定。长期许可的新增和撤销共用写入队列，并在队列内读取当前许可列表。提供方关闭时取消尚在等待 HMR 调度的写入，已开始的写入会在销毁完成前结束。拒绝会阻止再次询问，直到用户允许重询；重新添加的长期许可在执行检查和权限页中均优先于旧拒绝记录。
撤销使保留目标应用句柄的 Agent 连接失效，并等待原生清理完成。重新授权使用新连接，因此旧应用对象和窗口快照不能重新生效。[原生对象决策](2026-09-17-experimental-cua-object-api.zh.md)继续负责进程身份、坐标语义和与执行器分离的设计。
## 考虑过的替代方案
**依赖模型指引或隐藏控件。** 保留的对象和直接工具调用仍能到达原生操作。执行器必须在派发时检查用户决定。
**保存应用名称或未签名路径。** 替换程序可以复用二者。长期许可需要可验证的签名身份；任务授权支持未签名的开发应用，但不承诺跨进程复用。
**保存应用名称或未签名路径。** 替换程序可以复用二者。长期许可需要可验证的签名身份；会话授权支持未签名的开发应用，但不承诺跨进程复用。
**新增审批数据库或 Session 控制流。** Profile settings 已负责用户配置，User Questions 已负责交互回答。
**新增审批数据库或 Session 控制流。** Profile settings 已负责用户配置，User Questions 已负责交互回答。权限界面查询 provider，并监听其配置 namespace，无须另一份授权存储或事件通道。
**根据浏览器访问地址禁用权限按钮。** 用户可能通过屏幕共享操作运行 DSH 的电脑，并在该电脑上完成授权。端口转发和代理使访问地址无法可靠判断浏览器与 DSH 是否运行在同一台电脑上。
**把最近的 GUI 祖先当作授权对象。** 源码构建的 Desktop 可以位于 Terminal 和 helper 之间，但 macOS 仍将辅助功能和屏幕录制归属于 Terminal。责任进程记录提供系统认定的归属，仅凭父进程链无法确定。可选内部接口有兼容性风险，因此查询失败时省略应用信息，不再猜测。
**撤销后保留原生状态。** 原生应用查找可能返回旧句柄并保留窗口快照。关闭 Agent 连接可以通过现有传输清理释放其原生状态；保留其他应用的句柄则需要新增按应用销毁状态的原生协议。
## 影响
撤销一个应用也会使同一 Agent 连接中其他应用的句柄失效。这些应用需要重新选择；长期许可仍可复用，任务许可随之失效。其他任务使用独立连接，除非也持有被撤销应用的许可，否则不受影响。
拒绝或撤销一个应用也会使同一 Agent 连接中其他应用的句柄失效。这些应用需要重新选择；会话许可和长期许可仍可复用。其他会话使用独立连接，除非也持有被撤销应用的许可，否则不受影响。
显式启用可以启动共享 helper 进行只读权限检查。helper 在发布就绪状态、让启动器退出之前，记录 macOS 责任进程对应的应用，因此父进程改变后仍能提供应用名称和图标。可选的本机查询通过动态加载解析，无法取得归属时显示通用系统弹窗指引。授权归属仍以 macOS 提示为准。启用不会申请 macOS 授权或检查应用内容。
授权决定和失败结果由策略回归测试及 CUA 录制会话验证。桌面安装包的二进制分发、签名和打包宿主权限不属于源码启动流程。
apps/cli/package.json
+1
View File
@@ -105,6 +105,7 @@
"@deepseek-ai/dsh-experimental-terminal-bundle": "workspace:*",
"@deepseek-ai/dsh-experimental-session-search": "workspace:*",
"@deepseek-ai/dsh-experimental-tool-worktree": "workspace:*",
"@deepseek-ai/dsh-cua-bundle": "workspace:*",
"@deepseek-ai/dsh-experimental-plugin-market-bundle": "workspace:*"
},
"devDependencies": {
apps/desktop/README.i18n.yaml
+2 -2
View File
@@ -42,8 +42,8 @@
en: 98cf848762482d48
zh: 74f9100a39ec8c2f
/deepseek-harness-desktop/develop:
en: fe6d23e6a3d3bc66
zh: d60c8d3a55747916
en: faf7676ba18bbf2f
zh: f1d7b75ad348c548
/deepseek-harness-desktop/develop/startup-onboarding:
en: a51346b8d880ddba
zh: 7913214c1c66063c
apps/desktop/README.md
+2
View File
@@ -202,6 +202,8 @@ After an explicit build, `start:desktop` reconstructs the disposable project and
pnpm run start:desktop
```
Desktop package preparation filters optional native packages by the installer target’s OS and architecture. Windows package sets omit the macOS Computer Use tarball, including when their inputs were packed on macOS.
The Web counterparts are `pnpm run dev:web` and `pnpm run start:web`, documented in the [development guide](../../docs/development.md). Workspace development runs the current CLI and private Desktop Host packages under Electron RunAsNode. Plugin management and recovery use `$DSH_HOME/profiles/desktop`, separate from the disposable workspace runtime. The Host uses runtime module resolution in both development and packaged builds without creating official-package fallback links; developer-installed packages, including links, retain native priority. Use an unpacked application to exercise Electron RunAsNode, bundled pnpm, bundled dsh resources, plugin installation and repair paths.
The [native/renderer keyboard tests](tests/keyboard.spec.ts) compile as part of the repository Client typecheck. Their Desktop imports are limited to Cordis-free input, persistence, IPC, browser-guest, and overlay modules.
apps/desktop/README.zh.md
+2
View File
@@ -202,6 +202,8 @@ pnpm run dev:desktop
pnpm run start:desktop
```
桌面包准备过程按安装包目标的操作系统和架构筛选可选原生包。Windows 的包集合不包含 macOS Computer Use tarball，即使输入包是在 macOS 上打包的。
Web 侧的对应命令是 `pnpm run dev:web` 与 `pnpm run start:web`，见[开发指南](../../docs/development.zh.md)。Workspace 开发使用 Electron RunAsNode 运行当前 CLI 与私有 Desktop Host 包，插件管理和恢复使用 `$DSH_HOME/profiles/desktop`，与一次性工作区运行时分离。Host 在开发与打包构建中都使用 runtime 模块解析，不创建官方包的 fallback 链接；开发者安装的包（包括链接）保留原生优先级。需要验证 Electron RunAsNode、内置 pnpm、内置 dsh 资源、插件安装和修复时，应运行未封装安装器的应用目录。
[原生输入与渲染进程键盘测试](tests/keyboard.spec.ts)直接纳入仓库 Client 类型检查。它只导入不依赖 Cordis 的 Desktop 输入、持久化、IPC、浏览器 guest 和蒙层模块。
apps/desktop/scripts/prepare-dsh.ts
+1 -1
View File
@@ -134,7 +134,7 @@ async function main(): Promise<void> {
await packagingStep(process.env.DSH_DESKTOP_PACKAGING_RUN_DIR, 'runtime:install', () => runPnpm(['install', '--prod', '--frozen-lockfile', '--trust-lockfile']))
const packageSet = readDesktopCorePackageSet(BUILD_ROOT, release.version)
const targetName = resolveDesktopBuildTarget()
const target = { platform: process.platform, arch: desktopTargetPlatform(targetName).arch }
const target = desktopTargetPlatform(targetName)
const modules = join(BUILD_ROOT, 'node_modules')
const officeManifest = JSON.parse(readFileSync(join(modules, '@deepseek-ai/libreoffice-kit/package.json'), 'utf8'))
const officeEngine = selectOfficeEngine(officeManifest, target)
apps/desktop/scripts/prepare-package-set.ts
+28 -6
View File
@@ -25,7 +25,7 @@ import {
} from '../src/core-package-set.ts'
import { capture } from '../../../scripts/release/process.ts'
import { tarballFiles } from '../../../scripts/release/tarball.ts'
import { resolveDesktopTargetBuildPaths } from './desktop-build-paths.mjs'
import { desktopTargetPlatform, resolveDesktopBuildTarget, resolveDesktopTargetBuildPaths } from './desktop-build-paths.mjs'
const DSH_PACKAGE = '@deepseek-ai/dsh'
const ROOT_PACKAGES = [DSH_PACKAGE, DESKTOP_HOST_PACKAGE] as const
@@ -41,6 +41,17 @@ export interface PackedDesktopPackage {
readonly manifest: Readonly<Record<string, unknown>>
}
function supportsTarget(manifest: Readonly<Record<string, unknown>>, field: 'os' | 'cpu', target: string): boolean {
const values = manifest[field]
if (values === undefined) return true
if (!Array.isArray(values) || !values.every((value): value is string => typeof value === 'string')) {
throw new Error(`desktop package set: ${String(manifest.name)} has invalid ${field}`)
}
if (values.includes(`!${target}`)) return false
return values.length === 0 || (values.length === 1 && values[0] === 'any') || values.includes(target)
|| values.every(value => value.startsWith('!'))
}
function dependencyNames(manifest: Readonly<Record<string, unknown>>, section: string): string[] {
const value = manifest[section]
if (value === undefined) return []
@@ -54,10 +65,12 @@ function dependencyNames(manifest: Readonly<Record<string, unknown>>, section: s
* Select workspace dependencies rooted at dsh and its private Host; npm resolves external packages.
* Reads the repository workspace manifest and package manifests to distinguish required local packages from npm-resolved externals.
* @param available - Packed packages indexed by package name.
* @param target - Platform and architecture of the installer payload.
* @returns Selected packages sorted by name.
*/
export function selectDesktopPackageClosure(
available: ReadonlyMap<string, PackedDesktopPackage>,
target: ReturnType<typeof desktopTargetPlatform>,
): PackedDesktopPackage[] {
const workspace = yaml.load(readFileSync(join(REPOSITORY_ROOT, 'pnpm-workspace.yaml'), 'utf8')) as { packages: string[] }
const workspaceNames = new Set(globSync(workspace.packages.map(pattern => `${pattern}/package.json`), { cwd: REPOSITORY_ROOT })
@@ -77,7 +90,9 @@ export function selectDesktopPackageClosure(
}
}
for (const dependency of dependencyNames(packed.manifest, OPTIONAL_DEPENDENCY_SECTION)) {
if (available.has(dependency)) visit(dependency)
const optional = available.get(dependency)
if (optional && supportsTarget(optional.manifest, 'os', target.platform)
&& supportsTarget(optional.manifest, 'cpu', target.arch)) visit(dependency)
}
}
for (const name of ROOT_PACKAGES) {
@@ -127,9 +142,16 @@ export function assertDesktopHostPackageFiles(files: readonly string[]): void {
}
}
/** Prepare a package set from release tarball directories. */
export function prepareDesktopPackageSet(inputs: readonly string[], output: string): void {
const selected = selectDesktopPackageClosure(packedPackages(inputs))
/**
* Prepare a package set from release tarball directories.
* @param inputs - Release tarball directories.
* @param output - Destination package-set directory.
* @param target - Platform and architecture of the installer payload.
*/
export function prepareDesktopPackageSet(
inputs: readonly string[], output: string, target: ReturnType<typeof desktopTargetPlatform>,
): void {
const selected = selectDesktopPackageClosure(packedPackages(inputs), target)
const host = selected.find(packed => packed.manifest.name === DESKTOP_HOST_PACKAGE)
if (host === undefined) throw new Error(`desktop package set: selected closure omits ${DESKTOP_HOST_PACKAGE}`)
assertDesktopHostPackageFiles(tarballFiles(host.tarball))
@@ -171,7 +193,7 @@ function main(): void {
})
const inputs = (values.from ?? defaultInputs).map(path => resolve(REPOSITORY_ROOT, path))
const output = values.out === undefined ? buildPaths.packageSet : resolve(REPOSITORY_ROOT, values.out)
prepareDesktopPackageSet(inputs, output)
prepareDesktopPackageSet(inputs, output, desktopTargetPlatform(resolveDesktopBuildTarget()))
console.log(`desktop package set: prepared ${output}`)
}
apps/desktop/tests/cua-optional-package.spec.ts
+121
View File
@@ -0,0 +1,121 @@
/** Cold pnpm requests exercise shipped CUA dependency/platform metadata with inert package contents. */
import { createHash } from 'node:crypto'
import { createServer } from 'node:http'
import { mkdir, mkdtemp, readFile, rm, writeFile } from 'node:fs/promises'
import { tmpdir } from 'node:os'
import { join, resolve } from 'node:path'
import { execa } from 'execa'
import { expect, it, onTestFinished } from 'vitest'
import { pnpmCommand } from '../../../scripts/release/process.ts'
const repository = resolve(import.meta.dirname, '../../..')
const providerName = '@deepseek-ai/dsh-experimental-cua-macos'
const bundleName = '@deepseek-ai/dsh-cua-bundle'
const readManifest = async (path: string): Promise<Record<string, unknown>> =>
JSON.parse(await readFile(join(repository, path, 'package.json'), 'utf8')) as Record<string, unknown>
it('fetches the macOS optional tarball only for Darwin, with an empty cache for every target', async () => {
const root = await mkdtemp(join(tmpdir(), 'cua-cold-install-'))
onTestFinished(() => rm(root, { recursive: true, force: true }))
const userconfig = join(root, 'npmrc')
await writeFile(userconfig, '')
const environment = Object.fromEntries(Object.entries(process.env).filter(([key]) =>
!/KEY|SECRET|TOKEN|PASSWORD/iu.test(key) && !/^(npm|pnpm|corepack)_/iu.test(key)
&& key !== 'NODE_OPTIONS' && key !== 'NODE_PATH'))
const [pnpm, ...prefix] = pnpmCommand()
const command = async (args: string[], cwd: string) => {
const result = await execa(pnpm, [...prefix, `--config.userconfig=${userconfig}`, ...args], {
cwd, env: { ...environment, CI: 'true', XDG_CACHE_HOME: join(cwd, 'xdg-cache'),
XDG_DATA_HOME: join(cwd, 'xdg-data'), XDG_STATE_HOME: join(cwd, 'xdg-state'),
XDG_CONFIG_HOME: join(cwd, 'xdg-config') }, extendEnv: false, reject: false, timeout: 60_000,
})
expect(result.timedOut, result.stderr).toBe(false)
expect(result.signal, result.stderr).toBeUndefined()
expect(result.exitCode, `${result.stdout}\n${result.stderr}`).toBe(0)
return result.stdout
}
const cli = await readManifest('apps/cli')
const bundle = await readManifest('packages/experimental/cua-bundle')
const native = await readManifest('packages/experimental/cua-macos')
expect(cli.dependencies).toHaveProperty(bundleName)
expect(bundle.dependencies).not.toHaveProperty(providerName)
expect(bundle.optionalDependencies).toEqual({ [providerName]: 'workspace:*' })
expect(native.os).toEqual(['darwin'])
const version = '1.0.0'
const bundleDependencies = Object.fromEntries(Object.keys(bundle.dependencies as Record<string, string>).map(name => [name, version]))
const manifests = [
{ name: cli.name, version, dependencies: { [bundleName]: version } },
{ name: bundleName, version, dependencies: bundleDependencies, optionalDependencies: { [providerName]: version } },
...Object.keys(bundleDependencies).map(name => ({ name, version })),
{ name: providerName, version, os: native.os, files: native.files },
]
const packages = new Map<string, { manifest: Record<string, unknown>; bytes: Buffer }>()
for (const manifest of manifests) {
const name = String(manifest.name)
const directory = join(root, name.replace('@deepseek-ai/', ''))
await mkdir(directory)
await writeFile(join(directory, 'package.json'), JSON.stringify(manifest))
if (name === providerName) {
await mkdir(join(directory, 'native/bin/darwin-arm64'), { recursive: true })
for (const file of ['.gitignore', '.npmignore']) {
await writeFile(join(directory, 'native', file), await readFile(join(repository, 'packages/experimental/cua-macos/native', file)))
}
await writeFile(join(directory, 'native/bin/darwin-arm64/dsh-cua-native'), 'inert native payload')
}
await command(['pack', '--out', 'package.tgz'], directory)
const bytes = await readFile(join(directory, 'package.tgz'))
packages.set(name, { manifest, bytes })
if (name === providerName) {
const result = await execa('tar', ['-tzf', join(directory, 'package.tgz')])
expect(result.stdout).toContain('package/native/bin/darwin-arm64/dsh-cua-native')
}
}
const requests: string[] = []
let unrestricted = false
const server = createServer((request, response) => {
const path = decodeURIComponent(request.url ?? '')
requests.push(path)
const tarball = path.endsWith('.tgz')
const name = path.slice(1).replace(/\.tgz$/u, '')
const entry = packages.get(name)
if (!entry) { response.writeHead(404); response.end(); return }
if (tarball) { response.end(entry.bytes); return }
const manifest = { ...entry.manifest }
// pnpm also infers macOS from this package name; an explicit any overrides that inference.
if (unrestricted && name === providerName) manifest.os = ['any']
response.setHeader('Content-Type', 'application/json')
response.end(JSON.stringify({ name, 'dist-tags': { latest: version }, versions: {
[version]: { ...manifest, dist: {
tarball: `http://${request.headers.host}/${name}.tgz`,
integrity: `sha512-${createHash('sha512').update(entry.bytes).digest('base64')}`,
} },
} }))
})
onTestFinished(async () => {
server.closeAllConnections()
await new Promise<void>((resolve, reject) => { server.close((error) => { if (error) reject(error); else resolve() }) })
})
await new Promise<void>((resolve, reject) => { server.once('error', reject); server.listen(0, '127.0.0.1', resolve) })
const address = server.address()
if (!address || typeof address === 'string') throw new Error('fixture registry has no TCP address')
for (const target of ['darwin', 'win32', 'linux', 'linux-unrestricted']) {
unrestricted = target === 'linux-unrestricted'
const project = join(root, target)
await mkdir(project)
await writeFile(join(project, 'package.json'), JSON.stringify({ private: true, dependencies: { [String(cli.name)]: version } }))
await writeFile(join(project, 'pnpm-workspace.yaml'), [
'autoInstallPeers: false', 'supportedArchitectures:', ` os: [${unrestricted ? 'linux' : target}]`,
' cpu: [arm64]', 'minimumReleaseAge: 0', '',
].join('\n'))
requests.length = 0
const output = await command([
'install', '--ignore-scripts', '--config.enable-global-virtual-store=false',
`--config.registry=http://127.0.0.1:${address.port}/`, `--config.store-dir=${join(project, 'store')}`,
`--config.cache-dir=${join(project, 'cache')}`, `--config.state-dir=${join(project, 'state')}`,
], project)
expect(requests).toContain(`/${bundleName}.tgz`)
expect(requests).toContain('/@deepseek-ai/dsh-cua.tgz')
expect(requests.includes(`/${providerName}.tgz`), `${target}: ${JSON.stringify(requests)}\n${output}`)
.toBe(target === 'darwin' || unrestricted)
}
}, 120_000)
apps/desktop/tests/prepare-package-set.spec.ts
+75 -5
View File
@@ -1,6 +1,12 @@
import { afterEach, describe, expect, it, vi } from 'vitest'
import { afterEach, describe, expect, it, onTestFinished, vi } from 'vitest'
import { execFileSync } from 'node:child_process'
import { mkdirSync, mkdtempSync, readFileSync, readdirSync, rmSync, writeFileSync } from 'node:fs'
import { tmpdir } from 'node:os'
import { join } from 'node:path'
import { DESKTOP_HOST_RUNTIME_FILES, DESKTOP_PACKAGES_DIR } from '../src/core-package-set.ts'
import {
assertDesktopHostPackageFiles,
prepareDesktopPackageSet,
selectDesktopPackageClosure,
type PackedDesktopPackage,
} from '../scripts/prepare-package-set.ts'
@@ -37,7 +43,7 @@ describe('desktop package-set selection', () => {
['@deepseek-ai/platform-package', packed('@deepseek-ai/platform-package')],
['@deepseek-ai/unused', packed('@deepseek-ai/unused')],
])
expect(selectDesktopPackageClosure(available).map(entry => entry.manifest.name)).toEqual([
expect(selectDesktopPackageClosure(available, { platform: 'win32', arch: 'x64' }).map(entry => entry.manifest.name)).toEqual([
'@deepseek-ai/cordis',
'@deepseek-ai/dsh',
'@deepseek-ai/dsh-base',
@@ -57,10 +63,10 @@ describe('desktop package-set selection', () => {
dependencies: { '@deepseek-ai/dsh': '^1.0.0' },
})],
])
expect(() => selectDesktopPackageClosure(available)).toThrow(/unpacked package/u)
expect(() => selectDesktopPackageClosure(available, { platform: 'win32', arch: 'x64' })).toThrow(/unpacked package/u)
expect(() => selectDesktopPackageClosure(new Map([
['@deepseek-ai/dsh', packed('@deepseek-ai/dsh')],
]))).toThrow(/omit @deepseek-ai\/dsh-desktop-host/u)
]), { platform: 'win32', arch: 'x64' })).toThrow(/omit @deepseek-ai\/dsh-desktop-host/u)
})
it('leaves independently published Office packages to npm resolution', () => {
@@ -73,7 +79,7 @@ describe('desktop package-set selection', () => {
})],
['@deepseek-ai/dsh-desktop-host', packed('@deepseek-ai/dsh-desktop-host')],
])
expect(selectDesktopPackageClosure(available).map(entry => entry.manifest.name)).toEqual([
expect(selectDesktopPackageClosure(available, { platform: 'win32', arch: 'x64' }).map(entry => entry.manifest.name)).toEqual([
'@deepseek-ai/dsh', '@deepseek-ai/dsh-desktop-host',
])
})
@@ -92,3 +98,67 @@ describe('desktop package-set selection', () => {
expect(() => { assertDesktopHostPackageFiles(files.slice(0, 1)) }).toThrow(/lib\/cli\.js/u)
})
})
const macProvider = '@deepseek-ai/dsh-experimental-cua-macos'
const cuaBundle = '@deepseek-ai/dsh-cua-bundle'
const targetManifests = () => new Map<string, PackedDesktopPackage>([
['@deepseek-ai/dsh', packed('@deepseek-ai/dsh', { dependencies: { [cuaBundle]: '1.0.0' } })],
['@deepseek-ai/dsh-desktop-host', packed('@deepseek-ai/dsh-desktop-host')],
[cuaBundle, packed(cuaBundle, {
optionalDependencies: { [macProvider]: '1.0.0' }, dependencies: { '@deepseek-ai/dsh-cua': '1.0.0' },
})],
['@deepseek-ai/dsh-cua', packed('@deepseek-ai/dsh-cua')],
[macProvider, packed(macProvider, { os: ['darwin'] })],
])
it.each([
['darwin', 'arm64', true], ['darwin', 'x64', true], ['win32', 'x64', false],
] as const)('selects the CUA native tarball for target %s/%s independently of the host', (platform, arch, included) => {
const selected = selectDesktopPackageClosure(targetManifests(), { platform, arch }).map(entry => entry.manifest.name)
expect(selected.includes(macProvider)).toBe(included)
expect(selected).toContain('@deepseek-ai/dsh-cua')
expect(selected).toContain(cuaBundle)
})
it.each([
[{ cpu: ['arm64'] }, false], [{ os: ['!win32'] }, false], [{ os: ['!linux'] }, true],
[{ os: [] }, true], [{ os: ['any'] }, true], [{ os: ['any', '!win32'] }, false],
[{ cpu: ['x64'] }, true],
])('honors optional package platform constraints %j', (constraints, included) => {
const packages = targetManifests()
packages.set(macProvider, packed(macProvider, constraints))
expect(selectDesktopPackageClosure(packages, { platform: 'win32', arch: 'x64' })
.some(entry => entry.manifest.name === macProvider)).toBe(included)
})
it.each([{ os: 'darwin' }, { cpu: [42] }])('rejects malformed platform constraints %j', (constraints) => {
const packages = targetManifests()
packages.set(macProvider, packed(macProvider, constraints))
expect(() => selectDesktopPackageClosure(packages, { platform: 'win32', arch: 'x64' })).toThrow(/has invalid/u)
})
it('copies no macOS tarball into the Windows package set', () => {
const root = mkdtempSync(join(tmpdir(), 'desktop-platform-packages-'))
onTestFinished(() => { rmSync(root, { recursive: true, force: true }) })
const inputs = join(root, 'inputs')
mkdirSync(inputs)
for (const [name, entry] of targetManifests()) {
const stem = name.replace('@deepseek-ai/', '')
const folder = join(root, stem, 'package')
mkdirSync(folder, { recursive: true })
writeFileSync(join(folder, 'package.json'), JSON.stringify(entry.manifest))
if (name === '@deepseek-ai/dsh-desktop-host') {
for (const file of DESKTOP_HOST_RUNTIME_FILES) {
mkdirSync(join(folder, file, '..'), { recursive: true })
writeFileSync(join(folder, file), '')
}
}
execFileSync('tar', ['-czf', join(inputs, `${stem}.tgz`), '-C', join(folder, '..'), 'package'])
}
const output = join(root, 'output')
prepareDesktopPackageSet([inputs], output, { platform: 'win32', arch: 'x64' })
expect(readdirSync(join(output, DESKTOP_PACKAGES_DIR))).not.toContain('dsh-experimental-cua-macos.tgz')
prepareDesktopPackageSet([inputs], output, { platform: 'darwin', arch: 'arm64' })
expect(readFileSync(join(output, DESKTOP_PACKAGES_DIR, 'dsh-experimental-cua-macos.tgz')))
.toEqual(readFileSync(join(inputs, 'dsh-experimental-cua-macos.tgz')))
})
apps/web/tests/cua-optin.e2e.ts
+128
View File
@@ -0,0 +1,128 @@
/** Optional CUA uses existing plugin activation and offers a separate persistent PTC action. */
import { mkdtemp, readFile, rm, writeFile } from 'node:fs/promises'
import { tmpdir } from 'node:os'
import { join } from 'node:path'
import { fileURLToPath } from 'node:url'
import { chromium, type Browser } from 'playwright'
import { readProfileManifest } from '@deepseek-ai/dsh-app-boot'
import { SessionId } from '@deepseek-ai/dsh-session'
import type {} from '@deepseek-ai/dsh-ptc-runtime'
import type {} from '@deepseek-ai/dsh-agent-preset-registry'
import { expect, it } from 'vitest'
import { launchWebScaffold, type WebScaffold } from './scaffold.ts'
import { connectFreshWorkspaceZh, ZH_BROWSER_LOCALE } from './support.ts'
it.each([undefined, 'minimal'] as const)('adds CUA to PTC sessions while preserving default %s until the explicit persistent-runtime action', async (selectedDefault) => {
const directory = await mkdtemp(join(tmpdir(), 'cua-optin-web-'))
let ownedScaffold: WebScaffold | undefined
let browser: Browser | undefined
try {
const patch = join(directory, 'native-fixture.yml')
// The executable cannot answer native permission queries, so setup must offer retry in the detail page.
await writeFile(patch, JSON.stringify([{ id: 'cua', disabled: false, config: { helperPath: process.execPath } }]))
// Install the test provider explicitly so the helper override also exercises session wiring on Linux.
const scaffold = ownedScaffold = await launchWebScaffold({
profile: { packages: [{ dir: fileURLToPath(new URL('../../../packages/experimental/cua-macos', import.meta.url)), enabled: false }] },
harnessHome: join(directory, 'home'), extraOverlayPath: [
fileURLToPath(new URL('./pin-browse-picker.overlay.yml', import.meta.url)), patch,
] })
const initialDefault = selectedDefault ?? 'standard'
if (selectedDefault !== undefined) await scaffold.ctx.settings.update('agent-preset-registry', { selectedDefault })
browser = await chromium.launch()
const page = await browser.newPage({ viewport: { width: 1680, height: 1000 }, locale: ZH_BROWSER_LOCALE })
await page.goto(scaffold.authenticatedUrl)
await connectFreshWorkspaceZh(page, scaffold.workspaceCwd)
const originalAgent = scaffold.ctx.agents.list()[0]!
const ptcAgent = (await scaffold.ctx.agents.create({ sessionId: SessionId('cua-existing-ptc'),
setup: async (ctx) => { await scaffold.ctx.agentPresets.mount(ctx, 'ptc') },
})).agent
const runtimeFiber = () => [...scaffold.ctx.loader.entries()].find(entry => entry.options.id === 'ptc-runtime')?.fiber
const originalRuntime = scaffold.ctx.ptcRuntime
const originalRuntimeFiber = runtimeFiber()
expect(originalRuntimeFiber).toBeDefined()
expect(scaffold.ctx.ptcRuntime.programFormat).toBe('function-body')
const settingsPath = join(scaffold.harnessHome, 'profiles/scaffold/cordis.patch.yml')
const before = await readFile(settingsPath, 'utf8')
const selected = () => readProfileManifest('scaffold', join(scaffold.harnessHome, 'profiles/scaffold')).dsh?.profile?.bundles
await page.getByRole('navigation', { name: '全局面板' }).getByRole('button', { name: '插件', exact: true }).click()
const card = page.locator('[data-plugin-package="@deepseek-ai/dsh-cua-bundle"]')
const toggle = card.getByRole('switch', { name: '启用 Computer Use', exact: true })
expect(await toggle.getAttribute('aria-checked')).toBe('false')
expect(await card.textContent()).toContain('请切换到 PTC 模式')
// Bundle transitions await native cleanup under the browser action deadline.
await toggle.click()
await toggle.and(page.locator('[aria-checked="true"]')).waitFor({ state: 'visible' })
const setup = page.getByRole('dialog', { name: '使用 Computer Use 前需要设置', exact: true })
await setup.waitFor()
expect(await setup.getByText('Computer Use 插件已开启，但暂时无法检测系统权限。请前往插件详情页查看原因并重试。', { exact: true }).count()).toBe(1)
await setup.getByRole('button', { name: '稍后', exact: true }).click()
expect(selected()).toContain('@deepseek-ai/dsh-cua-bundle')
expect(await readFile(settingsPath, 'utf8')).toBe(before)
expect(runtimeFiber() === originalRuntimeFiber).toBe(true)
expect(scaffold.ctx.ptcRuntime.programFormat).toBe('function-body')
expect(scaffold.ctx.agentPresets.defaultId).toBe(initialDefault)
expect(scaffold.ctx.agents.get(originalAgent.id) === originalAgent).toBe(true)
expect(scaffold.ctx.tools.schemas(originalAgent).some(tool => tool.name === 'cua')).toBe(false)
expect(scaffold.ctx.tools.schemas(ptcAgent).some(tool => tool.name === 'cua')).toBe(true)
expect((await scaffold.ctx.systemPrompt.assemble({ scope: ptcAgent })).tools.map(tool => tool.name)).toEqual(['run_code'])
const fresh = await scaffold.ctx.agents.create({ sessionId: SessionId('cua-new-ptc'),
setup: async (ctx) => { await scaffold.ctx.agentPresets.mount(ctx, 'ptc') },
})
expect(scaffold.ctx.tools.schemas(fresh.agent).some(tool => tool.name === 'cua')).toBe(true)
await toggle.click()
await toggle.and(page.locator('[aria-checked="false"]')).waitFor({ state: 'visible' })
expect(scaffold.ctx.get('computerUse')).toBeUndefined()
for (const agent of [ptcAgent, fresh.agent]) expect(scaffold.ctx.tools.schemas(agent).some(tool => tool.name === 'cua')).toBe(false)
expect(runtimeFiber() === originalRuntimeFiber).toBe(true)
expect(scaffold.ctx.ptcRuntime.programFormat).toBe('function-body')
expect(await readFile(settingsPath, 'utf8')).toBe(before)
await card.getByRole('button', { name: '查看 Computer Use', exact: true }).click()
const detailToggle = page.locator('[data-plugin-panel]').getByRole('switch', { name: '启用 Computer Use', exact: true })
await detailToggle.click()
await detailToggle.and(page.locator('[aria-checked="true"]')).waitFor({ state: 'visible' })
const panel = page.locator('[data-cua-settings]')
await panel.waitFor()
const defaultSetting = panel.getByRole('group', { name: '新会话默认模式', exact: true })
await expect.poll(() => defaultSetting.getByRole('button', { name: '设为 PTC', exact: true }).isEnabled()).toBe(true)
await defaultSetting.getByRole('button', { name: '设为 PTC', exact: true }).click()
await expect.poll(() => scaffold.ctx.agentPresets.defaultId).toBe('ptc')
expect(scaffold.ctx.agentPresets.composedPreset(originalAgent.ctx)).toBe(initialDefault)
expect(runtimeFiber() === originalRuntimeFiber).toBe(true)
expect(scaffold.ctx.ptcRuntime.programFormat).toBe('function-body')
const confirmed = await scaffold.ctx.agents.create({ sessionId: SessionId('cua-confirmed-default'),
setup: async (ctx) => { await scaffold.ctx.agentPresets.mount(ctx) },
})
expect(scaffold.ctx.tools.schemas(confirmed.agent).some(tool => tool.name === 'cua')).toBe(true)
const saved = await readFile(settingsPath, 'utf8')
expect(saved).toContain('selectedDefault: ptc')
expect(saved).not.toContain('executionMode:')
const persistent = panel.getByRole('group', { name: 'PTC 执行方式', exact: true })
await persistent.getByRole('button', { name: '设为持久化', exact: true }).click()
await persistent.getByRole('button', { name: '已设为持久化', exact: true }).waitFor()
expect(scaffold.ctx.ptcRuntime.programFormat).toBe('repl')
expect(scaffold.ctx.ptcRuntime === originalRuntime).toBe(false)
const persistentRuntimeFiber = runtimeFiber()
const runtime = scaffold.ctx.ptcRuntime, owner = {}
const program = 'globalThis.counter = (globalThis.counter ?? 0) + 1'
expect((await runtime.run(runtime.resolve({ program, bindings: [], stateOwner: owner }))).value).toBe(1)
const configured = await readFile(settingsPath, 'utf8')
expect(configured).toContain('executionMode: persistent-in-memory')
expect(scaffold.ctx.agentPresets.defaultId).toBe('ptc')
expect(scaffold.ctx.agentPresets.composedPreset(originalAgent.ctx)).toBe(initialDefault)
await detailToggle.click()
await detailToggle.and(page.locator('[aria-checked="false"]')).waitFor({ state: 'visible' })
expect(await readFile(settingsPath, 'utf8')).toBe(configured)
expect(runtimeFiber() === persistentRuntimeFiber).toBe(true)
expect((await runtime.run(runtime.resolve({ program, bindings: [], stateOwner: owner }))).value).toBe(2)
await detailToggle.click()
await detailToggle.and(page.locator('[aria-checked="true"]')).waitFor({ state: 'visible' })
expect(await readFile(settingsPath, 'utf8')).toBe(configured)
expect(runtimeFiber() === persistentRuntimeFiber).toBe(true)
expect((await runtime.run(runtime.resolve({ program, bindings: [], stateOwner: owner }))).value).toBe(3)
await runtime.release(owner)
} finally {
try { await browser?.close() } finally {
try { await ownedScaffold?.close() } finally { await rm(directory, { recursive: true, force: true }) }
}
}
})
apps/web/tests/cua-permissions.e2e.ts
+150
View File
@@ -0,0 +1,150 @@
/** Built permission UI talks to the real provider; the isolated helper substitutes only macOS privilege observations. */
import { mkdir, mkdtemp, readFile, rm, writeFile } from 'node:fs/promises'
import { tmpdir } from 'node:os'
import { join } from 'node:path'
import { fileURLToPath } from 'node:url'
import { ToolCallId } from '@deepseek-ai/dsh-llm'
import type {} from '@deepseek-ai/dsh-user-approval'
import type {} from '@deepseek-ai/dsh-agent-preset-registry'
import { chromium, type Browser } from 'playwright'
import { expect, it, onTestFailed, onTestFinished } from 'vitest'
import { captureStableAria, compareOrRefreshGolden, launchWebScaffold, webSnapshotMode, watchConsole, type WebScaffold } from './scaffold.ts'
import { connectFreshWorkspace, newEnglishPage, saveFailureShot } from './support.ts'
import { installNativeFixture, startFixtureHost } from '../../../packages/experimental/cua-macos/tests/native-fixture.ts'
const bundle = fileURLToPath(new URL('../../../packages/experimental/cua-bundle', import.meta.url))
const expected = fileURLToPath(new URL('./expected/cua-permissions/missing.expected.md', import.meta.url))
// System-permission onboarding belongs to the macOS-only bundle; rendering and policy units remain portable.
it.skipIf(process.platform !== 'darwin')('keeps system requests pending until fresh observation and persists application revocation', async () => {
const root = await mkdtemp(join(tmpdir(), 'dsh-cua-permissions-web-'))
const resources: { scaffold?: WebScaffold; browser?: Browser; closeFixture?: () => Promise<void> } = {}
onTestFinished(async () => {
try { await resources.browser?.close() } finally {
try { await resources.scaffold?.close() } finally {
try { await resources.closeFixture?.() } finally { await rm(root, { recursive: true, force: true }) }
}
}
})
const helper = await installNativeFixture(root)
resources.closeFixture = await startFixtureHost(helper)
const state = join(root, 'permissions.json'), audit = join(root, 'helper-events.jsonl')
const launchApplication = { name: 'Visual Studio Code.app', path: '/Applications/Visual Studio Code.app',
icon: 'data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAIAAACQd1PeAAAACXBIWXMAAAPoAAAD6AG1e1JrAAAADElEQVQImWNgZGIGAAAOAAeCcsnOAAAAAElFTkSuQmCC' }
await writeFile(state, JSON.stringify({ accessibility: false, screenRecording: false, launchApplication }))
await writeFile(join(root, 'observe-helper'), '')
const fixtureBundle = join(root, 'fixture-bundle')
await mkdir(fixtureBundle)
await writeFile(join(fixtureBundle, 'package.json'), JSON.stringify({ name: 'cua-permission-fixture', version: '1.0.0',
dsh: { bundle: { patch: 'cordis.patch.yml' } } }))
await writeFile(join(fixtureBundle, 'cordis.patch.yml'), `- id: cua\n config: ${JSON.stringify({ helperPath: helper })}\n`)
const scaffold = await launchWebScaffold({
profile: { packages: [{ dir: bundle, enabled: true }, { dir: fixtureBundle, enabled: true }] },
})
resources.scaffold = scaffold
await scaffold.ctx.settings.mutate('cua', [{ op: 'set', path: ['allowedApps'],
value: [{ id: 'a'.repeat(64), name: 'Fixture', executablePath: '/Applications/Fixture.app' }] }])
const browser = await chromium.launch()
resources.browser = browser
const page = await newEnglishPage(browser), tripwire = watchConsole(page)
onTestFailed(() => saveFailureShot(page, 'web-e2e-cua-permissions'))
await page.goto(scaffold.authenticatedUrl)
await connectFreshWorkspace(page, scaffold.workspaceCwd)
const agent = scaffold.ctx.agents.list()[0]!
await scaffold.ctx.agentPresets.select(agent, 'ptc')
scaffold.ctx.approval.setPolicy(agent, 'never')
await scaffold.ctx.settings.mutate('cua', [{ op: 'set', path: ['allowedApps'], value: [] }])
// Application consent remains independent of the session's tool-approval policy.
const grant = scaffold.ctx.tools.execute({ agent, signal: new AbortController().signal,
callId: ToolCallId('cua-app-approval'), name: 'run_code',
arguments: { code: 'return await tools.cua({ method: "getApp", params: { selector: "Fixture" } })',
description: 'Request application access' } })
const approval = page.locator('[data-question-key]')
await Promise.race([approval.waitFor(), grant.then((result) => {
throw new Error(`Application selection completed before consent: ${JSON.stringify(result)}`)
})])
expect(await approval.textContent()).toContain('Fixture')
await approval.getByRole('radio', { name: 'Always allow', exact: true }).click()
await approval.getByRole('button', { name: 'Submit', exact: true }).click()
expect((await grant).isError).toBe(false)
await approval.waitFor({ state: 'hidden' })
await page.getByRole('button', { name: 'Plugins', exact: true }).click()
const card = page.locator('[data-plugin-package="@deepseek-ai/dsh-cua-bundle"]')
expect(await card.getByRole('button', { name: /Accessibility permission|Screen Recording permission/ }).count()).toBe(0)
await page.setViewportSize({ width: 1680, height: 600 })
const plugins = page.locator('[data-plugin-panel]')
const openDetails = page.getByRole('button', { name: 'View Computer Use', exact: true })
await openDetails.scrollIntoViewIfNeeded()
expect(await plugins.evaluate(element => element.scrollTop)).toBeGreaterThan(0)
await openDetails.click()
const panel = page.locator('[data-cua-settings]')
const accessibility = panel.getByRole('group', { name: 'Accessibility', exact: true })
await accessibility.getByRole('button', { name: 'Request permission', exact: true }).waitFor()
expect(await panel.getByRole('region', { name: 'Authorize Computer Use', exact: true }).getByRole('group').count()).toBe(4)
expect(await panel.getByRole('region', { name: 'Manage applications Computer Use can access', exact: true }).count()).toBe(1)
await expect.poll(() => panel.locator('li img').evaluate((image: HTMLImageElement) => image.naturalWidth)).toBe(1)
expect(await panel.getByText('/fixture/app', { exact: true }).count()).toBe(0)
expect(await accessibility.getByText('Visual Studio Code.app', { exact: true }).count()).toBe(1)
expect(await accessibility.textContent()).toContain('In System Settings, allow')
expect(await accessibility.textContent()).toContain('to read interface controls, click, and type.')
expect(await panel.getByRole('group', { name: 'Screen Recording', exact: true }).getByText('Visual Studio Code.app', { exact: true }).count()).toBe(1)
await expect.poll(() => accessibility.locator('img').evaluate((image: HTMLImageElement) => image.naturalWidth)).toBe(1)
await expect.poll(() => panel.getAttribute('aria-busy')).toBe('false')
await compareOrRefreshGolden(expected, await captureStableAria(page, '[data-cua-settings]', scaffold.workspaceCwd), webSnapshotMode())
try {
for (const width of [1680, 390]) {
await page.setViewportSize({ width, height: 1000 })
for (const colorScheme of ['light', 'dark'] as const) {
await page.emulateMedia({ colorScheme })
await expect.poll(() => page.evaluate(() => document.body.hasAttribute('data-ds-dark-theme'))).toBe(colorScheme === 'dark')
const measure = () => panel.evaluate(element => ({
clientWidth: element.clientWidth, scrollWidth: element.scrollWidth,
overflowing: [...element.querySelectorAll('*')].filter(child =>
child.getBoundingClientRect().right > element.getBoundingClientRect().right + 1,
).map(child => ({ tag: child.tagName, className: child.className, text: child.textContent?.slice(0, 80) })),
}))
await expect.poll(async () => {
const layout = await measure()
return layout.scrollWidth <= layout.clientWidth ? true : { width, colorScheme, ...layout }
}).toBe(true)
if (webSnapshotMode() === 'refresh') {
const directory = fileURLToPath(new URL('../../../.artifacts', import.meta.url))
await mkdir(directory, { recursive: true })
const path = join(directory, `cua-permission-design-${process.pid}-${width}-${colorScheme}.png`)
await panel.screenshot({ path, animations: 'disabled' })
console.log(`CUA permission design: ${path}`)
}
}
}
} finally {
await page.emulateMedia({ colorScheme: null })
await page.setViewportSize({ width: 1680, height: 1000 })
}
await accessibility.getByRole('button', { name: 'Request permission', exact: true }).click()
await expect.poll(() => panel.getAttribute('aria-busy')).toBe('false')
expect(await accessibility.getByRole('button', { name: 'Request permission', exact: true }).count()).toBe(1)
await writeFile(state, JSON.stringify({ accessibility: true, screenRecording: false, launchApplication }))
// Headless pages retain document focus; model returning from System Settings at the window-event boundary.
await page.evaluate(() => { window.dispatchEvent(new Event('focus')) })
await accessibility.getByText('Allowed', { exact: true }).waitFor()
const screen = panel.getByRole('group', { name: 'Screen Recording', exact: true })
expect(await accessibility.getByRole('button').count()).toBe(1)
await accessibility.getByRole('button', { name: 'Allowed', exact: true }).click()
expect(await screen.getByRole('button').count()).toBe(1)
await expect.poll(() => panel.getAttribute('aria-busy')).toBe('false')
await panel.getByRole('button', { name: 'Revoke', exact: true }).click()
await panel.getByRole('heading', { name: 'Always-allowed applications', exact: true }).waitFor({ state: 'hidden' })
await panel.getByRole('heading', { name: 'Applications not allowed', exact: true }).waitFor()
await panel.getByRole('button', { name: 'Allow asking again', exact: true }).click()
await panel.getByText('Fixture', { exact: true }).waitFor({ state: 'hidden' })
const saved = await readFile(join(scaffold.harnessHome, 'profiles/scaffold/cordis.patch.yml'), 'utf8')
expect(saved).toContain('allowedApps: []')
const events = (await readFile(audit, 'utf8')).trim().split('\n').map(line => JSON.parse(line) as { event: string; method: string; params: object })
.filter(record => record.event === 'call').map(({ method, params }) => ({ method, params }))
expect(events).toContainEqual({ method: 'requestPermission', params: { permission: 'accessibility' } })
expect(events.filter(event => event.method === 'openPermissionSettings')).toEqual([
{ method: 'openPermissionSettings', params: { permission: 'accessibility' } },
])
expect(events.every(event => ['permissionStatus', 'requestPermission', 'openPermissionSettings', 'applicationIcons', 'getApp'].includes(event.method))).toBe(true)
expect(tripwire.pageErrors).toEqual([])
}, 60_000)
apps/web/tests/expected/cua-permissions/missing.expected.md
+26
View File
@@ -0,0 +1,26 @@
- region "Authorize Computer Use":
- heading "Authorize Computer Use" [level=3]
- group "Default mode for new sessions":
- strong: Default mode for new sessions
- paragraph: Computer Use is available only in PTC mode.
- button "Set to PTC"
- group "PTC execution mode":
- strong: PTC execution mode
- paragraph: Keep variables and application objects across tool calls. Switching restarts the PTC runtime.
- button "Use persistent mode"
- group "Accessibility":
- strong: Accessibility
- paragraph: In System Settings, allow Visual Studio Code.app to read interface controls, click, and type.
- button "Request permission"
- group "Screen Recording":
- strong: Screen Recording
- paragraph: In System Settings, allow Visual Studio Code.app to capture application windows.
- button "Request permission"
- paragraph: Enable the application in System Settings. Follow the macOS prompt if it names another application.
- region "Manage applications Computer Use can access":
- heading "Manage applications Computer Use can access" [level=3]
- heading "Always-allowed applications" [level=4]
- list:
- listitem:
- strong: Fixture
- button "Revoke"
apps/web/tests/expected/plugin-config/official.expected.md
+4
View File
@@ -38,6 +38,10 @@
- button "查看 开发者工具": 开发者工具
- text: 实验性 查看调试会话原始数据、聊天消息分组数据，以及调试 NodeJS 后端
- switch "启用 开发者工具"
- listitem:
- button "查看 Computer Use": Computer Use
- text: 目前仅支持 macOS，Windows 支持正在开发中。为已有及新建 PTC 会话加载 Computer Use。请切换到 PTC 模式来使用，并授予辅助功能和屏幕录制权限。
- switch "启用 Computer Use"
- heading "已安装插件" [level=3]
- list:
- listitem:
apps/web/tests/expected/plugin-manager/live-enabled.expected.md
+4
View File
@@ -38,6 +38,10 @@
- button "查看 开发者工具": 开发者工具
- text: 实验性 查看调试会话原始数据、聊天消息分组数据，以及调试 NodeJS 后端
- switch "启用 开发者工具"
- listitem:
- button "查看 Computer Use": Computer Use
- text: 目前仅支持 macOS，Windows 支持正在开发中。为已有及新建 PTC 会话加载 Computer Use。请切换到 PTC 模式来使用，并授予辅助功能和屏幕录制权限。
- switch "启用 Computer Use"
- heading "已安装插件" [level=3]
- list:
- listitem:
apps/web/tests/expected/plugin-manager/manager.expected.md
+4
View File
@@ -38,6 +38,10 @@
- button "查看 开发者工具": 开发者工具
- text: 实验性 查看调试会话原始数据、聊天消息分组数据，以及调试 NodeJS 后端
- switch "启用 开发者工具"
- listitem:
- button "查看 Computer Use": Computer Use
- text: 目前仅支持 macOS，Windows 支持正在开发中。为已有及新建 PTC 会话加载 Computer Use。请切换到 PTC 模式来使用，并授予辅助功能和屏幕录制权限。
- switch "启用 Computer Use"
- heading "已安装插件" [level=3]
- list:
- listitem:
apps/web/tests/expected/plugin-manager/refresh.expected.md
+2 -2
View File
@@ -6,7 +6,7 @@ Focus at 500ms:
- tooltip "刷新"
failure: pending
- button "刷新" [disabled]
{"busy":"true","spinners":1,"cards":11,"skeletons":0,"alerts":[],"retry":0,"toasts":[]}
{"busy":"true","spinners":1,"cards":12,"skeletons":0,"alerts":[],"retry":0,"toasts":[]}
failure: settled
- button "刷新"
{"busy":"false","spinners":0,"cards":11,"skeletons":0,"alerts":[],"retry":0,"toasts":["刷新失败，请重试"]}
{"busy":"false","spinners":0,"cards":12,"skeletons":0,"alerts":[],"retry":0,"toasts":["刷新失败，请重试"]}
apps/web/tests/preview-boot.e2e.ts
+16 -3
View File
@@ -681,15 +681,28 @@ async function toggleOptionalBundles(page: Page): Promise<void> {
}
})
expect(result.optional.map(entry => entry.name).sort()).toEqual([...OPTIONAL_BUNDLES].sort())
expect(result.optional.filter(entry => entry.enabled || entry.availability !== 'installation' || entry.error !== null)).toEqual([])
expect(result.optional.filter(entry => entry.enabled || entry.availability !== 'installation')).toEqual([])
expect(result.optional.filter(entry => entry.error !== null)).toEqual([])
// Metadata resolves through the Worker's Node internal modules: every optional bundle shows its title and icon.
expect(result.optional.filter(entry => entry.metaError !== null || entry.title !== 'object' || entry.icon !== 'string')).toEqual([])
expect(result.inventoryErrors).toEqual([])
console.info(`preview boot: optional bundle toggles ${JSON.stringify(result.toggles.map(({ name, enable, disable, error }) => ({ name, enable, disable, error })))}`)
expect(result.toggles.filter(toggle => toggle.enable !== 'applied' || toggle.disable !== 'applied')
const cua = result.toggles.find(toggle => toggle.name === '@deepseek-ai/dsh-cua-bundle')!
expect(cua).toMatchObject({
enable: 'applied', disable: 'applied',
error: null,
restored: true,
})
expect(cua.changed).toBeGreaterThan(0)
expect(cua.inactive).toEqual([])
expect(cua.plugins).toContainEqual(expect.objectContaining({
moduleName: '@deepseek-ai/dsh-experimental-cua-macos', enabled: false, fiberPhase: null,
}))
const supported = result.toggles.filter(toggle => toggle.name !== '@deepseek-ai/dsh-cua-bundle')
expect(supported.filter(toggle => toggle.enable !== 'applied' || toggle.disable !== 'applied')
.map(({ name, enable, disable }) => ({ name, enable, disable }))).toEqual([])
// Each bundle changes Loader rows, every enabled row it changes runs, and switching it off restores the prior rows.
expect(result.toggles.filter(toggle => toggle.changed === 0 || toggle.inactive.length > 0 || !toggle.restored)
expect(supported.filter(toggle => toggle.changed === 0 || toggle.inactive.length > 0 || !toggle.restored)
.map(({ name, changed, inactive, restored }) => ({ name, changed, inactive, restored }))).toEqual([])
const pluginsWith = (bundle: string): Array<{ moduleName: string; enabled: boolean; fiberPhase: string | null }> =>
result.toggles.find(toggle => toggle.name === bundle)?.plugins ?? []
apps/web/tsconfig.json
+2
View File
@@ -82,6 +82,8 @@
"tests/plugin-config.e2e.ts",
"tests/input-plugin-extensions.e2e.ts",
"tests/plugin-manager.e2e.ts",
"tests/cua-optin.e2e.ts",
"tests/cua-permissions.e2e.ts",
"tests/schedule-header.e2e.ts",
"tests/worktree-opt-in.e2e.ts",
"tests/session-inspector.e2e.ts",
docs/capability-seams.i18n.yaml
+2 -2
View File
@@ -6,5 +6,5 @@
en: 41d772192075e133
zh: 5ef3da69b4de1d8a
/capability-seams-and-core-services:
en: f209321ab0da4f90
zh: 4af5c135eec51ce1
en: 753b3da3abc6aeea
zh: aab02ec16c3f744b
docs/capability-seams.md
+5
View File
@@ -52,6 +52,8 @@ flowchart LR
svc_computerUse["ctx.computerUse<br/>Computer-use provider registration"]
pkg_experimental_computer_use_cua_driver_mcp["experimental-computer-use-cua-driver-mcp"]
pkg_experimental_computer_use_cua_driver_native["experimental-computer-use-cua-driver-native"]
pkg_cua["cua"]
svc_cuaAccess["ctx.cuaAccess<br/>Computer Use permission controls"]
pkg_office_to_pdf["office-to-pdf"]
svc_officeToPdf["ctx.officeToPdf<br/>Office to PDF conversion"]
pkg_client_ui_sidebar_documentpreview["client-ui-sidebar-documentpreview"]
@@ -334,6 +336,7 @@ flowchart LR
pkg_cordis_host_runner --> svc_dynamicCordisRunner
pkg_credentials --> svc_credentials
pkg_credentials_local --> svc_credentials
pkg_cua --> svc_cuaAccess
pkg_deepseek_account --> svc_deepseekAccount
pkg_deepseek_account_platform --> svc_deepseekAccount
pkg_deepseek_llm_api_extensions --> svc_deepseekLlmApiExtensions
@@ -484,6 +487,7 @@ flowchart LR
svc_credentials --> pkg_api_settings_controller
svc_credentials --> pkg_llm_deepseek
svc_credentials --> pkg_llm_pi_ai
svc_cuaAccess --> pkg_cua
svc_deepseekAccount --> pkg_api_account_controller
svc_deepseekAccount --> pkg_llm_deepseek
svc_deepseekLlmApiExtensions --> pkg_llm_deepseek
@@ -628,6 +632,7 @@ flowchart LR
| `ctx.mcpResources` | `seam` | [`mcp-resources`](../packages/mcp/mcp-resources) | [`mcp-client`](../packages/mcp/mcp-client) | [`mcp-resources`](../packages/mcp/mcp-resources) | - | Connection-owned providers serve shared resource tools in the calling agent scope. |
| `ctx.browserUse` | `seam` | [`browser-use`](../packages/browser-use/browser-use) | [`experimental-browser-use-playwright-mcp`](../packages/experimental/browser-use-playwright-mcp), [`experimental-browser-use-chrome-devtools-mcp`](../packages/experimental/browser-use-chrome-devtools-mcp), [`experimental-browser-use-stagehand-native`](../packages/experimental/browser-use-stagehand-native) | [`experimental-browser-use-playwright-mcp`](../packages/experimental/browser-use-playwright-mcp), [`experimental-browser-use-chrome-devtools-mcp`](../packages/experimental/browser-use-chrome-devtools-mcp), [`experimental-browser-use-stagehand-native`](../packages/experimental/browser-use-stagehand-native) | - | One provider-owned name per service instance. Providers own their tools and browser resources per live Session; the shared service has no browser operation API. |
| `ctx.computerUse` | `seam` | [`computer-use`](../packages/computer-use/computer-use) | [`experimental-computer-use-cua-driver-mcp`](../packages/experimental/computer-use-cua-driver-mcp), [`experimental-computer-use-cua-driver-native`](../packages/experimental/computer-use-cua-driver-native) | [`experimental-computer-use-cua-driver-mcp`](../packages/experimental/computer-use-cua-driver-mcp), [`experimental-computer-use-cua-driver-native`](../packages/experimental/computer-use-cua-driver-native) | - | One provider-owned name per service instance. Each provider also owns its model tools; the service has no common action API, runtime selection, or Session workflow lock. |
| `ctx.cuaAccess` | `core` | [`cua`](../packages/experimental/cua) | - | [`cua`](../packages/experimental/cua) | - | The native provider owns application consent and cancellation; the optional Client page queries and changes those decisions through its Remote. |
| `ctx.officeToPdf` | `core` | [`office-to-pdf`](../packages/document/office-to-pdf) | - | [`client-ui-sidebar-documentpreview`](../packages/client/ui-sidebar-documentpreview) | - | Authorized Office bytes are converted on the Host using the declared native target engine, or Node WASM when no native target is declared. |
| `ctx.attachments` | `seam` | [`attachment`](../packages/attachment/attachment) | [`attachment-local`](../packages/attachment/attachment-local) | [`api-session-controller`](../packages/api/session-controller), [`tool-fs`](../packages/fs/tool-fs), [`llm-pi-ai`](../packages/llm/llm-pi-ai), [`llm-deepseek`](../packages/llm/llm-deepseek) | - | The host commits accepted images before session events; provider adapters resolve authorized durable references into provider-native content. |
| `ctx.fileUploads` | `core` | [`client-file-upload`](../packages/client/file-upload) | - | [`api-session-controller`](../packages/api/session-controller) | - | Owns streaming intake, durable storage, and staged receipt lifetime; the Session controller binds receipts to accepted submissions. |
docs/capability-seams.zh.md
+5
View File
@@ -54,6 +54,8 @@ flowchart LR
svc_computerUse["ctx.computerUse<br/>Computer-use provider registration"]
pkg_experimental_computer_use_cua_driver_mcp["experimental-computer-use-cua-driver-mcp"]
pkg_experimental_computer_use_cua_driver_native["experimental-computer-use-cua-driver-native"]
pkg_cua["cua"]
svc_cuaAccess["ctx.cuaAccess<br/>Computer Use permission controls"]
pkg_office_to_pdf["office-to-pdf"]
svc_officeToPdf["ctx.officeToPdf<br/>Office to PDF conversion"]
pkg_client_ui_sidebar_documentpreview["client-ui-sidebar-documentpreview"]
@@ -336,6 +338,7 @@ flowchart LR
pkg_cordis_host_runner --> svc_dynamicCordisRunner
pkg_credentials --> svc_credentials
pkg_credentials_local --> svc_credentials
pkg_cua --> svc_cuaAccess
pkg_deepseek_account --> svc_deepseekAccount
pkg_deepseek_account_platform --> svc_deepseekAccount
pkg_deepseek_llm_api_extensions --> svc_deepseekLlmApiExtensions
@@ -486,6 +489,7 @@ flowchart LR
svc_credentials --> pkg_api_settings_controller
svc_credentials --> pkg_llm_deepseek
svc_credentials --> pkg_llm_pi_ai
svc_cuaAccess --> pkg_cua
svc_deepseekAccount --> pkg_api_account_controller
svc_deepseekAccount --> pkg_llm_deepseek
svc_deepseekLlmApiExtensions --> pkg_llm_deepseek
@@ -630,6 +634,7 @@ flowchart LR
| `ctx.mcpResources` | `seam` | [`mcp-resources`](../packages/mcp/mcp-resources) | [`mcp-client`](../packages/mcp/mcp-client) | [`mcp-resources`](../packages/mcp/mcp-resources) | - | 连接所有者提供的操作在调用 agent 的作用域内服务于共享资源工具。 |
| `ctx.browserUse` | `seam` | [`browser-use`](../packages/browser-use/browser-use) | [`experimental-browser-use-playwright-mcp`](../packages/experimental/browser-use-playwright-mcp), [`experimental-browser-use-chrome-devtools-mcp`](../packages/experimental/browser-use-chrome-devtools-mcp), [`experimental-browser-use-stagehand-native`](../packages/experimental/browser-use-stagehand-native) | [`experimental-browser-use-playwright-mcp`](../packages/experimental/browser-use-playwright-mcp), [`experimental-browser-use-chrome-devtools-mcp`](../packages/experimental/browser-use-chrome-devtools-mcp), [`experimental-browser-use-stagehand-native`](../packages/experimental/browser-use-stagehand-native) | - | 每个服务实例注册一个提供方拥有的名称。提供方按实时 Session 拥有自己的工具与浏览器资源；共享服务不提供浏览器操作 API。 |
| `ctx.computerUse` | `seam` | [`computer-use`](../packages/computer-use/computer-use) | [`experimental-computer-use-cua-driver-mcp`](../packages/experimental/computer-use-cua-driver-mcp), [`experimental-computer-use-cua-driver-native`](../packages/experimental/computer-use-cua-driver-native) | [`experimental-computer-use-cua-driver-mcp`](../packages/experimental/computer-use-cua-driver-mcp), [`experimental-computer-use-cua-driver-native`](../packages/experimental/computer-use-cua-driver-native) | - | 每个服务实例只注册一个提供方自定的名称。各提供方也拥有自己的模型工具；服务不提供通用操作 API、运行时选择或 Session 流程锁。 |
| `ctx.cuaAccess` | `core` | [`cua`](../packages/experimental/cua) | - | [`cua`](../packages/experimental/cua) | - | 原生 provider 负责应用授权和取消操作；可选的 Client 权限页面通过 Remote 查询和修改授权。 |
| `ctx.officeToPdf` | `core` | [`office-to-pdf`](../packages/document/office-to-pdf) | - | [`client-ui-sidebar-documentpreview`](../packages/client/ui-sidebar-documentpreview) | - | 已授权的 Office 字节在宿主上使用已声明的原生目标引擎转换；未声明原生目标时使用 Node WASM。 |
| `ctx.attachments` | `seam` | [`attachment`](../packages/attachment/attachment) | [`attachment-local`](../packages/attachment/attachment-local) | [`api-session-controller`](../packages/api/session-controller), [`tool-fs`](../packages/fs/tool-fs), [`llm-pi-ai`](../packages/llm/llm-pi-ai), [`llm-deepseek`](../packages/llm/llm-deepseek) | - | 宿主会在会话事件之前提交已接受的图片；提供方适配器将已授权的持久引用解析为提供方原生内容。 |
| `ctx.fileUploads` | `core` | [`client-file-upload`](../packages/client/file-upload) | - | [`api-session-controller`](../packages/api/session-controller) | - | 负责流式接收、持久存储和暂存回执生命周期；Session Controller 将回执绑定到已接受的提交。 |
docs/config-catalog.md
+2 -1
View File
@@ -4762,6 +4762,7 @@ These load from a `cordis.yml` entry with no `config:` block; they declare no co
| `@deepseek-ai/dsh-computer-use` | — | [`packages/computer-use/computer-use/src/index.ts`](../packages/computer-use/computer-use/src/index.ts) |
| `@deepseek-ai/dsh-config-editor` | `loader` · `profileContext` | [`packages/boot/config-editor/src/index.ts`](../packages/boot/config-editor/src/index.ts) |
| `@deepseek-ai/dsh-cordis-client-runner` | — | [`packages/extensions/cordis-client-runner/src/index.ts`](../packages/extensions/cordis-client-runner/src/index.ts) |
| `@deepseek-ai/dsh-cua` | — | [`packages/experimental/cua/src/index.ts`](../packages/experimental/cua/src/index.ts) |
| `@deepseek-ai/dsh-deepseek-llm-api-extensions` | — | [`packages/llm/deepseek-llm-api-extensions/src/index.ts`](../packages/llm/deepseek-llm-api-extensions/src/index.ts) |
| `@deepseek-ai/dsh-experimental-auto-review` | `approval` · `llm` · `permissionPresets` · `sessions` · `tools` · `workingDirectory` | [`packages/experimental/auto-review/src/index.ts`](../packages/experimental/auto-review/src/index.ts) |
| `@deepseek-ai/dsh-experimental-client-ui-agent-team` | — | [`packages/experimental/client-ui-agent-team/src/index.ts`](../packages/experimental/client-ui-agent-team/src/index.ts) |
@@ -4849,7 +4850,7 @@ Imported as libraries by other packages; a `cordis.yml` cannot load them.
| `@deepseek-ai/dsh-client-ui-slots` | — | [`packages/client/ui-slots/src/index.ts`](../packages/client/ui-slots/src/index.ts) |
| `@deepseek-ai/dsh-client-web` | — | [`packages/client/web/src/index.ts`](../packages/client/web/src/index.ts) |
| `@deepseek-ai/dsh-cmdline` | — | [`packages/boot/cmdline/src/index.ts`](../packages/boot/cmdline/src/index.ts) |
| `@deepseek-ai/dsh-cua` | — | [`packages/experimental/cua/src/index.ts`](../packages/experimental/cua/src/index.ts) |
| `@deepseek-ai/dsh-cua-bundle` | — | [`packages/experimental/cua-bundle/src/index.ts`](../packages/experimental/cua-bundle/src/index.ts) |
| `@deepseek-ai/dsh-deque` | — | [`packages/util/deque/src/index.ts`](../packages/util/deque/src/index.ts) |
| `@deepseek-ai/dsh-experimental-agent-team-profile` | — | [`packages/experimental/agent-team-profile/src/index.ts`](../packages/experimental/agent-team-profile/src/index.ts) |
| `@deepseek-ai/dsh-experimental-badge-skill-bundle` | — | [`packages/experimental/badge-skill-bundle/src/index.ts`](../packages/experimental/badge-skill-bundle/src/index.ts) |
docs/config-catalog.zh.md
+2 -1
View File
@@ -4764,6 +4764,7 @@ export interface Config {
| `@deepseek-ai/dsh-computer-use` | — | [`packages/computer-use/computer-use/src/index.ts`](../packages/computer-use/computer-use/src/index.ts) |
| `@deepseek-ai/dsh-config-editor` | `loader` · `profileContext` | [`packages/boot/config-editor/src/index.ts`](../packages/boot/config-editor/src/index.ts) |
| `@deepseek-ai/dsh-cordis-client-runner` | — | [`packages/extensions/cordis-client-runner/src/index.ts`](../packages/extensions/cordis-client-runner/src/index.ts) |
| `@deepseek-ai/dsh-cua` | — | [`packages/experimental/cua/src/index.ts`](../packages/experimental/cua/src/index.ts) |
| `@deepseek-ai/dsh-deepseek-llm-api-extensions` | — | [`packages/llm/deepseek-llm-api-extensions/src/index.ts`](../packages/llm/deepseek-llm-api-extensions/src/index.ts) |
| `@deepseek-ai/dsh-experimental-auto-review` | `approval` · `llm` · `permissionPresets` · `sessions` · `tools` · `workingDirectory` | [`packages/experimental/auto-review/src/index.ts`](../packages/experimental/auto-review/src/index.ts) |
| `@deepseek-ai/dsh-experimental-client-ui-agent-team` | — | [`packages/experimental/client-ui-agent-team/src/index.ts`](../packages/experimental/client-ui-agent-team/src/index.ts) |
@@ -4851,7 +4852,7 @@ export interface Config {
| `@deepseek-ai/dsh-client-ui-slots` | — | [`packages/client/ui-slots/src/index.ts`](../packages/client/ui-slots/src/index.ts) |
| `@deepseek-ai/dsh-client-web` | — | [`packages/client/web/src/index.ts`](../packages/client/web/src/index.ts) |
| `@deepseek-ai/dsh-cmdline` | — | [`packages/boot/cmdline/src/index.ts`](../packages/boot/cmdline/src/index.ts) |
| `@deepseek-ai/dsh-cua` | — | [`packages/experimental/cua/src/index.ts`](../packages/experimental/cua/src/index.ts) |
| `@deepseek-ai/dsh-cua-bundle` | — | [`packages/experimental/cua-bundle/src/index.ts`](../packages/experimental/cua-bundle/src/index.ts) |
| `@deepseek-ai/dsh-deque` | — | [`packages/util/deque/src/index.ts`](../packages/util/deque/src/index.ts) |
| `@deepseek-ai/dsh-experimental-agent-team-profile` | — | [`packages/experimental/agent-team-profile/src/index.ts`](../packages/experimental/agent-team-profile/src/index.ts) |
| `@deepseek-ai/dsh-experimental-badge-skill-bundle` | — | [`packages/experimental/badge-skill-bundle/src/index.ts`](../packages/experimental/badge-skill-bundle/src/index.ts) |
docs/event-producer-consumer.md
+2 -2
View File
@@ -9,9 +9,9 @@ This matrix shows which packages dispatch each harness-owned event and which pac
| Event | Mode | Declared in | Dispatchers | Listeners |
| --- | --- | --- | --- | --- |
| `agent-loop/config-start-failed` | `emit` | [`packages/core/agent-loop/src/index.ts:196`](../packages/core/agent-loop/src/index.ts) | [`agent-loop`](../packages/core/agent-loop) (`events.dispatch`) | - |
| `agent-preset/selected` | `emit` | [`packages/preset/agent-preset-registry/src/types.ts:66`](../packages/preset/agent-preset-registry/src/types.ts) | [`agent-preset-registry`](../packages/preset/agent-preset-registry) (`emit`) | `remotes` |
| `agent-preset/selected` | `emit` | [`packages/preset/agent-preset-registry/src/types.ts:66`](../packages/preset/agent-preset-registry/src/types.ts) | [`agent-preset-registry`](../packages/preset/agent-preset-registry) (`emit`) | [`cua`](../packages/experimental/cua), `remotes` |
| `agent/assistant-stream` | `emit` | [`packages/core/agent/src/runtime-types.ts:363`](../packages/core/agent/src/runtime-types.ts) | [`agent-loop`](../packages/core/agent-loop) (`emit`) | [`headless`](../packages/bundle/headless), `session-controller` |
| `agent/created` | `serial` | [`packages/core/agent/src/runtime-types.ts:261`](../packages/core/agent/src/runtime-types.ts) | [`agent`](../packages/core/agent) (`serial`) | `agent-team`, `browser-use-runtime`, `claude-code-mods`, [`file-reference-local`](../packages/context/file-reference-local), [`goal`](../packages/goal/goal), [`goal-round-driver`](../packages/goal/goal-round-driver), [`hooks-claude-code`](../packages/experimental/hooks-claude-code), [`hooks-codex`](../packages/experimental/hooks-codex), [`loader-smoke`](../packages/test-support/loader-smoke), `session-controller`, `tool-agent-team`, [`tool-subagent`](../packages/subagent/tool-subagent) |
| `agent/created` | `serial` | [`packages/core/agent/src/runtime-types.ts:261`](../packages/core/agent/src/runtime-types.ts) | [`agent`](../packages/core/agent) (`serial`) | `agent-team`, `browser-use-runtime`, `claude-code-mods`, [`cua`](../packages/experimental/cua), [`file-reference-local`](../packages/context/file-reference-local), [`goal`](../packages/goal/goal), [`goal-round-driver`](../packages/goal/goal-round-driver), [`hooks-claude-code`](../packages/experimental/hooks-claude-code), [`hooks-codex`](../packages/experimental/hooks-codex), [`loader-smoke`](../packages/test-support/loader-smoke), `session-controller`, `tool-agent-team`, [`tool-subagent`](../packages/subagent/tool-subagent) |
| `agent/disposed` | `emit` | [`packages/core/agent/src/runtime-types.ts:270`](../packages/core/agent/src/runtime-types.ts) | [`agent`](../packages/core/agent) (`events.dispatch`) | [`agent-loop`](../packages/core/agent-loop), `claude-code-mods`, [`file-reference-local`](../packages/context/file-reference-local), [`goal-round-driver`](../packages/goal/goal-round-driver), `session-controller`, [`subagent`](../packages/subagent/subagent), `tool-agent-team`, [`tool-subagent`](../packages/subagent/tool-subagent) |
| `agent/error` | `emit` | [`packages/core/agent/src/runtime-types.ts:393`](../packages/core/agent/src/runtime-types.ts) | [`agent-loop`](../packages/core/agent-loop) (`emit`) | [`acp`](../packages/acp/acp), [`deepseek-account`](../packages/credentials/deepseek-account), [`goal-round-driver`](../packages/goal/goal-round-driver), [`headless`](../packages/bundle/headless), `server`, `session-controller`, [`session-telemetry`](../packages/session/session-telemetry), [`subagent`](../packages/subagent/subagent) |
| `agent/inbox/claimed` | `emit` | [`packages/core/agent/src/runtime-types.ts:299`](../packages/core/agent/src/runtime-types.ts) | [`agent-loop`](../packages/core/agent-loop) (`emit`) | [`acp`](../packages/acp/acp), [`goal-round-driver`](../packages/goal/goal-round-driver), [`subagent`](../packages/subagent/subagent), [`tool-jobs`](../packages/jobs/tool-jobs), [`user-questions`](../packages/interaction/user-questions) |
docs/event-producer-consumer.zh.md
+2 -2
View File
@@ -11,9 +11,9 @@
| Event | Mode | Declared in | Dispatchers | Listeners |
| --- | --- | --- | --- | --- |
| `agent-loop/config-start-failed` | `emit` | [`packages/core/agent-loop/src/index.ts:196`](../packages/core/agent-loop/src/index.ts) | [`agent-loop`](../packages/core/agent-loop) (`events.dispatch`) | - |
| `agent-preset/selected` | `emit` | [`packages/preset/agent-preset-registry/src/types.ts:66`](../packages/preset/agent-preset-registry/src/types.ts) | [`agent-preset-registry`](../packages/preset/agent-preset-registry) (`emit`) | `remotes` |
| `agent-preset/selected` | `emit` | [`packages/preset/agent-preset-registry/src/types.ts:66`](../packages/preset/agent-preset-registry/src/types.ts) | [`agent-preset-registry`](../packages/preset/agent-preset-registry) (`emit`) | [`cua`](../packages/experimental/cua), `remotes` |
| `agent/assistant-stream` | `emit` | [`packages/core/agent/src/runtime-types.ts:363`](../packages/core/agent/src/runtime-types.ts) | [`agent-loop`](../packages/core/agent-loop) (`emit`) | [`headless`](../packages/bundle/headless), `session-controller` |
| `agent/created` | `serial` | [`packages/core/agent/src/runtime-types.ts:261`](../packages/core/agent/src/runtime-types.ts) | [`agent`](../packages/core/agent) (`serial`) | `agent-team`, `browser-use-runtime`, `claude-code-mods`, [`file-reference-local`](../packages/context/file-reference-local), [`goal`](../packages/goal/goal), [`goal-round-driver`](../packages/goal/goal-round-driver), [`hooks-claude-code`](../packages/experimental/hooks-claude-code), [`hooks-codex`](../packages/experimental/hooks-codex), [`loader-smoke`](../packages/test-support/loader-smoke), `session-controller`, `tool-agent-team`, [`tool-subagent`](../packages/subagent/tool-subagent) |
| `agent/created` | `serial` | [`packages/core/agent/src/runtime-types.ts:261`](../packages/core/agent/src/runtime-types.ts) | [`agent`](../packages/core/agent) (`serial`) | `agent-team`, `browser-use-runtime`, `claude-code-mods`, [`cua`](../packages/experimental/cua), [`file-reference-local`](../packages/context/file-reference-local), [`goal`](../packages/goal/goal), [`goal-round-driver`](../packages/goal/goal-round-driver), [`hooks-claude-code`](../packages/experimental/hooks-claude-code), [`hooks-codex`](../packages/experimental/hooks-codex), [`loader-smoke`](../packages/test-support/loader-smoke), `session-controller`, `tool-agent-team`, [`tool-subagent`](../packages/subagent/tool-subagent) |
| `agent/disposed` | `emit` | [`packages/core/agent/src/runtime-types.ts:270`](../packages/core/agent/src/runtime-types.ts) | [`agent`](../packages/core/agent) (`events.dispatch`) | [`agent-loop`](../packages/core/agent-loop), `claude-code-mods`, [`file-reference-local`](../packages/context/file-reference-local), [`goal-round-driver`](../packages/goal/goal-round-driver), `session-controller`, [`subagent`](../packages/subagent/subagent), `tool-agent-team`, [`tool-subagent`](../packages/subagent/tool-subagent) |
| `agent/error` | `emit` | [`packages/core/agent/src/runtime-types.ts:393`](../packages/core/agent/src/runtime-types.ts) | [`agent-loop`](../packages/core/agent-loop) (`emit`) | [`acp`](../packages/acp/acp), [`deepseek-account`](../packages/credentials/deepseek-account), [`goal-round-driver`](../packages/goal/goal-round-driver), [`headless`](../packages/bundle/headless), `server`, `session-controller`, [`session-telemetry`](../packages/session/session-telemetry), [`subagent`](../packages/subagent/subagent) |
| `agent/inbox/claimed` | `emit` | [`packages/core/agent/src/runtime-types.ts:299`](../packages/core/agent/src/runtime-types.ts) | [`agent-loop`](../packages/core/agent-loop) (`emit`) | [`acp`](../packages/acp/acp), [`goal-round-driver`](../packages/goal/goal-round-driver), [`subagent`](../packages/subagent/subagent), [`tool-jobs`](../packages/jobs/tool-jobs), [`user-questions`](../packages/interaction/user-questions) |
docs/module-graph.md
+6 -9
View File
@@ -248,6 +248,7 @@ flowchart TD
end
subgraph group_experimental["packages/experimental"]
pkg_cua["cua"]
pkg_cua_bundle["cua-bundle"]
pkg_experimental_agent_team["experimental-agent-team"]
pkg_experimental_agent_team_profile["experimental-agent-team-profile"]
pkg_experimental_api_speech_to_text["experimental-api-speech-to-text"]
@@ -760,6 +761,9 @@ flowchart TD
pkg_deepseek_account_platform --> pkg_credentials
pkg_deepseek_account_platform --> pkg_deepseek_account
pkg_deepseek_account_platform --> pkg_host_webserver
pkg_cua --> pkg_attachment
pkg_cua --> pkg_scope
pkg_cua --> pkg_user_questions
pkg_experimental_ptc_runtime_python --> pkg_ptc_runtime
pkg_experimental_ptc_runtime_python --> pkg_sandbox
pkg_experimental_ptc_runtime_python --> pkg_sandbox_policy
@@ -903,14 +907,6 @@ flowchart TD
pkg_workspace_changes --> pkg_session
pkg_workspace_changes --> pkg_subprocess
pkg_workspace_changes --> pkg_tools
pkg_cua --> pkg_agent
pkg_cua --> pkg_attachment
pkg_cua --> pkg_computer_use
pkg_cua --> pkg_llm
pkg_cua --> pkg_subprocess
pkg_cua --> pkg_system_prompt
pkg_cua --> pkg_tools
pkg_cua --> pkg_user_questions
pkg_experimental_browser_use_chrome_devtools_mcp --> pkg_agent
pkg_experimental_browser_use_chrome_devtools_mcp --> pkg_browser_use
pkg_experimental_browser_use_chrome_devtools_mcp --> pkg_system_prompt
@@ -1537,6 +1533,7 @@ flowchart TD
| [`client-web`](../packages/client/web) | `client` | — |
| [`credentials`](../packages/credentials/credentials) | `credentials` | — |
| [`office-to-pdf`](../packages/document/office-to-pdf) | `document` | — |
| [`cua-bundle`](../packages/experimental/cua-bundle) | `experimental` | — |
| [`experimental-agent-team-profile`](../packages/experimental/agent-team-profile) | `experimental` | — |
| [`experimental-badge-skill-bundle`](../packages/experimental/badge-skill-bundle) | `experimental` | — |
| [`experimental-client-ui-agent-team`](../packages/experimental/client-ui-agent-team) | `experimental` | — |
@@ -1674,6 +1671,7 @@ flowchart TD
| [`api-account-controller`](../packages/api/account-controller) | `api` | [`agent`](../packages/core/agent), [`deepseek-account`](../packages/credentials/deepseek-account), [`typert-protocol`](../packages/typert/protocol) |
| [`compaction`](../packages/compaction/compaction) | `compaction` | [`brand`](../packages/util/brand), [`commands`](../packages/interaction/commands), [`llm`](../packages/llm/llm), [`session`](../packages/core/session) |
| [`deepseek-account-platform`](../packages/credentials/deepseek-account-platform) | `credentials` | [`authorization`](../packages/credentials/authorization), [`credentials`](../packages/credentials/credentials), [`deepseek-account`](../packages/credentials/deepseek-account), [`host-webserver`](../packages/host/webserver) |
| [`cua`](../packages/experimental/cua) | `experimental` | [`attachment`](../packages/attachment/attachment), [`scope`](../packages/core/scope), [`user-questions`](../packages/interaction/user-questions) |
| [`experimental-ptc-runtime-python`](../packages/experimental/ptc-runtime-python) | `experimental` | [`ptc-runtime`](../packages/ptc-runtime/ptc-runtime), [`sandbox`](../packages/sandbox/sandbox), [`sandbox-policy`](../packages/sandbox/sandbox-policy), [`timeout`](../packages/util/timeout), [`util-values`](../packages/util/values) |
| [`experimental-worktree`](../packages/experimental/worktree) | `experimental` | [`agent`](../packages/core/agent), [`fs`](../packages/fs/fs), [`sandbox`](../packages/sandbox/sandbox), [`sandbox-policy`](../packages/sandbox/sandbox-policy), [`subprocess`](../packages/subprocess/subprocess), [`working-directory`](../packages/session/working-directory) |
| [`command-feedback`](../packages/feedback/command-feedback) | `feedback` | [`anonymous-user-id`](../packages/identity/anonymous-user-id), [`commands`](../packages/interaction/commands), [`session`](../packages/core/session), [`typert-protocol`](../packages/typert/protocol) |
@@ -1701,7 +1699,6 @@ flowchart TD
| [`file-reference-local`](../packages/context/file-reference-local) | `context` | [`agent`](../packages/core/agent), [`file-reference`](../packages/context/file-reference), [`system-prompt`](../packages/core/system-prompt), [`tools`](../packages/core/tools), [`working-directory`](../packages/session/working-directory) |
| [`tool-present`](../packages/deliverables/tool-present) | `deliverables` | [`agent`](../packages/core/agent), [`fs`](../packages/fs/fs), [`llm`](../packages/llm/llm), [`session`](../packages/core/session), [`tools`](../packages/core/tools), [`working-directory`](../packages/session/working-directory) |
| [`workspace-changes`](../packages/deliverables/workspace-changes) | `deliverables` | [`agent`](../packages/core/agent), [`fs`](../packages/fs/fs), [`session`](../packages/core/session), [`subprocess`](../packages/subprocess/subprocess), [`tools`](../packages/core/tools) |
| [`cua`](../packages/experimental/cua) | `experimental` | [`agent`](../packages/core/agent), [`attachment`](../packages/attachment/attachment), [`computer-use`](../packages/computer-use/computer-use), [`llm`](../packages/llm/llm), [`subprocess`](../packages/subprocess/subprocess), [`system-prompt`](../packages/core/system-prompt), [`tools`](../packages/core/tools), [`user-questions`](../packages/interaction/user-questions) |
| [`experimental-browser-use-chrome-devtools-mcp`](../packages/experimental/browser-use-chrome-devtools-mcp) | `experimental` | [`agent`](../packages/core/agent), [`browser-use`](../packages/browser-use/browser-use), [`system-prompt`](../packages/core/system-prompt), [`tools`](../packages/core/tools), [`working-directory`](../packages/session/working-directory) |
| [`experimental-browser-use-playwright-mcp`](../packages/experimental/browser-use-playwright-mcp) | `experimental` | [`agent`](../packages/core/agent), [`browser-use`](../packages/browser-use/browser-use), [`system-prompt`](../packages/core/system-prompt), [`tools`](../packages/core/tools), [`working-directory`](../packages/session/working-directory) |
| [`experimental-browser-use-stagehand-native`](../packages/experimental/browser-use-stagehand-native) | `experimental` | [`agent`](../packages/core/agent), [`browser-use`](../packages/browser-use/browser-use), [`system-prompt`](../packages/core/system-prompt), [`tools`](../packages/core/tools) |
docs/module-graph.zh.md
+6 -9
View File
@@ -250,6 +250,7 @@ flowchart TD
end
subgraph group_experimental["packages/experimental"]
pkg_cua["cua"]
pkg_cua_bundle["cua-bundle"]
pkg_experimental_agent_team["experimental-agent-team"]
pkg_experimental_agent_team_profile["experimental-agent-team-profile"]
pkg_experimental_api_speech_to_text["experimental-api-speech-to-text"]
@@ -762,6 +763,9 @@ flowchart TD
pkg_deepseek_account_platform --> pkg_credentials
pkg_deepseek_account_platform --> pkg_deepseek_account
pkg_deepseek_account_platform --> pkg_host_webserver
pkg_cua --> pkg_attachment
pkg_cua --> pkg_scope
pkg_cua --> pkg_user_questions
pkg_experimental_ptc_runtime_python --> pkg_ptc_runtime
pkg_experimental_ptc_runtime_python --> pkg_sandbox
pkg_experimental_ptc_runtime_python --> pkg_sandbox_policy
@@ -905,14 +909,6 @@ flowchart TD
pkg_workspace_changes --> pkg_session
pkg_workspace_changes --> pkg_subprocess
pkg_workspace_changes --> pkg_tools
pkg_cua --> pkg_agent
pkg_cua --> pkg_attachment
pkg_cua --> pkg_computer_use
pkg_cua --> pkg_llm
pkg_cua --> pkg_subprocess
pkg_cua --> pkg_system_prompt
pkg_cua --> pkg_tools
pkg_cua --> pkg_user_questions
pkg_experimental_browser_use_chrome_devtools_mcp --> pkg_agent
pkg_experimental_browser_use_chrome_devtools_mcp --> pkg_browser_use
pkg_experimental_browser_use_chrome_devtools_mcp --> pkg_system_prompt
@@ -1539,6 +1535,7 @@ flowchart TD
| [`client-web`](../packages/client/web) | `client` | — |
| [`credentials`](../packages/credentials/credentials) | `credentials` | — |
| [`office-to-pdf`](../packages/document/office-to-pdf) | `document` | — |
| [`cua-bundle`](../packages/experimental/cua-bundle) | `experimental` | — |
| [`experimental-agent-team-profile`](../packages/experimental/agent-team-profile) | `experimental` | — |
| [`experimental-badge-skill-bundle`](../packages/experimental/badge-skill-bundle) | `experimental` | — |
| [`experimental-client-ui-agent-team`](../packages/experimental/client-ui-agent-team) | `experimental` | — |
@@ -1676,6 +1673,7 @@ flowchart TD
| [`api-account-controller`](../packages/api/account-controller) | `api` | [`agent`](../packages/core/agent), [`deepseek-account`](../packages/credentials/deepseek-account), [`typert-protocol`](../packages/typert/protocol) |
| [`compaction`](../packages/compaction/compaction) | `compaction` | [`brand`](../packages/util/brand), [`commands`](../packages/interaction/commands), [`llm`](../packages/llm/llm), [`session`](../packages/core/session) |
| [`deepseek-account-platform`](../packages/credentials/deepseek-account-platform) | `credentials` | [`authorization`](../packages/credentials/authorization), [`credentials`](../packages/credentials/credentials), [`deepseek-account`](../packages/credentials/deepseek-account), [`host-webserver`](../packages/host/webserver) |
| [`cua`](../packages/experimental/cua) | `experimental` | [`attachment`](../packages/attachment/attachment), [`scope`](../packages/core/scope), [`user-questions`](../packages/interaction/user-questions) |
| [`experimental-ptc-runtime-python`](../packages/experimental/ptc-runtime-python) | `experimental` | [`ptc-runtime`](../packages/ptc-runtime/ptc-runtime), [`sandbox`](../packages/sandbox/sandbox), [`sandbox-policy`](../packages/sandbox/sandbox-policy), [`timeout`](../packages/util/timeout), [`util-values`](../packages/util/values) |
| [`experimental-worktree`](../packages/experimental/worktree) | `experimental` | [`agent`](../packages/core/agent), [`fs`](../packages/fs/fs), [`sandbox`](../packages/sandbox/sandbox), [`sandbox-policy`](../packages/sandbox/sandbox-policy), [`subprocess`](../packages/subprocess/subprocess), [`working-directory`](../packages/session/working-directory) |
| [`command-feedback`](../packages/feedback/command-feedback) | `feedback` | [`anonymous-user-id`](../packages/identity/anonymous-user-id), [`commands`](../packages/interaction/commands), [`session`](../packages/core/session), [`typert-protocol`](../packages/typert/protocol) |
@@ -1703,7 +1701,6 @@ flowchart TD
| [`file-reference-local`](../packages/context/file-reference-local) | `context` | [`agent`](../packages/core/agent), [`file-reference`](../packages/context/file-reference), [`system-prompt`](../packages/core/system-prompt), [`tools`](../packages/core/tools), [`working-directory`](../packages/session/working-directory) |
| [`tool-present`](../packages/deliverables/tool-present) | `deliverables` | [`agent`](../packages/core/agent), [`fs`](../packages/fs/fs), [`llm`](../packages/llm/llm), [`session`](../packages/core/session), [`tools`](../packages/core/tools), [`working-directory`](../packages/session/working-directory) |
| [`workspace-changes`](../packages/deliverables/workspace-changes) | `deliverables` | [`agent`](../packages/core/agent), [`fs`](../packages/fs/fs), [`session`](../packages/core/session), [`subprocess`](../packages/subprocess/subprocess), [`tools`](../packages/core/tools) |
| [`cua`](../packages/experimental/cua) | `experimental` | [`agent`](../packages/core/agent), [`attachment`](../packages/attachment/attachment), [`computer-use`](../packages/computer-use/computer-use), [`llm`](../packages/llm/llm), [`subprocess`](../packages/subprocess/subprocess), [`system-prompt`](../packages/core/system-prompt), [`tools`](../packages/core/tools), [`user-questions`](../packages/interaction/user-questions) |
| [`experimental-browser-use-chrome-devtools-mcp`](../packages/experimental/browser-use-chrome-devtools-mcp) | `experimental` | [`agent`](../packages/core/agent), [`browser-use`](../packages/browser-use/browser-use), [`system-prompt`](../packages/core/system-prompt), [`tools`](../packages/core/tools), [`working-directory`](../packages/session/working-directory) |
| [`experimental-browser-use-playwright-mcp`](../packages/experimental/browser-use-playwright-mcp) | `experimental` | [`agent`](../packages/core/agent), [`browser-use`](../packages/browser-use/browser-use), [`system-prompt`](../packages/core/system-prompt), [`tools`](../packages/core/tools), [`working-directory`](../packages/session/working-directory) |
| [`experimental-browser-use-stagehand-native`](../packages/experimental/browser-use-stagehand-native) | `experimental` | [`agent`](../packages/core/agent), [`browser-use`](../packages/browser-use/browser-use), [`system-prompt`](../packages/core/system-prompt), [`tools`](../packages/core/tools) |
docs/subsystems/computer-use.i18n.yaml
+2 -2
View File
@@ -3,8 +3,8 @@
# After editing either side, bring the other along and re-record with:
# pnpm run verify-translation-pairing --write docs/subsystems/computer-use.md
/computer-use:
en: a82a6d1ec3fa9724
zh: 17483872a54d6899
en: aff84e3ac8c41594
zh: bee7fc09854bfa8b
/computer-use/choose-a-provider:
en: fa2e0d7b41c0450b
zh: af9a799711f901e6
docs/subsystems/computer-use.md
+76
View File
@@ -4,9 +4,19 @@ English | [中文](computer-use.zh.md)
Computer use lets a model observe and operate the local desktop through a configured provider. The shared DSH capability is called **computer use**; **Cua Driver** names the upstream implementation.
[Computer Use](../../packages/experimental/cua-bundle/README.md) provides an optional experimental switch for macOS CUA permission setup.
The [`macOS CUA provider`](../../packages/experimental/cua-macos/README.md) uses the [shared CUA objects and executor](../../packages/experimental/cua/README.md), which enforce application consent through User Questions and profile configuration.
The CUA provider owns `ctx.cuaAccess` for system-permission status, explicit system requests, and application revocation. Its [permission page](../../packages/experimental/cua/README.md) projects these controls; app/window operations enforce the provider's consent policy before dispatch.
| Permission type | Meaning |
|---|---|
| `CuaApplicationId`, `CuaApplicationGrant` | An executable/signature identity and its displayed saved decision. |
| `CuaSystemPermission` | Accessibility or Screen Recording, selected by the human. |
| `CuaPermissionProcess`, `CuaSystemPermissionState` | Native privilege checks, observed helper/parent processes, and the application identified by macOS process responsibility at helper startup. Application metadata is absent if the optional system query or application lookup is unavailable. |
| `CuaApplicationState` | Saved grants, refused applications, and whether persistence is available. Session-only grants remain internal to the provider. System observations and application icons are queried separately so the page can display each result as it arrives. |
## Choose a provider
Mount [`dsh-computer-use`](../../packages/computer-use/computer-use/README.md) and one provider in the same composition. Both Cua Driver providers are experimental public npm packages and require explicit activation.
@@ -56,4 +66,70 @@ register(name: ComputerUseProviderName): () => Promise<void>
```
Source: [`packages/computer-use/computer-use/src/index.ts`](../../packages/computer-use/computer-use/src/index.ts)
<a id="ctxcuaaccess--cuaaccess"></a>
### `ctx.cuaAccess` — `CuaAccess`
Browser controls for provider-owned native permission and application decisions.
```ts cordis-catalog
/** Read PTC execution state separately from native permission checks.
* @returns Whether the shared runtime retains memory and has an editable profile entry.
*/
@Remote ptcConfiguration(): { persistent: boolean; writable: boolean }
/** Persist in-memory PTC execution through the existing profile editor and Loader lifecycle.
* @returns Completion after the shared runtime reload; already-persistent runtimes are left intact.
*/
@Remote async enablePersistentPtc(): Promise<void>
/**
* Read application decisions without waiting for the native helper.
* @returns Saved grants and refused applications with their persistence owner.
*/
@Remote applications(): CuaApplicationState
/**
* Inspect system privileges independently of application decisions and icons.
* @returns Fresh native preflight and launch metadata.
*/
@Remote async systemStatus(): Promise<CuaSystemPermissionState>
/**
* Load presentation icons separately from application decisions.
* @returns Optional native PNG icons keyed by executable path for saved and blocked applications.
*/
@Remote async applicationIcons(): Promise<Record<string, string>>
/**
* Request a system privilege explicitly selected by the user.
* @param permission macOS privilege to request.
* @returns Preflight state after requesting; a false value never implies consent.
*/
@Remote async requestPermission(permission: CuaSystemPermission): Promise<CuaSystemPermissionState>
/**
* Open the macOS privacy pane selected by the user.
* @param permission Corresponding system privilege.
* @returns Completion after macOS accepted the open request.
*/
@Remote async openSettings(permission: CuaSystemPermission): Promise<void>
/**
* Remove saved and session access and join affected operations.
* @param id Application identity displayed on the permission page.
* @returns Completion after persistence and cancellation; dispatched input cannot be undone.
*/
@Remote revoke(id: CuaApplicationId): Promise<void>
/**
* Clear this application's refusals across sessions so each session may ask for access again.
* @param id Refused or revoked application displayed on the permission page.
* @returns Completion after clearing refusals; no permission is granted and no action is replayed.
*/
@Remote async retryApproval(id: CuaApplicationId): Promise<void>
```
Source: [`packages/experimental/cua/src/access.ts`](../../packages/experimental/cua/src/access.ts)
<!-- END GENERATED cordis-surface -->
docs/subsystems/computer-use.zh.md
+76
View File
@@ -4,9 +4,19 @@
计算机操作让模型通过配置的提供方观察并操作本地桌面。DSH 的共享能力称为 **computer use（计算机操作）**；**Cua Driver** 是上游实现的名称。
[Computer Use](../../packages/experimental/cua-bundle/README.zh.md) 提供 macOS CUA 及权限设置 的可选实验性开关。
[macOS CUA 提供方](../../packages/experimental/cua-macos/README.zh.md)使用[共享 CUA 对象与执行器](../../packages/experimental/cua/README.zh.md)，通过 User Questions 和配置档设置执行应用授权。
CUA provider 通过 `ctx.cuaAccess` 提供系统权限状态、用户主动申请和应用授权撤销。[权限页面](../../packages/experimental/cua/README.zh.md)投影这些控件；app/window 操作在分派前执行 provider 的授权检查。
| 权限类型 | 含义 |
|---|---|
| `CuaApplicationId`、`CuaApplicationGrant` | 可执行文件和签名身份，以及显示给用户的已保存决定。 |
| `CuaSystemPermission` | 用户选择申请的辅助功能或屏幕录制权限。 |
| `CuaPermissionProcess`、`CuaSystemPermissionState` | 观测到的 helper、父进程和原生权限状态，以及 helper 启动时由 macOS 责任进程记录识别出的应用。可选系统查询或应用查找不可用时，省略应用信息。 |
| `CuaApplicationState` | 持久授权、拒绝记录，以及能否持久保存。会话授权只保留在 provider 内部。系统权限和应用图标分别查询，页面在各项结果返回后分别显示。 |
## 选择提供方
在同一组合中挂载 [`dsh-computer-use`](../../packages/computer-use/computer-use/README.zh.md) 和一个提供方。两个 Cua Driver 提供方都是公开发布到 npm 的实验性包，均需显式启用。
@@ -56,4 +66,70 @@ register(name: ComputerUseProviderName): () => Promise<void>
```
Source: [`packages/computer-use/computer-use/src/index.ts`](../../packages/computer-use/computer-use/src/index.ts)
<a id="ctxcuaaccess--cuaaccess"></a>
### `ctx.cuaAccess` — `CuaAccess`
Browser controls for provider-owned native permission and application decisions.
```ts cordis-catalog
/** Read PTC execution state separately from native permission checks.
* @returns Whether the shared runtime retains memory and has an editable profile entry.
*/
@Remote ptcConfiguration(): { persistent: boolean; writable: boolean }
/** Persist in-memory PTC execution through the existing profile editor and Loader lifecycle.
* @returns Completion after the shared runtime reload; already-persistent runtimes are left intact.
*/
@Remote async enablePersistentPtc(): Promise<void>
/**
* Read application decisions without waiting for the native helper.
* @returns Saved grants and refused applications with their persistence owner.
*/
@Remote applications(): CuaApplicationState
/**
* Inspect system privileges independently of application decisions and icons.
* @returns Fresh native preflight and launch metadata.
*/
@Remote async systemStatus(): Promise<CuaSystemPermissionState>
/**
* Load presentation icons separately from application decisions.
* @returns Optional native PNG icons keyed by executable path for saved and blocked applications.
*/
@Remote async applicationIcons(): Promise<Record<string, string>>
/**
* Request a system privilege explicitly selected by the user.
* @param permission macOS privilege to request.
* @returns Preflight state after requesting; a false value never implies consent.
*/
@Remote async requestPermission(permission: CuaSystemPermission): Promise<CuaSystemPermissionState>
/**
* Open the macOS privacy pane selected by the user.
* @param permission Corresponding system privilege.
* @returns Completion after macOS accepted the open request.
*/
@Remote async openSettings(permission: CuaSystemPermission): Promise<void>
/**
* Remove saved and session access and join affected operations.
* @param id Application identity displayed on the permission page.
* @returns Completion after persistence and cancellation; dispatched input cannot be undone.
*/
@Remote revoke(id: CuaApplicationId): Promise<void>
/**
* Clear this application's refusals across sessions so each session may ask for access again.
* @param id Refused or revoked application displayed on the permission page.
* @returns Completion after clearing refusals; no permission is granted and no action is replayed.
*/
@Remote async retryApproval(id: CuaApplicationId): Promise<void>
```
Source: [`packages/experimental/cua/src/access.ts`](../../packages/experimental/cua/src/access.ts)
<!-- END GENERATED cordis-surface -->
docs/upgrade-guide/v0.2.1-rc.2/cua-preset-scope/guide.i18n.yaml
+16
View File
@@ -0,0 +1,16 @@
# Bilingual-pair consistency record for guide.md (docs/i18n/README.md): per heading
# section, a hash of its English and Chinese blocks outside code blocks and generated regions.
# After editing either side, bring the other along and re-record with:
# pnpm run verify-translation-pairing --write docs/upgrade-guide/v0.2.1-rc.2/cua-preset-scope/guide.md
/:
en: 36d6b79b48c08dd5
zh: 814caece159e9b0e
/cua-defaults-to-ptc-session-scopes:
en: a53b6884785bdf56
zh: 7d27fc9b029e0e3e
/cua-defaults-to-ptc-session-scopes/change:
en: dac96a29e3751249
zh: d48cb030f43c07e4
/cua-defaults-to-ptc-session-scopes/migration:
en: f3d82fe45e41c712
zh: e55e424f5a2c096a
docs/upgrade-guide/v0.2.1-rc.2/cua-preset-scope/guide.md
+18
View File
@@ -0,0 +1,18 @@
---
kind: upgrade-guide
description: "Standalone native CUA compositions must explicitly select global tool registration."
---
# CUA defaults to PTC session scopes
English | [中文](guide.zh.md)
## Change
The native CUA provider defaults `agentPreset` to `ptc`. It registers tools for existing and new PTC Agents, including when a saved application-grants configuration omits the new field. Custom compositions without an Agent preset registry no longer receive global CUA tools by default.
## Migration
1. For the standard Computer Use switch, keep the default; existing application grants remain usable.
2. For a standalone composition that needs global tools, add `agentPreset: false` under the native CUA entry’s `config` in its profile patch or overlay. Preserve its other fields, including `allowedApps` and `helperPath`.
3. Restart or let HMR apply the edit, then confirm that the intended Agent exposes `cua` in native mode or through the PTC SDK.
docs/upgrade-guide/v0.2.1-rc.2/cua-preset-scope/guide.zh.md
+18
View File
@@ -0,0 +1,18 @@
---
kind: upgrade-guide
description: "独立原生 CUA 组合需要显式选择全局工具注册。"
---
# CUA 默认使用 PTC 会话作用域
[English](guide.md) | 中文
## 变更
原生 CUA 提供方的 `agentPreset` 默认为 `ptc`。工具注册到已有和新建 PTC Agent，即使保存的应用授权配置没有这个新字段也会生效。不包含 Agent preset 注册表的自定义组合不再默认获得全局 CUA 工具。
## 迁移
1. 使用标准 Computer Use 开关时保持默认值，已有应用授权仍可使用。
2. 独立组合若需要全局工具，在 profile patch 或 overlay 的原生 CUA 条目 `config` 下添加 `agentPreset: false`，保留 `allowedApps`、`helperPath` 等其他字段。
3. 重启或等待 HMR 应用后，确认目标 Agent 能在原生模式或 PTC SDK 中使用 `cua`。
package.json
+1
View File
@@ -291,6 +291,7 @@
"@deepseek-ai/dsh-working-directory": "workspace:*",
"@deepseek-ai/dsh-experimental-worktree": "workspace:*",
"@deepseek-ai/dsh-experimental-tool-worktree": "workspace:*",
"@deepseek-ai/dsh-cua-bundle": "workspace:*",
"@deepseek-ai/dsh-experimental-subagent-oar": "workspace:*",
"@deepseek-ai/dsh-experimental-tool-subagent-oar": "workspace:*",
"@deepseek-ai/dsh-experimental-cua-macos": "workspace:*"
packages/boot/app-boot/README.i18n.yaml
+2 -2
View File
@@ -24,8 +24,8 @@
en: 4a424cd401556f20
zh: 0b34dd18e0db6cea
/deepseek-ai-dsh-app-boot/use-this-package/profiles:
en: d2d520668668292c
zh: b7deeab27032a3ad
en: 0a1b5f4f40d835a7
zh: d8f4b54160aa033b
/deepseek-ai-dsh-app-boot/use-this-package/previewing-the-effective-configuration:
en: 646a0ccdd7881ad1
zh: e3696acb1573dd7b
packages/boot/app-boot/README.md
+1 -1
View File
@@ -64,7 +64,7 @@ The enabled `dsh-hmr` plugin watches the profile manifest and both user patch fi
Inserted plugin names may be absolute filesystem paths, file URLs, or package specifiers. Patch loading converts absolute paths and patch-relative `./` or `../` paths to file URLs within `insert` rows and their nested groups; existing-entry name assertions and replacement `config` values remain literal.
Before mounting profile rows, the `dsh` launcher computes one immutable runtime resolution from the installation and ordered bundle dependency graphs. Every profile launcher uses runtime resolution, including plain Node, packaged executables, and the Electron Host. It installs the runtime resolution through Node's ESM and CommonJS resolvers without creating fallback links.
Before mounting profile rows, the `dsh` launcher computes one immutable runtime resolution from the installation and ordered bundle dependency graphs. Every profile launcher uses runtime resolution, including plain Node, packaged executables, and the Electron Host. It installs the runtime resolution through Node's ESM and CommonJS resolvers without creating fallback links. Traversal includes installed optional dependencies and skips those absent from the target installation.
`sanitizeProfile(binName, profileDir, bundles)` provides filesystem recovery without loading plugins or parsing patches. Desktop uses it for native fatal recovery. Call it only after stopping the profile and excluding concurrent profile writes. It renames the profile’s `cordis.patch.yml` to a unique `.bak-<timestamp>` sibling and restores the supplied bundle list, preserving installed packages and other manifest fields. The timestamp is Unix time in milliseconds; collisions append an ordinal (`-1`, `-2`, …) without changing it. It returns the backup path, or `undefined` when no patch exists; missing profiles remain absent. Profile initialization recreates an empty patch on the next launch. The home-level patch is unchanged. Invalid profile JSON fails before mutation; later errors propagate and retain completed changes for retry.
packages/boot/app-boot/README.zh.md
+1 -1
View File
@@ -64,7 +64,7 @@ profile 导入插件前，DSH 会将其声明的 DSH 版本要求与 `getDshRunt
插入条目的插件名可以是绝对文件系统路径、文件 URL 或包标识符。patch 加载会把 `insert` 条目及其嵌套分组中的绝对路径以及相对于 patch 文件的 `./` 或 `../` 路径转换为文件 URL；对已有条目名称的断言及替换用的 `config` 值保持原样。
挂载 profile 条目前，`dsh` launcher 会从安装依赖图与有序 bundle 依赖图计算一份不可变的 runtime resolution。普通 Node、打包可执行文件与 Electron Host 等所有 profile 启动器都使用 runtime 解析，将 runtime resolution 安装到 Node 的 ESM 与 CommonJS 解析器中，不创建 fallback 链接。
挂载 profile 条目前，`dsh` launcher 会从安装依赖图与有序 bundle 依赖图计算一份不可变的 runtime resolution。普通 Node、打包可执行文件与 Electron Host 等所有 profile 启动器都使用 runtime 解析，将 runtime resolution 安装到 Node 的 ESM 与 CommonJS 解析器中，不创建 fallback 链接。依赖遍历包含已安装的可选依赖，跳过目标安装中缺席的包。
`sanitizeProfile(binName, profileDir, bundles)` 提供文件恢复，无需加载插件或解析 patch。Desktop 在原生致命错误恢复中调用它。调用前必须停止 profile 并排除并发 profile 写入。它将 profile 的 `cordis.patch.yml` 重命名为带唯一 `.bak-<timestamp>` 后缀的同目录备份，并恢复调用方指定的 bundle 列表，保留已安装包和其他 manifest 字段。时间戳为 Unix 毫秒数；同名备份已存在时追加序号（`-1`、`-2`、……），时间戳保持不变。返回值为备份路径；patch 不存在时返回 `undefined`，缺失的 profile 不会被创建。下次启动的 profile 初始化会重新创建空 patch。home 级 patch 不变。无效 profile JSON 在修改前报错；后续错误向调用方抛出，保留已完成的修改供重试。
packages/boot/app-boot/src/profile.ts
+3 -1
View File
@@ -221,6 +221,7 @@ export const DEFAULT_PROFILE_BUNDLES: readonly string[] = ['@deepseek-ai/dsh-bas
* [admission](../../../../.agents/notes/implemented/architecture/2026-09-21-experimental-capabilities-as-optional-bundles.md)).
*/
export const OPTIONAL_BUNDLES: readonly string[] = [
'@deepseek-ai/dsh-cua-bundle',
'@deepseek-ai/dsh-experimental-session-search',
'@deepseek-ai/dsh-experimental-ralph-bundle',
'@deepseek-ai/dsh-experimental-terminal-bundle',
@@ -374,7 +375,8 @@ function readPackageManifest(anchor: string): ProfileManifest {
/** Return dependency names that may be imported by a loader-visible plugin. */
function profileDependencyNames(manifest: ProfileManifest): string[] {
return [...Object.keys(manifest.dependencies ?? {}), ...Object.keys(manifest.peerDependencies ?? {})]
return [...Object.keys(manifest.dependencies ?? {}), ...Object.keys(manifest.peerDependencies ?? {}),
...Object.keys(manifest.optionalDependencies ?? {})]
}
/** Resolve the installation packages that the runtime resolver supplies to every profile. */
packages/boot/app-boot/tests/profile-resolution-service.spec.ts
+25
View File
@@ -334,3 +334,28 @@ describe('profile package metadata service', () => {
expect(getEnvironmentData(key)).toBe(previous)
})
})
it.each(['installation', 'bundle'] as const)('exposes installed optional dependencies from %s and skips absent ones', async (scope) => {
const root = realpathSync.native(mkdtempSync(join(tmpdir(), 'dsh-optional-resolution-')))
roots.push(root)
const installAnchor = join(root, 'install', 'package.json')
file(installAnchor, JSON.stringify({ name: 'installation', version: '1.0.0' }))
const profileDir = join(root, 'profiles', 'test')
initProfile(profileDir, ['optional-bundle'])
const bundleDir = join(profileDir, 'node_modules', 'optional-bundle')
const bundleManifest = { name: 'optional-bundle', version: '1.0.0', dsh: { bundle: { patch: './cordis.patch.yml' } } }
file(join(bundleDir, 'package.json'), JSON.stringify(bundleManifest))
file(join(bundleDir, 'cordis.patch.yml'), '[]\n')
const anchor = scope === 'installation' ? installAnchor : join(bundleDir, 'package.json')
file(anchor, JSON.stringify({ ...JSON.parse(readFileSync(anchor, 'utf8')) as object,
optionalDependencies: { 'metadata-lib': '*', 'absent-platform': '*' },
}))
const optionalDir = join(dirname(anchor), 'node_modules', 'metadata-lib')
pkg(optionalDir, '1.0.0')
const active = await createRuntimeResolution({
installAnchor, home: root, profile: loadProfileDirectory('dsh', profileDir, installAnchor),
})
expect(active.entries).toContainEqual(expect.objectContaining({ name: 'metadata-lib', packageDir: optionalDir }))
expect(active.entries.some(entry => entry.name === 'absent-platform')).toBe(false)
})
packages/computer-use/README.i18n.yaml
+4 -4
View File
@@ -9,8 +9,8 @@
en: 77ea90624a004773
zh: fe6dbf8414a5aff0
/packages-computer-use/summary:
en: 3820da56a601d3f1
zh: 358e65f8a7cf7446
en: 7e5fcf9bb48c9677
zh: abee2be288115045
/packages-computer-use/table-of-contents:
en: 1eb35b015a6c423f
zh: a8a3dceee2c9e943
@@ -18,8 +18,8 @@
en: 9aa86e6451f60c3d
zh: c5e6816e90f2d74b
/packages-computer-use/related-documentation:
en: ad08d3417a0d4630
zh: cfe8e03033c2fede
en: 58198f9f17e6120f
zh: 188fa0ec5c4678f3
/packages-computer-use/dev-note:
en: b75c8458928b72b9
zh: 003ba4e17b4b5269
packages/computer-use/README.md
+3 -2
View File
@@ -9,7 +9,7 @@ English | [中文](README.zh.md)
## Summary
Computer-use providers let models observe and operate a desktop. This group owns exclusive provider registration. Each provider owns its operations, tools, and platform requirements; the experimental Cua Driver providers live in the experimental group.
Computer-use providers let models observe and operate a desktop. This group contains the shared registry that prevents multiple providers from being enabled in one composition. DSH's [CUA implementation](../experimental/cua/README.md), platform providers, and opt-in bundle currently live in `experimental`, alongside the Cua Driver adapters. The CUA package documents the platform requirements for promotion to this group.
## Table of Contents
@@ -32,7 +32,8 @@ Choose one provider and mount the shared registration service.
## Related documentation
- [Computer use](../../docs/subsystems/computer-use.md) — capability ownership and provider choices.
- [Experimental packages](../experimental/README.md) — Cua Driver MCP and native providers.
- [Computer Use bundle](../experimental/cua-bundle/README.md) — enable native CUA and access its permission controls.
- [Experimental packages](../experimental/README.md) — native CUA packages and Cua Driver adapters.
<a id="dev-note"></a>
## Dev Note
packages/computer-use/README.zh.md
+3 -2
View File
@@ -9,7 +9,7 @@ kind: "package-group"
## 概述
计算机操作提供方让模型观察并操作桌面。本组负责提供方的独占注册。各提供方拥有自己的操作、工具和平台要求；实验性 Cua Driver 提供方位于 experimental 组。
计算机操作提供方让模型观察并操作桌面。本组包含共享注册器，防止同一组合同时启用多个提供方。DSH 的 [CUA 实现](../experimental/cua/README.zh.md)、平台提供方和启用组合包目前位于 `experimental`，Cua Driver 适配包也位于该组。CUA 包文档说明了迁入本组所需的平台支持条件。
## 目录
@@ -32,7 +32,8 @@ kind: "package-group"
## 相关文档
- [计算机操作](../../docs/subsystems/computer-use.zh.md) — 能力归属和提供方选择。
- [实验性包](../experimental/README.zh.md) — Cua Driver MCP 和原生提供方。
- [Computer Use 组合包](../experimental/cua-bundle/README.zh.md)：启用原生 CUA 并使用权限管理功能。
- [实验性包](../experimental/README.zh.md)：原生 CUA 包和 Cua Driver 适配包。
<a id="dev-note"></a>
## 开发备注
packages/computer-use/computer-use/README.i18n.yaml
+4 -4
View File
@@ -9,8 +9,8 @@
en: 2efee0c5f0d577bc
zh: fef4b81bfc121052
/deepseek-ai-dsh-computer-use/summary:
en: 4de047849e68701f
zh: 90a32c0e279b4b05
en: 4cb331c550852b23
zh: d638995ab250397b
/deepseek-ai-dsh-computer-use/table-of-contents:
en: d152484eb41ac6b4
zh: 405b30feaf48c087
@@ -21,8 +21,8 @@
en: 181679887837591a
zh: cc2e96766ce0aa63
/deepseek-ai-dsh-computer-use/further-exploration:
en: b1bdbcbc150036b3
zh: 1d356e0522587abc
en: 987295d677466c04
zh: 9d7b40a04786490e
/deepseek-ai-dsh-computer-use/model-experience:
en: 8780c250dfaefaaa
zh: 1c4b3598a131fb65
packages/computer-use/computer-use/README.md
+4 -1
View File
@@ -9,7 +9,7 @@ English | [中文](README.zh.md)
## Summary
A deployment can enable one computer-use provider at a time. Loading another provider fails with the registered provider name. Each provider supplies its own tools and desktop operations. This package adds no model-visible tools and does not coordinate concurrent Sessions.
A deployment can enable one computer-use provider at a time. Loading another provider fails with the registered provider name. Both DSH's native CUA and the Cua Driver adapters use this registry. Each provider supplies its own tools and desktop operations. This package adds no model-visible tools and does not coordinate concurrent Sessions.
## Table of Contents
@@ -53,6 +53,9 @@ One private name owns the slot. Cordis effects remove contributions when their p
## Further Exploration
- [Computer use](../../../docs/subsystems/computer-use.md) — provider selection and shared-desktop limits.
- [Computer Use bundle](../../experimental/cua-bundle/README.md) — enable DSH's native CUA and permission controls.
- [Shared CUA library](../../experimental/cua/README.md) — application/window objects, tool execution, and application consent.
- [macOS CUA provider](../../experimental/cua-macos/README.md) — DSH's Swift implementation and native helper lifecycle.
- [Cua Driver MCP provider](../../experimental/computer-use-cua-driver-mcp/README.md) — use an installed driver.
- [Cua Driver native provider](../../experimental/computer-use-cua-driver-native/README.md) — use the npm runtime.
packages/computer-use/computer-use/README.zh.md
+4 -1
View File
@@ -9,7 +9,7 @@ kind: "package-reference"
## 概述
部署每次可启用一个计算机操作提供方。加载另一个提供方时会失败，并报告已注册的提供方名称。各提供方提供自己的工具和桌面操作。本包不添加模型可见工具，也不协调并发 Session。
部署每次可启用一个计算机操作提供方。加载另一个提供方时会失败，并报告已注册的提供方名称。DSH 原生 CUA 和 Cua Driver 适配包都使用此注册器。各提供方提供自己的工具和桌面操作。本包不添加模型可见工具，也不协调并发 Session。
## 目录
@@ -53,6 +53,9 @@ kind: "package-reference"
## 延伸阅读
- [计算机操作](../../../docs/subsystems/computer-use.zh.md) — 提供方选择和共享桌面限制。
- [Computer Use 组合包](../../experimental/cua-bundle/README.zh.md)：启用 DSH 原生 CUA 和权限管理功能。
- [共享 CUA 库](../../experimental/cua/README.zh.md)：应用与窗口对象、工具执行和应用授权。
- [macOS CUA 提供方](../../experimental/cua-macos/README.zh.md)：DSH 的 Swift 实现和原生辅助进程生命周期。
- [Cua Driver MCP 提供方](../../experimental/computer-use-cua-driver-mcp/README.zh.md) — 使用已安装的驱动。
- [Cua Driver 原生提供方](../../experimental/computer-use-cua-driver-native/README.zh.md) — 使用 npm 运行时。
packages/experimental/README.i18n.yaml
+2 -2
View File
@@ -18,8 +18,8 @@
en: b55aeb927737b90d
zh: 0b215fcb5c2c7908
/packages-experimental/packages:
en: 372b182dbf0f662b
zh: 7a8031c506394785
en: b16e94185c9b8b24
zh: 784faa79e027cd14
/packages-experimental/related-documentation:
en: 1cb9d3448d44e48d
zh: cd326d70a746b896
packages/experimental/README.md
+2 -1
View File
@@ -69,8 +69,9 @@ Promotion requires a named product role, documented public behavior and limitati
| [`claude-code-mods`](claude-code-mods/README.md) | Run Claude Code mods as plugins: their hook chains on harness extension points and a band above the prompt | `ctx.claudeCodeMods` |
| [`client-ui-claude-code-mods`](client-ui-claude-code-mods/README.md) | The Web band that draws a mod's tree above the prompt and sends button clicks back | — |
| [`ptc-runtime-python`](ptc-runtime-python/README.md) | CPython subprocess backend for the PTC execution seam | `ctx.ptcRuntime` |
| [CUA](cua/README.md) | Shared application/window objects, consent, and logged execution. | Library |
| [CUA](cua/README.md) | Application/window objects, authorized execution, and permission controls. | `ctx.cuaAccess` |
| [macOS CUA](cua-macos/README.md) | Swift native execution and platform preparation. | `ctx.computerUse` |
| [`cua-bundle`](cua-bundle/README.md) | Optional Computer Use switch for PTC sessions | — |
| [`computer-use-cua-driver-mcp`](computer-use-cua-driver-mcp/README.md) | Use an installed Cua Driver through MCP | `ctx.computerUse` |
| [`computer-use-cua-driver-native`](computer-use-cua-driver-native/README.md) | Embed the Cua Driver native npm runtime | `ctx.computerUse` |
| [`browser-use-playwright-mcp`](browser-use-playwright-mcp/README.md) | Playwright browser tools over MCP | `ctx.browserUse` |
packages/experimental/README.zh.md
+2 -1
View File
@@ -69,8 +69,9 @@ kind: "package-group"
| [`claude-code-mods`](claude-code-mods/README.zh.md) | 把 Claude Code 模组作为插件运行：钩子链落在 harness 扩展点上，并在提示框上方绘制横幅 | `ctx.claudeCodeMods` |
| [`client-ui-claude-code-mods`](client-ui-claude-code-mods/README.zh.md) | 在提示框上方绘制模组树并把按钮点击发回的 Web 横幅 | — |
| [`ptc-runtime-python`](ptc-runtime-python/README.zh.md) | PTC 执行 seam 的 CPython 子进程后端 | `ctx.ptcRuntime` |
| [CUA](cua/README.zh.md) | 共享应用与窗口对象、授权和已记录的执行。 | 库 |
| [CUA](cua/README.zh.md) | 应用与窗口对象、授权执行和权限控件。 | `ctx.cuaAccess` |
| [macOS CUA](cua-macos/README.zh.md) | Swift 原生执行与平台组件准备。 | `ctx.computerUse` |
| [`cua-bundle`](cua-bundle/README.zh.md) | 面向 PTC 会话、默认关闭的 Computer Use 开关 | — |
| [`computer-use-cua-driver-mcp`](computer-use-cua-driver-mcp/README.zh.md) | 通过 MCP 使用已安装的 Cua Driver | `ctx.computerUse` |
| [`computer-use-cua-driver-native`](computer-use-cua-driver-native/README.zh.md) | 嵌入 Cua Driver 原生 npm 运行时 | `ctx.computerUse` |
| [`browser-use-playwright-mcp`](browser-use-playwright-mcp/README.zh.md) | 通过 MCP 提供 Playwright 浏览器工具 | `ctx.browserUse` |
packages/experimental/cua-bundle/README.i18n.yaml
+46
View File
@@ -0,0 +1,46 @@
# Bilingual-pair consistency record for README.md (docs/i18n/README.md): per heading
# section, a hash of its English and Chinese blocks outside code blocks and generated regions.
# After editing either side, bring the other along and re-record with:
# pnpm run verify-translation-pairing --write packages/experimental/cua-bundle/README.md
/:
en: cc518dcd97629a96
zh: 465e999011282dc3
/deepseek-ai-dsh-cua-bundle:
en: 9084fb48d0f6489c
zh: ce692972d91cf663
/deepseek-ai-dsh-cua-bundle/summary:
en: 06d8bf4ee1ef6f3b
zh: 8568f649fb846615
/deepseek-ai-dsh-cua-bundle/table-of-contents:
en: b563e4ec4727d0de
zh: d0228da978238e05
/deepseek-ai-dsh-cua-bundle/use-this-package:
en: 32932a00acbfe4b6
zh: 34b6fa6933e4e8c9
/deepseek-ai-dsh-cua-bundle/understand-the-implementation:
en: 057771fe13aaa337
zh: 4fdb2b98ff3e7a78
/deepseek-ai-dsh-cua-bundle/further-exploration:
en: 9f15d56b5b16e1ca
zh: 75eb08e8828c8877
/deepseek-ai-dsh-cua-bundle/model-experience:
en: 215e7ba838619b7b
zh: a311da3843709f90
/deepseek-ai-dsh-cua-bundle/model-experience/computer-use-in-ptc-sessions:
en: e8f11f45b375771e
zh: e8f11f45b375771e
/deepseek-ai-dsh-cua-bundle/model-experience/computer-use-in-ptc-sessions/what-the-model-sees:
en: 591af735d1eb92f6
zh: 3632bffa33c77b28
/deepseek-ai-dsh-cua-bundle/model-experience/computer-use-in-ptc-sessions/token-effect:
en: e8de232e96220fe7
zh: 5c791307e2123667
/deepseek-ai-dsh-cua-bundle/model-experience/computer-use-in-ptc-sessions/kv-cache-effect:
en: 417c40acd93df532
zh: 69cc5cf7140c75a9
/deepseek-ai-dsh-cua-bundle/known-limitations-and-deferred-work:
en: 41f8c54d76d92c95
zh: 44c9662e437f4cbe
/deepseek-ai-dsh-cua-bundle/known-limitations-and-deferred-work/dev-note:
en: 0a1253c6f7b5e8b9
zh: ce96645b3e75874e
packages/experimental/cua-bundle/README.md
+94
View File
@@ -0,0 +1,94 @@
---
description: "Enable optional macOS Computer Use and its permission page."
kind: "package-bundle"
---
# @deepseek-ai/dsh-cua-bundle
English | [中文](README.zh.md)
## Summary
Enable macOS Computer Use from Plugins for existing and new PTC sessions. Windows support is in development. This bundle remains in `experimental` until that support is complete, then moves to the `computer-use` group. It is off by default; disabled profiles prepare no native components and register neither CUA tools nor their model guidance.
## Table of Contents
- [Use this package](#use-this-package)
- [Understand the implementation](#understand-the-implementation)
- [Further Exploration](#further-exploration)
- [Model Experience](#model-experience)
- [Known Limitations and Deferred Work](#known-limitations-and-deferred-work)
-----
<a id="use-this-package"></a>
## Use this package
This workflow targets developers running macOS Web or unpackaged Desktop from source. Open Plugins → Computer Use and enable the switch. The permission page identifies the responsible application reported by macOS when the optional system query is available. Request access and enable that application in System Settings; follow the application named by macOS if it differs from the detected source. For Web, run these commands from the repository root:
```sh
pnpm run build
pnpm dsh web
```
For the source Desktop launcher and its separate development profile, follow [Desktop development](../../../apps/desktop/README.md). Enable Computer Use in each profile that needs it.
The switch saves bundle selection in the profile’s `package.json`. Computer Use is installed only in PTC sessions. Enabling from the plugin list offers setup guidance when system permissions need attention; enabling from its detail page displays the permission controls directly. The detail page offers **Set to PTC** for the new-session default; existing sessions keep their preset. The switch leaves PTC execution mode and other settings unchanged. The detail page’s **Use persistent mode** action separately enables in-memory state across PTC calls; its effects and reload behavior are described in the [permission page](../cua/README.md). Native preparation runs when the provider loads and reuses the source-hashed executable when available.
Disabling unloads the provider and withdraws CUA tools from its Agent scopes. Runtime configuration, the new-session default and saved grants remain. Session-only application decisions end with provider disposal.
Computer Use requires a macOS DSH Host. On other operating systems, installation omits the optional macOS provider and the bundle skips its Loader entry. Shared CUA logic and the permission-page entry remain available, but Computer Use operations require the provider. A browser on another operating system can connect to a supported Mac Host.
If preparation fails, the saved selection remains and CUA operations stay unavailable. Read the Host startup diagnostic, correct its cause, then restart DSH.
Explicit activation runs a read-only permission check, which may start a helper without opening a system dialog. Open the [permission page](../cua/README.md) for system requests, settings navigation and automatic refresh. Tasks ask separately before accessing an application. Saved application decisions use the existing profile configuration; session decisions remain in memory across turns and connections until DSH restarts or Computer Use is disabled. Revoke access on the same page.
-----
<a id="understand-the-implementation"></a>
## Understand the implementation
<details>
<summary>Maintainer details</summary>
The bundle patch mounts the native provider and permission UI with `agentPreset: ptc`. Existing Agent scopes register the tools without changing preset documents. The detail-page action uses the existing settings form to save `agent-preset-registry.selectedDefault: ptc` only when clicked. [CUA](../cua/README.md) owns registration and disposal.
</details>
-----
<a id="further-exploration"></a>
## Further Exploration
[Computer use](../../../docs/subsystems/computer-use.md)
-----
<a id="model-experience"></a>
## Model Experience
### Computer Use in PTC sessions
#### What the model sees
Existing and new PTC sessions receive `cua` through the generated SDK and `run_code`, plus CUA guidance. Disabling withdraws these contributions and keeps PTC presentation. Other presets receive no CUA contributions. [CUA](../cua/README.md) and [PTC](../../ptc-runtime/ptc-runtime-node/README.md#persistent-repl) own the interfaces and output semantics.
#### Token effect
CUA activation adds its guidance. Disabling removes that guidance; the existing PTC execution instructions remain.
#### KV Cache effect
CUA activation and removal change the PTC SDK and system guidance; subsequent requests use the updated prefix. The top-level `run_code` transport remains present.
## Known Limitations and Deferred Work
<a id="known-limitations-and-deferred-work"></a>
- Source Web and Desktop share permission controls on macOS. Packaged Desktop helper distribution, signing, and permission attribution are deferred.
- PTC execution is shared by the profile and is configured independently. A Host without HMR applies saved changes at restart. Saving or revoking application grants while another HMR transaction unloads the provider can block; queued-write cancellation is a separate pending lifecycle fix.
<a id="dev-note"></a>
### Dev Note
None.
packages/experimental/cua-bundle/README.zh.md
+94
View File
@@ -0,0 +1,94 @@
---
description: "开启可选 macOS Computer Use 及权限页面。"
kind: "package-bundle"
---
# @deepseek-ai/dsh-cua-bundle
[English](README.md) | 中文
## 概述
从插件页为已有和新建 PTC 会话开启 macOS Computer Use。Windows 支持正在开发中。本组合包暂放在 `experimental`，待 Windows 支持完成后迁回 `computer-use` 组。默认关闭，关闭时不准备本机组件、不注册 CUA 工具或添加其模型指引。
## 目录
- [使用本包](#use-this-package)
- [理解实现](#understand-the-implementation)
- [进一步探索](#further-exploration)
- [模型体验](#model-experience)
- [已知限制与延后工作](#known-limitations-and-deferred-work)
-----
<a id="use-this-package"></a>
## 使用本包
本流程面向从源码运行 macOS Web 或未打包 Desktop 的开发者。打开“插件 → Computer Use”并启用。权限页通过可选系统查询显示 macOS 报告的责任应用，指引用户申请权限并在系统设置中开启对应开关。若 macOS 显示其他应用名称，以系统提示为准。Web 版在仓库根目录运行以下命令：
```sh
pnpm run build
pnpm dsh web
```
源码 Desktop 的启动方式和独立开发配置档见[桌面开发说明](../../../apps/desktop/README.zh.md)。需要使用 Computer Use 的配置档分别开启该功能。
开关将组合包选择保存到 profile 的 `package.json`。Computer Use 仅安装在 PTC 会话中。从插件列表启用时，系统权限未就绪会显示设置引导；从详情页启用后直接显示权限控件。详情页提供“设为 PTC”按钮来修改新会话默认模式，已有会话保留原 preset。开关保持 PTC 执行方式和其他设置不变。详情页的“设为持久化”按钮单独开启 PTC 跨调用的内存状态保留，作用范围和重载行为见[权限页面](../cua/README.zh.md)。提供方加载时准备本机组件，已有与源码摘要匹配的可执行文件时直接复用。
关闭会卸载提供方，并从其 Agent 作用域中撤回 CUA 工具。执行器配置、新会话默认值和已保存的授权保留。会话级应用授权随提供方卸载而失效。
Computer Use 要求 DSH Host 运行在 macOS 上。其他系统安装时跳过可选的 macOS 提供方，组合包也不加载该提供方。共享 CUA 逻辑和权限页入口仍保留，Computer Use 操作需要提供方才能使用。其他操作系统上的浏览器可以连接受支持的 Mac Host。
准备失败时，已保存的选择保留，CUA 操作不可用。查看 Host 的启动诊断，修复原因后重启 DSH。
显式启用会进行只读权限检查，可能启动 helper，但不会打开系统授权弹窗。[权限页面](../cua/README.zh.md) 提供系统申请入口、设置跳转和自动检测。任务访问应用前会单独询问用户。长期应用授权保存在现有 profile 配置中，会话授权保存在内存中，跨 turn 和连接复用，重启 DSH 或关闭 Computer Use 后失效。可以在同一页面撤销授权。
-----
<a id="understand-the-implementation"></a>
## 理解实现
<details>
<summary>维护者说明</summary>
组合包 patch 通过 `agentPreset: ptc` 挂载原生提供方和权限界面。工具通过现有 Agent 作用域注册，preset 文档保持不变。详情页按钮复用现有设置表单，仅在点击后保存 `agent-preset-registry.selectedDefault: ptc`。[CUA](../cua/README.zh.md)负责注册和销毁。
</details>
-----
<a id="further-exploration"></a>
## 进一步探索
[Computer use](../../../docs/subsystems/computer-use.zh.md)
-----
<a id="model-experience"></a>
## 模型体验
### Computer Use in PTC sessions
#### What the model sees
已有和新建 PTC 会话通过生成的 SDK 与 `run_code` 使用 `cua`，并获得 CUA 指引。关闭撤回这些贡献，保留 PTC 展示方式。其他 preset 不获得 CUA 贡献。完整接口与输出约定由 [CUA](../cua/README.zh.md) 和 [PTC](../../ptc-runtime/ptc-runtime-node/README.zh.md#persistent-repl) 说明。
#### Token effect
CUA 生效时添加其指引。关闭撤回这些指引，现有 PTC 执行说明保留。
#### KV Cache effect
CUA 生效和卸载会改变 PTC SDK 与系统指引，后续请求使用更新后的前缀。顶层 `run_code` 工具保持存在。
## 已知限制与延后工作
<a id="known-limitations-and-deferred-work"></a>
- 源码 Web 与 Desktop 共用 macOS 权限控件。桌面安装包的 helper 分发、签名和权限归属留待后续处理。
- PTC 执行方式由 profile 共享，需单独配置。未启用 HMR 的 Host 在重启时应用已保存的变更。保存或撤销应用授权与另一次卸载提供方的 HMR 同时发生时可能阻塞；排队写入取消另行修复。
<a id="dev-note"></a>
### 开发备注
None.
packages/experimental/cua-bundle/cordis.patch.yml
+10
View File
@@ -0,0 +1,10 @@
- insert:
- id: cua-provider-registry
name: '@deepseek-ai/dsh-computer-use'
- id: cua
name: '@deepseek-ai/dsh-experimental-cua-macos'
disabled: !!js process.platform !== 'darwin'
config:
agentPreset: ptc
- id: ui-cua
name: '@deepseek-ai/dsh-cua'
packages/experimental/cua-bundle/icon.svg
+12
View File
@@ -0,0 +1,12 @@
<svg width="36" height="36" viewBox="0 0 36 36" fill="none" xmlns="http://www.w3.org/2000/svg">
<circle cx="15.5" cy="15.5" r="8.5" fill="url(#a)" />
<circle cx="15.5" cy="15.5" r="5" fill="#FFFFFF" />
<path d="M13.2 15.5L14.9 17.2L18.2 13.9" stroke="#145AF3" stroke-width="1.8" stroke-linecap="round" stroke-linejoin="round" />
<path d="M22 22L29 29" stroke="#145AF3" stroke-width="3.2" stroke-linecap="round" />
<defs>
<linearGradient id="a" x1="7" y1="7" x2="24" y2="24" gradientUnits="userSpaceOnUse">
<stop stop-color="#7CB7FF" />
<stop offset="1" stop-color="#45D9E7" />
</linearGradient>
</defs>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 642 B

packages/experimental/cua-bundle/locale/en.json
+6
View File
@@ -0,0 +1,6 @@
{
"meta": {
"title": "Computer Use",
"description": "Currently supports macOS; Windows support is in development. Loads Computer Use for existing and new PTC sessions. Switch to PTC mode and grant Accessibility and Screen Recording permissions to use it."
}
}
packages/experimental/cua-bundle/locale/zh.json
+6
View File
@@ -0,0 +1,6 @@
{
"meta": {
"title": "Computer Use",
"description": "目前仅支持 macOS，Windows 支持正在开发中。为已有及新建 PTC 会话加载 Computer Use。请切换到 PTC 模式来使用，并授予辅助功能和屏幕录制权限。"
}
}
packages/experimental/cua-bundle/package.json
+56
View File
@@ -0,0 +1,56 @@
{
"name": "@deepseek-ai/dsh-cua-bundle",
"description": "Optional macOS Computer Use and permission setup",
"version": "0.2.1-rc.2",
"icon": "./icon.svg",
"publishConfig": {
"access": "public"
},
"repository": {
"type": "git",
"url": "git+https://github.com/deepseek-ai/deepseek-harness.git",
"directory": "packages/experimental/cua-bundle"
},
"type": "module",
"main": "lib/index.js",
"types": "lib/types/index.d.ts",
"exports": {
".": {
"types": "./lib/types/index.d.ts",
"default": "./lib/index.js"
},
"./cordis.patch.yml": "./cordis.patch.yml",
"./locale/*.json": "./locale/*.json",
"./src/*": "./src/*",
"./package.json": "./package.json"
},
"files": [
"icon.svg",
"locale/*.json",
"lib/index.js",
"cordis.patch.yml",
"lib/types/**/*.d.ts"
],
"license": "MIT",
"dsh": {
"bundle": {
"patch": "./cordis.patch.yml"
}
},
"dependencies": {
"@deepseek-ai/dsh-computer-use": "workspace:*",
"@deepseek-ai/dsh-cua": "workspace:*"
},
"peerDependencies": {
"@deepseek-ai/cordis": "workspace:~"
},
"devDependencies": {
"@deepseek-ai/cordis": "workspace:~",
"@deepseek-ai/cordis-plugin-include": "workspace:~",
"js-yaml": "^4.2.0",
"@deepseek-ai/cordis-plugin-loader": "workspace:~"
},
"optionalDependencies": {
"@deepseek-ai/dsh-experimental-cua-macos": "workspace:*"
}
}
packages/experimental/cua-bundle/src/index.ts
+2
View File
@@ -0,0 +1,2 @@
/** Optional Computer Use composition; configuration rows live in cordis.patch.yml. */
export {}
packages/experimental/cua-bundle/tests/platform.spec.ts
+35
View File
@@ -0,0 +1,35 @@
/** The shipped patch leaves shared entries available while skipping an absent native provider. */
import { readFile } from 'node:fs/promises'
import { Context } from '@deepseek-ai/cordis'
import Loader, { interpolate, type EntryOptions } from '@deepseek-ai/cordis-plugin-loader'
import { entryListSchema } from '@deepseek-ai/cordis-plugin-include'
import { load } from 'js-yaml'
import { expect, it, vi } from 'vitest'
it.each(['darwin', 'win32', 'linux'])('loads the native provider only on Darwin for %s', async (platform) => {
const patch = load(await readFile(new URL('../cordis.patch.yml', import.meta.url), 'utf8'), {
schema: entryListSchema,
}) as { insert: EntryOptions[] }[]
const ctx = new Context()
await ctx.plugin(Loader)
if (!ctx.loader.internal) throw new Error('Loader module importer is unavailable')
const imported = vi.spyOn(ctx.loader.internal, 'import').mockImplementation(async (name: string) => {
if (platform !== 'darwin' && name === '@deepseek-ai/dsh-experimental-cua-macos') {
throw new Error('native provider is absent from this installation')
}
return { name, apply: () => {} }
})
try {
for (const row of patch[0]!.insert) {
await ctx.loader.create({ ...row, disabled: Boolean(interpolate({ process: { platform } }, row.disabled)) })
}
await ctx.loader.await()
const imports = imported.mock.calls.map(call => call[0])
expect(imports).toContain('@deepseek-ai/dsh-cua')
expect(imports).toContain('@deepseek-ai/dsh-computer-use')
expect(imports.includes('@deepseek-ai/dsh-experimental-cua-macos')).toBe(platform === 'darwin')
} finally {
await ctx.fiber.dispose()
imported.mockRestore()
}
})
packages/experimental/cua-bundle/tsconfig.json
+15
View File
@@ -0,0 +1,15 @@
{
"extends": "../../../tsconfig.base.json",
"compilerOptions": {
"rootDir": "src",
"outDir": "lib/types"
},
"include": [
"src"
],
"references": [
{
"path": "../../../vendor/cordis"
}
]
}
packages/experimental/cua-macos/README.i18n.yaml
+6 -6
View File
@@ -12,11 +12,11 @@
en: 4549f02fc53854a5
zh: 6f74f1130b6623a8
/deepseek-ai-dsh-experimental-cua-macos/table-of-contents:
en: d152484eb41ac6b4
zh: 7b30b973a849247d
en: 9353d17b05f2f88c
zh: 6b2c6b4ee0046294
/deepseek-ai-dsh-experimental-cua-macos/use-this-package:
en: d9a238f761453123
zh: 849fd5fdccf45b95
en: 7126e254aaf42772
zh: fa05ac2346e022ff
/deepseek-ai-dsh-experimental-cua-macos/use-this-package/connect-an-external-execution-environment:
en: e7909c9b43e44fbc
zh: 98949877d4ea3a86
@@ -45,8 +45,8 @@
en: a2c9acb5a4320dda
zh: e8d1586af60215c4
/deepseek-ai-dsh-experimental-cua-macos/known-limitations-and-deferred-work:
en: 9d8af4c39fe407a3
zh: afec49eedfd702cb
en: de72c8c1184cb8dd
zh: 33fd21a93e51fee1
/deepseek-ai-dsh-experimental-cua-macos/known-limitations-and-deferred-work/dev-note:
en: b836f01bd0bceb4a
zh: 8122fa08b2aa95fa
packages/experimental/cua-macos/README.md
+13 -6
View File
@@ -20,12 +20,18 @@ Inspect macOS permissions, select running applications and windows, read accessi
- [Known Limitations and Deferred Work](#known-limitations-and-deferred-work)
- [Dev Note](#dev-note)
The helper supplies bounded 64-pixel PNG application icons for permission-page application lists. Icon reads do not launch an application or inspect its windows; executable and signing checks still determine authorization.
The shared helper queries the process macOS considers responsible for it and captures that application’s name and icon before announcing readiness. A Desktop app launched from Terminal can therefore identify Terminal even when another GUI application sits between it and the helper. Permission queries retain this metadata after the responsible application exits and across connections sharing the helper. If the system query or application metadata is unavailable, the field is omitted and the UI uses the system prompt for guidance.
-----
<a id="use-this-package"></a>
## Use this package
Enable this experimental provider through explicit source composition on macOS 14 or later. Windows support is in development. Shipped profiles do not mount it. Prepare a native helper and load the registry/provider overlay:
This package is an optional dependency restricted to macOS. Prepare the matching executable under `native/bin` before packing; that directory is included in the archive. Installed packages require the prepared helper and do not compile Swift.
For repository Web use, enable [Computer Use](../cua-bundle/README.md) in Plugins. This prepares the local executable on macOS 14 or later; Windows support is in development. Subsequent builds check it while the existing Web profile keeps the bundle selected. Disabled profiles do not prepare native components. Explicit source composition is also available:
```sh
swift build --package-path packages/experimental/cua-macos/native --configuration release
@@ -34,10 +40,11 @@ export DSH_CUA_HELPER="$cua_native_root/.build/release/dsh-cua-native"
pnpm dsh headless --patch packages/experimental/cua-macos/examples/macos.cordis.yml --dump-config
```
The dump command checks profile composition without a model request. For a real task, replace `--dump-config` with a prompt and configure the profile’s model credentials. The overlay mounts the computer-use registry and this provider; remove an existing competing provider before using it. The responsible host/helper needs Accessibility permission for AX/input and Screen Recording permission for capture. `getCapabilities` reports permissions and session lock state without opening a system dialog. Locked or unavailable desktop sessions reject UI observations and input before dispatch; the user must unlock macOS before resuming. `listApps` and `cua.getState` list running applications. `findApps(query)` searches standard macOS application directories by name, bundle ID or absolute path. `getApp` selects a unique name, bundle ID, path or running PID, obtains application consent, and launches a stopped app in the background when needed. A stopped executable is identified before approval and checked again before and after launch; unsigned code only receives task access. The returned object reads initial AX without activating the app. Launch cancellation may leave the application running. Model guidance starts with background observation and input, without activation during initialization or before each action. For hidden or minimized windows, try `app.show()` or `window.show()` and read fresh state. This requests unhide and unminimize without activation, raising or switching Spaces; app-level restoration selects the focused, main or sole AX window. The result names the selected `windowId`, or null when no window was selected, and reports `foregroundChanged` by comparing the front app and top window before and after the request. Null means foreground evidence was unavailable; later focus changes remain possible. If restoration fails or the window remains unusable, explain the needed foreground step, call `app.activate()` or an observed window’s `raise` action, then observe again. Necessary foreground access takes priority over preserving focus. Applications may change focus after input; background delivery does not guarantee an undisturbed user session.
The dump command checks profile composition without a model request. For a real task, replace `--dump-config` with a prompt and configure the profile’s model credentials. The overlay mounts the computer-use registry and this provider; remove an existing competing provider before using it. The responsible host/helper needs Accessibility permission for AX/input and Screen Recording permission for capture. `getCapabilities` reports permissions and session lock state without opening a system dialog. Locked or unavailable desktop sessions reject UI observations and input before dispatch; the user must unlock macOS before resuming. `listApps` and `cua.getState` list running applications. `findApps(query)` searches standard macOS application directories by name, bundle ID or absolute path. `getApp` selects a unique name, bundle ID, path or running PID, obtains application consent, and launches a stopped app in the background when needed. A stopped executable is identified before approval and checked again before and after launch; unsigned code only receives session access for the selected instance. The returned object reads initial AX without activating the app. Launch cancellation may leave the application running. Model guidance starts with background observation and input, without activation during initialization or before each action. For hidden or minimized windows, try `app.show()` or `window.show()` and read fresh state. This requests unhide and unminimize without activation, raising or switching Spaces; app-level restoration selects the focused, main or sole AX window. The result names the selected `windowId`, or null when no window was selected, and reports `foregroundChanged` by comparing the front app and top window before and after the request. Null means foreground evidence was unavailable; later focus changes remain possible. If restoration fails or the window remains unusable, explain the needed foreground step, call `app.activate()` or an observed window’s `raise` action, then observe again. Necessary foreground access takes priority over preserving focus. Applications may change focus after input; background delivery does not guarantee an undisturbed user session.
| Field | Default | Meaning |
| --- | --- | --- |
| `agentPreset` | `ptc` | Register tools and guidance only for Agents bound to this preset; `false` registers globally. |
| `allowedApps` | `[]` | Live profile-local application consent, keyed by executable path and verified signing requirement. |
| `helperPath` | package-owned executable | Optional absolute native executable override. |
| `buildTimeoutMs` | `300000` | Maximum native preparation time. |
@@ -60,7 +67,7 @@ Keypad names (`KP_0` through `KP_9`, `KP_Enter`, and keypad operators) retain th
Agent disposal and plugin unload cancel and await the complete tool call, including image admission and attachment storage. An image write already in progress must settle before disposal returns; a canceled call cannot return a successful screenshot.
Application consent is enforced before AX reads, window-title reads, screenshots, activation, and input through the CUA tool. User Questions offers task access or denial, plus saved access when the application has a verifiable signature and the composition provides settings persistence. Task access is bound to the selected process handle; saved access is bound to its canonical executable path and designated code-signing requirement. Denial prevents repeated prompts in that task. Removing a saved `allowedApps` entry cancels affected operations and invalidates their helper handles; reauthorization requires fresh selection. Other handles on the same Agent connection also expire, while saved grants remain reusable. Agent/provider disposal joins cleanup of the corresponding native connection. See the [application consent decision](../../../.agents/notes/implemented/architecture/2026-09-26-cua-application-consent.md). These checks govern CUA calls; persistent PTC and separately authorized shell tools retain their own execution authority.
Application consent is enforced before AX reads, window-title reads, screenshots, activation, and input through the CUA tool. The first selection asks the human for session access, denial, or saved access when the application has a verifiable signature. Session access is kept in memory by session and application identity, survives agent and helper replacement, and expires when DSH restarts or Computer Use is disabled. Signed application identities combine the canonical executable path and designated code-signing requirement; unverifiable applications remain bound to their process instance. Denial prevents repeated prompts in that session. Revocation reads and removes saved access in the same queue as grant creation, invalidates each affected Agent connection, and awaits its native cleanup without replaying input. Every app handle on those Agent connections becomes unusable; other applications need fresh selection, but their session and saved grants remain reusable. See the [application consent decision](../../../.agents/notes/implemented/architecture/2026-09-26-cua-application-consent.md). These checks govern CUA calls; persistent PTC and separately authorized shell tools are not an operating-system sandbox.
Raw `getState` exposes validated state fields and the complete text for the requested disclosure options. The native helper also returns structured element diagnostics for native checks; the tool projection omits that duplicate array from model text and structured tool values.
@@ -148,7 +155,7 @@ The plugin contributes one `cua` operation tool with method-specific parameters.
##### Operation guidance (verbatim)
```markdown
Use cua with method getCapabilities and params {} to inspect permissions and sessionLocked. If sessionLocked is true or an operation reports SESSION_LOCKED, stop UI attempts and ask the user to unlock macOS. A null sessionLocked means no usable console session was verified. After unlocking, read fresh state; screenshots from a locked desktop are not current observations. Use listApps or cua.getState to discover running applications, and findApps(query) to search installed applications by name, bundle ID or absolute path. getApp requests user consent before launching a stopped application in the background, or before observations and input on a running one. If access is denied or revoked, stop using that application until the user explicitly grants access again. Never switch tools to bypass a refusal. Keep the returned appHandle; every app operation requires it. listWindows returns window IDs and point bounds. Raw getState returns AX text and snapshotId. Menu-bar entries are shown by default; includeMenus=true expands their contents. details=true adds identifiers, global-point bounds and complete action names. Both options also apply to getAXState. Each read replaces the indices for that app/window scope. Use its element indices only with that snapshotId. Raw screenshot and coordinate operations require a windowId. Coordinates are pixels in the latest returned screenshot, measured from its top-left. Set relative:true for integer thousandths from 0 to 1000 instead. The Host converts both modes to native window points; AX detail bounds remain global macOS points. A verified window can expose no AX elements. Its empty snapshot permits screenshots and coordinate input; element-index actions require an element from a fresh AX observation.
Use cua with method getCapabilities and params {} to inspect permissions and sessionLocked. If sessionLocked is true or an operation reports SESSION_LOCKED, stop UI attempts and ask the user to unlock macOS. A null sessionLocked means no usable console session was verified. After unlocking, read fresh state; screenshots from a locked desktop are not current observations. Use listApps or cua.getState to discover running applications, and findApps(query) to search installed applications by name, bundle ID or absolute path. getApp requests user consent before launching a stopped application in the background, or before observations and input on a running one. If access is denied or revoked, stop using that application until the user explicitly permits another authorization request. Never switch tools to bypass a refusal. Keep the returned appHandle; every app operation requires it. listWindows returns window IDs and point bounds. Raw getState returns AX text and snapshotId. Menu-bar entries are shown by default; includeMenus=true expands their contents. details=true adds identifiers, global-point bounds and complete action names. Both options also apply to getAXState. Each read replaces the indices for that app/window scope. Use its element indices only with that snapshotId. Raw screenshot and coordinate operations require a windowId. Coordinates are pixels in the latest returned screenshot, measured from its top-left. Set relative:true for integer thousandths from 0 to 1000 instead. The Host converts both modes to native window points; AX detail bounds remain global macOS points. A verified window can expose no AX elements. Its empty snapshot permits screenshots and coordinate input; element-index actions require an element from a fresh AX observation.
Initialize the object API once in persistent PTC. Reuse its variables in later calls; put temporary declarations inside { ... } blocks to avoid redeclaration errors. All object methods are async. getAXState returns text, not a structured state object. Its text separates values and placeholders, and reports subroles, help, URLs, disabled/focused/selected/expanded state and UTF-16 selection ranges when available. Writability is true, false or unknown on relevant controls; omitted state fields do not assert false. selection=unavailable means the reported range exceeds the observed text. Compact output omits ordinary Press/Pick (click), ShowMenu (right click) and ScrollToVisible actions; details=true lists every available action. Use the displayed action name with performSecondaryAction; ambiguous aliases require the complete name from details=true. Truncation reports the reached provider limit. Narrow the window or increase that provider limit before reading again.
@@ -185,7 +192,7 @@ App and Window methods:
Raw operation tools use { method, params }, including params: {} for parameterless methods. getApp selects without activating the app. Keep applications in the background for observation and routine input. Do not activate or raise an app during initialization, before each action, or merely because another app is foreground. Prefer advertised AX actions. If a window is hidden or minimized, try app.show() or window.show() and read fresh state. show requests unhide and unminimize without activation, raising or switching Spaces; it chooses the explicit, focused, main or sole AX window. A null windowId means no window was selected for restoration. foregroundChanged compares the front app and top window before and after the request; null means unavailable, and delayed focus changes remain possible. If restoration is unsupported or the window remains unusable, including another Space or Stage Manager, explain the needed foreground step, use app.activate() or an advertised raise action, and read fresh state. Completing the task takes priority when foreground access is necessary. Avoid repeated activation when the user switches away; continue background-capable steps and report what requires foreground access. Applications may still change focus in response to input.
Await operations sequentially. After input, read fresh state to verify the effect. A successful action means dispatched, not verified completion. Cancellation may leave partial effects; never retry uncertain input without observing. STALE_APP requires getApp again; STALE_SNAPSHOT requires getState again. Absolute coordinate input requires a fresh screenshot of that window. Relative input can use a successful AX observation or screenshot. SCREENSHOT_REQUIRED means capture a fresh screenshot before using pixel coordinates. For click and hover, coordinate targets must be [x, y], for example hover([650, 400], {relative:true}). A bare number such as hover(650) is always an AX element index. The relative option applies only to coordinate targets, not AX indices. Window resizing since the latest observation rejects coordinate input; movement alone preserves window-local coordinates. WINDOW_CHANGED requires fresh state before any retry; use an advertised AX action or explicitly restore the window before screenshot or coordinate input, then observe again. PERMISSION_DENIED requires the user to grant macOS Accessibility or Screen Recording access in System Settings. After granting access, observe fresh state before continuing; do not automatically replay prior input.
Await operations sequentially. After input, read fresh state to verify the effect. A successful action means dispatched, not verified completion. Cancellation may leave partial effects; never retry uncertain input without observing. STALE_APP requires getApp again; STALE_SNAPSHOT requires getState again. Absolute coordinate input requires a fresh screenshot of that window. Relative input can use a successful AX observation or screenshot. SCREENSHOT_REQUIRED means capture a fresh screenshot before using pixel coordinates. For click and hover, coordinate targets must be [x, y], for example hover([650, 400], {relative:true}). A bare number such as hover(650) is always an AX element index. The relative option applies only to coordinate targets, not AX indices. Window resizing since the latest observation rejects coordinate input; movement alone preserves window-local coordinates. WINDOW_CHANGED requires fresh state before any retry; use an advertised AX action or explicitly restore the window before screenshot or coordinate input, then observe again. PERMISSION_DENIED requires the user to grant the required macOS system permissions for macOS Accessibility or Screen Recording. After granting access, observe fresh state before continuing; do not automatically replay prior input.
Hover first uses advertised AX alternate UI without moving the system cursor, preserving revealed descendants until the pointer target leaves their region. When unavailable, it moves the visible system cursor without pressing a button; that route requires the target point to be unobscured in the foreground app. FOREGROUND_REQUIRED requires explicit activation or raising and a fresh observation before retrying. Returning, AX reads and screenshots do not restore the previous point. Wait for the application's hover delay, observe, then hover into the revealed controls before clicking. Use successive hover calls for intermediate points. The result reports hoverRoute and systemPointerMoved. Other input or application changes can interrupt the interaction; app objects do not reserve applications between calls.
@@ -211,7 +218,7 @@ The queue orders individual native requests. Holding an app object does not rese
- macOS Stage Manager can reduce background window bounds and screenshots to thumbnails. Full-window observation and coordinate input in this state are not guaranteed; model guidance requires explaining foreground needs and avoiding repeated focus reclamation.
- Background input depends on private SkyLight symbols; macOS updates can make them unavailable. Validate each target OS version.
- Observed failures include dispatched Unicode text not appearing in Blender and background Command+A not selecting all text in AppKit. Inspect application state after input; a successful tool result confirms dispatch only.
- Native binaries require a local build. Signing, distribution, and system-permission onboarding remain deferred.
- Native binaries require a local build. Signing and Desktop distribution remain future work. The [permission page](../cua/README.md) offers macOS prompts and app revocation; packaged Desktop permission attribution remains deferred.
- Persistent PTC is an external dependency assumption, not an implementation in this package. Real-model task success requires separate evaluation beyond native and protocol fixtures.
-----
packages/experimental/cua-macos/README.zh.md
+13 -6
View File
@@ -20,12 +20,18 @@ kind: "package-reference"
- [已知限制与后续工作](#known-limitations-and-deferred-work)
- [开发备注](#dev-note)
helper 为权限页的应用列表提供 64 像素 PNG 图标。读取图标不会启动应用或检查窗口，授权仍由可执行文件和签名校验决定。
共享 helper 查询 macOS 为它记录的责任进程，在发布就绪状态前保存对应应用的名称和图标。因此，从 Terminal 启动 Desktop 时，即使中间还有另一个 GUI 应用，也能识别到 Terminal。责任应用退出后，权限查询仍保留这些信息，后续复用该 helper 的连接读取同一份记录。系统查询或应用信息不可用时省略该字段，由界面引导用户按系统弹窗选择授权对象。
-----
<a id="use-this-package"></a>
## 使用本包
该实验性提供方需要在 macOS 14 及以上通过源码配置显式启用。Windows 支持正在开发中。默认配置不加载它。先准备原生组件，再加载注册表和提供方覆盖层：
本包是仅限 macOS 的可选依赖。打包前需在 `native/bin` 下准备对应架构的可执行文件，该目录会包含在发布包中。安装版需要准备好的 helper，不会编译 Swift。
从仓库运行 Web 时，在插件页开启 [Computer Use](../cua-bundle/README.zh.md)。首次启用会在 macOS 14 及以上准备本机组件，Windows 支持正在开发中。现有 Web profile 保持选中时，后续构建继续检查更新。未开启的 profile 不准备原生组件。也可以显式加载源码覆盖层：
```sh
swift build --package-path packages/experimental/cua-macos/native --configuration release
@@ -34,10 +40,11 @@ export DSH_CUA_HELPER="$cua_native_root/.build/release/dsh-cua-native"
pnpm dsh headless --patch packages/experimental/cua-macos/examples/macos.cordis.yml --dump-config
```
配置输出命令不发起模型请求。执行真实任务时，将 `--dump-config` 替换为任务提示，并配置该 profile 的模型凭据。配置挂载 computer-use 注册表和本 provider；使用前需移除已有的竞争 provider。负责执行的宿主或辅助进程需要辅助功能权限来读取 AX 和输入，需要屏幕录制权限来截图。`getCapabilities` 查询权限和锁屏状态，不弹出系统授权窗口。桌面会话锁定或不可用时，界面观察与输入会在分派前拒绝，需由用户解锁 macOS 后继续。`listApps` 和 `cua.getState` 列出运行中的应用。`findApps(query)` 按名称、bundle ID 或绝对路径搜索 macOS 标准应用目录。`getApp` 按唯一名称、bundle ID、路径或运行中的 PID 选择应用，取得应用许可后，在需要时后台启动尚未运行的应用。启动前先确认可执行文件身份，授权后在启动前后再次核对；未签名代码只取得任务许可。返回的对象读取初始 AX，不自动激活应用。启动被取消时，应用可能已经运行。模型指引要求先在后台观察和输入，不在初始化或每次动作前激活应用。窗口隐藏或最小化时，先尝试 `app.show()` 或 `window.show()`，再重新观察。它请求取消隐藏和恢复最小化，不主动激活、抬起窗口或切换桌面空间；应用级恢复选择焦点窗口、主窗口或唯一 AX 窗口。结果通过 `windowId` 指明选中的窗口，未选中时为 null；`foregroundChanged` 比较操作前后的前台应用和最上层窗口，无法取得完整观察时为 null，返回后仍可能发生焦点变化。恢复失败或窗口仍不可用时，说明需要前台的步骤，再调用 `app.activate()` 或窗口的 `raise` 动作并重新观察。必须使用前台时，以完成任务为先。应用收到输入后仍可能改变焦点，后台投递不保证用户操作始终不受干扰。
配置输出命令不发起模型请求。执行真实任务时，将 `--dump-config` 替换为任务提示，并配置该 profile 的模型凭据。配置挂载 computer-use 注册表和本 provider；使用前需移除已有的竞争 provider。负责执行的宿主或辅助进程需要辅助功能权限来读取 AX 和输入，需要屏幕录制权限来截图。`getCapabilities` 查询权限和锁屏状态，不弹出系统授权窗口。桌面会话锁定或不可用时，界面观察与输入会在分派前拒绝，需由用户解锁 macOS 后继续。`listApps` 和 `cua.getState` 列出运行中的应用。`findApps(query)` 按名称、bundle ID 或绝对路径搜索 macOS 标准应用目录。`getApp` 按唯一名称、bundle ID、路径或运行中的 PID 选择应用，取得应用许可后，在需要时后台启动尚未运行的应用。启动前先确认可执行文件身份，授权后在启动前后再次核对；未签名代码只能取得所选实例的会话许可。返回的对象读取初始 AX，不自动激活应用。启动被取消时，应用可能已经运行。模型指引要求先在后台观察和输入，不在初始化或每次动作前激活应用。窗口隐藏或最小化时，先尝试 `app.show()` 或 `window.show()`，再重新观察。它请求取消隐藏和恢复最小化，不主动激活、抬起窗口或切换桌面空间；应用级恢复选择焦点窗口、主窗口或唯一 AX 窗口。结果通过 `windowId` 指明选中的窗口，未选中时为 null；`foregroundChanged` 比较操作前后的前台应用和最上层窗口，无法取得完整观察时为 null，返回后仍可能发生焦点变化。恢复失败或窗口仍不可用时，说明需要前台的步骤，再调用 `app.activate()` 或窗口的 `raise` 动作并重新观察。必须使用前台时，以完成任务为先。应用收到输入后仍可能改变焦点，后台投递不保证用户操作始终不受干扰。
| 字段 | 默认值 | 含义 |
| --- | --- | --- |
| `agentPreset` | `ptc` | 仅为绑定该 preset 的 Agent 注册工具和指引；`false` 表示全局注册。 |
| `allowedApps` | `[]` | 当前配置档的动态应用授权，绑定可执行文件路径和已验证的代码签名要求。 |
| `helperPath` | 包内可执行文件 | 可选的原生可执行文件绝对路径覆盖。 |
| `buildTimeoutMs` | `300000` | 原生组件准备的最长时间。 |
@@ -60,7 +67,7 @@ pnpm dsh headless --patch packages/experimental/cua-macos/examples/macos.cordis.
Agent 销毁和插件卸载会取消并等待完整工具调用，包括图像接入检查和附件存储。已经开始的图像写入必须结束后，销毁才会返回；已取消的调用不会返回成功截图。
CUA 工具在读取 AX、窗口标题、截图、激活和输入之前检查应用授权。User Questions 提供任务级允许和拒绝选项；应用签名可验证且组合支持配置持久化时，也可保存授权。任务授权绑定所选进程句柄，长期授权绑定真实可执行路径和指定代码签名要求。拒绝后，同一任务不再反复询问。删除保存的 `allowedApps` 条目会取消相关操作并使原生组件中的句柄失效；重新授权需要重新选择应用。同一 Agent 连接中的其他句柄也会失效，已保存的授权仍可复用。Agent 或提供方销毁时等待对应连接的原生状态清理。设计理由见[应用授权决策](../../../.agents/notes/implemented/architecture/2026-09-26-cua-application-consent.zh.md)。这些检查约束 CUA 调用；持久 PTC 和另行授权的 shell 工具保留各自的执行权限。
CUA 工具在读取 AX、窗口标题、截图、激活和输入前检查应用授权。首次选择应用时询问用户，可选择本会话允许、拒绝；签名可验证的应用还可保存长期授权。会话许可按会话和应用身份保存在内存中，Agent 或 helper 重建后仍可复用，重启 DSH 或关闭 Computer Use 后失效。已签名应用的身份由规范化可执行文件路径和代码签名指定要求组成；签名不可验证的应用仍绑定进程实例。拒绝后，同一会话不会反复询问。撤销与长期许可的保存共用队列，在队列内读取并移除许可，然后使相关 Agent 连接失效并等待原生清理完成，不自动重放输入。同一 Agent 连接中的其他应用也需要重新选择，其他应用的会话许可和长期许可仍可复用。设计理由见[应用授权决策](../../../.agents/notes/implemented/architecture/2026-09-26-cua-application-consent.zh.md)。这些检查的范围是 CUA 调用。persistent PTC 与已授权的 shell 仍按各自执行权限运行。
原始 `getState` 操作返回经过校验的状态字段，以及按本次披露选项生成的完整文本。原生 helper 还返回供原生检查使用的结构化元素诊断数组；工具投影会从模型文本和结构化工具值中移除这份重复数组。
@@ -148,7 +155,7 @@ AppKit sheet 可以通过其窗口 ID 单独选择，也会出现在所属文档
##### 操作指导文本（原文）
```markdown
Use cua with method getCapabilities and params {} to inspect permissions and sessionLocked. If sessionLocked is true or an operation reports SESSION_LOCKED, stop UI attempts and ask the user to unlock macOS. A null sessionLocked means no usable console session was verified. After unlocking, read fresh state; screenshots from a locked desktop are not current observations. Use listApps or cua.getState to discover running applications, and findApps(query) to search installed applications by name, bundle ID or absolute path. getApp requests user consent before launching a stopped application in the background, or before observations and input on a running one. If access is denied or revoked, stop using that application until the user explicitly grants access again. Never switch tools to bypass a refusal. Keep the returned appHandle; every app operation requires it. listWindows returns window IDs and point bounds. Raw getState returns AX text and snapshotId. Menu-bar entries are shown by default; includeMenus=true expands their contents. details=true adds identifiers, global-point bounds and complete action names. Both options also apply to getAXState. Each read replaces the indices for that app/window scope. Use its element indices only with that snapshotId. Raw screenshot and coordinate operations require a windowId. Coordinates are pixels in the latest returned screenshot, measured from its top-left. Set relative:true for integer thousandths from 0 to 1000 instead. The Host converts both modes to native window points; AX detail bounds remain global macOS points. A verified window can expose no AX elements. Its empty snapshot permits screenshots and coordinate input; element-index actions require an element from a fresh AX observation.
Use cua with method getCapabilities and params {} to inspect permissions and sessionLocked. If sessionLocked is true or an operation reports SESSION_LOCKED, stop UI attempts and ask the user to unlock macOS. A null sessionLocked means no usable console session was verified. After unlocking, read fresh state; screenshots from a locked desktop are not current observations. Use listApps or cua.getState to discover running applications, and findApps(query) to search installed applications by name, bundle ID or absolute path. getApp requests user consent before launching a stopped application in the background, or before observations and input on a running one. If access is denied or revoked, stop using that application until the user explicitly permits another authorization request. Never switch tools to bypass a refusal. Keep the returned appHandle; every app operation requires it. listWindows returns window IDs and point bounds. Raw getState returns AX text and snapshotId. Menu-bar entries are shown by default; includeMenus=true expands their contents. details=true adds identifiers, global-point bounds and complete action names. Both options also apply to getAXState. Each read replaces the indices for that app/window scope. Use its element indices only with that snapshotId. Raw screenshot and coordinate operations require a windowId. Coordinates are pixels in the latest returned screenshot, measured from its top-left. Set relative:true for integer thousandths from 0 to 1000 instead. The Host converts both modes to native window points; AX detail bounds remain global macOS points. A verified window can expose no AX elements. Its empty snapshot permits screenshots and coordinate input; element-index actions require an element from a fresh AX observation.
Initialize the object API once in persistent PTC. Reuse its variables in later calls; put temporary declarations inside { ... } blocks to avoid redeclaration errors. All object methods are async. getAXState returns text, not a structured state object. Its text separates values and placeholders, and reports subroles, help, URLs, disabled/focused/selected/expanded state and UTF-16 selection ranges when available. Writability is true, false or unknown on relevant controls; omitted state fields do not assert false. selection=unavailable means the reported range exceeds the observed text. Compact output omits ordinary Press/Pick (click), ShowMenu (right click) and ScrollToVisible actions; details=true lists every available action. Use the displayed action name with performSecondaryAction; ambiguous aliases require the complete name from details=true. Truncation reports the reached provider limit. Narrow the window or increase that provider limit before reading again.
@@ -185,7 +192,7 @@ App and Window methods:
Raw operation tools use { method, params }, including params: {} for parameterless methods. getApp selects without activating the app. Keep applications in the background for observation and routine input. Do not activate or raise an app during initialization, before each action, or merely because another app is foreground. Prefer advertised AX actions. If a window is hidden or minimized, try app.show() or window.show() and read fresh state. show requests unhide and unminimize without activation, raising or switching Spaces; it chooses the explicit, focused, main or sole AX window. A null windowId means no window was selected for restoration. foregroundChanged compares the front app and top window before and after the request; null means unavailable, and delayed focus changes remain possible. If restoration is unsupported or the window remains unusable, including another Space or Stage Manager, explain the needed foreground step, use app.activate() or an advertised raise action, and read fresh state. Completing the task takes priority when foreground access is necessary. Avoid repeated activation when the user switches away; continue background-capable steps and report what requires foreground access. Applications may still change focus in response to input.
Await operations sequentially. After input, read fresh state to verify the effect. A successful action means dispatched, not verified completion. Cancellation may leave partial effects; never retry uncertain input without observing. STALE_APP requires getApp again; STALE_SNAPSHOT requires getState again. Absolute coordinate input requires a fresh screenshot of that window. Relative input can use a successful AX observation or screenshot. SCREENSHOT_REQUIRED means capture a fresh screenshot before using pixel coordinates. For click and hover, coordinate targets must be [x, y], for example hover([650, 400], {relative:true}). A bare number such as hover(650) is always an AX element index. The relative option applies only to coordinate targets, not AX indices. Window resizing since the latest observation rejects coordinate input; movement alone preserves window-local coordinates. WINDOW_CHANGED requires fresh state before any retry; use an advertised AX action or explicitly restore the window before screenshot or coordinate input, then observe again. PERMISSION_DENIED requires the user to grant macOS Accessibility or Screen Recording access in System Settings. After granting access, observe fresh state before continuing; do not automatically replay prior input.
Await operations sequentially. After input, read fresh state to verify the effect. A successful action means dispatched, not verified completion. Cancellation may leave partial effects; never retry uncertain input without observing. STALE_APP requires getApp again; STALE_SNAPSHOT requires getState again. Absolute coordinate input requires a fresh screenshot of that window. Relative input can use a successful AX observation or screenshot. SCREENSHOT_REQUIRED means capture a fresh screenshot before using pixel coordinates. For click and hover, coordinate targets must be [x, y], for example hover([650, 400], {relative:true}). A bare number such as hover(650) is always an AX element index. The relative option applies only to coordinate targets, not AX indices. Window resizing since the latest observation rejects coordinate input; movement alone preserves window-local coordinates. WINDOW_CHANGED requires fresh state before any retry; use an advertised AX action or explicitly restore the window before screenshot or coordinate input, then observe again. PERMISSION_DENIED requires the user to grant the required macOS system permissions for macOS Accessibility or Screen Recording. After granting access, observe fresh state before continuing; do not automatically replay prior input.
Hover first uses advertised AX alternate UI without moving the system cursor, preserving revealed descendants until the pointer target leaves their region. When unavailable, it moves the visible system cursor without pressing a button; that route requires the target point to be unobscured in the foreground app. FOREGROUND_REQUIRED requires explicit activation or raising and a fresh observation before retrying. Returning, AX reads and screenshots do not restore the previous point. Wait for the application's hover delay, observe, then hover into the revealed controls before clicking. Use successive hover calls for intermediate points. The result reports hoverRoute and systemPointerMoved. Other input or application changes can interrupt the interaction; app objects do not reserve applications between calls.
@@ -211,7 +218,7 @@ Use selectText only on controls advertising selectionWritable. It selects a uniq
- macOS 台前调度可能将后台窗口的尺寸与截图变为缩略图，当前无法保证这种状态下的完整画面与坐标输入；模型指引要求说明前台需求并避免反复抢回焦点。
- 后台输入依赖私有 SkyLight 符号，macOS 升级可能使其不可用，需在目标系统版本验证。
- 实测 Blender Unicode 输入曾出现分派成功但界面未显示文本，后台 AppKit 的 Command+A 曾未选中全文。输入后必须检查应用状态；工具返回成功只表示已分派。
- 原生二进制需要本地构建。签名、分发和系统权限引导留待后续。
- 原生二进制需要本地构建，签名和桌面分发留待后续。[权限页面](../cua/README.zh.md)提供 macOS 申请入口和应用授权撤销；桌面安装包的权限归属留待后续处理。
- 持久 PTC 是对外部运行环境的假设，本包不实现它。真实模型任务成功率需要在原生测试和协议测试之外单独评估。
-----
packages/experimental/cua-macos/examples/macos.cordis.yml
+1
View File
@@ -4,4 +4,5 @@
- id: cua
name: '../src/index.ts'
config:
agentPreset: false
helperPath: !!js process.env.DSH_CUA_HELPER
packages/experimental/cua-macos/native/.npmignore
+3
View File
@@ -0,0 +1,3 @@
target/
.build/
.swiftpm/
packages/experimental/cua-macos/native/README.i18n.yaml
+8 -8
View File
@@ -15,17 +15,17 @@
en: e8cbc43d38786c40
zh: 52b3ba85d98011b9
/dsh-cua-native-helper/build-and-service:
en: 824e4a2b9036d9bf
zh: 37c34a454abc6965
en: 02b25ac1126b6094
zh: 3995a1f46354531b
/dsh-cua-native-helper/request-protocol:
en: 3202c567a6e1a654
zh: 143eebd0096a0487
en: c8ad13843989b314
zh: 2a42acb2faee8803
/dsh-cua-native-helper/observation-and-input:
en: c56adf89d249f4f9
zh: 2a36a99e0a2badef
en: c527eedc3a0ee968
zh: ce8cc12d5a62e8ae
/dsh-cua-native-helper/verification:
en: 0e42ed266cc287d8
zh: bc5e6ca20e82dae3
en: a2ac6f2d581a1e8b
zh: f82c7d0c13539421
/dsh-cua-native-helper/further-exploration:
en: 265e1927952ce6d7
zh: 7c35d9fd98b6e6c8
packages/experimental/cua-macos/native/README.md
+6 -2
View File
@@ -27,7 +27,7 @@ Build with `swift build --package-path packages/experimental/cua-macos/native --
The shared service uses an owner-only Unix socket under `/tmp/dsh-cua-<uid>`, keyed by the executable SHA-256. A version lock elects one process; an additional real-executable-path lock excludes a different build at the same path. The first frame is protocol-1 `hello` with the runtime key and this connection’s byte, AX, operation, screenshot, event-spacing and cleanup limits. `ready` reports the verified runtime key, PID and instance ID. Every connection owns an independent AXWorker. The stdin/stdout entry is a diagnostic proxy to this service.
AppKit runs on the main event loop. Blocking AX and input use each connection’s serial actor; all connections share a FIFO for complete desktop operations. Socket writes do not hold that operation slot. A request cancellation waits for native work to stop; it does not cancel the connection’s lifetime latch. Screenshot timeout ends only the request. An unfinished system capture retains the single capture slot until its real callback arrives, rejects new captures with CAPTURE_BUSY and leaves AX/input available. Only an operation or cleanup that cannot stop can invalidate the shared process. The proxy accepts `--max-request-bytes`, `--max-response-bytes`, `--ax-timeout-ms`, `--timeout-ms`, `--screenshot-timeout-ms`, `--shutdown-grace-ms` and `--text-event-interval-ms`; the Host sets the same limits in hello. An AX deadline bounds an individual system call. Cancellation can follow partial input, and uncertain input is never replayed automatically.
AppKit runs on the main event loop. Blocking socket accepts and handshakes use background queues; server callbacks explicitly carry `@MainActor @Sendable` isolation and weak ownership. Shutdown releases accepted connections even when their handshake is incomplete. Blocking AX and input use each connection’s serial actor; all connections share a FIFO for complete desktop operations. Socket writes do not hold that operation slot. A request cancellation waits for native work to stop; it does not cancel the connection’s lifetime latch. Screenshot timeout ends only the request. An unfinished system capture retains the single capture slot until its real callback arrives, rejects new captures with CAPTURE_BUSY and leaves AX/input available. Only an operation or cleanup that cannot stop can invalidate the shared process. The proxy accepts `--max-request-bytes`, `--max-response-bytes`, `--ax-timeout-ms`, `--timeout-ms`, `--screenshot-timeout-ms`, `--shutdown-grace-ms` and `--text-event-interval-ms`; the Host sets the same limits in hello. An AX deadline bounds an individual system call. Cancellation can follow partial input, and uncertain input is never replayed automatically.
-----
@@ -57,7 +57,7 @@ Every app-targeted operation requires `appHandle` from `getApp` in the same help
Stable error codes are `INVALID_REQUEST` (malformed envelope), `INVALID_ARGUMENT` (invalid parameters), `UNSUPPORTED` (unknown operation), `APP_UNAVAILABLE`, `AMBIGUOUS_APP`, `STALE_APP`, `STALE_SNAPSHOT`, `WINDOW_UNAVAILABLE`, `WINDOW_CHANGED`, `PERMISSION_DENIED`, `FOREGROUND_REQUIRED` (hover target is not the unobscured foreground window), `PANEL_NOT_READY` (system panel recipient unavailable before input), and `NATIVE_ERROR` (remaining operating-system failures). An error never promises that a partially dispatched input was rolled back. The host must not retry input automatically.
Application selection verifies the executable path and code-signing requirement before returning an authorization identity to the Host. Unsigned running applications receive a process-lifetime identity; stopped unsigned applications use an executable digest and cannot receive persistent consent. The Host obtains human consent before launch, observation or input.
Application selection returns an authorization record to the Host: a digest of canonical executable path and verified designated signing requirement, display name, executable path, and persistable flag. Unsigned or ad-hoc-signed applications cannot receive saved consent; running instances use process-lifetime identity, while stopped applications use an executable digest. The Host strips this record from model results and obtains human consent before launch, observation or input. This pipe protocol is trusted Host transport, not a sandbox against a separately authorized shell process.
Protocol declarations are generated from [the shared protocol](../../cua/src/protocol.ts). Run `pnpm exec tsx packages/experimental/cua-macos/native/generate-protocol.ts` from the repository root; package tests verify generated JSON and Swift field/enum tables. Model-submittable parameters exclude Host resource bounds and the private launch continuation.
@@ -79,6 +79,8 @@ Capture and pointer delivery read CG bounds before and after an exact-window AX
The responsible host/helper needs macOS Accessibility permission for AX state and input, and Screen Recording permission for screenshots. Capability preflight and permission errors do not grant permissions. Capture refuses a window that moves or resizes while the image is being read. Unassociated system panels and accessory-only applications are not selectable.
Host-only permission controls are `permissionStatus {}`, `requestPermission {permission}`, and `openPermissionSettings {permission}`, where permission is `accessibility` or `screenRecording`. They are excluded from model tools and from `getCapabilities.methods`. Permission status includes observed helper/parent process paths; these paths do not identify the TCC responsible application. The operating-system prompt names the application and asks for the user’s decision. The helper identifies the responsible process with the optional libSystem SPI `responsibility_get_pid_responsible_for_pid`, then captures its application name and icon before publishing readiness. The function is resolved dynamically, following the same signature used by [Chromium](https://chromium.googlesource.com/chromium/src/+/refs/tags/134.0.6998.28/base/process/process_info_mac.mm). If the symbol, process, or application metadata is unavailable, attribution stays unknown; it is not inferred from the nearest GUI ancestor. Permission checks remain independent of this metadata.
Observation reads each non-layout node’s descriptive properties and children through `AXUIElementCopyMultipleAttributeValues`. Per-attribute errors retain the individual-read rules; unsupported batch APIs use individual reads. Values are retained only during that observation, and input reads live attributes. Application state can change between nodes; a batch does not make the whole tree atomic.
System file-panel keyboard routing resolves the remote recipient through the selected app’s focused AX sheet. Only Apple’s Open and Save Panel Service is accepted, and its focused ancestry must still contain the identified remote window. Element pointer fallbacks resolve an app-owned window or sheet through AX parents when AXWindow is unavailable. Ordinary clicks retain AX action priority over pointer input; visual click markers need no system-cursor movement.
@@ -103,6 +105,8 @@ Use `python3 packages/experimental/cua-macos/native/tests/test_swift.py` for off
Run `python3 packages/experimental/cua-macos/native/tests/shared_helper_e2e.py --helper /absolute/helper` to verify concurrent startup, a shared service PID, independent handles, same-application handoff, request-local screenshot deadlines, connection cleanup and signal shutdown with an owned AppKit fixture. Existing Accessibility and Screen Recording grants are required. `--without-desktop` limits the check to protocol and process lifetime.
Run `python3 packages/experimental/cua-macos/native/tests/launcher_identity_e2e.py --helper /absolute/helper` in a GUI login session to verify a Launch Services application that starts a nested GUI launcher and then the helper. The helper identifies the outer responsible application and retains its name and icon after the launchers exit. This check owns both applications and the helper, cleans up its Launch Services registrations, and does not request Accessibility or Screen Recording access.
Run `python3 packages/experimental/cua-macos/native/tests/axless_e2e.py` after building and unlocking the Mac to verify an owned window whose application removes and later restores its AX roots. The check requires existing Accessibility/Screen Recording grants, rejects stale element indices, captures the AXless window, verifies a coordinate click in the fixture state file, and checks restoration. It never changes system permission settings.
Run `python3 packages/experimental/cua-macos/native/tests/observation_benchmark.py --reference-helper /absolute/reference --candidate-helper /absolute/candidate --artifacts /private/evidence` to compare complete observations of the same owned AppKit window with 35 and 400 synthetic rows. Each helper receives one warmup and three alternating measured requests. All result fields must match except helper-generated app and snapshot IDs. The report keeps raw timings, byte counts, hashes and replies; it sets no CI speed threshold and makes no memory claim. Fixture startup can activate the application. Run without other desktop automation or compilation jobs.
packages/experimental/cua-macos/native/README.zh.md
+6 -2
View File
@@ -27,7 +27,7 @@ kind: "package-reference"
共享服务在 `/tmp/dsh-cua-<uid>` 下使用仅所有者可访问的 Unix socket，以可执行文件的 SHA-256 区分实例。版本锁选出一个进程，另一个真实可执行路径锁排除同一路径的不同构建。首帧为协议版本 1 的 `hello`，包含运行时标识以及该连接的字节、AX、操作、截图、事件间隔和清理限制。`ready` 返回已核实的运行时标识、PID 和实例 ID。每条连接拥有独立的 AXWorker。stdin/stdout 入口是该服务的诊断代理。
AppKit 在主事件循环上运行。阻塞式 AX 和输入使用每条连接的串行 actor，所有连接共享用于完整桌面操作的 FIFO。socket 写出不占用操作槽位。请求取消会等待原生工作停止，不取消连接的生命期控制器。截图超时只结束该请求。未完成的系统捕获保留唯一捕获槽位，直至真实回调到达；期间新截图返回 CAPTURE_BUSY，AX 和输入仍可用。只有无法停止的操作或清理才可能使共享进程失效。代理接受 `--max-request-bytes`、`--max-response-bytes`、`--ax-timeout-ms`、`--timeout-ms`、`--screenshot-timeout-ms`、`--shutdown-grace-ms` 和 `--text-event-interval-ms`；Host 在 hello 中设置相同限制。AX 超时约束单次系统调用。取消可能发生在部分输入之后，结果不确定的输入不自动重放。
AppKit 在主事件循环上运行。阻塞式 socket 接收与握手使用后台队列，服务回调明确声明 `@MainActor @Sendable` 隔离，并弱引用服务对象。关闭时会释放已接收的连接，包括尚未完成握手的连接。阻塞式 AX 和输入使用每条连接的串行 actor，所有连接共享用于完整桌面操作的 FIFO。socket 写出不占用操作槽位。请求取消会等待原生工作停止，不取消连接的生命期控制器。截图超时只结束该请求。未完成的系统捕获保留唯一捕获槽位，直至真实回调到达；期间新截图返回 CAPTURE_BUSY，AX 和输入仍可用。只有无法停止的操作或清理才可能使共享进程失效。代理接受 `--max-request-bytes`、`--max-response-bytes`、`--ax-timeout-ms`、`--timeout-ms`、`--screenshot-timeout-ms`、`--shutdown-grace-ms` 和 `--text-event-interval-ms`；Host 在 hello 中设置相同限制。AX 超时约束单次系统调用。取消可能发生在部分输入之后，结果不确定的输入不自动重放。
-----
@@ -57,7 +57,7 @@ AppKit 在主事件循环上运行。阻塞式 AX 和输入使用每条连接的
稳定错误码包括 `INVALID_REQUEST`（信封无效）、`INVALID_ARGUMENT`（参数无效）、`UNSUPPORTED`（操作未知）、`APP_UNAVAILABLE`、`AMBIGUOUS_APP`、`STALE_APP`、`STALE_SNAPSHOT`、`WINDOW_UNAVAILABLE`、`WINDOW_CHANGED`、`PERMISSION_DENIED`、`FOREGROUND_REQUIRED`（悬停目标不是未被遮挡的前台窗口）、`PANEL_NOT_READY`（输入前系统面板接收方尚不可用）及 `NATIVE_ERROR`（其余操作系统失败）。错误不保证已分派的部分输入被回滚，Host 不得自动重试输入。
应用选择先核实可执行路径和代码签名要求，再向 Host 返回授权身份。未签名的运行中应用使用进程生命期身份，未启动的未签名应用使用可执行文件摘要，不能获得持久授权。Host 在启动、观察和输入前取得用户许可。
应用选择向 Host 返回授权记录：规范化可执行路径与经验证的代码签名指定要求的摘要、显示名称、可执行路径和可持久化标记。未签名或临时签名的应用不能获得长期授权；运行中实例使用进程生命期身份，未启动应用使用可执行文件摘要。Host 从模型结果中移除此记录，并在启动、观察和输入前取得用户许可。此管道协议用于可信 Host 传输，无法隔离另一个已获授权的 shell 进程。
协议声明从[共享协议](../../cua/src/protocol.ts)生成。在仓库根目录运行 `pnpm exec tsx packages/experimental/cua-macos/native/generate-protocol.ts`，包测试检查生成的 JSON 和 Swift 字段/枚举表。模型参数不包含 Host 资源上限和私有启动续接操作。
@@ -79,6 +79,8 @@ AppKit 在主事件循环上运行。阻塞式 AX 和输入使用每条连接的
负责执行的 host/helper 需要 macOS 辅助功能权限以读取 AX 和输入，需要屏幕录制权限以截图。能力预检和权限错误不授予权限。截图会拒绝在读取图片期间移动或缩放的窗口。无法确认归属的系统面板及仅后台辅助类型的应用不可选择。
仅供 Host 使用的权限操作为 `permissionStatus {}`、`requestPermission {permission}` 和 `openPermissionSettings {permission}`，其中 permission 为 `accessibility` 或 `screenRecording`。这些操作不出现在模型工具及 `getCapabilities.methods` 中。权限状态包含观察到的 helper 和父进程路径，这些路径不能确定 TCC 归属的应用。系统授权提示负责指出应用并询问用户决定。helper 使用可选的 libSystem 内部接口 `responsibility_get_pid_responsible_for_pid` 查询责任进程，在报告就绪前记录对应应用的名称和图标。接口通过动态加载解析，函数签名与 [Chromium](https://chromium.googlesource.com/chromium/src/+/refs/tags/134.0.6998.28/base/process/process_info_mac.mm) 使用的一致。符号、进程或应用信息不可用时，归属保持未知，不根据最近的 GUI 祖先推断。权限检查独立于这些信息。
观察通过 `AXUIElementCopyMultipleAttributeValues` 读取每个非布局节点的描述属性和子节点。逐属性错误保留单独读取的规则；不支持批量 API 时使用单独读取。值只在本次观察期间保留，输入读取即时属性。应用状态可能在节点之间变化，批量读取不使整棵树具备原子性。
系统文件面板的键盘路由通过所选应用的 AX 焦点 sheet 解析远端接收方。只接受 Apple 的 Open and Save Panel Service，且其焦点祖先链仍须包含已识别的远端窗口。AXWindow 不可用时，元素鼠标后备路径沿 AX 父节点解析应用所属窗口或 sheet。普通点击保持 AX 动作优先，可视点击标记不需要系统鼠标移动。
@@ -103,6 +105,8 @@ AppKit 在主事件循环上运行。阻塞式 AX 和输入使用每条连接的
运行 `python3 packages/experimental/cua-macos/native/tests/shared_helper_e2e.py --helper /absolute/helper`，通过测试自建的 AppKit fixture 核实并发启动、共享服务 PID、独立句柄、同一应用交接、请求级截图期限、连接清理和信号退出。需要已有辅助功能和屏幕录制权限。`--without-desktop` 将检查限定为协议与进程生命期。
在 GUI 登录会话中运行 `python3 packages/experimental/cua-macos/native/tests/launcher_identity_e2e.py --helper /absolute/helper`，验证由 Launch Services 启动的应用先启动另一个 GUI 启动器，再启动 helper。helper 识别外层责任应用，并在启动器退出后保留其名称和图标。检查只使用测试自建的两个应用和 helper，结束后清理对应的 Launch Services 注册，不申请辅助功能或屏幕录制权限。
构建并解锁 Mac 后运行 `python3 packages/experimental/cua-macos/native/tests/axless_e2e.py`，验证应用移除并恢复 AX 根节点的窗口。检查需要已有辅助功能/屏幕录制权限，会拒绝过期元素索引，捕获无 AX 窗口，通过 fixture 状态文件核实坐标点击，并检查恢复。不会修改系统权限设置。
运行 `python3 packages/experimental/cua-macos/native/tests/observation_benchmark.py --reference-helper /absolute/reference --candidate-helper /absolute/candidate --artifacts /private/evidence`，对比同一自建 AppKit 窗口中 35 行和 400 行合成内容的完整观察。每个 helper 先预热一次，再交替测量三次。除 helper 生成的应用和快照 ID 外，结果字段必须一致。报告保留原始耗时、字节数、哈希和回复，不设置 CI 性能阈值，也不声称内存表现。fixture 启动可能激活应用。运行时避免其他桌面自动化或编译任务。
packages/experimental/cua-macos/native/Sources/CuaNative/NativePlatform.swift
+26 -1
View File
@@ -36,6 +36,22 @@ struct ApplicationCatalogSnapshot: Sendable {
}
return ApplicationCatalogSnapshot(applications: apps, frontPID: NSWorkspace.shared.frontmostApplication?.processIdentifier)
}
/// A bounded PNG for human recognition; authorization continues to use executable/signing identity.
static func icon(at path: String) -> String? {
guard let bitmap = NSBitmapImageRep(bitmapDataPlanes: nil, pixelsWide: 64, pixelsHigh: 64,
bitsPerSample: 8, samplesPerPixel: 4, hasAlpha: true, isPlanar: false,
colorSpaceName: .deviceRGB, bytesPerRow: 0, bitsPerPixel: 0),
let context = NSGraphicsContext(bitmapImageRep: bitmap) else { return nil }
NSGraphicsContext.saveGraphicsState()
defer { NSGraphicsContext.restoreGraphicsState() }
NSGraphicsContext.current = context
var url = URL(fileURLWithPath: path)
while url.path != "/" && url.pathExtension != "app" { url.deleteLastPathComponent() }
let iconPath = url.pathExtension == "app" ? url.path : path
NSWorkspace.shared.icon(forFile: iconPath).draw(in: NSRect(x: 0, y: 0, width: 64, height: 64),
from: .zero, operation: .copy, fraction: 1)
return bitmap.representation(using: .png, properties: [:]).map { "data:image/png;base64," + $0.base64EncodedString() }
}
static func activate(_ identity: ProcessIdentity) throws {
try NativePlatform.verify(identity)
guard let app = NSRunningApplication(processIdentifier: identity.pid), !app.isTerminated else {
@@ -47,9 +63,18 @@ struct ApplicationCatalogSnapshot: Sendable {
}
enum NativePlatform {
/// Optional libSystem SPI: unavailable attribution stays unknown rather than guessing from parentage.
static func responsiblePID(_ pid: Int32) -> Int32? {
typealias Resolve = @convention(c) (pid_t) -> pid_t
guard let symbol = dlsym(UnsafeMutableRawPointer(bitPattern: -2), "responsibility_get_pid_responsible_for_pid") else { return nil }
let resolve = unsafeBitCast(symbol, to: Resolve.self)
let responsible = resolve(pid)
return responsible > 0 ? responsible : nil
}
static func executablePath(_ pid: Int32) -> String? {
var buffer = [CChar](repeating: 0, count: 4 * Int(MAXPATHLEN))
return proc_pidpath(pid, &buffer, UInt32(buffer.count)) > 0 ? String(cString: buffer) : nil
guard proc_pidpath(pid, &buffer, UInt32(buffer.count)) > 0 else { return nil }
return String(decoding: buffer.prefix { $0 != 0 }.map { UInt8(bitPattern: $0) }, as: UTF8.self)
}
static func identity(_ pid: Int32) throws -> ProcessIdentity {
var info = proc_bsdinfo()
packages/experimental/cua-macos/native/Sources/CuaNative/NativeProtocol.generated.swift
+6
View File
@@ -3,6 +3,7 @@ enum NativeProtocol {
static let modelMethods: [String] = ["activate", "click", "drag", "findApps", "getApp", "getCapabilities", "getState", "hover", "listApps", "listWindows", "paste", "performSecondaryAction", "pressKey", "screenshot", "scroll", "selectText", "setValue", "show", "typeText"]
static let fields: [String: Set<String>] = [
"activate": ["appHandle"],
"applicationIcons": ["paths"],
"click": ["appHandle", "clickCount", "elementIndex", "holdMs", "modifiers", "mouseButton", "snapshotId", "windowId", "x", "y"],
"drag": ["appHandle", "durationMs", "fromX", "fromY", "modifiers", "mouseButton", "snapshotId", "steps", "toX", "toY", "windowId"],
"findApps": ["maxAppMatches", "query"],
@@ -13,9 +14,12 @@ enum NativeProtocol {
"launchApp": ["appHandle"],
"listApps": [],
"listWindows": ["appHandle"],
"openPermissionSettings": ["permission"],
"paste": ["appHandle", "format", "maxClipboardBytes", "maxDepth", "maxElements", "pasteReadTimeoutMs", "text"],
"performSecondaryAction": ["action", "appHandle", "elementIndex", "snapshotId"],
"permissionStatus": [],
"pressKey": ["appHandle", "key", "modifiers"],
"requestPermission": ["permission"],
"screenshot": ["appHandle", "windowId"],
"scroll": ["appHandle", "deltaLinesX", "deltaLinesY", "snapshotId", "windowId", "x", "y"],
"selectText": ["appHandle", "elementIndex", "prefix", "selectionType", "snapshotId", "suffix", "text"],
@@ -26,8 +30,10 @@ enum NativeProtocol {
static let stringEnums: [String: [String: Set<String>]] = [
"click": ["modifiers": ["command", "control", "fn", "option", "shift"], "mouseButton": ["left", "middle", "right"]],
"drag": ["modifiers": ["command", "control", "fn", "option", "shift"], "mouseButton": ["left", "middle", "right"]],
"openPermissionSettings": ["permission": ["accessibility", "screenRecording"]],
"paste": ["format": ["html", "text"]],
"pressKey": ["modifiers": ["command", "control", "fn", "option", "shift"]],
"requestPermission": ["permission": ["accessibility", "screenRecording"]],
"selectText": ["selectionType": ["cursor_after", "cursor_before", "text"]],
]
}
packages/experimental/cua-macos/native/Sources/CuaNative/Permissions.swift
+69
View File
@@ -0,0 +1,69 @@
// System consent is requested only by the Host's user-facing permission controls.
import AppKit
@preconcurrency import ApplicationServices
import Darwin
import Foundation
@MainActor struct SystemPermissions {
private let launchApplication: JSONValue?
// Capture the system-reported responsible app before startup completes and the launcher can exit.
init() { launchApplication = Self.responsibleApplication(for: getpid()) }
func status() -> JSONValue {
var result: [String: JSONValue] = [
"accessibility": .bool(AXIsProcessTrusted()),
"screenRecording": .bool(CGPreflightScreenCaptureAccess()),
"helper": Self.process(getpid()),
"parent": Self.process(getppid()),
]
if let launchApplication { result["launchApplication"] = launchApplication }
return .object(result)
}
func request(_ permission: String) throws -> JSONValue {
switch permission {
case "accessibility":
if !AXIsProcessTrusted() {
let options = [kAXTrustedCheckOptionPrompt.takeUnretainedValue() as String: true] as CFDictionary
_ = AXIsProcessTrustedWithOptions(options)
}
case "screenRecording":
if !CGPreflightScreenCaptureAccess() { _ = CGRequestScreenCaptureAccess() }
default: throw RequestParameters.invalid("permission must be accessibility or screenRecording")
}
return status()
}
func openSettings(_ permission: String) throws -> JSONValue {
let pane: String
switch permission {
case "accessibility": pane = "Privacy_Accessibility"
case "screenRecording": pane = "Privacy_ScreenCapture"
default: throw RequestParameters.invalid("permission must be accessibility or screenRecording")
}
guard let url = URL(string: "x-apple.systempreferences:com.apple.preference.security?\(pane)"),
NSWorkspace.shared.open(url) else {
throw NativeFailure(code: "SETTINGS_UNAVAILABLE", message: "Open System Settings > Privacy & Security manually")
}
return .object(["opened": .bool(true)])
}
/// Resolve macOS process responsibility, including launchers outside the nearest GUI application.
static func responsibleApplication(for pid: Int32) -> JSONValue? {
guard let responsible = NativePlatform.responsiblePID(pid),
let app = NSRunningApplication(processIdentifier: responsible), !app.isTerminated,
let bundle = app.bundleURL else { return nil }
var result: [String: JSONValue] = ["name": .string(bundle.lastPathComponent), "path": .string(bundle.path)]
if let icon = ApplicationCatalog.icon(at: bundle.path) { result["icon"] = .string(icon) }
return .object(result)
}
private static func process(_ pid: Int32) -> JSONValue {
let path = NativePlatform.executablePath(pid)
var result: [String: JSONValue] = ["pid": .number(Double(pid))]
if let path { result["path"] = .string(path) }
if let name = NSRunningApplication(processIdentifier: pid)?.localizedName { result["name"] = .string(name) }
return .object(result)
}
}
packages/experimental/cua-macos/native/Sources/CuaNative/RequestAdmission.swift
+7
View File
@@ -26,6 +26,13 @@ struct RequestParameters: Sendable {
guard let value = values[key]?.string else { throw Self.invalid("\(key) must be a string") }
return value
}
func strings(_ key: String) throws -> [String] {
guard let raw = values[key], case .array(let items) = raw else { throw Self.invalid("\(key) must be a string array") }
return try items.map { value in
guard let path = value.string, !path.isEmpty else { throw Self.invalid("\(key) must contain nonempty strings") }
return path
}
}
func optionalString(_ key: String) throws -> String? { contains(key) ? try string(key) : nil }
func optionalBoolean(_ key: String) throws -> Bool? {
guard let raw = values[key] else { return nil }
packages/experimental/cua-macos/native/Sources/CuaNative/RequestCoordinator.swift
+18 -2
View File
@@ -9,6 +9,7 @@ import Darwin
private let cancellation: CancellationLatch
private let worker: AXWorker
private let capture: WindowCapture
private let permissions: SystemPermissions
private let clipboard = ClipboardPaste()
private let input: JSONLInput
private let writer: JSONLWriter
@@ -27,11 +28,12 @@ import Darwin
private var nextCancelled = false
init(configuration: HelperConfiguration, capture: WindowCapture, channel: SharedSocket,
cancellation: CancellationLatch, scheduler: RequestScheduler, writer: JSONLWriter,
cancellation: CancellationLatch, scheduler: RequestScheduler, writer: JSONLWriter, permissions: SystemPermissions,
onClose: @escaping @MainActor @Sendable () -> Void,
onFatal: @escaping @Sendable (NativeFailure, Double) -> Void) {
self.configuration = configuration
self.capture = capture
self.permissions = permissions
self.cancellation = cancellation
self.channel = channel
self.scheduler = scheduler
@@ -159,11 +161,25 @@ import Darwin
}
private func execute(_ request: NativeRequest, trace: ScreenshotTrace?) async throws -> JSONValue {
let params = try RequestParameters(request)
try cancellation.check()
if request.method == "screenshot" { try capture.requireAvailable() }
let requiresCatalog = ["getApp", "listApps", "findApps", "typeText", "pressKey"].contains(request.method)
let catalog = requiresCatalog ? ApplicationCatalog.snapshot() : ApplicationCatalogSnapshot(applications: [], frontPID: nil)
let result: JSONValue
if request.method == "findApps" {
if request.method == "permissionStatus" {
result = permissions.status()
} else if request.method == "applicationIcons" {
var icons: [String: JSONValue] = [:]
for path in try params.strings("paths") {
try cancellation.check()
if let icon = ApplicationCatalog.icon(at: path) { icons[path] = .string(icon) }
}
result = .object(icons)
} else if request.method == "requestPermission" {
result = try permissions.request(params.string("permission"))
} else if request.method == "openPermissionSettings" {
result = try permissions.openSettings(params.string("permission"))
} else if request.method == "findApps" {
result = .array(try ApplicationDiscovery.applications(matching: params.string("query"), exact: false,
limit: params.integer("maxAppMatches", in: 1...1000), cancellation: cancellation).map { $0.json(catalog: catalog) })
} else {
packages/experimental/cua-macos/native/Sources/CuaNative/SharedHelperServer.swift
+69 -28
View File
@@ -67,6 +67,29 @@ final class NativeSignals {
deinit { sources.forEach { $0.cancel() } }
}
// Blocking accepts own only Sendable state; server access belongs to the MainActor callback.
private func acceptConnections(listener: SharedSocket, lifetime: SharedHelperLifetime,
onAccept: @escaping @MainActor @Sendable (UUID, SharedSocket, CancellationLatch) -> Void) {
while !lifetime.isStopping {
let descriptor = accept(listener.descriptor, nil, nil)
if descriptor < 0 && errno == EINTR { continue }
guard descriptor >= 0 else {
if !lifetime.isStopping { lifetime.abort(grace: lifetime.shutdownGrace) }
return
}
let channel = SharedSocket(descriptor: descriptor)
var uid: uid_t = 0
var gid: gid_t = 0
guard getpeereid(descriptor, &uid, &gid) == 0, uid == getuid() else { channel.shutdown(); continue }
let id = UUID()
let cancellation = CancellationLatch()
guard lifetime.add(id, channel: channel, cancellation: cancellation) else { channel.shutdown(); return }
Task { @MainActor [onAccept, id, channel, cancellation] in
onAccept(id, channel, cancellation)
}
}
}
@MainActor final class SharedHelperServer {
private let options: SharedHelperOptions
private let versionLock: SharedHelperLock
@@ -75,6 +98,7 @@ final class NativeSignals {
private let lifetime: SharedHelperLifetime
private let scheduler = RequestScheduler()
private let capture = WindowCapture()
private let permissions = SystemPermissions()
private let instanceID = UUID().uuidString
private var connections: [UUID: RequestCoordinator] = [:]
private var startupDeadline: NativeDeadline?
@@ -99,12 +123,18 @@ final class NativeSignals {
func start() throws {
let lifetime = self.lifetime
signals = NativeSignals { [weak self] number in
lifetime.abort(grace: lifetime.shutdownGrace, status: 128 + number)
Task { @MainActor in self?.closeConnections(); self?.exitIfEmpty() }
let onStop: @MainActor @Sendable () -> Void = { [weak self] in
guard let self else { return }
self.closeConnections()
self.exitIfEmpty()
}
startupDeadline = NativeDeadline(seconds: options.startupTimeout) { [weak self] in
Task { @MainActor in self?.exitIfEmpty() }
signals = NativeSignals { [lifetime, onStop] number in
lifetime.abort(grace: lifetime.shutdownGrace, status: 128 + number)
Task { @MainActor [onStop] in onStop() }
}
let onIdle: @MainActor @Sendable () -> Void = { [weak self] in self?.exitIfEmpty() }
startupDeadline = NativeDeadline(seconds: options.startupTimeout) { [onIdle] in
Task { @MainActor [onIdle] in onIdle() }
}
let ready = options.ready(pid: getpid(), instanceID: instanceID)
let data = try JSONEncoder().encode(ready)
@@ -113,38 +143,45 @@ final class NativeSignals {
try FileHandle.standardOutput.write(contentsOf: data + Data([10]))
try FileHandle.standardOutput.close()
let listener = self.listener
DispatchQueue.global(qos: .utility).async { [weak self] in
while !lifetime.isStopping {
let descriptor = accept(listener.descriptor, nil, nil)
if descriptor < 0 && errno == EINTR { continue }
guard descriptor >= 0 else {
if !lifetime.isStopping { lifetime.abort(grace: lifetime.shutdownGrace) }
return
}
let channel = SharedSocket(descriptor: descriptor)
var uid: uid_t = 0
var gid: gid_t = 0
guard getpeereid(descriptor, &uid, &gid) == 0, uid == getuid() else { channel.shutdown(); continue }
let id = UUID()
let cancellation = CancellationLatch()
guard lifetime.add(id, channel: channel, cancellation: cancellation) else { channel.shutdown(); return }
Task { @MainActor in self?.admit(id, channel: channel, cancellation: cancellation) }
let onAccept: @MainActor @Sendable (UUID, SharedSocket, CancellationLatch) -> Void = { [weak self] id, channel, cancellation in
guard let self else {
channel.shutdown()
lifetime.remove(id)
return
}
self.admit(id, channel: channel, cancellation: cancellation)
}
DispatchQueue.global(qos: .utility).async { [listener, lifetime, onAccept] in
acceptConnections(listener: listener, lifetime: lifetime, onAccept: onAccept)
}
}
private func admit(_ id: UUID, channel: SharedSocket, cancellation: CancellationLatch) {
guard !lifetime.isStopping else { channel.shutdown(); disconnected(id); return }
let options = self.options
DispatchQueue.global(qos: .utility).async { [weak self] in
let lifetime = self.lifetime
let onHello: @MainActor @Sendable (HelperConfiguration) async -> Void = { [weak self] configuration in
guard let self else {
channel.shutdown()
lifetime.remove(id)
return
}
await self.connected(id, channel: channel, cancellation: cancellation, configuration: configuration)
}
let onFailure: @MainActor @Sendable () -> Void = { [weak self] in
lifetime.remove(id)
self?.exitIfEmpty()
}
DispatchQueue.global(qos: .utility).async { [options, channel, onHello, onFailure] in
do {
let configuration = try HelperConfiguration(hello: channel.readHello(timeout: options.startupTimeout), runtimeKey: options.runtimeKey)
Task { @MainActor in await self?.connected(id, channel: channel, cancellation: cancellation, configuration: configuration) }
Task { @MainActor [onHello, configuration] in await onHello(configuration) }
} catch {
let failure = error as? NativeFailure ?? NativeFailure(code: "NATIVE_PROTOCOL", message: "Invalid shared helper hello")
let response = JSONValue.object(["type": .string("error"), "code": .string(failure.code), "message": .string(failure.message)])
if let data = try? JSONEncoder().encode(response) { try? channel.write(data + Data([10]), timeout: options.startupTimeout) }
channel.shutdown()
Task { @MainActor in self?.disconnected(id) }
Task { @MainActor [onFailure] in onFailure() }
}
}
}
@@ -155,14 +192,18 @@ final class NativeSignals {
let writer = JSONLWriter(limit: configuration.maxResponseBytes, channel: channel, writeTimeout: configuration.shutdownGraceSeconds)
do { try await writer.write(options.ready(pid: getpid(), instanceID: instanceID)) }
catch { channel.shutdown(); disconnected(id); return }
// Signal shutdown can run while the ready frame is being written.
guard !lifetime.isStopping else { channel.shutdown(); disconnected(id); return }
let lifetime = self.lifetime
let onFatal: @MainActor @Sendable () -> Void = { [weak self] in self?.closeConnections() }
let coordinator = RequestCoordinator(configuration: configuration, capture: capture, channel: channel, cancellation: cancellation,
scheduler: scheduler, writer: writer, onClose: { [weak self] in
scheduler: scheduler, writer: writer, permissions: permissions, onClose: { [weak self] in
channel.shutdown()
self?.disconnected(id)
}, onFatal: { [weak self] _, grace in
guard let self else { lifetime.remove(id); return }
self.disconnected(id)
}, onFatal: { [lifetime, onFatal] _, grace in
lifetime.abort(grace: grace)
Task { @MainActor in self?.closeConnections() }
Task { @MainActor [onFatal] in onFatal() }
})
connections[id] = coordinator
startupDeadline?.cancel()
packages/experimental/cua-macos/native/protocol.schema.json
+125
View File
@@ -1037,6 +1037,131 @@
"params"
],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"id": {
"type": "string"
},
"method": {
"type": "string",
"const": "permissionStatus"
},
"params": {
"type": "object",
"properties": {},
"additionalProperties": false
}
},
"required": [
"id",
"method",
"params"
],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"id": {
"type": "string"
},
"method": {
"type": "string",
"const": "applicationIcons"
},
"params": {
"type": "object",
"properties": {
"paths": {
"type": "array",
"items": {
"type": "string",
"minLength": 1
}
}
},
"required": [
"paths"
],
"additionalProperties": false
}
},
"required": [
"id",
"method",
"params"
],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"id": {
"type": "string"
},
"method": {
"type": "string",
"const": "requestPermission"
},
"params": {
"type": "object",
"properties": {
"permission": {
"type": "string",
"enum": [
"accessibility",
"screenRecording"
]
}
},
"required": [
"permission"
],
"additionalProperties": false
}
},
"required": [
"id",
"method",
"params"
],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"id": {
"type": "string"
},
"method": {
"type": "string",
"const": "openPermissionSettings"
},
"params": {
"type": "object",
"properties": {
"permission": {
"type": "string",
"enum": [
"accessibility",
"screenRecording"
]
}
},
"required": [
"permission"
],
"additionalProperties": false
}
},
"required": [
"id",
"method",
"params"
],
"additionalProperties": false
}
]
}
packages/experimental/cua-macos/native/tests/LauncherFixture.swift
+29
View File
@@ -0,0 +1,29 @@
// An owned GUI launcher with no windows; its child outlives the launch application.
import AppKit
import Foundation
import Darwin
@MainActor final class Launcher: NSObject, NSApplicationDelegate {
private let child = Process()
func applicationDidFinishLaunching(_ notification: Notification) {
let arguments = Array(CommandLine.arguments.dropFirst())
guard let executable = arguments.first else { exit(64) }
child.executableURL = URL(fileURLWithPath: executable)
child.arguments = Array(arguments.dropFirst())
child.standardOutput = FileHandle.standardOutput
child.standardError = FileHandle.standardError
child.terminationHandler = { @Sendable process in exit(process.terminationStatus) }
do { try child.run() }
catch { fputs("Launching fixture child failed: \(error)\n", stderr); exit(1) }
}
}
@main enum LauncherMain {
@MainActor static func main() {
let app = NSApplication.shared
app.setActivationPolicy(.accessory)
let launcher = Launcher()
app.delegate = launcher
withExtendedLifetime(launcher) { app.run() }
}
}
packages/experimental/cua-macos/native/tests/launcher_identity_e2e.py
+76
View File
@@ -0,0 +1,76 @@
"""Verify system responsibility across nested GUI launchers and helper reuse; no TCC grants are required."""
import argparse
import base64
from contextlib import ExitStack
import hashlib
import json
from pathlib import Path
import plistlib
import shutil
import struct
import subprocess
import tempfile
import uuid
from native_test_support import BINARY, ROOT
from shared_helper_e2e import Connection, wait_released
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--helper", type=Path, default=BINARY)
args = parser.parse_args()
with tempfile.TemporaryDirectory(prefix="cua-", dir="/tmp") as temporary, ExitStack() as cleanup:
root = Path(temporary)
helper = root / "helper"
shutil.copy2(args.helper.resolve(strict=True), helper)
key = hashlib.sha256(helper.read_bytes()).hexdigest()
runtime = root / "r"
bundle = root / "Visual Studio Code.app"
binaries = bundle / "Contents" / "MacOS"
binaries.mkdir(parents=True)
launcher = binaries / "launcher"
subprocess.run(["xcrun", "swiftc", "-parse-as-library", str(ROOT / "tests/LauncherFixture.swift"), "-o", str(launcher)],
check=True, timeout=60)
nested = root / "Harness Dev.app"
nested_launcher = nested / "Contents" / "MacOS" / "launcher"
nested_launcher.parent.mkdir(parents=True)
shutil.copy2(launcher, nested_launcher)
identity = uuid.uuid4().hex
register = "/System/Library/Frameworks/CoreServices.framework/Frameworks/LaunchServices.framework/Support/lsregister"
for role, application, name in [("owner", bundle, "Code"), ("nested", nested, "Harness Dev")]:
with (application / "Contents" / "Info.plist").open("wb") as output:
plistlib.dump({"CFBundleExecutable": "launcher",
"CFBundleIdentifier": f"dev.dsh.cua.launcher-fixture.{role}-{identity}",
"CFBundleName": name, "CFBundleDisplayName": name, "CFBundlePackageType": "APPL",
"LSUIElement": True}, output)
cleanup.callback(subprocess.run, [register, "-u", str(application)], check=True,
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, timeout=10)
ready_file, stderr_file = root / "ready.json", root / "stderr.log"
# Launch Services establishes the outer app's responsibility; its GUI child inherits it.
subprocess.run(["/usr/bin/open", "-n", "-g", "-W", "--stdout", str(ready_file), "--stderr", str(stderr_file),
str(bundle), "--args", str(nested_launcher), str(helper), "--ensure-shared", "--runtime-key", key,
"--runtime-directory", str(runtime), "--startup-timeout-ms", "10000"],
check=True, capture_output=True, text=True, timeout=25)
ready = json.loads(ready_file.read_text())
owner = ready["pid"]
cleanup.callback(wait_released, runtime / (key + ".lock"), owner)
first = Connection(runtime / (key + ".sock"), key)
cleanup.callback(first.close)
state = first.call("permissionStatus", {})
assert state["parent"]["pid"] == 1, state["parent"]
source = state["launchApplication"]
assert source["name"] == bundle.name, source["name"]
assert Path(source["path"]).resolve() == bundle.resolve(), source["path"]
assert source["icon"].startswith("data:image/png;base64,")
image = base64.b64decode(source["icon"].split(",", 1)[1], validate=True)
assert image[:8] == b"\x89PNG\r\n\x1a\n"
assert struct.unpack(">II", image[16:24]) == (64, 64)
second = Connection(runtime / (key + ".sock"), key)
cleanup.callback(second.close)
assert second.call("permissionStatus", {})["launchApplication"] == source
print("PASS: system-responsible app wins over the nearest GUI ancestor and survives launcher exit and helper reuse")
if __name__ == "__main__":
main()
packages/experimental/cua-macos/native/tests/shared_helper_e2e.py
+43 -14
View File
@@ -5,6 +5,7 @@ Accessibility/Screen Recording grants are required unless --without-desktop is s
"""
import argparse
import concurrent.futures
from contextlib import ExitStack
import fcntl
import hashlib
import json
@@ -128,6 +129,15 @@ def main():
assert a.ready == b.ready == ready[0]
for client in clients:
client.call("protocol-unsupported", {}, error="UNSUPPORTED")
with socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) as invalid:
invalid.settimeout(15)
invalid.connect(str(socket_path))
invalid.sendall(b'{"type":"hello","protocol":2}\n')
with invalid.makefile("rb") as reader:
failure = json.loads(reader.readline())
assert failure["type"] == "error" and failure["code"] == "NATIVE_PROTOCOL", failure
assert reader.readline() == b"", "Rejected handshake retained its connection"
b.call("protocol-after-invalid-hello", {}, error="UNSUPPORTED")
# Coalesced frames retain socket order, including cancellation before execution starts.
a.socket.sendall(b'{"id":"coalesced","method":"listApps","params":{}}\n'
b'{"type":"cancel","id":"coalesced"}\n')
@@ -218,20 +228,39 @@ def main():
wait_released(lock_path, owner_pid)
owner_pid = None
daemon = subprocess.Popen([base[0], "--serve", *base[1:]], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
try:
assert select.select([daemon.stdout], [], [], 15)[0], "Daemon did not report ready"
signalled_ready = json.loads(daemon.stdout.readline())
assert signalled_ready["type"] == "ready", signalled_ready
daemon.send_signal(signal.SIGTERM)
status = daemon.wait(timeout=15)
assert status == 128 + signal.SIGTERM, (status, daemon.stderr.read())
wait_released(lock_path, daemon.pid)
finally:
stop(daemon)
daemon.stdout.close()
daemon.stderr.close()
print("PASS: concurrent launch, shared PID, client isolation, cleanup/lock release and SIGTERM")
for shutdown in ["signal", "incomplete-hello", "idle"]:
serve_options = base[1:] if shutdown != "idle" else [*base[1:-1], "100"]
daemon = subprocess.Popen([base[0], "--serve", *serve_options], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
try:
with ExitStack() as connections:
assert select.select([daemon.stdout], [], [], 15)[0], "Daemon did not report ready"
signalled_ready = json.loads(daemon.stdout.readline())
assert signalled_ready["type"] == "ready", signalled_ready
if shutdown == "incomplete-hello":
pending = connections.enter_context(socket.socket(socket.AF_UNIX, socket.SOCK_STREAM))
pending.settimeout(15)
pending.connect(str(socket_path))
pending.sendall(b'{"type":"hello"')
# A later completed handshake observes the accept loop progressing past this socket.
active = Connection(socket_path, key)
connections.callback(active.socket.close)
connections.callback(active.reader.close)
if shutdown != "idle":
daemon.send_signal(signal.SIGTERM)
status = daemon.wait(timeout=15)
expected_status = 0 if shutdown == "idle" else 128 + signal.SIGTERM
assert status == expected_status, (status, daemon.stderr.read())
wait_released(lock_path, daemon.pid)
# The forced-exit deadline does not unlink these files; require normal cleanup.
assert not socket_path.exists(), "Shutdown left its listening socket"
assert not (runtime / (key + ".ready")).exists(), "Shutdown left its ready record"
if shutdown == "incomplete-hello":
assert pending.recv(1) == b"", "Signal shutdown retained an incomplete handshake"
finally:
stop(daemon)
daemon.stdout.close()
daemon.stderr.close()
print("PASS: concurrent launch, shared PID, client isolation, rejected/incomplete handshakes, cleanup/lock release, SIGTERM and idle exit")
finally:
errors = []
for client in reversed(clients):
packages/experimental/cua-macos/native/tests/swift/PermissionIdentityTests.swift
+36
View File
@@ -1,5 +1,7 @@
// The selected catalog entry must still identify the executable inspected for consent.
import AppKit
import Darwin
import Foundation
import Testing
@testable import CuaNative
@@ -22,3 +24,37 @@ import Testing
#expect(error.code == "APP_UNAVAILABLE")
}
}
@MainActor @Test func applicationIconsUseBoundedNativePNGs() throws {
let encoded = try #require(ApplicationCatalog.icon(at: "/System/Library/CoreServices/Finder.app/Contents/MacOS/Finder"))
#expect(encoded.hasPrefix("data:image/png;base64,"))
let data = try #require(Data(base64Encoded: String(encoded.dropFirst("data:image/png;base64,".count))))
let bitmap = try #require(NSBitmapImageRep(data: data))
#expect(bitmap.pixelsWide == 64)
#expect(bitmap.pixelsHigh == 64)
#expect(encoded.utf8.count <= 65536)
}
@Test func resolvesTheCurrentProcessResponsibility() throws {
let responsible = try #require(NativePlatform.responsiblePID(getpid()))
#expect(responsible > 0)
#expect(NativePlatform.responsiblePID(responsible) == responsible)
}
@MainActor @Test func unknownResponsibilityDoesNotInventAnApplication() {
#expect(SystemPermissions.responsibleApplication(for: 1) == nil)
#expect(SystemPermissions.responsibleApplication(for: Int32.max) == nil)
#expect(NativePlatform.responsiblePID(Int32.max) == nil)
}
@MainActor @Test func permissionStatusKeepsProcessDiagnosticsSeparateFromLaunchMetadata() {
let permissions = SystemPermissions()
let state = permissions.status()
#expect(state["helper"]["pid"] == .number(Double(getpid())))
#expect(state["parent"]["pid"] == .number(Double(getppid())))
if let source = state["launchApplication"].object {
#expect(source["name"]?.string?.isEmpty == false)
#expect(source["path"]?.string?.hasSuffix(".app") == true)
}
#expect(permissions.status()["launchApplication"] == state["launchApplication"])
}
packages/experimental/cua-macos/native/tests/swift/ProtocolTests.swift
+6
View File
@@ -116,3 +116,9 @@ func rejectsInvalidWindowIDs(_ value: String) throws {
#expect(throws: NativeFailure.self) { try params.optionalBoolean("includeMenus") }
}
}
@Test(arguments: ["null", "true", "[1]", "[\"\"]"])
func rejectsMalformedApplicationIconPaths(_ value: String) throws {
let request = try NativeRequest(data: Data("{\"id\":\"icons\",\"method\":\"applicationIcons\",\"params\":{\"paths\":\(value)}}".utf8))
#expect(throws: NativeFailure.self) { try RequestParameters(request).strings("paths") }
}
packages/experimental/cua-macos/package.json
+4 -1
View File
@@ -84,5 +84,8 @@
},
"scripts": {
"build:native": "swift build --package-path native --configuration release"
}
},
"os": [
"darwin"
]
}
packages/experimental/cua-macos/src/guidance.ts
+2 -2
View File
@@ -1,6 +1,6 @@
/** macOS operation guidance supplied with the shared CUA object API. */
/** Instructions for raw operation calls and optional object-API bindings. */
export const GUIDANCE = `Use cua with method getCapabilities and params {} to inspect permissions and sessionLocked. If sessionLocked is true or an operation reports SESSION_LOCKED, stop UI attempts and ask the user to unlock macOS. A null sessionLocked means no usable console session was verified. After unlocking, read fresh state; screenshots from a locked desktop are not current observations. Use listApps or cua.getState to discover running applications, and findApps(query) to search installed applications by name, bundle ID or absolute path. getApp requests user consent before launching a stopped application in the background, or before observations and input on a running one. If access is denied or revoked, stop using that application until the user explicitly grants access again. Never switch tools to bypass a refusal. Keep the returned appHandle; every app operation requires it. listWindows returns window IDs and point bounds. Raw getState returns AX text and snapshotId. Menu-bar entries are shown by default; includeMenus=true expands their contents. details=true adds identifiers, global-point bounds and complete action names. Both options also apply to getAXState. Each read replaces the indices for that app/window scope. Use its element indices only with that snapshotId. Raw screenshot and coordinate operations require a windowId. Coordinates are pixels in the latest returned screenshot, measured from its top-left. Set relative:true for integer thousandths from 0 to 1000 instead. The Host converts both modes to native window points; AX detail bounds remain global macOS points. A verified window can expose no AX elements. Its empty snapshot permits screenshots and coordinate input; element-index actions require an element from a fresh AX observation.
export const GUIDANCE = `Use cua with method getCapabilities and params {} to inspect permissions and sessionLocked. If sessionLocked is true or an operation reports SESSION_LOCKED, stop UI attempts and ask the user to unlock macOS. A null sessionLocked means no usable console session was verified. After unlocking, read fresh state; screenshots from a locked desktop are not current observations. Use listApps or cua.getState to discover running applications, and findApps(query) to search installed applications by name, bundle ID or absolute path. getApp requests user consent before launching a stopped application in the background, or before observations and input on a running one. If access is denied or revoked, stop using that application until the user explicitly permits another authorization request. Never switch tools to bypass a refusal. Keep the returned appHandle; every app operation requires it. listWindows returns window IDs and point bounds. Raw getState returns AX text and snapshotId. Menu-bar entries are shown by default; includeMenus=true expands their contents. details=true adds identifiers, global-point bounds and complete action names. Both options also apply to getAXState. Each read replaces the indices for that app/window scope. Use its element indices only with that snapshotId. Raw screenshot and coordinate operations require a windowId. Coordinates are pixels in the latest returned screenshot, measured from its top-left. Set relative:true for integer thousandths from 0 to 1000 instead. The Host converts both modes to native window points; AX detail bounds remain global macOS points. A verified window can expose no AX elements. Its empty snapshot permits screenshots and coordinate input; element-index actions require an element from a fresh AX observation.
Initialize the object API once in persistent PTC. Reuse its variables in later calls; put temporary declarations inside { ... } blocks to avoid redeclaration errors. All object methods are async. getAXState returns text, not a structured state object. Its text separates values and placeholders, and reports subroles, help, URLs, disabled/focused/selected/expanded state and UTF-16 selection ranges when available. Writability is true, false or unknown on relevant controls; omitted state fields do not assert false. selection=unavailable means the reported range exceeds the observed text. Compact output omits ordinary Press/Pick (click), ShowMenu (right click) and ScrollToVisible actions; details=true lists every available action. Use the displayed action name with performSecondaryAction; ambiguous aliases require the complete name from details=true. Truncation reports the reached provider limit. Narrow the window or increase that provider limit before reading again.
@@ -37,7 +37,7 @@ App and Window methods:
Raw operation tools use { method, params }, including params: {} for parameterless methods. getApp selects without activating the app. Keep applications in the background for observation and routine input. Do not activate or raise an app during initialization, before each action, or merely because another app is foreground. Prefer advertised AX actions. If a window is hidden or minimized, try app.show() or window.show() and read fresh state. show requests unhide and unminimize without activation, raising or switching Spaces; it chooses the explicit, focused, main or sole AX window. A null windowId means no window was selected for restoration. foregroundChanged compares the front app and top window before and after the request; null means unavailable, and delayed focus changes remain possible. If restoration is unsupported or the window remains unusable, including another Space or Stage Manager, explain the needed foreground step, use app.activate() or an advertised raise action, and read fresh state. Completing the task takes priority when foreground access is necessary. Avoid repeated activation when the user switches away; continue background-capable steps and report what requires foreground access. Applications may still change focus in response to input.
Await operations sequentially. After input, read fresh state to verify the effect. A successful action means dispatched, not verified completion. Cancellation may leave partial effects; never retry uncertain input without observing. STALE_APP requires getApp again; STALE_SNAPSHOT requires getState again. Absolute coordinate input requires a fresh screenshot of that window. Relative input can use a successful AX observation or screenshot. SCREENSHOT_REQUIRED means capture a fresh screenshot before using pixel coordinates. For click and hover, coordinate targets must be [x, y], for example hover([650, 400], {relative:true}). A bare number such as hover(650) is always an AX element index. The relative option applies only to coordinate targets, not AX indices. Window resizing since the latest observation rejects coordinate input; movement alone preserves window-local coordinates. WINDOW_CHANGED requires fresh state before any retry; use an advertised AX action or explicitly restore the window before screenshot or coordinate input, then observe again. PERMISSION_DENIED requires the user to grant macOS Accessibility or Screen Recording access in System Settings. After granting access, observe fresh state before continuing; do not automatically replay prior input.
Await operations sequentially. After input, read fresh state to verify the effect. A successful action means dispatched, not verified completion. Cancellation may leave partial effects; never retry uncertain input without observing. STALE_APP requires getApp again; STALE_SNAPSHOT requires getState again. Absolute coordinate input requires a fresh screenshot of that window. Relative input can use a successful AX observation or screenshot. SCREENSHOT_REQUIRED means capture a fresh screenshot before using pixel coordinates. For click and hover, coordinate targets must be [x, y], for example hover([650, 400], {relative:true}). A bare number such as hover(650) is always an AX element index. The relative option applies only to coordinate targets, not AX indices. Window resizing since the latest observation rejects coordinate input; movement alone preserves window-local coordinates. WINDOW_CHANGED requires fresh state before any retry; use an advertised AX action or explicitly restore the window before screenshot or coordinate input, then observe again. PERMISSION_DENIED requires the user to grant the required macOS system permissions for macOS Accessibility or Screen Recording. After granting access, observe fresh state before continuing; do not automatically replay prior input.
Hover first uses advertised AX alternate UI without moving the system cursor, preserving revealed descendants until the pointer target leaves their region. When unavailable, it moves the visible system cursor without pressing a button; that route requires the target point to be unobscured in the foreground app. FOREGROUND_REQUIRED requires explicit activation or raising and a fresh observation before retrying. Returning, AX reads and screenshots do not restore the previous point. Wait for the application's hover delay, observe, then hover into the revealed controls before clicking. Use successive hover calls for intermediate points. The result reports hoverRoute and systemPointerMoved. Other input or application changes can interrupt the interaction; app objects do not reserve applications between calls.
packages/experimental/cua-macos/tests/large-image-ptc.spec.ts
+1 -1
View File
@@ -73,7 +73,7 @@ it.skipIf(process.platform === 'win32')('delivers a valid PNG larger than 4 MiB
const configPath = join(root, 'cordis.yml')
const configs: Record<string, object> = {
tools: { mode: 'ptc' }, attachments: { dshHome: join(root, 'home') }, workingDirectory: { defaultDirectory: root },
fs: { cwd: root }, sandboxPolicy: { mode: 'danger-full-access', workspaceRoot: root }, cua: { helperPath, allowedApps: [{ 'id': 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa', 'name': 'Fixture', 'executablePath': '/fixture/app' }] },
fs: { cwd: root }, sandboxPolicy: { mode: 'danger-full-access', workspaceRoot: root }, cua: { agentPreset: false, helperPath, allowedApps: [{ 'id': 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa', 'name': 'Fixture', 'executablePath': '/fixture/app' }] },
}
await writeFile(configPath, Object.keys(modules).flatMap(name => [
`- name: 'cordis:${name}'`, ...configs[name] === undefined ? [] : [` config: ${JSON.stringify(configs[name])}`],
packages/experimental/cua-macos/tests/loader-composition.spec.ts
+1 -1
View File
@@ -85,7 +85,7 @@ it.skipIf(process.platform === 'win32')('shares one application between retained
await writeFile(configPath, Object.keys(modules).flatMap(name => [
`- name: 'cordis:${name}'`,
...name === 'attachments' ? [' config:', ` dshHome: ${JSON.stringify(root)}`] : [],
...name === 'cua' ? [' config:', ` helperPath: ${JSON.stringify(helperPath)}`, ' allowedApps: [{"id": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", "name": "Fixture", "executablePath": "/fixture/app"}]'] : [],
...name === 'cua' ? [' config:', ' agentPreset: false', ` helperPath: ${JSON.stringify(helperPath)}`, ' allowedApps: [{"id": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", "name": "Fixture", "executablePath": "/fixture/app"}]'] : [],
]).join('\n') + '\n')
context.baseUrl = pathToFileURL(root).href + '/'
await context.plugin(Loader)
packages/experimental/cua-macos/tests/persistent-bridge.built.e2e.ts
+1 -1
View File
@@ -50,7 +50,7 @@ it.skipIf(process.platform === 'win32' || !existsSync(bridge) || !existsSync(boo
await ctx.plugin(NodeRuntime, { executionMode: 'persistent-in-memory', launch: { kind: 'node-script', executable: process.execPath, bootstrapPath: bootstrap } })
await ctx.plugin(Loader)
ctx.loader.builtins.cua = Cua
const entry = await ctx.loader.create({ name: 'cordis:cua', config: { helperPath, allowedApps: [{ 'id': 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa', 'name': 'Fixture', 'executablePath': '/fixture/app' }] } })
const entry = await ctx.loader.create({ name: 'cordis:cua', config: { agentPreset: false, helperPath, allowedApps: [{ 'id': 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa', 'name': 'Fixture', 'executablePath': '/fixture/app' }] } })
await ctx.loader.resolve(entry).fiber!.await()
await mountAgentLoopTestHarness(ctx)
ctx.llm.registerAdapter(['bridge-fixture'], new VisualAdapter())
packages/experimental/cua-macos/tests/protocol-generation.spec.ts
+4
View File
@@ -18,6 +18,10 @@ describe('native protocol generation', () => {
{ method: 'paste', params: { appHandle: 'app', text: 'text', pasteReadTimeoutMs: 1000 } },
{ method: 'findApps', params: { query: 'Editor', maxAppMatches: 1 } },
{ method: 'launchApp', params: { appHandle: 'pending' } },
{ method: 'applicationIcons', params: { paths: ['/Applications/Fixture.app'] } },
{ method: 'permissionStatus', params: {} },
{ method: 'requestPermission', params: { permission: 'accessibility' } },
{ method: 'openPermissionSettings', params: { permission: 'screenRecording' } },
])('rejects Host-only request data at model admission: $method', (request) => {
expect(Request.safeParse(request).success).toBe(false)
})
packages/experimental/cua-macos/tests/provider-lifecycle.spec.ts
+31 -9
View File
@@ -55,6 +55,7 @@ vi.mock('@deepseek-ai/dsh-cua/native-client', async importOriginal => ({
}))
const config = {
agentPreset: false,
helperPath: resolve('fixture-helper'), buildTimeoutMs: 300_000, timeoutMs: 10_000, connectTimeoutMs: 5000, screenshotTimeoutMs: 5000, axTimeoutMs: 3000, maxRequestBytes: 1024, maxResponseBytes: 4096, maxClipboardBytes: 8192, pasteReadTimeoutMs: 1000, maxAppMatches: 20,
graceMs: 10, textEventIntervalMs: 8, maxClickHoldMs: 7000, maxElements: 64, maxDepth: 8, dragDurationMs: 200, dragSteps: 10,
} satisfies Omit<NativeProvider.Config, 'allowedApps'>
@@ -99,9 +100,9 @@ async function setup(model?: { provider: string; model: string }) {
async function selectApplication(f: Awaited<ReturnType<typeof setup>>) {
await f.ctx.plugin(UserQuestions)
f.ctx.on('user-questions/request', async () => ({ answers: [{ id: 'cua-app-access', selected: ['Allow for this task'] }] }))
f.ctx.on('user-questions/request', async () => ({ answers: [{ id: 'cua-app-access', selected: ['Allow for this session'] }] }))
native.request.mockResolvedValueOnce({ appHandle: 'app-one', pid: 17, name: 'Fixture',
authorization: { id: 'a'.repeat(64), name: 'Fixture', executablePath: '/fixture/app', persistable: true } })
authorization: { id: 'a'.repeat(64), name: 'Fixture', executablePath: '/fixture/app', persistable: false } })
expect((await f.execute('getApp', { selector: 'Fixture' })).isError).toBe(false)
}
@@ -111,7 +112,7 @@ it('rejects a relative helper path before publishing computer-use operations', a
await expect(NativeProvider.apply(ctx, { ...config, allowedApps: { get: () => [] }, helperPath: 'relative-helper' })).rejects.toThrow('helperPath must be absolute')
})
it.each(['Allow for this task', 'Deny'])('obtains %s before a stopped application can launch', async (choice) => {
it.each(['Allow for this session', 'Deny'])('obtains %s before a stopped application can launch', async (choice) => {
const f = await setup()
await f.ctx.plugin(UserQuestions)
const asked: PromiseWithResolvers<void> = Promise.withResolvers()
@@ -136,7 +137,7 @@ it.each(['Allow for this task', 'Deny'])('obtains %s before a stopped applicatio
it('rejects a launch result that changes the approved handle', async () => {
const f = await setup()
await f.ctx.plugin(UserQuestions)
f.ctx.on('user-questions/request', async () => ({ answers: [{ id: 'cua-app-access', selected: ['Allow for this task'] }] }))
f.ctx.on('user-questions/request', async () => ({ answers: [{ id: 'cua-app-access', selected: ['Allow for this session'] }] }))
native.request.mockResolvedValueOnce({ appHandle: 'pending-app', pid: null, name: 'Fixture', authorization: {
id: 'a'.repeat(64), name: 'Fixture', executablePath: '/fixture/app', persistable: false,
} }).mockResolvedValueOnce({ appHandle: 'another-app', pid: 17, name: 'Fixture' })
@@ -263,6 +264,24 @@ it('keeps Agent disposal pending until its helper cleanup completes', async () =
} finally { release.resolve(); await disposing }
})
it('retains the computer-use registration until permission helper cleanup settles', async () => {
const f = await setup()
native.request.mockResolvedValue({ accessibility: false, screenRecording: false, helper: { pid: 9 }, parent: { pid: 8 } })
await f.ctx.cuaAccess.systemStatus()
const entered = Promise.withResolvers<undefined>(), release = Promise.withResolvers<undefined>()
native.dispose.mockImplementation(async () => { entered.resolve(undefined); await release.promise })
onTestFinished(() => { release.resolve(undefined) })
const registry = f.ctx.computerUse
const closing = f.provider.dispose()
try {
await entered.promise
expect(registry.providerName).toBe('dsh-cua')
release.resolve(undefined)
await closing
expect(registry.providerName).toBeUndefined()
} finally { release.resolve(undefined); await closing }
})
it('joins helper cleanup already started by Agent disposal when the provider unloads', async () => {
const f = await setup()
await f.execute()
@@ -503,6 +522,8 @@ it.each([
{ owner: 'provider', stage: 'storage' },
{ owner: 'agent', stage: 'admission' },
{ owner: 'agent', stage: 'storage' },
{ owner: 'revocation', stage: 'admission' },
{ owner: 'revocation', stage: 'storage' },
])('waits for screenshot $stage and cancels its result during $owner disposal', async ({ owner, stage }) => {
const f = await setup({ provider: 'image-fixture', model: 'vision' })
await selectApplication(f)
@@ -532,7 +553,9 @@ it.each([
const reached = await Promise.race([entered.promise.then(() => true), pending.then(() => false)])
expect(reached).toBe(true)
let closed = false
closing = (owner === 'provider' ? f.provider.dispose() : f.handle.dispose()).then(() => { closed = true })
closing = (owner === 'provider' ? f.provider.dispose()
: owner === 'agent' ? f.handle.dispose()
: f.ctx.cuaAccess.revoke('a'.repeat(64) as import('@deepseek-ai/dsh-cua/types').CuaApplicationId)).then(() => { closed = true })
await setImmediate()
expect(admissionSignal?.aborted).toBe(true)
expect(closed).toBe(false)
@@ -629,7 +652,6 @@ it('rejects an unsupported native host before starting preparation', async () =>
expect(native.prepare).not.toHaveBeenCalled()
})
it('stores signed application consent in the current profile and refuses unavailable settings', async () => {
const f = await setup()
await f.provider.dispose()
@@ -655,7 +677,7 @@ it('stores signed application consent in the current profile and refuses unavail
await mountRootInclude(f.ctx, join(dir, 'cordis.yml'), readProfilePatches('test', profile))
await f.ctx.loader.await()
await f.ctx.plugin(UserQuestions)
const answer = vi.fn(async () => ({ answers: [{ id: 'cua-app-access', selected: ['Allow for this task'] }] }))
const answer = vi.fn(async () => ({ answers: [{ id: 'cua-app-access', selected: ['Allow for this session'] }] }))
f.ctx.on('user-questions/request', answer)
native.request.mockResolvedValue({ appHandle: 'task-app', pid: 17, name: 'Fixture',
authorization: { id: 'a'.repeat(64), name: 'Fixture', executablePath: '/fixture/app', persistable: true } })
@@ -666,7 +688,7 @@ it('stores signed application consent in the current profile and refuses unavail
await f.ctx.hmr.runExclusive(async () => {})
expect(f.ctx.settings.describe().find(row => row.ns === 'cua')?.autoGenerate).toBe(false)
answer.mockResolvedValue({ answers: [{ id: 'cua-app-access', selected: ['Always allow'] }] })
const authorization = { id: 'a'.repeat(64), name: 'Fixture', executablePath: '/fixture/app', persistable: true }
const authorization = { id: 'b'.repeat(64), name: 'Fixture', executablePath: '/fixture/app', persistable: true }
native.request.mockResolvedValue({ appHandle: 'signed-app', pid: 17, name: 'Fixture', authorization })
expect((await f.execute('getApp', { selector: 'Fixture' })).isError).toBe(false)
expect(await readFile(profile.patchPath, 'utf8')).toContain(authorization.id)
@@ -677,7 +699,7 @@ it('stores signed application consent in the current profile and refuses unavail
expect(revoked.isError).toBe(true)
expect(JSON.stringify(revoked.content)).toContain('APP_ACCESS_REVOKED')
vi.spyOn(f.ctx.settings, 'describe').mockReturnValue([])
native.request.mockResolvedValue({ appHandle: 'different-app', pid: 18, name: 'Other fixture', authorization: { ...authorization, id: 'b'.repeat(64) } })
native.request.mockResolvedValue({ appHandle: 'different-app', pid: 18, name: 'Other fixture', authorization: { ...authorization, id: 'c'.repeat(64) } })
const unavailable = await f.execute('getApp', { selector: 'Fixture' })
expect(unavailable.isError).toBe(true)
expect(JSON.stringify(unavailable.content)).toContain('APP_ACCESS_DENIED')
packages/experimental/cua-macos/tsconfig.json
+1 -1
View File
@@ -12,7 +12,7 @@
"path": "../../../vendor/cordis"
},
{
"path": "../cua"
"path": "../cua/tsconfig.host.json"
},
{
"path": "../../subprocess/subprocess"
packages/experimental/cua/README.i18n.yaml
+13 -10
View File
@@ -3,8 +3,8 @@
# After editing either side, bring the other along and re-record with:
# pnpm run verify-translation-pairing --write packages/experimental/cua/README.md
/:
en: aa3d316ef17feae7
zh: 5a30d743ce257d5b
en: ed2539c6d12f814a
zh: 55a6c9d7f0d21fb7
/deepseek-ai-dsh-cua:
en: 0ef7ee882840cf1b
zh: 01b9e768eff2d302
@@ -12,14 +12,17 @@
en: f4d9cc48587d62a2
zh: d1bac6c1fe7eb29e
/deepseek-ai-dsh-cua/table-of-contents:
en: d152484eb41ac6b4
zh: 1db86285abf6c499
en: 4081cf444a77bf20
zh: def91d58969ec281
/deepseek-ai-dsh-cua/use-this-package:
en: bdb95a169555b365
zh: f4bdfbba6809b566
en: ceb54d964dc355b6
zh: 0892066fbceef7b9
/deepseek-ai-dsh-cua/use-this-package/permission-controls:
en: be53792edc381ea1
zh: 037407055d9f34a0
/deepseek-ai-dsh-cua/understand-the-implementation:
en: effcee57d8854c32
zh: d02c6b76e7249051
en: 1fde7df44f040131
zh: 59a842fa402ccb35
/deepseek-ai-dsh-cua/further-exploration:
en: b8b0aece886fadaa
zh: ecd8c88d601f2179
@@ -39,8 +42,8 @@
en: 2d5df52dafa1d940
zh: 06e815759b9150c3
/deepseek-ai-dsh-cua/known-limitations-and-deferred-work:
en: da6aefcb279aea83
zh: b45b7bb300b9f792
en: ae98f1b2718b3b94
zh: e6bf4e96ae08e669
/deepseek-ai-dsh-cua/known-limitations-and-deferred-work/dev-note:
en: 0a1253c6f7b5e8b9
zh: 9f56528562c7c914
packages/experimental/cua/README.md
+27 -2
View File
@@ -1,6 +1,6 @@
---
description: "Reuse Computer Use application and window objects with authorized, logged platform operations."
kind: "package-library"
kind: "package-reference"
---
# @deepseek-ai/dsh-cua
@@ -20,6 +20,8 @@ Inspect applications, retain window objects across PTC calls, and send actions t
- [Known Limitations and Deferred Work](#known-limitations-and-deferred-work)
- [Dev Note](#dev-note)
Application consent uses the existing User Questions options and remains independent of the session tool-approval policy. The provider accepts only one exact offered label without custom text. Native icons accompany permission-page observations and remain outside saved identities and model results. System checks, saved grants and refusals, and icons use separate queries.
-----
<a id="use-this-package"></a>
@@ -33,12 +35,24 @@ Observation methods emit and return their result. Use `void await app.getAXState
`tools.cua` returns operation values directly. Screenshot values retain the encoded image Base64 and add a durable `attachment` reference for the tool renderer; model content contains coordinate metadata and one image, without the encoded bytes. Before capture, the tool requires an attachment store and a resolved model route declaring image input. Admission and storage failures reject the call. The object bridge preserves the screenshot bytes and validated metadata without uploading or emitting the same image again.
Application consent precedes launch, observation, capture, and input. Task access belongs to the selected instance; saved grants require platform-verified executable identity. Revocation closes affected Agent connections, so their old objects require fresh selection. No input is replayed after cancellation, timeout, or reconnect.
Application consent precedes launch, observation, capture, and input. Session access is kept in memory by session and application identity, survives agent and helper replacement, and expires when DSH restarts or Computer Use is disabled. Saved grants require platform-verified executable identity. Refusal and revocation close affected Agent connections, so their old objects require fresh selection. No input is replayed after cancellation, timeout, or reconnect.
Actions return `dispatched:true, verified:false`; read fresh state to verify the result. Coordinate actions default to pixels in the latest returned screenshot; `relative: true` accepts integers from 0 to 1000 after a successful window observation. The Host resolves attachment normalization and the selected model route’s image projection before returning the screenshot, then converts input to native window points. Resizing invalidates the screenshot mapping; `SCREENSHOT_REQUIRED` requires a new screenshot for absolute coordinates. Resizing rejects old coordinates with `WINDOW_CHANGED`; movement alone preserves window-local coordinates.
`click(target, { holdMs: 800 })` holds the mouse button for 800 milliseconds before release. `holdMs` accepts nonnegative integers and applies to each press of a double-click; zero releases without an additional hold. The provider config `maxClickHoldMs` limits explicit holds and defaults to 5000 ms. Explicit holds use pointer input for both AX indices and coordinates; omission preserves ordinary click behavior. The operation deadline includes the hold, and cancellation or timeout releases the button. Received input cannot be undone; forced helper termination cannot guarantee release.
### Permission controls
The [Computer Use bundle](../cua-bundle/README.md) mounts this page with its native provider. **Authorize Computer Use** groups the new-session default, PTC execution mode, Accessibility, and Screen Recording inside one border, with dividers between the rows. Each system-permission row shows the system-reported responsible application’s `.app` bundle filename and icon together in a bordered label. **Manage applications Computer Use can access** places the application lists inside a separate bordered group. Choose **Request permission**, find that application in the macOS permission list, and enable its switch; if macOS names another application, follow that prompt. With no detected source, the page directs the user to the application named by macOS. Each system-permission row has one control: **Request permission** when access is missing, or an outlined **Allowed** button with a settings icon after authorization. Clicking it opens the corresponding privacy pane, where the user can revoke access; hovering shows **Open System Settings**. Unknown status uses the same button with an **Unknown** label; pending initial observations retain a single placeholder. The page refreshes on entry, when the window regains focus, and after permission actions. A failed query offers **Retry**. Follow any macOS quit-and-reopen prompt and restart DSH as needed.
System-permission operations affect the computer running DSH. When accessing DSH remotely, complete authorization on that computer.
Enabling from the plugin list shows setup guidance when system permissions need attention, with **Later** and **Open settings** actions. Enabling from the detail page displays the permission controls directly. The detail page provides **Set to PTC** under **Default mode for new sessions**; only this action saves the default. Existing sessions keep their preset. **Use persistent mode** in the **PTC execution mode** row separately saves `ptc-runtime.executionMode: persistent-in-memory`. Switching reloads the shared PTC runtime and may interrupt an active call; subsequent calls retain variables and application objects in memory. Already-persistent runtimes show a disabled **Persistent mode enabled** button.
The page lists always-allowed applications and refusals from any session. A refusal in one session does not cancel another session’s grant or prevent a new session from asking. Session-only grants remain effective within their sessions and are omitted from this page. **Revoke** removes the selected application's access and cancels affected work. Input already delivered to an application cannot be undone; observe the application before any further action. **Allow asking again** clears that application’s refusal records across all sessions; the next application selection still requires a new decision.
The Host marker has no configurable fields; platform providers own native configuration.
-----
<a id="understand-the-implementation"></a>
@@ -47,10 +61,20 @@ Actions return `dispatched:true, verified:false`; read fresh state to verify the
<details>
<summary>Shared execution and platform ownership</summary>
The permission Remote also exposes `ptcConfiguration()` and `enablePersistentPtc()` for the detail-page runtime setting. Reads inspect the existing runtime and editable profile entry. Explicit writes use `configEditor.edit()` to preserve other fields and apply normal Loader reload; an already-persistent runtime is left intact. This setting is separate from system and application authorization.
The provider calls [installCua](src/index.ts) with its prepared helper, connection factory, and guidance. [NativeClient](src/native-client.ts) owns request correlation, byte limits, cancellation acknowledgement, and failure settlement. The platform connector owns startup and connection cleanup. The shared executor retains one connection per Agent and joins in-flight image writes during disposal.
`agentPreset` defaults to `ptc`, restricting tools and guidance to Agents bound to that preset. The provider uses the existing `agent/created` and `agent-preset/selected` events and Agent-owned scopes, including Agents already present when it loads. Provider unload and Agent disposal release those registrations. Preset definitions may register after the provider; a preset name filters bound Agents and supplies no tools until a matching Agent exists. Set `agentPreset: false` for global registration in standalone compositions without presets. The Host continues to own permissions, native connections and cancellation; preset definitions remain unchanged.
[The operation schema](src/protocol.ts) is the authority for model parameters and separately declared Host-only fields. Native implementations enforce target identity, window geometry, resource limits, and operating-system access at execution. No invariant companion is published because this library has no separately projected registry or durable state.
The root Host entry exports `installCua`, `CuaConfig`, and the Typert-discovered `CuaAccess` service; its `apply()` only marks the optional Client contribution. `/client` loads the permission UI through the existing module loader. Separate Host and Client compiler projects keep React, CSS, and browser registrations out of headless imports. `/api` owns the app/window API and its types; `/bridge`, `/native-client`, and `/protocol` retain runtime and platform consumers. `/types` exports only the four named boundary types required by generated Remote declarations. The Client uses package-local types. `/typert` supplies Host reflection and `/remote` supplies generated browser descriptors and codecs; permission authority remains on the Host.
The optional browser contribution mounts the native provider's permission Remote and reuses the Plugins activation and configuration slots. The four setup rows remain visible while unknown fields use placeholders. The default-mode form, runtime configuration, system checks, application decisions, and application icons fill in as each query completes. Application names and actions do not wait for icons; icon positions retain their space. User actions run in order without waiting for pending observation queries. Disposal waits for both actions and queries to settle before releasing the separately owned Remote mount. Query errors stay with the affected section, and activation guidance depends only on system checks. Late results after disposal are discarded. The controller consumes the generated `ctx.remote.cuaAccess` namespace directly and handles unsuccessful Remote results in its existing query and action paths. It projects queries into the page and setup dialog; reconnects and updates to the Host-reported settings namespace refresh it. Operation notices live in the shell overlay, so closing the page does not remove them. The provider owns permission checks, persisted decisions, and cancellation; the browser owns no authorization state.
The detail page reads the effective default (`selectedDefault ?? default`) from the existing `configForms` service. The default button writes only `selectedDefault` through the form's revision-checked settings path. The execution-mode row uses CUA's `ptcConfiguration()` and `enablePersistentPtc()` controls: the Host edits only `ptc-runtime.executionMode` through the existing configuration editor, preserving its ordinary Loader reload lifecycle. Permission checks and runtime state settle independently. Loading, read-only and unavailable settings cannot submit a change; a failed write remains beside the setting for retry.
</details>
-----
@@ -84,6 +108,7 @@ Results append to history. Retaining objects does not rewrite previous messages;
<a id="known-limitations-and-deferred-work"></a>
- In-memory session decisions have no capacity limit or eviction. They remain until the provider unloads, so a long-running Host retains entries for every session/application pair that requested access.
- Native execution currently uses the macOS provider. Windows support is planned as a separate provider package with a C# helper, retaining the shared app/window API, authorization, snapshot validity, and action-result semantics. Its implementation must document system-specific limitations and distinguish cross-compilation evidence from Windows desktop validation.
- Persistent variables do not reserve an application across a sequence of operations. Other Agents and the user can change its state. Hover and observations do not restore the previous pointer position; [continuous interaction](../../../.agents/notes/implemented/architecture/2026-10-13-cua-continuous-hover.md) uses no cross-call ownership.
- The package stays in `experimental` until Windows support and platform validation permit promotion to the `computer-use` group.
packages/experimental/cua/README.zh.md
+27 -2
View File
@@ -1,6 +1,6 @@
---
description: "复用 Computer Use 应用和窗口对象，通过已授权并记录日志的平台操作完成任务。"
kind: "package-library"
kind: "package-reference"
---
# @deepseek-ai/dsh-cua
@@ -20,6 +20,8 @@ kind: "package-library"
- [已知限制与后续工作](#known-limitations-and-deferred-work)
- [开发备注](#dev-note)
应用授权使用现有 User Questions 选项，仍独立于会话的工具审批策略。提供方只接受一个准确匹配所给选项的标签，附带自由文本的回答不会授权。本机图标用于权限页展示，不进入保存的身份信息或模型结果。系统权限、持久授权与拒绝记录、图标分别查询。
-----
<a id="use-this-package"></a>
@@ -33,12 +35,24 @@ kind: "package-library"
`tools.cua` 直接返回操作结果。截图结果保留图片 Base64，并增加供工具渲染使用的持久 `attachment` 引用；模型内容只包含坐标元数据和一张图片，不包含编码字节。截图前要求附件服务可用，并确认当前模型支持图片输入。图片接入或保存失败会拒绝调用。对象桥接保留截图字节和已校验的元数据，不重复上传或输出同一张图片。
应用授权发生在启动、观察、截图和输入之前。任务许可绑定已选择的应用实例；长期许可要求平台验证可执行文件身份。撤销会关闭受影响的 Agent 连接，因此旧对象必须重新选择目标。取消、超时或重新连接后均不重放输入。
应用授权发生在启动、观察、截图和输入之前。会话许可按会话和应用身份保存在内存中，Agent 或 helper 重建后仍可复用，重启 DSH 或关闭 Computer Use 后失效。长期许可要求平台验证可执行文件身份。拒绝或撤销会关闭受影响的 Agent 连接，因此旧对象必须重新选择目标。取消、超时或重新连接后均不重放输入。
动作返回 `dispatched:true, verified:false`；请读取新状态以验证结果。坐标操作默认使用最近返回截图中的像素；成功观察窗口后，传入 `relative: true` 可使用 0～1000 的整数比例。Host 在返回截图前解析附件归一化和当前模型的图片缩放规则，再将输入换算为原生窗口 point。窗口尺寸变化会使截图映射失效，绝对坐标遇到 `SCREENSHOT_REQUIRED` 时需要重新截图。窗口缩放后以 `WINDOW_CHANGED` 拒绝旧坐标；单纯移动窗口不改变窗口内坐标。
`click(target, { holdMs: 800 })` 按住鼠标按钮 800 毫秒后松开。`holdMs` 接受非负整数，双击时应用于每次按下；0 表示按下后立即松开，不额外等待。提供方配置 `maxClickHoldMs` 限制显式按住时长，默认 5000 毫秒。指定时长后，AX 索引和坐标目标都使用鼠标事件；省略时保留普通点击行为。按住时间计入操作超时，取消或超时会松开按钮。已接收的输入无法撤销；辅助进程被强制终止时无法保证松开。
### 权限控件
[Computer Use 组合](../cua-bundle/README.zh.md)将本页与原生 provider 一起挂载。“给 Computer Use 授权”把新会话默认模式、PTC 执行方式、辅助功能和屏幕录制放在同一个边框内，四排之间使用分隔线。两条系统权限要求在能识别责任应用时，把该应用的 `.app` 包名称和图标放在同一个标签中。“管理 Computer Use 可访问的应用”使用独立边框包住应用列表。点击“申请权限”，在 macOS 权限列表中找到该应用并开启对应开关；若 macOS 显示其他名称，以系统提示为准。来源无法识别时，页面引导用户查找系统提示的应用。每条系统权限要求只显示一个控件：未授权时为“申请权限”，已授权时为带设置图标的“已允许”描边按钮，点击直接进入对应隐私设置页，由用户撤销权限，悬停提示“打开系统设置”。无法确认状态时使用同样的按钮，文字为“未确认”。首次查询期间保留一个占位。进入详情页、返回窗口或完成授权操作后会自动刷新；查询失败时可点击“重试”。若系统要求退出并重新打开应用，请按提示操作并重新启动 DSH。
系统权限操作作用于运行 DSH 的电脑。远程访问时，请在该电脑上完成授权。
从插件列表启用后，系统权限尚未就绪时会显示设置引导，提供“稍后”和“前往设置”。从详情页启用后直接显示权限控件。详情页的“新会话默认模式”提供“设为 PTC”按钮，仅点击此按钮才保存默认值。已有会话保留原 preset。“PTC 执行方式”一行的“设为持久化”单独保存 `ptc-runtime.executionMode: persistent-in-memory`。切换会重载共享 PTC 执行器，可能打断当前调用；后续调用在内存中保留变量和应用对象。已经使用持久化模式时，按钮显示“已设为持久化”并禁用。
页面展示“始终允许的应用”和“不允许访问的应用”。后者汇总任一会话的拒绝记录，不取消其他会话已有的授权，也不阻止新会话询问授权。会话授权仍在对应会话中生效，详情页不展示这类应用。“撤销”会移除对应应用的访问许可并取消相关操作。已发送到应用的输入无法撤回，继续操作前需重新观察应用。“允许再次询问”清除该应用在所有会话中的拒绝记录；下次选择应用时仍需用户重新决定。
本插件没有配置字段。
-----
<a id="understand-the-implementation"></a>
@@ -47,10 +61,20 @@ kind: "package-library"
<details>
<summary>共享执行与平台职责</summary>
权限 Remote 还通过 `ptcConfiguration()` 和 `enablePersistentPtc()` 支持详情页的执行方式设置。查询读取现有执行器和可编辑的 profile 条目。显式写入使用 `configEditor.edit()` 保留其他字段并按正常 Loader 规则重载；已是持久化模式的执行器保持不变。此设置独立于系统权限和应用授权。
提供方将准备好的 helper、连接工厂和指引传给 [installCua](src/index.ts)。[NativeClient](src/native-client.ts) 负责请求关联、字节上限、取消确认和失败处理。平台连接器负责启动和连接清理。共享执行器为每个 Agent 保留一个连接，并在销毁时等待进行中的图像写入。
`agentPreset` 默认为 `ptc`，将工具和指引限制到绑定该 preset 的 Agent。提供方使用现有 `agent/created`、`agent-preset/selected` 事件及 Agent 拥有的作用域，也处理启用前已存在的 Agent。提供方卸载和 Agent 销毁都会释放这些注册。preset 定义可以晚于提供方注册；名称用于筛选已绑定的 Agent，在匹配的 Agent 出现前不提供工具。不使用 preset 的独立组合可设置 `agentPreset: false` 来全局注册。Host 仍拥有权限、本机连接和取消操作，preset 定义保持不变。
[操作 schema](src/protocol.ts) 定义模型参数，并单独声明 Host 专用字段。原生实现在执行时检查目标身份、窗口几何、资源上限和操作系统访问条件。本库没有独立投影的注册表或持久状态，因此不发布 invariant 模块。
根 Host 入口导出 `installCua`、`CuaConfig` 和供 Typert 发现的 `CuaAccess` 服务，`apply()` 只标记可选 Client 贡献。`/client` 通过现有模块加载器加载权限界面。Host 和 Client 使用独立编译项目，headless 导入不会加载 React、CSS 或浏览器注册。`/api` 提供应用、窗口 API 及其类型；`/bridge`、`/native-client` 和 `/protocol` 保留运行时和平台消费者。`/types` 仅导出生成 Remote 声明所需的四个命名类型，Client 使用包内类型。`/typert` 提供 Host 反射，`/remote` 提供浏览器描述和编解码，权限判断仍由 Host 执行。
可选浏览器插件挂载原生 provider 的权限 Remote，复用插件页的启用引导和配置插槽。四排使用条件始终可见，未加载的字段保留占位。默认模式、执行方式、系统权限、应用列表和应用图标分别查询，结果返回后各自显示。应用名称和操作按钮不等待图标，图标位置保留空位。用户操作按顺序执行，不等待尚未完成的查询；卸载时等待操作和查询全部结束，再释放独立持有的 Remote 挂载。查询错误只显示在对应部分，启用引导只等待系统权限结果。卸载后的迟到结果会被丢弃。控制器直接接收生成的 `ctx.remote.cuaAccess`，在现有查询和操作流程中处理失败响应，并把查询结果投影到页面和设置弹窗；重新连接或 Host 返回的配置 namespace 更新时刷新状态。操作提示挂载到 shell overlay，关闭页面后仍可显示。provider 负责权限检查、持久授权和取消；浏览器不保存授权决定。
详情页通过现有 `configForms` 服务读取实际默认值（`selectedDefault ?? default`）。默认模式按钮仅写入 `selectedDefault`，沿用表单带版本检查的设置写入流程。执行方式一行调用 CUA 的 `ptcConfiguration()` 和 `enablePersistentPtc()`：Host 通过现有配置编辑器仅修改 `ptc-runtime.executionMode`，保留普通 Loader 重载规则。权限检查和执行方式状态独立返回。设置仍在加载、只读或不可用时不能提交；写入失败时在设置旁显示错误，供用户重试。
</details>
-----
@@ -84,6 +108,7 @@ kind: "package-library"
<a id="known-limitations-and-deferred-work"></a>
- 会话授权决策保存在内存中，没有容量上限或淘汰机制，直到提供方卸载才清空。长期运行的 Host 会保留每个曾请求访问的会话与应用组合。
- 当前原生执行使用 macOS 提供方。Windows 支持计划使用独立的提供方包和 C# helper，保留共享应用与窗口 API、授权、快照有效性和动作结果语义。实现须说明系统特有的限制，并区分交叉编译证据和 Windows 桌面验证。
- 持久变量不会在一组连续操作期间独占应用。其他 Agent 和用户可能改变其状态。 悬停和观察不恢复先前的鼠标位置；[连续交互](../../../.agents/notes/implemented/architecture/2026-10-13-cua-continuous-hover.zh.md)不分配跨调用所有权。
- 本包保留在 `experimental`，Windows 支持和平台验证完成后可迁至 `computer-use` 组。
packages/experimental/cua/package.json
+64 -13
View File
@@ -28,6 +28,18 @@
"types": "./lib/types/bridge.d.ts",
"default": "./lib/bridge.js"
},
"./typert": {
"types": "./lib/typert.host.d.ts",
"default": "./lib/typert.host.js"
},
"./remote": {
"types": "./lib/typert.remote-client.d.ts",
"default": "./lib/typert.remote-client.js"
},
"./types": {
"types": "./lib/types/remote-types.d.ts",
"default": "./lib/remote-types.js"
},
"./native-client": {
"types": "./lib/types/native-client.d.ts",
"default": "./lib/native-client.js"
@@ -35,34 +47,39 @@
"./protocol": {
"types": "./lib/types/protocol.d.ts",
"default": "./lib/protocol.js"
},
"./client": {
"types": "./lib/types/client/index.d.ts",
"default": "./lib/client.js"
}
},
"files": [
"lib/index.js",
"lib/client.js",
"lib/remote-types.js",
"lib/api.js",
"lib/bridge.js",
"lib/native-client.js",
"lib/protocol.js",
"lib/types/**/*.d.ts"
"lib/types/**/*.d.ts",
"lib/typert.host.js",
"lib/typert.host.d.ts",
"lib/typert.remote-client.js",
"lib/typert.remote-client.d.ts"
],
"license": "MIT",
"peerDependencies": {
"@deepseek-ai/cordis": "workspace:~",
"@deepseek-ai/dsh-computer-use": "workspace:*",
"@deepseek-ai/dsh-system-prompt": "workspace:*",
"@deepseek-ai/dsh-tools": "workspace:*",
"@deepseek-ai/dsh-agent": "workspace:*",
"@deepseek-ai/dsh-subprocess": "workspace:*",
"@deepseek-ai/dsh-user-questions": "workspace:*",
"@deepseek-ai/dsh-attachment": "workspace:*",
"@deepseek-ai/dsh-llm": "workspace:*"
"@deepseek-ai/dsh-scope": "workspace:*"
},
"dependencies": {
"@deepseek-ai/dsh-util-values": "workspace:*",
"@deepseek-ai/schemastery": "workspace:~",
"zod": "^4.4.3",
"@deepseek-ai/dsh-brand": "workspace:*",
"@deepseek-ai/dsh-atomic-write": "workspace:*"
"@deepseek-ai/dsh-atomic-write": "workspace:*",
"@deepseek-ai/dsh-typert-protocol": "workspace:*",
"@deepseek-ai/dsh-computer-use": "workspace:*"
},
"devDependencies": {
"@deepseek-ai/dsh-working-directory": "workspace:*",
@@ -74,7 +91,6 @@
"@deepseek-ai/dsh-agent-loop-testkit": "workspace:*",
"@deepseek-ai/dsh-attachment": "workspace:*",
"@deepseek-ai/dsh-attachment-local": "workspace:*",
"@deepseek-ai/dsh-computer-use": "workspace:*",
"@deepseek-ai/dsh-llm": "workspace:*",
"@deepseek-ai/dsh-session": "workspace:*",
"@deepseek-ai/dsh-session-projection": "workspace:*",
@@ -89,11 +105,46 @@
"@deepseek-ai/dsh-sandbox-policy": "workspace:*",
"sharp": "^0.35.3",
"@deepseek-ai/dsh-ptc-runtime": "workspace:*",
"@deepseek-ai/dsh-user-questions": "workspace:*",
"@deepseek-ai/dsh-settings": "workspace:*",
"@deepseek-ai/dsh-user-questions": "workspace:*",
"@deepseek-ai/dsh-app-boot": "workspace:*",
"@deepseek-ai/dsh-config-editor": "workspace:*",
"@deepseek-ai/dsh-hmr": "workspace:*",
"@deepseek-ai/cordis-plugin-timer": "workspace:~"
"@deepseek-ai/cordis-plugin-timer": "workspace:~",
"@deepseek-ai/dsh-scope": "workspace:*",
"@deepseek-ai/dsh-agent-preset-registry": "workspace:*",
"@deepseek-ai/dsh-api-remotes": "workspace:*",
"@deepseek-ai/dsh-api-gateway": "workspace:*",
"@deepseek-ai/dsh-client-locale": "workspace:*",
"@deepseek-ai/dsh-client-store": "workspace:*",
"@deepseek-ai/dsh-client-ui-primitives": "workspace:*",
"@deepseek-ai/dsh-client-ui-renderer": "workspace:*",
"@deepseek-ai/dsh-client-ui-slots": "workspace:*",
"@deepseek-ai/dsh-client-ui-plugin-manager": "workspace:*",
"@deepseek-ai/dsh-client-ui-settings": "workspace:*",
"@deepseek-ai/dsh-client-ui-layout": "workspace:*",
"@deepseek-ai/dsh-client-test-runtime": "workspace:*",
"react": "^18.2.0",
"react-dom": "^18.2.0",
"@testing-library/react": "^16.1.0",
"@deepseek-ai/dsh-loader-smoke": "workspace:*",
"@deepseek-ai/dsh-util-values": "workspace:*",
"@deepseek-ai/dsh-brand": "workspace:*"
},
"dsh": {
"client": {
"inject": [
"@deepseek-ai/dsh-api-remotes",
"@deepseek-ai/dsh-client-locale",
"@deepseek-ai/dsh-client-ui-plugin-manager",
"@deepseek-ai/dsh-client-ui-settings",
"@deepseek-ai/dsh-client-ui-layout"
],
"platform": "web"
}
},
"scripts": {
"bundle": "tsdown",
"watch": "tsdown --watch"
}
}
packages/experimental/cua/src/access-types.ts
+30
View File
@@ -0,0 +1,30 @@
/** Client-safe Computer Use permission observations and saved application decisions. */
import type { CuaApplicationGrant } from './application-types.ts'
export type { CuaApplicationGrant, CuaApplicationId } from './application-types.ts'
/** macOS privileges used by the native provider. */
export type CuaSystemPermission = 'accessibility' | 'screenRecording'
/** Observed process; neither the helper nor its parent is assumed to be the TCC owner. */
export interface CuaPermissionProcess {
pid: number
path?: string | undefined
name?: string | undefined
}
/** Fresh permission observations and startup metadata from the shared native helper. */
export interface CuaSystemPermissionState {
accessibility: boolean
screenRecording: boolean
helper: CuaPermissionProcess
parent: CuaPermissionProcess
/** Application macOS reports as responsible for the helper, captured at startup; absent when it cannot be identified. */
launchApplication?: { name: string; path: string; icon?: string | undefined } | undefined
}
/** Saved grants and refused applications, independent of native permission checks and icons. */
export interface CuaApplicationState {
/** Owning configuration entry for permission updates; null when this composition has no Loader entry. */
settingsNamespace: string | null
allowedApps: readonly CuaApplicationGrant[]
/** Refusals from any session, excluding saved grants; other sessions may still hold session access. */
blockedApps: readonly CuaApplicationGrant[]
persistentAvailable: boolean
}
packages/experimental/cua/src/access.ts
+167
View File
@@ -0,0 +1,167 @@
/** Browser controls for the provider's application decisions and native permission work. */
import type { Context } from '@deepseek-ai/cordis'
import type {} from '@deepseek-ai/dsh-config-editor'
import type {} from '@deepseek-ai/dsh-ptc-runtime'
import { Remote, TypertRemoteService } from '@deepseek-ai/dsh-typert-protocol'
import { ApplicationAccess, type AccessOptions } from './application-access.ts'
import type { JsonValue } from '@deepseek-ai/dsh-util-values'
import { z } from 'zod'
import { ApplicationIcon } from './protocol.ts'
import { NativeClient, type NativeClientConfig, type NativeChannelFactory } from './native-client.ts'
import type { CuaApplicationId, CuaApplicationState, CuaSystemPermission, CuaSystemPermissionState } from './access-types.ts'
export { ApplicationAuthorizationResult } from './application-access.ts'
export type * from './access-types.ts'
interface ControlOptions extends AccessOptions {
native: NativeClientConfig
createChannel: NativeChannelFactory
settingsNamespace: string | null
}
type Control = (method: string, params: Record<string, JsonValue>) => Promise<JsonValue>
const permissionProcess = z.object({ pid: z.number().int().positive(), path: z.string().optional(), name: z.string().optional() })
const systemPermissions = z.object({
accessibility: z.boolean(), screenRecording: z.boolean(), helper: permissionProcess, parent: permissionProcess,
launchApplication: z.object({ name: z.string().min(1), path: z.string().min(1), icon: ApplicationIcon.optional() }).optional(),
})
declare module '@deepseek-ai/cordis' {
interface Context {
/** Native permission controls and executor-owned application consent. */
cuaAccess: CuaAccess
}
}
/** Browser controls for provider-owned native permission and application decisions. */
export class CuaAccess extends TypertRemoteService {
/**
* @param ctx Provider context.
* @param policy Application decision owner.
* @param settingsNamespace Owning configuration entry.
* @param control Serialized native permission dispatch.
*/
constructor(ctx: Context, private readonly policy: ApplicationAccess,
private readonly settingsNamespace: string | null, private readonly control: Control) { super(ctx, 'cuaAccess') }
/** Read PTC execution state separately from native permission checks.
* @returns Whether the shared runtime retains memory and has an editable profile entry.
*/
@Remote
ptcConfiguration(): { persistent: boolean; writable: boolean } {
return {
persistent: this.ctx.get('ptcRuntime')?.programFormat === 'repl',
writable: this.ctx.get('configEditor')?.entries().some(entry => entry.options.id === 'ptc-runtime') ?? false,
}
}
/** Persist in-memory PTC execution through the existing profile editor and Loader lifecycle.
* @returns Completion after the shared runtime reload; already-persistent runtimes are left intact.
*/
@Remote
async enablePersistentPtc(): Promise<void> {
if (this.ptcConfiguration().persistent) return
const editor = this.ctx.get('configEditor')
const entry = editor?.entries().find(entry => entry.options.id === 'ptc-runtime')
if (editor === undefined || entry === undefined) throw new Error('PTC runtime has no editable profile entry')
await editor.edit(entry, current => ({ ...current, executionMode: 'persistent-in-memory' }))
}
/**
* Read application decisions without waiting for the native helper.
* @returns Saved grants and refused applications with their persistence owner.
*/
@Remote
applications(): CuaApplicationState {
return { settingsNamespace: this.settingsNamespace, ...this.policy.status() }
}
/**
* Inspect system privileges independently of application decisions and icons.
* @returns Fresh native preflight and launch metadata.
*/
@Remote
async systemStatus(): Promise<CuaSystemPermissionState> {
return systemPermissions.parse(await this.control('permissionStatus', {}))
}
/**
* Load presentation icons separately from application decisions.
* @returns Optional native PNG icons keyed by executable path for saved and blocked applications.
*/
@Remote
async applicationIcons(): Promise<Record<string, string>> {
const state = this.policy.status()
const paths = [...new Set([...state.allowedApps, ...state.blockedApps].map(app => app.executablePath))]
return paths.length === 0 ? {} : z.record(z.string(), ApplicationIcon).parse(await this.control('applicationIcons', { paths }))
}
/**
* Request a system privilege explicitly selected by the user.
* @param permission macOS privilege to request.
* @returns Preflight state after requesting; a false value never implies consent.
*/
@Remote
async requestPermission(permission: CuaSystemPermission): Promise<CuaSystemPermissionState> {
return systemPermissions.parse(await this.control('requestPermission', { permission }))
}
/**
* Open the macOS privacy pane selected by the user.
* @param permission Corresponding system privilege.
* @returns Completion after macOS accepted the open request.
*/
@Remote
async openSettings(permission: CuaSystemPermission): Promise<void> {
z.object({ opened: z.literal(true) }).parse(await this.control('openPermissionSettings', { permission }))
}
/**
* Remove saved and session access and join affected operations.
* @param id Application identity displayed on the permission page.
* @returns Completion after persistence and cancellation; dispatched input cannot be undone.
*/
@Remote
revoke(id: CuaApplicationId): Promise<void> { return this.policy.revoke(id) }
/**
* Clear this application's refusals across sessions so each session may ask for access again.
* @param id Refused or revoked application displayed on the permission page.
* @returns Completion after clearing refusals; no permission is granted and no action is replayed.
*/
/* oxlint-disable typescript/require-await -- Remote callers receive policy failures as rejected promises. */
@Remote
async retryApproval(id: CuaApplicationId): Promise<void> { this.policy.retryApproval(id) }
/* oxlint-enable typescript/require-await */
}
/**
* Install human-facing controls while keeping executor capabilities private to the provider.
* @param ctx Provider context that owns decisions and operation lifetimes.
* @param options Native transport and profile persistence.
* @returns Consent selection, guarded dispatch, and task cleanup capabilities.
*/
export function installCuaAccess(ctx: Context, options: ControlOptions): Pick<ApplicationAccess, 'select' | 'run' | 'forget' | 'dispose'> {
const policy = new ApplicationAccess(ctx, options)
const lifetime = new AbortController()
let client: NativeClient | undefined
let tail: Promise<unknown> = Promise.resolve()
const control: Control = (method, params) => {
lifetime.signal.throwIfAborted()
const perform = async () => {
lifetime.signal.throwIfAborted()
if (client?.disposed) await client.dispose()
lifetime.signal.throwIfAborted()
if (!client || client.disposed) client = new NativeClient(ctx, options.native, options.createChannel)
return client.request(method, params, lifetime.signal)
}
const operation = tail.then(perform, perform)
tail = operation
return operation
}
const dispose = ctx.effect(() => async () => {
lifetime.abort()
await Promise.allSettled([policy.dispose(), tail])
await client?.dispose()
}, 'cua-access.dispose')
new CuaAccess(ctx, policy, options.settingsNamespace, control)
return { select: policy.select.bind(policy), run: policy.run.bind(policy), forget: policy.forget.bind(policy), dispose }
}
packages/experimental/cua/src/application-access.ts
+96 -28
View File
@@ -6,6 +6,7 @@ import { z } from 'zod'
import { NativeError, type NativeClient } from './native-client.ts'
import type { CuaAppHandle } from './types.ts'
import type { CuaApplicationGrant, CuaApplicationId } from './application-types.ts'
import type { CuaApplicationState } from './access-types.ts'
/** Native identity admission occurs before any application observation or input. */
export const ApplicationAuthorizationResult = z.object({
@@ -19,7 +20,7 @@ export interface AccessOptions {
}
interface Selection {
identity: CuaApplicationGrant
scope: 'task' | 'persistent'
scope: 'session' | 'persistent'
lifetime: AbortController
}
interface ClientSelections {
@@ -37,9 +38,9 @@ interface ActiveOperation {
/** Owns application consent and cancels operations when saved access is removed. */
export class ApplicationAccess {
private readonly clients = new Map<NativeClient, ClientSelections>()
private readonly rejected = new Map<Agent, Map<CuaApplicationId, CuaApplicationGrant>>()
private readonly decisions = new Map<Agent['id'], Map<CuaApplicationId, { identity: CuaApplicationGrant; allowed: boolean }>>()
private readonly active = new Set<ActiveOperation>()
private readonly selections = new WeakMap<Agent, Map<CuaApplicationId, Promise<void>>>()
private readonly selections = new Map<Agent['id'], Map<CuaApplicationId, Promise<void>>>()
private readonly lifetime = new AbortController()
private mutationTail: Promise<unknown> = Promise.resolve()
private disposal: Promise<void> | undefined
@@ -59,13 +60,66 @@ export class ApplicationAccess {
this.lifetime.abort()
await Promise.allSettled([this.mutationTail, ...[...this.active].map(operation => operation.done.promise)])
this.clients.clear()
this.rejected.clear()
this.decisions.clear()
})()
}
/**
* Project application decisions without requesting native permissions.
* @returns Current saved grants and refused applications for the permission page.
*/
status(): Omit<CuaApplicationState, 'settingsNamespace'> {
const allowedApps = [...this.options.grants()]
const blockedApps = new Map<CuaApplicationId, CuaApplicationGrant>()
for (const decisions of this.decisions.values()) for (const [id, decision] of decisions) {
if (decision.allowed || allowedApps.some(grant => grant.id === id)) continue
blockedApps.set(id, decision.identity)
}
return { allowedApps,
blockedApps: [...blockedApps.values()], persistentAvailable: this.options.save !== undefined }
}
/**
* Remove saved and session consent and cancel affected native operations.
* @param id Application identity shown in the permission page.
* @returns Completion after persistence and affected operations settle; already dispatched input cannot be undone.
*/
async revoke(id: CuaApplicationId): Promise<void> {
this.lifetime.signal.throwIfAborted()
let clients: readonly NativeClient[] = []
await this.mutate(async () => {
const grants = this.options.grants()
if (grants.some(grant => grant.id === id)) {
const save = this.options.save
if (!save) throw new Error('Application permissions cannot be saved in this composition')
await save(grants.filter(grant => grant.id !== id))
}
clients = this.invalidate(id)
})
const results = await Promise.allSettled([
...clients.map(client => client.dispose()),
...[...this.active].filter(operation => operation.identity.id === id).map(operation => operation.done.promise),
])
const errors: unknown[] = []
for (const result of results) if (result.status === 'rejected') errors.push(result.reason)
if (errors.length) throw new AggregateError(errors, 'Application access was revoked, but helper cleanup failed')
}
/**
* Clear explicit refusal; the next getApp still requires a new user decision.
* @param id Application identity selected by the human.
*/
retryApproval(id: CuaApplicationId): void {
this.lifetime.signal.throwIfAborted()
for (const decisions of this.decisions.values()) if (decisions.get(id)?.allowed === false) decisions.delete(id)
for (const state of this.clients.values()) for (const [handle, selected] of state.handles) {
if (selected.identity.id === id && selected.lifetime.signal.aborted) state.handles.delete(handle)
}
}
/**
* Obtain consent for a freshly selected native handle. Free text never grants access.
* @param agent Exact live task requesting access.
* @param agent Live agent whose session owns the decision.
* @param client Helper that issued the handle.
* @param handle Helper handle for a running instance or an approved pending launch.
* @param identity Native executable/signing identity.
@@ -79,19 +133,21 @@ export class ApplicationAccess {
const existing = this.clients.get(client)?.handles.get(handle)
if (!client.disposed && existing && this.permitted(existing)) return
const saved = identity.persistable && this.options.grants().some(grant => grant.id === identity.id)
let scope: Selection['scope'] = 'persistent'
if (!saved && this.rejected.get(agent)?.has(identity.id)) throw this.denied(identity)
const decision = this.decisions.get(agent.id)?.get(identity.id)
let scope: Selection['scope'] = saved ? 'persistent' : 'session'
if (!saved && decision?.allowed === false) throw this.denied(identity)
if (client.disposed) throw new NativeError('NATIVE_CLOSED', 'Helper is closed; acquire fresh application handles')
if (!saved) {
if (!saved && !decision?.allowed) {
const questions = this.ctx.get('userQuestions')
if (!questions) throw new NativeError('APP_APPROVAL_UNAVAILABLE', 'No human answerer is available for application access')
const options = [{ label: 'Allow for this task' }, { label: 'Deny' }]
const options = [{ label: 'Allow for this session' }, { label: 'Deny' }]
if (identity.persistable && this.options.save) options.push({ label: 'Always allow' })
let result: AskUserQuestionAnswer
try {
result = await questions.ask({ agent, signal, questions: [{ id: 'cua-app-access',
question: `Allow Computer Use to launch if needed, read, capture, and control ${identity.name}?`,
detail: `Application: ${identity.executablePath}\nTask access applies to the selected application instance, including its launch if needed. macOS permissions and approval of sensitive actions remain separate. Saved access can be revoked by removing its allowedApps entry from this provider configuration. Task access ends with the task.`, options }] })
detail: 'Session access lasts until DSH restarts or Computer Use is disabled.',
options }] })
} catch (error) {
if (error instanceof UserQuestionError && error.code === 'ASK_CANCELLED') {
this.invalidate(identity.id, agent)
@@ -105,8 +161,9 @@ export class ApplicationAccess {
signal.throwIfAborted()
const answer = result.answers.find(answer => answer.id === 'cua-app-access')
const chosen = answer && !answer.custom?.trim() && answer.selected.length === 1 ? answer.selected[0] : undefined
if (chosen === 'Allow for this task') scope = 'task'
if (chosen === 'Allow for this session') scope = 'session'
else if (chosen === 'Always allow' && identity.persistable && this.options.save) {
scope = 'persistent'
const { persistable: _persistable, ...grant } = identity
await this.persist((grants) => {
signal.throwIfAborted()
@@ -129,7 +186,8 @@ export class ApplicationAccess {
}
state.handles.set(handle, { identity: grant, scope, lifetime: new AbortController() })
this.clients.set(client, state)
this.rejected.get(agent)?.delete(identity.id)
if (scope === 'session') this.decide(agent.id, grant, true)
else this.decisions.get(agent.id)?.delete(identity.id)
}))
}
@@ -155,11 +213,14 @@ export class ApplicationAccess {
}
private async selectInOrder(agent: Agent, id: CuaApplicationId, select: () => Promise<void>): Promise<void> {
const selections = this.selections.get(agent) ?? new Map<CuaApplicationId, Promise<void>>()
this.selections.set(agent, selections)
const selections = this.selections.get(agent.id) ?? new Map<CuaApplicationId, Promise<void>>()
this.selections.set(agent.id, selections)
const operation = (selections.get(id) ?? Promise.resolve()).then(select, select)
selections.set(id, operation)
try { await operation } finally { if (selections.get(id) === operation) selections.delete(id) }
try { await operation } finally {
if (selections.get(id) === operation) selections.delete(id)
if (selections.size === 0) this.selections.delete(agent.id)
}
}
private async withOperation<T>(agent: Agent, client: NativeClient, identity: CuaApplicationGrant, signal: AbortSignal,
@@ -172,6 +233,9 @@ export class ApplicationAccess {
const result = await dispatch(lifetime)
lifetime.throwIfAborted()
return result
} catch (error) {
if (operation.controller.signal.aborted) throw operation.controller.signal.reason
throw error
} finally {
this.active.delete(operation)
operation.done.resolve()
@@ -179,40 +243,44 @@ export class ApplicationAccess {
}
/**
* Forget the disposed task's transient decisions and selected handles.
* @param agent Disposed task whose transient decisions are released.
* Release the disposed agent's handles while retaining its session decisions.
* @param agent Disposed agent whose selected handles are released.
*/
forget(agent: Agent): void {
for (const [client, state] of this.clients) if (state.agent === agent) this.clients.delete(client)
this.rejected.delete(agent)
}
private permitted(selection: Selection): boolean {
return !selection.lifetime.signal.aborted
&& (selection.scope === 'task' || this.options.grants().some(grant => grant.id === selection.identity.id))
&& (selection.scope === 'session' || this.options.grants().some(grant => grant.id === selection.identity.id))
}
private denied(identity: CuaApplicationGrant): NativeError {
return new NativeError('APP_ACCESS_DENIED', `The user denied access to ${identity.name}. Do not retry or use another tool to bypass this decision.`)
}
private reject(agent: Agent, identity: CuaApplicationGrant): void {
const rejected = this.rejected.get(agent) ?? new Map<CuaApplicationId, CuaApplicationGrant>()
rejected.set(identity.id, { id: identity.id, name: identity.name, executablePath: identity.executablePath })
this.rejected.set(agent, rejected)
private decide(sessionId: Agent['id'], identity: CuaApplicationGrant, allowed: boolean): void {
const decisions = this.decisions.get(sessionId) ?? new Map<CuaApplicationId, { identity: CuaApplicationGrant; allowed: boolean }>()
decisions.set(identity.id, { identity: { id: identity.id, name: identity.name, executablePath: identity.executablePath }, allowed })
this.decisions.set(sessionId, decisions)
}
private invalidate(id: CuaApplicationId, agent?: Agent): void {
private invalidate(id: CuaApplicationId, agent?: Agent): NativeClient[] {
const clients = new Set<NativeClient>()
for (const [sessionId, decisions] of this.decisions) {
const decision = decisions.get(id)
if (decision && (agent === undefined || sessionId === agent.id)) this.decide(sessionId, decision.identity, false)
}
for (const [client, state] of this.clients) for (const selection of state.handles.values()) if (selection.identity.id === id
&& (agent === undefined || state.agent === agent)) {
&& (agent === undefined || state.agent.id === agent.id)) {
selection.lifetime.abort(new NativeError('APP_ACCESS_REVOKED', 'Application access was revoked; stop using this application'))
this.reject(state.agent, selection.identity)
this.decide(state.agent.id, selection.identity, false)
clients.add(client)
}
for (const operation of this.active) if (operation.identity.id === id && (agent === undefined || operation.agent === agent)) {
this.reject(operation.agent, operation.identity)
for (const operation of this.active) if (operation.identity.id === id && (agent === undefined || operation.agent.id === agent.id)) {
this.decide(operation.agent.id, operation.identity, false)
operation.controller.abort(this.denied(operation.identity))
clients.add(operation.client)
}
for (const client of clients) client.invalidate()
return [...clients]
}
private reconcile(): void {
for (const state of this.clients.values()) for (const selection of state.handles.values()) {
packages/experimental/cua/src/client/PermissionPanel.module.css
+48
View File
@@ -0,0 +1,48 @@
.panels { display: grid; gap: 24px; margin: 18px 0; font-size: 13px; }
.panel { display: grid; gap: 12px; min-width: 0; }
.card { min-width: 0; border: 0.5px solid var(--dsw-alias-border-l1); border-radius: var(--dsw-radius-md); }
.card > * + * { border-top: 0.5px solid var(--dsw-alias-border-l1); }
.panel p { margin: 0; color: var(--dsw-alias-label-secondary); line-height: 1.6; }
.panel h3 { margin: 0; font-size: 14px; font-weight: 500; }
.panel h4 { margin: 0; font-size: 13px; font-weight: 500; color: var(--dsw-alias-label-secondary); }
.permission, .apps li { display: flex; flex-wrap: wrap; align-items: center; gap: 12px; }
.permission, .group, .emptyState, .notice { padding: 16px; }
.permission { min-height: 80px; box-sizing: border-box; }
.defaultSetting > p { margin: 0 16px 16px; }
.permission > div:first-child, .apps li > div { flex: 1; min-width: 140px; }
.notice { display: flex; flex-wrap: wrap; align-items: center; gap: 8px; }
.group { display: grid; gap: 4px; }
.apps { display: grid; padding: 0; margin: 0; list-style: none; }
.apps li { padding: 12px 0; }
.apps li:last-child { padding-bottom: 0; }
.panel strong { font-weight: 500; }
.panel .error { color: var(--dsw-alias-state-error-primary); overflow-wrap: anywhere; }
.application { display: flex; align-items: center; gap: 10px; }
.application strong { min-width: 0; overflow-wrap: anywhere; }
.application img { flex-shrink: 0; object-fit: contain; }
.applicationBadge {
display: inline-flex;
align-items: center;
gap: 6px;
max-width: 100%;
padding: 2px 6px;
border: 0.5px solid var(--dsw-alias-border-l1);
border-radius: var(--dsw-radius-sm);
color: var(--dsw-alias-label-primary);
vertical-align: middle;
}
.applicationBadge img { flex-shrink: 0; object-fit: contain; }
.applicationBadge > span { min-width: 0; overflow-wrap: anywhere; }
.permissionAction, .buttonPlaceholder { width: 168px; flex-shrink: 0; }
.permissionAction[data-configured]:disabled { opacity: 1; color: var(--dsw-alias-label-primary); cursor: default; }
.buttonPlaceholder, .appPlaceholder, .appsPlaceholder {
display: inline-block;
max-width: 100%;
border-radius: var(--dsw-radius-xs);
background: var(--dsw-alias-bg-skeleton);
}
.buttonPlaceholder { height: 28px; }
.appPlaceholder { width: 160px; height: 20px; }
.appsPlaceholder { width: 240px; height: 40px; }
.iconPlaceholder { display: inline-block; width: 32px; height: 32px; flex-shrink: 0; }
Loaded 100 of 157 files, more files were not shown because too many files have changed in this diff. Show more