Merge branch 'master' into worktree/session-item-trailing-slot

This commit is contained in:
Yichen Jiang authored and GitHub committed 2026-10-19 23:07:23 +08:00
commit 9dbf907f93
171 files changed
+5558 -534

No files matched your search

@@ -9,11 +9,11 @@
en: 2ba6c43b20a3d347
zh: 5647179ddaa38ad9
/agent-note-ship-experimental-capabilities-as-optional-bundles/decision:
en: 6c8583bdefc5e36c
zh: 0928b4f2876bbfe4
en: d6328ba2b218f2ac
zh: 3731561313b1ad5e
/agent-note-ship-experimental-capabilities-as-optional-bundles/alternatives-considered:
en: d6f40bf02df37ab7
zh: e5b85ce9d41e8754
/agent-note-ship-experimental-capabilities-as-optional-bundles/consequences:
en: 38d27583533cd028
zh: 9bf11f5489c1aa43
en: 3064da835e892958
zh: bfcf967b6acc6f85
@@ -12,12 +12,14 @@ The Web plugin page offered two optional bundles, Agent Teams and voice input. A
`OPTIONAL_BUNDLES` owns the installation's default-off selections, including maintained optional capabilities and experimental capabilities. Every bundle declares `icon` and exports `./locale/*.json` with `meta.title` and `meta.description`; the Official group displays that metadata. A bundle's presentation does not determine its [experimental status](../../../../packages/experimental/README.md#status). The isolation check validates admission and still rejects experimental dependencies of nonexperimental bundles.
An optional bundle's dependencies are downloaded with every `dsh` installation. [Reasoning translation](2026-10-05-anonymous-reasoning-translation.md) uses three experimental packages without adding third-party runtime dependencies. Admission therefore considers installation cost and whether a switch supplies a usable composition without an additional configuration form. Browser and computer providers remain explicit compositions because they add substantial runtimes or require an external executable. Stagehand requires model credentials; Python PTC replaces a core provider and conflicts with TypeScript workflow consumers. Demonstrative hooks, webhooks, and mods stay outside the list. [Session Inspector](../feature/2026-09-24-session-inspector.md) owns the Inspector's distinct activation behavior.
An optional bundle's dependencies are downloaded with every `dsh` installation. [Reasoning translation](2026-10-05-anonymous-reasoning-translation.md) uses three experimental packages without adding third-party runtime dependencies. Admission therefore considers installation cost and whether a switch supplies a usable composition without an additional configuration form. External browser and Cua Driver providers remain explicit compositions because they add substantial runtimes or require an external executable. Stagehand requires model credentials; Python PTC replaces a core provider and conflicts with TypeScript workflow consumers. Demonstrative hooks, webhooks, and mods stay outside the list. [Session Inspector](../feature/2026-09-24-session-inspector.md) owns the Inspector's distinct activation behavior.
Bundles insert ordinary Host rows; optional model tools register in the global tool layer, so every preset, including minimal, receives them. Plugin management, profile composition, and reload need no bundle-specific behavior. Search, Ralph, and terminals supply their own isolated services in a Host group. Search opens one in-memory index on demand; the Host search policy stays unchanged. Badge skills and title providers act on the Host. Switching a bundle registers or removes its tools for live Agents, and later user patch layers retain precedence. A replacement title provider may register once the previous provider's disposal starts; the previous provider's late results never commit.
Claude Code and Codex install on demand through the separate `ON_DEMAND_BUNDLES` catalog; the [on-demand native-bundle decision](2026-10-05-official-on-demand-bundles.md) owns the cost, version, and offline-discovery trade-offs.
Computer Use is a shipped optional bundle in the experimental group. Its bundle uses the explicit naming exception `@deepseek-ai/dsh-cua-bundle`, alongside `@deepseek-ai/dsh-cua`. These names preserve profile references when the capability moves package groups; the experimental classification still controls isolation and promotion. Its repository-owned native component is prepared when the provider loads, without an external Cua Driver runtime.
## Alternatives considered
**Ship every provider as an optional bundle.** Composes and displays correctly, but grows every install by provider runtimes for a capability most installations never switch on; voice input's `sherpa-onnx-node` is the one accepted precedent.
@@ -31,3 +33,5 @@ Claude Code and Codex install on demand through the separate `ON_DEMAND_BUNDLES`
## Consequences
Official discovery does not imply experimental status or a different plugin lifecycle. Switching a bundle off keeps its installed files; failed activation retains the existing saved-selection behavior. Bundle rows are ordinary Host rows with their usual switches. Individual and combined composition tests cover the Host rows and every preset's tool catalog; live switching and recorded Sessions cover activation and model-visible behavior.
Computer Use keeps its permission service and native provider on the Host and registers tools in Agent scopes whose bound preset is `ptc`. Existing Agent events and `createScope()` maintain current, new and resumed Agents. Preset declarations remain unchanged. Only the explicit detail-page action changes the new-session default through the existing settings form; the activation switch does not configure PTC execution.
@@ -12,12 +12,14 @@ Web 插件页只提供两个可选 bundle:Agent Teams 与语音输入。Auto r
`OPTIONAL_BUNDLES` 管理安装随附且默认关闭的选项,包括维护中的可选能力与实验性能力。每个 bundle 声明 `icon`,并导出带 `meta.title` 与 `meta.description` 的 `./locale/*.json`;官方分组显示这些元数据。Bundle 的展示方式不决定其[实验性状态](../../../../packages/experimental/README.zh.md#status)。隔离检查校验准入,并继续拒绝非实验 bundle 的实验性依赖。
可选 bundle 的依赖随每次 `dsh` 安装下载。[思考过程翻译](2026-10-05-anonymous-reasoning-translation.zh.md)使用三个实验包,不增加第三方运行时依赖。因此准入需要考虑安装成本,以及开关能否在不增加配置表单的情况下提供可用组合。浏览器与电脑提供方仍使用显式组合,因为它们会增加较大的运行时或要求外部可执行文件。Stagehand 需要模型凭据;Python PTC 替换核心提供方,并与 TypeScript workflow 消费方冲突。演示性钩子、webhook 与模组不进入名单。[Session Inspector](../feature/2026-09-24-session-inspector.zh.md) 管理 Inspector 特有的激活行为。
可选 bundle 的依赖随每次 `dsh` 安装下载。[思考过程翻译](2026-10-05-anonymous-reasoning-translation.zh.md)使用三个实验包,不增加第三方运行时依赖。因此准入需要考虑安装成本,以及开关能否在不增加配置表单的情况下提供可用组合。外部浏览器与 Cua Driver 提供方仍使用显式组合,因为它们会增加较大的运行时或要求外部可执行文件。Stagehand 需要模型凭据;Python PTC 替换核心提供方,并与 TypeScript workflow 消费方冲突。演示性钩子、webhook 与模组不进入名单。[Session Inspector](../feature/2026-09-24-session-inspector.zh.md) 管理 Inspector 特有的激活行为。
Bundle 插入普通 Host 行;可选模型工具注册到全局工具层,因此包括 minimal 在内的所有预设都会获得它们。插件管理、profile 组合与重载无需针对 bundle 的特殊行为。搜索、Ralph 与终端在 Host 组内提供自己的隔离服务。搜索按需打开一个内存索引;Host 搜索策略保持不变。徽章技能与标题提供方作用于 Host。切换 bundle 会为运行中的 Agent 注册或移除其工具,后续用户 patch 层继续优先。前一个标题提供方开始清理后,替代提供方即可注册;前一个提供方的迟到结果不会提交。
Claude Code 与 Codex 通过独立的 `ON_DEMAND_BUNDLES` 目录按需安装,其成本、版本与离线发现取舍由[按需原生 bundle 决策](2026-10-05-official-on-demand-bundles.zh.md)管理。
Computer Use 是 experimental 包组内随安装提供的可选组合包。组合包与 `@deepseek-ai/dsh-cua` 一起采用显式命名例外 `@deepseek-ai/dsh-cua-bundle`,在能力移动分组时保留配置引用。依赖隔离和晋升仍由实验性分类约束。提供方加载时准备仓库自有的本机组件,无需外部 Cua Driver 运行时。
## 考虑过的替代方案
**把所有提供方都做成可选 bundle。** 组合与展示都正确,但会为多数安装从不启用的能力把提供方运行时塞进每次安装;语音输入的 `sherpa-onnx-node` 是唯一被接受的先例。
@@ -31,3 +33,5 @@ Claude Code 与 Codex 通过独立的 `ON_DEMAND_BUNDLES` 目录按需安装,
## 后果
官方发现不表示实验性状态,也不引入不同的插件生命周期。关闭 bundle 保留已安装文件;激活失败保留现有的已保存选择行为。Bundle 行是普通 Host 行,使用常规开关。独立与组合测试覆盖 Host 行及每个预设的工具目录;实时切换与已记录 Session 覆盖激活和模型可见行为。
Computer Use 将权限服务和原生提供方保留在 Host,在已绑定 `ptc` preset 的 Agent 作用域中注册工具。现有 Agent 事件和 `createScope()` 维护当前、新建和恢复的 Agent,preset 声明保持不变。只有详情页的显式操作通过现有设置表单修改新会话默认模式,启用开关不配置 PTC 执行方式。
@@ -9,11 +9,11 @@
en: b32a7ce6483b9b48
zh: ec0938ad032b99d8
/agent-note-application-consent-for-native-computer-use/decision:
en: 92247ff599c1103f
zh: 26845420a9711c83
en: 36c560c24c376f12
zh: 5e4ff72f66574e38
/agent-note-application-consent-for-native-computer-use/alternatives-considered:
en: 4693dcd217c4fedb
zh: 1dc9bf9067314006
en: 80cd5fd93a42d937
zh: 2a0e241b1b8c2e1d
/agent-note-application-consent-for-native-computer-use/consequences:
en: 0146a69ab2086edc
zh: 4257bc1ab4432c3f
en: 4bda3366af65250a
zh: 7c83561b4db07284
@@ -10,29 +10,38 @@ macOS privileges authorize an execution host, not a user's choice of target appl
## Decision
The CUA executor owns application consent for every object-API and native-tool dispatch. It uses the existing User Questions service for human decisions and the provider's existing profile configuration for saved grants. General tool approval and operating-system privileges remain separate requirements. Arbitrary persistent PTC code and separately authorized shell tools retain their own execution authority.
The CUA executor owns application consent for every object-API and native-tool dispatch. It uses the existing User Questions service for human decisions and the provider's existing profile configuration for saved grants. The existing question UI presents the offered labels; the executor rejects custom text or answers that do not select exactly one offered label. Permission-page icons and names are presentation metadata, excluded from saved identities and model results. General tool approval and operating-system privileges remain separate requirements. Arbitrary persistent PTC code and separately authorized shell tools retain their own execution authority.
A saved identity combines the canonical executable path with a verified designated signing requirement, represented by a SHA-256 digest. The path prevents authorization from following a copied application, and the signing requirement distinguishes replacements while allowing matching signed updates. Unverifiable and ad-hoc applications receive task access only, tied to a process instance and Agent connection lifetime. Labels do not establish identity.
System-permission buttons call the native helper on the DSH host and remain available to remote browsers. The [usage documentation](../../../../packages/experimental/cua/README.md#use-this-package) explains where to complete authorization; the page has no persistent remote-access notice.
A saved identity combines the canonical executable path with a verified designated signing requirement, represented by a SHA-256 digest. The path prevents authorization from following a copied application, and the signing requirement distinguishes replacements while allowing matching signed updates. Unverifiable and ad-hoc applications receive session access only, tied to a process instance. Labels do not establish identity.
Cold application selection prepares an executable identity and handle without launching. The same consent decision guards the private launch continuation, and the helper checks the executable again before launch and binds the resulting process afterward. Signed selections retain the normal designated-requirement identity; an unverifiable executable uses a content identity for this pending selection and cannot create a saved grant. Preparing metadata before asking avoids starting an application the user refuses.
Selections of the same application within one task are serialized, so a pending question does not produce competing decisions. Saved-grant additions read the current configuration inside one mutation queue. Refusals prevent another prompt within the task; an explicitly restored saved grant takes precedence at dispatch.
Session decisions are kept in memory by Session ID and application identity. Agent disposal and helper replacement release native handles without discarding these decisions; provider disposal or DSH restart clears them. This scope covers follow-up turns without relying on the model to decide when a user intent has ended.
Removing a saved grant invalidates Agent connections retaining the application's handles and cancels active operations; Agent/provider disposal awaits native connection cleanup. Reauthorization uses a fresh connection, so old app and window snapshots cannot become valid again. The [native object decision](2026-09-17-experimental-cua-object-api.md) continues to own process identity, coordinate semantics, and separation from the evaluator.
Selections of the same application within one session are serialized, so a pending question does not produce competing decisions. Saved-grant additions and revocations share one mutation queue and read the current grant list inside it. Provider shutdown cancels writes still waiting for HMR admission; admitted writes settle before disposal completes. Refusals prevent another prompt until the user permits asking again; a restored saved grant takes precedence in both dispatch and the permission-page projection.
Revocation invalidates the Agent connections that retain the application's handles, then awaits native cleanup. Reauthorization uses a fresh connection, so old app and window snapshots cannot become valid again. The [native object decision](2026-09-17-experimental-cua-object-api.md) continues to own process identity, coordinate semantics, and separation from the evaluator.
## Alternatives considered
**Trust model instructions or hide controls.** Retained objects and direct tool calls can still reach native operations. The executor must enforce the decision at dispatch.
**Persist application names or unsigned paths.** A replacement can reuse both. Durable grants require a verifiable signing identity; task-only access supports unsigned development applications without that promise.
**Persist application names or unsigned paths.** A replacement can reuse both. Durable grants require a verifiable signing identity; session access supports unsigned development applications without that promise.
**Add an approval database or Session control stream.** Profile settings already own user configuration, and User Questions already owns interactive answers.
**Add an approval database or Session control stream.** Profile settings already own user configuration, and User Questions already owns interactive answers. The permission UI observes the provider and its owning settings namespace without another authorization store or event channel.
**Disable permission buttons based on the browser address.** A user may operate the DSH host through screen sharing and complete authorization there. Port forwarding and proxies make the address an unreliable indication of whether the browser and DSH run on the same computer.
**Use the nearest GUI ancestor as the permission target.** A source-built Desktop app can sit between Terminal and the helper while macOS still attributes Accessibility and Screen Recording to Terminal. Process responsibility supplies the system’s attribution; ancestry alone does not. The optional SPI adds compatibility risk, so lookup failure omits the application instead of guessing.
**Keep the native context after revocation.** Its application lookup can return an earlier handle and retain window snapshots. Closing the Agent connection discards its native context through existing transport cleanup; preserving unrelated handles would require a native per-application retirement protocol.
## Consequences
Revoking one application also invalidates other application handles on the same Agent connection. Those applications require fresh selection; saved grants remain reusable, while task grants expire. Other tasks with independent connections remain unaffected unless they also hold access to the revoked application.
Refusing or revoking one application also invalidates other application handles on the same Agent connection. Those applications require fresh selection; session and saved grants remain reusable. Other sessions with independent connections remain unaffected unless they also hold access to the revoked application.
Explicit enablement may start the shared helper for a read-only permission probe. The helper captures the application identified by macOS process responsibility before readiness releases the launcher, retaining its name and icon after reparenting. The optional native query is dynamically resolved; unavailable attribution uses generic system-prompt guidance. The macOS prompt remains authoritative. Enabling does not request macOS authorization or inspect application content.
Application decisions and failure results are exercised through policy regressions and the recorded CUA scenario. Desktop binary distribution, signing, and packaged-host permissions remain outside the source-launch workflow.
@@ -10,29 +10,38 @@ macOS 权限授权执行宿主,不能表达用户允许操作哪些目标应
## 决策
CUA 执行器在每次对象 API 和原生工具调用时检查应用授权。人工决定复用 User Questions 服务,长期许可保存在 provider 的现有 profile 配置中。通用工具审批和操作系统权限仍是独立要求。任意 persistent PTC 代码和另行授权的 shell 工具保留各自的执行权限。
CUA 执行器在每次对象 API 和原生工具调用时检查应用授权。人工决定复用 User Questions 服务,长期许可保存在 provider 的现有 profile 配置中。通用工具审批和操作系统权限仍是独立要求。任意 persistent PTC 代码和另行授权的 shell 工具保留各自的执行权限。现有问答界面展示授权选项;执行器拒绝自由文本或未准确选中一个所给选项的回答。权限页图标和名称仅用于展示,不进入保存的身份信息或模型结果。
长期身份由规范化可执行文件路径和经过验证的签名指定要求组成,以 SHA-256 摘要表示。路径阻止许可跟随应用副本,签名要求区分替换程序,并允许符合相同签名要求的更新。签名不可验证或使用 ad-hoc 签名的应用只能获得任务授权,许可绑定进程实例和 Agent 连接生命周期。显示名称不用于确定身份。
系统权限按钮调用运行 DSH 的电脑上的原生 helper,远程访问时仍可使用。授权地点写在[使用说明](../../../../packages/experimental/cua/README.zh.md#use-this-package)中,页面不常驻远程访问提示。
长期身份由规范化可执行文件路径和经过验证的签名指定要求组成,以 SHA-256 摘要表示。路径阻止许可跟随应用副本,签名要求区分替换程序,并允许符合相同签名要求的更新。签名不可验证或使用 ad-hoc 签名的应用只能获得会话授权,许可绑定进程实例。显示名称不用于确定身份。
选择尚未运行的应用时,先准备可执行文件身份和句柄,不立即启动。同一授权决定保护内部启动续接,helper 在启动前再次核对可执行文件,并在启动后绑定实际进程。已签名应用沿用签名指定要求身份;签名不可验证的程序使用本次待启动选择的内容身份,不能生成长期许可。先准备元数据再询问,可以避免启动用户拒绝的应用。
同一任务对同一应用的选择串行执行,避免未完成的问题产生相互冲突的决定。长期许可的新增在同一写入队列内读取当前配置。拒绝会阻止任务内的再次询问;用户显式恢复的长期许可在执行检查时优先于旧拒绝记录。
会话决定按 Session ID 和应用身份保存在内存中。Agent 销毁或 helper 重建会释放原生句柄,但保留授权决定;提供方卸载或 DSH 重启后清除。会话范围可以覆盖用户的后续指令,无须由模型判断一次用户意图何时结束。
删除长期许可会使保留目标应用句柄的 Agent 连接失效,并取消活动操作;Agent 或提供方销毁时等待对应的原生连接清理。重新授权使用新连接,因此旧应用对象和窗口快照不能重新生效。[原生对象决策](2026-09-17-experimental-cua-object-api.zh.md)继续负责进程身份、坐标语义和与执行器分离的设计。
同一会话对同一应用的选择串行执行,避免未完成的问题产生相互冲突的决定。长期许可的新增和撤销共用写入队列,并在队列内读取当前许可列表。提供方关闭时取消尚在等待 HMR 调度的写入,已开始的写入会在销毁完成前结束。拒绝会阻止再次询问,直到用户允许重询;重新添加的长期许可在执行检查和权限页中均优先于旧拒绝记录。
撤销使保留目标应用句柄的 Agent 连接失效,并等待原生清理完成。重新授权使用新连接,因此旧应用对象和窗口快照不能重新生效。[原生对象决策](2026-09-17-experimental-cua-object-api.zh.md)继续负责进程身份、坐标语义和与执行器分离的设计。
## 考虑过的替代方案
**依赖模型指引或隐藏控件。** 保留的对象和直接工具调用仍能到达原生操作。执行器必须在派发时检查用户决定。
**保存应用名称或未签名路径。** 替换程序可以复用二者。长期许可需要可验证的签名身份;任务授权支持未签名的开发应用,但不承诺跨进程复用。
**保存应用名称或未签名路径。** 替换程序可以复用二者。长期许可需要可验证的签名身份;会话授权支持未签名的开发应用,但不承诺跨进程复用。
**新增审批数据库或 Session 控制流。** Profile settings 已负责用户配置,User Questions 已负责交互回答。
**新增审批数据库或 Session 控制流。** Profile settings 已负责用户配置,User Questions 已负责交互回答。权限界面查询 provider,并监听其配置 namespace,无须另一份授权存储或事件通道。
**根据浏览器访问地址禁用权限按钮。** 用户可能通过屏幕共享操作运行 DSH 的电脑,并在该电脑上完成授权。端口转发和代理使访问地址无法可靠判断浏览器与 DSH 是否运行在同一台电脑上。
**把最近的 GUI 祖先当作授权对象。** 源码构建的 Desktop 可以位于 Terminal 和 helper 之间,但 macOS 仍将辅助功能和屏幕录制归属于 Terminal。责任进程记录提供系统认定的归属,仅凭父进程链无法确定。可选内部接口有兼容性风险,因此查询失败时省略应用信息,不再猜测。
**撤销后保留原生状态。** 原生应用查找可能返回旧句柄并保留窗口快照。关闭 Agent 连接可以通过现有传输清理释放其原生状态;保留其他应用的句柄则需要新增按应用销毁状态的原生协议。
## 影响
撤销一个应用也会使同一 Agent 连接中其他应用的句柄失效。这些应用需要重新选择;长期许可仍可复用,任务许可随之失效。其他任务使用独立连接,除非也持有被撤销应用的许可,否则不受影响。
拒绝或撤销一个应用也会使同一 Agent 连接中其他应用的句柄失效。这些应用需要重新选择;会话许可和长期许可仍可复用。其他会话使用独立连接,除非也持有被撤销应用的许可,否则不受影响。
显式启用可以启动共享 helper 进行只读权限检查。helper 在发布就绪状态、让启动器退出之前,记录 macOS 责任进程对应的应用,因此父进程改变后仍能提供应用名称和图标。可选的本机查询通过动态加载解析,无法取得归属时显示通用系统弹窗指引。授权归属仍以 macOS 提示为准。启用不会申请 macOS 授权或检查应用内容。
授权决定和失败结果由策略回归测试及 CUA 录制会话验证。桌面安装包的二进制分发、签名和打包宿主权限不属于源码启动流程。
+1
View File
@@ -105,6 +105,7 @@
"@deepseek-ai/dsh-experimental-terminal-bundle": "workspace:*",
"@deepseek-ai/dsh-experimental-session-search": "workspace:*",
"@deepseek-ai/dsh-experimental-tool-worktree": "workspace:*",
"@deepseek-ai/dsh-cua-bundle": "workspace:*",
"@deepseek-ai/dsh-experimental-plugin-market-bundle": "workspace:*"
},
"devDependencies": {
+2 -2
View File
@@ -42,8 +42,8 @@
en: 98cf848762482d48
zh: 74f9100a39ec8c2f
/deepseek-harness-desktop/develop:
en: fe6d23e6a3d3bc66
zh: d60c8d3a55747916
en: faf7676ba18bbf2f
zh: f1d7b75ad348c548
/deepseek-harness-desktop/develop/startup-onboarding:
en: a51346b8d880ddba
zh: 7913214c1c66063c
+2
View File
@@ -202,6 +202,8 @@ After an explicit build, `start:desktop` reconstructs the disposable project and
pnpm run start:desktop
```
Desktop package preparation filters optional native packages by the installer target’s OS and architecture. Windows package sets omit the macOS Computer Use tarball, including when their inputs were packed on macOS.
The Web counterparts are `pnpm run dev:web` and `pnpm run start:web`, documented in the [development guide](../../docs/development.md). Workspace development runs the current CLI and private Desktop Host packages under Electron RunAsNode. Plugin management and recovery use `$DSH_HOME/profiles/desktop`, separate from the disposable workspace runtime. The Host uses runtime module resolution in both development and packaged builds without creating official-package fallback links; developer-installed packages, including links, retain native priority. Use an unpacked application to exercise Electron RunAsNode, bundled pnpm, bundled dsh resources, plugin installation and repair paths.
The [native/renderer keyboard tests](tests/keyboard.spec.ts) compile as part of the repository Client typecheck. Their Desktop imports are limited to Cordis-free input, persistence, IPC, browser-guest, and overlay modules.
+2
View File
@@ -202,6 +202,8 @@ pnpm run dev:desktop
pnpm run start:desktop
```
桌面包准备过程按安装包目标的操作系统和架构筛选可选原生包。Windows 的包集合不包含 macOS Computer Use tarball,即使输入包是在 macOS 上打包的。
Web 侧的对应命令是 `pnpm run dev:web` 与 `pnpm run start:web`,见[开发指南](../../docs/development.zh.md)。Workspace 开发使用 Electron RunAsNode 运行当前 CLI 与私有 Desktop Host 包,插件管理和恢复使用 `$DSH_HOME/profiles/desktop`,与一次性工作区运行时分离。Host 在开发与打包构建中都使用 runtime 模块解析,不创建官方包的 fallback 链接;开发者安装的包(包括链接)保留原生优先级。需要验证 Electron RunAsNode、内置 pnpm、内置 dsh 资源、插件安装和修复时,应运行未封装安装器的应用目录。
[原生输入与渲染进程键盘测试](tests/keyboard.spec.ts)直接纳入仓库 Client 类型检查。它只导入不依赖 Cordis 的 Desktop 输入、持久化、IPC、浏览器 guest 和蒙层模块。
+1 -1
View File
@@ -134,7 +134,7 @@ async function main(): Promise<void> {
await packagingStep(process.env.DSH_DESKTOP_PACKAGING_RUN_DIR, 'runtime:install', () => runPnpm(['install', '--prod', '--frozen-lockfile', '--trust-lockfile']))
const packageSet = readDesktopCorePackageSet(BUILD_ROOT, release.version)
const targetName = resolveDesktopBuildTarget()
const target = { platform: process.platform, arch: desktopTargetPlatform(targetName).arch }
const target = desktopTargetPlatform(targetName)
const modules = join(BUILD_ROOT, 'node_modules')
const officeManifest = JSON.parse(readFileSync(join(modules, '@deepseek-ai/libreoffice-kit/package.json'), 'utf8'))
const officeEngine = selectOfficeEngine(officeManifest, target)
+28 -6
View File
@@ -25,7 +25,7 @@ import {
} from '../src/core-package-set.ts'
import { capture } from '../../../scripts/release/process.ts'
import { tarballFiles } from '../../../scripts/release/tarball.ts'
import { resolveDesktopTargetBuildPaths } from './desktop-build-paths.mjs'
import { desktopTargetPlatform, resolveDesktopBuildTarget, resolveDesktopTargetBuildPaths } from './desktop-build-paths.mjs'
const DSH_PACKAGE = '@deepseek-ai/dsh'
const ROOT_PACKAGES = [DSH_PACKAGE, DESKTOP_HOST_PACKAGE] as const
@@ -41,6 +41,17 @@ export interface PackedDesktopPackage {
readonly manifest: Readonly<Record<string, unknown>>
}
function supportsTarget(manifest: Readonly<Record<string, unknown>>, field: 'os' | 'cpu', target: string): boolean {
const values = manifest[field]
if (values === undefined) return true
if (!Array.isArray(values) || !values.every((value): value is string => typeof value === 'string')) {
throw new Error(`desktop package set: ${String(manifest.name)} has invalid ${field}`)
}
if (values.includes(`!${target}`)) return false
return values.length === 0 || (values.length === 1 && values[0] === 'any') || values.includes(target)
|| values.every(value => value.startsWith('!'))
}
function dependencyNames(manifest: Readonly<Record<string, unknown>>, section: string): string[] {
const value = manifest[section]
if (value === undefined) return []
@@ -54,10 +65,12 @@ function dependencyNames(manifest: Readonly<Record<string, unknown>>, section: s
* Select workspace dependencies rooted at dsh and its private Host; npm resolves external packages.
* Reads the repository workspace manifest and package manifests to distinguish required local packages from npm-resolved externals.
* @param available - Packed packages indexed by package name.
* @param target - Platform and architecture of the installer payload.
* @returns Selected packages sorted by name.
*/
export function selectDesktopPackageClosure(
available: ReadonlyMap<string, PackedDesktopPackage>,
target: ReturnType<typeof desktopTargetPlatform>,
): PackedDesktopPackage[] {
const workspace = yaml.load(readFileSync(join(REPOSITORY_ROOT, 'pnpm-workspace.yaml'), 'utf8')) as { packages: string[] }
const workspaceNames = new Set(globSync(workspace.packages.map(pattern => `${pattern}/package.json`), { cwd: REPOSITORY_ROOT })
@@ -77,7 +90,9 @@ export function selectDesktopPackageClosure(
}
}
for (const dependency of dependencyNames(packed.manifest, OPTIONAL_DEPENDENCY_SECTION)) {
if (available.has(dependency)) visit(dependency)
const optional = available.get(dependency)
if (optional && supportsTarget(optional.manifest, 'os', target.platform)
&& supportsTarget(optional.manifest, 'cpu', target.arch)) visit(dependency)
}
}
for (const name of ROOT_PACKAGES) {
@@ -127,9 +142,16 @@ export function assertDesktopHostPackageFiles(files: readonly string[]): void {
}
}
/** Prepare a package set from release tarball directories. */
export function prepareDesktopPackageSet(inputs: readonly string[], output: string): void {
const selected = selectDesktopPackageClosure(packedPackages(inputs))
/**
* Prepare a package set from release tarball directories.
* @param inputs - Release tarball directories.
* @param output - Destination package-set directory.
* @param target - Platform and architecture of the installer payload.
*/
export function prepareDesktopPackageSet(
inputs: readonly string[], output: string, target: ReturnType<typeof desktopTargetPlatform>,
): void {
const selected = selectDesktopPackageClosure(packedPackages(inputs), target)
const host = selected.find(packed => packed.manifest.name === DESKTOP_HOST_PACKAGE)
if (host === undefined) throw new Error(`desktop package set: selected closure omits ${DESKTOP_HOST_PACKAGE}`)
assertDesktopHostPackageFiles(tarballFiles(host.tarball))
@@ -171,7 +193,7 @@ function main(): void {
})
const inputs = (values.from ?? defaultInputs).map(path => resolve(REPOSITORY_ROOT, path))
const output = values.out === undefined ? buildPaths.packageSet : resolve(REPOSITORY_ROOT, values.out)
prepareDesktopPackageSet(inputs, output)
prepareDesktopPackageSet(inputs, output, desktopTargetPlatform(resolveDesktopBuildTarget()))
console.log(`desktop package set: prepared ${output}`)
}
@@ -0,0 +1,121 @@
/** Cold pnpm requests exercise shipped CUA dependency/platform metadata with inert package contents. */
import { createHash } from 'node:crypto'
import { createServer } from 'node:http'
import { mkdir, mkdtemp, readFile, rm, writeFile } from 'node:fs/promises'
import { tmpdir } from 'node:os'
import { join, resolve } from 'node:path'
import { execa } from 'execa'
import { expect, it, onTestFinished } from 'vitest'
import { pnpmCommand } from '../../../scripts/release/process.ts'
const repository = resolve(import.meta.dirname, '../../..')
const providerName = '@deepseek-ai/dsh-experimental-cua-macos'
const bundleName = '@deepseek-ai/dsh-cua-bundle'
const readManifest = async (path: string): Promise<Record<string, unknown>> =>
JSON.parse(await readFile(join(repository, path, 'package.json'), 'utf8')) as Record<string, unknown>
it('fetches the macOS optional tarball only for Darwin, with an empty cache for every target', async () => {
const root = await mkdtemp(join(tmpdir(), 'cua-cold-install-'))
onTestFinished(() => rm(root, { recursive: true, force: true }))
const userconfig = join(root, 'npmrc')
await writeFile(userconfig, '')
const environment = Object.fromEntries(Object.entries(process.env).filter(([key]) =>
!/KEY|SECRET|TOKEN|PASSWORD/iu.test(key) && !/^(npm|pnpm|corepack)_/iu.test(key)
&& key !== 'NODE_OPTIONS' && key !== 'NODE_PATH'))
const [pnpm, ...prefix] = pnpmCommand()
const command = async (args: string[], cwd: string) => {
const result = await execa(pnpm, [...prefix, `--config.userconfig=${userconfig}`, ...args], {
cwd, env: { ...environment, CI: 'true', XDG_CACHE_HOME: join(cwd, 'xdg-cache'),
XDG_DATA_HOME: join(cwd, 'xdg-data'), XDG_STATE_HOME: join(cwd, 'xdg-state'),
XDG_CONFIG_HOME: join(cwd, 'xdg-config') }, extendEnv: false, reject: false, timeout: 60_000,
})
expect(result.timedOut, result.stderr).toBe(false)
expect(result.signal, result.stderr).toBeUndefined()
expect(result.exitCode, `${result.stdout}\n${result.stderr}`).toBe(0)
return result.stdout
}
const cli = await readManifest('apps/cli')
const bundle = await readManifest('packages/experimental/cua-bundle')
const native = await readManifest('packages/experimental/cua-macos')
expect(cli.dependencies).toHaveProperty(bundleName)
expect(bundle.dependencies).not.toHaveProperty(providerName)
expect(bundle.optionalDependencies).toEqual({ [providerName]: 'workspace:*' })
expect(native.os).toEqual(['darwin'])
const version = '1.0.0'
const bundleDependencies = Object.fromEntries(Object.keys(bundle.dependencies as Record<string, string>).map(name => [name, version]))
const manifests = [
{ name: cli.name, version, dependencies: { [bundleName]: version } },
{ name: bundleName, version, dependencies: bundleDependencies, optionalDependencies: { [providerName]: version } },
...Object.keys(bundleDependencies).map(name => ({ name, version })),
{ name: providerName, version, os: native.os, files: native.files },
]
const packages = new Map<string, { manifest: Record<string, unknown>; bytes: Buffer }>()
for (const manifest of manifests) {
const name = String(manifest.name)
const directory = join(root, name.replace('@deepseek-ai/', ''))
await mkdir(directory)
await writeFile(join(directory, 'package.json'), JSON.stringify(manifest))
if (name === providerName) {
await mkdir(join(directory, 'native/bin/darwin-arm64'), { recursive: true })
for (const file of ['.gitignore', '.npmignore']) {
await writeFile(join(directory, 'native', file), await readFile(join(repository, 'packages/experimental/cua-macos/native', file)))
}
await writeFile(join(directory, 'native/bin/darwin-arm64/dsh-cua-native'), 'inert native payload')
}
await command(['pack', '--out', 'package.tgz'], directory)
const bytes = await readFile(join(directory, 'package.tgz'))
packages.set(name, { manifest, bytes })
if (name === providerName) {
const result = await execa('tar', ['-tzf', join(directory, 'package.tgz')])
expect(result.stdout).toContain('package/native/bin/darwin-arm64/dsh-cua-native')
}
}
const requests: string[] = []
let unrestricted = false
const server = createServer((request, response) => {
const path = decodeURIComponent(request.url ?? '')
requests.push(path)
const tarball = path.endsWith('.tgz')
const name = path.slice(1).replace(/\.tgz$/u, '')
const entry = packages.get(name)
if (!entry) { response.writeHead(404); response.end(); return }
if (tarball) { response.end(entry.bytes); return }
const manifest = { ...entry.manifest }
// pnpm also infers macOS from this package name; an explicit any overrides that inference.
if (unrestricted && name === providerName) manifest.os = ['any']
response.setHeader('Content-Type', 'application/json')
response.end(JSON.stringify({ name, 'dist-tags': { latest: version }, versions: {
[version]: { ...manifest, dist: {
tarball: `http://${request.headers.host}/${name}.tgz`,
integrity: `sha512-${createHash('sha512').update(entry.bytes).digest('base64')}`,
} },
} }))
})
onTestFinished(async () => {
server.closeAllConnections()
await new Promise<void>((resolve, reject) => { server.close((error) => { if (error) reject(error); else resolve() }) })
})
await new Promise<void>((resolve, reject) => { server.once('error', reject); server.listen(0, '127.0.0.1', resolve) })
const address = server.address()
if (!address || typeof address === 'string') throw new Error('fixture registry has no TCP address')
for (const target of ['darwin', 'win32', 'linux', 'linux-unrestricted']) {
unrestricted = target === 'linux-unrestricted'
const project = join(root, target)
await mkdir(project)
await writeFile(join(project, 'package.json'), JSON.stringify({ private: true, dependencies: { [String(cli.name)]: version } }))
await writeFile(join(project, 'pnpm-workspace.yaml'), [
'autoInstallPeers: false', 'supportedArchitectures:', ` os: [${unrestricted ? 'linux' : target}]`,
' cpu: [arm64]', 'minimumReleaseAge: 0', '',
].join('\n'))
requests.length = 0
const output = await command([
'install', '--ignore-scripts', '--config.enable-global-virtual-store=false',
`--config.registry=http://127.0.0.1:${address.port}/`, `--config.store-dir=${join(project, 'store')}`,
`--config.cache-dir=${join(project, 'cache')}`, `--config.state-dir=${join(project, 'state')}`,
], project)
expect(requests).toContain(`/${bundleName}.tgz`)
expect(requests).toContain('/@deepseek-ai/dsh-cua.tgz')
expect(requests.includes(`/${providerName}.tgz`), `${target}: ${JSON.stringify(requests)}\n${output}`)
.toBe(target === 'darwin' || unrestricted)
}
}, 120_000)
+75 -5
View File
@@ -1,6 +1,12 @@
import { afterEach, describe, expect, it, vi } from 'vitest'
import { afterEach, describe, expect, it, onTestFinished, vi } from 'vitest'
import { execFileSync } from 'node:child_process'
import { mkdirSync, mkdtempSync, readFileSync, readdirSync, rmSync, writeFileSync } from 'node:fs'
import { tmpdir } from 'node:os'
import { join } from 'node:path'
import { DESKTOP_HOST_RUNTIME_FILES, DESKTOP_PACKAGES_DIR } from '../src/core-package-set.ts'
import {
assertDesktopHostPackageFiles,
prepareDesktopPackageSet,
selectDesktopPackageClosure,
type PackedDesktopPackage,
} from '../scripts/prepare-package-set.ts'
@@ -37,7 +43,7 @@ describe('desktop package-set selection', () => {
['@deepseek-ai/platform-package', packed('@deepseek-ai/platform-package')],
['@deepseek-ai/unused', packed('@deepseek-ai/unused')],
])
expect(selectDesktopPackageClosure(available).map(entry => entry.manifest.name)).toEqual([
expect(selectDesktopPackageClosure(available, { platform: 'win32', arch: 'x64' }).map(entry => entry.manifest.name)).toEqual([
'@deepseek-ai/cordis',
'@deepseek-ai/dsh',
'@deepseek-ai/dsh-base',
@@ -57,10 +63,10 @@ describe('desktop package-set selection', () => {
dependencies: { '@deepseek-ai/dsh': '^1.0.0' },
})],
])
expect(() => selectDesktopPackageClosure(available)).toThrow(/unpacked package/u)
expect(() => selectDesktopPackageClosure(available, { platform: 'win32', arch: 'x64' })).toThrow(/unpacked package/u)
expect(() => selectDesktopPackageClosure(new Map([
['@deepseek-ai/dsh', packed('@deepseek-ai/dsh')],
]))).toThrow(/omit @deepseek-ai\/dsh-desktop-host/u)
]), { platform: 'win32', arch: 'x64' })).toThrow(/omit @deepseek-ai\/dsh-desktop-host/u)
})
it('leaves independently published Office packages to npm resolution', () => {
@@ -73,7 +79,7 @@ describe('desktop package-set selection', () => {
})],
['@deepseek-ai/dsh-desktop-host', packed('@deepseek-ai/dsh-desktop-host')],
])
expect(selectDesktopPackageClosure(available).map(entry => entry.manifest.name)).toEqual([
expect(selectDesktopPackageClosure(available, { platform: 'win32', arch: 'x64' }).map(entry => entry.manifest.name)).toEqual([
'@deepseek-ai/dsh', '@deepseek-ai/dsh-desktop-host',
])
})
@@ -92,3 +98,67 @@ describe('desktop package-set selection', () => {
expect(() => { assertDesktopHostPackageFiles(files.slice(0, 1)) }).toThrow(/lib\/cli\.js/u)
})
})
const macProvider = '@deepseek-ai/dsh-experimental-cua-macos'
const cuaBundle = '@deepseek-ai/dsh-cua-bundle'
const targetManifests = () => new Map<string, PackedDesktopPackage>([
['@deepseek-ai/dsh', packed('@deepseek-ai/dsh', { dependencies: { [cuaBundle]: '1.0.0' } })],
['@deepseek-ai/dsh-desktop-host', packed('@deepseek-ai/dsh-desktop-host')],
[cuaBundle, packed(cuaBundle, {
optionalDependencies: { [macProvider]: '1.0.0' }, dependencies: { '@deepseek-ai/dsh-cua': '1.0.0' },
})],
['@deepseek-ai/dsh-cua', packed('@deepseek-ai/dsh-cua')],
[macProvider, packed(macProvider, { os: ['darwin'] })],
])
it.each([
['darwin', 'arm64', true], ['darwin', 'x64', true], ['win32', 'x64', false],
] as const)('selects the CUA native tarball for target %s/%s independently of the host', (platform, arch, included) => {
const selected = selectDesktopPackageClosure(targetManifests(), { platform, arch }).map(entry => entry.manifest.name)
expect(selected.includes(macProvider)).toBe(included)
expect(selected).toContain('@deepseek-ai/dsh-cua')
expect(selected).toContain(cuaBundle)
})
it.each([
[{ cpu: ['arm64'] }, false], [{ os: ['!win32'] }, false], [{ os: ['!linux'] }, true],
[{ os: [] }, true], [{ os: ['any'] }, true], [{ os: ['any', '!win32'] }, false],
[{ cpu: ['x64'] }, true],
])('honors optional package platform constraints %j', (constraints, included) => {
const packages = targetManifests()
packages.set(macProvider, packed(macProvider, constraints))
expect(selectDesktopPackageClosure(packages, { platform: 'win32', arch: 'x64' })
.some(entry => entry.manifest.name === macProvider)).toBe(included)
})
it.each([{ os: 'darwin' }, { cpu: [42] }])('rejects malformed platform constraints %j', (constraints) => {
const packages = targetManifests()
packages.set(macProvider, packed(macProvider, constraints))
expect(() => selectDesktopPackageClosure(packages, { platform: 'win32', arch: 'x64' })).toThrow(/has invalid/u)
})
it('copies no macOS tarball into the Windows package set', () => {
const root = mkdtempSync(join(tmpdir(), 'desktop-platform-packages-'))
onTestFinished(() => { rmSync(root, { recursive: true, force: true }) })
const inputs = join(root, 'inputs')
mkdirSync(inputs)
for (const [name, entry] of targetManifests()) {
const stem = name.replace('@deepseek-ai/', '')
const folder = join(root, stem, 'package')
mkdirSync(folder, { recursive: true })
writeFileSync(join(folder, 'package.json'), JSON.stringify(entry.manifest))
if (name === '@deepseek-ai/dsh-desktop-host') {
for (const file of DESKTOP_HOST_RUNTIME_FILES) {
mkdirSync(join(folder, file, '..'), { recursive: true })
writeFileSync(join(folder, file), '')
}
}
execFileSync('tar', ['-czf', join(inputs, `${stem}.tgz`), '-C', join(folder, '..'), 'package'])
}
const output = join(root, 'output')
prepareDesktopPackageSet([inputs], output, { platform: 'win32', arch: 'x64' })
expect(readdirSync(join(output, DESKTOP_PACKAGES_DIR))).not.toContain('dsh-experimental-cua-macos.tgz')
prepareDesktopPackageSet([inputs], output, { platform: 'darwin', arch: 'arm64' })
expect(readFileSync(join(output, DESKTOP_PACKAGES_DIR, 'dsh-experimental-cua-macos.tgz')))
.toEqual(readFileSync(join(inputs, 'dsh-experimental-cua-macos.tgz')))
})
+128
View File
@@ -0,0 +1,128 @@
/** Optional CUA uses existing plugin activation and offers a separate persistent PTC action. */
import { mkdtemp, readFile, rm, writeFile } from 'node:fs/promises'
import { tmpdir } from 'node:os'
import { join } from 'node:path'
import { fileURLToPath } from 'node:url'
import { chromium, type Browser } from 'playwright'
import { readProfileManifest } from '@deepseek-ai/dsh-app-boot'
import { SessionId } from '@deepseek-ai/dsh-session'
import type {} from '@deepseek-ai/dsh-ptc-runtime'
import type {} from '@deepseek-ai/dsh-agent-preset-registry'
import { expect, it } from 'vitest'
import { launchWebScaffold, type WebScaffold } from './scaffold.ts'
import { connectFreshWorkspaceZh, ZH_BROWSER_LOCALE } from './support.ts'
it.each([undefined, 'minimal'] as const)('adds CUA to PTC sessions while preserving default %s until the explicit persistent-runtime action', async (selectedDefault) => {
const directory = await mkdtemp(join(tmpdir(), 'cua-optin-web-'))
let ownedScaffold: WebScaffold | undefined
let browser: Browser | undefined
try {
const patch = join(directory, 'native-fixture.yml')
// The executable cannot answer native permission queries, so setup must offer retry in the detail page.
await writeFile(patch, JSON.stringify([{ id: 'cua', disabled: false, config: { helperPath: process.execPath } }]))
// Install the test provider explicitly so the helper override also exercises session wiring on Linux.
const scaffold = ownedScaffold = await launchWebScaffold({
profile: { packages: [{ dir: fileURLToPath(new URL('../../../packages/experimental/cua-macos', import.meta.url)), enabled: false }] },
harnessHome: join(directory, 'home'), extraOverlayPath: [
fileURLToPath(new URL('./pin-browse-picker.overlay.yml', import.meta.url)), patch,
] })
const initialDefault = selectedDefault ?? 'standard'
if (selectedDefault !== undefined) await scaffold.ctx.settings.update('agent-preset-registry', { selectedDefault })
browser = await chromium.launch()
const page = await browser.newPage({ viewport: { width: 1680, height: 1000 }, locale: ZH_BROWSER_LOCALE })
await page.goto(scaffold.authenticatedUrl)
await connectFreshWorkspaceZh(page, scaffold.workspaceCwd)
const originalAgent = scaffold.ctx.agents.list()[0]!
const ptcAgent = (await scaffold.ctx.agents.create({ sessionId: SessionId('cua-existing-ptc'),
setup: async (ctx) => { await scaffold.ctx.agentPresets.mount(ctx, 'ptc') },
})).agent
const runtimeFiber = () => [...scaffold.ctx.loader.entries()].find(entry => entry.options.id === 'ptc-runtime')?.fiber
const originalRuntime = scaffold.ctx.ptcRuntime
const originalRuntimeFiber = runtimeFiber()
expect(originalRuntimeFiber).toBeDefined()
expect(scaffold.ctx.ptcRuntime.programFormat).toBe('function-body')
const settingsPath = join(scaffold.harnessHome, 'profiles/scaffold/cordis.patch.yml')
const before = await readFile(settingsPath, 'utf8')
const selected = () => readProfileManifest('scaffold', join(scaffold.harnessHome, 'profiles/scaffold')).dsh?.profile?.bundles
await page.getByRole('navigation', { name: '全局面板' }).getByRole('button', { name: '插件', exact: true }).click()
const card = page.locator('[data-plugin-package="@deepseek-ai/dsh-cua-bundle"]')
const toggle = card.getByRole('switch', { name: '启用 Computer Use', exact: true })
expect(await toggle.getAttribute('aria-checked')).toBe('false')
expect(await card.textContent()).toContain('请切换到 PTC 模式')
// Bundle transitions await native cleanup under the browser action deadline.
await toggle.click()
await toggle.and(page.locator('[aria-checked="true"]')).waitFor({ state: 'visible' })
const setup = page.getByRole('dialog', { name: '使用 Computer Use 前需要设置', exact: true })
await setup.waitFor()
expect(await setup.getByText('Computer Use 插件已开启,但暂时无法检测系统权限。请前往插件详情页查看原因并重试。', { exact: true }).count()).toBe(1)
await setup.getByRole('button', { name: '稍后', exact: true }).click()
expect(selected()).toContain('@deepseek-ai/dsh-cua-bundle')
expect(await readFile(settingsPath, 'utf8')).toBe(before)
expect(runtimeFiber() === originalRuntimeFiber).toBe(true)
expect(scaffold.ctx.ptcRuntime.programFormat).toBe('function-body')
expect(scaffold.ctx.agentPresets.defaultId).toBe(initialDefault)
expect(scaffold.ctx.agents.get(originalAgent.id) === originalAgent).toBe(true)
expect(scaffold.ctx.tools.schemas(originalAgent).some(tool => tool.name === 'cua')).toBe(false)
expect(scaffold.ctx.tools.schemas(ptcAgent).some(tool => tool.name === 'cua')).toBe(true)
expect((await scaffold.ctx.systemPrompt.assemble({ scope: ptcAgent })).tools.map(tool => tool.name)).toEqual(['run_code'])
const fresh = await scaffold.ctx.agents.create({ sessionId: SessionId('cua-new-ptc'),
setup: async (ctx) => { await scaffold.ctx.agentPresets.mount(ctx, 'ptc') },
})
expect(scaffold.ctx.tools.schemas(fresh.agent).some(tool => tool.name === 'cua')).toBe(true)
await toggle.click()
await toggle.and(page.locator('[aria-checked="false"]')).waitFor({ state: 'visible' })
expect(scaffold.ctx.get('computerUse')).toBeUndefined()
for (const agent of [ptcAgent, fresh.agent]) expect(scaffold.ctx.tools.schemas(agent).some(tool => tool.name === 'cua')).toBe(false)
expect(runtimeFiber() === originalRuntimeFiber).toBe(true)
expect(scaffold.ctx.ptcRuntime.programFormat).toBe('function-body')
expect(await readFile(settingsPath, 'utf8')).toBe(before)
await card.getByRole('button', { name: '查看 Computer Use', exact: true }).click()
const detailToggle = page.locator('[data-plugin-panel]').getByRole('switch', { name: '启用 Computer Use', exact: true })
await detailToggle.click()
await detailToggle.and(page.locator('[aria-checked="true"]')).waitFor({ state: 'visible' })
const panel = page.locator('[data-cua-settings]')
await panel.waitFor()
const defaultSetting = panel.getByRole('group', { name: '新会话默认模式', exact: true })
await expect.poll(() => defaultSetting.getByRole('button', { name: '设为 PTC', exact: true }).isEnabled()).toBe(true)
await defaultSetting.getByRole('button', { name: '设为 PTC', exact: true }).click()
await expect.poll(() => scaffold.ctx.agentPresets.defaultId).toBe('ptc')
expect(scaffold.ctx.agentPresets.composedPreset(originalAgent.ctx)).toBe(initialDefault)
expect(runtimeFiber() === originalRuntimeFiber).toBe(true)
expect(scaffold.ctx.ptcRuntime.programFormat).toBe('function-body')
const confirmed = await scaffold.ctx.agents.create({ sessionId: SessionId('cua-confirmed-default'),
setup: async (ctx) => { await scaffold.ctx.agentPresets.mount(ctx) },
})
expect(scaffold.ctx.tools.schemas(confirmed.agent).some(tool => tool.name === 'cua')).toBe(true)
const saved = await readFile(settingsPath, 'utf8')
expect(saved).toContain('selectedDefault: ptc')
expect(saved).not.toContain('executionMode:')
const persistent = panel.getByRole('group', { name: 'PTC 执行方式', exact: true })
await persistent.getByRole('button', { name: '设为持久化', exact: true }).click()
await persistent.getByRole('button', { name: '已设为持久化', exact: true }).waitFor()
expect(scaffold.ctx.ptcRuntime.programFormat).toBe('repl')
expect(scaffold.ctx.ptcRuntime === originalRuntime).toBe(false)
const persistentRuntimeFiber = runtimeFiber()
const runtime = scaffold.ctx.ptcRuntime, owner = {}
const program = 'globalThis.counter = (globalThis.counter ?? 0) + 1'
expect((await runtime.run(runtime.resolve({ program, bindings: [], stateOwner: owner }))).value).toBe(1)
const configured = await readFile(settingsPath, 'utf8')
expect(configured).toContain('executionMode: persistent-in-memory')
expect(scaffold.ctx.agentPresets.defaultId).toBe('ptc')
expect(scaffold.ctx.agentPresets.composedPreset(originalAgent.ctx)).toBe(initialDefault)
await detailToggle.click()
await detailToggle.and(page.locator('[aria-checked="false"]')).waitFor({ state: 'visible' })
expect(await readFile(settingsPath, 'utf8')).toBe(configured)
expect(runtimeFiber() === persistentRuntimeFiber).toBe(true)
expect((await runtime.run(runtime.resolve({ program, bindings: [], stateOwner: owner }))).value).toBe(2)
await detailToggle.click()
await detailToggle.and(page.locator('[aria-checked="true"]')).waitFor({ state: 'visible' })
expect(await readFile(settingsPath, 'utf8')).toBe(configured)
expect(runtimeFiber() === persistentRuntimeFiber).toBe(true)
expect((await runtime.run(runtime.resolve({ program, bindings: [], stateOwner: owner }))).value).toBe(3)
await runtime.release(owner)
} finally {
try { await browser?.close() } finally {
try { await ownedScaffold?.close() } finally { await rm(directory, { recursive: true, force: true }) }
}
}
})
+150
View File
@@ -0,0 +1,150 @@
/** Built permission UI talks to the real provider; the isolated helper substitutes only macOS privilege observations. */
import { mkdir, mkdtemp, readFile, rm, writeFile } from 'node:fs/promises'
import { tmpdir } from 'node:os'
import { join } from 'node:path'
import { fileURLToPath } from 'node:url'
import { ToolCallId } from '@deepseek-ai/dsh-llm'
import type {} from '@deepseek-ai/dsh-user-approval'
import type {} from '@deepseek-ai/dsh-agent-preset-registry'
import { chromium, type Browser } from 'playwright'
import { expect, it, onTestFailed, onTestFinished } from 'vitest'
import { captureStableAria, compareOrRefreshGolden, launchWebScaffold, webSnapshotMode, watchConsole, type WebScaffold } from './scaffold.ts'
import { connectFreshWorkspace, newEnglishPage, saveFailureShot } from './support.ts'
import { installNativeFixture, startFixtureHost } from '../../../packages/experimental/cua-macos/tests/native-fixture.ts'
const bundle = fileURLToPath(new URL('../../../packages/experimental/cua-bundle', import.meta.url))
const expected = fileURLToPath(new URL('./expected/cua-permissions/missing.expected.md', import.meta.url))
// System-permission onboarding belongs to the macOS-only bundle; rendering and policy units remain portable.
it.skipIf(process.platform !== 'darwin')('keeps system requests pending until fresh observation and persists application revocation', async () => {
const root = await mkdtemp(join(tmpdir(), 'dsh-cua-permissions-web-'))
const resources: { scaffold?: WebScaffold; browser?: Browser; closeFixture?: () => Promise<void> } = {}
onTestFinished(async () => {
try { await resources.browser?.close() } finally {
try { await resources.scaffold?.close() } finally {
try { await resources.closeFixture?.() } finally { await rm(root, { recursive: true, force: true }) }
}
}
})
const helper = await installNativeFixture(root)
resources.closeFixture = await startFixtureHost(helper)
const state = join(root, 'permissions.json'), audit = join(root, 'helper-events.jsonl')
const launchApplication = { name: 'Visual Studio Code.app', path: '/Applications/Visual Studio Code.app',
icon: 'data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAIAAACQd1PeAAAACXBIWXMAAAPoAAAD6AG1e1JrAAAADElEQVQImWNgZGIGAAAOAAeCcsnOAAAAAElFTkSuQmCC' }
await writeFile(state, JSON.stringify({ accessibility: false, screenRecording: false, launchApplication }))
await writeFile(join(root, 'observe-helper'), '')
const fixtureBundle = join(root, 'fixture-bundle')
await mkdir(fixtureBundle)
await writeFile(join(fixtureBundle, 'package.json'), JSON.stringify({ name: 'cua-permission-fixture', version: '1.0.0',
dsh: { bundle: { patch: 'cordis.patch.yml' } } }))
await writeFile(join(fixtureBundle, 'cordis.patch.yml'), `- id: cua\n config: ${JSON.stringify({ helperPath: helper })}\n`)
const scaffold = await launchWebScaffold({
profile: { packages: [{ dir: bundle, enabled: true }, { dir: fixtureBundle, enabled: true }] },
})
resources.scaffold = scaffold
await scaffold.ctx.settings.mutate('cua', [{ op: 'set', path: ['allowedApps'],
value: [{ id: 'a'.repeat(64), name: 'Fixture', executablePath: '/Applications/Fixture.app' }] }])
const browser = await chromium.launch()
resources.browser = browser
const page = await newEnglishPage(browser), tripwire = watchConsole(page)
onTestFailed(() => saveFailureShot(page, 'web-e2e-cua-permissions'))
await page.goto(scaffold.authenticatedUrl)
await connectFreshWorkspace(page, scaffold.workspaceCwd)
const agent = scaffold.ctx.agents.list()[0]!
await scaffold.ctx.agentPresets.select(agent, 'ptc')
scaffold.ctx.approval.setPolicy(agent, 'never')
await scaffold.ctx.settings.mutate('cua', [{ op: 'set', path: ['allowedApps'], value: [] }])
// Application consent remains independent of the session's tool-approval policy.
const grant = scaffold.ctx.tools.execute({ agent, signal: new AbortController().signal,
callId: ToolCallId('cua-app-approval'), name: 'run_code',
arguments: { code: 'return await tools.cua({ method: "getApp", params: { selector: "Fixture" } })',
description: 'Request application access' } })
const approval = page.locator('[data-question-key]')
await Promise.race([approval.waitFor(), grant.then((result) => {
throw new Error(`Application selection completed before consent: ${JSON.stringify(result)}`)
})])
expect(await approval.textContent()).toContain('Fixture')
await approval.getByRole('radio', { name: 'Always allow', exact: true }).click()
await approval.getByRole('button', { name: 'Submit', exact: true }).click()
expect((await grant).isError).toBe(false)
await approval.waitFor({ state: 'hidden' })
await page.getByRole('button', { name: 'Plugins', exact: true }).click()
const card = page.locator('[data-plugin-package="@deepseek-ai/dsh-cua-bundle"]')
expect(await card.getByRole('button', { name: /Accessibility permission|Screen Recording permission/ }).count()).toBe(0)
await page.setViewportSize({ width: 1680, height: 600 })
const plugins = page.locator('[data-plugin-panel]')
const openDetails = page.getByRole('button', { name: 'View Computer Use', exact: true })
await openDetails.scrollIntoViewIfNeeded()
expect(await plugins.evaluate(element => element.scrollTop)).toBeGreaterThan(0)
await openDetails.click()
const panel = page.locator('[data-cua-settings]')
const accessibility = panel.getByRole('group', { name: 'Accessibility', exact: true })
await accessibility.getByRole('button', { name: 'Request permission', exact: true }).waitFor()
expect(await panel.getByRole('region', { name: 'Authorize Computer Use', exact: true }).getByRole('group').count()).toBe(4)
expect(await panel.getByRole('region', { name: 'Manage applications Computer Use can access', exact: true }).count()).toBe(1)
await expect.poll(() => panel.locator('li img').evaluate((image: HTMLImageElement) => image.naturalWidth)).toBe(1)
expect(await panel.getByText('/fixture/app', { exact: true }).count()).toBe(0)
expect(await accessibility.getByText('Visual Studio Code.app', { exact: true }).count()).toBe(1)
expect(await accessibility.textContent()).toContain('In System Settings, allow')
expect(await accessibility.textContent()).toContain('to read interface controls, click, and type.')
expect(await panel.getByRole('group', { name: 'Screen Recording', exact: true }).getByText('Visual Studio Code.app', { exact: true }).count()).toBe(1)
await expect.poll(() => accessibility.locator('img').evaluate((image: HTMLImageElement) => image.naturalWidth)).toBe(1)
await expect.poll(() => panel.getAttribute('aria-busy')).toBe('false')
await compareOrRefreshGolden(expected, await captureStableAria(page, '[data-cua-settings]', scaffold.workspaceCwd), webSnapshotMode())
try {
for (const width of [1680, 390]) {
await page.setViewportSize({ width, height: 1000 })
for (const colorScheme of ['light', 'dark'] as const) {
await page.emulateMedia({ colorScheme })
await expect.poll(() => page.evaluate(() => document.body.hasAttribute('data-ds-dark-theme'))).toBe(colorScheme === 'dark')
const measure = () => panel.evaluate(element => ({
clientWidth: element.clientWidth, scrollWidth: element.scrollWidth,
overflowing: [...element.querySelectorAll('*')].filter(child =>
child.getBoundingClientRect().right > element.getBoundingClientRect().right + 1,
).map(child => ({ tag: child.tagName, className: child.className, text: child.textContent?.slice(0, 80) })),
}))
await expect.poll(async () => {
const layout = await measure()
return layout.scrollWidth <= layout.clientWidth ? true : { width, colorScheme, ...layout }
}).toBe(true)
if (webSnapshotMode() === 'refresh') {
const directory = fileURLToPath(new URL('../../../.artifacts', import.meta.url))
await mkdir(directory, { recursive: true })
const path = join(directory, `cua-permission-design-${process.pid}-${width}-${colorScheme}.png`)
await panel.screenshot({ path, animations: 'disabled' })
console.log(`CUA permission design: ${path}`)
}
}
}
} finally {
await page.emulateMedia({ colorScheme: null })
await page.setViewportSize({ width: 1680, height: 1000 })
}
await accessibility.getByRole('button', { name: 'Request permission', exact: true }).click()
await expect.poll(() => panel.getAttribute('aria-busy')).toBe('false')
expect(await accessibility.getByRole('button', { name: 'Request permission', exact: true }).count()).toBe(1)
await writeFile(state, JSON.stringify({ accessibility: true, screenRecording: false, launchApplication }))
// Headless pages retain document focus; model returning from System Settings at the window-event boundary.
await page.evaluate(() => { window.dispatchEvent(new Event('focus')) })
await accessibility.getByText('Allowed', { exact: true }).waitFor()
const screen = panel.getByRole('group', { name: 'Screen Recording', exact: true })
expect(await accessibility.getByRole('button').count()).toBe(1)
await accessibility.getByRole('button', { name: 'Allowed', exact: true }).click()
expect(await screen.getByRole('button').count()).toBe(1)
await expect.poll(() => panel.getAttribute('aria-busy')).toBe('false')
await panel.getByRole('button', { name: 'Revoke', exact: true }).click()
await panel.getByRole('heading', { name: 'Always-allowed applications', exact: true }).waitFor({ state: 'hidden' })
await panel.getByRole('heading', { name: 'Applications not allowed', exact: true }).waitFor()
await panel.getByRole('button', { name: 'Allow asking again', exact: true }).click()
await panel.getByText('Fixture', { exact: true }).waitFor({ state: 'hidden' })
const saved = await readFile(join(scaffold.harnessHome, 'profiles/scaffold/cordis.patch.yml'), 'utf8')
expect(saved).toContain('allowedApps: []')
const events = (await readFile(audit, 'utf8')).trim().split('\n').map(line => JSON.parse(line) as { event: string; method: string; params: object })
.filter(record => record.event === 'call').map(({ method, params }) => ({ method, params }))
expect(events).toContainEqual({ method: 'requestPermission', params: { permission: 'accessibility' } })
expect(events.filter(event => event.method === 'openPermissionSettings')).toEqual([
{ method: 'openPermissionSettings', params: { permission: 'accessibility' } },
])
expect(events.every(event => ['permissionStatus', 'requestPermission', 'openPermissionSettings', 'applicationIcons', 'getApp'].includes(event.method))).toBe(true)
expect(tripwire.pageErrors).toEqual([])
}, 60_000)
@@ -0,0 +1,26 @@
- region "Authorize Computer Use":
- heading "Authorize Computer Use" [level=3]
- group "Default mode for new sessions":
- strong: Default mode for new sessions
- paragraph: Computer Use is available only in PTC mode.
- button "Set to PTC"
- group "PTC execution mode":
- strong: PTC execution mode
- paragraph: Keep variables and application objects across tool calls. Switching restarts the PTC runtime.
- button "Use persistent mode"
- group "Accessibility":
- strong: Accessibility
- paragraph: In System Settings, allow Visual Studio Code.app to read interface controls, click, and type.
- button "Request permission"
- group "Screen Recording":
- strong: Screen Recording
- paragraph: In System Settings, allow Visual Studio Code.app to capture application windows.
- button "Request permission"
- paragraph: Enable the application in System Settings. Follow the macOS prompt if it names another application.
- region "Manage applications Computer Use can access":
- heading "Manage applications Computer Use can access" [level=3]
- heading "Always-allowed applications" [level=4]
- list:
- listitem:
- strong: Fixture
- button "Revoke"
@@ -38,6 +38,10 @@
- button "查看 开发者工具": 开发者工具
- text: 实验性 查看调试会话原始数据、聊天消息分组数据,以及调试 NodeJS 后端
- switch "启用 开发者工具"
- listitem:
- button "查看 Computer Use": Computer Use
- text: 目前仅支持 macOS,Windows 支持正在开发中。为已有及新建 PTC 会话加载 Computer Use。请切换到 PTC 模式来使用,并授予辅助功能和屏幕录制权限。
- switch "启用 Computer Use"
- heading "已安装插件" [level=3]
- list:
- listitem:
@@ -38,6 +38,10 @@
- button "查看 开发者工具": 开发者工具
- text: 实验性 查看调试会话原始数据、聊天消息分组数据,以及调试 NodeJS 后端
- switch "启用 开发者工具"
- listitem:
- button "查看 Computer Use": Computer Use
- text: 目前仅支持 macOS,Windows 支持正在开发中。为已有及新建 PTC 会话加载 Computer Use。请切换到 PTC 模式来使用,并授予辅助功能和屏幕录制权限。
- switch "启用 Computer Use"
- heading "已安装插件" [level=3]
- list:
- listitem:
@@ -38,6 +38,10 @@
- button "查看 开发者工具": 开发者工具
- text: 实验性 查看调试会话原始数据、聊天消息分组数据,以及调试 NodeJS 后端
- switch "启用 开发者工具"
- listitem:
- button "查看 Computer Use": Computer Use
- text: 目前仅支持 macOS,Windows 支持正在开发中。为已有及新建 PTC 会话加载 Computer Use。请切换到 PTC 模式来使用,并授予辅助功能和屏幕录制权限。
- switch "启用 Computer Use"
- heading "已安装插件" [level=3]
- list:
- listitem:
@@ -6,7 +6,7 @@ Focus at 500ms:
- tooltip "刷新"
failure: pending
- button "刷新" [disabled]
{"busy":"true","spinners":1,"cards":11,"skeletons":0,"alerts":[],"retry":0,"toasts":[]}
{"busy":"true","spinners":1,"cards":12,"skeletons":0,"alerts":[],"retry":0,"toasts":[]}
failure: settled
- button "刷新"
{"busy":"false","spinners":0,"cards":11,"skeletons":0,"alerts":[],"retry":0,"toasts":["刷新失败,请重试"]}
{"busy":"false","spinners":0,"cards":12,"skeletons":0,"alerts":[],"retry":0,"toasts":["刷新失败,请重试"]}
+12 -3
View File
@@ -15,7 +15,7 @@ import {
acknowledgeReloadConnectionLoss, assertFixtureInventory, captureStableAria, compareOrRefreshGolden, fixtureUserPrompts,
launchWebScaffold, parseSeedFixture, realizeSeedFixture, renderSeedFixture, seedSession, watchConsole, webSnapshotMode, type WebScaffold,
} from './scaffold.ts'
import { openSettings, newEnglishPage, pinBrowserClock, saveFailureShot, WEB_FIXTURE_TIME } from './support.ts'
import { openSettings, newEnglishPage, pinBrowserClock, pinHostClock, saveFailureShot, WEB_FIXTURE_TIME } from './support.ts'
const SNAPSHOT_DIR = fileURLToPath(new URL('../../../snapshots/web/message-actions', import.meta.url))
// Borrowed read-only: this scenario needs any settled user+assistant pair, not
@@ -191,6 +191,7 @@ describe('web e2e: message IconActions and clocks on settled history', () => {
let browser: Browser
let page: Page
let tripwire: ReturnType<typeof watchConsole>
let hostClock: ReturnType<typeof pinHostClock> | undefined
beforeAll(async () => {
scaffold = await launchWebScaffold()
@@ -211,6 +212,10 @@ describe('web e2e: message IconActions and clocks on settled history', () => {
await seedSession(scaffold, raw, SEED_ID, undefined, { createdAt: WEB_FIXTURE_TIME - 90_000 })
browser = await chromium.launch()
page = await newEnglishPage(browser)
// Fork summaries use Host time; placeholders use browser time. Each pin
// starts at the fixture date, so keep these calls adjacent: setup between
// them can create enough clock skew to reverse the fork order.
hostClock = pinHostClock()
await pinBrowserClock(page)
tripwire = watchConsole(page)
await page.goto(scaffold.authenticatedUrl, { waitUntil: 'load' })
@@ -218,8 +223,12 @@ describe('web e2e: message IconActions and clocks on settled history', () => {
}, 120_000)
afterAll(async () => {
await browser?.close()
await scaffold?.close()
try {
await browser?.close()
await scaffold?.close()
} finally {
hostClock?.mockRestore()
}
})
it.skipIf(MODE === 'record')('enables branch only on the completed transcript tail', async () => {
+16 -3
View File
@@ -681,15 +681,28 @@ async function toggleOptionalBundles(page: Page): Promise<void> {
}
})
expect(result.optional.map(entry => entry.name).sort()).toEqual([...OPTIONAL_BUNDLES].sort())
expect(result.optional.filter(entry => entry.enabled || entry.availability !== 'installation' || entry.error !== null)).toEqual([])
expect(result.optional.filter(entry => entry.enabled || entry.availability !== 'installation')).toEqual([])
expect(result.optional.filter(entry => entry.error !== null)).toEqual([])
// Metadata resolves through the Worker's Node internal modules: every optional bundle shows its title and icon.
expect(result.optional.filter(entry => entry.metaError !== null || entry.title !== 'object' || entry.icon !== 'string')).toEqual([])
expect(result.inventoryErrors).toEqual([])
console.info(`preview boot: optional bundle toggles ${JSON.stringify(result.toggles.map(({ name, enable, disable, error }) => ({ name, enable, disable, error })))}`)
expect(result.toggles.filter(toggle => toggle.enable !== 'applied' || toggle.disable !== 'applied')
const cua = result.toggles.find(toggle => toggle.name === '@deepseek-ai/dsh-cua-bundle')!
expect(cua).toMatchObject({
enable: 'applied', disable: 'applied',
error: null,
restored: true,
})
expect(cua.changed).toBeGreaterThan(0)
expect(cua.inactive).toEqual([])
expect(cua.plugins).toContainEqual(expect.objectContaining({
moduleName: '@deepseek-ai/dsh-experimental-cua-macos', enabled: false, fiberPhase: null,
}))
const supported = result.toggles.filter(toggle => toggle.name !== '@deepseek-ai/dsh-cua-bundle')
expect(supported.filter(toggle => toggle.enable !== 'applied' || toggle.disable !== 'applied')
.map(({ name, enable, disable }) => ({ name, enable, disable }))).toEqual([])
// Each bundle changes Loader rows, every enabled row it changes runs, and switching it off restores the prior rows.
expect(result.toggles.filter(toggle => toggle.changed === 0 || toggle.inactive.length > 0 || !toggle.restored)
expect(supported.filter(toggle => toggle.changed === 0 || toggle.inactive.length > 0 || !toggle.restored)
.map(({ name, changed, inactive, restored }) => ({ name, changed, inactive, restored }))).toEqual([])
const pluginsWith = (bundle: string): Array<{ moduleName: string; enabled: boolean; fiberPhase: string | null }> =>
result.toggles.find(toggle => toggle.name === bundle)?.plugins ?? []
+2
View File
@@ -82,6 +82,8 @@
"tests/plugin-config.e2e.ts",
"tests/input-plugin-extensions.e2e.ts",
"tests/plugin-manager.e2e.ts",
"tests/cua-optin.e2e.ts",
"tests/cua-permissions.e2e.ts",
"tests/schedule-header.e2e.ts",
"tests/worktree-opt-in.e2e.ts",
"tests/session-inspector.e2e.ts",
+2 -2
View File
@@ -6,5 +6,5 @@
en: 41d772192075e133
zh: 5ef3da69b4de1d8a
/capability-seams-and-core-services:
en: f209321ab0da4f90
zh: 4af5c135eec51ce1
en: 753b3da3abc6aeea
zh: aab02ec16c3f744b
+5
View File
@@ -52,6 +52,8 @@ flowchart LR
svc_computerUse["ctx.computerUse<br/>Computer-use provider registration"]
pkg_experimental_computer_use_cua_driver_mcp["experimental-computer-use-cua-driver-mcp"]
pkg_experimental_computer_use_cua_driver_native["experimental-computer-use-cua-driver-native"]
pkg_cua["cua"]
svc_cuaAccess["ctx.cuaAccess<br/>Computer Use permission controls"]
pkg_office_to_pdf["office-to-pdf"]
svc_officeToPdf["ctx.officeToPdf<br/>Office to PDF conversion"]
pkg_client_ui_sidebar_documentpreview["client-ui-sidebar-documentpreview"]
@@ -334,6 +336,7 @@ flowchart LR
pkg_cordis_host_runner --> svc_dynamicCordisRunner
pkg_credentials --> svc_credentials
pkg_credentials_local --> svc_credentials
pkg_cua --> svc_cuaAccess
pkg_deepseek_account --> svc_deepseekAccount
pkg_deepseek_account_platform --> svc_deepseekAccount
pkg_deepseek_llm_api_extensions --> svc_deepseekLlmApiExtensions
@@ -484,6 +487,7 @@ flowchart LR
svc_credentials --> pkg_api_settings_controller
svc_credentials --> pkg_llm_deepseek
svc_credentials --> pkg_llm_pi_ai
svc_cuaAccess --> pkg_cua
svc_deepseekAccount --> pkg_api_account_controller
svc_deepseekAccount --> pkg_llm_deepseek
svc_deepseekLlmApiExtensions --> pkg_llm_deepseek
@@ -628,6 +632,7 @@ flowchart LR
| `ctx.mcpResources` | `seam` | [`mcp-resources`](../packages/mcp/mcp-resources) | [`mcp-client`](../packages/mcp/mcp-client) | [`mcp-resources`](../packages/mcp/mcp-resources) | - | Connection-owned providers serve shared resource tools in the calling agent scope. |
| `ctx.browserUse` | `seam` | [`browser-use`](../packages/browser-use/browser-use) | [`experimental-browser-use-playwright-mcp`](../packages/experimental/browser-use-playwright-mcp), [`experimental-browser-use-chrome-devtools-mcp`](../packages/experimental/browser-use-chrome-devtools-mcp), [`experimental-browser-use-stagehand-native`](../packages/experimental/browser-use-stagehand-native) | [`experimental-browser-use-playwright-mcp`](../packages/experimental/browser-use-playwright-mcp), [`experimental-browser-use-chrome-devtools-mcp`](../packages/experimental/browser-use-chrome-devtools-mcp), [`experimental-browser-use-stagehand-native`](../packages/experimental/browser-use-stagehand-native) | - | One provider-owned name per service instance. Providers own their tools and browser resources per live Session; the shared service has no browser operation API. |
| `ctx.computerUse` | `seam` | [`computer-use`](../packages/computer-use/computer-use) | [`experimental-computer-use-cua-driver-mcp`](../packages/experimental/computer-use-cua-driver-mcp), [`experimental-computer-use-cua-driver-native`](../packages/experimental/computer-use-cua-driver-native) | [`experimental-computer-use-cua-driver-mcp`](../packages/experimental/computer-use-cua-driver-mcp), [`experimental-computer-use-cua-driver-native`](../packages/experimental/computer-use-cua-driver-native) | - | One provider-owned name per service instance. Each provider also owns its model tools; the service has no common action API, runtime selection, or Session workflow lock. |
| `ctx.cuaAccess` | `core` | [`cua`](../packages/experimental/cua) | - | [`cua`](../packages/experimental/cua) | - | The native provider owns application consent and cancellation; the optional Client page queries and changes those decisions through its Remote. |
| `ctx.officeToPdf` | `core` | [`office-to-pdf`](../packages/document/office-to-pdf) | - | [`client-ui-sidebar-documentpreview`](../packages/client/ui-sidebar-documentpreview) | - | Authorized Office bytes are converted on the Host using the declared native target engine, or Node WASM when no native target is declared. |
| `ctx.attachments` | `seam` | [`attachment`](../packages/attachment/attachment) | [`attachment-local`](../packages/attachment/attachment-local) | [`api-session-controller`](../packages/api/session-controller), [`tool-fs`](../packages/fs/tool-fs), [`llm-pi-ai`](../packages/llm/llm-pi-ai), [`llm-deepseek`](../packages/llm/llm-deepseek) | - | The host commits accepted images before session events; provider adapters resolve authorized durable references into provider-native content. |
| `ctx.fileUploads` | `core` | [`client-file-upload`](../packages/client/file-upload) | - | [`api-session-controller`](../packages/api/session-controller) | - | Owns streaming intake, durable storage, and staged receipt lifetime; the Session controller binds receipts to accepted submissions. |
+5
View File
@@ -54,6 +54,8 @@ flowchart LR
svc_computerUse["ctx.computerUse<br/>Computer-use provider registration"]
pkg_experimental_computer_use_cua_driver_mcp["experimental-computer-use-cua-driver-mcp"]
pkg_experimental_computer_use_cua_driver_native["experimental-computer-use-cua-driver-native"]
pkg_cua["cua"]
svc_cuaAccess["ctx.cuaAccess<br/>Computer Use permission controls"]
pkg_office_to_pdf["office-to-pdf"]
svc_officeToPdf["ctx.officeToPdf<br/>Office to PDF conversion"]
pkg_client_ui_sidebar_documentpreview["client-ui-sidebar-documentpreview"]
@@ -336,6 +338,7 @@ flowchart LR
pkg_cordis_host_runner --> svc_dynamicCordisRunner
pkg_credentials --> svc_credentials
pkg_credentials_local --> svc_credentials
pkg_cua --> svc_cuaAccess
pkg_deepseek_account --> svc_deepseekAccount
pkg_deepseek_account_platform --> svc_deepseekAccount
pkg_deepseek_llm_api_extensions --> svc_deepseekLlmApiExtensions
@@ -486,6 +489,7 @@ flowchart LR
svc_credentials --> pkg_api_settings_controller
svc_credentials --> pkg_llm_deepseek
svc_credentials --> pkg_llm_pi_ai
svc_cuaAccess --> pkg_cua
svc_deepseekAccount --> pkg_api_account_controller
svc_deepseekAccount --> pkg_llm_deepseek
svc_deepseekLlmApiExtensions --> pkg_llm_deepseek
@@ -630,6 +634,7 @@ flowchart LR
| `ctx.mcpResources` | `seam` | [`mcp-resources`](../packages/mcp/mcp-resources) | [`mcp-client`](../packages/mcp/mcp-client) | [`mcp-resources`](../packages/mcp/mcp-resources) | - | 连接所有者提供的操作在调用 agent 的作用域内服务于共享资源工具。 |
| `ctx.browserUse` | `seam` | [`browser-use`](../packages/browser-use/browser-use) | [`experimental-browser-use-playwright-mcp`](../packages/experimental/browser-use-playwright-mcp), [`experimental-browser-use-chrome-devtools-mcp`](../packages/experimental/browser-use-chrome-devtools-mcp), [`experimental-browser-use-stagehand-native`](../packages/experimental/browser-use-stagehand-native) | [`experimental-browser-use-playwright-mcp`](../packages/experimental/browser-use-playwright-mcp), [`experimental-browser-use-chrome-devtools-mcp`](../packages/experimental/browser-use-chrome-devtools-mcp), [`experimental-browser-use-stagehand-native`](../packages/experimental/browser-use-stagehand-native) | - | 每个服务实例注册一个提供方拥有的名称。提供方按实时 Session 拥有自己的工具与浏览器资源;共享服务不提供浏览器操作 API。 |
| `ctx.computerUse` | `seam` | [`computer-use`](../packages/computer-use/computer-use) | [`experimental-computer-use-cua-driver-mcp`](../packages/experimental/computer-use-cua-driver-mcp), [`experimental-computer-use-cua-driver-native`](../packages/experimental/computer-use-cua-driver-native) | [`experimental-computer-use-cua-driver-mcp`](../packages/experimental/computer-use-cua-driver-mcp), [`experimental-computer-use-cua-driver-native`](../packages/experimental/computer-use-cua-driver-native) | - | 每个服务实例只注册一个提供方自定的名称。各提供方也拥有自己的模型工具;服务不提供通用操作 API、运行时选择或 Session 流程锁。 |
| `ctx.cuaAccess` | `core` | [`cua`](../packages/experimental/cua) | - | [`cua`](../packages/experimental/cua) | - | 原生 provider 负责应用授权和取消操作;可选的 Client 权限页面通过 Remote 查询和修改授权。 |
| `ctx.officeToPdf` | `core` | [`office-to-pdf`](../packages/document/office-to-pdf) | - | [`client-ui-sidebar-documentpreview`](../packages/client/ui-sidebar-documentpreview) | - | 已授权的 Office 字节在宿主上使用已声明的原生目标引擎转换;未声明原生目标时使用 Node WASM。 |
| `ctx.attachments` | `seam` | [`attachment`](../packages/attachment/attachment) | [`attachment-local`](../packages/attachment/attachment-local) | [`api-session-controller`](../packages/api/session-controller), [`tool-fs`](../packages/fs/tool-fs), [`llm-pi-ai`](../packages/llm/llm-pi-ai), [`llm-deepseek`](../packages/llm/llm-deepseek) | - | 宿主会在会话事件之前提交已接受的图片;提供方适配器将已授权的持久引用解析为提供方原生内容。 |
| `ctx.fileUploads` | `core` | [`client-file-upload`](../packages/client/file-upload) | - | [`api-session-controller`](../packages/api/session-controller) | - | 负责流式接收、持久存储和暂存回执生命周期;Session Controller 将回执绑定到已接受的提交。 |
+2 -1
View File
@@ -4762,6 +4762,7 @@ These load from a `cordis.yml` entry with no `config:` block; they declare no co
| `@deepseek-ai/dsh-computer-use` | — | [`packages/computer-use/computer-use/src/index.ts`](../packages/computer-use/computer-use/src/index.ts) |
| `@deepseek-ai/dsh-config-editor` | `loader` · `profileContext` | [`packages/boot/config-editor/src/index.ts`](../packages/boot/config-editor/src/index.ts) |
| `@deepseek-ai/dsh-cordis-client-runner` | — | [`packages/extensions/cordis-client-runner/src/index.ts`](../packages/extensions/cordis-client-runner/src/index.ts) |
| `@deepseek-ai/dsh-cua` | — | [`packages/experimental/cua/src/index.ts`](../packages/experimental/cua/src/index.ts) |
| `@deepseek-ai/dsh-deepseek-llm-api-extensions` | — | [`packages/llm/deepseek-llm-api-extensions/src/index.ts`](../packages/llm/deepseek-llm-api-extensions/src/index.ts) |
| `@deepseek-ai/dsh-experimental-auto-review` | `approval` · `llm` · `permissionPresets` · `sessions` · `tools` · `workingDirectory` | [`packages/experimental/auto-review/src/index.ts`](../packages/experimental/auto-review/src/index.ts) |
| `@deepseek-ai/dsh-experimental-client-ui-agent-team` | — | [`packages/experimental/client-ui-agent-team/src/index.ts`](../packages/experimental/client-ui-agent-team/src/index.ts) |
@@ -4849,7 +4850,7 @@ Imported as libraries by other packages; a `cordis.yml` cannot load them.
| `@deepseek-ai/dsh-client-ui-slots` | — | [`packages/client/ui-slots/src/index.ts`](../packages/client/ui-slots/src/index.ts) |
| `@deepseek-ai/dsh-client-web` | — | [`packages/client/web/src/index.ts`](../packages/client/web/src/index.ts) |
| `@deepseek-ai/dsh-cmdline` | — | [`packages/boot/cmdline/src/index.ts`](../packages/boot/cmdline/src/index.ts) |
| `@deepseek-ai/dsh-cua` | — | [`packages/experimental/cua/src/index.ts`](../packages/experimental/cua/src/index.ts) |
| `@deepseek-ai/dsh-cua-bundle` | — | [`packages/experimental/cua-bundle/src/index.ts`](../packages/experimental/cua-bundle/src/index.ts) |
| `@deepseek-ai/dsh-deque` | — | [`packages/util/deque/src/index.ts`](../packages/util/deque/src/index.ts) |
| `@deepseek-ai/dsh-experimental-agent-team-profile` | — | [`packages/experimental/agent-team-profile/src/index.ts`](../packages/experimental/agent-team-profile/src/index.ts) |
| `@deepseek-ai/dsh-experimental-badge-skill-bundle` | — | [`packages/experimental/badge-skill-bundle/src/index.ts`](../packages/experimental/badge-skill-bundle/src/index.ts) |
+2 -1
View File
@@ -4764,6 +4764,7 @@ export interface Config {
| `@deepseek-ai/dsh-computer-use` | — | [`packages/computer-use/computer-use/src/index.ts`](../packages/computer-use/computer-use/src/index.ts) |
| `@deepseek-ai/dsh-config-editor` | `loader` · `profileContext` | [`packages/boot/config-editor/src/index.ts`](../packages/boot/config-editor/src/index.ts) |
| `@deepseek-ai/dsh-cordis-client-runner` | — | [`packages/extensions/cordis-client-runner/src/index.ts`](../packages/extensions/cordis-client-runner/src/index.ts) |
| `@deepseek-ai/dsh-cua` | — | [`packages/experimental/cua/src/index.ts`](../packages/experimental/cua/src/index.ts) |
| `@deepseek-ai/dsh-deepseek-llm-api-extensions` | — | [`packages/llm/deepseek-llm-api-extensions/src/index.ts`](../packages/llm/deepseek-llm-api-extensions/src/index.ts) |
| `@deepseek-ai/dsh-experimental-auto-review` | `approval` · `llm` · `permissionPresets` · `sessions` · `tools` · `workingDirectory` | [`packages/experimental/auto-review/src/index.ts`](../packages/experimental/auto-review/src/index.ts) |
| `@deepseek-ai/dsh-experimental-client-ui-agent-team` | — | [`packages/experimental/client-ui-agent-team/src/index.ts`](../packages/experimental/client-ui-agent-team/src/index.ts) |
@@ -4851,7 +4852,7 @@ export interface Config {
| `@deepseek-ai/dsh-client-ui-slots` | — | [`packages/client/ui-slots/src/index.ts`](../packages/client/ui-slots/src/index.ts) |
| `@deepseek-ai/dsh-client-web` | — | [`packages/client/web/src/index.ts`](../packages/client/web/src/index.ts) |
| `@deepseek-ai/dsh-cmdline` | — | [`packages/boot/cmdline/src/index.ts`](../packages/boot/cmdline/src/index.ts) |
| `@deepseek-ai/dsh-cua` | — | [`packages/experimental/cua/src/index.ts`](../packages/experimental/cua/src/index.ts) |
| `@deepseek-ai/dsh-cua-bundle` | — | [`packages/experimental/cua-bundle/src/index.ts`](../packages/experimental/cua-bundle/src/index.ts) |
| `@deepseek-ai/dsh-deque` | — | [`packages/util/deque/src/index.ts`](../packages/util/deque/src/index.ts) |
| `@deepseek-ai/dsh-experimental-agent-team-profile` | — | [`packages/experimental/agent-team-profile/src/index.ts`](../packages/experimental/agent-team-profile/src/index.ts) |
| `@deepseek-ai/dsh-experimental-badge-skill-bundle` | — | [`packages/experimental/badge-skill-bundle/src/index.ts`](../packages/experimental/badge-skill-bundle/src/index.ts) |
+2 -2
View File
@@ -9,9 +9,9 @@ This matrix shows which packages dispatch each harness-owned event and which pac
| Event | Mode | Declared in | Dispatchers | Listeners |
| --- | --- | --- | --- | --- |
| `agent-loop/config-start-failed` | `emit` | [`packages/core/agent-loop/src/index.ts:196`](../packages/core/agent-loop/src/index.ts) | [`agent-loop`](../packages/core/agent-loop) (`events.dispatch`) | - |
| `agent-preset/selected` | `emit` | [`packages/preset/agent-preset-registry/src/types.ts:66`](../packages/preset/agent-preset-registry/src/types.ts) | [`agent-preset-registry`](../packages/preset/agent-preset-registry) (`emit`) | `remotes` |
| `agent-preset/selected` | `emit` | [`packages/preset/agent-preset-registry/src/types.ts:66`](../packages/preset/agent-preset-registry/src/types.ts) | [`agent-preset-registry`](../packages/preset/agent-preset-registry) (`emit`) | [`cua`](../packages/experimental/cua), `remotes` |
| `agent/assistant-stream` | `emit` | [`packages/core/agent/src/runtime-types.ts:363`](../packages/core/agent/src/runtime-types.ts) | [`agent-loop`](../packages/core/agent-loop) (`emit`) | [`headless`](../packages/bundle/headless), `session-controller` |
| `agent/created` | `serial` | [`packages/core/agent/src/runtime-types.ts:261`](../packages/core/agent/src/runtime-types.ts) | [`agent`](../packages/core/agent) (`serial`) | `agent-team`, `browser-use-runtime`, `claude-code-mods`, [`file-reference-local`](../packages/context/file-reference-local), [`goal`](../packages/goal/goal), [`goal-round-driver`](../packages/goal/goal-round-driver), [`hooks-claude-code`](../packages/experimental/hooks-claude-code), [`hooks-codex`](../packages/experimental/hooks-codex), [`loader-smoke`](../packages/test-support/loader-smoke), `session-controller`, `tool-agent-team`, [`tool-subagent`](../packages/subagent/tool-subagent) |
| `agent/created` | `serial` | [`packages/core/agent/src/runtime-types.ts:261`](../packages/core/agent/src/runtime-types.ts) | [`agent`](../packages/core/agent) (`serial`) | `agent-team`, `browser-use-runtime`, `claude-code-mods`, [`cua`](../packages/experimental/cua), [`file-reference-local`](../packages/context/file-reference-local), [`goal`](../packages/goal/goal), [`goal-round-driver`](../packages/goal/goal-round-driver), [`hooks-claude-code`](../packages/experimental/hooks-claude-code), [`hooks-codex`](../packages/experimental/hooks-codex), [`loader-smoke`](../packages/test-support/loader-smoke), `session-controller`, `tool-agent-team`, [`tool-subagent`](../packages/subagent/tool-subagent) |
| `agent/disposed` | `emit` | [`packages/core/agent/src/runtime-types.ts:270`](../packages/core/agent/src/runtime-types.ts) | [`agent`](../packages/core/agent) (`events.dispatch`) | [`agent-loop`](../packages/core/agent-loop), `claude-code-mods`, [`file-reference-local`](../packages/context/file-reference-local), [`goal-round-driver`](../packages/goal/goal-round-driver), `session-controller`, [`subagent`](../packages/subagent/subagent), `tool-agent-team`, [`tool-subagent`](../packages/subagent/tool-subagent) |
| `agent/error` | `emit` | [`packages/core/agent/src/runtime-types.ts:393`](../packages/core/agent/src/runtime-types.ts) | [`agent-loop`](../packages/core/agent-loop) (`emit`) | [`acp`](../packages/acp/acp), [`deepseek-account`](../packages/credentials/deepseek-account), [`goal-round-driver`](../packages/goal/goal-round-driver), [`headless`](../packages/bundle/headless), `server`, `session-controller`, [`session-telemetry`](../packages/session/session-telemetry), [`subagent`](../packages/subagent/subagent) |
| `agent/inbox/claimed` | `emit` | [`packages/core/agent/src/runtime-types.ts:299`](../packages/core/agent/src/runtime-types.ts) | [`agent-loop`](../packages/core/agent-loop) (`emit`) | [`acp`](../packages/acp/acp), [`goal-round-driver`](../packages/goal/goal-round-driver), [`subagent`](../packages/subagent/subagent), [`tool-jobs`](../packages/jobs/tool-jobs), [`user-questions`](../packages/interaction/user-questions) |
+2 -2
View File
@@ -11,9 +11,9 @@
| Event | Mode | Declared in | Dispatchers | Listeners |
| --- | --- | --- | --- | --- |
| `agent-loop/config-start-failed` | `emit` | [`packages/core/agent-loop/src/index.ts:196`](../packages/core/agent-loop/src/index.ts) | [`agent-loop`](../packages/core/agent-loop) (`events.dispatch`) | - |
| `agent-preset/selected` | `emit` | [`packages/preset/agent-preset-registry/src/types.ts:66`](../packages/preset/agent-preset-registry/src/types.ts) | [`agent-preset-registry`](../packages/preset/agent-preset-registry) (`emit`) | `remotes` |
| `agent-preset/selected` | `emit` | [`packages/preset/agent-preset-registry/src/types.ts:66`](../packages/preset/agent-preset-registry/src/types.ts) | [`agent-preset-registry`](../packages/preset/agent-preset-registry) (`emit`) | [`cua`](../packages/experimental/cua), `remotes` |
| `agent/assistant-stream` | `emit` | [`packages/core/agent/src/runtime-types.ts:363`](../packages/core/agent/src/runtime-types.ts) | [`agent-loop`](../packages/core/agent-loop) (`emit`) | [`headless`](../packages/bundle/headless), `session-controller` |
| `agent/created` | `serial` | [`packages/core/agent/src/runtime-types.ts:261`](../packages/core/agent/src/runtime-types.ts) | [`agent`](../packages/core/agent) (`serial`) | `agent-team`, `browser-use-runtime`, `claude-code-mods`, [`file-reference-local`](../packages/context/file-reference-local), [`goal`](../packages/goal/goal), [`goal-round-driver`](../packages/goal/goal-round-driver), [`hooks-claude-code`](../packages/experimental/hooks-claude-code), [`hooks-codex`](../packages/experimental/hooks-codex), [`loader-smoke`](../packages/test-support/loader-smoke), `session-controller`, `tool-agent-team`, [`tool-subagent`](../packages/subagent/tool-subagent) |
| `agent/created` | `serial` | [`packages/core/agent/src/runtime-types.ts:261`](../packages/core/agent/src/runtime-types.ts) | [`agent`](../packages/core/agent) (`serial`) | `agent-team`, `browser-use-runtime`, `claude-code-mods`, [`cua`](../packages/experimental/cua), [`file-reference-local`](../packages/context/file-reference-local), [`goal`](../packages/goal/goal), [`goal-round-driver`](../packages/goal/goal-round-driver), [`hooks-claude-code`](../packages/experimental/hooks-claude-code), [`hooks-codex`](../packages/experimental/hooks-codex), [`loader-smoke`](../packages/test-support/loader-smoke), `session-controller`, `tool-agent-team`, [`tool-subagent`](../packages/subagent/tool-subagent) |
| `agent/disposed` | `emit` | [`packages/core/agent/src/runtime-types.ts:270`](../packages/core/agent/src/runtime-types.ts) | [`agent`](../packages/core/agent) (`events.dispatch`) | [`agent-loop`](../packages/core/agent-loop), `claude-code-mods`, [`file-reference-local`](../packages/context/file-reference-local), [`goal-round-driver`](../packages/goal/goal-round-driver), `session-controller`, [`subagent`](../packages/subagent/subagent), `tool-agent-team`, [`tool-subagent`](../packages/subagent/tool-subagent) |
| `agent/error` | `emit` | [`packages/core/agent/src/runtime-types.ts:393`](../packages/core/agent/src/runtime-types.ts) | [`agent-loop`](../packages/core/agent-loop) (`emit`) | [`acp`](../packages/acp/acp), [`deepseek-account`](../packages/credentials/deepseek-account), [`goal-round-driver`](../packages/goal/goal-round-driver), [`headless`](../packages/bundle/headless), `server`, `session-controller`, [`session-telemetry`](../packages/session/session-telemetry), [`subagent`](../packages/subagent/subagent) |
| `agent/inbox/claimed` | `emit` | [`packages/core/agent/src/runtime-types.ts:299`](../packages/core/agent/src/runtime-types.ts) | [`agent-loop`](../packages/core/agent-loop) (`emit`) | [`acp`](../packages/acp/acp), [`goal-round-driver`](../packages/goal/goal-round-driver), [`subagent`](../packages/subagent/subagent), [`tool-jobs`](../packages/jobs/tool-jobs), [`user-questions`](../packages/interaction/user-questions) |
+6 -9
View File
@@ -248,6 +248,7 @@ flowchart TD
end
subgraph group_experimental["packages/experimental"]
pkg_cua["cua"]
pkg_cua_bundle["cua-bundle"]
pkg_experimental_agent_team["experimental-agent-team"]
pkg_experimental_agent_team_profile["experimental-agent-team-profile"]
pkg_experimental_api_speech_to_text["experimental-api-speech-to-text"]
@@ -760,6 +761,9 @@ flowchart TD
pkg_deepseek_account_platform --> pkg_credentials
pkg_deepseek_account_platform --> pkg_deepseek_account
pkg_deepseek_account_platform --> pkg_host_webserver
pkg_cua --> pkg_attachment
pkg_cua --> pkg_scope
pkg_cua --> pkg_user_questions
pkg_experimental_ptc_runtime_python --> pkg_ptc_runtime
pkg_experimental_ptc_runtime_python --> pkg_sandbox
pkg_experimental_ptc_runtime_python --> pkg_sandbox_policy
@@ -903,14 +907,6 @@ flowchart TD
pkg_workspace_changes --> pkg_session
pkg_workspace_changes --> pkg_subprocess
pkg_workspace_changes --> pkg_tools
pkg_cua --> pkg_agent
pkg_cua --> pkg_attachment
pkg_cua --> pkg_computer_use
pkg_cua --> pkg_llm
pkg_cua --> pkg_subprocess
pkg_cua --> pkg_system_prompt
pkg_cua --> pkg_tools
pkg_cua --> pkg_user_questions
pkg_experimental_browser_use_chrome_devtools_mcp --> pkg_agent
pkg_experimental_browser_use_chrome_devtools_mcp --> pkg_browser_use
pkg_experimental_browser_use_chrome_devtools_mcp --> pkg_system_prompt
@@ -1537,6 +1533,7 @@ flowchart TD
| [`client-web`](../packages/client/web) | `client` | — |
| [`credentials`](../packages/credentials/credentials) | `credentials` | — |
| [`office-to-pdf`](../packages/document/office-to-pdf) | `document` | — |
| [`cua-bundle`](../packages/experimental/cua-bundle) | `experimental` | — |
| [`experimental-agent-team-profile`](../packages/experimental/agent-team-profile) | `experimental` | — |
| [`experimental-badge-skill-bundle`](../packages/experimental/badge-skill-bundle) | `experimental` | — |
| [`experimental-client-ui-agent-team`](../packages/experimental/client-ui-agent-team) | `experimental` | — |
@@ -1674,6 +1671,7 @@ flowchart TD
| [`api-account-controller`](../packages/api/account-controller) | `api` | [`agent`](../packages/core/agent), [`deepseek-account`](../packages/credentials/deepseek-account), [`typert-protocol`](../packages/typert/protocol) |
| [`compaction`](../packages/compaction/compaction) | `compaction` | [`brand`](../packages/util/brand), [`commands`](../packages/interaction/commands), [`llm`](../packages/llm/llm), [`session`](../packages/core/session) |
| [`deepseek-account-platform`](../packages/credentials/deepseek-account-platform) | `credentials` | [`authorization`](../packages/credentials/authorization), [`credentials`](../packages/credentials/credentials), [`deepseek-account`](../packages/credentials/deepseek-account), [`host-webserver`](../packages/host/webserver) |
| [`cua`](../packages/experimental/cua) | `experimental` | [`attachment`](../packages/attachment/attachment), [`scope`](../packages/core/scope), [`user-questions`](../packages/interaction/user-questions) |
| [`experimental-ptc-runtime-python`](../packages/experimental/ptc-runtime-python) | `experimental` | [`ptc-runtime`](../packages/ptc-runtime/ptc-runtime), [`sandbox`](../packages/sandbox/sandbox), [`sandbox-policy`](../packages/sandbox/sandbox-policy), [`timeout`](../packages/util/timeout), [`util-values`](../packages/util/values) |
| [`experimental-worktree`](../packages/experimental/worktree) | `experimental` | [`agent`](../packages/core/agent), [`fs`](../packages/fs/fs), [`sandbox`](../packages/sandbox/sandbox), [`sandbox-policy`](../packages/sandbox/sandbox-policy), [`subprocess`](../packages/subprocess/subprocess), [`working-directory`](../packages/session/working-directory) |
| [`command-feedback`](../packages/feedback/command-feedback) | `feedback` | [`anonymous-user-id`](../packages/identity/anonymous-user-id), [`commands`](../packages/interaction/commands), [`session`](../packages/core/session), [`typert-protocol`](../packages/typert/protocol) |
@@ -1701,7 +1699,6 @@ flowchart TD
| [`file-reference-local`](../packages/context/file-reference-local) | `context` | [`agent`](../packages/core/agent), [`file-reference`](../packages/context/file-reference), [`system-prompt`](../packages/core/system-prompt), [`tools`](../packages/core/tools), [`working-directory`](../packages/session/working-directory) |
| [`tool-present`](../packages/deliverables/tool-present) | `deliverables` | [`agent`](../packages/core/agent), [`fs`](../packages/fs/fs), [`llm`](../packages/llm/llm), [`session`](../packages/core/session), [`tools`](../packages/core/tools), [`working-directory`](../packages/session/working-directory) |
| [`workspace-changes`](../packages/deliverables/workspace-changes) | `deliverables` | [`agent`](../packages/core/agent), [`fs`](../packages/fs/fs), [`session`](../packages/core/session), [`subprocess`](../packages/subprocess/subprocess), [`tools`](../packages/core/tools) |
| [`cua`](../packages/experimental/cua) | `experimental` | [`agent`](../packages/core/agent), [`attachment`](../packages/attachment/attachment), [`computer-use`](../packages/computer-use/computer-use), [`llm`](../packages/llm/llm), [`subprocess`](../packages/subprocess/subprocess), [`system-prompt`](../packages/core/system-prompt), [`tools`](../packages/core/tools), [`user-questions`](../packages/interaction/user-questions) |
| [`experimental-browser-use-chrome-devtools-mcp`](../packages/experimental/browser-use-chrome-devtools-mcp) | `experimental` | [`agent`](../packages/core/agent), [`browser-use`](../packages/browser-use/browser-use), [`system-prompt`](../packages/core/system-prompt), [`tools`](../packages/core/tools), [`working-directory`](../packages/session/working-directory) |
| [`experimental-browser-use-playwright-mcp`](../packages/experimental/browser-use-playwright-mcp) | `experimental` | [`agent`](../packages/core/agent), [`browser-use`](../packages/browser-use/browser-use), [`system-prompt`](../packages/core/system-prompt), [`tools`](../packages/core/tools), [`working-directory`](../packages/session/working-directory) |
| [`experimental-browser-use-stagehand-native`](../packages/experimental/browser-use-stagehand-native) | `experimental` | [`agent`](../packages/core/agent), [`browser-use`](../packages/browser-use/browser-use), [`system-prompt`](../packages/core/system-prompt), [`tools`](../packages/core/tools) |
+6 -9
View File
@@ -250,6 +250,7 @@ flowchart TD
end
subgraph group_experimental["packages/experimental"]
pkg_cua["cua"]
pkg_cua_bundle["cua-bundle"]
pkg_experimental_agent_team["experimental-agent-team"]
pkg_experimental_agent_team_profile["experimental-agent-team-profile"]
pkg_experimental_api_speech_to_text["experimental-api-speech-to-text"]
@@ -762,6 +763,9 @@ flowchart TD
pkg_deepseek_account_platform --> pkg_credentials
pkg_deepseek_account_platform --> pkg_deepseek_account
pkg_deepseek_account_platform --> pkg_host_webserver
pkg_cua --> pkg_attachment
pkg_cua --> pkg_scope
pkg_cua --> pkg_user_questions
pkg_experimental_ptc_runtime_python --> pkg_ptc_runtime
pkg_experimental_ptc_runtime_python --> pkg_sandbox
pkg_experimental_ptc_runtime_python --> pkg_sandbox_policy
@@ -905,14 +909,6 @@ flowchart TD
pkg_workspace_changes --> pkg_session
pkg_workspace_changes --> pkg_subprocess
pkg_workspace_changes --> pkg_tools
pkg_cua --> pkg_agent
pkg_cua --> pkg_attachment
pkg_cua --> pkg_computer_use
pkg_cua --> pkg_llm
pkg_cua --> pkg_subprocess
pkg_cua --> pkg_system_prompt
pkg_cua --> pkg_tools
pkg_cua --> pkg_user_questions
pkg_experimental_browser_use_chrome_devtools_mcp --> pkg_agent
pkg_experimental_browser_use_chrome_devtools_mcp --> pkg_browser_use
pkg_experimental_browser_use_chrome_devtools_mcp --> pkg_system_prompt
@@ -1539,6 +1535,7 @@ flowchart TD
| [`client-web`](../packages/client/web) | `client` | — |
| [`credentials`](../packages/credentials/credentials) | `credentials` | — |
| [`office-to-pdf`](../packages/document/office-to-pdf) | `document` | — |
| [`cua-bundle`](../packages/experimental/cua-bundle) | `experimental` | — |
| [`experimental-agent-team-profile`](../packages/experimental/agent-team-profile) | `experimental` | — |
| [`experimental-badge-skill-bundle`](../packages/experimental/badge-skill-bundle) | `experimental` | — |
| [`experimental-client-ui-agent-team`](../packages/experimental/client-ui-agent-team) | `experimental` | — |
@@ -1676,6 +1673,7 @@ flowchart TD
| [`api-account-controller`](../packages/api/account-controller) | `api` | [`agent`](../packages/core/agent), [`deepseek-account`](../packages/credentials/deepseek-account), [`typert-protocol`](../packages/typert/protocol) |
| [`compaction`](../packages/compaction/compaction) | `compaction` | [`brand`](../packages/util/brand), [`commands`](../packages/interaction/commands), [`llm`](../packages/llm/llm), [`session`](../packages/core/session) |
| [`deepseek-account-platform`](../packages/credentials/deepseek-account-platform) | `credentials` | [`authorization`](../packages/credentials/authorization), [`credentials`](../packages/credentials/credentials), [`deepseek-account`](../packages/credentials/deepseek-account), [`host-webserver`](../packages/host/webserver) |
| [`cua`](../packages/experimental/cua) | `experimental` | [`attachment`](../packages/attachment/attachment), [`scope`](../packages/core/scope), [`user-questions`](../packages/interaction/user-questions) |
| [`experimental-ptc-runtime-python`](../packages/experimental/ptc-runtime-python) | `experimental` | [`ptc-runtime`](../packages/ptc-runtime/ptc-runtime), [`sandbox`](../packages/sandbox/sandbox), [`sandbox-policy`](../packages/sandbox/sandbox-policy), [`timeout`](../packages/util/timeout), [`util-values`](../packages/util/values) |
| [`experimental-worktree`](../packages/experimental/worktree) | `experimental` | [`agent`](../packages/core/agent), [`fs`](../packages/fs/fs), [`sandbox`](../packages/sandbox/sandbox), [`sandbox-policy`](../packages/sandbox/sandbox-policy), [`subprocess`](../packages/subprocess/subprocess), [`working-directory`](../packages/session/working-directory) |
| [`command-feedback`](../packages/feedback/command-feedback) | `feedback` | [`anonymous-user-id`](../packages/identity/anonymous-user-id), [`commands`](../packages/interaction/commands), [`session`](../packages/core/session), [`typert-protocol`](../packages/typert/protocol) |
@@ -1703,7 +1701,6 @@ flowchart TD
| [`file-reference-local`](../packages/context/file-reference-local) | `context` | [`agent`](../packages/core/agent), [`file-reference`](../packages/context/file-reference), [`system-prompt`](../packages/core/system-prompt), [`tools`](../packages/core/tools), [`working-directory`](../packages/session/working-directory) |
| [`tool-present`](../packages/deliverables/tool-present) | `deliverables` | [`agent`](../packages/core/agent), [`fs`](../packages/fs/fs), [`llm`](../packages/llm/llm), [`session`](../packages/core/session), [`tools`](../packages/core/tools), [`working-directory`](../packages/session/working-directory) |
| [`workspace-changes`](../packages/deliverables/workspace-changes) | `deliverables` | [`agent`](../packages/core/agent), [`fs`](../packages/fs/fs), [`session`](../packages/core/session), [`subprocess`](../packages/subprocess/subprocess), [`tools`](../packages/core/tools) |
| [`cua`](../packages/experimental/cua) | `experimental` | [`agent`](../packages/core/agent), [`attachment`](../packages/attachment/attachment), [`computer-use`](../packages/computer-use/computer-use), [`llm`](../packages/llm/llm), [`subprocess`](../packages/subprocess/subprocess), [`system-prompt`](../packages/core/system-prompt), [`tools`](../packages/core/tools), [`user-questions`](../packages/interaction/user-questions) |
| [`experimental-browser-use-chrome-devtools-mcp`](../packages/experimental/browser-use-chrome-devtools-mcp) | `experimental` | [`agent`](../packages/core/agent), [`browser-use`](../packages/browser-use/browser-use), [`system-prompt`](../packages/core/system-prompt), [`tools`](../packages/core/tools), [`working-directory`](../packages/session/working-directory) |
| [`experimental-browser-use-playwright-mcp`](../packages/experimental/browser-use-playwright-mcp) | `experimental` | [`agent`](../packages/core/agent), [`browser-use`](../packages/browser-use/browser-use), [`system-prompt`](../packages/core/system-prompt), [`tools`](../packages/core/tools), [`working-directory`](../packages/session/working-directory) |
| [`experimental-browser-use-stagehand-native`](../packages/experimental/browser-use-stagehand-native) | `experimental` | [`agent`](../packages/core/agent), [`browser-use`](../packages/browser-use/browser-use), [`system-prompt`](../packages/core/system-prompt), [`tools`](../packages/core/tools) |
+2 -2
View File
@@ -3,8 +3,8 @@
# After editing either side, bring the other along and re-record with:
# pnpm run verify-translation-pairing --write docs/subsystems/computer-use.md
/computer-use:
en: a82a6d1ec3fa9724
zh: 17483872a54d6899
en: aff84e3ac8c41594
zh: bee7fc09854bfa8b
/computer-use/choose-a-provider:
en: fa2e0d7b41c0450b
zh: af9a799711f901e6
+76
View File
@@ -4,9 +4,19 @@ English | [中文](computer-use.zh.md)
Computer use lets a model observe and operate the local desktop through a configured provider. The shared DSH capability is called **computer use**; **Cua Driver** names the upstream implementation.
[Computer Use](../../packages/experimental/cua-bundle/README.md) provides an optional experimental switch for macOS CUA permission setup.
The [`macOS CUA provider`](../../packages/experimental/cua-macos/README.md) uses the [shared CUA objects and executor](../../packages/experimental/cua/README.md), which enforce application consent through User Questions and profile configuration.
The CUA provider owns `ctx.cuaAccess` for system-permission status, explicit system requests, and application revocation. Its [permission page](../../packages/experimental/cua/README.md) projects these controls; app/window operations enforce the provider's consent policy before dispatch.
| Permission type | Meaning |
|---|---|
| `CuaApplicationId`, `CuaApplicationGrant` | An executable/signature identity and its displayed saved decision. |
| `CuaSystemPermission` | Accessibility or Screen Recording, selected by the human. |
| `CuaPermissionProcess`, `CuaSystemPermissionState` | Native privilege checks, observed helper/parent processes, and the application identified by macOS process responsibility at helper startup. Application metadata is absent if the optional system query or application lookup is unavailable. |
| `CuaApplicationState` | Saved grants, refused applications, and whether persistence is available. Session-only grants remain internal to the provider. System observations and application icons are queried separately so the page can display each result as it arrives. |
## Choose a provider
Mount [`dsh-computer-use`](../../packages/computer-use/computer-use/README.md) and one provider in the same composition. Both Cua Driver providers are experimental public npm packages and require explicit activation.
@@ -56,4 +66,70 @@ register(name: ComputerUseProviderName): () => Promise<void>
```
Source: [`packages/computer-use/computer-use/src/index.ts`](../../packages/computer-use/computer-use/src/index.ts)
<a id="ctxcuaaccess--cuaaccess"></a>
### `ctx.cuaAccess` — `CuaAccess`
Browser controls for provider-owned native permission and application decisions.
```ts cordis-catalog
/** Read PTC execution state separately from native permission checks.
* @returns Whether the shared runtime retains memory and has an editable profile entry.
*/
@Remote ptcConfiguration(): { persistent: boolean; writable: boolean }
/** Persist in-memory PTC execution through the existing profile editor and Loader lifecycle.
* @returns Completion after the shared runtime reload; already-persistent runtimes are left intact.
*/
@Remote async enablePersistentPtc(): Promise<void>
/**
* Read application decisions without waiting for the native helper.
* @returns Saved grants and refused applications with their persistence owner.
*/
@Remote applications(): CuaApplicationState
/**
* Inspect system privileges independently of application decisions and icons.
* @returns Fresh native preflight and launch metadata.
*/
@Remote async systemStatus(): Promise<CuaSystemPermissionState>
/**
* Load presentation icons separately from application decisions.
* @returns Optional native PNG icons keyed by executable path for saved and blocked applications.
*/
@Remote async applicationIcons(): Promise<Record<string, string>>
/**
* Request a system privilege explicitly selected by the user.
* @param permission macOS privilege to request.
* @returns Preflight state after requesting; a false value never implies consent.
*/
@Remote async requestPermission(permission: CuaSystemPermission): Promise<CuaSystemPermissionState>
/**
* Open the macOS privacy pane selected by the user.
* @param permission Corresponding system privilege.
* @returns Completion after macOS accepted the open request.
*/
@Remote async openSettings(permission: CuaSystemPermission): Promise<void>
/**
* Remove saved and session access and join affected operations.
* @param id Application identity displayed on the permission page.
* @returns Completion after persistence and cancellation; dispatched input cannot be undone.
*/
@Remote revoke(id: CuaApplicationId): Promise<void>
/**
* Clear this application's refusals across sessions so each session may ask for access again.
* @param id Refused or revoked application displayed on the permission page.
* @returns Completion after clearing refusals; no permission is granted and no action is replayed.
*/
@Remote async retryApproval(id: CuaApplicationId): Promise<void>
```
Source: [`packages/experimental/cua/src/access.ts`](../../packages/experimental/cua/src/access.ts)
<!-- END GENERATED cordis-surface -->
+76
View File
@@ -4,9 +4,19 @@
计算机操作让模型通过配置的提供方观察并操作本地桌面。DSH 的共享能力称为 **computer use(计算机操作)**;**Cua Driver** 是上游实现的名称。
[Computer Use](../../packages/experimental/cua-bundle/README.zh.md) 提供 macOS CUA 及权限设置 的可选实验性开关。
[macOS CUA 提供方](../../packages/experimental/cua-macos/README.zh.md)使用[共享 CUA 对象与执行器](../../packages/experimental/cua/README.zh.md),通过 User Questions 和配置档设置执行应用授权。
CUA provider 通过 `ctx.cuaAccess` 提供系统权限状态、用户主动申请和应用授权撤销。[权限页面](../../packages/experimental/cua/README.zh.md)投影这些控件;app/window 操作在分派前执行 provider 的授权检查。
| 权限类型 | 含义 |
|---|---|
| `CuaApplicationId`、`CuaApplicationGrant` | 可执行文件和签名身份,以及显示给用户的已保存决定。 |
| `CuaSystemPermission` | 用户选择申请的辅助功能或屏幕录制权限。 |
| `CuaPermissionProcess`、`CuaSystemPermissionState` | 观测到的 helper、父进程和原生权限状态,以及 helper 启动时由 macOS 责任进程记录识别出的应用。可选系统查询或应用查找不可用时,省略应用信息。 |
| `CuaApplicationState` | 持久授权、拒绝记录,以及能否持久保存。会话授权只保留在 provider 内部。系统权限和应用图标分别查询,页面在各项结果返回后分别显示。 |
## 选择提供方
在同一组合中挂载 [`dsh-computer-use`](../../packages/computer-use/computer-use/README.zh.md) 和一个提供方。两个 Cua Driver 提供方都是公开发布到 npm 的实验性包,均需显式启用。
@@ -56,4 +66,70 @@ register(name: ComputerUseProviderName): () => Promise<void>
```
Source: [`packages/computer-use/computer-use/src/index.ts`](../../packages/computer-use/computer-use/src/index.ts)
<a id="ctxcuaaccess--cuaaccess"></a>
### `ctx.cuaAccess` — `CuaAccess`
Browser controls for provider-owned native permission and application decisions.
```ts cordis-catalog
/** Read PTC execution state separately from native permission checks.
* @returns Whether the shared runtime retains memory and has an editable profile entry.
*/
@Remote ptcConfiguration(): { persistent: boolean; writable: boolean }
/** Persist in-memory PTC execution through the existing profile editor and Loader lifecycle.
* @returns Completion after the shared runtime reload; already-persistent runtimes are left intact.
*/
@Remote async enablePersistentPtc(): Promise<void>
/**
* Read application decisions without waiting for the native helper.
* @returns Saved grants and refused applications with their persistence owner.
*/
@Remote applications(): CuaApplicationState
/**
* Inspect system privileges independently of application decisions and icons.
* @returns Fresh native preflight and launch metadata.
*/
@Remote async systemStatus(): Promise<CuaSystemPermissionState>
/**
* Load presentation icons separately from application decisions.
* @returns Optional native PNG icons keyed by executable path for saved and blocked applications.
*/
@Remote async applicationIcons(): Promise<Record<string, string>>
/**
* Request a system privilege explicitly selected by the user.
* @param permission macOS privilege to request.
* @returns Preflight state after requesting; a false value never implies consent.
*/
@Remote async requestPermission(permission: CuaSystemPermission): Promise<CuaSystemPermissionState>
/**
* Open the macOS privacy pane selected by the user.
* @param permission Corresponding system privilege.
* @returns Completion after macOS accepted the open request.
*/
@Remote async openSettings(permission: CuaSystemPermission): Promise<void>
/**
* Remove saved and session access and join affected operations.
* @param id Application identity displayed on the permission page.
* @returns Completion after persistence and cancellation; dispatched input cannot be undone.
*/
@Remote revoke(id: CuaApplicationId): Promise<void>
/**
* Clear this application's refusals across sessions so each session may ask for access again.
* @param id Refused or revoked application displayed on the permission page.
* @returns Completion after clearing refusals; no permission is granted and no action is replayed.
*/
@Remote async retryApproval(id: CuaApplicationId): Promise<void>
```
Source: [`packages/experimental/cua/src/access.ts`](../../packages/experimental/cua/src/access.ts)
<!-- END GENERATED cordis-surface -->
@@ -0,0 +1,16 @@
# Bilingual-pair consistency record for guide.md (docs/i18n/README.md): per heading
# section, a hash of its English and Chinese blocks outside code blocks and generated regions.
# After editing either side, bring the other along and re-record with:
# pnpm run verify-translation-pairing --write docs/upgrade-guide/v0.2.1-rc.2/cua-preset-scope/guide.md
/:
en: 36d6b79b48c08dd5
zh: 814caece159e9b0e
/cua-defaults-to-ptc-session-scopes:
en: a53b6884785bdf56
zh: 7d27fc9b029e0e3e
/cua-defaults-to-ptc-session-scopes/change:
en: dac96a29e3751249
zh: d48cb030f43c07e4
/cua-defaults-to-ptc-session-scopes/migration:
en: f3d82fe45e41c712
zh: e55e424f5a2c096a
@@ -0,0 +1,18 @@
---
kind: upgrade-guide
description: "Standalone native CUA compositions must explicitly select global tool registration."
---
# CUA defaults to PTC session scopes
English | [中文](guide.zh.md)
## Change
The native CUA provider defaults `agentPreset` to `ptc`. It registers tools for existing and new PTC Agents, including when a saved application-grants configuration omits the new field. Custom compositions without an Agent preset registry no longer receive global CUA tools by default.
## Migration
1. For the standard Computer Use switch, keep the default; existing application grants remain usable.
2. For a standalone composition that needs global tools, add `agentPreset: false` under the native CUA entry’s `config` in its profile patch or overlay. Preserve its other fields, including `allowedApps` and `helperPath`.
3. Restart or let HMR apply the edit, then confirm that the intended Agent exposes `cua` in native mode or through the PTC SDK.
@@ -0,0 +1,18 @@
---
kind: upgrade-guide
description: "独立原生 CUA 组合需要显式选择全局工具注册。"
---
# CUA 默认使用 PTC 会话作用域
[English](guide.md) | 中文
## 变更
原生 CUA 提供方的 `agentPreset` 默认为 `ptc`。工具注册到已有和新建 PTC Agent,即使保存的应用授权配置没有这个新字段也会生效。不包含 Agent preset 注册表的自定义组合不再默认获得全局 CUA 工具。
## 迁移
1. 使用标准 Computer Use 开关时保持默认值,已有应用授权仍可使用。
2. 独立组合若需要全局工具,在 profile patch 或 overlay 的原生 CUA 条目 `config` 下添加 `agentPreset: false`,保留 `allowedApps`、`helperPath` 等其他字段。
3. 重启或等待 HMR 应用后,确认目标 Agent 能在原生模式或 PTC SDK 中使用 `cua`。
+1
View File
@@ -291,6 +291,7 @@
"@deepseek-ai/dsh-working-directory": "workspace:*",
"@deepseek-ai/dsh-experimental-worktree": "workspace:*",
"@deepseek-ai/dsh-experimental-tool-worktree": "workspace:*",
"@deepseek-ai/dsh-cua-bundle": "workspace:*",
"@deepseek-ai/dsh-experimental-subagent-oar": "workspace:*",
"@deepseek-ai/dsh-experimental-tool-subagent-oar": "workspace:*",
"@deepseek-ai/dsh-experimental-cua-macos": "workspace:*"
+2 -2
View File
@@ -24,8 +24,8 @@
en: 4a424cd401556f20
zh: 0b34dd18e0db6cea
/deepseek-ai-dsh-app-boot/use-this-package/profiles:
en: d2d520668668292c
zh: b7deeab27032a3ad
en: 0a1b5f4f40d835a7
zh: d8f4b54160aa033b
/deepseek-ai-dsh-app-boot/use-this-package/previewing-the-effective-configuration:
en: 646a0ccdd7881ad1
zh: e3696acb1573dd7b
+1 -1
View File
@@ -64,7 +64,7 @@ The enabled `dsh-hmr` plugin watches the profile manifest and both user patch fi
Inserted plugin names may be absolute filesystem paths, file URLs, or package specifiers. Patch loading converts absolute paths and patch-relative `./` or `../` paths to file URLs within `insert` rows and their nested groups; existing-entry name assertions and replacement `config` values remain literal.
Before mounting profile rows, the `dsh` launcher computes one immutable runtime resolution from the installation and ordered bundle dependency graphs. Every profile launcher uses runtime resolution, including plain Node, packaged executables, and the Electron Host. It installs the runtime resolution through Node's ESM and CommonJS resolvers without creating fallback links.
Before mounting profile rows, the `dsh` launcher computes one immutable runtime resolution from the installation and ordered bundle dependency graphs. Every profile launcher uses runtime resolution, including plain Node, packaged executables, and the Electron Host. It installs the runtime resolution through Node's ESM and CommonJS resolvers without creating fallback links. Traversal includes installed optional dependencies and skips those absent from the target installation.
`sanitizeProfile(binName, profileDir, bundles)` provides filesystem recovery without loading plugins or parsing patches. Desktop uses it for native fatal recovery. Call it only after stopping the profile and excluding concurrent profile writes. It renames the profile’s `cordis.patch.yml` to a unique `.bak-<timestamp>` sibling and restores the supplied bundle list, preserving installed packages and other manifest fields. The timestamp is Unix time in milliseconds; collisions append an ordinal (`-1`, `-2`, …) without changing it. It returns the backup path, or `undefined` when no patch exists; missing profiles remain absent. Profile initialization recreates an empty patch on the next launch. The home-level patch is unchanged. Invalid profile JSON fails before mutation; later errors propagate and retain completed changes for retry.
+1 -1
View File
@@ -64,7 +64,7 @@ profile 导入插件前,DSH 会将其声明的 DSH 版本要求与 `getDshRunt
插入条目的插件名可以是绝对文件系统路径、文件 URL 或包标识符。patch 加载会把 `insert` 条目及其嵌套分组中的绝对路径以及相对于 patch 文件的 `./` 或 `../` 路径转换为文件 URL;对已有条目名称的断言及替换用的 `config` 值保持原样。
挂载 profile 条目前,`dsh` launcher 会从安装依赖图与有序 bundle 依赖图计算一份不可变的 runtime resolution。普通 Node、打包可执行文件与 Electron Host 等所有 profile 启动器都使用 runtime 解析,将 runtime resolution 安装到 Node 的 ESM 与 CommonJS 解析器中,不创建 fallback 链接。
挂载 profile 条目前,`dsh` launcher 会从安装依赖图与有序 bundle 依赖图计算一份不可变的 runtime resolution。普通 Node、打包可执行文件与 Electron Host 等所有 profile 启动器都使用 runtime 解析,将 runtime resolution 安装到 Node 的 ESM 与 CommonJS 解析器中,不创建 fallback 链接。依赖遍历包含已安装的可选依赖,跳过目标安装中缺席的包。
`sanitizeProfile(binName, profileDir, bundles)` 提供文件恢复,无需加载插件或解析 patch。Desktop 在原生致命错误恢复中调用它。调用前必须停止 profile 并排除并发 profile 写入。它将 profile 的 `cordis.patch.yml` 重命名为带唯一 `.bak-<timestamp>` 后缀的同目录备份,并恢复调用方指定的 bundle 列表,保留已安装包和其他 manifest 字段。时间戳为 Unix 毫秒数;同名备份已存在时追加序号(`-1`、`-2`、……),时间戳保持不变。返回值为备份路径;patch 不存在时返回 `undefined`,缺失的 profile 不会被创建。下次启动的 profile 初始化会重新创建空 patch。home 级 patch 不变。无效 profile JSON 在修改前报错;后续错误向调用方抛出,保留已完成的修改供重试。
+3 -1
View File
@@ -221,6 +221,7 @@ export const DEFAULT_PROFILE_BUNDLES: readonly string[] = ['@deepseek-ai/dsh-bas
* [admission](../../../../.agents/notes/implemented/architecture/2026-09-21-experimental-capabilities-as-optional-bundles.md)).
*/
export const OPTIONAL_BUNDLES: readonly string[] = [
'@deepseek-ai/dsh-cua-bundle',
'@deepseek-ai/dsh-experimental-session-search',
'@deepseek-ai/dsh-experimental-ralph-bundle',
'@deepseek-ai/dsh-experimental-terminal-bundle',
@@ -374,7 +375,8 @@ function readPackageManifest(anchor: string): ProfileManifest {
/** Return dependency names that may be imported by a loader-visible plugin. */
function profileDependencyNames(manifest: ProfileManifest): string[] {
return [...Object.keys(manifest.dependencies ?? {}), ...Object.keys(manifest.peerDependencies ?? {})]
return [...Object.keys(manifest.dependencies ?? {}), ...Object.keys(manifest.peerDependencies ?? {}),
...Object.keys(manifest.optionalDependencies ?? {})]
}
/** Resolve the installation packages that the runtime resolver supplies to every profile. */
@@ -334,3 +334,28 @@ describe('profile package metadata service', () => {
expect(getEnvironmentData(key)).toBe(previous)
})
})
it.each(['installation', 'bundle'] as const)('exposes installed optional dependencies from %s and skips absent ones', async (scope) => {
const root = realpathSync.native(mkdtempSync(join(tmpdir(), 'dsh-optional-resolution-')))
roots.push(root)
const installAnchor = join(root, 'install', 'package.json')
file(installAnchor, JSON.stringify({ name: 'installation', version: '1.0.0' }))
const profileDir = join(root, 'profiles', 'test')
initProfile(profileDir, ['optional-bundle'])
const bundleDir = join(profileDir, 'node_modules', 'optional-bundle')
const bundleManifest = { name: 'optional-bundle', version: '1.0.0', dsh: { bundle: { patch: './cordis.patch.yml' } } }
file(join(bundleDir, 'package.json'), JSON.stringify(bundleManifest))
file(join(bundleDir, 'cordis.patch.yml'), '[]\n')
const anchor = scope === 'installation' ? installAnchor : join(bundleDir, 'package.json')
file(anchor, JSON.stringify({ ...JSON.parse(readFileSync(anchor, 'utf8')) as object,
optionalDependencies: { 'metadata-lib': '*', 'absent-platform': '*' },
}))
const optionalDir = join(dirname(anchor), 'node_modules', 'metadata-lib')
pkg(optionalDir, '1.0.0')
const active = await createRuntimeResolution({
installAnchor, home: root, profile: loadProfileDirectory('dsh', profileDir, installAnchor),
})
expect(active.entries).toContainEqual(expect.objectContaining({ name: 'metadata-lib', packageDir: optionalDir }))
expect(active.entries.some(entry => entry.name === 'absent-platform')).toBe(false)
})
+2 -2
View File
@@ -15,8 +15,8 @@
en: d152484eb41ac6b4
zh: 405b30feaf48c087
/deepseek-ai-dsh-client-ui-schedule/use-this-package:
en: edba0e1573f3ce42
zh: 77875b469d94cc62
en: 7d44cd1640bebfad
zh: 1d6b6c28a3c27115
/deepseek-ai-dsh-client-ui-schedule/use-this-package/time-zone-data-source:
en: 6bd6882dbbbefcb0
zh: 03aa15f1a65d13dc
+1 -1
View File
@@ -25,7 +25,7 @@ Manage active and inactive tasks from the Automation tasks page and the right Si
<a id="use-this-package"></a>
## Use this package
The shipped Web composition mounts `ui-schedule` with the Host Schedule service. The clock context that supplies the model's current time is preset-owned: `standard`, `cordis`, and `ptc` declare it, and `minimal` declares neither it nor the reminder tools. The Automation tasks sidebar entry opens a global page without selecting or activating a Session. The page header's New action starts a new Session, where the reminder is created by asking the model; the page itself has no creation form. Its task list supports text search and one All, Enabled, and Inactive status filter row; search matches the stored task name, the instruction, and the internal Session id, not a resolved Session title, and the empty state keeps the heading's New action beside the notice. Selecting an active or inactive task opens a second detail level with Rules and Records tabs. Every list row shows the task's stored title with its status, frequency, and next run and never the original Session name; the detail's name control shows the stored title. Rules contains the complete reminder, status, next target for active tasks, original recurrence rule, and task id. Daily rule labels show the saved local time and exact stored IANA zone, preserving nonzero seconds and milliseconds rather than substituting the browser's zone. The Linked session entry in the detail tab strip names the linked Session by its catalog title, falling back to the Session id when no title is available. When the strip cannot fit the tab list beside the labelled entry, the entry becomes an icon button with the same button style as More task actions, with the Session chat icon and a Linked session tooltip that also shows while the entry is disabled; the entry keeps focus when it switches layouts; the tab list scrolls without a visible scrollbar. It opens an available, unarchived Session after metadata loads; pending Session or Workspace metadata, an archived Session, missing Session membership, or a Workspace query error keeps it disabled with an explanation. The click rechecks availability; browsing does not unarchive the Session. Only this action requests Session activation. Deletion is available from the More task actions menu in the detail's top strip, separate from navigation.
The shipped Web composition mounts `ui-schedule` with the Host Schedule service. The clock context that supplies the model's current time is preset-owned: `standard`, `cordis`, and `ptc` declare it, and `minimal` declares neither it nor the reminder tools. The Automation tasks sidebar entry opens a global page without selecting or activating a Session. The page header's New action starts a new Session, where the reminder is created by asking the model; the page itself has no creation form. Its task list supports text search and one All, Enabled, and Inactive status filter row; search matches the stored task name, the instruction, and the internal Session id, not a resolved Session title, and the empty state shows a decorative document-and-clock illustration above its notice and New action, with colors that follow the light or dark theme. Selecting an active or inactive task opens a second detail level with Rules and Records tabs. Every list row shows the task's stored title with its status, frequency, and next run and never the original Session name; the detail's name control shows the stored title. Rules contains the complete reminder, status, next target for active tasks, original recurrence rule, and task id. Daily rule labels show the saved local time and exact stored IANA zone, preserving nonzero seconds and milliseconds rather than substituting the browser's zone. The Linked session entry in the detail tab strip names the linked Session by its catalog title, falling back to the Session id when no title is available. When the strip cannot fit the tab list beside the labelled entry, the entry becomes an icon button with the same button style as More task actions, with the Session chat icon and a Linked session tooltip that also shows while the entry is disabled; the entry keeps focus when it switches layouts; the tab list scrolls without a visible scrollbar. It opens an available, unarchived Session after metadata loads; pending Session or Workspace metadata, an archived Session, missing Session membership, or a Workspace query error keeps it disabled with an explanation. The click rechecks availability; browsing does not unarchive the Session. Only this action requests Session activation. Deletion is available from the More task actions menu in the detail's top strip, separate from navigation.
The page header keeps the title at its natural width and moves New to another line when the two no longer fit together. The action label stays on one line.
+1 -1
View File
@@ -25,7 +25,7 @@ kind: "package-reference"
<a id="use-this-package"></a>
## 使用本包
随发行版交付的 Web 组合把 `ui-schedule` 与 Host Schedule 服务一起挂载。为模型提供当前时间的时钟上下文归 preset 所有:`standard`、`cordis` 与 `ptc` 声明它,`minimal` 既不声明它,也不声明提醒工具。侧栏的自动化任务入口打开全局页面,不选择或激活会话。页面页头的“新建”操作会打开一个新会话,提醒由模型在该会话中创建;页面本身没有创建表单。列表支持文本搜索和一行“全部/已开启/未运行”状态筛选;搜索匹配任务已存储的名称、指令和内部会话 id,不匹配解析出的会话标题,空状态在提示旁保留页头的“新建”操作。选择活动或未运行任务后,进入含“规则”和“任务运行记录”页签的第二层详情。每条列表行显示任务已存储的标题及其状态、频率和下次运行,不显示原会话名称;详情中的名称输入框显示任务已存储的标题。“规则”显示完整提醒、状态、活动任务的下一目标时间、原始重复规则和任务 id。每天的规则标签显示已保存的本地时间与精确的存储 IANA 时区,保留非零秒与毫秒,而不替换为浏览器时区。详情页签条内的“关联会话”入口按会话标题显示,取不到标题时回退显示会话 id。页签条放不下页签列表和带文字的入口时,入口收为与“更多任务操作”同样式的图标按钮,使用会话对话图标,悬停提示“关联会话”,入口禁用时也会显示;切换形态时保留焦点;页签列表可滚动但不显示滚动条。元数据加载后,它可打开可用且未归档的会话;会话或 Workspace 元数据尚未就绪、会话已归档、会话不在列表中或 Workspace 查询失败时,入口保持禁用并显示原因。点击时会重新检查可用性;浏览不会取消会话归档。仅此操作请求激活会话。删除位于详情顶部条带的“更多任务操作”菜单。
随发行版交付的 Web 组合把 `ui-schedule` 与 Host Schedule 服务一起挂载。为模型提供当前时间的时钟上下文归 preset 所有:`standard`、`cordis` 与 `ptc` 声明它,`minimal` 既不声明它,也不声明提醒工具。侧栏的自动化任务入口打开全局页面,不选择或激活会话。页面页头的“新建”操作会打开一个新会话,提醒由模型在该会话中创建;页面本身没有创建表单。列表支持文本搜索和一行“全部/已开启/未运行”状态筛选;搜索匹配任务已存储的名称、指令和内部会话 id,不匹配解析出的会话标题,空状态在提示和“新建”操作上方显示装饰性的文档与时钟插画,配色跟随浅色或深色主题。选择活动或未运行任务后,进入含“规则”和“任务运行记录”页签的第二层详情。每条列表行显示任务已存储的标题及其状态、频率和下次运行,不显示原会话名称;详情中的名称输入框显示任务已存储的标题。“规则”显示完整提醒、状态、活动任务的下一目标时间、原始重复规则和任务 id。每天的规则标签显示已保存的本地时间与精确的存储 IANA 时区,保留非零秒与毫秒,而不替换为浏览器时区。详情页签条内的“关联会话”入口按会话标题显示,取不到标题时回退显示会话 id。页签条放不下页签列表和带文字的入口时,入口收为与“更多任务操作”同样式的图标按钮,使用会话对话图标,悬停提示“关联会话”,入口禁用时也会显示;切换形态时保留焦点;页签列表可滚动但不显示滚动条。元数据加载后,它可打开可用且未归档的会话;会话或 Workspace 元数据尚未就绪、会话已归档、会话不在列表中或 Workspace 查询失败时,入口保持禁用并显示原因。点击时会重新检查可用性;浏览不会取消会话归档。仅此操作请求激活会话。删除位于详情顶部条带的“更多任务操作”菜单。
页头按标题的自然宽度排布;标题和“新建”无法并排容纳时,将按钮移到下一行。操作文案保持单行。
@@ -0,0 +1,37 @@
/** Decorative document-and-clock artwork with theme-owned colors. */
interface ScheduleEmptyIllustrationProps {
readonly className?: string | undefined
}
/**
* Render the empty-state illustration without adding an accessible announcement.
* @param props - optional CSS class applied to the root SVG.
* @returns the decorative inline SVG element.
*/
export function ScheduleEmptyIllustration({ className }: ScheduleEmptyIllustrationProps) {
return (
<svg className={className} width="170" height="170" viewBox="0 0 170 170" fill="none" aria-hidden="true" focusable="false">
<path d="M163.819 97.9382C163.819 126.572 144.48 135.329 115.847 135.329C87.2129 135.329 60.1276 126.572 60.1276 97.9382C60.1276 69.3047 101.003 46.0927 129.637 46.0927C158.27 46.0927 163.819 69.3047 163.819 97.9382Z" fill="var(--dsw-alias-illustration-background)" />
<path d="M9.49521 88.6735C20.8278 111.218 39.5801 110.429 62.2137 99.0519C84.8474 87.6748 102.791 70.018 91.4589 47.4731C80.1263 24.9282 38.6291 22.8935 15.9955 34.2706C-6.63818 45.6478 -1.83735 66.1286 9.49521 88.6735Z" fill="var(--dsw-alias-illustration-background)" />
<path d="M25.2636 34.442C24.727 29.8132 28.0445 25.6258 32.6733 25.0893L101.373 17.126C106.002 16.5895 110.189 19.9069 110.726 24.5358L120.107 105.466C120.643 110.095 117.326 114.282 112.697 114.818L43.9972 122.782C39.3684 123.318 35.181 120.001 34.6445 115.372L25.2636 34.442Z" fill="var(--dsw-alias-illustration-backing)" />
<path d="M40.169 25.1368C40.169 20.477 43.9466 16.6995 48.6064 16.6995H117.766C122.426 16.6995 126.204 20.477 126.204 25.1368V106.609C126.204 111.268 122.426 115.046 117.766 115.046H48.6064C43.9466 115.046 40.169 111.268 40.169 106.609V25.1368Z" fill="var(--dsw-alias-illustration-surface)" />
<path d="M110.258 44.9883V50.3516H72.0831V44.9883H110.258Z" fill="var(--dsw-alias-illustration-detail)" />
<path d="M110.258 63.2803V68.6436H72.0831V63.2803H110.258Z" fill="var(--dsw-alias-illustration-detail)" />
<path d="M91.171 81.5713V86.9346H72.0831V81.5713H91.171Z" fill="var(--dsw-alias-illustration-detail)" />
<path d="M56.1124 43.4998H64.0964V51.4838H56.1124V43.4998Z" fill="var(--dsw-alias-illustration-detail)" />
<path d="M56.1124 61.97H64.0964V69.954H56.1124V61.97Z" fill="var(--dsw-alias-illustration-detail)" />
<path d="M56.1124 80.2607H64.0964V88.2448H56.1124V80.2607Z" fill="var(--dsw-alias-illustration-detail)" />
<path d="M126.205 148.002C126.205 149.615 105.321 150.923 79.5592 150.923C53.7973 150.923 32.9132 149.615 32.9132 148.002C32.9132 146.388 53.7973 145.08 79.5592 145.08C105.321 145.08 126.205 146.388 126.205 148.002Z" fill="var(--dsw-alias-illustration-shadow)" />
<path d="M13.1544 119.318C14.7101 119.318 16.2657 119.43 17.8214 119.656C17.9942 119.682 18.1671 119.709 18.3399 119.737V120.238C18.1671 120.266 17.9942 120.293 17.8214 120.319C16.2657 120.545 14.7101 120.657 13.1544 120.657C11.5986 120.657 10.0423 120.545 8.48643 120.319C8.31357 120.293 8.14073 120.266 7.96787 120.238V119.737C8.14073 119.709 8.31357 119.682 8.48643 119.656C10.0423 119.43 11.5986 119.318 13.1544 119.318Z" fill="var(--dsw-alias-illustration-detail)" />
<path d="M125.197 2.96338C126.287 2.96338 127.378 3.076 128.468 3.30225C128.589 3.32738 128.71 3.3544 128.831 3.38232V3.88428C128.71 3.9122 128.589 3.93922 128.468 3.96436C127.378 4.1906 126.287 4.30322 125.197 4.30322C124.108 4.30319 123.018 4.19057 121.928 3.96436C121.807 3.93922 121.686 3.91221 121.565 3.88428V3.38232C121.686 3.3544 121.807 3.32738 121.928 3.30225C123.018 3.07603 124.108 2.96341 125.197 2.96338Z" fill="var(--dsw-alias-illustration-detail)" />
<path d="M145.187 32.6399C146.742 32.6399 148.298 32.7525 149.854 32.9788C150.026 33.0039 150.199 33.0309 150.372 33.0588V33.5608C150.199 33.5887 150.026 33.6157 149.854 33.6409C148.298 33.8671 146.742 33.9797 145.187 33.9797C143.631 33.9797 142.074 33.8671 140.519 33.6409C140.346 33.6157 140.173 33.5887 140 33.5608V33.0588C140.173 33.0309 140.346 33.0039 140.519 32.9788C142.074 32.7525 143.631 32.6399 145.187 32.6399Z" fill="var(--dsw-alias-illustration-detail)" />
<path d="M12.4776 119.987C12.4777 118.431 12.5903 116.875 12.8165 115.32C12.8416 115.147 12.8687 114.974 12.8966 114.801H13.3985C13.4265 114.974 13.4535 115.147 13.4786 115.32C13.7048 116.875 13.8175 118.431 13.8175 119.987C13.8175 121.542 13.7049 123.099 13.4786 124.654C13.4535 124.827 13.4265 125 13.3985 125.173H12.8966C12.8687 125 12.8416 124.827 12.8165 124.654C12.5903 123.099 12.4776 121.542 12.4776 119.987Z" fill="var(--dsw-alias-illustration-detail)" />
<path d="M124.522 3.63379C124.522 2.54381 124.634 1.45326 124.86 0.363281C124.886 0.242199 124.913 0.121083 124.941 2.19411e-08L125.442 0C125.47 0.121082 125.497 0.242199 125.523 0.363281C125.749 1.45326 125.861 2.54381 125.861 3.63379C125.861 4.72362 125.749 5.81349 125.523 6.90332C125.497 7.02442 125.47 7.1455 125.442 7.2666H124.941C124.913 7.1455 124.886 7.02442 124.86 6.90332C124.634 5.81349 124.522 4.72362 124.522 3.63379Z" fill="var(--dsw-alias-illustration-detail)" />
<path d="M144.509 33.3101C144.509 31.7544 144.622 30.1987 144.848 28.6431C144.873 28.4702 144.9 28.2974 144.928 28.1245H145.43C145.458 28.2974 145.485 28.4702 145.51 28.6431C145.736 30.1987 145.849 31.7544 145.849 33.3101C145.849 34.8659 145.736 36.4222 145.51 37.978C145.485 38.1509 145.458 38.3237 145.43 38.4966H144.928C144.9 38.3237 144.873 38.1509 144.848 37.978C144.622 36.4222 144.509 34.8659 144.509 33.3101Z" fill="var(--dsw-alias-illustration-detail)" />
<path d="M149.05 98.8527C149.05 112.66 137.857 123.853 124.05 123.853C110.242 123.853 99.0496 112.66 99.0496 98.8527C99.0496 85.0455 110.242 73.8527 124.05 73.8527C137.857 73.8527 149.05 85.0455 149.05 98.8527Z" fill="var(--dsw-alias-illustration-detail)" />
<path d="M144.192 98.8529C144.192 109.978 135.174 118.997 124.049 118.997C112.923 118.997 103.905 109.978 103.905 98.8529C103.905 87.7278 112.923 78.709 124.049 78.709C135.174 78.709 144.192 87.7278 144.192 98.8529Z" fill="var(--dsw-alias-illustration-inset)" />
<path d="M125.124 86.7281L125.123 99.2824L134.923 104.132L132.824 108.375L122.76 103.395C121.309 102.677 120.39 101.197 120.39 99.5782L120.389 86.7275L125.124 86.7281Z" fill="var(--dsw-alias-illustration-detail)" />
</svg>
)
}
@@ -388,6 +388,14 @@
color: var(--dsw-alias-label-caption);
}
.emptyIllustration {
flex: none;
width: 110px;
max-width: 100%;
height: auto;
margin-bottom: 12px;
}
/* The mock's `.empty h3 + .btn`: the empty state's action keeps the mock's 16px
above. */
.emptyAction {
@@ -13,6 +13,7 @@ import { TaskDetail, useTaskDetail, type TaskDetailInjected } from './TaskDetail
import { useRelativeClock } from './relative-clock.ts'
import { formatScheduleFrequency, nextRunParts, taskName, zoneLabel } from './schedule-format.ts'
import type { FrequencyZone } from './schedule-format.ts'
import { ScheduleEmptyIllustration } from './ScheduleEmptyIllustration.tsx'
import css from './TaskManagerPage.module.css'
/** Injected catalog and task actions for the management page. */
@@ -151,7 +152,7 @@ export function TaskManagerPage(props: TaskManagerPageProps) {
<div className={css.list}>
{selected === undefined && <CatalogFeedback status={status} populated={rows.length > 0} onRetry={onRetry} t={t} />}
{status === 'ready' && rows.length === 0 && <div className={css.empty} role="status">
<IconClockOutlineRegular size={24} className={css.emptyGlyph} />
<ScheduleEmptyIllustration className={css.emptyIllustration} />
<h2>{t(emptyTitle)}</h2>
<Button variant="outline" className={css.emptyAction} onClick={onNewTask}>
{t('empty.action')}
@@ -891,6 +891,24 @@ describe('Task manager catalog', () => {
expect(screen.getByRole('button', { name: en['statusFilter.all'] }).getAttribute('aria-pressed')).toBe('true')
})
it('renders a decorative document-and-clock illustration in an empty catalog', () => {
mount({ records: [], status: 'ready' })
const illustration = screen.getByRole('status').querySelector('svg')
expect(illustration?.getAttribute('aria-hidden')).toBe('true')
expect(illustration?.getAttribute('focusable')).toBe('false')
expect(illustration?.getAttribute('viewBox')).toBe('0 0 170 170')
const paths = illustration?.querySelectorAll('path')
expect(paths).toHaveLength(20)
// Roles follow the approved artwork's paint order, including overlapping paper and clock layers.
expect(paths?.[0]?.getAttribute('fill')).toBe('var(--dsw-alias-illustration-background)')
expect(paths?.[2]?.getAttribute('fill')).toBe('var(--dsw-alias-illustration-backing)')
expect(paths?.[3]?.getAttribute('fill')).toBe('var(--dsw-alias-illustration-surface)')
expect(paths?.[10]?.getAttribute('fill')).toBe('var(--dsw-alias-illustration-shadow)')
expect(paths?.[17]?.getAttribute('fill')).toBe('var(--dsw-alias-illustration-detail)')
expect(paths?.[18]?.getAttribute('fill')).toBe('var(--dsw-alias-illustration-inset)')
expect(emptyNewTaskButton().textContent).toBe(en['empty.action'])
})
it('distinguishes an empty catalog, loading, and a failed read', () => {
const h = mount({ records: [], status: 'loading' })
expect(screen.getByRole('status', { name: en['list.loading'] }).textContent).toBe('')
@@ -3761,6 +3779,10 @@ describe('Mock detail metrics shared with the task list', () => {
expect(heading).toMatch(/font-size:\s*14px;/)
expect(heading).toMatch(/font-weight:\s*400;/)
expect(rule('emptyGlyph')).toMatch(/margin-bottom:\s*12px;/)
expect(rule('emptyIllustration')).toMatch(/width:\s*110px;/)
expect(rule('emptyIllustration')).toMatch(/max-width:\s*100%;/)
expect(rule('emptyIllustration')).toMatch(/height:\s*auto;/)
expect(rule('emptyIllustration')).toMatch(/margin-bottom:\s*12px;/)
// The mock's `.empty h3 + .btn`.
expect(rule('emptyAction')).toMatch(/margin-top:\s*16px;/)
})
+2 -2
View File
@@ -33,8 +33,8 @@
en: 3eb9b0e3365b2db3
zh: c73acc55dd897266
/deepseek-ai-dsh-client-ui-theme/understand-the-implementation/stylesheets:
en: 2da5b2af3a965090
zh: 883c91afc6bf105d
en: c47fd726c865cdd3
zh: 7c62b6e5857c29a7
/deepseek-ai-dsh-client-ui-theme/understand-the-implementation/scrollbar-rebinding:
en: bf76e7c34a331595
zh: 5accd0260ef7c61e
+2
View File
@@ -69,6 +69,8 @@ In pointer modality, `html[data-input-modality='pointer'] body :focus-visible:no
Menu icons use `--dsw-alias-menu-icon`: neutral-bluish 800 in light mode and `label-primary-dimmed` in dark mode.
The `--dsw-alias-illustration-*` tokens separate layered decorative artwork backgrounds, foreground surfaces, backing layers, shadows, inset fills, and details. Clock faces use the inset fill. The dark palette keeps foreground surfaces brighter than backing layers and backgrounds; inset fills and shadows share the backing tone.
`base.css` suppresses only the outline of focused elements marked `data-dsh-automatic-focus` by the [primitive focus helper](../ui-primitives/README.md); ordinary keyboard focus styling, borders, shadows, and error states remain intact.
System toasts use `--dsw-alias-toast-bg` and `--dsw-alias-toast-label` for a shared background and text color across callers. Document previews pair `--dsw-alias-bg-document-preview` with `--dsw-alias-label-document-preview` so the backdrop and status text follow the same theme. Tooltip keycaps use `--dsw-alias-tooltip-key-bg`, a lighter fill derived from the tooltip background in each palette. Switch thumbs read `--dsw-alias-switch-thumb`: white in light mode and neutral-bluish 400 in dark mode, so an off switch stays lighter than its track without the glare of pure white.
+2
View File
@@ -69,6 +69,8 @@ Desktop 的固定快捷键 `Cmd`/`Ctrl` + `+`(或 `=`)和 `Cmd`/`Ctrl` + `-`
菜单图标使用 `--dsw-alias-menu-icon`:浅色模式为 neutral-bluish 800,深色模式为 `label-primary-dimmed`。
`--dsw-alias-illustration-*` token 分别定义分层装饰插画的背景、前层表面、后层、投影、内嵌填充与细节配色。表盘使用内嵌填充。深色配色让前层表面比后层和背景更亮;内嵌填充与投影沿用后层色调。
`base.css` 仅抑制[基础控件焦点工具](../ui-primitives/README.zh.md)通过 `data-dsh-automatic-focus` 标记的聚焦元素外轮廓线;正常键盘焦点样式、边框、阴影及错误状态保持不变。
系统提示使用 `--dsw-alias-toast-bg` 和 `--dsw-alias-toast-label`,在各调用方之间统一背景与文字颜色。文档预览配对使用 `--dsw-alias-bg-document-preview` 与 `--dsw-alias-label-document-preview`,使底色与状态文字遵循相同主题。Tooltip 键帽使用 `--dsw-alias-tooltip-key-bg`,由各主题的 tooltip 背景派生稍浅的填充。开关滑块读取 `--dsw-alias-switch-thumb`:浅色模式为纯白,深色模式为 neutral-bluish 400,使关闭态滑块比轨道更亮,同时避免纯白在暗色下过亮。
@@ -59,9 +59,11 @@ body {
--dsw-static-neutral-bluish-150: rgb(233, 236, 242);
--dsw-static-neutral-bluish-200: rgb(225, 229, 238);
--dsw-static-neutral-bluish-300: rgb(207, 211, 214);
--dsw-static-neutral-bluish-310: rgb(202, 210, 219);
--dsw-static-neutral-bluish-400: rgb(173, 178, 184);
--dsw-static-neutral-bluish-500: rgb(151, 157, 166);
--dsw-static-neutral-bluish-50: rgb(249, 250, 251);
--dsw-static-neutral-bluish-55: rgb(245, 247, 250);
--dsw-static-neutral-bluish-600: rgb(129, 133, 140);
--dsw-static-neutral-bluish-60: rgb(245, 246, 247);
--dsw-static-neutral-bluish-700: rgb(97, 102, 107);
@@ -71,6 +73,7 @@ body {
--dsw-static-neutral-bluish-850: rgb(44, 44, 46);
--dsw-static-neutral-bluish-875: rgb(35, 35, 36);
--dsw-static-neutral-bluish-900: rgb(27, 27, 28);
--dsw-static-neutral-bluish-90: rgb(237, 240, 244);
--dsw-static-neutral-bluish-950: rgb(21, 21, 23);
--dsw-static-red-100: rgb(254, 226, 226);
--dsw-static-red-400: rgb(242, 90, 90);
@@ -140,9 +143,11 @@ body[data-ds-dark-theme] {
--dsw-static-neutral-bluish-150: rgb(233, 236, 242);
--dsw-static-neutral-bluish-200: rgb(225, 229, 238);
--dsw-static-neutral-bluish-300: rgb(207, 211, 214);
--dsw-static-neutral-bluish-310: rgb(202, 210, 219);
--dsw-static-neutral-bluish-400: rgb(173, 178, 184);
--dsw-static-neutral-bluish-500: rgb(151, 157, 166);
--dsw-static-neutral-bluish-50: rgb(249, 250, 251);
--dsw-static-neutral-bluish-55: rgb(245, 247, 250);
--dsw-static-neutral-bluish-600: rgb(129, 133, 140);
--dsw-static-neutral-bluish-60: rgb(249, 250, 251);
--dsw-static-neutral-bluish-700: rgb(97, 102, 107);
@@ -152,6 +157,7 @@ body[data-ds-dark-theme] {
--dsw-static-neutral-bluish-850: rgb(44, 44, 46);
--dsw-static-neutral-bluish-875: rgb(35, 35, 36);
--dsw-static-neutral-bluish-900: rgb(27, 27, 28);
--dsw-static-neutral-bluish-90: rgb(237, 240, 244);
--dsw-static-neutral-bluish-950: rgb(21, 21, 23);
--dsw-static-red-100: rgb(254, 226, 226);
--dsw-static-red-400: rgb(242, 90, 90);
@@ -206,6 +212,12 @@ body {
--dsw-alias-button-tool-bar-fill-invisible: rgba(31, 31, 31, 0.36);
--dsw-alias-button-tool-bar-fill: rgba(84, 85, 87, 0.5);
--dsw-alias-button-tool-bar-hover: rgba(84, 85, 87, 0.6);
--dsw-alias-illustration-background: var(--dsw-static-neutral-bluish-55);
--dsw-alias-illustration-backing: var(--dsw-static-neutral-bluish-310);
--dsw-alias-illustration-detail: var(--dsw-static-neutral-bluish-310);
--dsw-alias-illustration-inset: var(--dsw-alias-illustration-surface);
--dsw-alias-illustration-shadow: var(--dsw-alias-illustration-surface);
--dsw-alias-illustration-surface: var(--dsw-static-neutral-bluish-90);
--dsw-alias-interactive-bg-active: rgba(38, 49, 72, 0.1);
--dsw-alias-interactive-bg-hover-accent: rgba(38, 49, 72, 0.14);
--dsw-alias-interactive-bg-hover-danger: rgba(236, 19, 19, 0.05);
@@ -324,6 +336,12 @@ body[data-ds-dark-theme] {
--dsw-alias-button-tool-bar-fill-invisible: rgba(31, 31, 31, 0.36);
--dsw-alias-button-tool-bar-fill: rgba(84, 85, 87, 0.5);
--dsw-alias-button-tool-bar-hover: rgba(84, 85, 87, 0.6);
--dsw-alias-illustration-background: var(--dsw-static-neutral-bluish-850);
--dsw-alias-illustration-backing: var(--dsw-static-neutral-bluish-750);
--dsw-alias-illustration-detail: var(--dsw-static-neutral-bluish-600);
--dsw-alias-illustration-inset: var(--dsw-alias-illustration-backing);
--dsw-alias-illustration-shadow: var(--dsw-alias-illustration-backing);
--dsw-alias-illustration-surface: var(--dsw-static-neutral-bluish-700);
--dsw-alias-interactive-bg-active: rgba(255, 255, 255, 0.14);
--dsw-alias-interactive-bg-hover-accent: rgba(255, 255, 255, 0.24);
--dsw-alias-interactive-bg-hover-danger: rgba(242, 90, 90, 0.15);
@@ -0,0 +1,140 @@
/** Decorative artwork resolves theme tokens to distinct foreground, backing, and detail colors. */
import { readFileSync } from 'node:fs'
import { resolve } from 'node:path'
import { expect, it } from 'vitest'
import { parseRules, varReferences } from './stylesheet-scan.ts'
const stylesheet = readFileSync(resolve(import.meta.dirname, '../src/styles/design-platform.css'), 'utf8')
const illustration = readFileSync(resolve(import.meta.dirname, '../../ui-schedule/src/client/ScheduleEmptyIllustration.tsx'), 'utf8')
const rules = parseRules(stylesheet)
const PREFIX = '--dsw-alias-illustration-'
const ROLES = ['background', 'backing', 'detail', 'inset', 'shadow', 'surface'] as const
type Rgb = [number, number, number]
function definitions(dark: boolean): Map<string, string> {
const selector = dark ? 'body[data-ds-dark-theme]' : 'body'
return new Map(rules.filter(rule => rule.selectors.includes(selector)).flatMap(rule => rule.declarations))
}
function resolveColor(palette: Map<string, string>, name: string): Rgb {
const seen = new Set<string>()
let current = name
while (palette.has(current) && !seen.has(current)) {
seen.add(current)
const value = palette.get(current)!
const [reference] = varReferences(value)
if (reference !== undefined) {
expect(value, current).toMatch(/^var\(\s*--[\w-]+\s*\)$/)
current = reference
continue
}
const rgb = /^rgb\(\s*(\d+)\s*,\s*(\d+)\s*,\s*(\d+)\s*\)$/.exec(value)
expect(rgb, `${name}: ${current} = ${value}`).not.toBeNull()
return [Number(rgb![1]), Number(rgb![2]), Number(rgb![3])]
}
throw new Error(`Unresolved color ${name}: ${current}`)
}
function illustrationColors(palette: Map<string, string>) {
return Object.fromEntries(ROLES.map(role => [role, resolveColor(palette, `${PREFIX}${role}`)]))
}
function luminance(rgb: Rgb): number {
const linear = rgb.map((channel) => {
const value = channel / 255
return value <= 0.04045 ? value / 12.92 : ((value + 0.055) / 1.055) ** 2.4
})
return linear[0]! * 0.2126 + linear[1]! * 0.7152 + linear[2]! * 0.0722
}
function expectDarkLayers(palette: Map<string, string>): void {
const color = (role: string): Rgb => resolveColor(palette, `${PREFIX}${role}`)
expect(luminance(color('surface'))).toBeGreaterThan(luminance(color('backing')))
expect(luminance(color('backing'))).toBeGreaterThan(luminance(color('background')))
for (const role of ['surface', 'backing', 'background']) expect(color('detail')).not.toEqual(color(role))
expect(color('inset')).toEqual(color('backing'))
expect(color('shadow')).toEqual(color('backing'))
}
function expectConsumerColors(source: string, palette: Map<string, string>): void {
const consumed = new Set(varReferences(source))
const defined = [...palette.keys()].filter(name => name.startsWith(PREFIX)).sort()
expect([...consumed].sort()).toEqual(defined)
expect(consumed.size).toBe(ROLES.length)
for (const name of consumed) resolveColor(palette, name)
}
const lightColors = {
background: [245, 247, 250],
backing: [202, 210, 219],
detail: [202, 210, 219],
inset: [237, 240, 244],
shadow: [237, 240, 244],
surface: [237, 240, 244],
}
it('preserves the original light artwork colors, including surface-colored insets and shadows', () => {
expect(illustrationColors(definitions(false))).toEqual(lightColors)
})
it.each([false, true])('defines the approved light static steps in the dark=%s palette', (dark) => {
const palette = definitions(dark)
expect(resolveColor(palette, '--dsw-static-neutral-bluish-55')).toEqual([245, 247, 250])
expect(resolveColor(palette, '--dsw-static-neutral-bluish-90')).toEqual([237, 240, 244])
expect(resolveColor(palette, '--dsw-static-neutral-bluish-310')).toEqual([202, 210, 219])
})
it('resolves the dark artwork colors through the actual static palette', () => {
expect(illustrationColors(definitions(true))).toEqual({
background: [44, 44, 46],
backing: [67, 69, 74],
detail: [129, 133, 140],
inset: [67, 69, 74],
shadow: [67, 69, 74],
surface: [97, 102, 107],
})
})
it('keeps dark surfaces brighter than backing and background, with distinct details and backing-colored insets and shadows', () => {
expectDarkLayers(definitions(true))
})
it.each([false, true])('resolves the illustration consumer names in the dark=%s palette', (dark) => {
expectConsumerColors(illustration, definitions(dark))
})
it('rejects light static-color drift even when alias names stay unchanged', () => {
const palette = definitions(false)
palette.set('--dsw-static-neutral-bluish-90', 'rgb(235, 238, 242)')
expect(() => { expect(illustrationColors(palette)).toEqual(lightColors) }).toThrow()
})
it.each([
['surface', 'backing'],
['backing', 'surface'],
['background', 'backing'],
['detail', 'surface'],
['inset', 'surface'],
['shadow', 'surface'],
])('rejects a dark %s role changed to %s', (role, target) => {
const palette = definitions(true)
palette.set(`${PREFIX}${role}`, `var(${PREFIX}${target})`)
expect(() => { expectDarkLayers(palette) }).toThrow()
})
it.each([false, true])('rejects renamed consumers and missing definitions in the dark=%s palette', (dark) => {
const palette = definitions(dark)
const renamed = illustration.replaceAll(`${PREFIX}inset`, `${PREFIX}clock-face`)
expect(() => { expectConsumerColors(renamed, palette) }).toThrow()
const misspelled = illustration.replace(PREFIX, '--dsw-alias-illustation-')
expect(() => { expectConsumerColors(misspelled, palette) }).toThrow()
palette.delete(`${PREFIX}inset`)
expect(() => { expectConsumerColors(illustration, palette) }).toThrow()
})
it.each(['--dsw-static-missing', `${PREFIX}inset`])('rejects an unresolved or cyclic inset reference to %s', (target) => {
const palette = definitions(true)
palette.set(`${PREFIX}inset`, `var(${target})`)
expect(() => resolveColor(palette, `${PREFIX}inset`)).toThrow('Unresolved color')
})
+4 -4
View File
@@ -9,8 +9,8 @@
en: 77ea90624a004773
zh: fe6dbf8414a5aff0
/packages-computer-use/summary:
en: 3820da56a601d3f1
zh: 358e65f8a7cf7446
en: 7e5fcf9bb48c9677
zh: abee2be288115045
/packages-computer-use/table-of-contents:
en: 1eb35b015a6c423f
zh: a8a3dceee2c9e943
@@ -18,8 +18,8 @@
en: 9aa86e6451f60c3d
zh: c5e6816e90f2d74b
/packages-computer-use/related-documentation:
en: ad08d3417a0d4630
zh: cfe8e03033c2fede
en: 58198f9f17e6120f
zh: 188fa0ec5c4678f3
/packages-computer-use/dev-note:
en: b75c8458928b72b9
zh: 003ba4e17b4b5269
+3 -2
View File
@@ -9,7 +9,7 @@ English | [中文](README.zh.md)
## Summary
Computer-use providers let models observe and operate a desktop. This group owns exclusive provider registration. Each provider owns its operations, tools, and platform requirements; the experimental Cua Driver providers live in the experimental group.
Computer-use providers let models observe and operate a desktop. This group contains the shared registry that prevents multiple providers from being enabled in one composition. DSH's [CUA implementation](../experimental/cua/README.md), platform providers, and opt-in bundle currently live in `experimental`, alongside the Cua Driver adapters. The CUA package documents the platform requirements for promotion to this group.
## Table of Contents
@@ -32,7 +32,8 @@ Choose one provider and mount the shared registration service.
## Related documentation
- [Computer use](../../docs/subsystems/computer-use.md) — capability ownership and provider choices.
- [Experimental packages](../experimental/README.md) — Cua Driver MCP and native providers.
- [Computer Use bundle](../experimental/cua-bundle/README.md) — enable native CUA and access its permission controls.
- [Experimental packages](../experimental/README.md) — native CUA packages and Cua Driver adapters.
<a id="dev-note"></a>
## Dev Note
+3 -2
View File
@@ -9,7 +9,7 @@ kind: "package-group"
## 概述
计算机操作提供方让模型观察并操作桌面。本组负责提供方的独占注册。各提供方拥有自己的操作、工具和平台要求;实验性 Cua Driver 提供方位于 experimental 组。
计算机操作提供方让模型观察并操作桌面。本组包含共享注册器,防止同一组合同时启用多个提供方。DSH 的 [CUA 实现](../experimental/cua/README.zh.md)、平台提供方和启用组合包目前位于 `experimental`,Cua Driver 适配包也位于该组。CUA 包文档说明了迁入本组所需的平台支持条件。
## 目录
@@ -32,7 +32,8 @@ kind: "package-group"
## 相关文档
- [计算机操作](../../docs/subsystems/computer-use.zh.md) — 能力归属和提供方选择。
- [实验性包](../experimental/README.zh.md) — Cua Driver MCP 和原生提供方。
- [Computer Use 组合包](../experimental/cua-bundle/README.zh.md):启用原生 CUA 并使用权限管理功能。
- [实验性包](../experimental/README.zh.md):原生 CUA 包和 Cua Driver 适配包。
<a id="dev-note"></a>
## 开发备注
@@ -9,8 +9,8 @@
en: 2efee0c5f0d577bc
zh: fef4b81bfc121052
/deepseek-ai-dsh-computer-use/summary:
en: 4de047849e68701f
zh: 90a32c0e279b4b05
en: 4cb331c550852b23
zh: d638995ab250397b
/deepseek-ai-dsh-computer-use/table-of-contents:
en: d152484eb41ac6b4
zh: 405b30feaf48c087
@@ -21,8 +21,8 @@
en: 181679887837591a
zh: cc2e96766ce0aa63
/deepseek-ai-dsh-computer-use/further-exploration:
en: b1bdbcbc150036b3
zh: 1d356e0522587abc
en: 987295d677466c04
zh: 9d7b40a04786490e
/deepseek-ai-dsh-computer-use/model-experience:
en: 8780c250dfaefaaa
zh: 1c4b3598a131fb65
+4 -1
View File
@@ -9,7 +9,7 @@ English | [中文](README.zh.md)
## Summary
A deployment can enable one computer-use provider at a time. Loading another provider fails with the registered provider name. Each provider supplies its own tools and desktop operations. This package adds no model-visible tools and does not coordinate concurrent Sessions.
A deployment can enable one computer-use provider at a time. Loading another provider fails with the registered provider name. Both DSH's native CUA and the Cua Driver adapters use this registry. Each provider supplies its own tools and desktop operations. This package adds no model-visible tools and does not coordinate concurrent Sessions.
## Table of Contents
@@ -53,6 +53,9 @@ One private name owns the slot. Cordis effects remove contributions when their p
## Further Exploration
- [Computer use](../../../docs/subsystems/computer-use.md) — provider selection and shared-desktop limits.
- [Computer Use bundle](../../experimental/cua-bundle/README.md) — enable DSH's native CUA and permission controls.
- [Shared CUA library](../../experimental/cua/README.md) — application/window objects, tool execution, and application consent.
- [macOS CUA provider](../../experimental/cua-macos/README.md) — DSH's Swift implementation and native helper lifecycle.
- [Cua Driver MCP provider](../../experimental/computer-use-cua-driver-mcp/README.md) — use an installed driver.
- [Cua Driver native provider](../../experimental/computer-use-cua-driver-native/README.md) — use the npm runtime.
@@ -9,7 +9,7 @@ kind: "package-reference"
## 概述
部署每次可启用一个计算机操作提供方。加载另一个提供方时会失败,并报告已注册的提供方名称。各提供方提供自己的工具和桌面操作。本包不添加模型可见工具,也不协调并发 Session。
部署每次可启用一个计算机操作提供方。加载另一个提供方时会失败,并报告已注册的提供方名称。DSH 原生 CUA 和 Cua Driver 适配包都使用此注册器。各提供方提供自己的工具和桌面操作。本包不添加模型可见工具,也不协调并发 Session。
## 目录
@@ -53,6 +53,9 @@ kind: "package-reference"
## 延伸阅读
- [计算机操作](../../../docs/subsystems/computer-use.zh.md) — 提供方选择和共享桌面限制。
- [Computer Use 组合包](../../experimental/cua-bundle/README.zh.md):启用 DSH 原生 CUA 和权限管理功能。
- [共享 CUA 库](../../experimental/cua/README.zh.md):应用与窗口对象、工具执行和应用授权。
- [macOS CUA 提供方](../../experimental/cua-macos/README.zh.md):DSH 的 Swift 实现和原生辅助进程生命周期。
- [Cua Driver MCP 提供方](../../experimental/computer-use-cua-driver-mcp/README.zh.md) — 使用已安装的驱动。
- [Cua Driver 原生提供方](../../experimental/computer-use-cua-driver-native/README.zh.md) — 使用 npm 运行时。
+2 -2
View File
@@ -18,8 +18,8 @@
en: b55aeb927737b90d
zh: 0b215fcb5c2c7908
/packages-experimental/packages:
en: 372b182dbf0f662b
zh: 7a8031c506394785
en: b16e94185c9b8b24
zh: 784faa79e027cd14
/packages-experimental/related-documentation:
en: 1cb9d3448d44e48d
zh: cd326d70a746b896
+2 -1
View File
@@ -69,8 +69,9 @@ Promotion requires a named product role, documented public behavior and limitati
| [`claude-code-mods`](claude-code-mods/README.md) | Run Claude Code mods as plugins: their hook chains on harness extension points and a band above the prompt | `ctx.claudeCodeMods` |
| [`client-ui-claude-code-mods`](client-ui-claude-code-mods/README.md) | The Web band that draws a mod's tree above the prompt and sends button clicks back | — |
| [`ptc-runtime-python`](ptc-runtime-python/README.md) | CPython subprocess backend for the PTC execution seam | `ctx.ptcRuntime` |
| [CUA](cua/README.md) | Shared application/window objects, consent, and logged execution. | Library |
| [CUA](cua/README.md) | Application/window objects, authorized execution, and permission controls. | `ctx.cuaAccess` |
| [macOS CUA](cua-macos/README.md) | Swift native execution and platform preparation. | `ctx.computerUse` |
| [`cua-bundle`](cua-bundle/README.md) | Optional Computer Use switch for PTC sessions | — |
| [`computer-use-cua-driver-mcp`](computer-use-cua-driver-mcp/README.md) | Use an installed Cua Driver through MCP | `ctx.computerUse` |
| [`computer-use-cua-driver-native`](computer-use-cua-driver-native/README.md) | Embed the Cua Driver native npm runtime | `ctx.computerUse` |
| [`browser-use-playwright-mcp`](browser-use-playwright-mcp/README.md) | Playwright browser tools over MCP | `ctx.browserUse` |
+2 -1
View File
@@ -69,8 +69,9 @@ kind: "package-group"
| [`claude-code-mods`](claude-code-mods/README.zh.md) | 把 Claude Code 模组作为插件运行:钩子链落在 harness 扩展点上,并在提示框上方绘制横幅 | `ctx.claudeCodeMods` |
| [`client-ui-claude-code-mods`](client-ui-claude-code-mods/README.zh.md) | 在提示框上方绘制模组树并把按钮点击发回的 Web 横幅 | — |
| [`ptc-runtime-python`](ptc-runtime-python/README.zh.md) | PTC 执行 seam 的 CPython 子进程后端 | `ctx.ptcRuntime` |
| [CUA](cua/README.zh.md) | 共享应用与窗口对象、授权和已记录的执行。 | 库 |
| [CUA](cua/README.zh.md) | 应用与窗口对象、授权执行和权限控件。 | `ctx.cuaAccess` |
| [macOS CUA](cua-macos/README.zh.md) | Swift 原生执行与平台组件准备。 | `ctx.computerUse` |
| [`cua-bundle`](cua-bundle/README.zh.md) | 面向 PTC 会话、默认关闭的 Computer Use 开关 | — |
| [`computer-use-cua-driver-mcp`](computer-use-cua-driver-mcp/README.zh.md) | 通过 MCP 使用已安装的 Cua Driver | `ctx.computerUse` |
| [`computer-use-cua-driver-native`](computer-use-cua-driver-native/README.zh.md) | 嵌入 Cua Driver 原生 npm 运行时 | `ctx.computerUse` |
| [`browser-use-playwright-mcp`](browser-use-playwright-mcp/README.zh.md) | 通过 MCP 提供 Playwright 浏览器工具 | `ctx.browserUse` |
@@ -0,0 +1,46 @@
# Bilingual-pair consistency record for README.md (docs/i18n/README.md): per heading
# section, a hash of its English and Chinese blocks outside code blocks and generated regions.
# After editing either side, bring the other along and re-record with:
# pnpm run verify-translation-pairing --write packages/experimental/cua-bundle/README.md
/:
en: cc518dcd97629a96
zh: 465e999011282dc3
/deepseek-ai-dsh-cua-bundle:
en: 9084fb48d0f6489c
zh: ce692972d91cf663
/deepseek-ai-dsh-cua-bundle/summary:
en: 06d8bf4ee1ef6f3b
zh: 8568f649fb846615
/deepseek-ai-dsh-cua-bundle/table-of-contents:
en: b563e4ec4727d0de
zh: d0228da978238e05
/deepseek-ai-dsh-cua-bundle/use-this-package:
en: 32932a00acbfe4b6
zh: 34b6fa6933e4e8c9
/deepseek-ai-dsh-cua-bundle/understand-the-implementation:
en: 057771fe13aaa337
zh: 4fdb2b98ff3e7a78
/deepseek-ai-dsh-cua-bundle/further-exploration:
en: 9f15d56b5b16e1ca
zh: 75eb08e8828c8877
/deepseek-ai-dsh-cua-bundle/model-experience:
en: 215e7ba838619b7b
zh: a311da3843709f90
/deepseek-ai-dsh-cua-bundle/model-experience/computer-use-in-ptc-sessions:
en: e8f11f45b375771e
zh: e8f11f45b375771e
/deepseek-ai-dsh-cua-bundle/model-experience/computer-use-in-ptc-sessions/what-the-model-sees:
en: 591af735d1eb92f6
zh: 3632bffa33c77b28
/deepseek-ai-dsh-cua-bundle/model-experience/computer-use-in-ptc-sessions/token-effect:
en: e8de232e96220fe7
zh: 5c791307e2123667
/deepseek-ai-dsh-cua-bundle/model-experience/computer-use-in-ptc-sessions/kv-cache-effect:
en: 417c40acd93df532
zh: 69cc5cf7140c75a9
/deepseek-ai-dsh-cua-bundle/known-limitations-and-deferred-work:
en: 41f8c54d76d92c95
zh: 44c9662e437f4cbe
/deepseek-ai-dsh-cua-bundle/known-limitations-and-deferred-work/dev-note:
en: 0a1253c6f7b5e8b9
zh: ce96645b3e75874e
@@ -0,0 +1,94 @@
---
description: "Enable optional macOS Computer Use and its permission page."
kind: "package-bundle"
---
# @deepseek-ai/dsh-cua-bundle
English | [中文](README.zh.md)
## Summary
Enable macOS Computer Use from Plugins for existing and new PTC sessions. Windows support is in development. This bundle remains in `experimental` until that support is complete, then moves to the `computer-use` group. It is off by default; disabled profiles prepare no native components and register neither CUA tools nor their model guidance.
## Table of Contents
- [Use this package](#use-this-package)
- [Understand the implementation](#understand-the-implementation)
- [Further Exploration](#further-exploration)
- [Model Experience](#model-experience)
- [Known Limitations and Deferred Work](#known-limitations-and-deferred-work)
-----
<a id="use-this-package"></a>
## Use this package
This workflow targets developers running macOS Web or unpackaged Desktop from source. Open Plugins → Computer Use and enable the switch. The permission page identifies the responsible application reported by macOS when the optional system query is available. Request access and enable that application in System Settings; follow the application named by macOS if it differs from the detected source. For Web, run these commands from the repository root:
```sh
pnpm run build
pnpm dsh web
```
For the source Desktop launcher and its separate development profile, follow [Desktop development](../../../apps/desktop/README.md). Enable Computer Use in each profile that needs it.
The switch saves bundle selection in the profile’s `package.json`. Computer Use is installed only in PTC sessions. Enabling from the plugin list offers setup guidance when system permissions need attention; enabling from its detail page displays the permission controls directly. The detail page offers **Set to PTC** for the new-session default; existing sessions keep their preset. The switch leaves PTC execution mode and other settings unchanged. The detail page’s **Use persistent mode** action separately enables in-memory state across PTC calls; its effects and reload behavior are described in the [permission page](../cua/README.md). Native preparation runs when the provider loads and reuses the source-hashed executable when available.
Disabling unloads the provider and withdraws CUA tools from its Agent scopes. Runtime configuration, the new-session default and saved grants remain. Session-only application decisions end with provider disposal.
Computer Use requires a macOS DSH Host. On other operating systems, installation omits the optional macOS provider and the bundle skips its Loader entry. Shared CUA logic and the permission-page entry remain available, but Computer Use operations require the provider. A browser on another operating system can connect to a supported Mac Host.
If preparation fails, the saved selection remains and CUA operations stay unavailable. Read the Host startup diagnostic, correct its cause, then restart DSH.
Explicit activation runs a read-only permission check, which may start a helper without opening a system dialog. Open the [permission page](../cua/README.md) for system requests, settings navigation and automatic refresh. Tasks ask separately before accessing an application. Saved application decisions use the existing profile configuration; session decisions remain in memory across turns and connections until DSH restarts or Computer Use is disabled. Revoke access on the same page.
-----
<a id="understand-the-implementation"></a>
## Understand the implementation
<details>
<summary>Maintainer details</summary>
The bundle patch mounts the native provider and permission UI with `agentPreset: ptc`. Existing Agent scopes register the tools without changing preset documents. The detail-page action uses the existing settings form to save `agent-preset-registry.selectedDefault: ptc` only when clicked. [CUA](../cua/README.md) owns registration and disposal.
</details>
-----
<a id="further-exploration"></a>
## Further Exploration
[Computer use](../../../docs/subsystems/computer-use.md)
-----
<a id="model-experience"></a>
## Model Experience
### Computer Use in PTC sessions
#### What the model sees
Existing and new PTC sessions receive `cua` through the generated SDK and `run_code`, plus CUA guidance. Disabling withdraws these contributions and keeps PTC presentation. Other presets receive no CUA contributions. [CUA](../cua/README.md) and [PTC](../../ptc-runtime/ptc-runtime-node/README.md#persistent-repl) own the interfaces and output semantics.
#### Token effect
CUA activation adds its guidance. Disabling removes that guidance; the existing PTC execution instructions remain.
#### KV Cache effect
CUA activation and removal change the PTC SDK and system guidance; subsequent requests use the updated prefix. The top-level `run_code` transport remains present.
## Known Limitations and Deferred Work
<a id="known-limitations-and-deferred-work"></a>
- Source Web and Desktop share permission controls on macOS. Packaged Desktop helper distribution, signing, and permission attribution are deferred.
- PTC execution is shared by the profile and is configured independently. A Host without HMR applies saved changes at restart. Saving or revoking application grants while another HMR transaction unloads the provider can block; queued-write cancellation is a separate pending lifecycle fix.
<a id="dev-note"></a>
### Dev Note
None.
@@ -0,0 +1,94 @@
---
description: "开启可选 macOS Computer Use 及权限页面。"
kind: "package-bundle"
---
# @deepseek-ai/dsh-cua-bundle
[English](README.md) | 中文
## 概述
从插件页为已有和新建 PTC 会话开启 macOS Computer Use。Windows 支持正在开发中。本组合包暂放在 `experimental`,待 Windows 支持完成后迁回 `computer-use` 组。默认关闭,关闭时不准备本机组件、不注册 CUA 工具或添加其模型指引。
## 目录
- [使用本包](#use-this-package)
- [理解实现](#understand-the-implementation)
- [进一步探索](#further-exploration)
- [模型体验](#model-experience)
- [已知限制与延后工作](#known-limitations-and-deferred-work)
-----
<a id="use-this-package"></a>
## 使用本包
本流程面向从源码运行 macOS Web 或未打包 Desktop 的开发者。打开“插件 → Computer Use”并启用。权限页通过可选系统查询显示 macOS 报告的责任应用,指引用户申请权限并在系统设置中开启对应开关。若 macOS 显示其他应用名称,以系统提示为准。Web 版在仓库根目录运行以下命令:
```sh
pnpm run build
pnpm dsh web
```
源码 Desktop 的启动方式和独立开发配置档见[桌面开发说明](../../../apps/desktop/README.zh.md)。需要使用 Computer Use 的配置档分别开启该功能。
开关将组合包选择保存到 profile 的 `package.json`。Computer Use 仅安装在 PTC 会话中。从插件列表启用时,系统权限未就绪会显示设置引导;从详情页启用后直接显示权限控件。详情页提供“设为 PTC”按钮来修改新会话默认模式,已有会话保留原 preset。开关保持 PTC 执行方式和其他设置不变。详情页的“设为持久化”按钮单独开启 PTC 跨调用的内存状态保留,作用范围和重载行为见[权限页面](../cua/README.zh.md)。提供方加载时准备本机组件,已有与源码摘要匹配的可执行文件时直接复用。
关闭会卸载提供方,并从其 Agent 作用域中撤回 CUA 工具。执行器配置、新会话默认值和已保存的授权保留。会话级应用授权随提供方卸载而失效。
Computer Use 要求 DSH Host 运行在 macOS 上。其他系统安装时跳过可选的 macOS 提供方,组合包也不加载该提供方。共享 CUA 逻辑和权限页入口仍保留,Computer Use 操作需要提供方才能使用。其他操作系统上的浏览器可以连接受支持的 Mac Host。
准备失败时,已保存的选择保留,CUA 操作不可用。查看 Host 的启动诊断,修复原因后重启 DSH。
显式启用会进行只读权限检查,可能启动 helper,但不会打开系统授权弹窗。[权限页面](../cua/README.zh.md) 提供系统申请入口、设置跳转和自动检测。任务访问应用前会单独询问用户。长期应用授权保存在现有 profile 配置中,会话授权保存在内存中,跨 turn 和连接复用,重启 DSH 或关闭 Computer Use 后失效。可以在同一页面撤销授权。
-----
<a id="understand-the-implementation"></a>
## 理解实现
<details>
<summary>维护者说明</summary>
组合包 patch 通过 `agentPreset: ptc` 挂载原生提供方和权限界面。工具通过现有 Agent 作用域注册,preset 文档保持不变。详情页按钮复用现有设置表单,仅在点击后保存 `agent-preset-registry.selectedDefault: ptc`。[CUA](../cua/README.zh.md)负责注册和销毁。
</details>
-----
<a id="further-exploration"></a>
## 进一步探索
[Computer use](../../../docs/subsystems/computer-use.zh.md)
-----
<a id="model-experience"></a>
## 模型体验
### Computer Use in PTC sessions
#### What the model sees
已有和新建 PTC 会话通过生成的 SDK 与 `run_code` 使用 `cua`,并获得 CUA 指引。关闭撤回这些贡献,保留 PTC 展示方式。其他 preset 不获得 CUA 贡献。完整接口与输出约定由 [CUA](../cua/README.zh.md) 和 [PTC](../../ptc-runtime/ptc-runtime-node/README.zh.md#persistent-repl) 说明。
#### Token effect
CUA 生效时添加其指引。关闭撤回这些指引,现有 PTC 执行说明保留。
#### KV Cache effect
CUA 生效和卸载会改变 PTC SDK 与系统指引,后续请求使用更新后的前缀。顶层 `run_code` 工具保持存在。
## 已知限制与延后工作
<a id="known-limitations-and-deferred-work"></a>
- 源码 Web 与 Desktop 共用 macOS 权限控件。桌面安装包的 helper 分发、签名和权限归属留待后续处理。
- PTC 执行方式由 profile 共享,需单独配置。未启用 HMR 的 Host 在重启时应用已保存的变更。保存或撤销应用授权与另一次卸载提供方的 HMR 同时发生时可能阻塞;排队写入取消另行修复。
<a id="dev-note"></a>
### 开发备注
None.
@@ -0,0 +1,10 @@
- insert:
- id: cua-provider-registry
name: '@deepseek-ai/dsh-computer-use'
- id: cua
name: '@deepseek-ai/dsh-experimental-cua-macos'
disabled: !!js process.platform !== 'darwin'
config:
agentPreset: ptc
- id: ui-cua
name: '@deepseek-ai/dsh-cua'
+12
View File
@@ -0,0 +1,12 @@
<svg width="36" height="36" viewBox="0 0 36 36" fill="none" xmlns="http://www.w3.org/2000/svg">
<circle cx="15.5" cy="15.5" r="8.5" fill="url(#a)" />
<circle cx="15.5" cy="15.5" r="5" fill="#FFFFFF" />
<path d="M13.2 15.5L14.9 17.2L18.2 13.9" stroke="#145AF3" stroke-width="1.8" stroke-linecap="round" stroke-linejoin="round" />
<path d="M22 22L29 29" stroke="#145AF3" stroke-width="3.2" stroke-linecap="round" />
<defs>
<linearGradient id="a" x1="7" y1="7" x2="24" y2="24" gradientUnits="userSpaceOnUse">
<stop stop-color="#7CB7FF" />
<stop offset="1" stop-color="#45D9E7" />
</linearGradient>
</defs>
</svg>

After

Width:  |  Height:  |  Size: 642 B

@@ -0,0 +1,6 @@
{
"meta": {
"title": "Computer Use",
"description": "Currently supports macOS; Windows support is in development. Loads Computer Use for existing and new PTC sessions. Switch to PTC mode and grant Accessibility and Screen Recording permissions to use it."
}
}
@@ -0,0 +1,6 @@
{
"meta": {
"title": "Computer Use",
"description": "目前仅支持 macOS,Windows 支持正在开发中。为已有及新建 PTC 会话加载 Computer Use。请切换到 PTC 模式来使用,并授予辅助功能和屏幕录制权限。"
}
}
@@ -0,0 +1,56 @@
{
"name": "@deepseek-ai/dsh-cua-bundle",
"description": "Optional macOS Computer Use and permission setup",
"version": "0.2.1-rc.2",
"icon": "./icon.svg",
"publishConfig": {
"access": "public"
},
"repository": {
"type": "git",
"url": "git+https://github.com/deepseek-ai/deepseek-harness.git",
"directory": "packages/experimental/cua-bundle"
},
"type": "module",
"main": "lib/index.js",
"types": "lib/types/index.d.ts",
"exports": {
".": {
"types": "./lib/types/index.d.ts",
"default": "./lib/index.js"
},
"./cordis.patch.yml": "./cordis.patch.yml",
"./locale/*.json": "./locale/*.json",
"./src/*": "./src/*",
"./package.json": "./package.json"
},
"files": [
"icon.svg",
"locale/*.json",
"lib/index.js",
"cordis.patch.yml",
"lib/types/**/*.d.ts"
],
"license": "MIT",
"dsh": {
"bundle": {
"patch": "./cordis.patch.yml"
}
},
"dependencies": {
"@deepseek-ai/dsh-computer-use": "workspace:*",
"@deepseek-ai/dsh-cua": "workspace:*"
},
"peerDependencies": {
"@deepseek-ai/cordis": "workspace:~"
},
"devDependencies": {
"@deepseek-ai/cordis": "workspace:~",
"@deepseek-ai/cordis-plugin-include": "workspace:~",
"js-yaml": "^4.2.0",
"@deepseek-ai/cordis-plugin-loader": "workspace:~"
},
"optionalDependencies": {
"@deepseek-ai/dsh-experimental-cua-macos": "workspace:*"
}
}
@@ -0,0 +1,2 @@
/** Optional Computer Use composition; configuration rows live in cordis.patch.yml. */
export {}
@@ -0,0 +1,35 @@
/** The shipped patch leaves shared entries available while skipping an absent native provider. */
import { readFile } from 'node:fs/promises'
import { Context } from '@deepseek-ai/cordis'
import Loader, { interpolate, type EntryOptions } from '@deepseek-ai/cordis-plugin-loader'
import { entryListSchema } from '@deepseek-ai/cordis-plugin-include'
import { load } from 'js-yaml'
import { expect, it, vi } from 'vitest'
it.each(['darwin', 'win32', 'linux'])('loads the native provider only on Darwin for %s', async (platform) => {
const patch = load(await readFile(new URL('../cordis.patch.yml', import.meta.url), 'utf8'), {
schema: entryListSchema,
}) as { insert: EntryOptions[] }[]
const ctx = new Context()
await ctx.plugin(Loader)
if (!ctx.loader.internal) throw new Error('Loader module importer is unavailable')
const imported = vi.spyOn(ctx.loader.internal, 'import').mockImplementation(async (name: string) => {
if (platform !== 'darwin' && name === '@deepseek-ai/dsh-experimental-cua-macos') {
throw new Error('native provider is absent from this installation')
}
return { name, apply: () => {} }
})
try {
for (const row of patch[0]!.insert) {
await ctx.loader.create({ ...row, disabled: Boolean(interpolate({ process: { platform } }, row.disabled)) })
}
await ctx.loader.await()
const imports = imported.mock.calls.map(call => call[0])
expect(imports).toContain('@deepseek-ai/dsh-cua')
expect(imports).toContain('@deepseek-ai/dsh-computer-use')
expect(imports.includes('@deepseek-ai/dsh-experimental-cua-macos')).toBe(platform === 'darwin')
} finally {
await ctx.fiber.dispose()
imported.mockRestore()
}
})
@@ -0,0 +1,15 @@
{
"extends": "../../../tsconfig.base.json",
"compilerOptions": {
"rootDir": "src",
"outDir": "lib/types"
},
"include": [
"src"
],
"references": [
{
"path": "../../../vendor/cordis"
}
]
}
@@ -12,11 +12,11 @@
en: 4549f02fc53854a5
zh: 6f74f1130b6623a8
/deepseek-ai-dsh-experimental-cua-macos/table-of-contents:
en: d152484eb41ac6b4
zh: 7b30b973a849247d
en: 9353d17b05f2f88c
zh: 6b2c6b4ee0046294
/deepseek-ai-dsh-experimental-cua-macos/use-this-package:
en: d9a238f761453123
zh: 849fd5fdccf45b95
en: 7126e254aaf42772
zh: fa05ac2346e022ff
/deepseek-ai-dsh-experimental-cua-macos/use-this-package/connect-an-external-execution-environment:
en: e7909c9b43e44fbc
zh: 98949877d4ea3a86
@@ -45,8 +45,8 @@
en: a2c9acb5a4320dda
zh: e8d1586af60215c4
/deepseek-ai-dsh-experimental-cua-macos/known-limitations-and-deferred-work:
en: 9d8af4c39fe407a3
zh: afec49eedfd702cb
en: de72c8c1184cb8dd
zh: 33fd21a93e51fee1
/deepseek-ai-dsh-experimental-cua-macos/known-limitations-and-deferred-work/dev-note:
en: b836f01bd0bceb4a
zh: 8122fa08b2aa95fa
+13 -6
View File
@@ -20,12 +20,18 @@ Inspect macOS permissions, select running applications and windows, read accessi
- [Known Limitations and Deferred Work](#known-limitations-and-deferred-work)
- [Dev Note](#dev-note)
The helper supplies bounded 64-pixel PNG application icons for permission-page application lists. Icon reads do not launch an application or inspect its windows; executable and signing checks still determine authorization.
The shared helper queries the process macOS considers responsible for it and captures that application’s name and icon before announcing readiness. A Desktop app launched from Terminal can therefore identify Terminal even when another GUI application sits between it and the helper. Permission queries retain this metadata after the responsible application exits and across connections sharing the helper. If the system query or application metadata is unavailable, the field is omitted and the UI uses the system prompt for guidance.
-----
<a id="use-this-package"></a>
## Use this package
Enable this experimental provider through explicit source composition on macOS 14 or later. Windows support is in development. Shipped profiles do not mount it. Prepare a native helper and load the registry/provider overlay:
This package is an optional dependency restricted to macOS. Prepare the matching executable under `native/bin` before packing; that directory is included in the archive. Installed packages require the prepared helper and do not compile Swift.
For repository Web use, enable [Computer Use](../cua-bundle/README.md) in Plugins. This prepares the local executable on macOS 14 or later; Windows support is in development. Subsequent builds check it while the existing Web profile keeps the bundle selected. Disabled profiles do not prepare native components. Explicit source composition is also available:
```sh
swift build --package-path packages/experimental/cua-macos/native --configuration release
@@ -34,10 +40,11 @@ export DSH_CUA_HELPER="$cua_native_root/.build/release/dsh-cua-native"
pnpm dsh headless --patch packages/experimental/cua-macos/examples/macos.cordis.yml --dump-config
```
The dump command checks profile composition without a model request. For a real task, replace `--dump-config` with a prompt and configure the profile’s model credentials. The overlay mounts the computer-use registry and this provider; remove an existing competing provider before using it. The responsible host/helper needs Accessibility permission for AX/input and Screen Recording permission for capture. `getCapabilities` reports permissions and session lock state without opening a system dialog. Locked or unavailable desktop sessions reject UI observations and input before dispatch; the user must unlock macOS before resuming. `listApps` and `cua.getState` list running applications. `findApps(query)` searches standard macOS application directories by name, bundle ID or absolute path. `getApp` selects a unique name, bundle ID, path or running PID, obtains application consent, and launches a stopped app in the background when needed. A stopped executable is identified before approval and checked again before and after launch; unsigned code only receives task access. The returned object reads initial AX without activating the app. Launch cancellation may leave the application running. Model guidance starts with background observation and input, without activation during initialization or before each action. For hidden or minimized windows, try `app.show()` or `window.show()` and read fresh state. This requests unhide and unminimize without activation, raising or switching Spaces; app-level restoration selects the focused, main or sole AX window. The result names the selected `windowId`, or null when no window was selected, and reports `foregroundChanged` by comparing the front app and top window before and after the request. Null means foreground evidence was unavailable; later focus changes remain possible. If restoration fails or the window remains unusable, explain the needed foreground step, call `app.activate()` or an observed window’s `raise` action, then observe again. Necessary foreground access takes priority over preserving focus. Applications may change focus after input; background delivery does not guarantee an undisturbed user session.
The dump command checks profile composition without a model request. For a real task, replace `--dump-config` with a prompt and configure the profile’s model credentials. The overlay mounts the computer-use registry and this provider; remove an existing competing provider before using it. The responsible host/helper needs Accessibility permission for AX/input and Screen Recording permission for capture. `getCapabilities` reports permissions and session lock state without opening a system dialog. Locked or unavailable desktop sessions reject UI observations and input before dispatch; the user must unlock macOS before resuming. `listApps` and `cua.getState` list running applications. `findApps(query)` searches standard macOS application directories by name, bundle ID or absolute path. `getApp` selects a unique name, bundle ID, path or running PID, obtains application consent, and launches a stopped app in the background when needed. A stopped executable is identified before approval and checked again before and after launch; unsigned code only receives session access for the selected instance. The returned object reads initial AX without activating the app. Launch cancellation may leave the application running. Model guidance starts with background observation and input, without activation during initialization or before each action. For hidden or minimized windows, try `app.show()` or `window.show()` and read fresh state. This requests unhide and unminimize without activation, raising or switching Spaces; app-level restoration selects the focused, main or sole AX window. The result names the selected `windowId`, or null when no window was selected, and reports `foregroundChanged` by comparing the front app and top window before and after the request. Null means foreground evidence was unavailable; later focus changes remain possible. If restoration fails or the window remains unusable, explain the needed foreground step, call `app.activate()` or an observed window’s `raise` action, then observe again. Necessary foreground access takes priority over preserving focus. Applications may change focus after input; background delivery does not guarantee an undisturbed user session.
| Field | Default | Meaning |
| --- | --- | --- |
| `agentPreset` | `ptc` | Register tools and guidance only for Agents bound to this preset; `false` registers globally. |
| `allowedApps` | `[]` | Live profile-local application consent, keyed by executable path and verified signing requirement. |
| `helperPath` | package-owned executable | Optional absolute native executable override. |
| `buildTimeoutMs` | `300000` | Maximum native preparation time. |
@@ -60,7 +67,7 @@ Keypad names (`KP_0` through `KP_9`, `KP_Enter`, and keypad operators) retain th
Agent disposal and plugin unload cancel and await the complete tool call, including image admission and attachment storage. An image write already in progress must settle before disposal returns; a canceled call cannot return a successful screenshot.
Application consent is enforced before AX reads, window-title reads, screenshots, activation, and input through the CUA tool. User Questions offers task access or denial, plus saved access when the application has a verifiable signature and the composition provides settings persistence. Task access is bound to the selected process handle; saved access is bound to its canonical executable path and designated code-signing requirement. Denial prevents repeated prompts in that task. Removing a saved `allowedApps` entry cancels affected operations and invalidates their helper handles; reauthorization requires fresh selection. Other handles on the same Agent connection also expire, while saved grants remain reusable. Agent/provider disposal joins cleanup of the corresponding native connection. See the [application consent decision](../../../.agents/notes/implemented/architecture/2026-09-26-cua-application-consent.md). These checks govern CUA calls; persistent PTC and separately authorized shell tools retain their own execution authority.
Application consent is enforced before AX reads, window-title reads, screenshots, activation, and input through the CUA tool. The first selection asks the human for session access, denial, or saved access when the application has a verifiable signature. Session access is kept in memory by session and application identity, survives agent and helper replacement, and expires when DSH restarts or Computer Use is disabled. Signed application identities combine the canonical executable path and designated code-signing requirement; unverifiable applications remain bound to their process instance. Denial prevents repeated prompts in that session. Revocation reads and removes saved access in the same queue as grant creation, invalidates each affected Agent connection, and awaits its native cleanup without replaying input. Every app handle on those Agent connections becomes unusable; other applications need fresh selection, but their session and saved grants remain reusable. See the [application consent decision](../../../.agents/notes/implemented/architecture/2026-09-26-cua-application-consent.md). These checks govern CUA calls; persistent PTC and separately authorized shell tools are not an operating-system sandbox.
Raw `getState` exposes validated state fields and the complete text for the requested disclosure options. The native helper also returns structured element diagnostics for native checks; the tool projection omits that duplicate array from model text and structured tool values.
@@ -148,7 +155,7 @@ The plugin contributes one `cua` operation tool with method-specific parameters.
##### Operation guidance (verbatim)
```markdown
Use cua with method getCapabilities and params {} to inspect permissions and sessionLocked. If sessionLocked is true or an operation reports SESSION_LOCKED, stop UI attempts and ask the user to unlock macOS. A null sessionLocked means no usable console session was verified. After unlocking, read fresh state; screenshots from a locked desktop are not current observations. Use listApps or cua.getState to discover running applications, and findApps(query) to search installed applications by name, bundle ID or absolute path. getApp requests user consent before launching a stopped application in the background, or before observations and input on a running one. If access is denied or revoked, stop using that application until the user explicitly grants access again. Never switch tools to bypass a refusal. Keep the returned appHandle; every app operation requires it. listWindows returns window IDs and point bounds. Raw getState returns AX text and snapshotId. Menu-bar entries are shown by default; includeMenus=true expands their contents. details=true adds identifiers, global-point bounds and complete action names. Both options also apply to getAXState. Each read replaces the indices for that app/window scope. Use its element indices only with that snapshotId. Raw screenshot and coordinate operations require a windowId. Coordinates are pixels in the latest returned screenshot, measured from its top-left. Set relative:true for integer thousandths from 0 to 1000 instead. The Host converts both modes to native window points; AX detail bounds remain global macOS points. A verified window can expose no AX elements. Its empty snapshot permits screenshots and coordinate input; element-index actions require an element from a fresh AX observation.
Use cua with method getCapabilities and params {} to inspect permissions and sessionLocked. If sessionLocked is true or an operation reports SESSION_LOCKED, stop UI attempts and ask the user to unlock macOS. A null sessionLocked means no usable console session was verified. After unlocking, read fresh state; screenshots from a locked desktop are not current observations. Use listApps or cua.getState to discover running applications, and findApps(query) to search installed applications by name, bundle ID or absolute path. getApp requests user consent before launching a stopped application in the background, or before observations and input on a running one. If access is denied or revoked, stop using that application until the user explicitly permits another authorization request. Never switch tools to bypass a refusal. Keep the returned appHandle; every app operation requires it. listWindows returns window IDs and point bounds. Raw getState returns AX text and snapshotId. Menu-bar entries are shown by default; includeMenus=true expands their contents. details=true adds identifiers, global-point bounds and complete action names. Both options also apply to getAXState. Each read replaces the indices for that app/window scope. Use its element indices only with that snapshotId. Raw screenshot and coordinate operations require a windowId. Coordinates are pixels in the latest returned screenshot, measured from its top-left. Set relative:true for integer thousandths from 0 to 1000 instead. The Host converts both modes to native window points; AX detail bounds remain global macOS points. A verified window can expose no AX elements. Its empty snapshot permits screenshots and coordinate input; element-index actions require an element from a fresh AX observation.
Initialize the object API once in persistent PTC. Reuse its variables in later calls; put temporary declarations inside { ... } blocks to avoid redeclaration errors. All object methods are async. getAXState returns text, not a structured state object. Its text separates values and placeholders, and reports subroles, help, URLs, disabled/focused/selected/expanded state and UTF-16 selection ranges when available. Writability is true, false or unknown on relevant controls; omitted state fields do not assert false. selection=unavailable means the reported range exceeds the observed text. Compact output omits ordinary Press/Pick (click), ShowMenu (right click) and ScrollToVisible actions; details=true lists every available action. Use the displayed action name with performSecondaryAction; ambiguous aliases require the complete name from details=true. Truncation reports the reached provider limit. Narrow the window or increase that provider limit before reading again.
@@ -185,7 +192,7 @@ App and Window methods:
Raw operation tools use { method, params }, including params: {} for parameterless methods. getApp selects without activating the app. Keep applications in the background for observation and routine input. Do not activate or raise an app during initialization, before each action, or merely because another app is foreground. Prefer advertised AX actions. If a window is hidden or minimized, try app.show() or window.show() and read fresh state. show requests unhide and unminimize without activation, raising or switching Spaces; it chooses the explicit, focused, main or sole AX window. A null windowId means no window was selected for restoration. foregroundChanged compares the front app and top window before and after the request; null means unavailable, and delayed focus changes remain possible. If restoration is unsupported or the window remains unusable, including another Space or Stage Manager, explain the needed foreground step, use app.activate() or an advertised raise action, and read fresh state. Completing the task takes priority when foreground access is necessary. Avoid repeated activation when the user switches away; continue background-capable steps and report what requires foreground access. Applications may still change focus in response to input.
Await operations sequentially. After input, read fresh state to verify the effect. A successful action means dispatched, not verified completion. Cancellation may leave partial effects; never retry uncertain input without observing. STALE_APP requires getApp again; STALE_SNAPSHOT requires getState again. Absolute coordinate input requires a fresh screenshot of that window. Relative input can use a successful AX observation or screenshot. SCREENSHOT_REQUIRED means capture a fresh screenshot before using pixel coordinates. For click and hover, coordinate targets must be [x, y], for example hover([650, 400], {relative:true}). A bare number such as hover(650) is always an AX element index. The relative option applies only to coordinate targets, not AX indices. Window resizing since the latest observation rejects coordinate input; movement alone preserves window-local coordinates. WINDOW_CHANGED requires fresh state before any retry; use an advertised AX action or explicitly restore the window before screenshot or coordinate input, then observe again. PERMISSION_DENIED requires the user to grant macOS Accessibility or Screen Recording access in System Settings. After granting access, observe fresh state before continuing; do not automatically replay prior input.
Await operations sequentially. After input, read fresh state to verify the effect. A successful action means dispatched, not verified completion. Cancellation may leave partial effects; never retry uncertain input without observing. STALE_APP requires getApp again; STALE_SNAPSHOT requires getState again. Absolute coordinate input requires a fresh screenshot of that window. Relative input can use a successful AX observation or screenshot. SCREENSHOT_REQUIRED means capture a fresh screenshot before using pixel coordinates. For click and hover, coordinate targets must be [x, y], for example hover([650, 400], {relative:true}). A bare number such as hover(650) is always an AX element index. The relative option applies only to coordinate targets, not AX indices. Window resizing since the latest observation rejects coordinate input; movement alone preserves window-local coordinates. WINDOW_CHANGED requires fresh state before any retry; use an advertised AX action or explicitly restore the window before screenshot or coordinate input, then observe again. PERMISSION_DENIED requires the user to grant the required macOS system permissions for macOS Accessibility or Screen Recording. After granting access, observe fresh state before continuing; do not automatically replay prior input.
Hover first uses advertised AX alternate UI without moving the system cursor, preserving revealed descendants until the pointer target leaves their region. When unavailable, it moves the visible system cursor without pressing a button; that route requires the target point to be unobscured in the foreground app. FOREGROUND_REQUIRED requires explicit activation or raising and a fresh observation before retrying. Returning, AX reads and screenshots do not restore the previous point. Wait for the application's hover delay, observe, then hover into the revealed controls before clicking. Use successive hover calls for intermediate points. The result reports hoverRoute and systemPointerMoved. Other input or application changes can interrupt the interaction; app objects do not reserve applications between calls.
@@ -211,7 +218,7 @@ The queue orders individual native requests. Holding an app object does not rese
- macOS Stage Manager can reduce background window bounds and screenshots to thumbnails. Full-window observation and coordinate input in this state are not guaranteed; model guidance requires explaining foreground needs and avoiding repeated focus reclamation.
- Background input depends on private SkyLight symbols; macOS updates can make them unavailable. Validate each target OS version.
- Observed failures include dispatched Unicode text not appearing in Blender and background Command+A not selecting all text in AppKit. Inspect application state after input; a successful tool result confirms dispatch only.
- Native binaries require a local build. Signing, distribution, and system-permission onboarding remain deferred.
- Native binaries require a local build. Signing and Desktop distribution remain future work. The [permission page](../cua/README.md) offers macOS prompts and app revocation; packaged Desktop permission attribution remains deferred.
- Persistent PTC is an external dependency assumption, not an implementation in this package. Real-model task success requires separate evaluation beyond native and protocol fixtures.
-----
+13 -6
View File
@@ -20,12 +20,18 @@ kind: "package-reference"
- [已知限制与后续工作](#known-limitations-and-deferred-work)
- [开发备注](#dev-note)
helper 为权限页的应用列表提供 64 像素 PNG 图标。读取图标不会启动应用或检查窗口,授权仍由可执行文件和签名校验决定。
共享 helper 查询 macOS 为它记录的责任进程,在发布就绪状态前保存对应应用的名称和图标。因此,从 Terminal 启动 Desktop 时,即使中间还有另一个 GUI 应用,也能识别到 Terminal。责任应用退出后,权限查询仍保留这些信息,后续复用该 helper 的连接读取同一份记录。系统查询或应用信息不可用时省略该字段,由界面引导用户按系统弹窗选择授权对象。
-----
<a id="use-this-package"></a>
## 使用本包
该实验性提供方需要在 macOS 14 及以上通过源码配置显式启用。Windows 支持正在开发中。默认配置不加载它。先准备原生组件,再加载注册表和提供方覆盖层:
本包是仅限 macOS 的可选依赖。打包前需在 `native/bin` 下准备对应架构的可执行文件,该目录会包含在发布包中。安装版需要准备好的 helper,不会编译 Swift。
从仓库运行 Web 时,在插件页开启 [Computer Use](../cua-bundle/README.zh.md)。首次启用会在 macOS 14 及以上准备本机组件,Windows 支持正在开发中。现有 Web profile 保持选中时,后续构建继续检查更新。未开启的 profile 不准备原生组件。也可以显式加载源码覆盖层:
```sh
swift build --package-path packages/experimental/cua-macos/native --configuration release
@@ -34,10 +40,11 @@ export DSH_CUA_HELPER="$cua_native_root/.build/release/dsh-cua-native"
pnpm dsh headless --patch packages/experimental/cua-macos/examples/macos.cordis.yml --dump-config
```
配置输出命令不发起模型请求。执行真实任务时,将 `--dump-config` 替换为任务提示,并配置该 profile 的模型凭据。配置挂载 computer-use 注册表和本 provider;使用前需移除已有的竞争 provider。负责执行的宿主或辅助进程需要辅助功能权限来读取 AX 和输入,需要屏幕录制权限来截图。`getCapabilities` 查询权限和锁屏状态,不弹出系统授权窗口。桌面会话锁定或不可用时,界面观察与输入会在分派前拒绝,需由用户解锁 macOS 后继续。`listApps` 和 `cua.getState` 列出运行中的应用。`findApps(query)` 按名称、bundle ID 或绝对路径搜索 macOS 标准应用目录。`getApp` 按唯一名称、bundle ID、路径或运行中的 PID 选择应用,取得应用许可后,在需要时后台启动尚未运行的应用。启动前先确认可执行文件身份,授权后在启动前后再次核对;未签名代码只取得任务许可。返回的对象读取初始 AX,不自动激活应用。启动被取消时,应用可能已经运行。模型指引要求先在后台观察和输入,不在初始化或每次动作前激活应用。窗口隐藏或最小化时,先尝试 `app.show()` 或 `window.show()`,再重新观察。它请求取消隐藏和恢复最小化,不主动激活、抬起窗口或切换桌面空间;应用级恢复选择焦点窗口、主窗口或唯一 AX 窗口。结果通过 `windowId` 指明选中的窗口,未选中时为 null;`foregroundChanged` 比较操作前后的前台应用和最上层窗口,无法取得完整观察时为 null,返回后仍可能发生焦点变化。恢复失败或窗口仍不可用时,说明需要前台的步骤,再调用 `app.activate()` 或窗口的 `raise` 动作并重新观察。必须使用前台时,以完成任务为先。应用收到输入后仍可能改变焦点,后台投递不保证用户操作始终不受干扰。
配置输出命令不发起模型请求。执行真实任务时,将 `--dump-config` 替换为任务提示,并配置该 profile 的模型凭据。配置挂载 computer-use 注册表和本 provider;使用前需移除已有的竞争 provider。负责执行的宿主或辅助进程需要辅助功能权限来读取 AX 和输入,需要屏幕录制权限来截图。`getCapabilities` 查询权限和锁屏状态,不弹出系统授权窗口。桌面会话锁定或不可用时,界面观察与输入会在分派前拒绝,需由用户解锁 macOS 后继续。`listApps` 和 `cua.getState` 列出运行中的应用。`findApps(query)` 按名称、bundle ID 或绝对路径搜索 macOS 标准应用目录。`getApp` 按唯一名称、bundle ID、路径或运行中的 PID 选择应用,取得应用许可后,在需要时后台启动尚未运行的应用。启动前先确认可执行文件身份,授权后在启动前后再次核对;未签名代码只能取得所选实例的会话许可。返回的对象读取初始 AX,不自动激活应用。启动被取消时,应用可能已经运行。模型指引要求先在后台观察和输入,不在初始化或每次动作前激活应用。窗口隐藏或最小化时,先尝试 `app.show()` 或 `window.show()`,再重新观察。它请求取消隐藏和恢复最小化,不主动激活、抬起窗口或切换桌面空间;应用级恢复选择焦点窗口、主窗口或唯一 AX 窗口。结果通过 `windowId` 指明选中的窗口,未选中时为 null;`foregroundChanged` 比较操作前后的前台应用和最上层窗口,无法取得完整观察时为 null,返回后仍可能发生焦点变化。恢复失败或窗口仍不可用时,说明需要前台的步骤,再调用 `app.activate()` 或窗口的 `raise` 动作并重新观察。必须使用前台时,以完成任务为先。应用收到输入后仍可能改变焦点,后台投递不保证用户操作始终不受干扰。
| 字段 | 默认值 | 含义 |
| --- | --- | --- |
| `agentPreset` | `ptc` | 仅为绑定该 preset 的 Agent 注册工具和指引;`false` 表示全局注册。 |
| `allowedApps` | `[]` | 当前配置档的动态应用授权,绑定可执行文件路径和已验证的代码签名要求。 |
| `helperPath` | 包内可执行文件 | 可选的原生可执行文件绝对路径覆盖。 |
| `buildTimeoutMs` | `300000` | 原生组件准备的最长时间。 |
@@ -60,7 +67,7 @@ pnpm dsh headless --patch packages/experimental/cua-macos/examples/macos.cordis.
Agent 销毁和插件卸载会取消并等待完整工具调用,包括图像接入检查和附件存储。已经开始的图像写入必须结束后,销毁才会返回;已取消的调用不会返回成功截图。
CUA 工具在读取 AX、窗口标题、截图、激活和输入之前检查应用授权。User Questions 提供任务级允许和拒绝选项;应用签名可验证且组合支持配置持久化时,也可保存授权。任务授权绑定所选进程句柄,长期授权绑定真实可执行路径和指定代码签名要求。拒绝后,同一任务不再反复询问。删除保存的 `allowedApps` 条目会取消相关操作并使原生组件中的句柄失效;重新授权需要重新选择应用。同一 Agent 连接中的其他句柄也会失效,已保存的授权仍可复用。Agent 或提供方销毁时等待对应连接的原生状态清理。设计理由见[应用授权决策](../../../.agents/notes/implemented/architecture/2026-09-26-cua-application-consent.zh.md)。这些检查约束 CUA 调用;持久 PTC 和另行授权的 shell 工具保留各自的执行权限。
CUA 工具在读取 AX、窗口标题、截图、激活和输入前检查应用授权。首次选择应用时询问用户,可选择本会话允许、拒绝;签名可验证的应用还可保存长期授权。会话许可按会话和应用身份保存在内存中,Agent 或 helper 重建后仍可复用,重启 DSH 或关闭 Computer Use 后失效。已签名应用的身份由规范化可执行文件路径和代码签名指定要求组成;签名不可验证的应用仍绑定进程实例。拒绝后,同一会话不会反复询问。撤销与长期许可的保存共用队列,在队列内读取并移除许可,然后使相关 Agent 连接失效并等待原生清理完成,不自动重放输入。同一 Agent 连接中的其他应用也需要重新选择,其他应用的会话许可和长期许可仍可复用。设计理由见[应用授权决策](../../../.agents/notes/implemented/architecture/2026-09-26-cua-application-consent.zh.md)。这些检查的范围是 CUA 调用。persistent PTC 与已授权的 shell 仍按各自执行权限运行。
原始 `getState` 操作返回经过校验的状态字段,以及按本次披露选项生成的完整文本。原生 helper 还返回供原生检查使用的结构化元素诊断数组;工具投影会从模型文本和结构化工具值中移除这份重复数组。
@@ -148,7 +155,7 @@ AppKit sheet 可以通过其窗口 ID 单独选择,也会出现在所属文档
##### 操作指导文本(原文)
```markdown
Use cua with method getCapabilities and params {} to inspect permissions and sessionLocked. If sessionLocked is true or an operation reports SESSION_LOCKED, stop UI attempts and ask the user to unlock macOS. A null sessionLocked means no usable console session was verified. After unlocking, read fresh state; screenshots from a locked desktop are not current observations. Use listApps or cua.getState to discover running applications, and findApps(query) to search installed applications by name, bundle ID or absolute path. getApp requests user consent before launching a stopped application in the background, or before observations and input on a running one. If access is denied or revoked, stop using that application until the user explicitly grants access again. Never switch tools to bypass a refusal. Keep the returned appHandle; every app operation requires it. listWindows returns window IDs and point bounds. Raw getState returns AX text and snapshotId. Menu-bar entries are shown by default; includeMenus=true expands their contents. details=true adds identifiers, global-point bounds and complete action names. Both options also apply to getAXState. Each read replaces the indices for that app/window scope. Use its element indices only with that snapshotId. Raw screenshot and coordinate operations require a windowId. Coordinates are pixels in the latest returned screenshot, measured from its top-left. Set relative:true for integer thousandths from 0 to 1000 instead. The Host converts both modes to native window points; AX detail bounds remain global macOS points. A verified window can expose no AX elements. Its empty snapshot permits screenshots and coordinate input; element-index actions require an element from a fresh AX observation.
Use cua with method getCapabilities and params {} to inspect permissions and sessionLocked. If sessionLocked is true or an operation reports SESSION_LOCKED, stop UI attempts and ask the user to unlock macOS. A null sessionLocked means no usable console session was verified. After unlocking, read fresh state; screenshots from a locked desktop are not current observations. Use listApps or cua.getState to discover running applications, and findApps(query) to search installed applications by name, bundle ID or absolute path. getApp requests user consent before launching a stopped application in the background, or before observations and input on a running one. If access is denied or revoked, stop using that application until the user explicitly permits another authorization request. Never switch tools to bypass a refusal. Keep the returned appHandle; every app operation requires it. listWindows returns window IDs and point bounds. Raw getState returns AX text and snapshotId. Menu-bar entries are shown by default; includeMenus=true expands their contents. details=true adds identifiers, global-point bounds and complete action names. Both options also apply to getAXState. Each read replaces the indices for that app/window scope. Use its element indices only with that snapshotId. Raw screenshot and coordinate operations require a windowId. Coordinates are pixels in the latest returned screenshot, measured from its top-left. Set relative:true for integer thousandths from 0 to 1000 instead. The Host converts both modes to native window points; AX detail bounds remain global macOS points. A verified window can expose no AX elements. Its empty snapshot permits screenshots and coordinate input; element-index actions require an element from a fresh AX observation.
Initialize the object API once in persistent PTC. Reuse its variables in later calls; put temporary declarations inside { ... } blocks to avoid redeclaration errors. All object methods are async. getAXState returns text, not a structured state object. Its text separates values and placeholders, and reports subroles, help, URLs, disabled/focused/selected/expanded state and UTF-16 selection ranges when available. Writability is true, false or unknown on relevant controls; omitted state fields do not assert false. selection=unavailable means the reported range exceeds the observed text. Compact output omits ordinary Press/Pick (click), ShowMenu (right click) and ScrollToVisible actions; details=true lists every available action. Use the displayed action name with performSecondaryAction; ambiguous aliases require the complete name from details=true. Truncation reports the reached provider limit. Narrow the window or increase that provider limit before reading again.
@@ -185,7 +192,7 @@ App and Window methods:
Raw operation tools use { method, params }, including params: {} for parameterless methods. getApp selects without activating the app. Keep applications in the background for observation and routine input. Do not activate or raise an app during initialization, before each action, or merely because another app is foreground. Prefer advertised AX actions. If a window is hidden or minimized, try app.show() or window.show() and read fresh state. show requests unhide and unminimize without activation, raising or switching Spaces; it chooses the explicit, focused, main or sole AX window. A null windowId means no window was selected for restoration. foregroundChanged compares the front app and top window before and after the request; null means unavailable, and delayed focus changes remain possible. If restoration is unsupported or the window remains unusable, including another Space or Stage Manager, explain the needed foreground step, use app.activate() or an advertised raise action, and read fresh state. Completing the task takes priority when foreground access is necessary. Avoid repeated activation when the user switches away; continue background-capable steps and report what requires foreground access. Applications may still change focus in response to input.
Await operations sequentially. After input, read fresh state to verify the effect. A successful action means dispatched, not verified completion. Cancellation may leave partial effects; never retry uncertain input without observing. STALE_APP requires getApp again; STALE_SNAPSHOT requires getState again. Absolute coordinate input requires a fresh screenshot of that window. Relative input can use a successful AX observation or screenshot. SCREENSHOT_REQUIRED means capture a fresh screenshot before using pixel coordinates. For click and hover, coordinate targets must be [x, y], for example hover([650, 400], {relative:true}). A bare number such as hover(650) is always an AX element index. The relative option applies only to coordinate targets, not AX indices. Window resizing since the latest observation rejects coordinate input; movement alone preserves window-local coordinates. WINDOW_CHANGED requires fresh state before any retry; use an advertised AX action or explicitly restore the window before screenshot or coordinate input, then observe again. PERMISSION_DENIED requires the user to grant macOS Accessibility or Screen Recording access in System Settings. After granting access, observe fresh state before continuing; do not automatically replay prior input.
Await operations sequentially. After input, read fresh state to verify the effect. A successful action means dispatched, not verified completion. Cancellation may leave partial effects; never retry uncertain input without observing. STALE_APP requires getApp again; STALE_SNAPSHOT requires getState again. Absolute coordinate input requires a fresh screenshot of that window. Relative input can use a successful AX observation or screenshot. SCREENSHOT_REQUIRED means capture a fresh screenshot before using pixel coordinates. For click and hover, coordinate targets must be [x, y], for example hover([650, 400], {relative:true}). A bare number such as hover(650) is always an AX element index. The relative option applies only to coordinate targets, not AX indices. Window resizing since the latest observation rejects coordinate input; movement alone preserves window-local coordinates. WINDOW_CHANGED requires fresh state before any retry; use an advertised AX action or explicitly restore the window before screenshot or coordinate input, then observe again. PERMISSION_DENIED requires the user to grant the required macOS system permissions for macOS Accessibility or Screen Recording. After granting access, observe fresh state before continuing; do not automatically replay prior input.
Hover first uses advertised AX alternate UI without moving the system cursor, preserving revealed descendants until the pointer target leaves their region. When unavailable, it moves the visible system cursor without pressing a button; that route requires the target point to be unobscured in the foreground app. FOREGROUND_REQUIRED requires explicit activation or raising and a fresh observation before retrying. Returning, AX reads and screenshots do not restore the previous point. Wait for the application's hover delay, observe, then hover into the revealed controls before clicking. Use successive hover calls for intermediate points. The result reports hoverRoute and systemPointerMoved. Other input or application changes can interrupt the interaction; app objects do not reserve applications between calls.
@@ -211,7 +218,7 @@ Use selectText only on controls advertising selectionWritable. It selects a uniq
- macOS 台前调度可能将后台窗口的尺寸与截图变为缩略图,当前无法保证这种状态下的完整画面与坐标输入;模型指引要求说明前台需求并避免反复抢回焦点。
- 后台输入依赖私有 SkyLight 符号,macOS 升级可能使其不可用,需在目标系统版本验证。
- 实测 Blender Unicode 输入曾出现分派成功但界面未显示文本,后台 AppKit 的 Command+A 曾未选中全文。输入后必须检查应用状态;工具返回成功只表示已分派。
- 原生二进制需要本地构建。签名、分发和系统权限引导留待后续。
- 原生二进制需要本地构建,签名和桌面分发留待后续。[权限页面](../cua/README.zh.md)提供 macOS 申请入口和应用授权撤销;桌面安装包的权限归属留待后续处理。
- 持久 PTC 是对外部运行环境的假设,本包不实现它。真实模型任务成功率需要在原生测试和协议测试之外单独评估。
-----
@@ -4,4 +4,5 @@
- id: cua
name: '../src/index.ts'
config:
agentPreset: false
helperPath: !!js process.env.DSH_CUA_HELPER
@@ -0,0 +1,3 @@
target/
.build/
.swiftpm/
@@ -15,17 +15,17 @@
en: e8cbc43d38786c40
zh: 52b3ba85d98011b9
/dsh-cua-native-helper/build-and-service:
en: 824e4a2b9036d9bf
zh: 37c34a454abc6965
en: 02b25ac1126b6094
zh: 3995a1f46354531b
/dsh-cua-native-helper/request-protocol:
en: 3202c567a6e1a654
zh: 143eebd0096a0487
en: c8ad13843989b314
zh: 2a42acb2faee8803
/dsh-cua-native-helper/observation-and-input:
en: c56adf89d249f4f9
zh: 2a36a99e0a2badef
en: c527eedc3a0ee968
zh: ce8cc12d5a62e8ae
/dsh-cua-native-helper/verification:
en: 0e42ed266cc287d8
zh: bc5e6ca20e82dae3
en: a2ac6f2d581a1e8b
zh: f82c7d0c13539421
/dsh-cua-native-helper/further-exploration:
en: 265e1927952ce6d7
zh: 7c35d9fd98b6e6c8
@@ -27,7 +27,7 @@ Build with `swift build --package-path packages/experimental/cua-macos/native --
The shared service uses an owner-only Unix socket under `/tmp/dsh-cua-<uid>`, keyed by the executable SHA-256. A version lock elects one process; an additional real-executable-path lock excludes a different build at the same path. The first frame is protocol-1 `hello` with the runtime key and this connection’s byte, AX, operation, screenshot, event-spacing and cleanup limits. `ready` reports the verified runtime key, PID and instance ID. Every connection owns an independent AXWorker. The stdin/stdout entry is a diagnostic proxy to this service.
AppKit runs on the main event loop. Blocking AX and input use each connection’s serial actor; all connections share a FIFO for complete desktop operations. Socket writes do not hold that operation slot. A request cancellation waits for native work to stop; it does not cancel the connection’s lifetime latch. Screenshot timeout ends only the request. An unfinished system capture retains the single capture slot until its real callback arrives, rejects new captures with CAPTURE_BUSY and leaves AX/input available. Only an operation or cleanup that cannot stop can invalidate the shared process. The proxy accepts `--max-request-bytes`, `--max-response-bytes`, `--ax-timeout-ms`, `--timeout-ms`, `--screenshot-timeout-ms`, `--shutdown-grace-ms` and `--text-event-interval-ms`; the Host sets the same limits in hello. An AX deadline bounds an individual system call. Cancellation can follow partial input, and uncertain input is never replayed automatically.
AppKit runs on the main event loop. Blocking socket accepts and handshakes use background queues; server callbacks explicitly carry `@MainActor @Sendable` isolation and weak ownership. Shutdown releases accepted connections even when their handshake is incomplete. Blocking AX and input use each connection’s serial actor; all connections share a FIFO for complete desktop operations. Socket writes do not hold that operation slot. A request cancellation waits for native work to stop; it does not cancel the connection’s lifetime latch. Screenshot timeout ends only the request. An unfinished system capture retains the single capture slot until its real callback arrives, rejects new captures with CAPTURE_BUSY and leaves AX/input available. Only an operation or cleanup that cannot stop can invalidate the shared process. The proxy accepts `--max-request-bytes`, `--max-response-bytes`, `--ax-timeout-ms`, `--timeout-ms`, `--screenshot-timeout-ms`, `--shutdown-grace-ms` and `--text-event-interval-ms`; the Host sets the same limits in hello. An AX deadline bounds an individual system call. Cancellation can follow partial input, and uncertain input is never replayed automatically.
-----
@@ -57,7 +57,7 @@ Every app-targeted operation requires `appHandle` from `getApp` in the same help
Stable error codes are `INVALID_REQUEST` (malformed envelope), `INVALID_ARGUMENT` (invalid parameters), `UNSUPPORTED` (unknown operation), `APP_UNAVAILABLE`, `AMBIGUOUS_APP`, `STALE_APP`, `STALE_SNAPSHOT`, `WINDOW_UNAVAILABLE`, `WINDOW_CHANGED`, `PERMISSION_DENIED`, `FOREGROUND_REQUIRED` (hover target is not the unobscured foreground window), `PANEL_NOT_READY` (system panel recipient unavailable before input), and `NATIVE_ERROR` (remaining operating-system failures). An error never promises that a partially dispatched input was rolled back. The host must not retry input automatically.
Application selection verifies the executable path and code-signing requirement before returning an authorization identity to the Host. Unsigned running applications receive a process-lifetime identity; stopped unsigned applications use an executable digest and cannot receive persistent consent. The Host obtains human consent before launch, observation or input.
Application selection returns an authorization record to the Host: a digest of canonical executable path and verified designated signing requirement, display name, executable path, and persistable flag. Unsigned or ad-hoc-signed applications cannot receive saved consent; running instances use process-lifetime identity, while stopped applications use an executable digest. The Host strips this record from model results and obtains human consent before launch, observation or input. This pipe protocol is trusted Host transport, not a sandbox against a separately authorized shell process.
Protocol declarations are generated from [the shared protocol](../../cua/src/protocol.ts). Run `pnpm exec tsx packages/experimental/cua-macos/native/generate-protocol.ts` from the repository root; package tests verify generated JSON and Swift field/enum tables. Model-submittable parameters exclude Host resource bounds and the private launch continuation.
@@ -79,6 +79,8 @@ Capture and pointer delivery read CG bounds before and after an exact-window AX
The responsible host/helper needs macOS Accessibility permission for AX state and input, and Screen Recording permission for screenshots. Capability preflight and permission errors do not grant permissions. Capture refuses a window that moves or resizes while the image is being read. Unassociated system panels and accessory-only applications are not selectable.
Host-only permission controls are `permissionStatus {}`, `requestPermission {permission}`, and `openPermissionSettings {permission}`, where permission is `accessibility` or `screenRecording`. They are excluded from model tools and from `getCapabilities.methods`. Permission status includes observed helper/parent process paths; these paths do not identify the TCC responsible application. The operating-system prompt names the application and asks for the user’s decision. The helper identifies the responsible process with the optional libSystem SPI `responsibility_get_pid_responsible_for_pid`, then captures its application name and icon before publishing readiness. The function is resolved dynamically, following the same signature used by [Chromium](https://chromium.googlesource.com/chromium/src/+/refs/tags/134.0.6998.28/base/process/process_info_mac.mm). If the symbol, process, or application metadata is unavailable, attribution stays unknown; it is not inferred from the nearest GUI ancestor. Permission checks remain independent of this metadata.
Observation reads each non-layout node’s descriptive properties and children through `AXUIElementCopyMultipleAttributeValues`. Per-attribute errors retain the individual-read rules; unsupported batch APIs use individual reads. Values are retained only during that observation, and input reads live attributes. Application state can change between nodes; a batch does not make the whole tree atomic.
System file-panel keyboard routing resolves the remote recipient through the selected app’s focused AX sheet. Only Apple’s Open and Save Panel Service is accepted, and its focused ancestry must still contain the identified remote window. Element pointer fallbacks resolve an app-owned window or sheet through AX parents when AXWindow is unavailable. Ordinary clicks retain AX action priority over pointer input; visual click markers need no system-cursor movement.
@@ -103,6 +105,8 @@ Use `python3 packages/experimental/cua-macos/native/tests/test_swift.py` for off
Run `python3 packages/experimental/cua-macos/native/tests/shared_helper_e2e.py --helper /absolute/helper` to verify concurrent startup, a shared service PID, independent handles, same-application handoff, request-local screenshot deadlines, connection cleanup and signal shutdown with an owned AppKit fixture. Existing Accessibility and Screen Recording grants are required. `--without-desktop` limits the check to protocol and process lifetime.
Run `python3 packages/experimental/cua-macos/native/tests/launcher_identity_e2e.py --helper /absolute/helper` in a GUI login session to verify a Launch Services application that starts a nested GUI launcher and then the helper. The helper identifies the outer responsible application and retains its name and icon after the launchers exit. This check owns both applications and the helper, cleans up its Launch Services registrations, and does not request Accessibility or Screen Recording access.
Run `python3 packages/experimental/cua-macos/native/tests/axless_e2e.py` after building and unlocking the Mac to verify an owned window whose application removes and later restores its AX roots. The check requires existing Accessibility/Screen Recording grants, rejects stale element indices, captures the AXless window, verifies a coordinate click in the fixture state file, and checks restoration. It never changes system permission settings.
Run `python3 packages/experimental/cua-macos/native/tests/observation_benchmark.py --reference-helper /absolute/reference --candidate-helper /absolute/candidate --artifacts /private/evidence` to compare complete observations of the same owned AppKit window with 35 and 400 synthetic rows. Each helper receives one warmup and three alternating measured requests. All result fields must match except helper-generated app and snapshot IDs. The report keeps raw timings, byte counts, hashes and replies; it sets no CI speed threshold and makes no memory claim. Fixture startup can activate the application. Run without other desktop automation or compilation jobs.
@@ -27,7 +27,7 @@ kind: "package-reference"
共享服务在 `/tmp/dsh-cua-<uid>` 下使用仅所有者可访问的 Unix socket,以可执行文件的 SHA-256 区分实例。版本锁选出一个进程,另一个真实可执行路径锁排除同一路径的不同构建。首帧为协议版本 1 的 `hello`,包含运行时标识以及该连接的字节、AX、操作、截图、事件间隔和清理限制。`ready` 返回已核实的运行时标识、PID 和实例 ID。每条连接拥有独立的 AXWorker。stdin/stdout 入口是该服务的诊断代理。
AppKit 在主事件循环上运行。阻塞式 AX 和输入使用每条连接的串行 actor,所有连接共享用于完整桌面操作的 FIFO。socket 写出不占用操作槽位。请求取消会等待原生工作停止,不取消连接的生命期控制器。截图超时只结束该请求。未完成的系统捕获保留唯一捕获槽位,直至真实回调到达;期间新截图返回 CAPTURE_BUSY,AX 和输入仍可用。只有无法停止的操作或清理才可能使共享进程失效。代理接受 `--max-request-bytes`、`--max-response-bytes`、`--ax-timeout-ms`、`--timeout-ms`、`--screenshot-timeout-ms`、`--shutdown-grace-ms` 和 `--text-event-interval-ms`;Host 在 hello 中设置相同限制。AX 超时约束单次系统调用。取消可能发生在部分输入之后,结果不确定的输入不自动重放。
AppKit 在主事件循环上运行。阻塞式 socket 接收与握手使用后台队列,服务回调明确声明 `@MainActor @Sendable` 隔离,并弱引用服务对象。关闭时会释放已接收的连接,包括尚未完成握手的连接。阻塞式 AX 和输入使用每条连接的串行 actor,所有连接共享用于完整桌面操作的 FIFO。socket 写出不占用操作槽位。请求取消会等待原生工作停止,不取消连接的生命期控制器。截图超时只结束该请求。未完成的系统捕获保留唯一捕获槽位,直至真实回调到达;期间新截图返回 CAPTURE_BUSY,AX 和输入仍可用。只有无法停止的操作或清理才可能使共享进程失效。代理接受 `--max-request-bytes`、`--max-response-bytes`、`--ax-timeout-ms`、`--timeout-ms`、`--screenshot-timeout-ms`、`--shutdown-grace-ms` 和 `--text-event-interval-ms`;Host 在 hello 中设置相同限制。AX 超时约束单次系统调用。取消可能发生在部分输入之后,结果不确定的输入不自动重放。
-----
@@ -57,7 +57,7 @@ AppKit 在主事件循环上运行。阻塞式 AX 和输入使用每条连接的
稳定错误码包括 `INVALID_REQUEST`(信封无效)、`INVALID_ARGUMENT`(参数无效)、`UNSUPPORTED`(操作未知)、`APP_UNAVAILABLE`、`AMBIGUOUS_APP`、`STALE_APP`、`STALE_SNAPSHOT`、`WINDOW_UNAVAILABLE`、`WINDOW_CHANGED`、`PERMISSION_DENIED`、`FOREGROUND_REQUIRED`(悬停目标不是未被遮挡的前台窗口)、`PANEL_NOT_READY`(输入前系统面板接收方尚不可用)及 `NATIVE_ERROR`(其余操作系统失败)。错误不保证已分派的部分输入被回滚,Host 不得自动重试输入。
应用选择先核实可执行路径和代码签名要求,再向 Host 返回授权身份。未签名的运行中应用使用进程生命期身份,未启动的未签名应用使用可执行文件摘要,不能获得持久授权。Host 在启动、观察和输入前取得用户许可。
应用选择向 Host 返回授权记录:规范化可执行路径与经验证的代码签名指定要求的摘要、显示名称、可执行路径和可持久化标记。未签名或临时签名的应用不能获得长期授权;运行中实例使用进程生命期身份,未启动应用使用可执行文件摘要。Host 从模型结果中移除此记录,并在启动、观察和输入前取得用户许可。此管道协议用于可信 Host 传输,无法隔离另一个已获授权的 shell 进程。
协议声明从[共享协议](../../cua/src/protocol.ts)生成。在仓库根目录运行 `pnpm exec tsx packages/experimental/cua-macos/native/generate-protocol.ts`,包测试检查生成的 JSON 和 Swift 字段/枚举表。模型参数不包含 Host 资源上限和私有启动续接操作。
@@ -79,6 +79,8 @@ AppKit 在主事件循环上运行。阻塞式 AX 和输入使用每条连接的
负责执行的 host/helper 需要 macOS 辅助功能权限以读取 AX 和输入,需要屏幕录制权限以截图。能力预检和权限错误不授予权限。截图会拒绝在读取图片期间移动或缩放的窗口。无法确认归属的系统面板及仅后台辅助类型的应用不可选择。
仅供 Host 使用的权限操作为 `permissionStatus {}`、`requestPermission {permission}` 和 `openPermissionSettings {permission}`,其中 permission 为 `accessibility` 或 `screenRecording`。这些操作不出现在模型工具及 `getCapabilities.methods` 中。权限状态包含观察到的 helper 和父进程路径,这些路径不能确定 TCC 归属的应用。系统授权提示负责指出应用并询问用户决定。helper 使用可选的 libSystem 内部接口 `responsibility_get_pid_responsible_for_pid` 查询责任进程,在报告就绪前记录对应应用的名称和图标。接口通过动态加载解析,函数签名与 [Chromium](https://chromium.googlesource.com/chromium/src/+/refs/tags/134.0.6998.28/base/process/process_info_mac.mm) 使用的一致。符号、进程或应用信息不可用时,归属保持未知,不根据最近的 GUI 祖先推断。权限检查独立于这些信息。
观察通过 `AXUIElementCopyMultipleAttributeValues` 读取每个非布局节点的描述属性和子节点。逐属性错误保留单独读取的规则;不支持批量 API 时使用单独读取。值只在本次观察期间保留,输入读取即时属性。应用状态可能在节点之间变化,批量读取不使整棵树具备原子性。
系统文件面板的键盘路由通过所选应用的 AX 焦点 sheet 解析远端接收方。只接受 Apple 的 Open and Save Panel Service,且其焦点祖先链仍须包含已识别的远端窗口。AXWindow 不可用时,元素鼠标后备路径沿 AX 父节点解析应用所属窗口或 sheet。普通点击保持 AX 动作优先,可视点击标记不需要系统鼠标移动。
@@ -103,6 +105,8 @@ AppKit 在主事件循环上运行。阻塞式 AX 和输入使用每条连接的
运行 `python3 packages/experimental/cua-macos/native/tests/shared_helper_e2e.py --helper /absolute/helper`,通过测试自建的 AppKit fixture 核实并发启动、共享服务 PID、独立句柄、同一应用交接、请求级截图期限、连接清理和信号退出。需要已有辅助功能和屏幕录制权限。`--without-desktop` 将检查限定为协议与进程生命期。
在 GUI 登录会话中运行 `python3 packages/experimental/cua-macos/native/tests/launcher_identity_e2e.py --helper /absolute/helper`,验证由 Launch Services 启动的应用先启动另一个 GUI 启动器,再启动 helper。helper 识别外层责任应用,并在启动器退出后保留其名称和图标。检查只使用测试自建的两个应用和 helper,结束后清理对应的 Launch Services 注册,不申请辅助功能或屏幕录制权限。
构建并解锁 Mac 后运行 `python3 packages/experimental/cua-macos/native/tests/axless_e2e.py`,验证应用移除并恢复 AX 根节点的窗口。检查需要已有辅助功能/屏幕录制权限,会拒绝过期元素索引,捕获无 AX 窗口,通过 fixture 状态文件核实坐标点击,并检查恢复。不会修改系统权限设置。
运行 `python3 packages/experimental/cua-macos/native/tests/observation_benchmark.py --reference-helper /absolute/reference --candidate-helper /absolute/candidate --artifacts /private/evidence`,对比同一自建 AppKit 窗口中 35 行和 400 行合成内容的完整观察。每个 helper 先预热一次,再交替测量三次。除 helper 生成的应用和快照 ID 外,结果字段必须一致。报告保留原始耗时、字节数、哈希和回复,不设置 CI 性能阈值,也不声称内存表现。fixture 启动可能激活应用。运行时避免其他桌面自动化或编译任务。
@@ -36,6 +36,22 @@ struct ApplicationCatalogSnapshot: Sendable {
}
return ApplicationCatalogSnapshot(applications: apps, frontPID: NSWorkspace.shared.frontmostApplication?.processIdentifier)
}
/// A bounded PNG for human recognition; authorization continues to use executable/signing identity.
static func icon(at path: String) -> String? {
guard let bitmap = NSBitmapImageRep(bitmapDataPlanes: nil, pixelsWide: 64, pixelsHigh: 64,
bitsPerSample: 8, samplesPerPixel: 4, hasAlpha: true, isPlanar: false,
colorSpaceName: .deviceRGB, bytesPerRow: 0, bitsPerPixel: 0),
let context = NSGraphicsContext(bitmapImageRep: bitmap) else { return nil }
NSGraphicsContext.saveGraphicsState()
defer { NSGraphicsContext.restoreGraphicsState() }
NSGraphicsContext.current = context
var url = URL(fileURLWithPath: path)
while url.path != "/" && url.pathExtension != "app" { url.deleteLastPathComponent() }
let iconPath = url.pathExtension == "app" ? url.path : path
NSWorkspace.shared.icon(forFile: iconPath).draw(in: NSRect(x: 0, y: 0, width: 64, height: 64),
from: .zero, operation: .copy, fraction: 1)
return bitmap.representation(using: .png, properties: [:]).map { "data:image/png;base64," + $0.base64EncodedString() }
}
static func activate(_ identity: ProcessIdentity) throws {
try NativePlatform.verify(identity)
guard let app = NSRunningApplication(processIdentifier: identity.pid), !app.isTerminated else {
@@ -47,9 +63,18 @@ struct ApplicationCatalogSnapshot: Sendable {
}
enum NativePlatform {
/// Optional libSystem SPI: unavailable attribution stays unknown rather than guessing from parentage.
static func responsiblePID(_ pid: Int32) -> Int32? {
typealias Resolve = @convention(c) (pid_t) -> pid_t
guard let symbol = dlsym(UnsafeMutableRawPointer(bitPattern: -2), "responsibility_get_pid_responsible_for_pid") else { return nil }
let resolve = unsafeBitCast(symbol, to: Resolve.self)
let responsible = resolve(pid)
return responsible > 0 ? responsible : nil
}
static func executablePath(_ pid: Int32) -> String? {
var buffer = [CChar](repeating: 0, count: 4 * Int(MAXPATHLEN))
return proc_pidpath(pid, &buffer, UInt32(buffer.count)) > 0 ? String(cString: buffer) : nil
guard proc_pidpath(pid, &buffer, UInt32(buffer.count)) > 0 else { return nil }
return String(decoding: buffer.prefix { $0 != 0 }.map { UInt8(bitPattern: $0) }, as: UTF8.self)
}
static func identity(_ pid: Int32) throws -> ProcessIdentity {
var info = proc_bsdinfo()
@@ -3,6 +3,7 @@ enum NativeProtocol {
static let modelMethods: [String] = ["activate", "click", "drag", "findApps", "getApp", "getCapabilities", "getState", "hover", "listApps", "listWindows", "paste", "performSecondaryAction", "pressKey", "screenshot", "scroll", "selectText", "setValue", "show", "typeText"]
static let fields: [String: Set<String>] = [
"activate": ["appHandle"],
"applicationIcons": ["paths"],
"click": ["appHandle", "clickCount", "elementIndex", "holdMs", "modifiers", "mouseButton", "snapshotId", "windowId", "x", "y"],
"drag": ["appHandle", "durationMs", "fromX", "fromY", "modifiers", "mouseButton", "snapshotId", "steps", "toX", "toY", "windowId"],
"findApps": ["maxAppMatches", "query"],
@@ -13,9 +14,12 @@ enum NativeProtocol {
"launchApp": ["appHandle"],
"listApps": [],
"listWindows": ["appHandle"],
"openPermissionSettings": ["permission"],
"paste": ["appHandle", "format", "maxClipboardBytes", "maxDepth", "maxElements", "pasteReadTimeoutMs", "text"],
"performSecondaryAction": ["action", "appHandle", "elementIndex", "snapshotId"],
"permissionStatus": [],
"pressKey": ["appHandle", "key", "modifiers"],
"requestPermission": ["permission"],
"screenshot": ["appHandle", "windowId"],
"scroll": ["appHandle", "deltaLinesX", "deltaLinesY", "snapshotId", "windowId", "x", "y"],
"selectText": ["appHandle", "elementIndex", "prefix", "selectionType", "snapshotId", "suffix", "text"],
@@ -26,8 +30,10 @@ enum NativeProtocol {
static let stringEnums: [String: [String: Set<String>]] = [
"click": ["modifiers": ["command", "control", "fn", "option", "shift"], "mouseButton": ["left", "middle", "right"]],
"drag": ["modifiers": ["command", "control", "fn", "option", "shift"], "mouseButton": ["left", "middle", "right"]],
"openPermissionSettings": ["permission": ["accessibility", "screenRecording"]],
"paste": ["format": ["html", "text"]],
"pressKey": ["modifiers": ["command", "control", "fn", "option", "shift"]],
"requestPermission": ["permission": ["accessibility", "screenRecording"]],
"selectText": ["selectionType": ["cursor_after", "cursor_before", "text"]],
]
}
@@ -0,0 +1,69 @@
// System consent is requested only by the Host's user-facing permission controls.
import AppKit
@preconcurrency import ApplicationServices
import Darwin
import Foundation
@MainActor struct SystemPermissions {
private let launchApplication: JSONValue?
// Capture the system-reported responsible app before startup completes and the launcher can exit.
init() { launchApplication = Self.responsibleApplication(for: getpid()) }
func status() -> JSONValue {
var result: [String: JSONValue] = [
"accessibility": .bool(AXIsProcessTrusted()),
"screenRecording": .bool(CGPreflightScreenCaptureAccess()),
"helper": Self.process(getpid()),
"parent": Self.process(getppid()),
]
if let launchApplication { result["launchApplication"] = launchApplication }
return .object(result)
}
func request(_ permission: String) throws -> JSONValue {
switch permission {
case "accessibility":
if !AXIsProcessTrusted() {
let options = [kAXTrustedCheckOptionPrompt.takeUnretainedValue() as String: true] as CFDictionary
_ = AXIsProcessTrustedWithOptions(options)
}
case "screenRecording":
if !CGPreflightScreenCaptureAccess() { _ = CGRequestScreenCaptureAccess() }
default: throw RequestParameters.invalid("permission must be accessibility or screenRecording")
}
return status()
}
func openSettings(_ permission: String) throws -> JSONValue {
let pane: String
switch permission {
case "accessibility": pane = "Privacy_Accessibility"
case "screenRecording": pane = "Privacy_ScreenCapture"
default: throw RequestParameters.invalid("permission must be accessibility or screenRecording")
}
guard let url = URL(string: "x-apple.systempreferences:com.apple.preference.security?\(pane)"),
NSWorkspace.shared.open(url) else {
throw NativeFailure(code: "SETTINGS_UNAVAILABLE", message: "Open System Settings > Privacy & Security manually")
}
return .object(["opened": .bool(true)])
}
/// Resolve macOS process responsibility, including launchers outside the nearest GUI application.
static func responsibleApplication(for pid: Int32) -> JSONValue? {
guard let responsible = NativePlatform.responsiblePID(pid),
let app = NSRunningApplication(processIdentifier: responsible), !app.isTerminated,
let bundle = app.bundleURL else { return nil }
var result: [String: JSONValue] = ["name": .string(bundle.lastPathComponent), "path": .string(bundle.path)]
if let icon = ApplicationCatalog.icon(at: bundle.path) { result["icon"] = .string(icon) }
return .object(result)
}
private static func process(_ pid: Int32) -> JSONValue {
let path = NativePlatform.executablePath(pid)
var result: [String: JSONValue] = ["pid": .number(Double(pid))]
if let path { result["path"] = .string(path) }
if let name = NSRunningApplication(processIdentifier: pid)?.localizedName { result["name"] = .string(name) }
return .object(result)
}
}
@@ -26,6 +26,13 @@ struct RequestParameters: Sendable {
guard let value = values[key]?.string else { throw Self.invalid("\(key) must be a string") }
return value
}
func strings(_ key: String) throws -> [String] {
guard let raw = values[key], case .array(let items) = raw else { throw Self.invalid("\(key) must be a string array") }
return try items.map { value in
guard let path = value.string, !path.isEmpty else { throw Self.invalid("\(key) must contain nonempty strings") }
return path
}
}
func optionalString(_ key: String) throws -> String? { contains(key) ? try string(key) : nil }
func optionalBoolean(_ key: String) throws -> Bool? {
guard let raw = values[key] else { return nil }
@@ -9,6 +9,7 @@ import Darwin
private let cancellation: CancellationLatch
private let worker: AXWorker
private let capture: WindowCapture
private let permissions: SystemPermissions
private let clipboard = ClipboardPaste()
private let input: JSONLInput
private let writer: JSONLWriter
@@ -27,11 +28,12 @@ import Darwin
private var nextCancelled = false
init(configuration: HelperConfiguration, capture: WindowCapture, channel: SharedSocket,
cancellation: CancellationLatch, scheduler: RequestScheduler, writer: JSONLWriter,
cancellation: CancellationLatch, scheduler: RequestScheduler, writer: JSONLWriter, permissions: SystemPermissions,
onClose: @escaping @MainActor @Sendable () -> Void,
onFatal: @escaping @Sendable (NativeFailure, Double) -> Void) {
self.configuration = configuration
self.capture = capture
self.permissions = permissions
self.cancellation = cancellation
self.channel = channel
self.scheduler = scheduler
@@ -159,11 +161,25 @@ import Darwin
}
private func execute(_ request: NativeRequest, trace: ScreenshotTrace?) async throws -> JSONValue {
let params = try RequestParameters(request)
try cancellation.check()
if request.method == "screenshot" { try capture.requireAvailable() }
let requiresCatalog = ["getApp", "listApps", "findApps", "typeText", "pressKey"].contains(request.method)
let catalog = requiresCatalog ? ApplicationCatalog.snapshot() : ApplicationCatalogSnapshot(applications: [], frontPID: nil)
let result: JSONValue
if request.method == "findApps" {
if request.method == "permissionStatus" {
result = permissions.status()
} else if request.method == "applicationIcons" {
var icons: [String: JSONValue] = [:]
for path in try params.strings("paths") {
try cancellation.check()
if let icon = ApplicationCatalog.icon(at: path) { icons[path] = .string(icon) }
}
result = .object(icons)
} else if request.method == "requestPermission" {
result = try permissions.request(params.string("permission"))
} else if request.method == "openPermissionSettings" {
result = try permissions.openSettings(params.string("permission"))
} else if request.method == "findApps" {
result = .array(try ApplicationDiscovery.applications(matching: params.string("query"), exact: false,
limit: params.integer("maxAppMatches", in: 1...1000), cancellation: cancellation).map { $0.json(catalog: catalog) })
} else {
@@ -67,6 +67,29 @@ final class NativeSignals {
deinit { sources.forEach { $0.cancel() } }
}
// Blocking accepts own only Sendable state; server access belongs to the MainActor callback.
private func acceptConnections(listener: SharedSocket, lifetime: SharedHelperLifetime,
onAccept: @escaping @MainActor @Sendable (UUID, SharedSocket, CancellationLatch) -> Void) {
while !lifetime.isStopping {
let descriptor = accept(listener.descriptor, nil, nil)
if descriptor < 0 && errno == EINTR { continue }
guard descriptor >= 0 else {
if !lifetime.isStopping { lifetime.abort(grace: lifetime.shutdownGrace) }
return
}
let channel = SharedSocket(descriptor: descriptor)
var uid: uid_t = 0
var gid: gid_t = 0
guard getpeereid(descriptor, &uid, &gid) == 0, uid == getuid() else { channel.shutdown(); continue }
let id = UUID()
let cancellation = CancellationLatch()
guard lifetime.add(id, channel: channel, cancellation: cancellation) else { channel.shutdown(); return }
Task { @MainActor [onAccept, id, channel, cancellation] in
onAccept(id, channel, cancellation)
}
}
}
@MainActor final class SharedHelperServer {
private let options: SharedHelperOptions
private let versionLock: SharedHelperLock
@@ -75,6 +98,7 @@ final class NativeSignals {
private let lifetime: SharedHelperLifetime
private let scheduler = RequestScheduler()
private let capture = WindowCapture()
private let permissions = SystemPermissions()
private let instanceID = UUID().uuidString
private var connections: [UUID: RequestCoordinator] = [:]
private var startupDeadline: NativeDeadline?
@@ -99,12 +123,18 @@ final class NativeSignals {
func start() throws {
let lifetime = self.lifetime
signals = NativeSignals { [weak self] number in
lifetime.abort(grace: lifetime.shutdownGrace, status: 128 + number)
Task { @MainActor in self?.closeConnections(); self?.exitIfEmpty() }
let onStop: @MainActor @Sendable () -> Void = { [weak self] in
guard let self else { return }
self.closeConnections()
self.exitIfEmpty()
}
startupDeadline = NativeDeadline(seconds: options.startupTimeout) { [weak self] in
Task { @MainActor in self?.exitIfEmpty() }
signals = NativeSignals { [lifetime, onStop] number in
lifetime.abort(grace: lifetime.shutdownGrace, status: 128 + number)
Task { @MainActor [onStop] in onStop() }
}
let onIdle: @MainActor @Sendable () -> Void = { [weak self] in self?.exitIfEmpty() }
startupDeadline = NativeDeadline(seconds: options.startupTimeout) { [onIdle] in
Task { @MainActor [onIdle] in onIdle() }
}
let ready = options.ready(pid: getpid(), instanceID: instanceID)
let data = try JSONEncoder().encode(ready)
@@ -113,38 +143,45 @@ final class NativeSignals {
try FileHandle.standardOutput.write(contentsOf: data + Data([10]))
try FileHandle.standardOutput.close()
let listener = self.listener
DispatchQueue.global(qos: .utility).async { [weak self] in
while !lifetime.isStopping {
let descriptor = accept(listener.descriptor, nil, nil)
if descriptor < 0 && errno == EINTR { continue }
guard descriptor >= 0 else {
if !lifetime.isStopping { lifetime.abort(grace: lifetime.shutdownGrace) }
return
}
let channel = SharedSocket(descriptor: descriptor)
var uid: uid_t = 0
var gid: gid_t = 0
guard getpeereid(descriptor, &uid, &gid) == 0, uid == getuid() else { channel.shutdown(); continue }
let id = UUID()
let cancellation = CancellationLatch()
guard lifetime.add(id, channel: channel, cancellation: cancellation) else { channel.shutdown(); return }
Task { @MainActor in self?.admit(id, channel: channel, cancellation: cancellation) }
let onAccept: @MainActor @Sendable (UUID, SharedSocket, CancellationLatch) -> Void = { [weak self] id, channel, cancellation in
guard let self else {
channel.shutdown()
lifetime.remove(id)
return
}
self.admit(id, channel: channel, cancellation: cancellation)
}
DispatchQueue.global(qos: .utility).async { [listener, lifetime, onAccept] in
acceptConnections(listener: listener, lifetime: lifetime, onAccept: onAccept)
}
}
private func admit(_ id: UUID, channel: SharedSocket, cancellation: CancellationLatch) {
guard !lifetime.isStopping else { channel.shutdown(); disconnected(id); return }
let options = self.options
DispatchQueue.global(qos: .utility).async { [weak self] in
let lifetime = self.lifetime
let onHello: @MainActor @Sendable (HelperConfiguration) async -> Void = { [weak self] configuration in
guard let self else {
channel.shutdown()
lifetime.remove(id)
return
}
await self.connected(id, channel: channel, cancellation: cancellation, configuration: configuration)
}
let onFailure: @MainActor @Sendable () -> Void = { [weak self] in
lifetime.remove(id)
self?.exitIfEmpty()
}
DispatchQueue.global(qos: .utility).async { [options, channel, onHello, onFailure] in
do {
let configuration = try HelperConfiguration(hello: channel.readHello(timeout: options.startupTimeout), runtimeKey: options.runtimeKey)
Task { @MainActor in await self?.connected(id, channel: channel, cancellation: cancellation, configuration: configuration) }
Task { @MainActor [onHello, configuration] in await onHello(configuration) }
} catch {
let failure = error as? NativeFailure ?? NativeFailure(code: "NATIVE_PROTOCOL", message: "Invalid shared helper hello")
let response = JSONValue.object(["type": .string("error"), "code": .string(failure.code), "message": .string(failure.message)])
if let data = try? JSONEncoder().encode(response) { try? channel.write(data + Data([10]), timeout: options.startupTimeout) }
channel.shutdown()
Task { @MainActor in self?.disconnected(id) }
Task { @MainActor [onFailure] in onFailure() }
}
}
}
@@ -155,14 +192,18 @@ final class NativeSignals {
let writer = JSONLWriter(limit: configuration.maxResponseBytes, channel: channel, writeTimeout: configuration.shutdownGraceSeconds)
do { try await writer.write(options.ready(pid: getpid(), instanceID: instanceID)) }
catch { channel.shutdown(); disconnected(id); return }
// Signal shutdown can run while the ready frame is being written.
guard !lifetime.isStopping else { channel.shutdown(); disconnected(id); return }
let lifetime = self.lifetime
let onFatal: @MainActor @Sendable () -> Void = { [weak self] in self?.closeConnections() }
let coordinator = RequestCoordinator(configuration: configuration, capture: capture, channel: channel, cancellation: cancellation,
scheduler: scheduler, writer: writer, onClose: { [weak self] in
scheduler: scheduler, writer: writer, permissions: permissions, onClose: { [weak self] in
channel.shutdown()
self?.disconnected(id)
}, onFatal: { [weak self] _, grace in
guard let self else { lifetime.remove(id); return }
self.disconnected(id)
}, onFatal: { [lifetime, onFatal] _, grace in
lifetime.abort(grace: grace)
Task { @MainActor in self?.closeConnections() }
Task { @MainActor [onFatal] in onFatal() }
})
connections[id] = coordinator
startupDeadline?.cancel()
@@ -1037,6 +1037,131 @@
"params"
],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"id": {
"type": "string"
},
"method": {
"type": "string",
"const": "permissionStatus"
},
"params": {
"type": "object",
"properties": {},
"additionalProperties": false
}
},
"required": [
"id",
"method",
"params"
],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"id": {
"type": "string"
},
"method": {
"type": "string",
"const": "applicationIcons"
},
"params": {
"type": "object",
"properties": {
"paths": {
"type": "array",
"items": {
"type": "string",
"minLength": 1
}
}
},
"required": [
"paths"
],
"additionalProperties": false
}
},
"required": [
"id",
"method",
"params"
],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"id": {
"type": "string"
},
"method": {
"type": "string",
"const": "requestPermission"
},
"params": {
"type": "object",
"properties": {
"permission": {
"type": "string",
"enum": [
"accessibility",
"screenRecording"
]
}
},
"required": [
"permission"
],
"additionalProperties": false
}
},
"required": [
"id",
"method",
"params"
],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"id": {
"type": "string"
},
"method": {
"type": "string",
"const": "openPermissionSettings"
},
"params": {
"type": "object",
"properties": {
"permission": {
"type": "string",
"enum": [
"accessibility",
"screenRecording"
]
}
},
"required": [
"permission"
],
"additionalProperties": false
}
},
"required": [
"id",
"method",
"params"
],
"additionalProperties": false
}
]
}
@@ -0,0 +1,29 @@
// An owned GUI launcher with no windows; its child outlives the launch application.
import AppKit
import Foundation
import Darwin
@MainActor final class Launcher: NSObject, NSApplicationDelegate {
private let child = Process()
func applicationDidFinishLaunching(_ notification: Notification) {
let arguments = Array(CommandLine.arguments.dropFirst())
guard let executable = arguments.first else { exit(64) }
child.executableURL = URL(fileURLWithPath: executable)
child.arguments = Array(arguments.dropFirst())
child.standardOutput = FileHandle.standardOutput
child.standardError = FileHandle.standardError
child.terminationHandler = { @Sendable process in exit(process.terminationStatus) }
do { try child.run() }
catch { fputs("Launching fixture child failed: \(error)\n", stderr); exit(1) }
}
}
@main enum LauncherMain {
@MainActor static func main() {
let app = NSApplication.shared
app.setActivationPolicy(.accessory)
let launcher = Launcher()
app.delegate = launcher
withExtendedLifetime(launcher) { app.run() }
}
}
@@ -0,0 +1,76 @@
"""Verify system responsibility across nested GUI launchers and helper reuse; no TCC grants are required."""
import argparse
import base64
from contextlib import ExitStack
import hashlib
import json
from pathlib import Path
import plistlib
import shutil
import struct
import subprocess
import tempfile
import uuid
from native_test_support import BINARY, ROOT
from shared_helper_e2e import Connection, wait_released
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--helper", type=Path, default=BINARY)
args = parser.parse_args()
with tempfile.TemporaryDirectory(prefix="cua-", dir="/tmp") as temporary, ExitStack() as cleanup:
root = Path(temporary)
helper = root / "helper"
shutil.copy2(args.helper.resolve(strict=True), helper)
key = hashlib.sha256(helper.read_bytes()).hexdigest()
runtime = root / "r"
bundle = root / "Visual Studio Code.app"
binaries = bundle / "Contents" / "MacOS"
binaries.mkdir(parents=True)
launcher = binaries / "launcher"
subprocess.run(["xcrun", "swiftc", "-parse-as-library", str(ROOT / "tests/LauncherFixture.swift"), "-o", str(launcher)],
check=True, timeout=60)
nested = root / "Harness Dev.app"
nested_launcher = nested / "Contents" / "MacOS" / "launcher"
nested_launcher.parent.mkdir(parents=True)
shutil.copy2(launcher, nested_launcher)
identity = uuid.uuid4().hex
register = "/System/Library/Frameworks/CoreServices.framework/Frameworks/LaunchServices.framework/Support/lsregister"
for role, application, name in [("owner", bundle, "Code"), ("nested", nested, "Harness Dev")]:
with (application / "Contents" / "Info.plist").open("wb") as output:
plistlib.dump({"CFBundleExecutable": "launcher",
"CFBundleIdentifier": f"dev.dsh.cua.launcher-fixture.{role}-{identity}",
"CFBundleName": name, "CFBundleDisplayName": name, "CFBundlePackageType": "APPL",
"LSUIElement": True}, output)
cleanup.callback(subprocess.run, [register, "-u", str(application)], check=True,
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, timeout=10)
ready_file, stderr_file = root / "ready.json", root / "stderr.log"
# Launch Services establishes the outer app's responsibility; its GUI child inherits it.
subprocess.run(["/usr/bin/open", "-n", "-g", "-W", "--stdout", str(ready_file), "--stderr", str(stderr_file),
str(bundle), "--args", str(nested_launcher), str(helper), "--ensure-shared", "--runtime-key", key,
"--runtime-directory", str(runtime), "--startup-timeout-ms", "10000"],
check=True, capture_output=True, text=True, timeout=25)
ready = json.loads(ready_file.read_text())
owner = ready["pid"]
cleanup.callback(wait_released, runtime / (key + ".lock"), owner)
first = Connection(runtime / (key + ".sock"), key)
cleanup.callback(first.close)
state = first.call("permissionStatus", {})
assert state["parent"]["pid"] == 1, state["parent"]
source = state["launchApplication"]
assert source["name"] == bundle.name, source["name"]
assert Path(source["path"]).resolve() == bundle.resolve(), source["path"]
assert source["icon"].startswith("data:image/png;base64,")
image = base64.b64decode(source["icon"].split(",", 1)[1], validate=True)
assert image[:8] == b"\x89PNG\r\n\x1a\n"
assert struct.unpack(">II", image[16:24]) == (64, 64)
second = Connection(runtime / (key + ".sock"), key)
cleanup.callback(second.close)
assert second.call("permissionStatus", {})["launchApplication"] == source
print("PASS: system-responsible app wins over the nearest GUI ancestor and survives launcher exit and helper reuse")
if __name__ == "__main__":
main()
@@ -5,6 +5,7 @@ Accessibility/Screen Recording grants are required unless --without-desktop is s
"""
import argparse
import concurrent.futures
from contextlib import ExitStack
import fcntl
import hashlib
import json
@@ -128,6 +129,15 @@ def main():
assert a.ready == b.ready == ready[0]
for client in clients:
client.call("protocol-unsupported", {}, error="UNSUPPORTED")
with socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) as invalid:
invalid.settimeout(15)
invalid.connect(str(socket_path))
invalid.sendall(b'{"type":"hello","protocol":2}\n')
with invalid.makefile("rb") as reader:
failure = json.loads(reader.readline())
assert failure["type"] == "error" and failure["code"] == "NATIVE_PROTOCOL", failure
assert reader.readline() == b"", "Rejected handshake retained its connection"
b.call("protocol-after-invalid-hello", {}, error="UNSUPPORTED")
# Coalesced frames retain socket order, including cancellation before execution starts.
a.socket.sendall(b'{"id":"coalesced","method":"listApps","params":{}}\n'
b'{"type":"cancel","id":"coalesced"}\n')
@@ -218,20 +228,39 @@ def main():
wait_released(lock_path, owner_pid)
owner_pid = None
daemon = subprocess.Popen([base[0], "--serve", *base[1:]], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
try:
assert select.select([daemon.stdout], [], [], 15)[0], "Daemon did not report ready"
signalled_ready = json.loads(daemon.stdout.readline())
assert signalled_ready["type"] == "ready", signalled_ready
daemon.send_signal(signal.SIGTERM)
status = daemon.wait(timeout=15)
assert status == 128 + signal.SIGTERM, (status, daemon.stderr.read())
wait_released(lock_path, daemon.pid)
finally:
stop(daemon)
daemon.stdout.close()
daemon.stderr.close()
print("PASS: concurrent launch, shared PID, client isolation, cleanup/lock release and SIGTERM")
for shutdown in ["signal", "incomplete-hello", "idle"]:
serve_options = base[1:] if shutdown != "idle" else [*base[1:-1], "100"]
daemon = subprocess.Popen([base[0], "--serve", *serve_options], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
try:
with ExitStack() as connections:
assert select.select([daemon.stdout], [], [], 15)[0], "Daemon did not report ready"
signalled_ready = json.loads(daemon.stdout.readline())
assert signalled_ready["type"] == "ready", signalled_ready
if shutdown == "incomplete-hello":
pending = connections.enter_context(socket.socket(socket.AF_UNIX, socket.SOCK_STREAM))
pending.settimeout(15)
pending.connect(str(socket_path))
pending.sendall(b'{"type":"hello"')
# A later completed handshake observes the accept loop progressing past this socket.
active = Connection(socket_path, key)
connections.callback(active.socket.close)
connections.callback(active.reader.close)
if shutdown != "idle":
daemon.send_signal(signal.SIGTERM)
status = daemon.wait(timeout=15)
expected_status = 0 if shutdown == "idle" else 128 + signal.SIGTERM
assert status == expected_status, (status, daemon.stderr.read())
wait_released(lock_path, daemon.pid)
# The forced-exit deadline does not unlink these files; require normal cleanup.
assert not socket_path.exists(), "Shutdown left its listening socket"
assert not (runtime / (key + ".ready")).exists(), "Shutdown left its ready record"
if shutdown == "incomplete-hello":
assert pending.recv(1) == b"", "Signal shutdown retained an incomplete handshake"
finally:
stop(daemon)
daemon.stdout.close()
daemon.stderr.close()
print("PASS: concurrent launch, shared PID, client isolation, rejected/incomplete handshakes, cleanup/lock release, SIGTERM and idle exit")
finally:
errors = []
for client in reversed(clients):
@@ -1,5 +1,7 @@
// The selected catalog entry must still identify the executable inspected for consent.
import AppKit
import Darwin
import Foundation
import Testing
@testable import CuaNative
@@ -22,3 +24,37 @@ import Testing
#expect(error.code == "APP_UNAVAILABLE")
}
}
@MainActor @Test func applicationIconsUseBoundedNativePNGs() throws {
let encoded = try #require(ApplicationCatalog.icon(at: "/System/Library/CoreServices/Finder.app/Contents/MacOS/Finder"))
#expect(encoded.hasPrefix("data:image/png;base64,"))
let data = try #require(Data(base64Encoded: String(encoded.dropFirst("data:image/png;base64,".count))))
let bitmap = try #require(NSBitmapImageRep(data: data))
#expect(bitmap.pixelsWide == 64)
#expect(bitmap.pixelsHigh == 64)
#expect(encoded.utf8.count <= 65536)
}
@Test func resolvesTheCurrentProcessResponsibility() throws {
let responsible = try #require(NativePlatform.responsiblePID(getpid()))
#expect(responsible > 0)
#expect(NativePlatform.responsiblePID(responsible) == responsible)
}
@MainActor @Test func unknownResponsibilityDoesNotInventAnApplication() {
#expect(SystemPermissions.responsibleApplication(for: 1) == nil)
#expect(SystemPermissions.responsibleApplication(for: Int32.max) == nil)
#expect(NativePlatform.responsiblePID(Int32.max) == nil)
}
@MainActor @Test func permissionStatusKeepsProcessDiagnosticsSeparateFromLaunchMetadata() {
let permissions = SystemPermissions()
let state = permissions.status()
#expect(state["helper"]["pid"] == .number(Double(getpid())))
#expect(state["parent"]["pid"] == .number(Double(getppid())))
if let source = state["launchApplication"].object {
#expect(source["name"]?.string?.isEmpty == false)
#expect(source["path"]?.string?.hasSuffix(".app") == true)
}
#expect(permissions.status()["launchApplication"] == state["launchApplication"])
}
@@ -116,3 +116,9 @@ func rejectsInvalidWindowIDs(_ value: String) throws {
#expect(throws: NativeFailure.self) { try params.optionalBoolean("includeMenus") }
}
}
@Test(arguments: ["null", "true", "[1]", "[\"\"]"])
func rejectsMalformedApplicationIconPaths(_ value: String) throws {
let request = try NativeRequest(data: Data("{\"id\":\"icons\",\"method\":\"applicationIcons\",\"params\":{\"paths\":\(value)}}".utf8))
#expect(throws: NativeFailure.self) { try RequestParameters(request).strings("paths") }
}
+4 -1
View File
@@ -84,5 +84,8 @@
},
"scripts": {
"build:native": "swift build --package-path native --configuration release"
}
},
"os": [
"darwin"
]
}
@@ -1,6 +1,6 @@
/** macOS operation guidance supplied with the shared CUA object API. */
/** Instructions for raw operation calls and optional object-API bindings. */
export const GUIDANCE = `Use cua with method getCapabilities and params {} to inspect permissions and sessionLocked. If sessionLocked is true or an operation reports SESSION_LOCKED, stop UI attempts and ask the user to unlock macOS. A null sessionLocked means no usable console session was verified. After unlocking, read fresh state; screenshots from a locked desktop are not current observations. Use listApps or cua.getState to discover running applications, and findApps(query) to search installed applications by name, bundle ID or absolute path. getApp requests user consent before launching a stopped application in the background, or before observations and input on a running one. If access is denied or revoked, stop using that application until the user explicitly grants access again. Never switch tools to bypass a refusal. Keep the returned appHandle; every app operation requires it. listWindows returns window IDs and point bounds. Raw getState returns AX text and snapshotId. Menu-bar entries are shown by default; includeMenus=true expands their contents. details=true adds identifiers, global-point bounds and complete action names. Both options also apply to getAXState. Each read replaces the indices for that app/window scope. Use its element indices only with that snapshotId. Raw screenshot and coordinate operations require a windowId. Coordinates are pixels in the latest returned screenshot, measured from its top-left. Set relative:true for integer thousandths from 0 to 1000 instead. The Host converts both modes to native window points; AX detail bounds remain global macOS points. A verified window can expose no AX elements. Its empty snapshot permits screenshots and coordinate input; element-index actions require an element from a fresh AX observation.
export const GUIDANCE = `Use cua with method getCapabilities and params {} to inspect permissions and sessionLocked. If sessionLocked is true or an operation reports SESSION_LOCKED, stop UI attempts and ask the user to unlock macOS. A null sessionLocked means no usable console session was verified. After unlocking, read fresh state; screenshots from a locked desktop are not current observations. Use listApps or cua.getState to discover running applications, and findApps(query) to search installed applications by name, bundle ID or absolute path. getApp requests user consent before launching a stopped application in the background, or before observations and input on a running one. If access is denied or revoked, stop using that application until the user explicitly permits another authorization request. Never switch tools to bypass a refusal. Keep the returned appHandle; every app operation requires it. listWindows returns window IDs and point bounds. Raw getState returns AX text and snapshotId. Menu-bar entries are shown by default; includeMenus=true expands their contents. details=true adds identifiers, global-point bounds and complete action names. Both options also apply to getAXState. Each read replaces the indices for that app/window scope. Use its element indices only with that snapshotId. Raw screenshot and coordinate operations require a windowId. Coordinates are pixels in the latest returned screenshot, measured from its top-left. Set relative:true for integer thousandths from 0 to 1000 instead. The Host converts both modes to native window points; AX detail bounds remain global macOS points. A verified window can expose no AX elements. Its empty snapshot permits screenshots and coordinate input; element-index actions require an element from a fresh AX observation.
Initialize the object API once in persistent PTC. Reuse its variables in later calls; put temporary declarations inside { ... } blocks to avoid redeclaration errors. All object methods are async. getAXState returns text, not a structured state object. Its text separates values and placeholders, and reports subroles, help, URLs, disabled/focused/selected/expanded state and UTF-16 selection ranges when available. Writability is true, false or unknown on relevant controls; omitted state fields do not assert false. selection=unavailable means the reported range exceeds the observed text. Compact output omits ordinary Press/Pick (click), ShowMenu (right click) and ScrollToVisible actions; details=true lists every available action. Use the displayed action name with performSecondaryAction; ambiguous aliases require the complete name from details=true. Truncation reports the reached provider limit. Narrow the window or increase that provider limit before reading again.
@@ -37,7 +37,7 @@ App and Window methods:
Raw operation tools use { method, params }, including params: {} for parameterless methods. getApp selects without activating the app. Keep applications in the background for observation and routine input. Do not activate or raise an app during initialization, before each action, or merely because another app is foreground. Prefer advertised AX actions. If a window is hidden or minimized, try app.show() or window.show() and read fresh state. show requests unhide and unminimize without activation, raising or switching Spaces; it chooses the explicit, focused, main or sole AX window. A null windowId means no window was selected for restoration. foregroundChanged compares the front app and top window before and after the request; null means unavailable, and delayed focus changes remain possible. If restoration is unsupported or the window remains unusable, including another Space or Stage Manager, explain the needed foreground step, use app.activate() or an advertised raise action, and read fresh state. Completing the task takes priority when foreground access is necessary. Avoid repeated activation when the user switches away; continue background-capable steps and report what requires foreground access. Applications may still change focus in response to input.
Await operations sequentially. After input, read fresh state to verify the effect. A successful action means dispatched, not verified completion. Cancellation may leave partial effects; never retry uncertain input without observing. STALE_APP requires getApp again; STALE_SNAPSHOT requires getState again. Absolute coordinate input requires a fresh screenshot of that window. Relative input can use a successful AX observation or screenshot. SCREENSHOT_REQUIRED means capture a fresh screenshot before using pixel coordinates. For click and hover, coordinate targets must be [x, y], for example hover([650, 400], {relative:true}). A bare number such as hover(650) is always an AX element index. The relative option applies only to coordinate targets, not AX indices. Window resizing since the latest observation rejects coordinate input; movement alone preserves window-local coordinates. WINDOW_CHANGED requires fresh state before any retry; use an advertised AX action or explicitly restore the window before screenshot or coordinate input, then observe again. PERMISSION_DENIED requires the user to grant macOS Accessibility or Screen Recording access in System Settings. After granting access, observe fresh state before continuing; do not automatically replay prior input.
Await operations sequentially. After input, read fresh state to verify the effect. A successful action means dispatched, not verified completion. Cancellation may leave partial effects; never retry uncertain input without observing. STALE_APP requires getApp again; STALE_SNAPSHOT requires getState again. Absolute coordinate input requires a fresh screenshot of that window. Relative input can use a successful AX observation or screenshot. SCREENSHOT_REQUIRED means capture a fresh screenshot before using pixel coordinates. For click and hover, coordinate targets must be [x, y], for example hover([650, 400], {relative:true}). A bare number such as hover(650) is always an AX element index. The relative option applies only to coordinate targets, not AX indices. Window resizing since the latest observation rejects coordinate input; movement alone preserves window-local coordinates. WINDOW_CHANGED requires fresh state before any retry; use an advertised AX action or explicitly restore the window before screenshot or coordinate input, then observe again. PERMISSION_DENIED requires the user to grant the required macOS system permissions for macOS Accessibility or Screen Recording. After granting access, observe fresh state before continuing; do not automatically replay prior input.
Hover first uses advertised AX alternate UI without moving the system cursor, preserving revealed descendants until the pointer target leaves their region. When unavailable, it moves the visible system cursor without pressing a button; that route requires the target point to be unobscured in the foreground app. FOREGROUND_REQUIRED requires explicit activation or raising and a fresh observation before retrying. Returning, AX reads and screenshots do not restore the previous point. Wait for the application's hover delay, observe, then hover into the revealed controls before clicking. Use successive hover calls for intermediate points. The result reports hoverRoute and systemPointerMoved. Other input or application changes can interrupt the interaction; app objects do not reserve applications between calls.
@@ -73,7 +73,7 @@ it.skipIf(process.platform === 'win32')('delivers a valid PNG larger than 4 MiB
const configPath = join(root, 'cordis.yml')
const configs: Record<string, object> = {
tools: { mode: 'ptc' }, attachments: { dshHome: join(root, 'home') }, workingDirectory: { defaultDirectory: root },
fs: { cwd: root }, sandboxPolicy: { mode: 'danger-full-access', workspaceRoot: root }, cua: { helperPath, allowedApps: [{ 'id': 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa', 'name': 'Fixture', 'executablePath': '/fixture/app' }] },
fs: { cwd: root }, sandboxPolicy: { mode: 'danger-full-access', workspaceRoot: root }, cua: { agentPreset: false, helperPath, allowedApps: [{ 'id': 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa', 'name': 'Fixture', 'executablePath': '/fixture/app' }] },
}
await writeFile(configPath, Object.keys(modules).flatMap(name => [
`- name: 'cordis:${name}'`, ...configs[name] === undefined ? [] : [` config: ${JSON.stringify(configs[name])}`],
Loaded 100 of 171 files, more files were not shown because too many files have changed in this diff. Show more