feat(cua): add the experimental Computer Use bundle

2026-10-17 16:35:13 +08:00
1 parent aa786c98d4
commit 6ae9ce5e2f
@@ -0,0 +1,46 @@
+# Bilingual-pair consistency record for README.md (docs/i18n/README.md): per heading
+# section, a hash of its English and Chinese blocks outside code blocks and generated regions.
+# After editing either side, bring the other along and re-record with:
+#   pnpm run verify-translation-pairing --write packages/experimental/cua-bundle/README.md
+/:
+  en: 6a4efed7f053f52c
+  zh: 9135d38515947b44
+/deepseek-ai-dsh-cua-bundle:
+  en: 9084fb48d0f6489c
+  zh: ce692972d91cf663
+/deepseek-ai-dsh-cua-bundle/summary:
+  en: 06d8bf4ee1ef6f3b
+  zh: 8568f649fb846615
+/deepseek-ai-dsh-cua-bundle/table-of-contents:
+  en: b563e4ec4727d0de
+  zh: d0228da978238e05
+/deepseek-ai-dsh-cua-bundle/use-this-package:
+  en: 5b315d6902074bdd
+  zh: b904b76441f44088
+/deepseek-ai-dsh-cua-bundle/understand-the-implementation:
+  en: 1d592dc9112502fe
+  zh: a922cd0878b6c27d
+/deepseek-ai-dsh-cua-bundle/further-exploration:
+  en: 9f15d56b5b16e1ca
+  zh: 75eb08e8828c8877
+/deepseek-ai-dsh-cua-bundle/model-experience:
+  en: 215e7ba838619b7b
+  zh: a311da3843709f90
+/deepseek-ai-dsh-cua-bundle/model-experience/computer-use-and-persistent-ptc:
+  en: b932589e03b6ff18
+  zh: b932589e03b6ff18
+/deepseek-ai-dsh-cua-bundle/model-experience/computer-use-and-persistent-ptc/what-the-model-sees:
+  en: 591af735d1eb92f6
+  zh: 3632bffa33c77b28
+/deepseek-ai-dsh-cua-bundle/model-experience/computer-use-and-persistent-ptc/token-effect:
+  en: 09a8fd5d9ab27cea
+  zh: 05de759d1f4f33c5
+/deepseek-ai-dsh-cua-bundle/model-experience/computer-use-and-persistent-ptc/kv-cache-effect:
+  en: 417c40acd93df532
+  zh: 69cc5cf7140c75a9
+/deepseek-ai-dsh-cua-bundle/known-limitations-and-deferred-work:
+  en: 8ddfd9245748b7c3
+  zh: 52015ab010abd0f8
+/deepseek-ai-dsh-cua-bundle/known-limitations-and-deferred-work/dev-note:
+  en: 0a1253c6f7b5e8b9
+  zh: ce96645b3e75874e
@@ -0,0 +1,94 @@
+---
+description: "Enable experimental Computer Use and persistent PTC from the plugin page."
+kind: "package-bundle"
+---
+
+# @deepseek-ai/dsh-cua-bundle
+
+English | [中文](README.zh.md)
+
+## Summary
+
+Enable macOS Computer Use from Plugins for existing and new PTC sessions. Windows support is in development. This bundle remains in `experimental` until that support is complete, then moves to the `computer-use` group. It is off by default; disabled profiles prepare no native components and register neither CUA tools nor their model guidance.
+
+## Table of Contents
+
+- [Use this package](#use-this-package)
+- [Understand the implementation](#understand-the-implementation)
+- [Further Exploration](#further-exploration)
+- [Model Experience](#model-experience)
+- [Known Limitations and Deferred Work](#known-limitations-and-deferred-work)
+
+-----
+
+<a id="use-this-package"></a>
+## Use this package
+
+This workflow targets developers running macOS Web or unpackaged Desktop from source. Open Plugins → Experimental plugins → Computer Use and enable the switch. The permission page identifies the responsible application reported by macOS when the optional system query is available, and guides the user through requesting access, enabling the matching system switch, and automatically refreshing on return. Follow the application named by macOS if it differs from the detected source. Persistent PTC retains variables across calls under the selected file sandbox policy. For Web, run these commands from the repository root:
+
+```sh
+pnpm run build
+pnpm dsh web
+```
+
+For the source Desktop launcher and its separate development profile, follow [Desktop development](../../../apps/desktop/README.md). Enable Computer Use in each profile that needs it.
+
+The switch saves bundle selection in the profile’s `package.json`. Enabling configures persistent PTC in the user’s `cordis.patch.yml`; matching values are left unchanged. Computer Use is designed for PTC and installed only in PTC sessions. When system permissions need attention, activation guidance opens the plugin details. The detail page offers **Set to PTC** for the new-session default; existing sessions keep their preset. HMR maintains these settings and the provider’s session tool registrations. Native components prepare on activation and source startup; subsequent `pnpm run build` checks the selected Web profile’s incremental native build. First preparation requires local compiler tools, and failure preserves diagnostics without publishing a new executable.
+
+Disabling unloads the provider and withdraws CUA tools from its Agent scopes. It retains the selected default, runtime configuration and retained PTC variables. Enabling again does not recreate a runtime already configured as persistent. A first change from per-call execution replaces the shared runtime through HMR and can interrupt an active PTC call.
+
+Computer Use is available only on a macOS DSH Host. Other Hosts refuse enablement before saving settings. An already selected bundle is skipped at startup with a diagnostic and remains available for deselection; saved user settings remain. A browser on another operating system can connect to a supported Mac Host.
+
+If preparation fails, the saved selection and PTC settings remain, while CUA operations stay unavailable. Refresh or reopen the plugin page to read the component diagnostic, correct its cause, then restart DSH.
+
+Explicit activation runs a read-only permission check, which may start a helper without opening a system dialog. Open the [permission page](../client-ui-cua/README.md) for system requests, settings navigation and automatic refresh. Tasks ask separately before accessing an application. Saved application decisions use the existing profile configuration; session decisions remain in memory across turns and connections until DSH restarts or Computer Use is disabled. Revoke access on the same page.
+
+-----
+
+<a id="understand-the-implementation"></a>
+## Understand the implementation
+
+<details>
+<summary>Maintainer details</summary>
+
+The bundle patch mounts the shared native provider and permission UI with `agentPreset: ptc`. The provider registers tools through existing Agent scopes, so other presets receive no CUA tool or guidance. Preset documents remain unchanged. `configureOnEnable` persists `ui-settings.enabled: true` and `ptc-runtime.executionMode: persistent-in-memory` through [Plugin Manager](../../boot/plugin-manager/README.md). The detail-page action uses the existing settings form to save `agent-preset-registry.selectedDefault: ptc` only when clicked, preserving the deployment `default`. The provider defaults to PTC scope even when a saved grants override omits that field. [CUA](../cua/README.md) owns registration and disposal.
+
+</details>
+
+-----
+
+<a id="further-exploration"></a>
+## Further Exploration
+
+[Computer use](../../../docs/subsystems/computer-use.md)
+
+-----
+
+<a id="model-experience"></a>
+## Model Experience
+
+### Computer Use and persistent PTC
+
+#### What the model sees
+
+Existing and new PTC sessions receive `cua` through the generated SDK and `run_code`, plus CUA guidance. Disabling withdraws these contributions and keeps PTC presentation. Other presets receive no CUA contributions. [CUA](../cua/README.md) and [PTC](../../ptc-runtime/ptc-runtime-node/README.md#persistent-repl) own the interfaces and output semantics.
+
+#### Token effect
+
+CUA activation adds its guidance. Disabling removes that guidance and retains persistent PTC instructions.
+
+#### KV Cache effect
+
+CUA activation and removal change the PTC SDK and system guidance; subsequent requests use the updated prefix. The top-level `run_code` transport remains present.
+
+## Known Limitations and Deferred Work
+
+<a id="known-limitations-and-deferred-work"></a>
+
+- Source Web and Desktop share permission controls on macOS. Packaged Desktop helper distribution, signing, and permission attribution are deferred.
+- PTC execution is shared by the profile. A Host without HMR applies saved changes at restart.
+
+<a id="dev-note"></a>
+### Dev Note
+
+None.
@@ -0,0 +1,94 @@
+---
+description: "从插件页开启实验性 Computer Use 和 persistent PTC。"
+kind: "package-bundle"
+---
+
+# @deepseek-ai/dsh-cua-bundle
+
+[English](README.md) | 中文
+
+## 概述
+
+从插件页为已有和新建 PTC 会话开启 macOS Computer Use。Windows 支持正在开发中。本组合包暂放在 `experimental`，待 Windows 支持完成后迁回 `computer-use` 组。默认关闭，关闭时不准备本机组件、不注册 CUA 工具或添加其模型指引。
+
+## 目录
+
+- [使用本包](#use-this-package)
+- [理解实现](#understand-the-implementation)
+- [进一步探索](#further-exploration)
+- [模型体验](#model-experience)
+- [已知限制与延后工作](#known-limitations-and-deferred-work)
+
+-----
+
+<a id="use-this-package"></a>
+## 使用本包
+
+本流程面向从源码运行 macOS Web 或未打包 Desktop 的开发者。打开侧栏“插件 → 实验性插件 → Computer Use”并启用。权限页通过可选系统查询显示 macOS 报告的责任应用，无法识别时使用通用指引，指引用户申请权限、开启对应系统开关，返回后自动检测。若 macOS 显示其他应用名称，以系统提示为准。persistent PTC 跨调用保留变量，并遵守当前文件沙箱策略。Web 版在仓库根目录运行以下命令：
+
+```sh
+pnpm run build
+pnpm dsh web
+```
+
+源码 Desktop 的启动方式和独立开发配置档见[桌面开发说明](../../../apps/desktop/README.zh.md)。需要使用 Computer Use 的配置档分别开启该功能。
+
+开关将组合包选择保存到 profile 的 `package.json`。开启时在用户 `cordis.patch.yml` 中配置 persistent PTC，已匹配的值不改写。Computer Use 针对 PTC 模式设计，仅安装在 PTC 会话中。系统权限尚未就绪时，启用引导会进入插件详情页。详情页提供“设为 PTC”按钮来修改新会话默认模式；已有会话保留原 preset。HMR 维护这些设置和提供方的会话工具注册。本机组件在启用及源码启动时准备，后续 `pnpm run build` 检查已选中 Web profile 的本机增量构建。首次准备需要本机编译工具，失败保留诊断且不发布新的可执行文件。
+
+关闭会卸载提供方，并从其 Agent 作用域中撤回 CUA 工具。用户选择的默认值、执行器配置和变量均保留。再次开启不会重建已经配置为 persistent 的执行器。首次从 per-call 切换时，HMR 会重载共享执行器，可能中断正在执行的 PTC 调用。
+
+Computer Use 仅支持 macOS DSH Host。其他 Host 会在保存设置前拒绝启用。已选中的组合包会在启动时跳过并报告原因，仍可取消选择；已保存的用户设置保留。其他操作系统上的浏览器可以连接受支持的 Mac Host。
+
+准备失败时，已保存的选择和 PTC 设置保留，CUA 操作暂不可用。刷新或重新打开插件页查看组件诊断，修复后重启 DSH。
+
+显式启用会进行只读权限检查，可能启动 helper，但不会打开系统授权弹窗。[权限页面](../client-ui-cua/README.zh.md) 提供系统申请入口、设置跳转和自动检测。任务访问应用前会单独询问用户。长期应用授权保存在现有 profile 配置中，会话授权保存在内存中，跨 turn 和连接复用，重启 DSH 或关闭 Computer Use 后失效。可以在同一页面撤销授权。
+
+-----
+
+<a id="understand-the-implementation"></a>
+## 理解实现
+
+<details>
+<summary>维护者说明</summary>
+
+组合包 patch 通过 `agentPreset: ptc` 挂载共享原生提供方和权限界面。提供方在现有 Agent 作用域中注册工具，其他 preset 不获得 CUA 工具或指引，preset 文档保持不变。`configureOnEnable` 通过[插件管理器](../../boot/plugin-manager/README.zh.md)保存 `ui-settings.enabled: true` 和 `ptc-runtime.executionMode: persistent-in-memory`。详情页按钮复用现有设置表单，仅在用户点击后保存 `agent-preset-registry.selectedDefault: ptc`，保留部署级 `default`。即使旧授权配置没有声明作用域字段，提供方也默认只为 PTC 会话注册工具。[CUA](../cua/README.zh.md)负责注册和销毁。
+
+</details>
+
+-----
+
+<a id="further-exploration"></a>
+## 进一步探索
+
+[Computer use](../../../docs/subsystems/computer-use.zh.md)
+
+-----
+
+<a id="model-experience"></a>
+## 模型体验
+
+### Computer Use and persistent PTC
+
+#### What the model sees
+
+已有和新建 PTC 会话通过生成的 SDK 与 `run_code` 使用 `cua`，并获得 CUA 指引。关闭撤回这些贡献，保留 PTC 展示方式。其他 preset 不获得 CUA 贡献。完整接口与输出约定由 [CUA](../cua/README.zh.md) 和 [PTC](../../ptc-runtime/ptc-runtime-node/README.zh.md#persistent-repl) 说明。
+
+#### Token effect
+
+CUA 生效时添加其指引。关闭撤回这些指引，保留 persistent PTC 说明。
+
+#### KV Cache effect
+
+CUA 生效和卸载会改变 PTC SDK 与系统指引，后续请求使用更新后的前缀。顶层 `run_code` 工具保持存在。
+
+## 已知限制与延后工作
+
+<a id="known-limitations-and-deferred-work"></a>
+
+- 源码 Web 与 Desktop 共用 macOS 权限控件。桌面安装包的 helper 分发、签名和权限归属留待后续处理。
+- PTC 执行方式由 profile 共享。未启用 HMR 的 Host 在重启时应用已保存的变更。
+
+<a id="dev-note"></a>
+### 开发备注
+
+None.
@@ -0,0 +1,9 @@
+- insert:
+    - id: cua-provider-registry
+      name: '@deepseek-ai/dsh-computer-use'
+    - id: cua
+      name: '@deepseek-ai/dsh-experimental-cua-macos'
+      config:
+        agentPreset: ptc
+    - id: ui-cua
+      name: '@deepseek-ai/dsh-client-ui-cua'
@@ -0,0 +1,12 @@
+<svg width="36" height="36" viewBox="0 0 36 36" fill="none" xmlns="http://www.w3.org/2000/svg">
+  <circle cx="15.5" cy="15.5" r="8.5" fill="url(#a)" />
+  <circle cx="15.5" cy="15.5" r="5" fill="#FFFFFF" />
+  <path d="M13.2 15.5L14.9 17.2L18.2 13.9" stroke="#145AF3" stroke-width="1.8" stroke-linecap="round" stroke-linejoin="round" />
+  <path d="M22 22L29 29" stroke="#145AF3" stroke-width="3.2" stroke-linecap="round" />
+  <defs>
+    <linearGradient id="a" x1="7" y1="7" x2="24" y2="24" gradientUnits="userSpaceOnUse">
+      <stop stop-color="#7CB7FF" />
+      <stop offset="1" stop-color="#45D9E7" />
+    </linearGradient>
+  </defs>
+</svg>
@@ -0,0 +1,6 @@
+{
+  "meta": {
+    "title": "Computer Use",
+    "description": "macOS only. Windows support is in development. Enabling makes PTC persistent and loads Computer Use for existing and new PTC sessions. Switch to PTC mode and grant Accessibility and Screen Recording permissions to use Computer Use."
+  }
+}
@@ -0,0 +1,6 @@
+{
+  "meta": {
+    "title": "Computer Use",
+    "description": "目前仅支持 macOS，Windows 支持正在开发中。开启会将 PTC 模式改为可持久化，并为已有及新建 PTC 会话加载 Computer Use。请切换到 PTC 模式来使用 Computer Use，并授予辅助功能和屏幕录制权限。"
+  }
+}
@@ -0,0 +1,66 @@
+{
+  "name": "@deepseek-ai/dsh-cua-bundle",
+  "description": "Optional macOS Computer Use with persistent PTC",
+  "version": "0.2.1-rc.2",
+  "icon": "./icon.svg",
+  "publishConfig": {
+    "access": "public"
+  },
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/deepseek-ai/deepseek-harness.git",
+    "directory": "packages/experimental/cua-bundle"
+  },
+  "type": "module",
+  "main": "lib/index.js",
+  "types": "lib/types/index.d.ts",
+  "exports": {
+    ".": {
+      "types": "./lib/types/index.d.ts",
+      "default": "./lib/index.js"
+    },
+    "./cordis.patch.yml": "./cordis.patch.yml",
+    "./locale/*.json": "./locale/*.json",
+    "./src/*": "./src/*",
+    "./package.json": "./package.json"
+  },
+  "files": [
+    "icon.svg",
+    "locale/*.json",
+    "lib/index.js",
+    "cordis.patch.yml",
+    "lib/types/**/*.d.ts"
+  ],
+  "license": "MIT",
+  "dsh": {
+    "bundle": {
+      "patch": "./cordis.patch.yml",
+      "experimental": true,
+      "configureOnEnable": {
+        "ui-settings": {
+          "enabled": true
+        },
+        "ptc-runtime": {
+          "executionMode": "persistent-in-memory"
+        }
+      },
+      "platforms": [
+        "darwin"
+      ]
+    }
+  },
+  "dependencies": {
+    "@deepseek-ai/dsh-experimental-cua-macos": "workspace:*",
+    "@deepseek-ai/dsh-computer-use": "workspace:*",
+    "@deepseek-ai/dsh-ptc-runtime-node": "workspace:*",
+    "@deepseek-ai/dsh-client-ui-cua": "workspace:*"
+  },
+  "peerDependencies": {
+    "@deepseek-ai/cordis": "workspace:~"
+  },
+  "devDependencies": {
+    "@deepseek-ai/cordis": "workspace:~",
+    "@deepseek-ai/cordis-plugin-include": "workspace:~",
+    "js-yaml": "^4.2.0"
+  }
+}
@@ -0,0 +1,2 @@
+/** Optional Computer Use composition; configuration rows live in cordis.patch.yml. */
+export {}
@@ -0,0 +1,15 @@
+{
+  "extends": "../../../tsconfig.base.json",
+  "compilerOptions": {
+    "rootDir": "src",
+    "outDir": "lib/types"
+  },
+  "include": [
+    "src"
+  ],
+  "references": [
+    {
+      "path": "../../../vendor/cordis"
+    }
+  ]
+}