diff --git a/.agents/notes/implemented/architecture/2026-09-21-experimental-capabilities-as-optional-bundles.i18n.yaml b/.agents/notes/implemented/architecture/2026-09-21-experimental-capabilities-as-optional-bundles.i18n.yaml index 448629299d..5d8a536a53 100644 --- a/.agents/notes/implemented/architecture/2026-09-21-experimental-capabilities-as-optional-bundles.i18n.yaml +++ b/.agents/notes/implemented/architecture/2026-09-21-experimental-capabilities-as-optional-bundles.i18n.yaml @@ -9,11 +9,11 @@ en: 2ba6c43b20a3d347 zh: 5647179ddaa38ad9 /agent-note-ship-experimental-capabilities-as-optional-bundles/decision: - en: 6c8583bdefc5e36c - zh: 0928b4f2876bbfe4 + en: d6328ba2b218f2ac + zh: 3731561313b1ad5e /agent-note-ship-experimental-capabilities-as-optional-bundles/alternatives-considered: en: d6f40bf02df37ab7 zh: e5b85ce9d41e8754 /agent-note-ship-experimental-capabilities-as-optional-bundles/consequences: - en: 38d27583533cd028 - zh: 9bf11f5489c1aa43 + en: 3064da835e892958 + zh: bfcf967b6acc6f85 diff --git a/.agents/notes/implemented/architecture/2026-09-21-experimental-capabilities-as-optional-bundles.md b/.agents/notes/implemented/architecture/2026-09-21-experimental-capabilities-as-optional-bundles.md index dbdd104da0..a2d39b0ffb 100644 --- a/.agents/notes/implemented/architecture/2026-09-21-experimental-capabilities-as-optional-bundles.md +++ b/.agents/notes/implemented/architecture/2026-09-21-experimental-capabilities-as-optional-bundles.md @@ -12,12 +12,14 @@ The Web plugin page offered two optional bundles, Agent Teams and voice input. A `OPTIONAL_BUNDLES` owns the installation's default-off selections, including maintained optional capabilities and experimental capabilities. Every bundle declares `icon` and exports `./locale/*.json` with `meta.title` and `meta.description`; the Official group displays that metadata. A bundle's presentation does not determine its [experimental status](../../../../packages/experimental/README.md#status). The isolation check validates admission and still rejects experimental dependencies of nonexperimental bundles. -An optional bundle's dependencies are downloaded with every `dsh` installation. [Reasoning translation](2026-10-05-anonymous-reasoning-translation.md) uses three experimental packages without adding third-party runtime dependencies. Admission therefore considers installation cost and whether a switch supplies a usable composition without an additional configuration form. Browser and computer providers remain explicit compositions because they add substantial runtimes or require an external executable. Stagehand requires model credentials; Python PTC replaces a core provider and conflicts with TypeScript workflow consumers. Demonstrative hooks, webhooks, and mods stay outside the list. [Session Inspector](../feature/2026-09-24-session-inspector.md) owns the Inspector's distinct activation behavior. +An optional bundle's dependencies are downloaded with every `dsh` installation. [Reasoning translation](2026-10-05-anonymous-reasoning-translation.md) uses three experimental packages without adding third-party runtime dependencies. Admission therefore considers installation cost and whether a switch supplies a usable composition without an additional configuration form. External browser and Cua Driver providers remain explicit compositions because they add substantial runtimes or require an external executable. Stagehand requires model credentials; Python PTC replaces a core provider and conflicts with TypeScript workflow consumers. Demonstrative hooks, webhooks, and mods stay outside the list. [Session Inspector](../feature/2026-09-24-session-inspector.md) owns the Inspector's distinct activation behavior. Bundles insert ordinary Host rows; optional model tools register in the global tool layer, so every preset, including minimal, receives them. Plugin management, profile composition, and reload need no bundle-specific behavior. Search, Ralph, and terminals supply their own isolated services in a Host group. Search opens one in-memory index on demand; the Host search policy stays unchanged. Badge skills and title providers act on the Host. Switching a bundle registers or removes its tools for live Agents, and later user patch layers retain precedence. A replacement title provider may register once the previous provider's disposal starts; the previous provider's late results never commit. Claude Code and Codex install on demand through the separate `ON_DEMAND_BUNDLES` catalog; the [on-demand native-bundle decision](2026-10-05-official-on-demand-bundles.md) owns the cost, version, and offline-discovery trade-offs. +Computer Use is a shipped optional bundle in the experimental group. Its bundle uses the explicit naming exception `@deepseek-ai/dsh-cua-bundle`, alongside `@deepseek-ai/dsh-cua`. These names preserve profile references when the capability moves package groups; the experimental classification still controls isolation and promotion. Its repository-owned native component is prepared when the provider loads, without an external Cua Driver runtime. + ## Alternatives considered **Ship every provider as an optional bundle.** Composes and displays correctly, but grows every install by provider runtimes for a capability most installations never switch on; voice input's `sherpa-onnx-node` is the one accepted precedent. @@ -31,3 +33,5 @@ Claude Code and Codex install on demand through the separate `ON_DEMAND_BUNDLES` ## Consequences Official discovery does not imply experimental status or a different plugin lifecycle. Switching a bundle off keeps its installed files; failed activation retains the existing saved-selection behavior. Bundle rows are ordinary Host rows with their usual switches. Individual and combined composition tests cover the Host rows and every preset's tool catalog; live switching and recorded Sessions cover activation and model-visible behavior. + +Computer Use keeps its permission service and native provider on the Host and registers tools in Agent scopes whose bound preset is `ptc`. Existing Agent events and `createScope()` maintain current, new and resumed Agents. Preset declarations remain unchanged. Only the explicit detail-page action changes the new-session default through the existing settings form; the activation switch does not configure PTC execution. diff --git a/.agents/notes/implemented/architecture/2026-09-21-experimental-capabilities-as-optional-bundles.zh.md b/.agents/notes/implemented/architecture/2026-09-21-experimental-capabilities-as-optional-bundles.zh.md index 6c9df974e8..15bc65b56f 100644 --- a/.agents/notes/implemented/architecture/2026-09-21-experimental-capabilities-as-optional-bundles.zh.md +++ b/.agents/notes/implemented/architecture/2026-09-21-experimental-capabilities-as-optional-bundles.zh.md @@ -12,12 +12,14 @@ Web 插件页只提供两个可选 bundle：Agent Teams 与语音输入。Auto r `OPTIONAL_BUNDLES` 管理安装随附且默认关闭的选项，包括维护中的可选能力与实验性能力。每个 bundle 声明 `icon`，并导出带 `meta.title` 与 `meta.description` 的 `./locale/*.json`；官方分组显示这些元数据。Bundle 的展示方式不决定其[实验性状态](../../../../packages/experimental/README.zh.md#status)。隔离检查校验准入，并继续拒绝非实验 bundle 的实验性依赖。 -可选 bundle 的依赖随每次 `dsh` 安装下载。[思考过程翻译](2026-10-05-anonymous-reasoning-translation.zh.md)使用三个实验包，不增加第三方运行时依赖。因此准入需要考虑安装成本，以及开关能否在不增加配置表单的情况下提供可用组合。浏览器与电脑提供方仍使用显式组合，因为它们会增加较大的运行时或要求外部可执行文件。Stagehand 需要模型凭据；Python PTC 替换核心提供方，并与 TypeScript workflow 消费方冲突。演示性钩子、webhook 与模组不进入名单。[Session Inspector](../feature/2026-09-24-session-inspector.zh.md) 管理 Inspector 特有的激活行为。 +可选 bundle 的依赖随每次 `dsh` 安装下载。[思考过程翻译](2026-10-05-anonymous-reasoning-translation.zh.md)使用三个实验包，不增加第三方运行时依赖。因此准入需要考虑安装成本，以及开关能否在不增加配置表单的情况下提供可用组合。外部浏览器与 Cua Driver 提供方仍使用显式组合，因为它们会增加较大的运行时或要求外部可执行文件。Stagehand 需要模型凭据；Python PTC 替换核心提供方，并与 TypeScript workflow 消费方冲突。演示性钩子、webhook 与模组不进入名单。[Session Inspector](../feature/2026-09-24-session-inspector.zh.md) 管理 Inspector 特有的激活行为。 Bundle 插入普通 Host 行；可选模型工具注册到全局工具层，因此包括 minimal 在内的所有预设都会获得它们。插件管理、profile 组合与重载无需针对 bundle 的特殊行为。搜索、Ralph 与终端在 Host 组内提供自己的隔离服务。搜索按需打开一个内存索引；Host 搜索策略保持不变。徽章技能与标题提供方作用于 Host。切换 bundle 会为运行中的 Agent 注册或移除其工具，后续用户 patch 层继续优先。前一个标题提供方开始清理后，替代提供方即可注册；前一个提供方的迟到结果不会提交。 Claude Code 与 Codex 通过独立的 `ON_DEMAND_BUNDLES` 目录按需安装，其成本、版本与离线发现取舍由[按需原生 bundle 决策](2026-10-05-official-on-demand-bundles.zh.md)管理。 +Computer Use 是 experimental 包组内随安装提供的可选组合包。组合包与 `@deepseek-ai/dsh-cua` 一起采用显式命名例外 `@deepseek-ai/dsh-cua-bundle`，在能力移动分组时保留配置引用。依赖隔离和晋升仍由实验性分类约束。提供方加载时准备仓库自有的本机组件，无需外部 Cua Driver 运行时。 + ## 考虑过的替代方案 **把所有提供方都做成可选 bundle。** 组合与展示都正确，但会为多数安装从不启用的能力把提供方运行时塞进每次安装；语音输入的 `sherpa-onnx-node` 是唯一被接受的先例。 @@ -31,3 +33,5 @@ Claude Code 与 Codex 通过独立的 `ON_DEMAND_BUNDLES` 目录按需安装， ## 后果 官方发现不表示实验性状态，也不引入不同的插件生命周期。关闭 bundle 保留已安装文件；激活失败保留现有的已保存选择行为。Bundle 行是普通 Host 行，使用常规开关。独立与组合测试覆盖 Host 行及每个预设的工具目录；实时切换与已记录 Session 覆盖激活和模型可见行为。 + +Computer Use 将权限服务和原生提供方保留在 Host，在已绑定 `ptc` preset 的 Agent 作用域中注册工具。现有 Agent 事件和 `createScope()` 维护当前、新建和恢复的 Agent，preset 声明保持不变。只有详情页的显式操作通过现有设置表单修改新会话默认模式，启用开关不配置 PTC 执行方式。 diff --git a/.agents/notes/implemented/architecture/2026-09-26-cua-application-consent.i18n.yaml b/.agents/notes/implemented/architecture/2026-09-26-cua-application-consent.i18n.yaml index a5e3035bd0..2f89f011b6 100644 --- a/.agents/notes/implemented/architecture/2026-09-26-cua-application-consent.i18n.yaml +++ b/.agents/notes/implemented/architecture/2026-09-26-cua-application-consent.i18n.yaml @@ -9,11 +9,11 @@ en: b32a7ce6483b9b48 zh: ec0938ad032b99d8 /agent-note-application-consent-for-native-computer-use/decision: - en: 92247ff599c1103f - zh: 26845420a9711c83 + en: 36c560c24c376f12 + zh: 5e4ff72f66574e38 /agent-note-application-consent-for-native-computer-use/alternatives-considered: - en: 4693dcd217c4fedb - zh: 1dc9bf9067314006 + en: 80cd5fd93a42d937 + zh: 2a0e241b1b8c2e1d /agent-note-application-consent-for-native-computer-use/consequences: - en: 0146a69ab2086edc - zh: 4257bc1ab4432c3f + en: 4bda3366af65250a + zh: 7c83561b4db07284 diff --git a/.agents/notes/implemented/architecture/2026-09-26-cua-application-consent.md b/.agents/notes/implemented/architecture/2026-09-26-cua-application-consent.md index 3fb91170e5..389a994714 100644 --- a/.agents/notes/implemented/architecture/2026-09-26-cua-application-consent.md +++ b/.agents/notes/implemented/architecture/2026-09-26-cua-application-consent.md @@ -10,29 +10,38 @@ macOS privileges authorize an execution host, not a user's choice of target appl ## Decision -The CUA executor owns application consent for every object-API and native-tool dispatch. It uses the existing User Questions service for human decisions and the provider's existing profile configuration for saved grants. General tool approval and operating-system privileges remain separate requirements. Arbitrary persistent PTC code and separately authorized shell tools retain their own execution authority. +The CUA executor owns application consent for every object-API and native-tool dispatch. It uses the existing User Questions service for human decisions and the provider's existing profile configuration for saved grants. The existing question UI presents the offered labels; the executor rejects custom text or answers that do not select exactly one offered label. Permission-page icons and names are presentation metadata, excluded from saved identities and model results. General tool approval and operating-system privileges remain separate requirements. Arbitrary persistent PTC code and separately authorized shell tools retain their own execution authority. -A saved identity combines the canonical executable path with a verified designated signing requirement, represented by a SHA-256 digest. The path prevents authorization from following a copied application, and the signing requirement distinguishes replacements while allowing matching signed updates. Unverifiable and ad-hoc applications receive task access only, tied to a process instance and Agent connection lifetime. Labels do not establish identity. +System-permission buttons call the native helper on the DSH host and remain available to remote browsers. The [usage documentation](../../../../packages/experimental/cua/README.md#use-this-package) explains where to complete authorization; the page has no persistent remote-access notice. + +A saved identity combines the canonical executable path with a verified designated signing requirement, represented by a SHA-256 digest. The path prevents authorization from following a copied application, and the signing requirement distinguishes replacements while allowing matching signed updates. Unverifiable and ad-hoc applications receive session access only, tied to a process instance. Labels do not establish identity. Cold application selection prepares an executable identity and handle without launching. The same consent decision guards the private launch continuation, and the helper checks the executable again before launch and binds the resulting process afterward. Signed selections retain the normal designated-requirement identity; an unverifiable executable uses a content identity for this pending selection and cannot create a saved grant. Preparing metadata before asking avoids starting an application the user refuses. -Selections of the same application within one task are serialized, so a pending question does not produce competing decisions. Saved-grant additions read the current configuration inside one mutation queue. Refusals prevent another prompt within the task; an explicitly restored saved grant takes precedence at dispatch. +Session decisions are kept in memory by Session ID and application identity. Agent disposal and helper replacement release native handles without discarding these decisions; provider disposal or DSH restart clears them. This scope covers follow-up turns without relying on the model to decide when a user intent has ended. -Removing a saved grant invalidates Agent connections retaining the application's handles and cancels active operations; Agent/provider disposal awaits native connection cleanup. Reauthorization uses a fresh connection, so old app and window snapshots cannot become valid again. The [native object decision](2026-09-17-experimental-cua-object-api.md) continues to own process identity, coordinate semantics, and separation from the evaluator. +Selections of the same application within one session are serialized, so a pending question does not produce competing decisions. Saved-grant additions and revocations share one mutation queue and read the current grant list inside it. Provider shutdown cancels writes still waiting for HMR admission; admitted writes settle before disposal completes. Refusals prevent another prompt until the user permits asking again; a restored saved grant takes precedence in both dispatch and the permission-page projection. + +Revocation invalidates the Agent connections that retain the application's handles, then awaits native cleanup. Reauthorization uses a fresh connection, so old app and window snapshots cannot become valid again. The [native object decision](2026-09-17-experimental-cua-object-api.md) continues to own process identity, coordinate semantics, and separation from the evaluator. ## Alternatives considered **Trust model instructions or hide controls.** Retained objects and direct tool calls can still reach native operations. The executor must enforce the decision at dispatch. -**Persist application names or unsigned paths.** A replacement can reuse both. Durable grants require a verifiable signing identity; task-only access supports unsigned development applications without that promise. +**Persist application names or unsigned paths.** A replacement can reuse both. Durable grants require a verifiable signing identity; session access supports unsigned development applications without that promise. -**Add an approval database or Session control stream.** Profile settings already own user configuration, and User Questions already owns interactive answers. +**Add an approval database or Session control stream.** Profile settings already own user configuration, and User Questions already owns interactive answers. The permission UI observes the provider and its owning settings namespace without another authorization store or event channel. + +**Disable permission buttons based on the browser address.** A user may operate the DSH host through screen sharing and complete authorization there. Port forwarding and proxies make the address an unreliable indication of whether the browser and DSH run on the same computer. + +**Use the nearest GUI ancestor as the permission target.** A source-built Desktop app can sit between Terminal and the helper while macOS still attributes Accessibility and Screen Recording to Terminal. Process responsibility supplies the system’s attribution; ancestry alone does not. The optional SPI adds compatibility risk, so lookup failure omits the application instead of guessing. **Keep the native context after revocation.** Its application lookup can return an earlier handle and retain window snapshots. Closing the Agent connection discards its native context through existing transport cleanup; preserving unrelated handles would require a native per-application retirement protocol. ## Consequences -Revoking one application also invalidates other application handles on the same Agent connection. Those applications require fresh selection; saved grants remain reusable, while task grants expire. Other tasks with independent connections remain unaffected unless they also hold access to the revoked application. +Refusing or revoking one application also invalidates other application handles on the same Agent connection. Those applications require fresh selection; session and saved grants remain reusable. Other sessions with independent connections remain unaffected unless they also hold access to the revoked application. +Explicit enablement may start the shared helper for a read-only permission probe. The helper captures the application identified by macOS process responsibility before readiness releases the launcher, retaining its name and icon after reparenting. The optional native query is dynamically resolved; unavailable attribution uses generic system-prompt guidance. The macOS prompt remains authoritative. Enabling does not request macOS authorization or inspect application content. Application decisions and failure results are exercised through policy regressions and the recorded CUA scenario. Desktop binary distribution, signing, and packaged-host permissions remain outside the source-launch workflow. diff --git a/.agents/notes/implemented/architecture/2026-09-26-cua-application-consent.zh.md b/.agents/notes/implemented/architecture/2026-09-26-cua-application-consent.zh.md index 73d4640db7..e7311dfc6d 100644 --- a/.agents/notes/implemented/architecture/2026-09-26-cua-application-consent.zh.md +++ b/.agents/notes/implemented/architecture/2026-09-26-cua-application-consent.zh.md @@ -10,29 +10,38 @@ macOS 权限授权执行宿主，不能表达用户允许操作哪些目标应 ## 决策 -CUA 执行器在每次对象 API 和原生工具调用时检查应用授权。人工决定复用 User Questions 服务，长期许可保存在 provider 的现有 profile 配置中。通用工具审批和操作系统权限仍是独立要求。任意 persistent PTC 代码和另行授权的 shell 工具保留各自的执行权限。 +CUA 执行器在每次对象 API 和原生工具调用时检查应用授权。人工决定复用 User Questions 服务，长期许可保存在 provider 的现有 profile 配置中。通用工具审批和操作系统权限仍是独立要求。任意 persistent PTC 代码和另行授权的 shell 工具保留各自的执行权限。现有问答界面展示授权选项；执行器拒绝自由文本或未准确选中一个所给选项的回答。权限页图标和名称仅用于展示，不进入保存的身份信息或模型结果。 -长期身份由规范化可执行文件路径和经过验证的签名指定要求组成，以 SHA-256 摘要表示。路径阻止许可跟随应用副本，签名要求区分替换程序，并允许符合相同签名要求的更新。签名不可验证或使用 ad-hoc 签名的应用只能获得任务授权，许可绑定进程实例和 Agent 连接生命周期。显示名称不用于确定身份。 +系统权限按钮调用运行 DSH 的电脑上的原生 helper，远程访问时仍可使用。授权地点写在[使用说明](../../../../packages/experimental/cua/README.zh.md#use-this-package)中，页面不常驻远程访问提示。 + +长期身份由规范化可执行文件路径和经过验证的签名指定要求组成，以 SHA-256 摘要表示。路径阻止许可跟随应用副本，签名要求区分替换程序，并允许符合相同签名要求的更新。签名不可验证或使用 ad-hoc 签名的应用只能获得会话授权，许可绑定进程实例。显示名称不用于确定身份。 选择尚未运行的应用时，先准备可执行文件身份和句柄，不立即启动。同一授权决定保护内部启动续接，helper 在启动前再次核对可执行文件，并在启动后绑定实际进程。已签名应用沿用签名指定要求身份；签名不可验证的程序使用本次待启动选择的内容身份，不能生成长期许可。先准备元数据再询问，可以避免启动用户拒绝的应用。 -同一任务对同一应用的选择串行执行，避免未完成的问题产生相互冲突的决定。长期许可的新增在同一写入队列内读取当前配置。拒绝会阻止任务内的再次询问；用户显式恢复的长期许可在执行检查时优先于旧拒绝记录。 +会话决定按 Session ID 和应用身份保存在内存中。Agent 销毁或 helper 重建会释放原生句柄，但保留授权决定；提供方卸载或 DSH 重启后清除。会话范围可以覆盖用户的后续指令，无须由模型判断一次用户意图何时结束。 -删除长期许可会使保留目标应用句柄的 Agent 连接失效，并取消活动操作；Agent 或提供方销毁时等待对应的原生连接清理。重新授权使用新连接，因此旧应用对象和窗口快照不能重新生效。[原生对象决策](2026-09-17-experimental-cua-object-api.zh.md)继续负责进程身份、坐标语义和与执行器分离的设计。 +同一会话对同一应用的选择串行执行，避免未完成的问题产生相互冲突的决定。长期许可的新增和撤销共用写入队列，并在队列内读取当前许可列表。提供方关闭时取消尚在等待 HMR 调度的写入，已开始的写入会在销毁完成前结束。拒绝会阻止再次询问，直到用户允许重询；重新添加的长期许可在执行检查和权限页中均优先于旧拒绝记录。 + +撤销使保留目标应用句柄的 Agent 连接失效，并等待原生清理完成。重新授权使用新连接，因此旧应用对象和窗口快照不能重新生效。[原生对象决策](2026-09-17-experimental-cua-object-api.zh.md)继续负责进程身份、坐标语义和与执行器分离的设计。 ## 考虑过的替代方案 **依赖模型指引或隐藏控件。** 保留的对象和直接工具调用仍能到达原生操作。执行器必须在派发时检查用户决定。 -**保存应用名称或未签名路径。** 替换程序可以复用二者。长期许可需要可验证的签名身份；任务授权支持未签名的开发应用，但不承诺跨进程复用。 +**保存应用名称或未签名路径。** 替换程序可以复用二者。长期许可需要可验证的签名身份；会话授权支持未签名的开发应用，但不承诺跨进程复用。 -**新增审批数据库或 Session 控制流。** Profile settings 已负责用户配置，User Questions 已负责交互回答。 +**新增审批数据库或 Session 控制流。** Profile settings 已负责用户配置，User Questions 已负责交互回答。权限界面查询 provider，并监听其配置 namespace，无须另一份授权存储或事件通道。 + +**根据浏览器访问地址禁用权限按钮。** 用户可能通过屏幕共享操作运行 DSH 的电脑，并在该电脑上完成授权。端口转发和代理使访问地址无法可靠判断浏览器与 DSH 是否运行在同一台电脑上。 + +**把最近的 GUI 祖先当作授权对象。** 源码构建的 Desktop 可以位于 Terminal 和 helper 之间，但 macOS 仍将辅助功能和屏幕录制归属于 Terminal。责任进程记录提供系统认定的归属，仅凭父进程链无法确定。可选内部接口有兼容性风险，因此查询失败时省略应用信息，不再猜测。 **撤销后保留原生状态。** 原生应用查找可能返回旧句柄并保留窗口快照。关闭 Agent 连接可以通过现有传输清理释放其原生状态；保留其他应用的句柄则需要新增按应用销毁状态的原生协议。 ## 影响 -撤销一个应用也会使同一 Agent 连接中其他应用的句柄失效。这些应用需要重新选择；长期许可仍可复用，任务许可随之失效。其他任务使用独立连接，除非也持有被撤销应用的许可，否则不受影响。 +拒绝或撤销一个应用也会使同一 Agent 连接中其他应用的句柄失效。这些应用需要重新选择；会话许可和长期许可仍可复用。其他会话使用独立连接，除非也持有被撤销应用的许可，否则不受影响。 +显式启用可以启动共享 helper 进行只读权限检查。helper 在发布就绪状态、让启动器退出之前，记录 macOS 责任进程对应的应用，因此父进程改变后仍能提供应用名称和图标。可选的本机查询通过动态加载解析，无法取得归属时显示通用系统弹窗指引。授权归属仍以 macOS 提示为准。启用不会申请 macOS 授权或检查应用内容。 授权决定和失败结果由策略回归测试及 CUA 录制会话验证。桌面安装包的二进制分发、签名和打包宿主权限不属于源码启动流程。 diff --git a/apps/cli/package.json b/apps/cli/package.json index 6f4f927b48..d7c3e78f72 100644 --- a/apps/cli/package.json +++ b/apps/cli/package.json @@ -105,6 +105,7 @@ "@deepseek-ai/dsh-experimental-terminal-bundle": "workspace:*", "@deepseek-ai/dsh-experimental-session-search": "workspace:*", "@deepseek-ai/dsh-experimental-tool-worktree": "workspace:*", + "@deepseek-ai/dsh-cua-bundle": "workspace:*", "@deepseek-ai/dsh-experimental-plugin-market-bundle": "workspace:*" }, "devDependencies": { diff --git a/apps/desktop/README.i18n.yaml b/apps/desktop/README.i18n.yaml index 4268cbc148..bda88b0fc6 100644 --- a/apps/desktop/README.i18n.yaml +++ b/apps/desktop/README.i18n.yaml @@ -42,8 +42,8 @@ en: 98cf848762482d48 zh: 74f9100a39ec8c2f /deepseek-harness-desktop/develop: - en: fe6d23e6a3d3bc66 - zh: d60c8d3a55747916 + en: faf7676ba18bbf2f + zh: f1d7b75ad348c548 /deepseek-harness-desktop/develop/startup-onboarding: en: a51346b8d880ddba zh: 7913214c1c66063c diff --git a/apps/desktop/README.md b/apps/desktop/README.md index f8951d5b98..45c94d62c7 100644 --- a/apps/desktop/README.md +++ b/apps/desktop/README.md @@ -202,6 +202,8 @@ After an explicit build, `start:desktop` reconstructs the disposable project and pnpm run start:desktop ``` +Desktop package preparation filters optional native packages by the installer target’s OS and architecture. Windows package sets omit the macOS Computer Use tarball, including when their inputs were packed on macOS. + The Web counterparts are `pnpm run dev:web` and `pnpm run start:web`, documented in the [development guide](../../docs/development.md). Workspace development runs the current CLI and private Desktop Host packages under Electron RunAsNode. Plugin management and recovery use `$DSH_HOME/profiles/desktop`, separate from the disposable workspace runtime. The Host uses runtime module resolution in both development and packaged builds without creating official-package fallback links; developer-installed packages, including links, retain native priority. Use an unpacked application to exercise Electron RunAsNode, bundled pnpm, bundled dsh resources, plugin installation and repair paths. The [native/renderer keyboard tests](tests/keyboard.spec.ts) compile as part of the repository Client typecheck. Their Desktop imports are limited to Cordis-free input, persistence, IPC, browser-guest, and overlay modules. diff --git a/apps/desktop/README.zh.md b/apps/desktop/README.zh.md index 0c6d3abda4..0504c7004c 100644 --- a/apps/desktop/README.zh.md +++ b/apps/desktop/README.zh.md @@ -202,6 +202,8 @@ pnpm run dev:desktop pnpm run start:desktop ``` +桌面包准备过程按安装包目标的操作系统和架构筛选可选原生包。Windows 的包集合不包含 macOS Computer Use tarball，即使输入包是在 macOS 上打包的。 + Web 侧的对应命令是 `pnpm run dev:web` 与 `pnpm run start:web`，见[开发指南](../../docs/development.zh.md)。Workspace 开发使用 Electron RunAsNode 运行当前 CLI 与私有 Desktop Host 包，插件管理和恢复使用 `$DSH_HOME/profiles/desktop`，与一次性工作区运行时分离。Host 在开发与打包构建中都使用 runtime 模块解析，不创建官方包的 fallback 链接；开发者安装的包（包括链接）保留原生优先级。需要验证 Electron RunAsNode、内置 pnpm、内置 dsh 资源、插件安装和修复时，应运行未封装安装器的应用目录。 [原生输入与渲染进程键盘测试](tests/keyboard.spec.ts)直接纳入仓库 Client 类型检查。它只导入不依赖 Cordis 的 Desktop 输入、持久化、IPC、浏览器 guest 和蒙层模块。 diff --git a/apps/desktop/scripts/prepare-dsh.ts b/apps/desktop/scripts/prepare-dsh.ts index 9de39d3ff2..365c5a45f2 100644 --- a/apps/desktop/scripts/prepare-dsh.ts +++ b/apps/desktop/scripts/prepare-dsh.ts @@ -134,7 +134,7 @@ async function main(): Promise { await packagingStep(process.env.DSH_DESKTOP_PACKAGING_RUN_DIR, 'runtime:install', () => runPnpm(['install', '--prod', '--frozen-lockfile', '--trust-lockfile'])) const packageSet = readDesktopCorePackageSet(BUILD_ROOT, release.version) const targetName = resolveDesktopBuildTarget() - const target = { platform: process.platform, arch: desktopTargetPlatform(targetName).arch } + const target = desktopTargetPlatform(targetName) const modules = join(BUILD_ROOT, 'node_modules') const officeManifest = JSON.parse(readFileSync(join(modules, '@deepseek-ai/libreoffice-kit/package.json'), 'utf8')) const officeEngine = selectOfficeEngine(officeManifest, target) diff --git a/apps/desktop/scripts/prepare-package-set.ts b/apps/desktop/scripts/prepare-package-set.ts index 7e7d62eea2..4cffa7018b 100644 --- a/apps/desktop/scripts/prepare-package-set.ts +++ b/apps/desktop/scripts/prepare-package-set.ts @@ -25,7 +25,7 @@ import { } from '../src/core-package-set.ts' import { capture } from '../../../scripts/release/process.ts' import { tarballFiles } from '../../../scripts/release/tarball.ts' -import { resolveDesktopTargetBuildPaths } from './desktop-build-paths.mjs' +import { desktopTargetPlatform, resolveDesktopBuildTarget, resolveDesktopTargetBuildPaths } from './desktop-build-paths.mjs' const DSH_PACKAGE = '@deepseek-ai/dsh' const ROOT_PACKAGES = [DSH_PACKAGE, DESKTOP_HOST_PACKAGE] as const @@ -41,6 +41,17 @@ export interface PackedDesktopPackage { readonly manifest: Readonly> } +function supportsTarget(manifest: Readonly>, field: 'os' | 'cpu', target: string): boolean { + const values = manifest[field] + if (values === undefined) return true + if (!Array.isArray(values) || !values.every((value): value is string => typeof value === 'string')) { + throw new Error(`desktop package set: ${String(manifest.name)} has invalid ${field}`) + } + if (values.includes(`!${target}`)) return false + return values.length === 0 || (values.length === 1 && values[0] === 'any') || values.includes(target) + || values.every(value => value.startsWith('!')) +} + function dependencyNames(manifest: Readonly>, section: string): string[] { const value = manifest[section] if (value === undefined) return [] @@ -54,10 +65,12 @@ function dependencyNames(manifest: Readonly>, section: s * Select workspace dependencies rooted at dsh and its private Host; npm resolves external packages. * Reads the repository workspace manifest and package manifests to distinguish required local packages from npm-resolved externals. * @param available - Packed packages indexed by package name. + * @param target - Platform and architecture of the installer payload. * @returns Selected packages sorted by name. */ export function selectDesktopPackageClosure( available: ReadonlyMap, + target: ReturnType, ): PackedDesktopPackage[] { const workspace = yaml.load(readFileSync(join(REPOSITORY_ROOT, 'pnpm-workspace.yaml'), 'utf8')) as { packages: string[] } const workspaceNames = new Set(globSync(workspace.packages.map(pattern => `${pattern}/package.json`), { cwd: REPOSITORY_ROOT }) @@ -77,7 +90,9 @@ export function selectDesktopPackageClosure( } } for (const dependency of dependencyNames(packed.manifest, OPTIONAL_DEPENDENCY_SECTION)) { - if (available.has(dependency)) visit(dependency) + const optional = available.get(dependency) + if (optional && supportsTarget(optional.manifest, 'os', target.platform) + && supportsTarget(optional.manifest, 'cpu', target.arch)) visit(dependency) } } for (const name of ROOT_PACKAGES) { @@ -127,9 +142,16 @@ export function assertDesktopHostPackageFiles(files: readonly string[]): void { } } -/** Prepare a package set from release tarball directories. */ -export function prepareDesktopPackageSet(inputs: readonly string[], output: string): void { - const selected = selectDesktopPackageClosure(packedPackages(inputs)) +/** + * Prepare a package set from release tarball directories. + * @param inputs - Release tarball directories. + * @param output - Destination package-set directory. + * @param target - Platform and architecture of the installer payload. + */ +export function prepareDesktopPackageSet( + inputs: readonly string[], output: string, target: ReturnType, +): void { + const selected = selectDesktopPackageClosure(packedPackages(inputs), target) const host = selected.find(packed => packed.manifest.name === DESKTOP_HOST_PACKAGE) if (host === undefined) throw new Error(`desktop package set: selected closure omits ${DESKTOP_HOST_PACKAGE}`) assertDesktopHostPackageFiles(tarballFiles(host.tarball)) @@ -171,7 +193,7 @@ function main(): void { }) const inputs = (values.from ?? defaultInputs).map(path => resolve(REPOSITORY_ROOT, path)) const output = values.out === undefined ? buildPaths.packageSet : resolve(REPOSITORY_ROOT, values.out) - prepareDesktopPackageSet(inputs, output) + prepareDesktopPackageSet(inputs, output, desktopTargetPlatform(resolveDesktopBuildTarget())) console.log(`desktop package set: prepared ${output}`) } diff --git a/apps/desktop/tests/cua-optional-package.spec.ts b/apps/desktop/tests/cua-optional-package.spec.ts new file mode 100644 index 0000000000..3f3f7ad572 --- /dev/null +++ b/apps/desktop/tests/cua-optional-package.spec.ts @@ -0,0 +1,121 @@ +/** Cold pnpm requests exercise shipped CUA dependency/platform metadata with inert package contents. */ +import { createHash } from 'node:crypto' +import { createServer } from 'node:http' +import { mkdir, mkdtemp, readFile, rm, writeFile } from 'node:fs/promises' +import { tmpdir } from 'node:os' +import { join, resolve } from 'node:path' +import { execa } from 'execa' +import { expect, it, onTestFinished } from 'vitest' +import { pnpmCommand } from '../../../scripts/release/process.ts' + +const repository = resolve(import.meta.dirname, '../../..') +const providerName = '@deepseek-ai/dsh-experimental-cua-macos' +const bundleName = '@deepseek-ai/dsh-cua-bundle' +const readManifest = async (path: string): Promise> => + JSON.parse(await readFile(join(repository, path, 'package.json'), 'utf8')) as Record + +it('fetches the macOS optional tarball only for Darwin, with an empty cache for every target', async () => { + const root = await mkdtemp(join(tmpdir(), 'cua-cold-install-')) + onTestFinished(() => rm(root, { recursive: true, force: true })) + const userconfig = join(root, 'npmrc') + await writeFile(userconfig, '') + const environment = Object.fromEntries(Object.entries(process.env).filter(([key]) => + !/KEY|SECRET|TOKEN|PASSWORD/iu.test(key) && !/^(npm|pnpm|corepack)_/iu.test(key) + && key !== 'NODE_OPTIONS' && key !== 'NODE_PATH')) + const [pnpm, ...prefix] = pnpmCommand() + const command = async (args: string[], cwd: string) => { + const result = await execa(pnpm, [...prefix, `--config.userconfig=${userconfig}`, ...args], { + cwd, env: { ...environment, CI: 'true', XDG_CACHE_HOME: join(cwd, 'xdg-cache'), + XDG_DATA_HOME: join(cwd, 'xdg-data'), XDG_STATE_HOME: join(cwd, 'xdg-state'), + XDG_CONFIG_HOME: join(cwd, 'xdg-config') }, extendEnv: false, reject: false, timeout: 60_000, + }) + expect(result.timedOut, result.stderr).toBe(false) + expect(result.signal, result.stderr).toBeUndefined() + expect(result.exitCode, `${result.stdout}\n${result.stderr}`).toBe(0) + return result.stdout + } + const cli = await readManifest('apps/cli') + const bundle = await readManifest('packages/experimental/cua-bundle') + const native = await readManifest('packages/experimental/cua-macos') + expect(cli.dependencies).toHaveProperty(bundleName) + expect(bundle.dependencies).not.toHaveProperty(providerName) + expect(bundle.optionalDependencies).toEqual({ [providerName]: 'workspace:*' }) + expect(native.os).toEqual(['darwin']) + const version = '1.0.0' + const bundleDependencies = Object.fromEntries(Object.keys(bundle.dependencies as Record).map(name => [name, version])) + const manifests = [ + { name: cli.name, version, dependencies: { [bundleName]: version } }, + { name: bundleName, version, dependencies: bundleDependencies, optionalDependencies: { [providerName]: version } }, + ...Object.keys(bundleDependencies).map(name => ({ name, version })), + { name: providerName, version, os: native.os, files: native.files }, + ] + const packages = new Map; bytes: Buffer }>() + for (const manifest of manifests) { + const name = String(manifest.name) + const directory = join(root, name.replace('@deepseek-ai/', '')) + await mkdir(directory) + await writeFile(join(directory, 'package.json'), JSON.stringify(manifest)) + if (name === providerName) { + await mkdir(join(directory, 'native/bin/darwin-arm64'), { recursive: true }) + for (const file of ['.gitignore', '.npmignore']) { + await writeFile(join(directory, 'native', file), await readFile(join(repository, 'packages/experimental/cua-macos/native', file))) + } + await writeFile(join(directory, 'native/bin/darwin-arm64/dsh-cua-native'), 'inert native payload') + } + await command(['pack', '--out', 'package.tgz'], directory) + const bytes = await readFile(join(directory, 'package.tgz')) + packages.set(name, { manifest, bytes }) + if (name === providerName) { + const result = await execa('tar', ['-tzf', join(directory, 'package.tgz')]) + expect(result.stdout).toContain('package/native/bin/darwin-arm64/dsh-cua-native') + } + } + const requests: string[] = [] + let unrestricted = false + const server = createServer((request, response) => { + const path = decodeURIComponent(request.url ?? '') + requests.push(path) + const tarball = path.endsWith('.tgz') + const name = path.slice(1).replace(/\.tgz$/u, '') + const entry = packages.get(name) + if (!entry) { response.writeHead(404); response.end(); return } + if (tarball) { response.end(entry.bytes); return } + const manifest = { ...entry.manifest } + // pnpm also infers macOS from this package name; an explicit any overrides that inference. + if (unrestricted && name === providerName) manifest.os = ['any'] + response.setHeader('Content-Type', 'application/json') + response.end(JSON.stringify({ name, 'dist-tags': { latest: version }, versions: { + [version]: { ...manifest, dist: { + tarball: `http://${request.headers.host}/${name}.tgz`, + integrity: `sha512-${createHash('sha512').update(entry.bytes).digest('base64')}`, + } }, + } })) + }) + onTestFinished(async () => { + server.closeAllConnections() + await new Promise((resolve, reject) => { server.close((error) => { if (error) reject(error); else resolve() }) }) + }) + await new Promise((resolve, reject) => { server.once('error', reject); server.listen(0, '127.0.0.1', resolve) }) + const address = server.address() + if (!address || typeof address === 'string') throw new Error('fixture registry has no TCP address') + for (const target of ['darwin', 'win32', 'linux', 'linux-unrestricted']) { + unrestricted = target === 'linux-unrestricted' + const project = join(root, target) + await mkdir(project) + await writeFile(join(project, 'package.json'), JSON.stringify({ private: true, dependencies: { [String(cli.name)]: version } })) + await writeFile(join(project, 'pnpm-workspace.yaml'), [ + 'autoInstallPeers: false', 'supportedArchitectures:', ` os: [${unrestricted ? 'linux' : target}]`, + ' cpu: [arm64]', 'minimumReleaseAge: 0', '', + ].join('\n')) + requests.length = 0 + const output = await command([ + 'install', '--ignore-scripts', '--config.enable-global-virtual-store=false', + `--config.registry=http://127.0.0.1:${address.port}/`, `--config.store-dir=${join(project, 'store')}`, + `--config.cache-dir=${join(project, 'cache')}`, `--config.state-dir=${join(project, 'state')}`, + ], project) + expect(requests).toContain(`/${bundleName}.tgz`) + expect(requests).toContain('/@deepseek-ai/dsh-cua.tgz') + expect(requests.includes(`/${providerName}.tgz`), `${target}: ${JSON.stringify(requests)}\n${output}`) + .toBe(target === 'darwin' || unrestricted) + } +}, 120_000) diff --git a/apps/desktop/tests/prepare-package-set.spec.ts b/apps/desktop/tests/prepare-package-set.spec.ts index 6db1ccbd01..c1b5edbd69 100644 --- a/apps/desktop/tests/prepare-package-set.spec.ts +++ b/apps/desktop/tests/prepare-package-set.spec.ts @@ -1,6 +1,12 @@ -import { afterEach, describe, expect, it, vi } from 'vitest' +import { afterEach, describe, expect, it, onTestFinished, vi } from 'vitest' +import { execFileSync } from 'node:child_process' +import { mkdirSync, mkdtempSync, readFileSync, readdirSync, rmSync, writeFileSync } from 'node:fs' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import { DESKTOP_HOST_RUNTIME_FILES, DESKTOP_PACKAGES_DIR } from '../src/core-package-set.ts' import { assertDesktopHostPackageFiles, + prepareDesktopPackageSet, selectDesktopPackageClosure, type PackedDesktopPackage, } from '../scripts/prepare-package-set.ts' @@ -37,7 +43,7 @@ describe('desktop package-set selection', () => { ['@deepseek-ai/platform-package', packed('@deepseek-ai/platform-package')], ['@deepseek-ai/unused', packed('@deepseek-ai/unused')], ]) - expect(selectDesktopPackageClosure(available).map(entry => entry.manifest.name)).toEqual([ + expect(selectDesktopPackageClosure(available, { platform: 'win32', arch: 'x64' }).map(entry => entry.manifest.name)).toEqual([ '@deepseek-ai/cordis', '@deepseek-ai/dsh', '@deepseek-ai/dsh-base', @@ -57,10 +63,10 @@ describe('desktop package-set selection', () => { dependencies: { '@deepseek-ai/dsh': '^1.0.0' }, })], ]) - expect(() => selectDesktopPackageClosure(available)).toThrow(/unpacked package/u) + expect(() => selectDesktopPackageClosure(available, { platform: 'win32', arch: 'x64' })).toThrow(/unpacked package/u) expect(() => selectDesktopPackageClosure(new Map([ ['@deepseek-ai/dsh', packed('@deepseek-ai/dsh')], - ]))).toThrow(/omit @deepseek-ai\/dsh-desktop-host/u) + ]), { platform: 'win32', arch: 'x64' })).toThrow(/omit @deepseek-ai\/dsh-desktop-host/u) }) it('leaves independently published Office packages to npm resolution', () => { @@ -73,7 +79,7 @@ describe('desktop package-set selection', () => { })], ['@deepseek-ai/dsh-desktop-host', packed('@deepseek-ai/dsh-desktop-host')], ]) - expect(selectDesktopPackageClosure(available).map(entry => entry.manifest.name)).toEqual([ + expect(selectDesktopPackageClosure(available, { platform: 'win32', arch: 'x64' }).map(entry => entry.manifest.name)).toEqual([ '@deepseek-ai/dsh', '@deepseek-ai/dsh-desktop-host', ]) }) @@ -92,3 +98,67 @@ describe('desktop package-set selection', () => { expect(() => { assertDesktopHostPackageFiles(files.slice(0, 1)) }).toThrow(/lib\/cli\.js/u) }) }) + +const macProvider = '@deepseek-ai/dsh-experimental-cua-macos' +const cuaBundle = '@deepseek-ai/dsh-cua-bundle' +const targetManifests = () => new Map([ + ['@deepseek-ai/dsh', packed('@deepseek-ai/dsh', { dependencies: { [cuaBundle]: '1.0.0' } })], + ['@deepseek-ai/dsh-desktop-host', packed('@deepseek-ai/dsh-desktop-host')], + [cuaBundle, packed(cuaBundle, { + optionalDependencies: { [macProvider]: '1.0.0' }, dependencies: { '@deepseek-ai/dsh-cua': '1.0.0' }, + })], + ['@deepseek-ai/dsh-cua', packed('@deepseek-ai/dsh-cua')], + [macProvider, packed(macProvider, { os: ['darwin'] })], +]) + +it.each([ + ['darwin', 'arm64', true], ['darwin', 'x64', true], ['win32', 'x64', false], +] as const)('selects the CUA native tarball for target %s/%s independently of the host', (platform, arch, included) => { + const selected = selectDesktopPackageClosure(targetManifests(), { platform, arch }).map(entry => entry.manifest.name) + expect(selected.includes(macProvider)).toBe(included) + expect(selected).toContain('@deepseek-ai/dsh-cua') + expect(selected).toContain(cuaBundle) +}) + +it.each([ + [{ cpu: ['arm64'] }, false], [{ os: ['!win32'] }, false], [{ os: ['!linux'] }, true], + [{ os: [] }, true], [{ os: ['any'] }, true], [{ os: ['any', '!win32'] }, false], + [{ cpu: ['x64'] }, true], +])('honors optional package platform constraints %j', (constraints, included) => { + const packages = targetManifests() + packages.set(macProvider, packed(macProvider, constraints)) + expect(selectDesktopPackageClosure(packages, { platform: 'win32', arch: 'x64' }) + .some(entry => entry.manifest.name === macProvider)).toBe(included) +}) + +it.each([{ os: 'darwin' }, { cpu: [42] }])('rejects malformed platform constraints %j', (constraints) => { + const packages = targetManifests() + packages.set(macProvider, packed(macProvider, constraints)) + expect(() => selectDesktopPackageClosure(packages, { platform: 'win32', arch: 'x64' })).toThrow(/has invalid/u) +}) + +it('copies no macOS tarball into the Windows package set', () => { + const root = mkdtempSync(join(tmpdir(), 'desktop-platform-packages-')) + onTestFinished(() => { rmSync(root, { recursive: true, force: true }) }) + const inputs = join(root, 'inputs') + mkdirSync(inputs) + for (const [name, entry] of targetManifests()) { + const stem = name.replace('@deepseek-ai/', '') + const folder = join(root, stem, 'package') + mkdirSync(folder, { recursive: true }) + writeFileSync(join(folder, 'package.json'), JSON.stringify(entry.manifest)) + if (name === '@deepseek-ai/dsh-desktop-host') { + for (const file of DESKTOP_HOST_RUNTIME_FILES) { + mkdirSync(join(folder, file, '..'), { recursive: true }) + writeFileSync(join(folder, file), '') + } + } + execFileSync('tar', ['-czf', join(inputs, `${stem}.tgz`), '-C', join(folder, '..'), 'package']) + } + const output = join(root, 'output') + prepareDesktopPackageSet([inputs], output, { platform: 'win32', arch: 'x64' }) + expect(readdirSync(join(output, DESKTOP_PACKAGES_DIR))).not.toContain('dsh-experimental-cua-macos.tgz') + prepareDesktopPackageSet([inputs], output, { platform: 'darwin', arch: 'arm64' }) + expect(readFileSync(join(output, DESKTOP_PACKAGES_DIR, 'dsh-experimental-cua-macos.tgz'))) + .toEqual(readFileSync(join(inputs, 'dsh-experimental-cua-macos.tgz'))) +}) diff --git a/apps/web/tests/cua-optin.e2e.ts b/apps/web/tests/cua-optin.e2e.ts new file mode 100644 index 0000000000..7a35661fe4 --- /dev/null +++ b/apps/web/tests/cua-optin.e2e.ts @@ -0,0 +1,128 @@ +/** Optional CUA uses existing plugin activation and offers a separate persistent PTC action. */ +import { mkdtemp, readFile, rm, writeFile } from 'node:fs/promises' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import { fileURLToPath } from 'node:url' +import { chromium, type Browser } from 'playwright' +import { readProfileManifest } from '@deepseek-ai/dsh-app-boot' +import { SessionId } from '@deepseek-ai/dsh-session' +import type {} from '@deepseek-ai/dsh-ptc-runtime' +import type {} from '@deepseek-ai/dsh-agent-preset-registry' +import { expect, it } from 'vitest' +import { launchWebScaffold, type WebScaffold } from './scaffold.ts' +import { connectFreshWorkspaceZh, ZH_BROWSER_LOCALE } from './support.ts' + +it.each([undefined, 'minimal'] as const)('adds CUA to PTC sessions while preserving default %s until the explicit persistent-runtime action', async (selectedDefault) => { + const directory = await mkdtemp(join(tmpdir(), 'cua-optin-web-')) + let ownedScaffold: WebScaffold | undefined + let browser: Browser | undefined + try { + const patch = join(directory, 'native-fixture.yml') + // The executable cannot answer native permission queries, so setup must offer retry in the detail page. + await writeFile(patch, JSON.stringify([{ id: 'cua', disabled: false, config: { helperPath: process.execPath } }])) + // Install the test provider explicitly so the helper override also exercises session wiring on Linux. + const scaffold = ownedScaffold = await launchWebScaffold({ + profile: { packages: [{ dir: fileURLToPath(new URL('../../../packages/experimental/cua-macos', import.meta.url)), enabled: false }] }, + harnessHome: join(directory, 'home'), extraOverlayPath: [ + fileURLToPath(new URL('./pin-browse-picker.overlay.yml', import.meta.url)), patch, + ] }) + const initialDefault = selectedDefault ?? 'standard' + if (selectedDefault !== undefined) await scaffold.ctx.settings.update('agent-preset-registry', { selectedDefault }) + browser = await chromium.launch() + const page = await browser.newPage({ viewport: { width: 1680, height: 1000 }, locale: ZH_BROWSER_LOCALE }) + await page.goto(scaffold.authenticatedUrl) + await connectFreshWorkspaceZh(page, scaffold.workspaceCwd) + const originalAgent = scaffold.ctx.agents.list()[0]! + const ptcAgent = (await scaffold.ctx.agents.create({ sessionId: SessionId('cua-existing-ptc'), + setup: async (ctx) => { await scaffold.ctx.agentPresets.mount(ctx, 'ptc') }, + })).agent + const runtimeFiber = () => [...scaffold.ctx.loader.entries()].find(entry => entry.options.id === 'ptc-runtime')?.fiber + const originalRuntime = scaffold.ctx.ptcRuntime + const originalRuntimeFiber = runtimeFiber() + expect(originalRuntimeFiber).toBeDefined() + expect(scaffold.ctx.ptcRuntime.programFormat).toBe('function-body') + const settingsPath = join(scaffold.harnessHome, 'profiles/scaffold/cordis.patch.yml') + const before = await readFile(settingsPath, 'utf8') + const selected = () => readProfileManifest('scaffold', join(scaffold.harnessHome, 'profiles/scaffold')).dsh?.profile?.bundles + await page.getByRole('navigation', { name: '全局面板' }).getByRole('button', { name: '插件', exact: true }).click() + const card = page.locator('[data-plugin-package="@deepseek-ai/dsh-cua-bundle"]') + const toggle = card.getByRole('switch', { name: '启用 Computer Use', exact: true }) + expect(await toggle.getAttribute('aria-checked')).toBe('false') + expect(await card.textContent()).toContain('请切换到 PTC 模式') + // Bundle transitions await native cleanup under the browser action deadline. + await toggle.click() + await toggle.and(page.locator('[aria-checked="true"]')).waitFor({ state: 'visible' }) + const setup = page.getByRole('dialog', { name: '使用 Computer Use 前需要设置', exact: true }) + await setup.waitFor() + expect(await setup.getByText('Computer Use 插件已开启，但暂时无法检测系统权限。请前往插件详情页查看原因并重试。', { exact: true }).count()).toBe(1) + await setup.getByRole('button', { name: '稍后', exact: true }).click() + expect(selected()).toContain('@deepseek-ai/dsh-cua-bundle') + expect(await readFile(settingsPath, 'utf8')).toBe(before) + expect(runtimeFiber() === originalRuntimeFiber).toBe(true) + expect(scaffold.ctx.ptcRuntime.programFormat).toBe('function-body') + expect(scaffold.ctx.agentPresets.defaultId).toBe(initialDefault) + expect(scaffold.ctx.agents.get(originalAgent.id) === originalAgent).toBe(true) + expect(scaffold.ctx.tools.schemas(originalAgent).some(tool => tool.name === 'cua')).toBe(false) + expect(scaffold.ctx.tools.schemas(ptcAgent).some(tool => tool.name === 'cua')).toBe(true) + expect((await scaffold.ctx.systemPrompt.assemble({ scope: ptcAgent })).tools.map(tool => tool.name)).toEqual(['run_code']) + const fresh = await scaffold.ctx.agents.create({ sessionId: SessionId('cua-new-ptc'), + setup: async (ctx) => { await scaffold.ctx.agentPresets.mount(ctx, 'ptc') }, + }) + expect(scaffold.ctx.tools.schemas(fresh.agent).some(tool => tool.name === 'cua')).toBe(true) + await toggle.click() + await toggle.and(page.locator('[aria-checked="false"]')).waitFor({ state: 'visible' }) + expect(scaffold.ctx.get('computerUse')).toBeUndefined() + for (const agent of [ptcAgent, fresh.agent]) expect(scaffold.ctx.tools.schemas(agent).some(tool => tool.name === 'cua')).toBe(false) + expect(runtimeFiber() === originalRuntimeFiber).toBe(true) + expect(scaffold.ctx.ptcRuntime.programFormat).toBe('function-body') + expect(await readFile(settingsPath, 'utf8')).toBe(before) + await card.getByRole('button', { name: '查看 Computer Use', exact: true }).click() + const detailToggle = page.locator('[data-plugin-panel]').getByRole('switch', { name: '启用 Computer Use', exact: true }) + await detailToggle.click() + await detailToggle.and(page.locator('[aria-checked="true"]')).waitFor({ state: 'visible' }) + const panel = page.locator('[data-cua-settings]') + await panel.waitFor() + const defaultSetting = panel.getByRole('group', { name: '新会话默认模式', exact: true }) + await expect.poll(() => defaultSetting.getByRole('button', { name: '设为 PTC', exact: true }).isEnabled()).toBe(true) + await defaultSetting.getByRole('button', { name: '设为 PTC', exact: true }).click() + await expect.poll(() => scaffold.ctx.agentPresets.defaultId).toBe('ptc') + expect(scaffold.ctx.agentPresets.composedPreset(originalAgent.ctx)).toBe(initialDefault) + expect(runtimeFiber() === originalRuntimeFiber).toBe(true) + expect(scaffold.ctx.ptcRuntime.programFormat).toBe('function-body') + const confirmed = await scaffold.ctx.agents.create({ sessionId: SessionId('cua-confirmed-default'), + setup: async (ctx) => { await scaffold.ctx.agentPresets.mount(ctx) }, + }) + expect(scaffold.ctx.tools.schemas(confirmed.agent).some(tool => tool.name === 'cua')).toBe(true) + const saved = await readFile(settingsPath, 'utf8') + expect(saved).toContain('selectedDefault: ptc') + expect(saved).not.toContain('executionMode:') + const persistent = panel.getByRole('group', { name: 'PTC 执行方式', exact: true }) + await persistent.getByRole('button', { name: '设为持久化', exact: true }).click() + await persistent.getByRole('button', { name: '已设为持久化', exact: true }).waitFor() + expect(scaffold.ctx.ptcRuntime.programFormat).toBe('repl') + expect(scaffold.ctx.ptcRuntime === originalRuntime).toBe(false) + const persistentRuntimeFiber = runtimeFiber() + const runtime = scaffold.ctx.ptcRuntime, owner = {} + const program = 'globalThis.counter = (globalThis.counter ?? 0) + 1' + expect((await runtime.run(runtime.resolve({ program, bindings: [], stateOwner: owner }))).value).toBe(1) + const configured = await readFile(settingsPath, 'utf8') + expect(configured).toContain('executionMode: persistent-in-memory') + expect(scaffold.ctx.agentPresets.defaultId).toBe('ptc') + expect(scaffold.ctx.agentPresets.composedPreset(originalAgent.ctx)).toBe(initialDefault) + await detailToggle.click() + await detailToggle.and(page.locator('[aria-checked="false"]')).waitFor({ state: 'visible' }) + expect(await readFile(settingsPath, 'utf8')).toBe(configured) + expect(runtimeFiber() === persistentRuntimeFiber).toBe(true) + expect((await runtime.run(runtime.resolve({ program, bindings: [], stateOwner: owner }))).value).toBe(2) + await detailToggle.click() + await detailToggle.and(page.locator('[aria-checked="true"]')).waitFor({ state: 'visible' }) + expect(await readFile(settingsPath, 'utf8')).toBe(configured) + expect(runtimeFiber() === persistentRuntimeFiber).toBe(true) + expect((await runtime.run(runtime.resolve({ program, bindings: [], stateOwner: owner }))).value).toBe(3) + await runtime.release(owner) + } finally { + try { await browser?.close() } finally { + try { await ownedScaffold?.close() } finally { await rm(directory, { recursive: true, force: true }) } + } + } +}) diff --git a/apps/web/tests/cua-permissions.e2e.ts b/apps/web/tests/cua-permissions.e2e.ts new file mode 100644 index 0000000000..18072ae8b1 --- /dev/null +++ b/apps/web/tests/cua-permissions.e2e.ts @@ -0,0 +1,150 @@ +/** Built permission UI talks to the real provider; the isolated helper substitutes only macOS privilege observations. */ +import { mkdir, mkdtemp, readFile, rm, writeFile } from 'node:fs/promises' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import { fileURLToPath } from 'node:url' +import { ToolCallId } from '@deepseek-ai/dsh-llm' +import type {} from '@deepseek-ai/dsh-user-approval' +import type {} from '@deepseek-ai/dsh-agent-preset-registry' +import { chromium, type Browser } from 'playwright' +import { expect, it, onTestFailed, onTestFinished } from 'vitest' +import { captureStableAria, compareOrRefreshGolden, launchWebScaffold, webSnapshotMode, watchConsole, type WebScaffold } from './scaffold.ts' +import { connectFreshWorkspace, newEnglishPage, saveFailureShot } from './support.ts' +import { installNativeFixture, startFixtureHost } from '../../../packages/experimental/cua-macos/tests/native-fixture.ts' + +const bundle = fileURLToPath(new URL('../../../packages/experimental/cua-bundle', import.meta.url)) +const expected = fileURLToPath(new URL('./expected/cua-permissions/missing.expected.md', import.meta.url)) + +// System-permission onboarding belongs to the macOS-only bundle; rendering and policy units remain portable. +it.skipIf(process.platform !== 'darwin')('keeps system requests pending until fresh observation and persists application revocation', async () => { + const root = await mkdtemp(join(tmpdir(), 'dsh-cua-permissions-web-')) + const resources: { scaffold?: WebScaffold; browser?: Browser; closeFixture?: () => Promise } = {} + onTestFinished(async () => { + try { await resources.browser?.close() } finally { + try { await resources.scaffold?.close() } finally { + try { await resources.closeFixture?.() } finally { await rm(root, { recursive: true, force: true }) } + } + } + }) + const helper = await installNativeFixture(root) + resources.closeFixture = await startFixtureHost(helper) + const state = join(root, 'permissions.json'), audit = join(root, 'helper-events.jsonl') + const launchApplication = { name: 'Visual Studio Code.app', path: '/Applications/Visual Studio Code.app', + icon: 'data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAIAAACQd1PeAAAACXBIWXMAAAPoAAAD6AG1e1JrAAAADElEQVQImWNgZGIGAAAOAAeCcsnOAAAAAElFTkSuQmCC' } + await writeFile(state, JSON.stringify({ accessibility: false, screenRecording: false, launchApplication })) + await writeFile(join(root, 'observe-helper'), '') + const fixtureBundle = join(root, 'fixture-bundle') + await mkdir(fixtureBundle) + await writeFile(join(fixtureBundle, 'package.json'), JSON.stringify({ name: 'cua-permission-fixture', version: '1.0.0', + dsh: { bundle: { patch: 'cordis.patch.yml' } } })) + await writeFile(join(fixtureBundle, 'cordis.patch.yml'), `- id: cua\n config: ${JSON.stringify({ helperPath: helper })}\n`) + const scaffold = await launchWebScaffold({ + profile: { packages: [{ dir: bundle, enabled: true }, { dir: fixtureBundle, enabled: true }] }, + }) + resources.scaffold = scaffold + await scaffold.ctx.settings.mutate('cua', [{ op: 'set', path: ['allowedApps'], + value: [{ id: 'a'.repeat(64), name: 'Fixture', executablePath: '/Applications/Fixture.app' }] }]) + const browser = await chromium.launch() + resources.browser = browser + const page = await newEnglishPage(browser), tripwire = watchConsole(page) + onTestFailed(() => saveFailureShot(page, 'web-e2e-cua-permissions')) + await page.goto(scaffold.authenticatedUrl) + await connectFreshWorkspace(page, scaffold.workspaceCwd) + const agent = scaffold.ctx.agents.list()[0]! + await scaffold.ctx.agentPresets.select(agent, 'ptc') + scaffold.ctx.approval.setPolicy(agent, 'never') + await scaffold.ctx.settings.mutate('cua', [{ op: 'set', path: ['allowedApps'], value: [] }]) + // Application consent remains independent of the session's tool-approval policy. + const grant = scaffold.ctx.tools.execute({ agent, signal: new AbortController().signal, + callId: ToolCallId('cua-app-approval'), name: 'run_code', + arguments: { code: 'return await tools.cua({ method: "getApp", params: { selector: "Fixture" } })', + description: 'Request application access' } }) + const approval = page.locator('[data-question-key]') + await Promise.race([approval.waitFor(), grant.then((result) => { + throw new Error(`Application selection completed before consent: ${JSON.stringify(result)}`) + })]) + expect(await approval.textContent()).toContain('Fixture') + await approval.getByRole('radio', { name: 'Always allow', exact: true }).click() + await approval.getByRole('button', { name: 'Submit', exact: true }).click() + expect((await grant).isError).toBe(false) + await approval.waitFor({ state: 'hidden' }) + await page.getByRole('button', { name: 'Plugins', exact: true }).click() + const card = page.locator('[data-plugin-package="@deepseek-ai/dsh-cua-bundle"]') + expect(await card.getByRole('button', { name: /Accessibility permission|Screen Recording permission/ }).count()).toBe(0) + await page.setViewportSize({ width: 1680, height: 600 }) + const plugins = page.locator('[data-plugin-panel]') + const openDetails = page.getByRole('button', { name: 'View Computer Use', exact: true }) + await openDetails.scrollIntoViewIfNeeded() + expect(await plugins.evaluate(element => element.scrollTop)).toBeGreaterThan(0) + await openDetails.click() + const panel = page.locator('[data-cua-settings]') + const accessibility = panel.getByRole('group', { name: 'Accessibility', exact: true }) + await accessibility.getByRole('button', { name: 'Request permission', exact: true }).waitFor() + expect(await panel.getByRole('region', { name: 'Authorize Computer Use', exact: true }).getByRole('group').count()).toBe(4) + expect(await panel.getByRole('region', { name: 'Manage applications Computer Use can access', exact: true }).count()).toBe(1) + await expect.poll(() => panel.locator('li img').evaluate((image: HTMLImageElement) => image.naturalWidth)).toBe(1) + expect(await panel.getByText('/fixture/app', { exact: true }).count()).toBe(0) + expect(await accessibility.getByText('Visual Studio Code.app', { exact: true }).count()).toBe(1) + expect(await accessibility.textContent()).toContain('In System Settings, allow') + expect(await accessibility.textContent()).toContain('to read interface controls, click, and type.') + expect(await panel.getByRole('group', { name: 'Screen Recording', exact: true }).getByText('Visual Studio Code.app', { exact: true }).count()).toBe(1) + await expect.poll(() => accessibility.locator('img').evaluate((image: HTMLImageElement) => image.naturalWidth)).toBe(1) + await expect.poll(() => panel.getAttribute('aria-busy')).toBe('false') + await compareOrRefreshGolden(expected, await captureStableAria(page, '[data-cua-settings]', scaffold.workspaceCwd), webSnapshotMode()) + try { + for (const width of [1680, 390]) { + await page.setViewportSize({ width, height: 1000 }) + for (const colorScheme of ['light', 'dark'] as const) { + await page.emulateMedia({ colorScheme }) + await expect.poll(() => page.evaluate(() => document.body.hasAttribute('data-ds-dark-theme'))).toBe(colorScheme === 'dark') + const measure = () => panel.evaluate(element => ({ + clientWidth: element.clientWidth, scrollWidth: element.scrollWidth, + overflowing: [...element.querySelectorAll('*')].filter(child => + child.getBoundingClientRect().right > element.getBoundingClientRect().right + 1, + ).map(child => ({ tag: child.tagName, className: child.className, text: child.textContent?.slice(0, 80) })), + })) + await expect.poll(async () => { + const layout = await measure() + return layout.scrollWidth <= layout.clientWidth ? true : { width, colorScheme, ...layout } + }).toBe(true) + if (webSnapshotMode() === 'refresh') { + const directory = fileURLToPath(new URL('../../../.artifacts', import.meta.url)) + await mkdir(directory, { recursive: true }) + const path = join(directory, `cua-permission-design-${process.pid}-${width}-${colorScheme}.png`) + await panel.screenshot({ path, animations: 'disabled' }) + console.log(`CUA permission design: ${path}`) + } + } + } + } finally { + await page.emulateMedia({ colorScheme: null }) + await page.setViewportSize({ width: 1680, height: 1000 }) + } + await accessibility.getByRole('button', { name: 'Request permission', exact: true }).click() + await expect.poll(() => panel.getAttribute('aria-busy')).toBe('false') + expect(await accessibility.getByRole('button', { name: 'Request permission', exact: true }).count()).toBe(1) + await writeFile(state, JSON.stringify({ accessibility: true, screenRecording: false, launchApplication })) + // Headless pages retain document focus; model returning from System Settings at the window-event boundary. + await page.evaluate(() => { window.dispatchEvent(new Event('focus')) }) + await accessibility.getByText('Allowed', { exact: true }).waitFor() + const screen = panel.getByRole('group', { name: 'Screen Recording', exact: true }) + expect(await accessibility.getByRole('button').count()).toBe(1) + await accessibility.getByRole('button', { name: 'Allowed', exact: true }).click() + expect(await screen.getByRole('button').count()).toBe(1) + await expect.poll(() => panel.getAttribute('aria-busy')).toBe('false') + await panel.getByRole('button', { name: 'Revoke', exact: true }).click() + await panel.getByRole('heading', { name: 'Always-allowed applications', exact: true }).waitFor({ state: 'hidden' }) + await panel.getByRole('heading', { name: 'Applications not allowed', exact: true }).waitFor() + await panel.getByRole('button', { name: 'Allow asking again', exact: true }).click() + await panel.getByText('Fixture', { exact: true }).waitFor({ state: 'hidden' }) + const saved = await readFile(join(scaffold.harnessHome, 'profiles/scaffold/cordis.patch.yml'), 'utf8') + expect(saved).toContain('allowedApps: []') + const events = (await readFile(audit, 'utf8')).trim().split('\n').map(line => JSON.parse(line) as { event: string; method: string; params: object }) + .filter(record => record.event === 'call').map(({ method, params }) => ({ method, params })) + expect(events).toContainEqual({ method: 'requestPermission', params: { permission: 'accessibility' } }) + expect(events.filter(event => event.method === 'openPermissionSettings')).toEqual([ + { method: 'openPermissionSettings', params: { permission: 'accessibility' } }, + ]) + expect(events.every(event => ['permissionStatus', 'requestPermission', 'openPermissionSettings', 'applicationIcons', 'getApp'].includes(event.method))).toBe(true) + expect(tripwire.pageErrors).toEqual([]) +}, 60_000) diff --git a/apps/web/tests/expected/cua-permissions/missing.expected.md b/apps/web/tests/expected/cua-permissions/missing.expected.md new file mode 100644 index 0000000000..4255b31e9f --- /dev/null +++ b/apps/web/tests/expected/cua-permissions/missing.expected.md @@ -0,0 +1,26 @@ +- region "Authorize Computer Use": + - heading "Authorize Computer Use" [level=3] + - group "Default mode for new sessions": + - strong: Default mode for new sessions + - paragraph: Computer Use is available only in PTC mode. + - button "Set to PTC" + - group "PTC execution mode": + - strong: PTC execution mode + - paragraph: Keep variables and application objects across tool calls. Switching restarts the PTC runtime. + - button "Use persistent mode" + - group "Accessibility": + - strong: Accessibility + - paragraph: In System Settings, allow Visual Studio Code.app to read interface controls, click, and type. + - button "Request permission" + - group "Screen Recording": + - strong: Screen Recording + - paragraph: In System Settings, allow Visual Studio Code.app to capture application windows. + - button "Request permission" + - paragraph: Enable the application in System Settings. Follow the macOS prompt if it names another application. +- region "Manage applications Computer Use can access": + - heading "Manage applications Computer Use can access" [level=3] + - heading "Always-allowed applications" [level=4] + - list: + - listitem: + - strong: Fixture + - button "Revoke" diff --git a/apps/web/tests/expected/plugin-config/official.expected.md b/apps/web/tests/expected/plugin-config/official.expected.md index 37d3a232b4..27c2e72887 100644 --- a/apps/web/tests/expected/plugin-config/official.expected.md +++ b/apps/web/tests/expected/plugin-config/official.expected.md @@ -38,6 +38,10 @@ - button "查看 开发者工具": 开发者工具 - text: 实验性 查看调试会话原始数据、聊天消息分组数据，以及调试 NodeJS 后端 - switch "启用 开发者工具" + - listitem: + - button "查看 Computer Use": Computer Use + - text: 目前仅支持 macOS，Windows 支持正在开发中。为已有及新建 PTC 会话加载 Computer Use。请切换到 PTC 模式来使用，并授予辅助功能和屏幕录制权限。 + - switch "启用 Computer Use" - heading "已安装插件" [level=3] - list: - listitem: diff --git a/apps/web/tests/expected/plugin-manager/live-enabled.expected.md b/apps/web/tests/expected/plugin-manager/live-enabled.expected.md index eab9dd48e1..cfe553e2bb 100644 --- a/apps/web/tests/expected/plugin-manager/live-enabled.expected.md +++ b/apps/web/tests/expected/plugin-manager/live-enabled.expected.md @@ -38,6 +38,10 @@ - button "查看 开发者工具": 开发者工具 - text: 实验性 查看调试会话原始数据、聊天消息分组数据，以及调试 NodeJS 后端 - switch "启用 开发者工具" + - listitem: + - button "查看 Computer Use": Computer Use + - text: 目前仅支持 macOS，Windows 支持正在开发中。为已有及新建 PTC 会话加载 Computer Use。请切换到 PTC 模式来使用，并授予辅助功能和屏幕录制权限。 + - switch "启用 Computer Use" - heading "已安装插件" [level=3] - list: - listitem: diff --git a/apps/web/tests/expected/plugin-manager/manager.expected.md b/apps/web/tests/expected/plugin-manager/manager.expected.md index f3dd4a7bc6..d58ad62b03 100644 --- a/apps/web/tests/expected/plugin-manager/manager.expected.md +++ b/apps/web/tests/expected/plugin-manager/manager.expected.md @@ -38,6 +38,10 @@ - button "查看 开发者工具": 开发者工具 - text: 实验性 查看调试会话原始数据、聊天消息分组数据，以及调试 NodeJS 后端 - switch "启用 开发者工具" + - listitem: + - button "查看 Computer Use": Computer Use + - text: 目前仅支持 macOS，Windows 支持正在开发中。为已有及新建 PTC 会话加载 Computer Use。请切换到 PTC 模式来使用，并授予辅助功能和屏幕录制权限。 + - switch "启用 Computer Use" - heading "已安装插件" [level=3] - list: - listitem: diff --git a/apps/web/tests/expected/plugin-manager/refresh.expected.md b/apps/web/tests/expected/plugin-manager/refresh.expected.md index 75dda78845..85d666fd3e 100644 --- a/apps/web/tests/expected/plugin-manager/refresh.expected.md +++ b/apps/web/tests/expected/plugin-manager/refresh.expected.md @@ -6,7 +6,7 @@ Focus at 500ms: - tooltip "刷新" failure: pending - button "刷新" [disabled] -{"busy":"true","spinners":1,"cards":11,"skeletons":0,"alerts":[],"retry":0,"toasts":[]} +{"busy":"true","spinners":1,"cards":12,"skeletons":0,"alerts":[],"retry":0,"toasts":[]} failure: settled - button "刷新" -{"busy":"false","spinners":0,"cards":11,"skeletons":0,"alerts":[],"retry":0,"toasts":["刷新失败，请重试"]} +{"busy":"false","spinners":0,"cards":12,"skeletons":0,"alerts":[],"retry":0,"toasts":["刷新失败，请重试"]} diff --git a/apps/web/tests/message-actions.e2e.ts b/apps/web/tests/message-actions.e2e.ts index d70558dab4..2f96305714 100644 --- a/apps/web/tests/message-actions.e2e.ts +++ b/apps/web/tests/message-actions.e2e.ts @@ -15,7 +15,7 @@ import { acknowledgeReloadConnectionLoss, assertFixtureInventory, captureStableAria, compareOrRefreshGolden, fixtureUserPrompts, launchWebScaffold, parseSeedFixture, realizeSeedFixture, renderSeedFixture, seedSession, watchConsole, webSnapshotMode, type WebScaffold, } from './scaffold.ts' -import { openSettings, newEnglishPage, pinBrowserClock, saveFailureShot, WEB_FIXTURE_TIME } from './support.ts' +import { openSettings, newEnglishPage, pinBrowserClock, pinHostClock, saveFailureShot, WEB_FIXTURE_TIME } from './support.ts' const SNAPSHOT_DIR = fileURLToPath(new URL('../../../snapshots/web/message-actions', import.meta.url)) // Borrowed read-only: this scenario needs any settled user+assistant pair, not @@ -191,6 +191,7 @@ describe('web e2e: message IconActions and clocks on settled history', () => { let browser: Browser let page: Page let tripwire: ReturnType + let hostClock: ReturnType | undefined beforeAll(async () => { scaffold = await launchWebScaffold() @@ -211,6 +212,10 @@ describe('web e2e: message IconActions and clocks on settled history', () => { await seedSession(scaffold, raw, SEED_ID, undefined, { createdAt: WEB_FIXTURE_TIME - 90_000 }) browser = await chromium.launch() page = await newEnglishPage(browser) + // Fork summaries use Host time; placeholders use browser time. Each pin + // starts at the fixture date, so keep these calls adjacent: setup between + // them can create enough clock skew to reverse the fork order. + hostClock = pinHostClock() await pinBrowserClock(page) tripwire = watchConsole(page) await page.goto(scaffold.authenticatedUrl, { waitUntil: 'load' }) @@ -218,8 +223,12 @@ describe('web e2e: message IconActions and clocks on settled history', () => { }, 120_000) afterAll(async () => { - await browser?.close() - await scaffold?.close() + try { + await browser?.close() + await scaffold?.close() + } finally { + hostClock?.mockRestore() + } }) it.skipIf(MODE === 'record')('enables branch only on the completed transcript tail', async () => { diff --git a/apps/web/tests/preview-boot.e2e.ts b/apps/web/tests/preview-boot.e2e.ts index 0e32a5a7d6..157d7b97c6 100644 --- a/apps/web/tests/preview-boot.e2e.ts +++ b/apps/web/tests/preview-boot.e2e.ts @@ -681,15 +681,28 @@ async function toggleOptionalBundles(page: Page): Promise { } }) expect(result.optional.map(entry => entry.name).sort()).toEqual([...OPTIONAL_BUNDLES].sort()) - expect(result.optional.filter(entry => entry.enabled || entry.availability !== 'installation' || entry.error !== null)).toEqual([]) + expect(result.optional.filter(entry => entry.enabled || entry.availability !== 'installation')).toEqual([]) + expect(result.optional.filter(entry => entry.error !== null)).toEqual([]) // Metadata resolves through the Worker's Node internal modules: every optional bundle shows its title and icon. expect(result.optional.filter(entry => entry.metaError !== null || entry.title !== 'object' || entry.icon !== 'string')).toEqual([]) expect(result.inventoryErrors).toEqual([]) console.info(`preview boot: optional bundle toggles ${JSON.stringify(result.toggles.map(({ name, enable, disable, error }) => ({ name, enable, disable, error })))}`) - expect(result.toggles.filter(toggle => toggle.enable !== 'applied' || toggle.disable !== 'applied') + const cua = result.toggles.find(toggle => toggle.name === '@deepseek-ai/dsh-cua-bundle')! + expect(cua).toMatchObject({ + enable: 'applied', disable: 'applied', + error: null, + restored: true, + }) + expect(cua.changed).toBeGreaterThan(0) + expect(cua.inactive).toEqual([]) + expect(cua.plugins).toContainEqual(expect.objectContaining({ + moduleName: '@deepseek-ai/dsh-experimental-cua-macos', enabled: false, fiberPhase: null, + })) + const supported = result.toggles.filter(toggle => toggle.name !== '@deepseek-ai/dsh-cua-bundle') + expect(supported.filter(toggle => toggle.enable !== 'applied' || toggle.disable !== 'applied') .map(({ name, enable, disable }) => ({ name, enable, disable }))).toEqual([]) // Each bundle changes Loader rows, every enabled row it changes runs, and switching it off restores the prior rows. - expect(result.toggles.filter(toggle => toggle.changed === 0 || toggle.inactive.length > 0 || !toggle.restored) + expect(supported.filter(toggle => toggle.changed === 0 || toggle.inactive.length > 0 || !toggle.restored) .map(({ name, changed, inactive, restored }) => ({ name, changed, inactive, restored }))).toEqual([]) const pluginsWith = (bundle: string): Array<{ moduleName: string; enabled: boolean; fiberPhase: string | null }> => result.toggles.find(toggle => toggle.name === bundle)?.plugins ?? [] diff --git a/apps/web/tsconfig.json b/apps/web/tsconfig.json index d1da30db67..9d25a02899 100644 --- a/apps/web/tsconfig.json +++ b/apps/web/tsconfig.json @@ -82,6 +82,8 @@ "tests/plugin-config.e2e.ts", "tests/input-plugin-extensions.e2e.ts", "tests/plugin-manager.e2e.ts", + "tests/cua-optin.e2e.ts", + "tests/cua-permissions.e2e.ts", "tests/schedule-header.e2e.ts", "tests/worktree-opt-in.e2e.ts", "tests/session-inspector.e2e.ts", diff --git a/docs/capability-seams.i18n.yaml b/docs/capability-seams.i18n.yaml index 573a76f8b7..ae1d83cdf3 100644 --- a/docs/capability-seams.i18n.yaml +++ b/docs/capability-seams.i18n.yaml @@ -6,5 +6,5 @@ en: 41d772192075e133 zh: 5ef3da69b4de1d8a /capability-seams-and-core-services: - en: f209321ab0da4f90 - zh: 4af5c135eec51ce1 + en: 753b3da3abc6aeea + zh: aab02ec16c3f744b diff --git a/docs/capability-seams.md b/docs/capability-seams.md index 865162e7df..f2b52ade47 100644 --- a/docs/capability-seams.md +++ b/docs/capability-seams.md @@ -52,6 +52,8 @@ flowchart LR svc_computerUse["ctx.computerUse
Computer-use provider registration"] pkg_experimental_computer_use_cua_driver_mcp["experimental-computer-use-cua-driver-mcp"] pkg_experimental_computer_use_cua_driver_native["experimental-computer-use-cua-driver-native"] + pkg_cua["cua"] + svc_cuaAccess["ctx.cuaAccess
Computer Use permission controls"] pkg_office_to_pdf["office-to-pdf"] svc_officeToPdf["ctx.officeToPdf
Office to PDF conversion"] pkg_client_ui_sidebar_documentpreview["client-ui-sidebar-documentpreview"] @@ -334,6 +336,7 @@ flowchart LR pkg_cordis_host_runner --> svc_dynamicCordisRunner pkg_credentials --> svc_credentials pkg_credentials_local --> svc_credentials + pkg_cua --> svc_cuaAccess pkg_deepseek_account --> svc_deepseekAccount pkg_deepseek_account_platform --> svc_deepseekAccount pkg_deepseek_llm_api_extensions --> svc_deepseekLlmApiExtensions @@ -484,6 +487,7 @@ flowchart LR svc_credentials --> pkg_api_settings_controller svc_credentials --> pkg_llm_deepseek svc_credentials --> pkg_llm_pi_ai + svc_cuaAccess --> pkg_cua svc_deepseekAccount --> pkg_api_account_controller svc_deepseekAccount --> pkg_llm_deepseek svc_deepseekLlmApiExtensions --> pkg_llm_deepseek @@ -628,6 +632,7 @@ flowchart LR | `ctx.mcpResources` | `seam` | [`mcp-resources`](../packages/mcp/mcp-resources) | [`mcp-client`](../packages/mcp/mcp-client) | [`mcp-resources`](../packages/mcp/mcp-resources) | - | Connection-owned providers serve shared resource tools in the calling agent scope. | | `ctx.browserUse` | `seam` | [`browser-use`](../packages/browser-use/browser-use) | [`experimental-browser-use-playwright-mcp`](../packages/experimental/browser-use-playwright-mcp), [`experimental-browser-use-chrome-devtools-mcp`](../packages/experimental/browser-use-chrome-devtools-mcp), [`experimental-browser-use-stagehand-native`](../packages/experimental/browser-use-stagehand-native) | [`experimental-browser-use-playwright-mcp`](../packages/experimental/browser-use-playwright-mcp), [`experimental-browser-use-chrome-devtools-mcp`](../packages/experimental/browser-use-chrome-devtools-mcp), [`experimental-browser-use-stagehand-native`](../packages/experimental/browser-use-stagehand-native) | - | One provider-owned name per service instance. Providers own their tools and browser resources per live Session; the shared service has no browser operation API. | | `ctx.computerUse` | `seam` | [`computer-use`](../packages/computer-use/computer-use) | [`experimental-computer-use-cua-driver-mcp`](../packages/experimental/computer-use-cua-driver-mcp), [`experimental-computer-use-cua-driver-native`](../packages/experimental/computer-use-cua-driver-native) | [`experimental-computer-use-cua-driver-mcp`](../packages/experimental/computer-use-cua-driver-mcp), [`experimental-computer-use-cua-driver-native`](../packages/experimental/computer-use-cua-driver-native) | - | One provider-owned name per service instance. Each provider also owns its model tools; the service has no common action API, runtime selection, or Session workflow lock. | +| `ctx.cuaAccess` | `core` | [`cua`](../packages/experimental/cua) | - | [`cua`](../packages/experimental/cua) | - | The native provider owns application consent and cancellation; the optional Client page queries and changes those decisions through its Remote. | | `ctx.officeToPdf` | `core` | [`office-to-pdf`](../packages/document/office-to-pdf) | - | [`client-ui-sidebar-documentpreview`](../packages/client/ui-sidebar-documentpreview) | - | Authorized Office bytes are converted on the Host using the declared native target engine, or Node WASM when no native target is declared. | | `ctx.attachments` | `seam` | [`attachment`](../packages/attachment/attachment) | [`attachment-local`](../packages/attachment/attachment-local) | [`api-session-controller`](../packages/api/session-controller), [`tool-fs`](../packages/fs/tool-fs), [`llm-pi-ai`](../packages/llm/llm-pi-ai), [`llm-deepseek`](../packages/llm/llm-deepseek) | - | The host commits accepted images before session events; provider adapters resolve authorized durable references into provider-native content. | | `ctx.fileUploads` | `core` | [`client-file-upload`](../packages/client/file-upload) | - | [`api-session-controller`](../packages/api/session-controller) | - | Owns streaming intake, durable storage, and staged receipt lifetime; the Session controller binds receipts to accepted submissions. | diff --git a/docs/capability-seams.zh.md b/docs/capability-seams.zh.md index 395986ab96..fede454e27 100644 --- a/docs/capability-seams.zh.md +++ b/docs/capability-seams.zh.md @@ -54,6 +54,8 @@ flowchart LR svc_computerUse["ctx.computerUse
Computer-use provider registration"] pkg_experimental_computer_use_cua_driver_mcp["experimental-computer-use-cua-driver-mcp"] pkg_experimental_computer_use_cua_driver_native["experimental-computer-use-cua-driver-native"] + pkg_cua["cua"] + svc_cuaAccess["ctx.cuaAccess
Computer Use permission controls"] pkg_office_to_pdf["office-to-pdf"] svc_officeToPdf["ctx.officeToPdf
Office to PDF conversion"] pkg_client_ui_sidebar_documentpreview["client-ui-sidebar-documentpreview"] @@ -336,6 +338,7 @@ flowchart LR pkg_cordis_host_runner --> svc_dynamicCordisRunner pkg_credentials --> svc_credentials pkg_credentials_local --> svc_credentials + pkg_cua --> svc_cuaAccess pkg_deepseek_account --> svc_deepseekAccount pkg_deepseek_account_platform --> svc_deepseekAccount pkg_deepseek_llm_api_extensions --> svc_deepseekLlmApiExtensions @@ -486,6 +489,7 @@ flowchart LR svc_credentials --> pkg_api_settings_controller svc_credentials --> pkg_llm_deepseek svc_credentials --> pkg_llm_pi_ai + svc_cuaAccess --> pkg_cua svc_deepseekAccount --> pkg_api_account_controller svc_deepseekAccount --> pkg_llm_deepseek svc_deepseekLlmApiExtensions --> pkg_llm_deepseek @@ -630,6 +634,7 @@ flowchart LR | `ctx.mcpResources` | `seam` | [`mcp-resources`](../packages/mcp/mcp-resources) | [`mcp-client`](../packages/mcp/mcp-client) | [`mcp-resources`](../packages/mcp/mcp-resources) | - | 连接所有者提供的操作在调用 agent 的作用域内服务于共享资源工具。 | | `ctx.browserUse` | `seam` | [`browser-use`](../packages/browser-use/browser-use) | [`experimental-browser-use-playwright-mcp`](../packages/experimental/browser-use-playwright-mcp), [`experimental-browser-use-chrome-devtools-mcp`](../packages/experimental/browser-use-chrome-devtools-mcp), [`experimental-browser-use-stagehand-native`](../packages/experimental/browser-use-stagehand-native) | [`experimental-browser-use-playwright-mcp`](../packages/experimental/browser-use-playwright-mcp), [`experimental-browser-use-chrome-devtools-mcp`](../packages/experimental/browser-use-chrome-devtools-mcp), [`experimental-browser-use-stagehand-native`](../packages/experimental/browser-use-stagehand-native) | - | 每个服务实例注册一个提供方拥有的名称。提供方按实时 Session 拥有自己的工具与浏览器资源；共享服务不提供浏览器操作 API。 | | `ctx.computerUse` | `seam` | [`computer-use`](../packages/computer-use/computer-use) | [`experimental-computer-use-cua-driver-mcp`](../packages/experimental/computer-use-cua-driver-mcp), [`experimental-computer-use-cua-driver-native`](../packages/experimental/computer-use-cua-driver-native) | [`experimental-computer-use-cua-driver-mcp`](../packages/experimental/computer-use-cua-driver-mcp), [`experimental-computer-use-cua-driver-native`](../packages/experimental/computer-use-cua-driver-native) | - | 每个服务实例只注册一个提供方自定的名称。各提供方也拥有自己的模型工具；服务不提供通用操作 API、运行时选择或 Session 流程锁。 | +| `ctx.cuaAccess` | `core` | [`cua`](../packages/experimental/cua) | - | [`cua`](../packages/experimental/cua) | - | 原生 provider 负责应用授权和取消操作；可选的 Client 权限页面通过 Remote 查询和修改授权。 | | `ctx.officeToPdf` | `core` | [`office-to-pdf`](../packages/document/office-to-pdf) | - | [`client-ui-sidebar-documentpreview`](../packages/client/ui-sidebar-documentpreview) | - | 已授权的 Office 字节在宿主上使用已声明的原生目标引擎转换；未声明原生目标时使用 Node WASM。 | | `ctx.attachments` | `seam` | [`attachment`](../packages/attachment/attachment) | [`attachment-local`](../packages/attachment/attachment-local) | [`api-session-controller`](../packages/api/session-controller), [`tool-fs`](../packages/fs/tool-fs), [`llm-pi-ai`](../packages/llm/llm-pi-ai), [`llm-deepseek`](../packages/llm/llm-deepseek) | - | 宿主会在会话事件之前提交已接受的图片；提供方适配器将已授权的持久引用解析为提供方原生内容。 | | `ctx.fileUploads` | `core` | [`client-file-upload`](../packages/client/file-upload) | - | [`api-session-controller`](../packages/api/session-controller) | - | 负责流式接收、持久存储和暂存回执生命周期；Session Controller 将回执绑定到已接受的提交。 | diff --git a/docs/config-catalog.md b/docs/config-catalog.md index 1b7b3903b3..9734ae8398 100644 --- a/docs/config-catalog.md +++ b/docs/config-catalog.md @@ -4762,6 +4762,7 @@ These load from a `cordis.yml` entry with no `config:` block; they declare no co | `@deepseek-ai/dsh-computer-use` | — | [`packages/computer-use/computer-use/src/index.ts`](../packages/computer-use/computer-use/src/index.ts) | | `@deepseek-ai/dsh-config-editor` | `loader` · `profileContext` | [`packages/boot/config-editor/src/index.ts`](../packages/boot/config-editor/src/index.ts) | | `@deepseek-ai/dsh-cordis-client-runner` | — | [`packages/extensions/cordis-client-runner/src/index.ts`](../packages/extensions/cordis-client-runner/src/index.ts) | +| `@deepseek-ai/dsh-cua` | — | [`packages/experimental/cua/src/index.ts`](../packages/experimental/cua/src/index.ts) | | `@deepseek-ai/dsh-deepseek-llm-api-extensions` | — | [`packages/llm/deepseek-llm-api-extensions/src/index.ts`](../packages/llm/deepseek-llm-api-extensions/src/index.ts) | | `@deepseek-ai/dsh-experimental-auto-review` | `approval` · `llm` · `permissionPresets` · `sessions` · `tools` · `workingDirectory` | [`packages/experimental/auto-review/src/index.ts`](../packages/experimental/auto-review/src/index.ts) | | `@deepseek-ai/dsh-experimental-client-ui-agent-team` | — | [`packages/experimental/client-ui-agent-team/src/index.ts`](../packages/experimental/client-ui-agent-team/src/index.ts) | @@ -4849,7 +4850,7 @@ Imported as libraries by other packages; a `cordis.yml` cannot load them. | `@deepseek-ai/dsh-client-ui-slots` | — | [`packages/client/ui-slots/src/index.ts`](../packages/client/ui-slots/src/index.ts) | | `@deepseek-ai/dsh-client-web` | — | [`packages/client/web/src/index.ts`](../packages/client/web/src/index.ts) | | `@deepseek-ai/dsh-cmdline` | — | [`packages/boot/cmdline/src/index.ts`](../packages/boot/cmdline/src/index.ts) | -| `@deepseek-ai/dsh-cua` | — | [`packages/experimental/cua/src/index.ts`](../packages/experimental/cua/src/index.ts) | +| `@deepseek-ai/dsh-cua-bundle` | — | [`packages/experimental/cua-bundle/src/index.ts`](../packages/experimental/cua-bundle/src/index.ts) | | `@deepseek-ai/dsh-deque` | — | [`packages/util/deque/src/index.ts`](../packages/util/deque/src/index.ts) | | `@deepseek-ai/dsh-experimental-agent-team-profile` | — | [`packages/experimental/agent-team-profile/src/index.ts`](../packages/experimental/agent-team-profile/src/index.ts) | | `@deepseek-ai/dsh-experimental-badge-skill-bundle` | — | [`packages/experimental/badge-skill-bundle/src/index.ts`](../packages/experimental/badge-skill-bundle/src/index.ts) | diff --git a/docs/config-catalog.zh.md b/docs/config-catalog.zh.md index bc917305a4..af4f431824 100644 --- a/docs/config-catalog.zh.md +++ b/docs/config-catalog.zh.md @@ -4764,6 +4764,7 @@ export interface Config { | `@deepseek-ai/dsh-computer-use` | — | [`packages/computer-use/computer-use/src/index.ts`](../packages/computer-use/computer-use/src/index.ts) | | `@deepseek-ai/dsh-config-editor` | `loader` · `profileContext` | [`packages/boot/config-editor/src/index.ts`](../packages/boot/config-editor/src/index.ts) | | `@deepseek-ai/dsh-cordis-client-runner` | — | [`packages/extensions/cordis-client-runner/src/index.ts`](../packages/extensions/cordis-client-runner/src/index.ts) | +| `@deepseek-ai/dsh-cua` | — | [`packages/experimental/cua/src/index.ts`](../packages/experimental/cua/src/index.ts) | | `@deepseek-ai/dsh-deepseek-llm-api-extensions` | — | [`packages/llm/deepseek-llm-api-extensions/src/index.ts`](../packages/llm/deepseek-llm-api-extensions/src/index.ts) | | `@deepseek-ai/dsh-experimental-auto-review` | `approval` · `llm` · `permissionPresets` · `sessions` · `tools` · `workingDirectory` | [`packages/experimental/auto-review/src/index.ts`](../packages/experimental/auto-review/src/index.ts) | | `@deepseek-ai/dsh-experimental-client-ui-agent-team` | — | [`packages/experimental/client-ui-agent-team/src/index.ts`](../packages/experimental/client-ui-agent-team/src/index.ts) | @@ -4851,7 +4852,7 @@ export interface Config { | `@deepseek-ai/dsh-client-ui-slots` | — | [`packages/client/ui-slots/src/index.ts`](../packages/client/ui-slots/src/index.ts) | | `@deepseek-ai/dsh-client-web` | — | [`packages/client/web/src/index.ts`](../packages/client/web/src/index.ts) | | `@deepseek-ai/dsh-cmdline` | — | [`packages/boot/cmdline/src/index.ts`](../packages/boot/cmdline/src/index.ts) | -| `@deepseek-ai/dsh-cua` | — | [`packages/experimental/cua/src/index.ts`](../packages/experimental/cua/src/index.ts) | +| `@deepseek-ai/dsh-cua-bundle` | — | [`packages/experimental/cua-bundle/src/index.ts`](../packages/experimental/cua-bundle/src/index.ts) | | `@deepseek-ai/dsh-deque` | — | [`packages/util/deque/src/index.ts`](../packages/util/deque/src/index.ts) | | `@deepseek-ai/dsh-experimental-agent-team-profile` | — | [`packages/experimental/agent-team-profile/src/index.ts`](../packages/experimental/agent-team-profile/src/index.ts) | | `@deepseek-ai/dsh-experimental-badge-skill-bundle` | — | [`packages/experimental/badge-skill-bundle/src/index.ts`](../packages/experimental/badge-skill-bundle/src/index.ts) | diff --git a/docs/event-producer-consumer.md b/docs/event-producer-consumer.md index 50b9819555..063f53ed2c 100644 --- a/docs/event-producer-consumer.md +++ b/docs/event-producer-consumer.md @@ -9,9 +9,9 @@ This matrix shows which packages dispatch each harness-owned event and which pac | Event | Mode | Declared in | Dispatchers | Listeners | | --- | --- | --- | --- | --- | | `agent-loop/config-start-failed` | `emit` | [`packages/core/agent-loop/src/index.ts:196`](../packages/core/agent-loop/src/index.ts) | [`agent-loop`](../packages/core/agent-loop) (`events.dispatch`) | - | -| `agent-preset/selected` | `emit` | [`packages/preset/agent-preset-registry/src/types.ts:66`](../packages/preset/agent-preset-registry/src/types.ts) | [`agent-preset-registry`](../packages/preset/agent-preset-registry) (`emit`) | `remotes` | +| `agent-preset/selected` | `emit` | [`packages/preset/agent-preset-registry/src/types.ts:66`](../packages/preset/agent-preset-registry/src/types.ts) | [`agent-preset-registry`](../packages/preset/agent-preset-registry) (`emit`) | [`cua`](../packages/experimental/cua), `remotes` | | `agent/assistant-stream` | `emit` | [`packages/core/agent/src/runtime-types.ts:363`](../packages/core/agent/src/runtime-types.ts) | [`agent-loop`](../packages/core/agent-loop) (`emit`) | [`headless`](../packages/bundle/headless), `session-controller` | -| `agent/created` | `serial` | [`packages/core/agent/src/runtime-types.ts:261`](../packages/core/agent/src/runtime-types.ts) | [`agent`](../packages/core/agent) (`serial`) | `agent-team`, `browser-use-runtime`, `claude-code-mods`, [`file-reference-local`](../packages/context/file-reference-local), [`goal`](../packages/goal/goal), [`goal-round-driver`](../packages/goal/goal-round-driver), [`hooks-claude-code`](../packages/experimental/hooks-claude-code), [`hooks-codex`](../packages/experimental/hooks-codex), [`loader-smoke`](../packages/test-support/loader-smoke), `session-controller`, `tool-agent-team`, [`tool-subagent`](../packages/subagent/tool-subagent) | +| `agent/created` | `serial` | [`packages/core/agent/src/runtime-types.ts:261`](../packages/core/agent/src/runtime-types.ts) | [`agent`](../packages/core/agent) (`serial`) | `agent-team`, `browser-use-runtime`, `claude-code-mods`, [`cua`](../packages/experimental/cua), [`file-reference-local`](../packages/context/file-reference-local), [`goal`](../packages/goal/goal), [`goal-round-driver`](../packages/goal/goal-round-driver), [`hooks-claude-code`](../packages/experimental/hooks-claude-code), [`hooks-codex`](../packages/experimental/hooks-codex), [`loader-smoke`](../packages/test-support/loader-smoke), `session-controller`, `tool-agent-team`, [`tool-subagent`](../packages/subagent/tool-subagent) | | `agent/disposed` | `emit` | [`packages/core/agent/src/runtime-types.ts:270`](../packages/core/agent/src/runtime-types.ts) | [`agent`](../packages/core/agent) (`events.dispatch`) | [`agent-loop`](../packages/core/agent-loop), `claude-code-mods`, [`file-reference-local`](../packages/context/file-reference-local), [`goal-round-driver`](../packages/goal/goal-round-driver), `session-controller`, [`subagent`](../packages/subagent/subagent), `tool-agent-team`, [`tool-subagent`](../packages/subagent/tool-subagent) | | `agent/error` | `emit` | [`packages/core/agent/src/runtime-types.ts:393`](../packages/core/agent/src/runtime-types.ts) | [`agent-loop`](../packages/core/agent-loop) (`emit`) | [`acp`](../packages/acp/acp), [`deepseek-account`](../packages/credentials/deepseek-account), [`goal-round-driver`](../packages/goal/goal-round-driver), [`headless`](../packages/bundle/headless), `server`, `session-controller`, [`session-telemetry`](../packages/session/session-telemetry), [`subagent`](../packages/subagent/subagent) | | `agent/inbox/claimed` | `emit` | [`packages/core/agent/src/runtime-types.ts:299`](../packages/core/agent/src/runtime-types.ts) | [`agent-loop`](../packages/core/agent-loop) (`emit`) | [`acp`](../packages/acp/acp), [`goal-round-driver`](../packages/goal/goal-round-driver), [`subagent`](../packages/subagent/subagent), [`tool-jobs`](../packages/jobs/tool-jobs), [`user-questions`](../packages/interaction/user-questions) | diff --git a/docs/event-producer-consumer.zh.md b/docs/event-producer-consumer.zh.md index 582feb2230..78f2a02da4 100644 --- a/docs/event-producer-consumer.zh.md +++ b/docs/event-producer-consumer.zh.md @@ -11,9 +11,9 @@ | Event | Mode | Declared in | Dispatchers | Listeners | | --- | --- | --- | --- | --- | | `agent-loop/config-start-failed` | `emit` | [`packages/core/agent-loop/src/index.ts:196`](../packages/core/agent-loop/src/index.ts) | [`agent-loop`](../packages/core/agent-loop) (`events.dispatch`) | - | -| `agent-preset/selected` | `emit` | [`packages/preset/agent-preset-registry/src/types.ts:66`](../packages/preset/agent-preset-registry/src/types.ts) | [`agent-preset-registry`](../packages/preset/agent-preset-registry) (`emit`) | `remotes` | +| `agent-preset/selected` | `emit` | [`packages/preset/agent-preset-registry/src/types.ts:66`](../packages/preset/agent-preset-registry/src/types.ts) | [`agent-preset-registry`](../packages/preset/agent-preset-registry) (`emit`) | [`cua`](../packages/experimental/cua), `remotes` | | `agent/assistant-stream` | `emit` | [`packages/core/agent/src/runtime-types.ts:363`](../packages/core/agent/src/runtime-types.ts) | [`agent-loop`](../packages/core/agent-loop) (`emit`) | [`headless`](../packages/bundle/headless), `session-controller` | -| `agent/created` | `serial` | [`packages/core/agent/src/runtime-types.ts:261`](../packages/core/agent/src/runtime-types.ts) | [`agent`](../packages/core/agent) (`serial`) | `agent-team`, `browser-use-runtime`, `claude-code-mods`, [`file-reference-local`](../packages/context/file-reference-local), [`goal`](../packages/goal/goal), [`goal-round-driver`](../packages/goal/goal-round-driver), [`hooks-claude-code`](../packages/experimental/hooks-claude-code), [`hooks-codex`](../packages/experimental/hooks-codex), [`loader-smoke`](../packages/test-support/loader-smoke), `session-controller`, `tool-agent-team`, [`tool-subagent`](../packages/subagent/tool-subagent) | +| `agent/created` | `serial` | [`packages/core/agent/src/runtime-types.ts:261`](../packages/core/agent/src/runtime-types.ts) | [`agent`](../packages/core/agent) (`serial`) | `agent-team`, `browser-use-runtime`, `claude-code-mods`, [`cua`](../packages/experimental/cua), [`file-reference-local`](../packages/context/file-reference-local), [`goal`](../packages/goal/goal), [`goal-round-driver`](../packages/goal/goal-round-driver), [`hooks-claude-code`](../packages/experimental/hooks-claude-code), [`hooks-codex`](../packages/experimental/hooks-codex), [`loader-smoke`](../packages/test-support/loader-smoke), `session-controller`, `tool-agent-team`, [`tool-subagent`](../packages/subagent/tool-subagent) | | `agent/disposed` | `emit` | [`packages/core/agent/src/runtime-types.ts:270`](../packages/core/agent/src/runtime-types.ts) | [`agent`](../packages/core/agent) (`events.dispatch`) | [`agent-loop`](../packages/core/agent-loop), `claude-code-mods`, [`file-reference-local`](../packages/context/file-reference-local), [`goal-round-driver`](../packages/goal/goal-round-driver), `session-controller`, [`subagent`](../packages/subagent/subagent), `tool-agent-team`, [`tool-subagent`](../packages/subagent/tool-subagent) | | `agent/error` | `emit` | [`packages/core/agent/src/runtime-types.ts:393`](../packages/core/agent/src/runtime-types.ts) | [`agent-loop`](../packages/core/agent-loop) (`emit`) | [`acp`](../packages/acp/acp), [`deepseek-account`](../packages/credentials/deepseek-account), [`goal-round-driver`](../packages/goal/goal-round-driver), [`headless`](../packages/bundle/headless), `server`, `session-controller`, [`session-telemetry`](../packages/session/session-telemetry), [`subagent`](../packages/subagent/subagent) | | `agent/inbox/claimed` | `emit` | [`packages/core/agent/src/runtime-types.ts:299`](../packages/core/agent/src/runtime-types.ts) | [`agent-loop`](../packages/core/agent-loop) (`emit`) | [`acp`](../packages/acp/acp), [`goal-round-driver`](../packages/goal/goal-round-driver), [`subagent`](../packages/subagent/subagent), [`tool-jobs`](../packages/jobs/tool-jobs), [`user-questions`](../packages/interaction/user-questions) | diff --git a/docs/module-graph.md b/docs/module-graph.md index 1eda478157..403450aac0 100644 --- a/docs/module-graph.md +++ b/docs/module-graph.md @@ -248,6 +248,7 @@ flowchart TD end subgraph group_experimental["packages/experimental"] pkg_cua["cua"] + pkg_cua_bundle["cua-bundle"] pkg_experimental_agent_team["experimental-agent-team"] pkg_experimental_agent_team_profile["experimental-agent-team-profile"] pkg_experimental_api_speech_to_text["experimental-api-speech-to-text"] @@ -760,6 +761,9 @@ flowchart TD pkg_deepseek_account_platform --> pkg_credentials pkg_deepseek_account_platform --> pkg_deepseek_account pkg_deepseek_account_platform --> pkg_host_webserver + pkg_cua --> pkg_attachment + pkg_cua --> pkg_scope + pkg_cua --> pkg_user_questions pkg_experimental_ptc_runtime_python --> pkg_ptc_runtime pkg_experimental_ptc_runtime_python --> pkg_sandbox pkg_experimental_ptc_runtime_python --> pkg_sandbox_policy @@ -903,14 +907,6 @@ flowchart TD pkg_workspace_changes --> pkg_session pkg_workspace_changes --> pkg_subprocess pkg_workspace_changes --> pkg_tools - pkg_cua --> pkg_agent - pkg_cua --> pkg_attachment - pkg_cua --> pkg_computer_use - pkg_cua --> pkg_llm - pkg_cua --> pkg_subprocess - pkg_cua --> pkg_system_prompt - pkg_cua --> pkg_tools - pkg_cua --> pkg_user_questions pkg_experimental_browser_use_chrome_devtools_mcp --> pkg_agent pkg_experimental_browser_use_chrome_devtools_mcp --> pkg_browser_use pkg_experimental_browser_use_chrome_devtools_mcp --> pkg_system_prompt @@ -1537,6 +1533,7 @@ flowchart TD | [`client-web`](../packages/client/web) | `client` | — | | [`credentials`](../packages/credentials/credentials) | `credentials` | — | | [`office-to-pdf`](../packages/document/office-to-pdf) | `document` | — | +| [`cua-bundle`](../packages/experimental/cua-bundle) | `experimental` | — | | [`experimental-agent-team-profile`](../packages/experimental/agent-team-profile) | `experimental` | — | | [`experimental-badge-skill-bundle`](../packages/experimental/badge-skill-bundle) | `experimental` | — | | [`experimental-client-ui-agent-team`](../packages/experimental/client-ui-agent-team) | `experimental` | — | @@ -1674,6 +1671,7 @@ flowchart TD | [`api-account-controller`](../packages/api/account-controller) | `api` | [`agent`](../packages/core/agent), [`deepseek-account`](../packages/credentials/deepseek-account), [`typert-protocol`](../packages/typert/protocol) | | [`compaction`](../packages/compaction/compaction) | `compaction` | [`brand`](../packages/util/brand), [`commands`](../packages/interaction/commands), [`llm`](../packages/llm/llm), [`session`](../packages/core/session) | | [`deepseek-account-platform`](../packages/credentials/deepseek-account-platform) | `credentials` | [`authorization`](../packages/credentials/authorization), [`credentials`](../packages/credentials/credentials), [`deepseek-account`](../packages/credentials/deepseek-account), [`host-webserver`](../packages/host/webserver) | +| [`cua`](../packages/experimental/cua) | `experimental` | [`attachment`](../packages/attachment/attachment), [`scope`](../packages/core/scope), [`user-questions`](../packages/interaction/user-questions) | | [`experimental-ptc-runtime-python`](../packages/experimental/ptc-runtime-python) | `experimental` | [`ptc-runtime`](../packages/ptc-runtime/ptc-runtime), [`sandbox`](../packages/sandbox/sandbox), [`sandbox-policy`](../packages/sandbox/sandbox-policy), [`timeout`](../packages/util/timeout), [`util-values`](../packages/util/values) | | [`experimental-worktree`](../packages/experimental/worktree) | `experimental` | [`agent`](../packages/core/agent), [`fs`](../packages/fs/fs), [`sandbox`](../packages/sandbox/sandbox), [`sandbox-policy`](../packages/sandbox/sandbox-policy), [`subprocess`](../packages/subprocess/subprocess), [`working-directory`](../packages/session/working-directory) | | [`command-feedback`](../packages/feedback/command-feedback) | `feedback` | [`anonymous-user-id`](../packages/identity/anonymous-user-id), [`commands`](../packages/interaction/commands), [`session`](../packages/core/session), [`typert-protocol`](../packages/typert/protocol) | @@ -1701,7 +1699,6 @@ flowchart TD | [`file-reference-local`](../packages/context/file-reference-local) | `context` | [`agent`](../packages/core/agent), [`file-reference`](../packages/context/file-reference), [`system-prompt`](../packages/core/system-prompt), [`tools`](../packages/core/tools), [`working-directory`](../packages/session/working-directory) | | [`tool-present`](../packages/deliverables/tool-present) | `deliverables` | [`agent`](../packages/core/agent), [`fs`](../packages/fs/fs), [`llm`](../packages/llm/llm), [`session`](../packages/core/session), [`tools`](../packages/core/tools), [`working-directory`](../packages/session/working-directory) | | [`workspace-changes`](../packages/deliverables/workspace-changes) | `deliverables` | [`agent`](../packages/core/agent), [`fs`](../packages/fs/fs), [`session`](../packages/core/session), [`subprocess`](../packages/subprocess/subprocess), [`tools`](../packages/core/tools) | -| [`cua`](../packages/experimental/cua) | `experimental` | [`agent`](../packages/core/agent), [`attachment`](../packages/attachment/attachment), [`computer-use`](../packages/computer-use/computer-use), [`llm`](../packages/llm/llm), [`subprocess`](../packages/subprocess/subprocess), [`system-prompt`](../packages/core/system-prompt), [`tools`](../packages/core/tools), [`user-questions`](../packages/interaction/user-questions) | | [`experimental-browser-use-chrome-devtools-mcp`](../packages/experimental/browser-use-chrome-devtools-mcp) | `experimental` | [`agent`](../packages/core/agent), [`browser-use`](../packages/browser-use/browser-use), [`system-prompt`](../packages/core/system-prompt), [`tools`](../packages/core/tools), [`working-directory`](../packages/session/working-directory) | | [`experimental-browser-use-playwright-mcp`](../packages/experimental/browser-use-playwright-mcp) | `experimental` | [`agent`](../packages/core/agent), [`browser-use`](../packages/browser-use/browser-use), [`system-prompt`](../packages/core/system-prompt), [`tools`](../packages/core/tools), [`working-directory`](../packages/session/working-directory) | | [`experimental-browser-use-stagehand-native`](../packages/experimental/browser-use-stagehand-native) | `experimental` | [`agent`](../packages/core/agent), [`browser-use`](../packages/browser-use/browser-use), [`system-prompt`](../packages/core/system-prompt), [`tools`](../packages/core/tools) | diff --git a/docs/module-graph.zh.md b/docs/module-graph.zh.md index 928a955aef..cc65f2db3c 100644 --- a/docs/module-graph.zh.md +++ b/docs/module-graph.zh.md @@ -250,6 +250,7 @@ flowchart TD end subgraph group_experimental["packages/experimental"] pkg_cua["cua"] + pkg_cua_bundle["cua-bundle"] pkg_experimental_agent_team["experimental-agent-team"] pkg_experimental_agent_team_profile["experimental-agent-team-profile"] pkg_experimental_api_speech_to_text["experimental-api-speech-to-text"] @@ -762,6 +763,9 @@ flowchart TD pkg_deepseek_account_platform --> pkg_credentials pkg_deepseek_account_platform --> pkg_deepseek_account pkg_deepseek_account_platform --> pkg_host_webserver + pkg_cua --> pkg_attachment + pkg_cua --> pkg_scope + pkg_cua --> pkg_user_questions pkg_experimental_ptc_runtime_python --> pkg_ptc_runtime pkg_experimental_ptc_runtime_python --> pkg_sandbox pkg_experimental_ptc_runtime_python --> pkg_sandbox_policy @@ -905,14 +909,6 @@ flowchart TD pkg_workspace_changes --> pkg_session pkg_workspace_changes --> pkg_subprocess pkg_workspace_changes --> pkg_tools - pkg_cua --> pkg_agent - pkg_cua --> pkg_attachment - pkg_cua --> pkg_computer_use - pkg_cua --> pkg_llm - pkg_cua --> pkg_subprocess - pkg_cua --> pkg_system_prompt - pkg_cua --> pkg_tools - pkg_cua --> pkg_user_questions pkg_experimental_browser_use_chrome_devtools_mcp --> pkg_agent pkg_experimental_browser_use_chrome_devtools_mcp --> pkg_browser_use pkg_experimental_browser_use_chrome_devtools_mcp --> pkg_system_prompt @@ -1539,6 +1535,7 @@ flowchart TD | [`client-web`](../packages/client/web) | `client` | — | | [`credentials`](../packages/credentials/credentials) | `credentials` | — | | [`office-to-pdf`](../packages/document/office-to-pdf) | `document` | — | +| [`cua-bundle`](../packages/experimental/cua-bundle) | `experimental` | — | | [`experimental-agent-team-profile`](../packages/experimental/agent-team-profile) | `experimental` | — | | [`experimental-badge-skill-bundle`](../packages/experimental/badge-skill-bundle) | `experimental` | — | | [`experimental-client-ui-agent-team`](../packages/experimental/client-ui-agent-team) | `experimental` | — | @@ -1676,6 +1673,7 @@ flowchart TD | [`api-account-controller`](../packages/api/account-controller) | `api` | [`agent`](../packages/core/agent), [`deepseek-account`](../packages/credentials/deepseek-account), [`typert-protocol`](../packages/typert/protocol) | | [`compaction`](../packages/compaction/compaction) | `compaction` | [`brand`](../packages/util/brand), [`commands`](../packages/interaction/commands), [`llm`](../packages/llm/llm), [`session`](../packages/core/session) | | [`deepseek-account-platform`](../packages/credentials/deepseek-account-platform) | `credentials` | [`authorization`](../packages/credentials/authorization), [`credentials`](../packages/credentials/credentials), [`deepseek-account`](../packages/credentials/deepseek-account), [`host-webserver`](../packages/host/webserver) | +| [`cua`](../packages/experimental/cua) | `experimental` | [`attachment`](../packages/attachment/attachment), [`scope`](../packages/core/scope), [`user-questions`](../packages/interaction/user-questions) | | [`experimental-ptc-runtime-python`](../packages/experimental/ptc-runtime-python) | `experimental` | [`ptc-runtime`](../packages/ptc-runtime/ptc-runtime), [`sandbox`](../packages/sandbox/sandbox), [`sandbox-policy`](../packages/sandbox/sandbox-policy), [`timeout`](../packages/util/timeout), [`util-values`](../packages/util/values) | | [`experimental-worktree`](../packages/experimental/worktree) | `experimental` | [`agent`](../packages/core/agent), [`fs`](../packages/fs/fs), [`sandbox`](../packages/sandbox/sandbox), [`sandbox-policy`](../packages/sandbox/sandbox-policy), [`subprocess`](../packages/subprocess/subprocess), [`working-directory`](../packages/session/working-directory) | | [`command-feedback`](../packages/feedback/command-feedback) | `feedback` | [`anonymous-user-id`](../packages/identity/anonymous-user-id), [`commands`](../packages/interaction/commands), [`session`](../packages/core/session), [`typert-protocol`](../packages/typert/protocol) | @@ -1703,7 +1701,6 @@ flowchart TD | [`file-reference-local`](../packages/context/file-reference-local) | `context` | [`agent`](../packages/core/agent), [`file-reference`](../packages/context/file-reference), [`system-prompt`](../packages/core/system-prompt), [`tools`](../packages/core/tools), [`working-directory`](../packages/session/working-directory) | | [`tool-present`](../packages/deliverables/tool-present) | `deliverables` | [`agent`](../packages/core/agent), [`fs`](../packages/fs/fs), [`llm`](../packages/llm/llm), [`session`](../packages/core/session), [`tools`](../packages/core/tools), [`working-directory`](../packages/session/working-directory) | | [`workspace-changes`](../packages/deliverables/workspace-changes) | `deliverables` | [`agent`](../packages/core/agent), [`fs`](../packages/fs/fs), [`session`](../packages/core/session), [`subprocess`](../packages/subprocess/subprocess), [`tools`](../packages/core/tools) | -| [`cua`](../packages/experimental/cua) | `experimental` | [`agent`](../packages/core/agent), [`attachment`](../packages/attachment/attachment), [`computer-use`](../packages/computer-use/computer-use), [`llm`](../packages/llm/llm), [`subprocess`](../packages/subprocess/subprocess), [`system-prompt`](../packages/core/system-prompt), [`tools`](../packages/core/tools), [`user-questions`](../packages/interaction/user-questions) | | [`experimental-browser-use-chrome-devtools-mcp`](../packages/experimental/browser-use-chrome-devtools-mcp) | `experimental` | [`agent`](../packages/core/agent), [`browser-use`](../packages/browser-use/browser-use), [`system-prompt`](../packages/core/system-prompt), [`tools`](../packages/core/tools), [`working-directory`](../packages/session/working-directory) | | [`experimental-browser-use-playwright-mcp`](../packages/experimental/browser-use-playwright-mcp) | `experimental` | [`agent`](../packages/core/agent), [`browser-use`](../packages/browser-use/browser-use), [`system-prompt`](../packages/core/system-prompt), [`tools`](../packages/core/tools), [`working-directory`](../packages/session/working-directory) | | [`experimental-browser-use-stagehand-native`](../packages/experimental/browser-use-stagehand-native) | `experimental` | [`agent`](../packages/core/agent), [`browser-use`](../packages/browser-use/browser-use), [`system-prompt`](../packages/core/system-prompt), [`tools`](../packages/core/tools) | diff --git a/docs/subsystems/computer-use.i18n.yaml b/docs/subsystems/computer-use.i18n.yaml index a0cf7675d0..fc82ebf720 100644 --- a/docs/subsystems/computer-use.i18n.yaml +++ b/docs/subsystems/computer-use.i18n.yaml @@ -3,8 +3,8 @@ # After editing either side, bring the other along and re-record with: # pnpm run verify-translation-pairing --write docs/subsystems/computer-use.md /computer-use: - en: a82a6d1ec3fa9724 - zh: 17483872a54d6899 + en: aff84e3ac8c41594 + zh: bee7fc09854bfa8b /computer-use/choose-a-provider: en: fa2e0d7b41c0450b zh: af9a799711f901e6 diff --git a/docs/subsystems/computer-use.md b/docs/subsystems/computer-use.md index 6331c0f492..bc8f1b0491 100644 --- a/docs/subsystems/computer-use.md +++ b/docs/subsystems/computer-use.md @@ -4,9 +4,19 @@ English | [中文](computer-use.zh.md) Computer use lets a model observe and operate the local desktop through a configured provider. The shared DSH capability is called **computer use**; **Cua Driver** names the upstream implementation. +[Computer Use](../../packages/experimental/cua-bundle/README.md) provides an optional experimental switch for macOS CUA permission setup. The [`macOS CUA provider`](../../packages/experimental/cua-macos/README.md) uses the [shared CUA objects and executor](../../packages/experimental/cua/README.md), which enforce application consent through User Questions and profile configuration. +The CUA provider owns `ctx.cuaAccess` for system-permission status, explicit system requests, and application revocation. Its [permission page](../../packages/experimental/cua/README.md) projects these controls; app/window operations enforce the provider's consent policy before dispatch. + +| Permission type | Meaning | +|---|---| +| `CuaApplicationId`, `CuaApplicationGrant` | An executable/signature identity and its displayed saved decision. | +| `CuaSystemPermission` | Accessibility or Screen Recording, selected by the human. | +| `CuaPermissionProcess`, `CuaSystemPermissionState` | Native privilege checks, observed helper/parent processes, and the application identified by macOS process responsibility at helper startup. Application metadata is absent if the optional system query or application lookup is unavailable. | +| `CuaApplicationState` | Saved grants, refused applications, and whether persistence is available. Session-only grants remain internal to the provider. System observations and application icons are queried separately so the page can display each result as it arrives. | + ## Choose a provider Mount [`dsh-computer-use`](../../packages/computer-use/computer-use/README.md) and one provider in the same composition. Both Cua Driver providers are experimental public npm packages and require explicit activation. @@ -56,4 +66,70 @@ register(name: ComputerUseProviderName): () => Promise ``` Source: [`packages/computer-use/computer-use/src/index.ts`](../../packages/computer-use/computer-use/src/index.ts) + + + +### `ctx.cuaAccess` — `CuaAccess` + +Browser controls for provider-owned native permission and application decisions. + +```ts cordis-catalog +/** Read PTC execution state separately from native permission checks. + * @returns Whether the shared runtime retains memory and has an editable profile entry. + */ +@Remote ptcConfiguration(): { persistent: boolean; writable: boolean } + +/** Persist in-memory PTC execution through the existing profile editor and Loader lifecycle. + * @returns Completion after the shared runtime reload; already-persistent runtimes are left intact. + */ +@Remote async enablePersistentPtc(): Promise + +/** + * Read application decisions without waiting for the native helper. + * @returns Saved grants and refused applications with their persistence owner. + */ +@Remote applications(): CuaApplicationState + +/** + * Inspect system privileges independently of application decisions and icons. + * @returns Fresh native preflight and launch metadata. + */ +@Remote async systemStatus(): Promise + +/** + * Load presentation icons separately from application decisions. + * @returns Optional native PNG icons keyed by executable path for saved and blocked applications. + */ +@Remote async applicationIcons(): Promise> + +/** + * Request a system privilege explicitly selected by the user. + * @param permission macOS privilege to request. + * @returns Preflight state after requesting; a false value never implies consent. + */ +@Remote async requestPermission(permission: CuaSystemPermission): Promise + +/** + * Open the macOS privacy pane selected by the user. + * @param permission Corresponding system privilege. + * @returns Completion after macOS accepted the open request. + */ +@Remote async openSettings(permission: CuaSystemPermission): Promise + +/** + * Remove saved and session access and join affected operations. + * @param id Application identity displayed on the permission page. + * @returns Completion after persistence and cancellation; dispatched input cannot be undone. + */ +@Remote revoke(id: CuaApplicationId): Promise + +/** + * Clear this application's refusals across sessions so each session may ask for access again. + * @param id Refused or revoked application displayed on the permission page. + * @returns Completion after clearing refusals; no permission is granted and no action is replayed. + */ +@Remote async retryApproval(id: CuaApplicationId): Promise +``` + +Source: [`packages/experimental/cua/src/access.ts`](../../packages/experimental/cua/src/access.ts) diff --git a/docs/subsystems/computer-use.zh.md b/docs/subsystems/computer-use.zh.md index cd4901bd1a..1323c6b033 100644 --- a/docs/subsystems/computer-use.zh.md +++ b/docs/subsystems/computer-use.zh.md @@ -4,9 +4,19 @@ 计算机操作让模型通过配置的提供方观察并操作本地桌面。DSH 的共享能力称为 **computer use（计算机操作）**；**Cua Driver** 是上游实现的名称。 +[Computer Use](../../packages/experimental/cua-bundle/README.zh.md) 提供 macOS CUA 及权限设置 的可选实验性开关。 [macOS CUA 提供方](../../packages/experimental/cua-macos/README.zh.md)使用[共享 CUA 对象与执行器](../../packages/experimental/cua/README.zh.md)，通过 User Questions 和配置档设置执行应用授权。 +CUA provider 通过 `ctx.cuaAccess` 提供系统权限状态、用户主动申请和应用授权撤销。[权限页面](../../packages/experimental/cua/README.zh.md)投影这些控件；app/window 操作在分派前执行 provider 的授权检查。 + +| 权限类型 | 含义 | +|---|---| +| `CuaApplicationId`、`CuaApplicationGrant` | 可执行文件和签名身份，以及显示给用户的已保存决定。 | +| `CuaSystemPermission` | 用户选择申请的辅助功能或屏幕录制权限。 | +| `CuaPermissionProcess`、`CuaSystemPermissionState` | 观测到的 helper、父进程和原生权限状态，以及 helper 启动时由 macOS 责任进程记录识别出的应用。可选系统查询或应用查找不可用时，省略应用信息。 | +| `CuaApplicationState` | 持久授权、拒绝记录，以及能否持久保存。会话授权只保留在 provider 内部。系统权限和应用图标分别查询，页面在各项结果返回后分别显示。 | + ## 选择提供方 在同一组合中挂载 [`dsh-computer-use`](../../packages/computer-use/computer-use/README.zh.md) 和一个提供方。两个 Cua Driver 提供方都是公开发布到 npm 的实验性包，均需显式启用。 @@ -56,4 +66,70 @@ register(name: ComputerUseProviderName): () => Promise ``` Source: [`packages/computer-use/computer-use/src/index.ts`](../../packages/computer-use/computer-use/src/index.ts) + + + +### `ctx.cuaAccess` — `CuaAccess` + +Browser controls for provider-owned native permission and application decisions. + +```ts cordis-catalog +/** Read PTC execution state separately from native permission checks. + * @returns Whether the shared runtime retains memory and has an editable profile entry. + */ +@Remote ptcConfiguration(): { persistent: boolean; writable: boolean } + +/** Persist in-memory PTC execution through the existing profile editor and Loader lifecycle. + * @returns Completion after the shared runtime reload; already-persistent runtimes are left intact. + */ +@Remote async enablePersistentPtc(): Promise + +/** + * Read application decisions without waiting for the native helper. + * @returns Saved grants and refused applications with their persistence owner. + */ +@Remote applications(): CuaApplicationState + +/** + * Inspect system privileges independently of application decisions and icons. + * @returns Fresh native preflight and launch metadata. + */ +@Remote async systemStatus(): Promise + +/** + * Load presentation icons separately from application decisions. + * @returns Optional native PNG icons keyed by executable path for saved and blocked applications. + */ +@Remote async applicationIcons(): Promise> + +/** + * Request a system privilege explicitly selected by the user. + * @param permission macOS privilege to request. + * @returns Preflight state after requesting; a false value never implies consent. + */ +@Remote async requestPermission(permission: CuaSystemPermission): Promise + +/** + * Open the macOS privacy pane selected by the user. + * @param permission Corresponding system privilege. + * @returns Completion after macOS accepted the open request. + */ +@Remote async openSettings(permission: CuaSystemPermission): Promise + +/** + * Remove saved and session access and join affected operations. + * @param id Application identity displayed on the permission page. + * @returns Completion after persistence and cancellation; dispatched input cannot be undone. + */ +@Remote revoke(id: CuaApplicationId): Promise + +/** + * Clear this application's refusals across sessions so each session may ask for access again. + * @param id Refused or revoked application displayed on the permission page. + * @returns Completion after clearing refusals; no permission is granted and no action is replayed. + */ +@Remote async retryApproval(id: CuaApplicationId): Promise +``` + +Source: [`packages/experimental/cua/src/access.ts`](../../packages/experimental/cua/src/access.ts) diff --git a/docs/upgrade-guide/v0.2.1-rc.2/cua-preset-scope/guide.i18n.yaml b/docs/upgrade-guide/v0.2.1-rc.2/cua-preset-scope/guide.i18n.yaml new file mode 100644 index 0000000000..7313c9c61a --- /dev/null +++ b/docs/upgrade-guide/v0.2.1-rc.2/cua-preset-scope/guide.i18n.yaml @@ -0,0 +1,16 @@ +# Bilingual-pair consistency record for guide.md (docs/i18n/README.md): per heading +# section, a hash of its English and Chinese blocks outside code blocks and generated regions. +# After editing either side, bring the other along and re-record with: +# pnpm run verify-translation-pairing --write docs/upgrade-guide/v0.2.1-rc.2/cua-preset-scope/guide.md +/: + en: 36d6b79b48c08dd5 + zh: 814caece159e9b0e +/cua-defaults-to-ptc-session-scopes: + en: a53b6884785bdf56 + zh: 7d27fc9b029e0e3e +/cua-defaults-to-ptc-session-scopes/change: + en: dac96a29e3751249 + zh: d48cb030f43c07e4 +/cua-defaults-to-ptc-session-scopes/migration: + en: f3d82fe45e41c712 + zh: e55e424f5a2c096a diff --git a/docs/upgrade-guide/v0.2.1-rc.2/cua-preset-scope/guide.md b/docs/upgrade-guide/v0.2.1-rc.2/cua-preset-scope/guide.md new file mode 100644 index 0000000000..7db56c3afd --- /dev/null +++ b/docs/upgrade-guide/v0.2.1-rc.2/cua-preset-scope/guide.md @@ -0,0 +1,18 @@ +--- +kind: upgrade-guide +description: "Standalone native CUA compositions must explicitly select global tool registration." +--- + +# CUA defaults to PTC session scopes + +English | [中文](guide.zh.md) + +## Change + +The native CUA provider defaults `agentPreset` to `ptc`. It registers tools for existing and new PTC Agents, including when a saved application-grants configuration omits the new field. Custom compositions without an Agent preset registry no longer receive global CUA tools by default. + +## Migration + +1. For the standard Computer Use switch, keep the default; existing application grants remain usable. +2. For a standalone composition that needs global tools, add `agentPreset: false` under the native CUA entry’s `config` in its profile patch or overlay. Preserve its other fields, including `allowedApps` and `helperPath`. +3. Restart or let HMR apply the edit, then confirm that the intended Agent exposes `cua` in native mode or through the PTC SDK. diff --git a/docs/upgrade-guide/v0.2.1-rc.2/cua-preset-scope/guide.zh.md b/docs/upgrade-guide/v0.2.1-rc.2/cua-preset-scope/guide.zh.md new file mode 100644 index 0000000000..ff6f6ccbc0 --- /dev/null +++ b/docs/upgrade-guide/v0.2.1-rc.2/cua-preset-scope/guide.zh.md @@ -0,0 +1,18 @@ +--- +kind: upgrade-guide +description: "独立原生 CUA 组合需要显式选择全局工具注册。" +--- + +# CUA 默认使用 PTC 会话作用域 + +[English](guide.md) | 中文 + +## 变更 + +原生 CUA 提供方的 `agentPreset` 默认为 `ptc`。工具注册到已有和新建 PTC Agent，即使保存的应用授权配置没有这个新字段也会生效。不包含 Agent preset 注册表的自定义组合不再默认获得全局 CUA 工具。 + +## 迁移 + +1. 使用标准 Computer Use 开关时保持默认值，已有应用授权仍可使用。 +2. 独立组合若需要全局工具，在 profile patch 或 overlay 的原生 CUA 条目 `config` 下添加 `agentPreset: false`，保留 `allowedApps`、`helperPath` 等其他字段。 +3. 重启或等待 HMR 应用后，确认目标 Agent 能在原生模式或 PTC SDK 中使用 `cua`。 diff --git a/package.json b/package.json index 3053d646eb..fd29dfb45f 100644 --- a/package.json +++ b/package.json @@ -291,6 +291,7 @@ "@deepseek-ai/dsh-working-directory": "workspace:*", "@deepseek-ai/dsh-experimental-worktree": "workspace:*", "@deepseek-ai/dsh-experimental-tool-worktree": "workspace:*", + "@deepseek-ai/dsh-cua-bundle": "workspace:*", "@deepseek-ai/dsh-experimental-subagent-oar": "workspace:*", "@deepseek-ai/dsh-experimental-tool-subagent-oar": "workspace:*", "@deepseek-ai/dsh-experimental-cua-macos": "workspace:*" diff --git a/packages/boot/app-boot/README.i18n.yaml b/packages/boot/app-boot/README.i18n.yaml index c13fb647d1..be702d4775 100644 --- a/packages/boot/app-boot/README.i18n.yaml +++ b/packages/boot/app-boot/README.i18n.yaml @@ -24,8 +24,8 @@ en: 4a424cd401556f20 zh: 0b34dd18e0db6cea /deepseek-ai-dsh-app-boot/use-this-package/profiles: - en: d2d520668668292c - zh: b7deeab27032a3ad + en: 0a1b5f4f40d835a7 + zh: d8f4b54160aa033b /deepseek-ai-dsh-app-boot/use-this-package/previewing-the-effective-configuration: en: 646a0ccdd7881ad1 zh: e3696acb1573dd7b diff --git a/packages/boot/app-boot/README.md b/packages/boot/app-boot/README.md index 59c38476da..4b43180589 100644 --- a/packages/boot/app-boot/README.md +++ b/packages/boot/app-boot/README.md @@ -64,7 +64,7 @@ The enabled `dsh-hmr` plugin watches the profile manifest and both user patch fi Inserted plugin names may be absolute filesystem paths, file URLs, or package specifiers. Patch loading converts absolute paths and patch-relative `./` or `../` paths to file URLs within `insert` rows and their nested groups; existing-entry name assertions and replacement `config` values remain literal. -Before mounting profile rows, the `dsh` launcher computes one immutable runtime resolution from the installation and ordered bundle dependency graphs. Every profile launcher uses runtime resolution, including plain Node, packaged executables, and the Electron Host. It installs the runtime resolution through Node's ESM and CommonJS resolvers without creating fallback links. +Before mounting profile rows, the `dsh` launcher computes one immutable runtime resolution from the installation and ordered bundle dependency graphs. Every profile launcher uses runtime resolution, including plain Node, packaged executables, and the Electron Host. It installs the runtime resolution through Node's ESM and CommonJS resolvers without creating fallback links. Traversal includes installed optional dependencies and skips those absent from the target installation. `sanitizeProfile(binName, profileDir, bundles)` provides filesystem recovery without loading plugins or parsing patches. Desktop uses it for native fatal recovery. Call it only after stopping the profile and excluding concurrent profile writes. It renames the profile’s `cordis.patch.yml` to a unique `.bak-` sibling and restores the supplied bundle list, preserving installed packages and other manifest fields. The timestamp is Unix time in milliseconds; collisions append an ordinal (`-1`, `-2`, …) without changing it. It returns the backup path, or `undefined` when no patch exists; missing profiles remain absent. Profile initialization recreates an empty patch on the next launch. The home-level patch is unchanged. Invalid profile JSON fails before mutation; later errors propagate and retain completed changes for retry. diff --git a/packages/boot/app-boot/README.zh.md b/packages/boot/app-boot/README.zh.md index 73e383e04a..eee5749ba6 100644 --- a/packages/boot/app-boot/README.zh.md +++ b/packages/boot/app-boot/README.zh.md @@ -64,7 +64,7 @@ profile 导入插件前，DSH 会将其声明的 DSH 版本要求与 `getDshRunt 插入条目的插件名可以是绝对文件系统路径、文件 URL 或包标识符。patch 加载会把 `insert` 条目及其嵌套分组中的绝对路径以及相对于 patch 文件的 `./` 或 `../` 路径转换为文件 URL；对已有条目名称的断言及替换用的 `config` 值保持原样。 -挂载 profile 条目前，`dsh` launcher 会从安装依赖图与有序 bundle 依赖图计算一份不可变的 runtime resolution。普通 Node、打包可执行文件与 Electron Host 等所有 profile 启动器都使用 runtime 解析，将 runtime resolution 安装到 Node 的 ESM 与 CommonJS 解析器中，不创建 fallback 链接。 +挂载 profile 条目前，`dsh` launcher 会从安装依赖图与有序 bundle 依赖图计算一份不可变的 runtime resolution。普通 Node、打包可执行文件与 Electron Host 等所有 profile 启动器都使用 runtime 解析，将 runtime resolution 安装到 Node 的 ESM 与 CommonJS 解析器中，不创建 fallback 链接。依赖遍历包含已安装的可选依赖，跳过目标安装中缺席的包。 `sanitizeProfile(binName, profileDir, bundles)` 提供文件恢复，无需加载插件或解析 patch。Desktop 在原生致命错误恢复中调用它。调用前必须停止 profile 并排除并发 profile 写入。它将 profile 的 `cordis.patch.yml` 重命名为带唯一 `.bak-` 后缀的同目录备份，并恢复调用方指定的 bundle 列表，保留已安装包和其他 manifest 字段。时间戳为 Unix 毫秒数；同名备份已存在时追加序号（`-1`、`-2`、……），时间戳保持不变。返回值为备份路径；patch 不存在时返回 `undefined`，缺失的 profile 不会被创建。下次启动的 profile 初始化会重新创建空 patch。home 级 patch 不变。无效 profile JSON 在修改前报错；后续错误向调用方抛出，保留已完成的修改供重试。 diff --git a/packages/boot/app-boot/src/profile.ts b/packages/boot/app-boot/src/profile.ts index 3c7aa74762..db23fbf232 100644 --- a/packages/boot/app-boot/src/profile.ts +++ b/packages/boot/app-boot/src/profile.ts @@ -221,6 +221,7 @@ export const DEFAULT_PROFILE_BUNDLES: readonly string[] = ['@deepseek-ai/dsh-bas * [admission](../../../../.agents/notes/implemented/architecture/2026-09-21-experimental-capabilities-as-optional-bundles.md)). */ export const OPTIONAL_BUNDLES: readonly string[] = [ + '@deepseek-ai/dsh-cua-bundle', '@deepseek-ai/dsh-experimental-session-search', '@deepseek-ai/dsh-experimental-ralph-bundle', '@deepseek-ai/dsh-experimental-terminal-bundle', @@ -374,7 +375,8 @@ function readPackageManifest(anchor: string): ProfileManifest { /** Return dependency names that may be imported by a loader-visible plugin. */ function profileDependencyNames(manifest: ProfileManifest): string[] { - return [...Object.keys(manifest.dependencies ?? {}), ...Object.keys(manifest.peerDependencies ?? {})] + return [...Object.keys(manifest.dependencies ?? {}), ...Object.keys(manifest.peerDependencies ?? {}), + ...Object.keys(manifest.optionalDependencies ?? {})] } /** Resolve the installation packages that the runtime resolver supplies to every profile. */ diff --git a/packages/boot/app-boot/tests/profile-resolution-service.spec.ts b/packages/boot/app-boot/tests/profile-resolution-service.spec.ts index 9ca8a7e0d6..bed3b6c8c1 100644 --- a/packages/boot/app-boot/tests/profile-resolution-service.spec.ts +++ b/packages/boot/app-boot/tests/profile-resolution-service.spec.ts @@ -334,3 +334,28 @@ describe('profile package metadata service', () => { expect(getEnvironmentData(key)).toBe(previous) }) }) + + +it.each(['installation', 'bundle'] as const)('exposes installed optional dependencies from %s and skips absent ones', async (scope) => { + const root = realpathSync.native(mkdtempSync(join(tmpdir(), 'dsh-optional-resolution-'))) + roots.push(root) + const installAnchor = join(root, 'install', 'package.json') + file(installAnchor, JSON.stringify({ name: 'installation', version: '1.0.0' })) + const profileDir = join(root, 'profiles', 'test') + initProfile(profileDir, ['optional-bundle']) + const bundleDir = join(profileDir, 'node_modules', 'optional-bundle') + const bundleManifest = { name: 'optional-bundle', version: '1.0.0', dsh: { bundle: { patch: './cordis.patch.yml' } } } + file(join(bundleDir, 'package.json'), JSON.stringify(bundleManifest)) + file(join(bundleDir, 'cordis.patch.yml'), '[]\n') + const anchor = scope === 'installation' ? installAnchor : join(bundleDir, 'package.json') + file(anchor, JSON.stringify({ ...JSON.parse(readFileSync(anchor, 'utf8')) as object, + optionalDependencies: { 'metadata-lib': '*', 'absent-platform': '*' }, + })) + const optionalDir = join(dirname(anchor), 'node_modules', 'metadata-lib') + pkg(optionalDir, '1.0.0') + const active = await createRuntimeResolution({ + installAnchor, home: root, profile: loadProfileDirectory('dsh', profileDir, installAnchor), + }) + expect(active.entries).toContainEqual(expect.objectContaining({ name: 'metadata-lib', packageDir: optionalDir })) + expect(active.entries.some(entry => entry.name === 'absent-platform')).toBe(false) +}) diff --git a/packages/client/ui-schedule/README.i18n.yaml b/packages/client/ui-schedule/README.i18n.yaml index 205e4c6b84..8a2a17dc02 100644 --- a/packages/client/ui-schedule/README.i18n.yaml +++ b/packages/client/ui-schedule/README.i18n.yaml @@ -15,8 +15,8 @@ en: d152484eb41ac6b4 zh: 405b30feaf48c087 /deepseek-ai-dsh-client-ui-schedule/use-this-package: - en: edba0e1573f3ce42 - zh: 77875b469d94cc62 + en: 7d44cd1640bebfad + zh: 1d6b6c28a3c27115 /deepseek-ai-dsh-client-ui-schedule/use-this-package/time-zone-data-source: en: 6bd6882dbbbefcb0 zh: 03aa15f1a65d13dc diff --git a/packages/client/ui-schedule/README.md b/packages/client/ui-schedule/README.md index 220af1e6c7..6f02ed050a 100644 --- a/packages/client/ui-schedule/README.md +++ b/packages/client/ui-schedule/README.md @@ -25,7 +25,7 @@ Manage active and inactive tasks from the Automation tasks page and the right Si ## Use this package -The shipped Web composition mounts `ui-schedule` with the Host Schedule service. The clock context that supplies the model's current time is preset-owned: `standard`, `cordis`, and `ptc` declare it, and `minimal` declares neither it nor the reminder tools. The Automation tasks sidebar entry opens a global page without selecting or activating a Session. The page header's New action starts a new Session, where the reminder is created by asking the model; the page itself has no creation form. Its task list supports text search and one All, Enabled, and Inactive status filter row; search matches the stored task name, the instruction, and the internal Session id, not a resolved Session title, and the empty state keeps the heading's New action beside the notice. Selecting an active or inactive task opens a second detail level with Rules and Records tabs. Every list row shows the task's stored title with its status, frequency, and next run and never the original Session name; the detail's name control shows the stored title. Rules contains the complete reminder, status, next target for active tasks, original recurrence rule, and task id. Daily rule labels show the saved local time and exact stored IANA zone, preserving nonzero seconds and milliseconds rather than substituting the browser's zone. The Linked session entry in the detail tab strip names the linked Session by its catalog title, falling back to the Session id when no title is available. When the strip cannot fit the tab list beside the labelled entry, the entry becomes an icon button with the same button style as More task actions, with the Session chat icon and a Linked session tooltip that also shows while the entry is disabled; the entry keeps focus when it switches layouts; the tab list scrolls without a visible scrollbar. It opens an available, unarchived Session after metadata loads; pending Session or Workspace metadata, an archived Session, missing Session membership, or a Workspace query error keeps it disabled with an explanation. The click rechecks availability; browsing does not unarchive the Session. Only this action requests Session activation. Deletion is available from the More task actions menu in the detail's top strip, separate from navigation. +The shipped Web composition mounts `ui-schedule` with the Host Schedule service. The clock context that supplies the model's current time is preset-owned: `standard`, `cordis`, and `ptc` declare it, and `minimal` declares neither it nor the reminder tools. The Automation tasks sidebar entry opens a global page without selecting or activating a Session. The page header's New action starts a new Session, where the reminder is created by asking the model; the page itself has no creation form. Its task list supports text search and one All, Enabled, and Inactive status filter row; search matches the stored task name, the instruction, and the internal Session id, not a resolved Session title, and the empty state shows a decorative document-and-clock illustration above its notice and New action, with colors that follow the light or dark theme. Selecting an active or inactive task opens a second detail level with Rules and Records tabs. Every list row shows the task's stored title with its status, frequency, and next run and never the original Session name; the detail's name control shows the stored title. Rules contains the complete reminder, status, next target for active tasks, original recurrence rule, and task id. Daily rule labels show the saved local time and exact stored IANA zone, preserving nonzero seconds and milliseconds rather than substituting the browser's zone. The Linked session entry in the detail tab strip names the linked Session by its catalog title, falling back to the Session id when no title is available. When the strip cannot fit the tab list beside the labelled entry, the entry becomes an icon button with the same button style as More task actions, with the Session chat icon and a Linked session tooltip that also shows while the entry is disabled; the entry keeps focus when it switches layouts; the tab list scrolls without a visible scrollbar. It opens an available, unarchived Session after metadata loads; pending Session or Workspace metadata, an archived Session, missing Session membership, or a Workspace query error keeps it disabled with an explanation. The click rechecks availability; browsing does not unarchive the Session. Only this action requests Session activation. Deletion is available from the More task actions menu in the detail's top strip, separate from navigation. The page header keeps the title at its natural width and moves New to another line when the two no longer fit together. The action label stays on one line. diff --git a/packages/client/ui-schedule/README.zh.md b/packages/client/ui-schedule/README.zh.md index 88575c4405..b7de6727ed 100644 --- a/packages/client/ui-schedule/README.zh.md +++ b/packages/client/ui-schedule/README.zh.md @@ -25,7 +25,7 @@ kind: "package-reference" ## 使用本包 -随发行版交付的 Web 组合把 `ui-schedule` 与 Host Schedule 服务一起挂载。为模型提供当前时间的时钟上下文归 preset 所有：`standard`、`cordis` 与 `ptc` 声明它，`minimal` 既不声明它，也不声明提醒工具。侧栏的自动化任务入口打开全局页面，不选择或激活会话。页面页头的“新建”操作会打开一个新会话，提醒由模型在该会话中创建；页面本身没有创建表单。列表支持文本搜索和一行“全部/已开启/未运行”状态筛选；搜索匹配任务已存储的名称、指令和内部会话 id，不匹配解析出的会话标题，空状态在提示旁保留页头的“新建”操作。选择活动或未运行任务后，进入含“规则”和“任务运行记录”页签的第二层详情。每条列表行显示任务已存储的标题及其状态、频率和下次运行，不显示原会话名称；详情中的名称输入框显示任务已存储的标题。“规则”显示完整提醒、状态、活动任务的下一目标时间、原始重复规则和任务 id。每天的规则标签显示已保存的本地时间与精确的存储 IANA 时区，保留非零秒与毫秒，而不替换为浏览器时区。详情页签条内的“关联会话”入口按会话标题显示，取不到标题时回退显示会话 id。页签条放不下页签列表和带文字的入口时，入口收为与“更多任务操作”同样式的图标按钮，使用会话对话图标，悬停提示“关联会话”，入口禁用时也会显示；切换形态时保留焦点；页签列表可滚动但不显示滚动条。元数据加载后，它可打开可用且未归档的会话；会话或 Workspace 元数据尚未就绪、会话已归档、会话不在列表中或 Workspace 查询失败时，入口保持禁用并显示原因。点击时会重新检查可用性；浏览不会取消会话归档。仅此操作请求激活会话。删除位于详情顶部条带的“更多任务操作”菜单。 +随发行版交付的 Web 组合把 `ui-schedule` 与 Host Schedule 服务一起挂载。为模型提供当前时间的时钟上下文归 preset 所有：`standard`、`cordis` 与 `ptc` 声明它，`minimal` 既不声明它，也不声明提醒工具。侧栏的自动化任务入口打开全局页面，不选择或激活会话。页面页头的“新建”操作会打开一个新会话，提醒由模型在该会话中创建；页面本身没有创建表单。列表支持文本搜索和一行“全部/已开启/未运行”状态筛选；搜索匹配任务已存储的名称、指令和内部会话 id，不匹配解析出的会话标题，空状态在提示和“新建”操作上方显示装饰性的文档与时钟插画，配色跟随浅色或深色主题。选择活动或未运行任务后，进入含“规则”和“任务运行记录”页签的第二层详情。每条列表行显示任务已存储的标题及其状态、频率和下次运行，不显示原会话名称；详情中的名称输入框显示任务已存储的标题。“规则”显示完整提醒、状态、活动任务的下一目标时间、原始重复规则和任务 id。每天的规则标签显示已保存的本地时间与精确的存储 IANA 时区，保留非零秒与毫秒，而不替换为浏览器时区。详情页签条内的“关联会话”入口按会话标题显示，取不到标题时回退显示会话 id。页签条放不下页签列表和带文字的入口时，入口收为与“更多任务操作”同样式的图标按钮，使用会话对话图标，悬停提示“关联会话”，入口禁用时也会显示；切换形态时保留焦点；页签列表可滚动但不显示滚动条。元数据加载后，它可打开可用且未归档的会话；会话或 Workspace 元数据尚未就绪、会话已归档、会话不在列表中或 Workspace 查询失败时，入口保持禁用并显示原因。点击时会重新检查可用性；浏览不会取消会话归档。仅此操作请求激活会话。删除位于详情顶部条带的“更多任务操作”菜单。 页头按标题的自然宽度排布；标题和“新建”无法并排容纳时，将按钮移到下一行。操作文案保持单行。 diff --git a/packages/client/ui-schedule/src/client/ScheduleEmptyIllustration.tsx b/packages/client/ui-schedule/src/client/ScheduleEmptyIllustration.tsx new file mode 100644 index 0000000000..e6cee2a839 --- /dev/null +++ b/packages/client/ui-schedule/src/client/ScheduleEmptyIllustration.tsx @@ -0,0 +1,37 @@ +/** Decorative document-and-clock artwork with theme-owned colors. */ + +interface ScheduleEmptyIllustrationProps { + readonly className?: string | undefined +} + +/** + * Render the empty-state illustration without adding an accessible announcement. + * @param props - optional CSS class applied to the root SVG. + * @returns the decorative inline SVG element. + */ +export function ScheduleEmptyIllustration({ className }: ScheduleEmptyIllustrationProps) { + return ( + + ) +} diff --git a/packages/client/ui-schedule/src/client/TaskManagerPage.module.css b/packages/client/ui-schedule/src/client/TaskManagerPage.module.css index ba3ee2a030..ebdb77b4b8 100644 --- a/packages/client/ui-schedule/src/client/TaskManagerPage.module.css +++ b/packages/client/ui-schedule/src/client/TaskManagerPage.module.css @@ -388,6 +388,14 @@ color: var(--dsw-alias-label-caption); } +.emptyIllustration { + flex: none; + width: 110px; + max-width: 100%; + height: auto; + margin-bottom: 12px; +} + /* The mock's `.empty h3 + .btn`: the empty state's action keeps the mock's 16px above. */ .emptyAction { diff --git a/packages/client/ui-schedule/src/client/TaskManagerPage.tsx b/packages/client/ui-schedule/src/client/TaskManagerPage.tsx index 8f09c96137..7d18063ae8 100644 --- a/packages/client/ui-schedule/src/client/TaskManagerPage.tsx +++ b/packages/client/ui-schedule/src/client/TaskManagerPage.tsx @@ -13,6 +13,7 @@ import { TaskDetail, useTaskDetail, type TaskDetailInjected } from './TaskDetail import { useRelativeClock } from './relative-clock.ts' import { formatScheduleFrequency, nextRunParts, taskName, zoneLabel } from './schedule-format.ts' import type { FrequencyZone } from './schedule-format.ts' +import { ScheduleEmptyIllustration } from './ScheduleEmptyIllustration.tsx' import css from './TaskManagerPage.module.css' /** Injected catalog and task actions for the management page. */ @@ -151,7 +152,7 @@ export function TaskManagerPage(props: TaskManagerPageProps) {
{selected === undefined && 0} onRetry={onRetry} t={t} />} {status === 'ready' && rows.length === 0 &&
- +

{t(emptyTitle)}

} +
+ {!configured && (status === 'unavailable' ?

{t(`${kind}Unavailable`)}

+ : status === 'ready' && !writable &&

{t(`${kind}ReadOnly`)}

)} + {error !== null &&

{error}

} +
+} + +function Applications({ title, apps, icons, onAction, label, disabled }: { + title: string + apps: readonly CuaApplicationGrant[] + icons: Record | null + disabled: boolean + label: string + onAction: PermissionProps['revoke'] +}) { + return
+

{title}

+
    {apps.map(app =>
  • +
    + {icons?.[app.executablePath] !== undefined ? + : } + {app.name} +
    + +
    • )}
+
+} + +function PermissionDescription({ application, permission, loading, t }: Pick & { + application: CuaSystemPermissionState['launchApplication'] + permission: 'accessibility' | 'screenRecording' + loading: boolean +}) { + if (!loading && application === undefined) return

{t(permission === 'accessibility' ? 'accessibilityUse' : 'screenRecordingUse')}

+ const [before, after] = t(permission === 'accessibility' ? 'accessibilityPermission' : 'screenRecordingPermission').split('{application}') + return

+ {before} + + {application === undefined ? : <> + {application.icon !== undefined && } + {application.name} + } + + {after} +

+} + +function PermissionAction({ permission, granted, loading, busy, request, openSettings, t }: Pick & { + permission: 'accessibility' | 'screenRecording' + granted: boolean | null + loading: boolean + busy: boolean +}) { + if (loading) return + if (granted === false) return + return +} + +/** @param props Host observation and explicit user actions. @returns Permission controls without automatic system prompts. */ +export function PermissionPanel(props: PermissionPanelProps) { + const { t, refresh } = props + const state = props.useCuaPermissions(value => value) + const observed = state.system.value + const apps = state.applications.value + const busy = state.system.busy + const appsBusy = state.applications.busy + const loading = observed === null && (busy || state.system.error === null) + const application = !busy && state.system.error !== null ? undefined : observed?.launchApplication + useEffect(() => { + refresh() + window.addEventListener('focus', refresh) + return () => { window.removeEventListener('focus', refresh) } + }, [refresh]) + return
+
+

{t('title')}

+
+ + + {!busy && state.system.error !== null &&
+ {t('queryFailed', { message: state.system.error })} + +
} + {(['accessibility', 'screenRecording'] as const).map(permission =>
+
+ {t(permission)} + +
+ +
)} +
+ {observed !== null && (!observed.accessibility || !observed.screenRecording) &&

+ {t(application === undefined ? 'permissionHintUnknown' : 'permissionHint')} +

} +
+
+

{t('applicationsTitle')}

+
+ {(['applications', 'icons'] as const).map(key => !state[key].busy && state[key].error !== null &&
+ {t(key === 'applications' ? 'applicationsFailed' : 'iconsFailed', { message: state[key].error })} + +
)} + {apps === null ?
: <> + {apps.allowedApps.length === 0 && apps.blockedApps.length === 0 &&

{t('noApps')}

} + {apps.allowedApps.length > 0 && } + {apps.blockedApps.length > 0 && } + } +
+
+
+} + +/** + * @param props Permission observation and activation navigation callbacks. + * @returns Setup guidance when system permissions need attention. + */ +export function PermissionSetup({ t, refresh, useCuaPermissions, onDismiss, onOpenDetails }: PermissionSetupProps) { + const state = useCuaPermissions(value => value.system) + useEffect(() => { refresh() }, [refresh]) + const ready = state.value !== null && state.error === null + && state.value.accessibility && state.value.screenRecording + useEffect(() => { if (ready && !state.busy) onDismiss() }, [ready, state.busy, onDismiss]) + return + + + }> +

{t(state.error === null ? 'setupBody' : 'setupQueryFailed')}

+ +} + +/** @param props Shared operation outcome, independent of page lifetime. @returns The existing global toast presentation. */ +export function PermissionToast(props: PermissionProps) { + const toast = props.useCuaPermissions(value => value.toast) + if (!toast) return null + return +} diff --git a/packages/experimental/cua/src/client/controller.ts b/packages/experimental/cua/src/client/controller.ts new file mode 100644 index 0000000000..0ecdbdf4d0 --- /dev/null +++ b/packages/experimental/cua/src/client/controller.ts @@ -0,0 +1,121 @@ +/** One query projection for the permission page and activation guidance. */ +import type { Context } from '@deepseek-ai/cordis' +import type {} from '@deepseek-ai/dsh-api-remotes/client' +import type {} from '@deepseek-ai/dsh-cua/remote' +import type { RemoteResult } from '@deepseek-ai/dsh-typert-protocol' +import { createSnapshotStore, type SnapshotStore } from '@deepseek-ai/dsh-client-store' +import type { CuaApplicationId, CuaApplicationState, CuaSystemPermission, CuaSystemPermissionState } from '../access-types.ts' + +interface PermissionData { + runtime: { persistent: boolean; writable: boolean } + system: CuaSystemPermissionState + applications: CuaApplicationState + icons: Record +} +/** Query failures retain the last observation without presenting it as current. */ +interface Query { + value: T | null + busy: boolean + error: string | null +} +/** Independently settled observations; the overall busy flag follows the latest refresh. */ +export type PermissionView = { [K in keyof PermissionData]: Query } & { + busy: boolean + toast: { id: number; kind: 'requested' | 'opened' | 'revoked' | 'retryAllowed' } | { id: number; kind: 'failed'; message: string } | null +} +/** Slot actions and the renderer-bound permission projection. */ +export interface PermissionFace { + hooks: { cuaPermissions: SnapshotStore } + refresh: () => void + request: (permission: CuaSystemPermission) => void + openSettings: (permission: CuaSystemPermission) => void + revoke: (id: CuaApplicationId) => void + retryApproval: (id: CuaApplicationId) => void + dismissToast: () => void +} + +/** Serializes user actions and drops superseded refresh results or results arriving after disposal. */ +export class PermissionController { + /** Last observed permissions and the lifetime of the current refresh/action. */ + readonly state = createSnapshotStore({ + runtime: { value: null, busy: false, error: null }, + system: { value: null, busy: false, error: null }, + applications: { value: null, busy: false, error: null }, + icons: { value: null, busy: false, error: null }, + busy: false, toast: null, + }) + private tail: Promise = Promise.resolve() + private reads: Promise = Promise.resolve() + private closed = false + private sequence = 0 + /** @param remote Generated CUA Remote namespace; failed results are handled with query and action failures. */ + constructor(private readonly remote: Context['remote']['cuaAccess']) {} + /** + * Bind the permission projection to its user actions. + * @returns Reactive state and controls shared by the page, prompt, and toast host. + */ + face(): PermissionFace { + return { + hooks: { cuaPermissions: this.state }, + refresh: () => { this.enqueue() }, + request: (permission) => { this.enqueue(() => this.remote.requestPermission(permission), 'requested') }, + openSettings: (permission) => { this.enqueue(() => this.remote.openSettings(permission), 'opened') }, + revoke: (id) => { this.enqueue(() => this.remote.revoke(id), 'revoked') }, + retryApproval: (id) => { this.enqueue(() => this.remote.retryApproval(id), 'retryAllowed') }, + dismissToast: () => { if (!this.closed) this.state.update((state) => { state.toast = null }) }, + } + } + /** Prevent publication after UI withdrawal and join the accepted Remote work. */ + async dispose(): Promise { this.closed = true; await this.tail; await this.reads } + private enqueue(action?: () => Promise>, kind?: 'requested' | 'opened' | 'revoked' | 'retryAllowed'): void { + if (this.closed) return + const current = () => !this.closed + const sequence = ++this.sequence + this.state.update((state) => { + state.busy = true + for (const key of ['runtime', 'system', 'applications', 'icons'] as const) state[key].busy = true + }) + this.tail = this.tail.then(async () => { + if (!current()) return + try { + if (action) { + const result = await action() + if (!result.ok) throw result.error + } + if (!current()) return + if (kind) this.state.update((state) => { state.toast = { id: sequence, kind } }) + } catch (error) { + if (!current()) return + const message = error instanceof Error ? error.message : String(error) + this.state.update((state) => { state.toast = { id: sequence, kind: 'failed', message } }) + } + const latest = () => current() && this.sequence === sequence + const refresh = Promise.all([ + this.read('runtime', this.remote.ptcConfiguration, latest), + this.read('system', this.remote.systemStatus, latest), + this.read('applications', this.remote.applications, latest), + this.read('icons', this.remote.applicationIcons, latest), + ]).then(() => { + if (latest()) this.state.update((state) => { state.busy = false }) + }) + this.reads = Promise.all([this.reads, refresh]).then(() => {}) + }) + } + + private async read( + key: K, read: () => Promise>, current: () => boolean, + ): Promise { + if (!current()) return + try { + const result = await read() + if (!current()) return + if (!result.ok) throw result.error + this.state.update((state) => { state[key].value = result.value; state[key].error = null; state[key].busy = false }) + } catch (error) { + if (!current()) return + this.state.update((state) => { + state[key].error = error instanceof Error ? error.message : String(error); state[key].busy = false + }) + } + } +} diff --git a/packages/experimental/cua/src/client/index.ts b/packages/experimental/cua/src/client/index.ts new file mode 100644 index 0000000000..a4b192ce58 --- /dev/null +++ b/packages/experimental/cua/src/client/index.ts @@ -0,0 +1,15 @@ +/** Bind the source-owned permission UI to its generated Remote contribution. */ +import type { Context } from '@deepseek-ai/cordis' +import cuaRemote from '@deepseek-ai/dsh-cua/remote' +import { mountCuaPermissions } from './mount.ts' + +export { inject } from './mount.ts' + +/** + * Activate the optional Computer Use permission controls. + * @param ctx Browser plugin context. + * @returns Cleanup of the UI and Remote contribution. + */ +export async function apply(ctx: Context): Promise<() => Promise> { + return mountCuaPermissions(ctx, cuaRemote) +} diff --git a/packages/experimental/cua/src/client/locales.ts b/packages/experimental/cua/src/client/locales.ts new file mode 100644 index 0000000000..0702e20597 --- /dev/null +++ b/packages/experimental/cua/src/client/locales.ts @@ -0,0 +1,76 @@ +/** Product copy for native permission onboarding and application revocation. */ +export const NS = 'cuaPermissions' +/** English permission-page dictionary. */ +export const en = { + title: 'Authorize Computer Use', + applicationsTitle: 'Manage applications Computer Use can access', + setupTitle: 'Set up Computer Use before use', + setupBody: 'Computer Use is enabled. Before using it, switch to PTC mode and grant Accessibility and Screen Recording permissions. Open the plugin details to complete setup.', + setupQueryFailed: 'Computer Use is enabled, but system permissions could not be checked. Open the plugin details to see the error and retry.', + defaultTitle: 'Default mode for new sessions', + defaultBody: 'Computer Use is available only in PTC mode.', + defaultConfirm: 'Set to PTC', defaultSet: 'PTC is the default', + defaultUnavailable: 'Could not read the default mode. Check it in Settings.', + defaultReadOnly: 'Default-mode settings are read-only.', + defaultNotSaved: 'The default mode was not saved. Try again.', + defaultFailed: 'Could not change the default mode: {message}', + persistentTitle: 'PTC execution mode', + persistentBody: 'Keep variables and application objects across tool calls. Switching restarts the PTC runtime.', + persistentConfirm: 'Use persistent mode', persistentSet: 'Persistent mode enabled', + persistentUnavailable: 'Could not read the PTC execution mode. Check it in Settings.', + persistentReadOnly: 'PTC execution settings are read-only.', + persistentNotSaved: 'The PTC execution mode was not saved. Try again.', + persistentFailed: 'Could not change the PTC execution mode: {message}', + openSetup: 'Open settings', later: 'Later', close: 'Close', + accessibility: 'Accessibility', screenRecording: 'Screen Recording', + accessibilityUse: 'Read interface controls, click, and type.', screenRecordingUse: 'Capture the selected application window.', + accessibilityPermission: 'In System Settings, allow {application} to read interface controls, click, and type.', + screenRecordingPermission: 'In System Settings, allow {application} to capture application windows.', + permissionHint: 'Enable the application in System Settings. Follow the macOS prompt if it names another application.', + permissionHintUnknown: 'Enable these permissions for the application named in the macOS prompt.', + granted: 'Allowed', unknown: 'Unknown', request: 'Request permission', settings: 'Open System Settings', retry: 'Retry', + allowedApps: 'Always-allowed applications', + blockedApps: 'Applications not allowed', retryApproval: 'Allow asking again', + retryAllowed: 'The next application selection will ask again; access has not been granted', + noApps: 'No always-allowed or blocked applications.', + revoke: 'Revoke', revoked: 'Application access revoked', requested: 'Review the macOS prompt, or open System Settings if no prompt appears', opened: 'System Settings opened', + applicationsFailed: 'Could not load application access: {message}', iconsFailed: 'Could not load application icons: {message}', + failed: 'Could not update permissions: {message}', queryFailed: 'Could not check permissions: {message}', +} +/** Keys shared by all supported locale dictionaries. */ +export type CuaPermissionLocaleKey = keyof typeof en +/** Chinese permission-page dictionary. */ +export const zh: Record = { + title: '给 Computer Use 授权', applicationsTitle: '管理 Computer Use 可访问的应用', setupTitle: '使用 Computer Use 前需要设置', + setupBody: 'Computer Use 插件已开启。使用前请切换到 PTC 模式，并授予辅助功能和屏幕录制权限。请前往插件详情页完成设置。', + setupQueryFailed: 'Computer Use 插件已开启，但暂时无法检测系统权限。请前往插件详情页查看原因并重试。', + defaultTitle: '新会话默认模式', + defaultBody: 'Computer Use 仅在 PTC 模式中可用。', + defaultConfirm: '设为 PTC', defaultSet: '已设为 PTC', + defaultUnavailable: '无法读取默认模式，请在设置中检查', + defaultReadOnly: '默认模式设置为只读', + defaultNotSaved: '默认模式未保存，请重试', + defaultFailed: '无法切换默认模式：{message}', + persistentTitle: 'PTC 执行方式', + persistentBody: '跨工具调用保留变量和应用对象。切换会重启 PTC 执行器。', + persistentConfirm: '设为持久化', persistentSet: '已设为持久化', + persistentUnavailable: '无法读取 PTC 执行方式，请在设置中检查', + persistentReadOnly: 'PTC 执行方式设置为只读', + persistentNotSaved: 'PTC 执行方式未保存，请重试', + persistentFailed: '无法切换 PTC 执行方式：{message}', + openSetup: '前往设置', later: '稍后', close: '关闭', + accessibility: '辅助功能', screenRecording: '屏幕录制', + accessibilityUse: '读取界面控件、点击和输入文字', screenRecordingUse: '截取所选应用的窗口', + accessibilityPermission: '在系统设置中允许 {application} 读取界面、点击和输入。', + screenRecordingPermission: '在系统设置中允许 {application} 截取应用窗口。', + permissionHint: '在系统设置中开启对应应用；若 macOS 提示其他应用，以系统提示为准', + permissionHintUnknown: '在系统设置中，为 macOS 提示的应用开启对应权限', + granted: '已允许', unknown: '未确认', request: '申请权限', settings: '打开系统设置', retry: '重试', + allowedApps: '始终允许的应用', + blockedApps: '不允许访问的应用', retryApproval: '允许再次询问', + retryAllowed: '下次选择此应用时将重新询问，当前尚未授权', + noApps: '暂无始终允许或不允许访问的应用。', + revoke: '撤销', revoked: '已撤销应用访问权限', requested: '请查看 macOS 提示；若未出现，请打开系统设置', opened: '已打开系统设置', + applicationsFailed: '无法读取应用授权：{message}', iconsFailed: '无法读取应用图标：{message}', + failed: '无法更新权限：{message}', queryFailed: '无法检测权限：{message}', +} diff --git a/packages/experimental/cua/src/client/mount.ts b/packages/experimental/cua/src/client/mount.ts new file mode 100644 index 0000000000..bc43e9bd8b --- /dev/null +++ b/packages/experimental/cua/src/client/mount.ts @@ -0,0 +1,69 @@ +/** Optional Computer Use controls mounted through existing plugin-page and overlay slots. */ +import type { Context } from '@deepseek-ai/cordis' +import type {} from '@deepseek-ai/dsh-cua/remote' +import type { TypertRemoteContribution } from '@deepseek-ai/dsh-typert-protocol' +import type {} from '@deepseek-ai/dsh-api-remotes/client' +import type {} from '@deepseek-ai/dsh-client-locale/client' +import type {} from '@deepseek-ai/dsh-client-ui-renderer/client' +import type {} from '@deepseek-ai/dsh-client-ui-layout/client' +import type {} from '@deepseek-ai/dsh-client-ui-plugin-manager/client' +import type {} from '@deepseek-ai/dsh-client-ui-settings/client' +import { PermissionController } from './controller.ts' +import { PermissionPanel, PermissionSetup, PermissionToast } from './PermissionPanel.tsx' +import { en, NS, zh, type CuaPermissionLocaleKey } from './locales.ts' + +declare module '@deepseek-ai/dsh-client-ui-slots' { + interface LocaleNamespaceMap { + /** Native permission controls and application consent. */ + cuaPermissions: CuaPermissionLocaleKey + } +} + +/** Existing browser registries needed by the optional permission UI. */ +export const inject = ['remote', 'slots', 'locale', 'configForms'] + +/** + * Mount the optional permission Remote and its browser controls. + * @param ctx Browser plugin context. + * @param contribution Generated permission Remote definitions. + * @returns Cleanup for the optional Remote and UI registrations. + */ +export async function mountCuaPermissions(ctx: Context, contribution: TypertRemoteContribution): Promise<() => Promise> { + const transport = ctx.plugin(async (owner) => { await owner.remote.$mount(contribution) }) + const ui = ctx.inject(['remote.cuaAccess', 'slots', 'locale'], (ctx) => { + ctx.effect(() => ctx.locale.register(NS, { en, zh })) + const controller = new PermissionController(ctx.remote.cuaAccess) + const face = controller.face() + const preset = ctx.configForms.get<{ default: string; selectedDefault?: string }>('agent-preset-registry') + const settings = { + ...face, + hooks: { ...face.hooks, cuaDefaultPreset: preset }, + setPtcDefault: () => preset.set('selectedDefault', 'ptc'), + setPersistentPtc: async () => { + const result = await ctx.remote.cuaAccess.enablePersistentPtc(); if (!result.ok) throw result.error + face.refresh() + return true + }, + } + ctx.effect(() => () => controller.dispose()) + ctx.on('connection/reset', face.refresh) + ctx.effect(() => ctx.remote.$on('settings/document-updated', (namespace) => { + if (namespace === controller.state.getSnapshot().applications.value?.settingsNamespace) face.refresh() + })) + ctx.slots.inject('plugins.bundle.config', () => ctx.slots.register({ + name: 'plugins.bundle.config', key: '@deepseek-ai/dsh-cua-bundle', locale: NS, inject: () => settings, + }, PermissionPanel)) + ctx.slots.inject('plugins.bundle.activation', () => ctx.slots.register({ + name: 'plugins.bundle.activation', key: '@deepseek-ai/dsh-cua-bundle', locale: NS, inject: () => face, + }, PermissionSetup)) + ctx.slots.inject('shell.overlay', () => ctx.slots.register({ + name: 'shell.overlay', id: 'cua-permission-outcome', locale: NS, inject: () => face, + }, PermissionToast)) + }) + const dispose = ctx.effect(function* () { + yield transport.dispose + yield ui.dispose + }, 'cua-permissions.mount') + try { await Promise.all([transport, ui]) } catch (error) { await dispose(); throw error } + return dispose +} diff --git a/packages/experimental/cua/src/css-modules.d.ts b/packages/experimental/cua/src/css-modules.d.ts new file mode 100644 index 0000000000..8811db1264 --- /dev/null +++ b/packages/experimental/cua/src/css-modules.d.ts @@ -0,0 +1,4 @@ +declare module '*.module.css' { + const classes: Record + export default classes +} diff --git a/packages/experimental/cua/src/index.ts b/packages/experimental/cua/src/index.ts index edf3121a0a..1cec00f09a 100644 --- a/packages/experimental/cua/src/index.ts +++ b/packages/experimental/cua/src/index.ts @@ -13,22 +13,27 @@ import type { ContentBlock } from '@deepseek-ai/dsh-llm' import type { JsonValue } from '@deepseek-ai/dsh-util-values' import { z } from 'zod' import { NativeClient, NativeError, type NativeClientConfig, type NativeChannelFactory } from './native-client.ts' -import { ApplicationAuthorizationResult, ApplicationAccess } from './application-access.ts' -import type { CuaApplicationGrant, CuaApplicationId } from './application-types.ts' +import { ApplicationAuthorizationResult, installCuaAccess } from './access.ts' +import type { CuaApplicationGrant, CuaApplicationId } from './access-types.ts' import type { CuaAppHandle, CuaOperation } from './types.ts' import { AppResult, NativeScreenshotResult, Request, ScreenshotResult, StateResult, WindowsResult } from './protocol.ts' import { CuaCoordinates } from './coordinates.ts' -import type {} from '@deepseek-ai/dsh-system-prompt' -import type { ToolExecution } from '@deepseek-ai/dsh-tools' +import { createScope, type Scope } from '@deepseek-ai/dsh-scope' +import type {} from '@deepseek-ai/dsh-agent-preset-registry' +import type { PromptSection } from '@deepseek-ai/dsh-system-prompt' +import type { ToolDefinition, ToolExecution } from '@deepseek-ai/dsh-tools' import type {} from '@deepseek-ai/dsh-subprocess' import type {} from '@deepseek-ai/dsh-computer-use' import type {} from '@deepseek-ai/dsh-ptc-runtime' import type {} from '@deepseek-ai/dsh-settings' import type {} from '@deepseek-ai/cordis-plugin-loader' +export { CuaAccess } from './access.ts' /** Native executable and per-operation resource bounds. */ export interface CuaConfig { + /** Register tools for Agents using this preset; defaults to ptc. False enables global registration. */ + agentPreset: string | false /** Signed applications explicitly approved for future tasks in this profile. */ allowedApps: Volatile & { /** Unbranded digest at the profile configuration boundary; admitted by the Config schema. */ @@ -74,6 +79,7 @@ export interface CuaConfig { /** Validated host configuration; no driver is downloaded at runtime. */ export const CuaConfig = Schema.object({ + agentPreset: Schema.union([Schema.string().min(1), Schema.const(false)]).default('ptc'), allowedApps: Schema.array(Schema.object({ id: Schema.string().pattern(/^[a-f0-9]{64}$/).required(), name: Schema.string().required(), executablePath: Schema.string().min(1).required(), @@ -161,7 +167,9 @@ export function installCua(ctx: Context, config: CuaConfig, backend: CuaBackend) } const entryId = ctx.fiber.entry?.options.id ctx.inject(['settings'], scope => scope.effect(() => scope.settings.configure({ auto: false }, ctx.fiber))) - const access = new ApplicationAccess(ctx, { + const access = installCuaAccess(ctx, { + settingsNamespace: entryId ?? null, + native, createChannel: backend.createChannel, grants: () => config.allowedApps.get().map(grant => ({ ...grant, id: grant.id as CuaApplicationId })), get save() { const settings = ctx.get('settings') @@ -280,7 +288,7 @@ export function installCua(ctx: Context, config: CuaConfig, backend: CuaBackend) if (method === 'listWindows') execution.coordinates.observe(parsedParams.appHandle as CuaAppHandle, WindowsResult.parse(result)) return result } - yield ctx.tools.register({ + const tool: ToolDefinition = { name: 'cua', description: backend.description, parameters: { ...z.record(z.string(), z.json()).parse(z.toJSONSchema(Request)), type: 'object' }, output: { schema: {}, render: renderResult }, @@ -323,14 +331,50 @@ export function installCua(ctx: Context, config: CuaConfig, backend: CuaBackend) pending.delete(operation) } }, - }) - yield ctx.systemPrompt.section({ + } + const prompt: PromptSection = { name: 'computer-use:dsh-cua', order: ctx.systemPrompt.getSectionOrder('TOOL_COMPUTER_USE'), text: () => ctx.get('ptcRuntime')?.programFormat !== 'repl' ? guidance : `${guidance} Persistent PTC initialization: var { connectCua } = await import(${JSON.stringify(new URL('../lib/bridge.js', import.meta.url).href)}); var cua = connectCua(() => tools.cua, text => console.log(text));`, - }) + } + const register = (scope: Context): void => { + scope.tools.register(tool) + scope.systemPrompt.section(prompt) + } + if (config.agentPreset === false) register(ctx) + else { + const consumer = ctx.inject(['agentPresets'], async (presetCtx) => { + const scopes = new Map() + const sync = async (agent: Agent): Promise => { + const previous = scopes.get(agent) + if (presetCtx.agentPresets.composedPreset(agent.ctx) !== config.agentPreset) { + scopes.delete(agent) + await previous?.dispose() + return + } + if (previous !== undefined) return + const scope = createScope(agent.ctx, agent) + scopes.set(agent, scope) + scope.ctx.effect(() => () => { + if (scopes.get(agent) === scope) scopes.delete(agent) + }, 'cua.agent-tools') + register(scope.ctx) + } + presetCtx.effect(() => async () => { await Promise.all([...scopes.values()].map(scope => scope.dispose())) }, 'cua.agent-tools') + presetCtx.on('agent/created', async ({ agent }) => { await sync(agent) }) + presetCtx.on('agent-preset/selected', (id) => { + const agent = presetCtx.agents.get(id) + if (agent !== undefined) void sync(agent).catch((error: unknown) => { presetCtx.logger.warn(`CUA tool scope update failed: ${String(error)}`) }) + }) + await Promise.all(presetCtx.agents.list().map(sync)) + }) + yield () => consumer.dispose() + } }, 'dsh-cua.runtime') } + +/** Host marker for the optional Client contribution; platform providers install native execution separately. */ +export function apply(): void {} diff --git a/packages/experimental/cua/src/protocol.ts b/packages/experimental/cua/src/protocol.ts index 0af1ee0fda..3e1d9f6450 100644 --- a/packages/experimental/cua/src/protocol.ts +++ b/packages/experimental/cua/src/protocol.ts @@ -81,8 +81,13 @@ export const NativeResourceParameters = { }).strict(), findApps: z.object({ maxAppMatches: z.number().int().min(1).max(1000) }).strict(), } -/** Executor-only continuation after application consent; absent from the model Request union. */ -export const NativeOnlyOperations = [operation('launchApp', app)] as const +/** Host-only launch continuation and user-facing system permission controls; absent from model arguments. */ +export const NativeOnlyOperations = [ + operation('launchApp', app), operation('permissionStatus', {}), + operation('applicationIcons', { paths: z.array(z.string().min(1)) }), + operation('requestPermission', { permission: z.enum(['accessibility', 'screenRecording']) }), + operation('openPermissionSettings', { permission: z.enum(['accessibility', 'screenRecording']) }), +] as const /** Process identity returned only after a successful app selection. */ export const AppResult = z.object({ @@ -127,3 +132,6 @@ export const ActionResult = z.object({ dispatched: z.literal(true), verified: z. export const HoverResult = ActionResult.extend({ hoverRoute: z.enum(['accessibility', 'system-pointer']), systemPointerMoved: z.boolean() }) /** Background restoration dispatch and sampled foreground changes; null means the observation was unavailable. */ export const ShowResult = ActionResult.extend({ windowId: id.nullable(), foregroundChanged: z.boolean().nullable() }) + +/** Bounded native PNG thumbnail for human presentation, excluded from model results and saved grants. */ +export const ApplicationIcon = z.string().max(65536).regex(/^data:image\/png;base64,[A-Za-z0-9+/]+={0,2}$/) diff --git a/packages/experimental/cua/src/remote-types.ts b/packages/experimental/cua/src/remote-types.ts new file mode 100644 index 0000000000..6c16dde03d --- /dev/null +++ b/packages/experimental/cua/src/remote-types.ts @@ -0,0 +1,3 @@ +/** Named Remote boundary types required by Typert's generated Client declarations. */ +export type { CuaApplicationId } from './application-types.ts' +export type { CuaApplicationState, CuaSystemPermission, CuaSystemPermissionState } from './access-types.ts' diff --git a/packages/experimental/cua/tests/access.spec.ts b/packages/experimental/cua/tests/access.spec.ts new file mode 100644 index 0000000000..8cffbcc4e9 --- /dev/null +++ b/packages/experimental/cua/tests/access.spec.ts @@ -0,0 +1,419 @@ +/** Application decisions are checked at dispatch and revocation cancels owned work. */ +import { setImmediate } from 'node:timers/promises' +import { Context } from '@deepseek-ai/cordis' +import { mountAgentLoopTestDependencies, mountAgentLoopTestHarness } from '@deepseek-ai/dsh-agent-loop-testkit' +import { SessionId } from '@deepseek-ai/dsh-session' +import UserQuestions, { type AskUserQuestionAnswer } from '@deepseek-ai/dsh-user-questions' +import { expect, it, onTestFinished, vi } from 'vitest' +import { ApplicationAuthorizationResult, installCuaAccess } from '../src/access.ts' +import type { CuaApplicationGrant } from '../src/access-types.ts' +import type { CuaAppHandle } from '../src/types.ts' +import { NativeClient } from '../src/native-client.ts' + +const native = vi.hoisted(() => ({ instances: [] as Array<{ + disposed: boolean + request: ReturnType + dispose: ReturnType Promise>> +}> })) + +vi.mock('../src/native-client.ts', async original => ({ + ...await original(), + NativeClient: class { + disposed = false + request = vi.fn(async (method: string) => method === 'openPermissionSettings' ? { opened: true } + : method === 'applicationIcons' ? {} : { accessibility: false, screenRecording: false, helper: { pid: 9 }, parent: { pid: 8 } }) + invalidate() { this.disposed = true } + dispose = vi.fn(async () => { this.disposed = true }) + constructor() { native.instances.push(this) } + }, +})) + +const unusedConnector = () => { throw new Error('Mocked native client does not connect') } + +const nativeConfig = { helperPath: '/fixture', cwd: '/', timeoutMs: 1000, connectTimeoutMs: 5000, screenshotTimeoutMs: 5000, axTimeoutMs: 100, + maxRequestBytes: 4096, maxResponseBytes: 4096, graceMs: 100, textEventIntervalMs: 1, maxClickHoldMs: 5000 } +const identity = ApplicationAuthorizationResult.parse({ id: 'a'.repeat(64), name: 'Test application', executablePath: '/Applications/Fixture.app/Contents/MacOS/Fixture', persistable: true }) +const handle = 'fixture-app' as CuaAppHandle +const signal = new AbortController().signal + +async function setup(choice = 'Allow for this session', options: { writable?: boolean; questions?: boolean; initial?: readonly CuaApplicationGrant[] } = {}) { + native.instances.length = 0 + const ctx = new Context() + onTestFinished(() => ctx.fiber.dispose()) + await mountAgentLoopTestDependencies(ctx) + await mountAgentLoopTestHarness(ctx) + if (options.questions !== false) await ctx.plugin(UserQuestions) + const owner = await ctx.agents.create({ sessionId: SessionId('cua-access-test') }) + const answer = vi.fn<() => Promise>(async () => ({ answers: [{ id: 'cua-app-access', selected: [choice] }] })) + ctx.on('user-questions/request', answer) + let grants: readonly CuaApplicationGrant[] = options.initial ?? [] + const save = vi.fn(async (next: readonly CuaApplicationGrant[]) => { grants = next }) + let writable = options.writable !== false + const access = installCuaAccess(ctx, { native: nativeConfig, createChannel: unusedConnector, settingsNamespace: 'cua', grants: () => grants, get save() { return writable ? save : undefined } }) + const client = new NativeClient(ctx, nativeConfig, unusedConnector) + return { ctx, owner, agent: owner.agent, answer, save, access, client, grants: () => grants, + setWritable(value: boolean) { writable = value }, + } +} + +it('requires another user decision after the permission page permits asking again', async () => { + const f = await setup('Deny') + await expect(f.access.select(f.agent, f.client, handle, identity, signal)).rejects.toThrow('APP_ACCESS_DENIED') + expect(f.ctx.cuaAccess.applications().blockedApps).toHaveLength(1) + await f.ctx.cuaAccess.retryApproval(identity.id) + expect(f.ctx.cuaAccess.applications().blockedApps).toEqual([]) + await expect(f.access.run(f.client, handle, signal, async () => null)).rejects.toThrow('APP_ACCESS_REQUIRED') + f.answer.mockResolvedValueOnce({ answers: [{ id: 'cua-app-access', selected: ['Allow for this session'] }] }) + const retryClient = new NativeClient(f.ctx, nativeConfig, unusedConnector) + await f.access.select(f.agent, retryClient, handle, identity, signal) + expect(f.answer).toHaveBeenCalledTimes(2) + await f.ctx.cuaAccess.revoke(identity.id) + await f.ctx.cuaAccess.retryApproval(identity.id) + await expect(f.access.run(f.client, handle, signal, async () => null)).rejects.toThrow('APP_ACCESS_REQUIRED') +}) + +it('persists a signed identity and revokes it before further dispatch', async () => { + const f = await setup('Always allow') + await f.access.select(f.agent, f.client, handle, identity, signal) + expect(f.grants()).toEqual([{ id: identity.id, name: identity.name, executablePath: identity.executablePath }]) + await f.ctx.cuaAccess.revoke(identity.id) + expect(f.grants()).toEqual([]) + await expect(f.access.run(f.client, handle, signal, async () => null)).rejects.toThrow('APP_ACCESS_REVOKED') + await expect(f.access.select(f.agent, f.client, handle, identity, signal)).rejects.toThrow('APP_ACCESS_DENIED') +}) + +it('joins an in-flight native operation when application access is revoked', async () => { + const f = await setup() + await f.access.select(f.agent, f.client, handle, identity, signal) + const entered = Promise.withResolvers() + const pending = f.access.run(f.client, handle, signal, async (operationSignal) => { + const done = Promise.withResolvers() + operationSignal.addEventListener('abort', () => { done.reject(new Error('native cancelled')) }, { once: true }) + entered.resolve(undefined) + return done.promise + }) + const rejected = expect(pending).rejects.toThrow('APP_ACCESS_DENIED') + await entered.promise + await f.ctx.cuaAccess.revoke(identity.id) + await rejected + await expect(f.access.run(f.client, handle, signal, async () => null)).rejects.toThrow('APP_ACCESS_REVOKED') +}) + +it.each(['revoke', 'dispose'] as const)('waits for native cancellation to finish during %s', async (action) => { + const f = await setup() + await f.access.select(f.agent, f.client, handle, identity, signal) + const entered = Promise.withResolvers(), cancelled = Promise.withResolvers() + const release = Promise.withResolvers() + onTestFinished(() => { release.resolve(undefined) }) + const pending = f.access.run(f.client, handle, signal, async (operationSignal) => { + operationSignal.addEventListener('abort', () => { cancelled.resolve(undefined) }, { once: true }) + entered.resolve(operationSignal) + await release.promise + }) + const rejected = expect(pending).rejects.toThrow() + const operationSignal = await entered.promise + let finished = false + const closing = (action === 'revoke' ? f.ctx.cuaAccess.revoke(identity.id) : f.access.dispose()) + .then(() => { finished = true }) + try { + await cancelled.promise + await setImmediate() + expect(operationSignal.aborted).toBe(true) + expect(finished).toBe(false) + release.resolve(undefined) + await closing + await rejected + expect(finished).toBe(true) + } finally { + release.resolve(undefined) + await Promise.all([closing, rejected]) + } +}) + +it('fails closed without a human answerer and never offers persistence without its owner', async () => { + const f = await setup('Always allow', { writable: false, questions: false }) + await expect(f.access.select(f.agent, f.client, handle, identity, signal)).rejects.toThrow('APP_APPROVAL_UNAVAILABLE') + await f.ctx.plugin(UserQuestions) + await expect(f.access.select(f.agent, f.client, handle, identity, signal)).rejects.toThrow('APP_ACCESS_DENIED') + expect(f.save).not.toHaveBeenCalled() + expect(f.ctx.cuaAccess.applications().persistentAvailable).toBe(false) +}) + +it('projects task decisions, observes live config removal, and forgets a disposed task', async () => { + const f = await setup('Always allow') + await f.access.select(f.agent, f.client, handle, identity, signal) + expect(f.ctx.cuaAccess.applications().allowedApps).toHaveLength(1) + await f.save([]) + f.ctx.emit('loader/volatile-update', [['allowedApps']]) + await expect(f.access.run(f.client, handle, signal, async () => null)).rejects.toThrow('APP_ACCESS_REVOKED') + await f.owner.dispose() + f.access.forget(f.agent) +}) + +it('refuses to promise revocation of a saved decision without persistence', async () => { + const { persistable: _persistable, ...grant } = identity + const f = await setup('Deny', { initial: [grant], writable: false }) + await expect(f.ctx.cuaAccess.revoke(identity.id)).rejects.toThrow('cannot be saved') + expect(f.grants()).toEqual([grant]) +}) + +it('checks native permissions without converting an OS request into a grant', async () => { + const f = await setup() + await f.access.select(f.agent, f.client, handle, identity, signal) + const before = f.ctx.cuaAccess.applications() + expect(before).toEqual({ settingsNamespace: 'cua', allowedApps: [], blockedApps: [], persistentAvailable: true }) + expect(await f.ctx.cuaAccess.systemStatus()).toMatchObject({ accessibility: false, screenRecording: false }) + expect(await f.ctx.cuaAccess.requestPermission('accessibility')).toMatchObject({ accessibility: false }) + await f.ctx.cuaAccess.openSettings('screenRecording') + const transport = native.instances[1]! + expect(transport.request).toHaveBeenCalledWith('openPermissionSettings', { permission: 'screenRecording' }, expect.any(AbortSignal)) + transport.request.mockRejectedValueOnce(new Error('transport ended')) + await expect(f.ctx.cuaAccess.systemStatus()).rejects.toThrow('transport ended') + transport.disposed = true + expect((await f.ctx.cuaAccess.systemStatus()).screenRecording).toBe(false) + expect(transport.dispose).toHaveBeenCalledOnce() + await f.access.dispose() + await expect(f.ctx.cuaAccess.systemStatus()).rejects.toThrow() + await expect(f.ctx.cuaAccess.revoke(identity.id)).rejects.toThrow() +}) + +it('does not open a replacement control connection after provider disposal begins', async () => { + const f = await setup() + await f.ctx.cuaAccess.systemStatus() + const control = native.instances.at(-1)! + control.disposed = true + const released = Promise.withResolvers(), entered = Promise.withResolvers() + onTestFinished(() => { released.resolve(undefined) }) + control.dispose.mockImplementationOnce(async () => { entered.resolve(undefined); await released.promise }) + const pending = f.ctx.cuaAccess.systemStatus() + const rejected = expect(pending).rejects.toThrow() + await entered.promise + const count = native.instances.length + const closing = f.access.dispose() + released.resolve(undefined) + await Promise.all([closing, rejected]) + expect(native.instances).toHaveLength(count) +}) + + +it('isolates other tasks and applications while revoking and forgetting one decision', async () => { + const f = await setup() + const owner = await f.ctx.agents.create({ sessionId: SessionId('another-cua-task') }) + const otherClient = new NativeClient(f.ctx, nativeConfig, unusedConnector) + const other = { ...identity, id: 'b'.repeat(64) as typeof identity.id } + const otherHandle = 'other-app' as CuaAppHandle + await f.access.select(f.agent, f.client, handle, identity, signal) + await f.access.select(f.agent, f.client, otherHandle, other, signal) + await f.access.select(owner.agent, otherClient, otherHandle, other, signal) + f.ctx.emit('loader/volatile-update', [['allowedApps']]) + await f.ctx.cuaAccess.retryApproval(identity.id) + const started: PromiseWithResolvers = Promise.withResolvers() + const release: PromiseWithResolvers = Promise.withResolvers() + onTestFinished(() => { release.resolve() }) + const continuing = f.access.run(otherClient, otherHandle, signal, async (operationSignal) => { + started.resolve() + await release.promise + return !operationSignal.aborted + }) + await started.promise + await f.ctx.cuaAccess.revoke(identity.id) + release.resolve() + expect(await continuing).toBe(true) + f.access.forget(f.agent) + await expect(f.access.run(otherClient, otherHandle, signal, async () => 'still allowed')).resolves.toBe('still allowed') + await owner.dispose() +}) + +it('removes a grant whose save is already pending when another task revokes the application', async () => { + const f = await setup() + await f.access.select(f.agent, f.client, handle, identity, signal) + const owner = await f.ctx.agents.create({ sessionId: SessionId('cua-concurrent-save') }) + const client = new NativeClient(f.ctx, nativeConfig, unusedConnector) + const entered: PromiseWithResolvers = Promise.withResolvers() + const release: PromiseWithResolvers = Promise.withResolvers() + onTestFinished(() => { release.resolve() }) + const save = f.save.getMockImplementation()! + f.save.mockImplementationOnce(async (grants) => { entered.resolve(); await release.promise; await save(grants) }) + f.answer.mockResolvedValueOnce({ answers: [{ id: 'cua-app-access', selected: ['Always allow'] }] }) + const selection = f.access.select(owner.agent, client, handle, identity, signal) + const settledSelection = Promise.allSettled([selection]) + await entered.promise + const revoked = f.ctx.cuaAccess.revoke(identity.id) + release.resolve() + await revoked + await settledSelection + expect(f.grants()).toEqual([]) + expect(f.save).toHaveBeenCalledTimes(2) + expect(client.disposed).toBe(true) + expect(f.client.disposed).toBe(true) + const replacement = new NativeClient(f.ctx, nativeConfig, unusedConnector) + await expect(f.access.select(owner.agent, replacement, handle, identity, signal)).rejects.toThrow('APP_ACCESS_DENIED') +}) + +it('keeps revoked helper handles invalid after the application is approved again', async () => { + const f = await setup() + await f.access.select(f.agent, f.client, handle, identity, signal) + await f.ctx.cuaAccess.revoke(identity.id) + expect(f.client.disposed).toBe(true) + await f.ctx.cuaAccess.retryApproval(identity.id) + await expect(f.access.select(f.agent, f.client, handle, identity, signal)).rejects.toThrow('NATIVE_CLOSED') + const replacement = new NativeClient(f.ctx, nativeConfig, unusedConnector) + const replacementHandle = 'replacement-app' as CuaAppHandle + await f.access.select(f.agent, replacement, replacementHandle, identity, signal) + const oldDispatch = vi.fn(async () => 'stale window') + await expect(f.access.run(f.client, handle, signal, oldDispatch)).rejects.toThrow('APP_ACCESS_REQUIRED') + expect(oldDispatch).not.toHaveBeenCalled() + await expect(f.access.run(replacement, replacementHandle, signal, async () => 'fresh window')).resolves.toBe('fresh window') +}) + +it('projects restored persistent grants without showing the same application as blocked', async () => { + const f = await setup('Deny') + await expect(f.access.select(f.agent, f.client, handle, identity, signal)).rejects.toThrow('APP_ACCESS_DENIED') + const { persistable: _persistable, ...grant } = identity + await f.save([grant]) + expect(f.ctx.cuaAccess.applications().blockedApps).toEqual([]) + const replacement = new NativeClient(f.ctx, nativeConfig, unusedConnector) + await f.access.select(f.agent, replacement, handle, identity, signal) + expect(f.ctx.cuaAccess.applications().allowedApps).toEqual([grant]) + expect(f.answer).toHaveBeenCalledOnce() +}) + +it('keeps access revoked when stopping the helper reports a cleanup failure', async () => { + const f = await setup('Always allow') + await f.access.select(f.agent, f.client, handle, identity, signal) + vi.spyOn(f.client, 'dispose').mockRejectedValueOnce(new Error('process range did not exit')) + await expect(f.ctx.cuaAccess.revoke(identity.id)).rejects.toThrow('helper cleanup failed') + expect(f.grants()).toEqual([]) + expect(f.client.disposed).toBe(true) + const dispatch = vi.fn(async () => 'unexpected input') + await expect(f.access.run(f.client, handle, signal, dispatch)).rejects.toThrow('APP_ACCESS_REVOKED') + expect(dispatch).not.toHaveBeenCalled() +}) + +it('rejects an answer for a helper that exits while the human question is pending', async () => { + const f = await setup() + const answer = Promise.withResolvers() + const entered: PromiseWithResolvers = Promise.withResolvers() + onTestFinished(() => { answer.resolve({ answers: [{ id: 'cua-app-access', selected: ['Deny'] }] }) }) + f.answer.mockImplementationOnce(() => { entered.resolve(); return answer.promise }) + const selected = f.access.select(f.agent, f.client, handle, identity, signal) + const rejected = expect(selected).rejects.toThrow('NATIVE_CLOSED') + await entered.promise + f.client.invalidate() + answer.resolve({ answers: [{ id: 'cua-app-access', selected: ['Allow for this session'] }] }) + await rejected + const replacement = new NativeClient(f.ctx, nativeConfig, unusedConnector) + await f.access.select(f.agent, replacement, handle, identity, signal) + expect(f.answer).toHaveBeenCalledTimes(2) +}) + +it('keeps other session grants after revocation while requiring fresh native handles', async () => { + const f = await setup() + const other = { ...identity, id: 'b'.repeat(64) as typeof identity.id } + const otherHandle = 'other-application' as CuaAppHandle + await f.access.select(f.agent, f.client, handle, identity, signal) + await f.access.select(f.agent, f.client, otherHandle, other, signal) + await f.ctx.cuaAccess.revoke(identity.id) + await expect(f.access.run(f.client, otherHandle, signal, async () => 'old handle')).rejects.toThrow('NATIVE_CLOSED') + const state = f.ctx.cuaAccess.applications() + expect(state.blockedApps.map(app => app.id)).toEqual([identity.id]) + const replacement = new NativeClient(f.ctx, nativeConfig, unusedConnector) + await f.access.select(f.agent, replacement, otherHandle, other, signal) + await expect(f.access.run(replacement, otherHandle, signal, async () => 'new handle')).resolves.toBe('new handle') + expect(f.answer).toHaveBeenCalledTimes(2) +}) + +it('does not let a stale revoked task invalidate another task after a saved grant is restored', async () => { + const f = await setup() + await f.access.select(f.agent, f.client, handle, identity, signal) + await f.ctx.cuaAccess.revoke(identity.id) + const { persistable: _persistable, ...grant } = identity + await f.save([grant]) + const owner = await f.ctx.agents.create({ sessionId: SessionId('cua-restored-grant') }) + const replacement = new NativeClient(f.ctx, nativeConfig, unusedConnector) + await f.access.select(owner.agent, replacement, handle, identity, signal) + await expect(f.access.run(f.client, handle, signal, async () => 'stale')).rejects.toThrow('APP_ACCESS_REVOKED') + f.ctx.emit('loader/volatile-update', [['allowedApps']]) + expect(replacement.disposed).toBe(false) + await expect(f.access.run(replacement, handle, signal, async () => 'restored')).resolves.toBe('restored') +}) + +it('reads application decisions without native work and retrieves icons separately', async () => { + const f = await setup('Always allow') + await f.access.select(f.agent, f.client, handle, identity, signal) + expect(f.ctx.cuaAccess.applications().allowedApps).toEqual(f.grants()) + expect(native.instances).toHaveLength(1) + await f.ctx.cuaAccess.systemStatus() + const control = native.instances.at(-1)! + const pending = Promise.withResolvers>() + onTestFinished(() => { pending.resolve({}) }) + control.request.mockReturnValueOnce(pending.promise) + const icons = f.ctx.cuaAccess.applicationIcons() + expect(f.ctx.cuaAccess.applications().allowedApps).toEqual(f.grants()) + pending.resolve({ [identity.executablePath]: 'data:image/png;base64,aWNvbg==' }) + expect(await icons).toEqual({ [identity.executablePath]: 'data:image/png;base64,aWNvbg==' }) + expect(f.grants()[0]).not.toHaveProperty('icon') +}) + +it('skips native icon work for an empty application list and propagates icon failures independently', async () => { + const f = await setup() + expect(await f.ctx.cuaAccess.applicationIcons()).toEqual({}) + expect(native.instances).toHaveLength(1) + await f.access.select(f.agent, f.client, handle, identity, signal) + expect(await f.ctx.cuaAccess.applicationIcons()).toEqual({}) + expect(native.instances).toHaveLength(1) + await f.save([{ id: identity.id, name: identity.name, executablePath: identity.executablePath }]) + await f.ctx.cuaAccess.systemStatus() + native.instances.at(-1)!.request.mockRejectedValueOnce(new Error('icons unavailable')) + await expect(f.ctx.cuaAccess.applicationIcons()).rejects.toThrow('icons unavailable') + expect(f.ctx.cuaAccess.applications().allowedApps).toHaveLength(1) + expect((await f.ctx.cuaAccess.systemStatus()).accessibility).toBe(false) +}) + +it('preserves captured launch metadata separately from the current helper parent', async () => { + const f = await setup() + await f.ctx.cuaAccess.systemStatus() + const control = native.instances.at(-1)! + const launchApplication = { name: 'Visual Studio Code', path: '/Applications/Visual Studio Code.app', icon: 'data:image/png;base64,aWNvbg==' } + control.request.mockResolvedValueOnce({ accessibility: false, screenRecording: false, + helper: { pid: 9 }, parent: { pid: 1, path: '/sbin/launchd' }, launchApplication }) + expect(await f.ctx.cuaAccess.systemStatus()).toMatchObject({ parent: { pid: 1 }, launchApplication }) +}) + + +it('omits session-only grants from the permission page and retains their access after another session saves a grant', async () => { + const f = await setup() + await f.access.select(f.agent, f.client, handle, identity, signal) + const other = { ...identity, id: 'b'.repeat(64) as typeof identity.id, name: 'Another application', executablePath: '/Applications/Other.app' } + await f.access.select(f.agent, f.client, 'other-application' as CuaAppHandle, other, signal) + expect(f.ctx.cuaAccess.applications()).toEqual({ settingsNamespace: 'cua', allowedApps: [], blockedApps: [], persistentAvailable: true }) + const owner = await f.ctx.agents.create({ sessionId: SessionId('cua-always-allow') }) + const client = new NativeClient(f.ctx, nativeConfig, unusedConnector) + f.answer.mockResolvedValueOnce({ answers: [{ id: 'cua-app-access', selected: ['Always allow'] }] }) + await f.access.select(owner.agent, client, handle, identity, signal) + const state = f.ctx.cuaAccess.applications() + expect(state.allowedApps.map(app => app.id)).toEqual([identity.id]) + expect(state.blockedApps).toEqual([]) + await expect(f.access.run(f.client, handle, signal, async () => 'still allowed')).resolves.toBe('still allowed') +}) + +it('configures persistent PTC through the existing profile editor only on explicit request', async () => { + const f = await setup() + expect(f.ctx.cuaAccess.ptcConfiguration()).toEqual({ persistent: false, writable: false }) + await expect(f.ctx.cuaAccess.enablePersistentPtc()).rejects.toThrow('no editable profile entry') + const entry = { options: { id: 'ptc-runtime' } } + const runtime = { programFormat: 'function-body' } + const edit = vi.fn(async (received: object, change: (current: Record) => Record) => { + expect(received).toBe(entry) + expect(change({ timeoutMs: 4200 })).toEqual({ timeoutMs: 4200, executionMode: 'persistent-in-memory' }) + runtime.programFormat = 'repl' + }) + f.ctx.provide('ptcRuntime', runtime) + f.ctx.provide('configEditor', { entries: () => [entry], edit }) + expect(f.ctx.cuaAccess.ptcConfiguration()).toEqual({ persistent: false, writable: true }) + expect(edit).not.toHaveBeenCalled() + await f.ctx.cuaAccess.enablePersistentPtc() + expect(f.ctx.cuaAccess.ptcConfiguration()).toEqual({ persistent: true, writable: true }) + await f.ctx.cuaAccess.enablePersistentPtc() + expect(edit).toHaveBeenCalledOnce() +}) diff --git a/packages/experimental/cua/tests/agent-scopes.spec.ts b/packages/experimental/cua/tests/agent-scopes.spec.ts new file mode 100644 index 0000000000..fd1a349130 --- /dev/null +++ b/packages/experimental/cua/tests/agent-scopes.spec.ts @@ -0,0 +1,106 @@ +/** CUA tool and prompt ownership follows live Agent presets and provider disposal. */ +import { resolve } from 'node:path' +import { Context } from '@deepseek-ai/cordis' +import Loader from '@deepseek-ai/cordis-plugin-loader' +import AgentPresets from '@deepseek-ai/dsh-agent-preset-registry' +import { mountAgentLoopTestDependencies, mountAgentLoopTestHarness } from '@deepseek-ai/dsh-agent-loop-testkit' +import ComputerUseRegistry from '@deepseek-ai/dsh-computer-use' +import Subprocess from '@deepseek-ai/dsh-subprocess' +import { SessionId } from '@deepseek-ai/dsh-session' +import { expect, it, onTestFinished, vi } from 'vitest' +import { CuaConfig, installCua } from '../src/index.ts' + +const provider = { + name: 'test-cua-scopes', + inject: ['computerUse', 'subprocess', 'tools', 'systemPrompt', 'agents'], + Config: CuaConfig, + apply(ctx: Context, config: CuaConfig) { + installCua(ctx, config, { + helperPath: resolve('unused-cua-helper'), + createChannel() { throw new Error('Scope tests must not start a native helper') }, + description: 'Fixture computer interaction', guidance: 'Fixture CUA guidance', + }) + }, +} + +async function setup(presetIds = ['standard', 'ptc']) { + const ctx = new Context() + onTestFinished(() => ctx.fiber.dispose()) + ctx.baseUrl = import.meta.url + await ctx.plugin(Loader) + await mountAgentLoopTestDependencies(ctx) + await mountAgentLoopTestHarness(ctx) + await ctx.plugin(ComputerUseRegistry) + await ctx.plugin(Subprocess, []) + await ctx.plugin(AgentPresets, { default: 'standard' }) + for (const id of presetIds) { + await ctx.plugin({ inject: ['agentPresets'], async* apply(owner: Context) { + yield await owner.agentPresets.register({ id, plugins: [] }) + } }) + } + const create = (id: string, preset: string) => ctx.agents.create({ + sessionId: SessionId(id), setup: async (owner) => { await ctx.agentPresets.mount(owner, preset) }, + }) + return { ctx, create } +} + +it('adds and removes CUA for existing and new PTC Agents without changing other presets', async () => { + const { ctx, create } = await setup() + const existing = await create('existing', 'ptc') + const standard = await create('standard', 'standard') + expect(ctx.tools.get('cua', existing.agent)).toBeUndefined() + const mounted = await ctx.plugin(provider, {}) + await expect.poll(() => ctx.tools.get('cua', existing.agent)).toBeDefined() + expect(ctx.tools.get('cua', standard.agent)).toBeUndefined() + expect((await ctx.systemPrompt.assemble({ scope: existing.agent })).sections) + .toContainEqual(expect.objectContaining({ name: 'computer-use:dsh-cua', text: 'Fixture CUA guidance\n\nMaximum click holdMs: 5000 ms.' })) + await ctx.agentPresets.select(existing.agent, 'ptc') + expect(ctx.tools.schemas(existing.agent).filter(tool => tool.name === 'cua')).toHaveLength(1) + const fresh = await create('fresh', 'ptc') + expect(ctx.tools.get('cua', fresh.agent)).toBeDefined() + await ctx.agentPresets.select(existing.agent, 'standard') + await expect.poll(() => ctx.tools.get('cua', existing.agent)).toBeUndefined() + expect((await ctx.systemPrompt.assemble({ scope: existing.agent })).sections) + .not.toContainEqual(expect.objectContaining({ name: 'computer-use:dsh-cua' })) + await ctx.agentPresets.select(existing.agent, 'ptc') + await expect.poll(() => ctx.tools.get('cua', existing.agent)).toBeDefined() + await fresh.dispose() + expect(ctx.tools.get('cua', fresh.agent)).toBeUndefined() + await mounted.dispose() + expect(ctx.tools.get('cua', existing.agent)).toBeUndefined() + expect((await ctx.systemPrompt.assemble({ scope: existing.agent })).sections) + .not.toContainEqual(expect.objectContaining({ name: 'computer-use:dsh-cua' })) + const reloaded = await ctx.plugin(provider, {}) + await expect.poll(() => ctx.tools.get('cua', existing.agent)).toBeDefined() + expect(ctx.tools.get('cua', standard.agent)).toBeUndefined() + await reloaded.dispose() +}) + +it('ignores unloaded Session selections and reports a failed live scope update', async () => { + const { ctx, create } = await setup() + const handle = await create('active', 'ptc') + const mounted = await ctx.plugin(provider, {}) + await expect.poll(() => ctx.tools.get('cua', handle.agent)).toBeDefined() + const warn = vi.spyOn(mounted.ctx.logger, 'warn').mockImplementation(() => {}) + const lookup = vi.spyOn(ctx.agentPresets, 'composedPreset') + onTestFinished(() => { lookup.mockRestore(); warn.mockRestore() }) + ctx.emit('agent-preset/selected', SessionId('unloaded'), 'ptc') + expect(lookup).not.toHaveBeenCalled() + lookup.mockImplementationOnce(() => { throw new Error('Preset lookup failed during reload') }) + ctx.emit('agent-preset/selected', handle.agent.id, 'ptc') + await expect.poll(() => warn.mock.calls).toEqual([['CUA tool scope update failed: Error: Preset lookup failed during reload']]) + expect(ctx.tools.get('cua', handle.agent)).toBeDefined() + await ctx.agentPresets.select(handle.agent, 'standard') + await expect.poll(() => ctx.tools.get('cua', handle.agent)).toBeUndefined() +}) + +it('accepts a matching preset registered after the CUA provider', async () => { + const { ctx, create } = await setup([]) + await ctx.plugin(provider, {}) + expect(ctx.tools.get('cua')).toBeUndefined() + await ctx.plugin({ inject: ['agentPresets'], async* apply(owner: Context) { + yield await owner.agentPresets.register({ id: 'ptc', plugins: [] }) + } }) + const handle = await create('late-preset', 'ptc') + expect(ctx.tools.get('cua', handle.agent)).toBeDefined() +}) diff --git a/packages/experimental/cua/tests/application-access.spec.ts b/packages/experimental/cua/tests/application-access.spec.ts index 38d5faa62d..74d2ddefbf 100644 --- a/packages/experimental/cua/tests/application-access.spec.ts +++ b/packages/experimental/cua/tests/application-access.spec.ts @@ -36,7 +36,7 @@ const identity = ApplicationAuthorizationResult.parse({ id: 'a'.repeat(64), name const handle = 'fixture-app' as CuaAppHandle const signal = new AbortController().signal -async function setup(choice = 'Allow for this task', options: { writable?: boolean; questions?: boolean; initial?: readonly CuaApplicationGrant[] } = {}) { +async function setup(choice = 'Allow for this session', options: { writable?: boolean; questions?: boolean; initial?: readonly CuaApplicationGrant[] } = {}) { native.instances.length = 0 const ctx = new Context() onTestFinished(() => ctx.fiber.dispose()) @@ -63,7 +63,7 @@ it('refuses raw handle dispatch until getApp obtains consent', async () => { expect(dispatch).not.toHaveBeenCalled() }) -it('retains task consent for the exact helper handle without saving it', async () => { +it('reuses session consent across application handles and connections without saving it', async () => { const f = await setup() await f.access.select(f.agent, f.client, handle, identity, signal) await expect(f.access.run(f.client, handle, signal, async () => 'read')).resolves.toBe('read') @@ -72,6 +72,11 @@ it('retains task consent for the exact helper handle without saving it', async ( expect(f.save).not.toHaveBeenCalled() const replacement = new NativeClient(f.ctx, nativeConfig, unusedConnector) await expect(f.access.run(replacement, handle, signal, async () => null)).rejects.toThrow('APP_ACCESS_REQUIRED') + await f.access.select(f.agent, replacement, 'fresh-app' as CuaAppHandle, identity, signal) + await f.access.select(f.agent, replacement, 'restarted-app' as CuaAppHandle, identity, signal) + expect(f.answer).toHaveBeenCalledOnce() + await expect(f.access.run(f.client, handle, signal, async () => null)).rejects.toThrow('APP_ACCESS_REQUIRED') + await expect(f.access.run(replacement, 'restarted-app' as CuaAppHandle, signal, async () => 'read')).resolves.toBe('read') }) it('does not repeatedly ask after an application is denied', async () => { @@ -120,10 +125,10 @@ it('reuses only saved signed identities when a task or helper is replaced', asyn it.each([ { answers: [] }, - { answers: [{ id: 'another-question', selected: ['Allow for this task'] }] }, - { answers: [{ id: 'cua-app-access', selected: ['Allow for this task', 'Deny'] }] }, + { answers: [{ id: 'another-question', selected: ['Allow for this session'] }] }, + { answers: [{ id: 'cua-app-access', selected: ['Allow for this session', 'Deny'] }] }, { answers: [{ id: 'cua-app-access', selected: [], custom: 'Always allow' }] }, - { answers: [{ id: 'cua-app-access', selected: ['Allow for this task'], custom: 'different scope' }] }, + { answers: [{ id: 'cua-app-access', selected: ['Allow for this session'], custom: 'different scope' }] }, ])('does not interpret an incomplete or free-text answer as consent: %j', async (answer) => { const f = await setup() f.answer.mockResolvedValueOnce(answer) @@ -199,8 +204,8 @@ it('reuses one accepted question for duplicate selections', async () => { f.answer.mockImplementationOnce(() => { entered.resolve(); return answer.promise }) const first = f.access.select(f.agent, f.client, handle, identity, signal) await entered.promise - const second = f.access.select(f.agent, f.client, handle, identity, signal) - answer.resolve({ answers: [{ id: 'cua-app-access', selected: ['Allow for this task'] }] }) + const second = f.access.select(f.agent, f.client, 'another-handle' as CuaAppHandle, identity, signal) + answer.resolve({ answers: [{ id: 'cua-app-access', selected: ['Allow for this session'] }] }) await Promise.all([first, second]) expect(f.answer).toHaveBeenCalledOnce() await expect(f.access.run(f.client, handle, signal, async () => 'allowed')).resolves.toBe('allowed') @@ -244,7 +249,7 @@ it('cancels native work when saved consent is removed from configuration', async entered.resolve(undefined) return done.promise }) - const rejected = expect(pending).rejects.toThrow('native cancelled') + const rejected = expect(pending).rejects.toThrow('APP_ACCESS_DENIED') await entered.promise await f.save([]) f.ctx.emit('loader/volatile-update', [['allowedApps']]) @@ -279,7 +284,7 @@ it('rejects an answer for a helper that exits while the human question is pendin const rejected = expect(selected).rejects.toThrow('NATIVE_CLOSED') await entered.promise f.client.invalidate() - answer.resolve({ answers: [{ id: 'cua-app-access', selected: ['Allow for this task'] }] }) + answer.resolve({ answers: [{ id: 'cua-app-access', selected: ['Allow for this session'] }] }) await rejected }) @@ -317,3 +322,57 @@ it('keeps an active task operation running when a different task denies the same await expect(pending).resolves.toBeNull() } finally { release.resolve(null); await pending } }) + + +it.each(['Allow for this session', 'Deny'])('retains %s when the same session gets a new agent', async (choice) => { + const f = await setup(choice) + const select = f.access.select(f.agent, f.client, handle, identity, signal) + if (choice === 'Deny') await expect(select).rejects.toThrow('APP_ACCESS_DENIED') + else await select + await f.owner.dispose() + f.access.forget(f.agent) + const owner = await f.ctx.agents.create({ sessionId: f.agent.id }) + const client = new NativeClient(f.ctx, nativeConfig, unusedConnector) + const resumed = f.access.select(owner.agent, client, handle, identity, signal) + if (choice === 'Deny') await expect(resumed).rejects.toThrow('APP_ACCESS_DENIED') + else { + await resumed + await expect(f.access.run(client, handle, signal, async () => 'resumed')).resolves.toBe('resumed') + } + expect(f.answer).toHaveBeenCalledOnce() +}) + +it('asks separately for another session and a changed application identity', async () => { + const f = await setup() + await f.access.select(f.agent, f.client, handle, identity, signal) + const owner = await f.ctx.agents.create({ sessionId: SessionId('separate-session') }) + const client = new NativeClient(f.ctx, nativeConfig, unusedConnector) + await f.access.select(owner.agent, client, handle, identity, signal) + expect(f.answer).toHaveBeenCalledTimes(2) + await f.access.select(f.agent, f.client, 'changed-app' as CuaAppHandle, + { ...identity, id: 'b'.repeat(64) as typeof identity.id }, signal) + expect(f.answer).toHaveBeenCalledTimes(3) +}) + +it('revokes session consent even after its agent and connection are released', async () => { + const f = await setup() + await f.access.select(f.agent, f.client, handle, identity, signal) + f.access.forget(f.agent) + await f.access.revoke(identity.id) + const replacement = new NativeClient(f.ctx, nativeConfig, unusedConnector) + await expect(f.access.select(f.agent, replacement, handle, identity, signal)).rejects.toThrow('APP_ACCESS_DENIED') + expect(f.answer).toHaveBeenCalledOnce() + f.access.retryApproval(identity.id) + await f.access.select(f.agent, replacement, handle, identity, signal) + expect(f.answer).toHaveBeenCalledTimes(2) +}) + +it('clears session consent when the provider is disabled and recreated', async () => { + const f = await setup() + await f.access.select(f.agent, f.client, handle, identity, signal) + await f.access.dispose() + const access = new ApplicationAccess(f.ctx, { grants: f.grants, save: f.save }) + const client = new NativeClient(f.ctx, nativeConfig, unusedConnector) + await access.select(f.agent, client, handle, identity, signal) + expect(f.answer).toHaveBeenCalledTimes(2) +}) diff --git a/packages/experimental/cua/tests/controller.client.spec.ts b/packages/experimental/cua/tests/controller.client.spec.ts new file mode 100644 index 0000000000..7a2f146ca2 --- /dev/null +++ b/packages/experimental/cua/tests/controller.client.spec.ts @@ -0,0 +1,281 @@ +/** Permission refreshes preserve observations and join user actions before withdrawal. */ +import { setImmediate } from 'node:timers/promises' +import type { Context } from '@deepseek-ai/cordis' +import type {} from '@deepseek-ai/dsh-cua/remote' +import { RemoteError } from '@deepseek-ai/dsh-typert-protocol' +import { expect, it, onTestFinished, vi } from 'vitest' +import type { CuaApplicationId, CuaApplicationState, CuaSystemPermissionState } from '../src/access-types.ts' +import { PermissionController } from '../src/client/controller.ts' + +const value: CuaSystemPermissionState = { accessibility: true, screenRecording: false, helper: { pid: 9 }, parent: { pid: 8 } } +const applications: CuaApplicationState = { + allowedApps: [], blockedApps: [], settingsNamespace: 'cua', persistentAvailable: true } +type CuaRemote = Context['remote']['cuaAccess'] +function fixture() { + const actions = { + ptcConfiguration: vi.fn(async () => ({ ok: true, value: { persistent: false, writable: true } })), + systemStatus: vi.fn(async () => ({ ok: true, value })), + applications: vi.fn(async () => ({ ok: true, value: applications })), + applicationIcons: vi.fn(async () => ({ ok: true, value: {} })), + requestPermission: vi.fn(async () => ({ ok: true, value })), + openSettings: vi.fn(async () => ({ ok: true, value: undefined })), + revoke: vi.fn(async () => ({ ok: true, value: undefined })), + retryApproval: vi.fn(async () => ({ ok: true, value: undefined })), + enablePersistentPtc: vi.fn(async () => ({ ok: true, value: undefined })), + } + const controller = new PermissionController(actions) + return { actions, controller, face: controller.face() } +} + +it('refreshes after a user request without treating the request itself as a grant', async () => { + const f = fixture() + f.face.request('screenRecording') + await vi.waitFor(() => { expect(f.controller.state.getSnapshot().busy).toBe(false) }) + expect(f.actions.requestPermission).toHaveBeenCalledWith('screenRecording') + expect(f.controller.state.getSnapshot().system.value?.screenRecording).toBe(false) + expect(f.controller.state.getSnapshot().toast?.kind).toBe('requested') + f.face.dismissToast() + expect(f.controller.state.getSnapshot().toast).toBeNull() + await f.controller.dispose() +}) + +it('retains the last observation while marking a failed refresh', async () => { + const f = fixture() + f.face.refresh() + await vi.waitFor(() => { expect(f.controller.state.getSnapshot().system.value).toEqual(value) }) + f.actions.systemStatus.mockRejectedValueOnce(new Error('connection closed')) + f.face.refresh() + await vi.waitFor(() => { expect(f.controller.state.getSnapshot().system.error).toBe('connection closed') }) + expect(f.controller.state.getSnapshot().system.value).toEqual(value) + await f.controller.dispose() +}) + +it.each([new Error('write refused'), 'write refused'])('reports settings and revocation outcomes through the shared toast state', async (error) => { + const f = fixture() + f.face.openSettings('accessibility') + await vi.waitFor(() => { expect(f.controller.state.getSnapshot().toast?.kind).toBe('opened') }) + f.face.revoke('a'.repeat(64) as CuaApplicationId) + await vi.waitFor(() => { expect(f.controller.state.getSnapshot().toast?.kind).toBe('revoked') }) + f.face.retryApproval('a'.repeat(64) as CuaApplicationId) + await vi.waitFor(() => { expect(f.controller.state.getSnapshot().toast?.kind).toBe('retryAllowed') }) + f.actions.revoke.mockRejectedValueOnce(error) + f.face.revoke('a'.repeat(64) as CuaApplicationId) + await vi.waitFor(() => { expect(f.controller.state.getSnapshot().toast).toMatchObject({ kind: 'failed', message: 'write refused' }) }) + await f.controller.dispose() +}) + +it('joins a pending read without publishing after disposal or starting queued actions', async () => { + const f = fixture() + const pending = Promise.withResolvers() + const entered = Promise.withResolvers() + f.actions.systemStatus.mockImplementationOnce(async () => { + entered.resolve(undefined) + return { ok: true, value: await pending.promise } + }) + f.face.refresh() + await entered.promise + f.face.request('accessibility') + const before = f.controller.state.getSnapshot() + const closed = f.controller.dispose() + pending.resolve(value) + await closed + expect(f.controller.state.getSnapshot()).toBe(before) + expect(f.actions.requestPermission).not.toHaveBeenCalled() + f.face.refresh() + expect(f.actions.systemStatus).toHaveBeenCalledOnce() +}) + +it('revokes access while icons are pending, serializes writes, and joins reads on disposal', async () => { + const f = fixture(), icons = Promise.withResolvers>() + const action = Promise.withResolvers(), entered = Promise.withResolvers() + onTestFinished(async () => { icons.resolve({}); action.resolve(undefined); await f.controller.dispose() }) + f.actions.applicationIcons.mockImplementationOnce(async () => ({ ok: true, value: await icons.promise })) + f.actions.revoke.mockImplementationOnce(async () => { + entered.resolve(undefined) + await action.promise + return { ok: true, value: undefined } + }) + f.face.refresh() + await vi.waitFor(() => { expect(f.controller.state.getSnapshot().applications.busy).toBe(false) }) + expect(f.controller.state.getSnapshot().icons.busy).toBe(true) + f.face.revoke('a'.repeat(64) as CuaApplicationId) + await entered.promise + f.face.retryApproval('a'.repeat(64) as CuaApplicationId) + await setImmediate() + expect(f.actions.retryApproval).not.toHaveBeenCalled() + action.resolve(undefined) + await vi.waitFor(() => { expect(f.controller.state.getSnapshot().busy).toBe(false) }) + expect(f.actions.retryApproval).toHaveBeenCalledOnce() + expect(f.controller.state.getSnapshot().toast?.kind).toBe('retryAllowed') + const before = f.controller.state.getSnapshot() + let disposed = false + const closed = f.controller.dispose().then(() => { disposed = true }) + await setImmediate() + expect(disposed).toBe(false) + icons.resolve({ '/old/app': 'old-icon' }) + await closed + expect(f.controller.state.getSnapshot()).toBe(before) +}) + +it('ignores stale toast completion after the permission UI is disposed', async () => { + const f = fixture() + f.face.request('accessibility') + await vi.waitFor(() => { expect(f.controller.state.getSnapshot().toast?.kind).toBe('requested') }) + await f.controller.dispose() + const before = f.controller.state.getSnapshot() + f.face.dismissToast() + f.face.refresh() + expect(f.controller.state.getSnapshot()).toBe(before) +}) + +it.each(['success', 'failure'])('settles an in-flight action %s after disposal without publishing', async (outcome) => { + const f = fixture(), pending = Promise.withResolvers(), entered = Promise.withResolvers() + onTestFinished(async () => { pending.resolve(undefined); await f.controller.dispose() }) + f.actions.requestPermission.mockImplementationOnce(async () => { + entered.resolve(undefined) + await pending.promise + return { ok: true, value } + }) + f.face.request('accessibility') + await entered.promise + const before = f.controller.state.getSnapshot(), closed = f.controller.dispose() + if (outcome === 'failure') pending.reject(new Error('permission request ended after withdrawal')) + else pending.resolve(undefined) + await closed + expect(f.controller.state.getSnapshot()).toBe(before) + expect(f.actions.systemStatus).not.toHaveBeenCalled() +}) + +it('contains a late read rejection and supports errors without an Error instance', async () => { + const f = fixture() + f.actions.systemStatus.mockRejectedValueOnce('offline') + f.face.refresh() + await vi.waitFor(() => { expect(f.controller.state.getSnapshot().system.error).toBe('offline') }) + const pending = Promise.withResolvers(), entered = Promise.withResolvers() + f.actions.systemStatus.mockImplementationOnce(async () => { + entered.resolve(undefined) + return { ok: true, value: await pending.promise } + }) + f.face.refresh() + await entered.promise + const before = f.controller.state.getSnapshot(), closed = f.controller.dispose() + pending.reject(new Error('late disconnect')) + await closed + expect(f.controller.state.getSnapshot()).toBe(before) +}) + + +it.each(['success', 'failure'])('does not publish an older refresh %s while the activation refresh is queued', async (outcome) => { + const f = fixture() + const first = Promise.withResolvers(), firstEntered = Promise.withResolvers() + const latest = Promise.withResolvers(), latestEntered = Promise.withResolvers() + onTestFinished(async () => { first.resolve(value); latest.resolve(value); await f.controller.dispose() }) + f.actions.systemStatus.mockImplementationOnce(async () => { + firstEntered.resolve(undefined) + return { ok: true, value: await first.promise } + }) + f.actions.systemStatus.mockImplementationOnce(async () => { + latestEntered.resolve(undefined) + return { ok: true, value: await latest.promise } + }) + const observations: Array> = [] + const unsubscribe = f.controller.state.subscribe(() => { observations.push(f.controller.state.getSnapshot()) }) + onTestFinished(unsubscribe) + f.face.refresh() + await firstEntered.promise + f.face.refresh() + if (outcome === 'failure') first.reject(new Error('provider is reloading')) + else first.resolve({ ...value, screenRecording: true }) + await latestEntered.promise + expect(observations.every(state => state.system.error === null && state.system.value === null && state.busy)).toBe(true) + latest.resolve(value) + await vi.waitFor(() => { expect(f.controller.state.getSnapshot()).toMatchObject({ busy: false, system: { error: null, value } }) }) + expect(observations.every(state => state.system.error === null)).toBe(true) +}) + +it('keeps a user-action failure visible when a refresh is queued behind it', async () => { + const f = fixture(), action = Promise.withResolvers(), entered = Promise.withResolvers() + onTestFinished(async () => { action.resolve(undefined); await f.controller.dispose() }) + f.actions.requestPermission.mockImplementationOnce(async () => { + entered.resolve(undefined) + await action.promise + return { ok: true, value } + }) + f.face.request('accessibility') + await entered.promise + f.face.refresh() + action.reject(new Error('permission request failed')) + await vi.waitFor(() => { expect(f.controller.state.getSnapshot().busy).toBe(false) }) + expect(f.controller.state.getSnapshot()).toMatchObject({ system: { value, error: null }, + toast: { kind: 'failed', message: 'permission request failed' } }) +}) + + +it('publishes each observation before the other queries finish', async () => { + const f = fixture() + const system = Promise.withResolvers() + const apps = Promise.withResolvers() + const icons = Promise.withResolvers>() + onTestFinished(async () => { system.resolve(value); apps.resolve(applications); icons.resolve({}); await f.controller.dispose() }) + f.actions.systemStatus.mockImplementationOnce(async () => ({ ok: true, value: await system.promise })) + f.actions.applications.mockImplementationOnce(async () => ({ ok: true, value: await apps.promise })) + f.actions.applicationIcons.mockImplementationOnce(async () => ({ ok: true, value: await icons.promise })) + f.face.refresh() + apps.resolve(applications) + await vi.waitFor(() => { + expect(f.controller.state.getSnapshot().applications).toEqual({ value: applications, error: null, busy: false }) + }) + expect(f.controller.state.getSnapshot().system).toMatchObject({ value: null, busy: true }) + system.resolve(value) + await vi.waitFor(() => { expect(f.controller.state.getSnapshot().system).toEqual({ value, error: null, busy: false }) }) + expect(f.controller.state.getSnapshot().icons).toMatchObject({ value: null, busy: true }) + icons.reject(new Error('icon unavailable')) + await vi.waitFor(() => { expect(f.controller.state.getSnapshot().busy).toBe(false) }) + expect(f.controller.state.getSnapshot()).toMatchObject({ + system: { value, error: null }, applications: { value: applications, error: null }, + icons: { value: null, error: 'icon unavailable', busy: false }, + }) +}) + +it('keeps application decisions available when system inspection fails', async () => { + const f = fixture() + f.actions.systemStatus.mockRejectedValueOnce(new Error('helper offline')) + f.face.refresh() + await vi.waitFor(() => { expect(f.controller.state.getSnapshot().busy).toBe(false) }) + expect(f.controller.state.getSnapshot()).toMatchObject({ + system: { value: null, error: 'helper offline' }, + applications: { value: applications, error: null, busy: false }, + }) + await f.controller.dispose() +}) + + +it.each([ + ['ptcConfiguration', 'runtime'], ['systemStatus', 'system'], + ['applications', 'applications'], ['applicationIcons', 'icons'], +] as const)('contains an unsuccessful %s response in its own query section', async (method, key) => { + const f = fixture() + onTestFinished(() => f.controller.dispose()) + f.face.refresh() + await vi.waitFor(() => { expect(f.controller.state.getSnapshot().busy).toBe(false) }) + const previous = f.controller.state.getSnapshot()[key].value + f.actions[method].mockResolvedValueOnce({ ok: false, error: new RemoteError('gateway/internal', 'query refused', {}) }) + f.face.refresh() + await vi.waitFor(() => { expect(f.controller.state.getSnapshot().busy).toBe(false) }) + expect(f.controller.state.getSnapshot()[key]).toEqual({ value: previous, busy: false, error: 'query refused' }) + expect(f.controller.state.getSnapshot().toast).toBeNull() +}) + +it.each(['requestPermission', 'openSettings', 'revoke', 'retryApproval'] as const)( + 'reports an unsuccessful %s response instead of a success notice', async (method) => { + const f = fixture() + onTestFinished(() => f.controller.dispose()) + f.actions[method].mockResolvedValueOnce({ ok: false, error: new RemoteError('gateway/internal', 'action refused', {}) }) + if (method === 'requestPermission') f.face.request('accessibility') + else if (method === 'openSettings') f.face.openSettings('accessibility') + else f.face[method]('a'.repeat(64) as CuaApplicationId) + await vi.waitFor(() => { expect(f.controller.state.getSnapshot().busy).toBe(false) }) + expect(f.controller.state.getSnapshot().toast).toMatchObject({ kind: 'failed', message: 'action refused' }) + expect(f.actions.systemStatus).toHaveBeenCalledOnce() + }, +) diff --git a/packages/experimental/cua/tests/export-types.spec.ts b/packages/experimental/cua/tests/export-types.spec.ts new file mode 100644 index 0000000000..399fd84355 --- /dev/null +++ b/packages/experimental/cua/tests/export-types.spec.ts @@ -0,0 +1,29 @@ +/** CUA's public types distinguish object consumers from its generated permission transport. */ +import { expect, expectTypeOf, it } from 'vitest' +import { Context } from '@deepseek-ai/cordis' +import * as Cua from '../src/index.ts' +import type { CuaApi } from '@deepseek-ai/dsh-cua/api' +import type { CuaApplicationState } from '@deepseek-ai/dsh-cua/types' +// @ts-expect-error Permission payloads are not part of the platform-provider root entry. +import type { CuaSystemPermissionState as RemovedRoot } from '@deepseek-ai/dsh-cua' +// @ts-expect-error The object API does not expose permission-page state. +import type { CuaSystemPermissionState as RemovedApi } from '@deepseek-ai/dsh-cua/api' +// @ts-expect-error Object interfaces use /api; /types is the generated Remote type entry. +import type { CuaApi as RemovedTypes } from '@deepseek-ai/dsh-cua/types' +// @ts-expect-error The nested grant structure is internal, not an independently exported type. +import type { CuaApplicationGrant as RemovedGrant } from '@deepseek-ai/dsh-cua/types' + +it('keeps application operations and the necessary Remote payload types available', () => { + expectTypeOf().toHaveProperty('getApp') + expectTypeOf().toHaveProperty('allowedApps') + expectTypeOf<[RemovedRoot, RemovedApi, RemovedTypes, RemovedGrant]>().toBeArray() +}) + +it('mounts the Host marker without installing native or browser services', async () => { + const ctx = new Context() + try { + await ctx.plugin(Cua) + expect(ctx.get('cuaAccess')).toBeUndefined() + expect(ctx.get('computerUse')).toBeUndefined() + } finally { await ctx.fiber.dispose() } +}) diff --git a/packages/experimental/cua/tests/exports.built.e2e.ts b/packages/experimental/cua/tests/exports.built.e2e.ts new file mode 100644 index 0000000000..9300b946d5 --- /dev/null +++ b/packages/experimental/cua/tests/exports.built.e2e.ts @@ -0,0 +1,18 @@ +/** Published CUA entries retain headless isolation and the existing generated transport. */ +import { execFile } from 'node:child_process' +import { fileURLToPath } from 'node:url' +import { promisify } from 'node:util' +import { resolveExampleLaunch } from '@deepseek-ai/dsh-loader-smoke' +import { expect, it, onTestFinished } from 'vitest' + +it('loads the built Host, object API and native exports without Client dependencies', async () => { + const fixture = fileURLToPath(new URL('./fixtures/exports.mjs', import.meta.url)) + const launch = resolveExampleLaunch({ srcBin: fixture, libBin: fixture, mode: 'lib' }) + const controller = new AbortController() + const child = promisify(execFile)(launch.command, launch.args, { signal: controller.signal }) + onTestFinished(async () => { + controller.abort() + await child.catch(() => { /* A failed or cancelled fixture is asserted by the test. */ }) + }) + expect((await child).stdout).toBe('headless exports and generated Remote passed\n') +}) diff --git a/packages/experimental/cua/tests/fixtures/exports.mjs b/packages/experimental/cua/tests/fixtures/exports.mjs new file mode 100644 index 0000000000..aa1625007c --- /dev/null +++ b/packages/experimental/cua/tests/fixtures/exports.mjs @@ -0,0 +1,28 @@ +/** Built package smoke: headless consumers must not import the browser implementation. */ +import assert from 'node:assert/strict' +import { registerHooks } from 'node:module' + +const guard = registerHooks({ + resolve(specifier, context, next) { + assert(!/^(?:react(?:-dom)?(?:\/|$)|@deepseek-ai\/dsh-client-)/.test(specifier), `Headless import reached ${specifier}`) + const result = next(specifier, context) + assert(!/\/cua\/lib\/(?:client(?:\/|\.)|types\/client\/)/.test(result.url), `Headless import reached ${result.url}`) + return result + }, +}) +const { Context } = await import('@deepseek-ai/cordis') +const host = await import('@deepseek-ai/dsh-cua') +const { createCuaApi } = await import('@deepseek-ai/dsh-cua/api') +const { connectCua } = await import('@deepseek-ai/dsh-cua/bridge') +const { NativeClient } = await import('@deepseek-ai/dsh-cua/native-client') +const { Request } = await import('@deepseek-ai/dsh-cua/protocol') +for (const callable of [host.apply, host.installCua, host.CuaAccess, createCuaApi, connectCua, NativeClient]) assert.equal(typeof callable, 'function') +assert.equal(Request.parse({ method: 'getCapabilities', params: {} }).method, 'getCapabilities') +const ctx = new Context() +try { await ctx.plugin(host); assert.equal(ctx.get('cuaAccess'), undefined) } +finally { await ctx.fiber.dispose() } +guard.deregister() +const remote = await import('@deepseek-ai/dsh-cua/remote') +assert.equal(remote.default.package, '@deepseek-ai/dsh-cua') +assert.equal(remote.default.descriptors.length, 9) +process.stdout.write('headless exports and generated Remote passed\n') diff --git a/packages/experimental/cua/tests/mount.client.spec.ts b/packages/experimental/cua/tests/mount.client.spec.ts new file mode 100644 index 0000000000..02145516f6 --- /dev/null +++ b/packages/experimental/cua/tests/mount.client.spec.ts @@ -0,0 +1,192 @@ +/** The optional permission namespace and existing page slots share one disposal lifetime. */ +import assert from 'node:assert/strict' +import { setImmediate } from 'node:timers/promises' +import { Context, Service } from '@deepseek-ai/cordis' +import { LocaleRuntime } from '@deepseek-ai/dsh-client-locale/client' +import { SlotRegistry } from '@deepseek-ai/dsh-client-ui-renderer/client' +import { createSnapshotStore } from '@deepseek-ai/dsh-client-store' +import { RemoteError, type RemoteResult, type TypertRemoteContribution } from '@deepseek-ai/dsh-typert-protocol' +import type { CuaApplicationState, CuaSystemPermissionState } from '../src/access-types.ts' +import { expect, it, vi } from 'vitest' +import type { PermissionFace } from '../src/client/controller.ts' +import { mountCuaPermissions, inject } from '../src/client/mount.ts' + +const contribution: TypertRemoteContribution = { package: '@deepseek-ai/dsh-cua', descriptors: [] } +const value: CuaApplicationState & { system: CuaSystemPermissionState } = { + system: { accessibility: false, screenRecording: false, helper: { pid: 2 }, parent: { pid: 1 } }, + allowedApps: [], blockedApps: [], settingsNamespace: 'cua', persistentAvailable: true } + +function actionsOf(value: Record): asserts value is Record & PermissionFace { + assert(typeof value.refresh === 'function' && typeof value.request === 'function' && typeof value.openSettings === 'function') + assert(typeof value.revoke === 'function' && typeof value.dismissToast === 'function') + assert(typeof value.retryApproval === 'function') + assert(typeof value.hooks === 'object' && value.hooks !== null) +} + +async function fixture(fail = false) { + const ctx = new Context(), unmount = vi.fn(async () => {}), unsubscribe = vi.fn() + let settingsListener: ((namespace: string, revision: number) => void) | undefined + class Remote extends Service { + constructor() { super(ctx, 'remote') } + async $mount(received: TypertRemoteContribution) { + expect(received).toBe(contribution) + const owned = this.ctx.effect(() => unmount) + return () => owned() + } + $on(_event: string, listener: (namespace: string, revision: number) => void) { settingsListener = listener; return unsubscribe } + } + new Remote() + const setDefault = vi.fn(async () => true) + const setPersistent = vi.fn<() => Promise>>().mockResolvedValue({ ok: true, value: undefined }) + const runtimeConfiguration = vi.fn<() => Promise>>() + .mockResolvedValue({ ok: true, value: { persistent: false, writable: true } }) + const preset = createSnapshotStore({ status: 'ready' as const, value: { default: 'standard' }, + base: {}, user: {}, revision: 1, writable: true }) + ctx.provide('configForms', { get: (namespace: string) => { + expect(namespace).toBe('agent-preset-registry') + return { ...preset, set: setDefault } + } }) + const status = vi.fn<() => Promise>>().mockResolvedValue({ ok: true, value }) + const systemStatus = vi.fn<() => Promise>>().mockResolvedValue({ ok: true, value: value.system }) + const applicationIcons = vi.fn<() => Promise>>>().mockResolvedValue({ ok: true, value: {} }) + const requestPermission = vi.fn<() => Promise>>() + .mockResolvedValue({ ok: true, value: value.system }) + const openSettings = vi.fn<() => Promise>>().mockResolvedValue({ ok: true, value: undefined }) + const revoke = vi.fn<() => Promise>>().mockResolvedValue({ ok: true, value: undefined }) + const retryApproval = vi.fn<() => Promise>>().mockResolvedValue({ ok: true, value: undefined }) + ctx.provide('remote.cuaAccess', { + ptcConfiguration: runtimeConfiguration, enablePersistentPtc: setPersistent, + applications: status, systemStatus, applicationIcons, requestPermission, openSettings, revoke, retryApproval }) + ctx.provide('locale', new LocaleRuntime(ctx)) + await ctx.plugin(SlotRegistry) + ctx.slots.register({ name: 'root', children: { + 'plugins.bundle.activation': { kind: 'keyed', scope: 'root' }, + 'plugins.bundle.config': { kind: 'keyed', scope: 'root' }, 'shell.overlay': { kind: 'list', scope: 'root' }, + } } as never, () => null) + if (fail) vi.spyOn(ctx.slots, 'inject').mockImplementationOnce(() => { throw new Error('slot unavailable') }) + return { ctx, unmount, unsubscribe, status, systemStatus, applicationIcons, + requestPermission, openSettings, revoke, retryApproval, setDefault, setPersistent, runtimeConfiguration, + updateSettings(namespace: string) { settingsListener!(namespace, 1) }, + } +} + +it('withdraws the permission Remote and all controls with the optional plugin', async () => { + const f = await fixture() + try { + const fiber = f.ctx.plugin({ inject: [...inject], apply: ctx => mountCuaPermissions(ctx, contribution) }) + await fiber + const entry = f.ctx.slots.entries('plugins.bundle.config')[0]! + const actions = entry.inject!() + actionsOf(actions) + actions.refresh() + await vi.waitFor(() => { expect(actions.hooks.cuaPermissions.getSnapshot().applications.value).toEqual(value) }) + actions.request('accessibility') + await vi.waitFor(() => { expect(actions.hooks.cuaPermissions.getSnapshot().toast?.kind).toBe('requested') }) + actions.openSettings('screenRecording') + await vi.waitFor(() => { expect(actions.hooks.cuaPermissions.getSnapshot().toast?.kind).toBe('opened') }) + actions.revoke('a'.repeat(64) as import('../src/access-types.ts').CuaApplicationId) + await vi.waitFor(() => { expect(actions.hooks.cuaPermissions.getSnapshot().toast?.kind).toBe('revoked') }) + actions.retryApproval('a'.repeat(64) as import('../src/access-types.ts').CuaApplicationId) + await vi.waitFor(() => { expect(actions.hooks.cuaPermissions.getSnapshot().toast?.kind).toBe('retryAllowed') }) + const failed = { ok: false as const, error: new RemoteError('gateway/internal', 'offline', {}) } + for (const [target, invoke] of [ + [f.runtimeConfiguration, actions.refresh], [f.status, actions.refresh], [f.systemStatus, actions.refresh], + [f.applicationIcons, actions.refresh], [f.requestPermission, () => { actions.request('accessibility') }], + [f.openSettings, () => { actions.openSettings('accessibility') }], + [f.revoke, () => { actions.revoke('a'.repeat(64) as import('../src/access-types.ts').CuaApplicationId) }], + [f.retryApproval, () => { actions.retryApproval('a'.repeat(64) as import('../src/access-types.ts').CuaApplicationId) }], + ] as const) { + actions.refresh() + await vi.waitFor(() => { expect(actions.hooks.cuaPermissions.getSnapshot().busy).toBe(false) }) + target.mockResolvedValueOnce(failed) + invoke() + await vi.waitFor(() => { + const state = actions.hooks.cuaPermissions.getSnapshot() + const error = target === f.runtimeConfiguration ? state.runtime.error : target === f.status ? state.applications.error + : target === f.systemStatus ? state.system.error + : target === f.applicationIcons ? state.icons.error : state.toast?.kind === 'failed' ? state.toast.message : null + expect(error).toContain('offline') + }) + } + for (const name of ['plugins.bundle.activation', 'shell.overlay'] as const) { + const other = f.ctx.slots.entries(name)[0]! + const otherActions = other.inject!() + actionsOf(otherActions) + expect(otherActions.hooks.cuaPermissions).toBe(actions.hooks.cuaPermissions) + } + const settings = f.ctx.slots.entries('plugins.bundle.config')[0]!.inject!() + assert(typeof settings.setPtcDefault === 'function') + expect(f.ctx.slots.entries('plugins.bundle.activation')[0]!.inject!()).not.toHaveProperty('setPtcDefault') + expect(f.setDefault).not.toHaveBeenCalled() + await (settings.setPtcDefault as () => Promise)() + expect(f.setDefault).toHaveBeenCalledExactlyOnceWith('selectedDefault', 'ptc') + assert(typeof settings.setPersistentPtc === 'function') + expect(f.ctx.slots.entries('plugins.bundle.activation')[0]!.inject!()).not.toHaveProperty('setPersistentPtc') + expect(f.setPersistent).not.toHaveBeenCalled() + await (settings.setPersistentPtc as () => Promise)() + expect(f.setPersistent).toHaveBeenCalledExactlyOnceWith() + f.setPersistent.mockResolvedValueOnce(failed) + await expect((settings.setPersistentPtc as () => Promise)()).rejects.toThrow('offline') + await fiber.dispose() + expect(f.ctx.slots.entries('conversation.composer')).toHaveLength(0) + expect(f.ctx.slots.entries('plugins.bundle.config')).toHaveLength(0) + expect(f.ctx.slots.entries('shell.overlay')).toHaveLength(0) + expect(f.unsubscribe).toHaveBeenCalledOnce() + expect(f.unmount).toHaveBeenCalledOnce() + } finally { await f.ctx.fiber.dispose() } +}) + +it('withdraws the Remote contribution when UI registration fails', async () => { + const f = await fixture(true) + try { + await expect(mountCuaPermissions(f.ctx, contribution)).rejects.toThrow('slot unavailable') + expect(f.unmount).toHaveBeenCalledOnce() + } finally { await f.ctx.fiber.dispose() } +}) + +it.each(['plugin', 'root'])('keeps the Remote mounted until an accepted revoke completes during %s disposal', async (owner) => { + const f = await fixture() + const response = Promise.withResolvers>(), entered = Promise.withResolvers() + f.revoke.mockImplementationOnce(() => { entered.resolve(undefined); return response.promise }) + f.unmount.mockImplementation(async () => { response.resolve({ ok: false, error: new RemoteError('gateway/internal', 'unmounted', {}) }) }) + let closing: Promise | undefined + try { + const fiber = await f.ctx.plugin({ inject: [...inject], apply: ctx => mountCuaPermissions(ctx, contribution) }) + const actions = f.ctx.slots.entries('plugins.bundle.config')[0]!.inject!() + actionsOf(actions) + actions.revoke('a'.repeat(64) as import('../src/access-types.ts').CuaApplicationId) + await entered.promise + let closed = false + closing = Promise.resolve(owner === 'plugin' ? fiber.dispose() : f.ctx.fiber.dispose()).then(() => { closed = true }) + await setImmediate() + expect(f.unmount).not.toHaveBeenCalled() + expect(closed).toBe(false) + response.resolve({ ok: true, value: undefined }) + await closing + expect(f.unmount).toHaveBeenCalledOnce() + } finally { + response.resolve({ ok: true, value: undefined }) + await closing + await f.ctx.fiber.dispose() + } +}) + +it('refreshes only the settings namespace reported by the permission owner', async () => { + const f = await fixture() + f.status.mockResolvedValue({ ok: true, value: { ...value, settingsNamespace: 'custom-cua-entry' } }) + try { + await f.ctx.plugin({ inject: [...inject], apply: ctx => mountCuaPermissions(ctx, contribution) }) + const actions = f.ctx.slots.entries('plugins.bundle.config')[0]!.inject!() + actionsOf(actions) + f.updateSettings('locale') + expect(f.status).not.toHaveBeenCalled() + actions.refresh() + await vi.waitFor(() => { expect(actions.hooks.cuaPermissions.getSnapshot().busy).toBe(false) }) + expect(f.status).toHaveBeenCalledOnce() + f.updateSettings('cua') + f.updateSettings('locale') + expect(f.status).toHaveBeenCalledOnce() + f.updateSettings('custom-cua-entry') + await vi.waitFor(() => { expect(f.status).toHaveBeenCalledTimes(2) }) + } finally { await f.ctx.fiber.dispose() } +}) diff --git a/packages/experimental/cua/tests/panel.client.spec.tsx b/packages/experimental/cua/tests/panel.client.spec.tsx new file mode 100644 index 0000000000..6c2b06668f --- /dev/null +++ b/packages/experimental/cua/tests/panel.client.spec.tsx @@ -0,0 +1,376 @@ +// @vitest-environment jsdom +/** Permission UI distinguishes missing privileges, uncertainty, and application consent. */ +import { act, cleanup, fireEvent, render, screen, waitFor, within } from '@testing-library/react' +import { bindSnapshotSelector, makeTranslate } from '@deepseek-ai/dsh-client-test-runtime' +import { createSnapshotStore } from '@deepseek-ai/dsh-client-store' +import type { CuaApplicationId, CuaApplicationState, CuaSystemPermissionState } from '../src/access-types.ts' +import type { ConfigFormSnapshot } from '@deepseek-ai/dsh-client-ui-settings/client' +import { afterEach, expect, it, vi } from 'vitest' +import { PermissionPanel, PermissionSetup, PermissionToast, type PermissionPanelProps } from '../src/client/PermissionPanel.tsx' +import type { PermissionView } from '../src/client/controller.ts' +import { en, zh } from '../src/client/locales.ts' + +afterEach(() => { cleanup(); vi.useRealTimers() }) +const value: CuaApplicationState & { system: CuaSystemPermissionState } = { system: { accessibility: false, screenRecording: true, + helper: { pid: 9, path: '/native/helper' }, parent: { pid: 8, path: '/host/node' } }, +allowedApps: [], blockedApps: [], settingsNamespace: 'cua', persistentAvailable: true } + +function observe(value: CuaApplicationState & { system: CuaSystemPermissionState }) { + return { system: { value: value.system, busy: false, error: null }, + applications: { value, busy: false, error: null }, icons: { value: {}, busy: false, error: null }, busy: false } +} + +function fixture() { + const state = createSnapshotStore({ + runtime: { value: { persistent: false, writable: true }, busy: false, error: null }, + system: { value: null, busy: true, error: null }, applications: { value: null, busy: true, error: null }, + icons: { value: null, busy: true, error: null }, busy: true, toast: null, + }) + const preset = createSnapshotStore>({ + status: 'ready', value: { default: 'standard', selectedDefault: 'ptc' }, + base: {}, user: {}, revision: 1, writable: true, + }) + const refresh = vi.fn() + const props = { useCuaPermissions: bindSnapshotSelector(state), t: makeTranslate(en), + setPersistentPtc: vi.fn(async () => true), + useCuaDefaultPreset: bindSnapshotSelector(preset), setPtcDefault: vi.fn(async () => true), + refresh, request: vi.fn(), openSettings: vi.fn(), revoke: vi.fn(), + retryApproval: vi.fn(), dismissToast: vi.fn(), + } satisfies PermissionPanelProps + return { state, preset, props, + publish(patch: Partial) { act(() => { state.set({ ...state.getSnapshot(), ...patch }) }) }, + } +} + +it('requests only a missing privilege and leaves OS attribution to the system prompt', () => { + const f = fixture() + const shown = render() + expect(f.props.refresh).toHaveBeenCalledOnce() + f.publish({ ...observe(value) }) + fireEvent.click(screen.getByRole('button', { name: en.request })) + expect(f.props.request).toHaveBeenCalledWith('accessibility') + expect(screen.queryByRole('button', { name: en.settings })).toBeNull() + const allowed = screen.getByRole('button', { name: en.granted }) + expect(allowed.getAttribute('title')).toBe(en.settings) + fireEvent.click(allowed) + expect(f.props.openSettings.mock.calls).toEqual([['screenRecording']]) + fireEvent.focus(window) + expect(f.props.refresh).toHaveBeenCalledTimes(2) + shown.unmount() + fireEvent.focus(window) + expect(f.props.refresh).toHaveBeenCalledTimes(2) +}) + +it.each([en, zh])('changes the default from the detail page and retains a refused write for retry', async (dictionary) => { + const f = fixture() + f.preset.set({ ...f.preset.getSnapshot(), value: { default: 'standard' } }) + f.publish({ ...observe(value) }) + render() + const group = screen.getByRole('group', { name: dictionary.defaultTitle }) + expect(within(group).getByText(dictionary.defaultBody)).toBeTruthy() + expect(f.props.setPtcDefault).not.toHaveBeenCalled() + const write = Promise.withResolvers() + f.props.setPtcDefault.mockReturnValueOnce(write.promise) + fireEvent.click(within(group).getByRole('button', { name: dictionary.defaultConfirm })) + expect(within(group).getByRole('button', { name: dictionary.defaultConfirm }).hasAttribute('disabled')).toBe(true) + await act(async () => { write.resolve(false); await write.promise }) + expect(screen.getByRole('alert').textContent).toBe(dictionary.defaultNotSaved) + fireEvent.click(within(group).getByRole('button', { name: dictionary.defaultConfirm })) + await waitFor(() => { expect(screen.queryByRole('alert')).toBeNull() }) + expect(f.props.setPtcDefault).toHaveBeenCalledTimes(2) + act(() => { f.preset.set({ ...f.preset.getSnapshot(), value: { default: 'standard', selectedDefault: 'ptc' } }) }) + expect(within(group).getByRole('button', { name: dictionary.defaultSet }).hasAttribute('disabled')).toBe(true) +}) + +it('dismisses activation when permissions are ready without changing the default', () => { + const f = fixture(), onDismiss = vi.fn() + f.preset.set({ ...f.preset.getSnapshot(), value: { default: 'standard', selectedDefault: 'minimal' } }) + f.publish({ ...observe({ ...value, system: { ...value.system, accessibility: true } }) }) + render() + expect(onDismiss).toHaveBeenCalledOnce() + expect(screen.queryByRole('dialog')).toBeNull() + expect(f.props.setPtcDefault).not.toHaveBeenCalled() +}) + +it.each([new Error('offline'), 'offline'])('reports a failed default-mode request on the detail page', async (error) => { + const f = fixture() + f.preset.set({ ...f.preset.getSnapshot(), value: { default: 'standard' } }) + f.props.setPtcDefault.mockRejectedValueOnce(error) + render() + fireEvent.click(screen.getByRole('button', { name: en.defaultConfirm })) + await waitFor(() => { expect(screen.getByRole('alert').textContent).toContain('offline') }) +}) + +it('waits for the settings document and recognizes PTC as the deployment default', () => { + const f = fixture() + f.preset.set({ ...f.preset.getSnapshot(), status: 'loading', value: undefined }) + render() + expect(screen.queryByRole('button', { name: en.defaultConfirm })).toBeNull() + act(() => { f.preset.set({ ...f.preset.getSnapshot(), status: 'ready', value: { default: 'ptc' } }) }) + expect(screen.getByRole('button', { name: en.defaultSet }).hasAttribute('disabled')).toBe(true) + expect(f.props.setPtcDefault).not.toHaveBeenCalled() +}) + +it.each(['unavailable', 'ready'] as const)('explains why the %s settings document cannot change the default', (status) => { + const f = fixture() + f.preset.set({ ...f.preset.getSnapshot(), status, writable: false, + value: status === 'ready' ? { default: 'standard' } : undefined }) + render() + expect(screen.getByRole('alert').textContent).toBe(status === 'ready' ? en.defaultReadOnly : en.defaultUnavailable) + expect(screen.getByRole('button', { name: en.defaultConfirm }).hasAttribute('disabled')).toBe(true) + expect(f.props.setPtcDefault).not.toHaveBeenCalled() +}) + +it('shows query failures as unknown while retaining previously displayed apps', () => { + const f = fixture(), app = { id: 'a'.repeat(64) as CuaApplicationId, name: 'Fixture', executablePath: '/fixture/app' } + render() + f.publish({ ...observe({ ...value, allowedApps: [app] }), system: { value: value.system, busy: false, error: 'offline' }, busy: false }) + expect(screen.getAllByRole('button', { name: en.revoke })).toHaveLength(1) + expect(screen.getAllByText(en.unknown)).toHaveLength(2) + expect(screen.queryByText(en.granted)).toBeNull() + expect(screen.getByRole('alert').textContent).toContain('offline') + fireEvent.click(screen.getAllByRole('button', { name: en.revoke })[0]!) + expect(f.props.revoke).toHaveBeenCalledWith(app.id) + fireEvent.click(screen.getByRole('button', { name: en.retry })) + expect(f.props.refresh).toHaveBeenCalledTimes(2) + f.publish({ system: { ...f.state.getSnapshot().system, busy: true }, busy: true }) + expect(screen.queryByRole('button', { name: en.retry })).toBeNull() + f.publish({ system: { ...f.state.getSnapshot().system, busy: false, error: null }, busy: false }) + expect(screen.queryByRole('button', { name: en.retry })).toBeNull() + f.publish(observe(value)) + expect(screen.getByText(en.noApps)).toBeTruthy() + expect(screen.queryByRole('button', { name: en.revoke })).toBeNull() +}) + +it('keeps unknown initial status distinct from a confirmed missing privilege in Chinese', () => { + const f = fixture() + render() + f.publish({ system: { ...f.state.getSnapshot().system, busy: false, error: 'connection lost' }, busy: false }) + expect(screen.queryByText(zh.granted)).toBeNull() + expect(screen.queryByRole('button', { name: zh.request })).toBeNull() + f.publish({ ...observe({ ...value, system: { ...value.system, accessibility: true, screenRecording: false } }) }) + fireEvent.click(screen.getByRole('button', { name: zh.request })) + expect(f.props.request).toHaveBeenCalledWith('screenRecording') +}) + +it('offers setup after inspection, permits dismissal, and dismisses once both privileges are observed', () => { + const f = fixture(), onDismiss = vi.fn(), onOpenDetails = vi.fn() + render() + expect(screen.queryByRole('dialog')).toBeNull() + f.publish({ ...observe(value) }) + const dialog = screen.getByRole('dialog', { name: en.setupTitle }) + fireEvent.click(within(dialog).getByRole('button', { name: en.openSetup })) + expect(onOpenDetails).toHaveBeenCalledOnce() + fireEvent.click(within(dialog).getByRole('button', { name: en.later })) + expect(onDismiss).toHaveBeenCalledOnce() + f.publish({ ...observe({ ...value, system: { ...value.system, accessibility: true } }) }) + expect(screen.queryByRole('dialog')).toBeNull() + expect(onDismiss).toHaveBeenCalledTimes(2) +}) + +it('offers the details page when permission inspection fails', () => { + const f = fixture() + render() + f.publish({ system: { ...f.state.getSnapshot().system, busy: false, error: 'offline' }, busy: false }) + expect(screen.getByRole('dialog')).toBeTruthy() + expect(screen.queryByRole('alert')).toBeNull() + expect(screen.getByText(en.setupQueryFailed)).toBeTruthy() + expect(screen.queryByText(en.setupBody)).toBeNull() +}) + +it('exposes a new-request action for a blocked application without calling grant or revoke', () => { + const f = fixture(), app = { id: 'a'.repeat(64) as CuaApplicationId, name: 'Fixture', executablePath: '/fixture/app' } + render() + f.publish({ ...observe({ ...value, blockedApps: [app] }) }) + expect(screen.queryByText(en.noApps)).toBeNull() + fireEvent.click(screen.getByRole('button', { name: en.retryApproval })) + expect(f.props.retryApproval).toHaveBeenCalledWith(app.id) + expect(f.props.revoke).not.toHaveBeenCalled() + expect(f.props.request).not.toHaveBeenCalled() +}) + +it('renders action outcomes outside the permission page and clears finished toasts', async () => { + vi.useFakeTimers() + const f = fixture() + render() + expect(screen.queryByRole('alert')).toBeNull() + f.publish({ toast: { id: 1, kind: 'revoked' } }) + expect(screen.getByRole('alert').textContent).toBe(en.revoked) + await act(async () => { await vi.advanceTimersByTimeAsync(4000) }) + expect(f.props.dismissToast).toHaveBeenCalledOnce() + f.publish({ toast: { id: 2, kind: 'failed', message: 'write refused' } }) + expect(screen.getByRole('alert').textContent).toContain('write refused') +}) + +it('keeps permission guidance open while a fresh observation is pending', () => { + const f = fixture() + render() + f.publish({ ...observe(value) }) + const dialog = screen.getByRole('dialog') + f.publish({ system: { ...f.state.getSnapshot().system, busy: true }, busy: true }) + expect(screen.getByRole('dialog')).toBe(dialog) +}) + +it('shows application icons and names while keeping the executable path in the tooltip', () => { + const f = fixture() + render() + f.publish({ ...observe({ ...value, allowedApps: [{ id: 'a'.repeat(64) as CuaApplicationId, + name: 'Calculator', executablePath: '/Applications/Calculator.app/Contents/MacOS/Calculator' }] }), + icons: { value: { '/Applications/Calculator.app/Contents/MacOS/Calculator': 'data:image/png;base64,aWNvbg==' }, busy: false, error: null }, busy: false }) + expect(screen.getByText('Calculator')).toBeTruthy() + expect(screen.queryByText('/Applications/Calculator.app/Contents/MacOS/Calculator')).toBeNull() + const identity = screen.getByTitle('/Applications/Calculator.app/Contents/MacOS/Calculator') + expect(identity.querySelector('img')?.getAttribute('src')).toBe('data:image/png;base64,aWNvbg==') +}) + +it.each([en, zh])('shows four requirements with the target application beside each system permission', (dictionary) => { + const f = fixture() + render() + const application = { name: 'Visual Studio Code.app', path: '/Applications/Visual Studio Code.app', icon: 'data:image/png;base64,aWNvbg==' } + f.publish({ ...observe({ ...value, system: { ...value.system, parent: { pid: 1, path: '/sbin/launchd' }, launchApplication: application } }) }) + const requirements = screen.getByRole('region', { name: dictionary.title }) + expect(within(requirements).getAllByRole('group')).toHaveLength(4) + expect(screen.getByRole('region', { name: dictionary.applicationsTitle })).toBeTruthy() + const accessibility = within(requirements).getByRole('group', { name: dictionary.accessibility }) + expect(within(accessibility).getByText(application.name)).toBeTruthy() + expect(accessibility.textContent).toContain(dictionary.accessibilityPermission.replace('{application}', application.name)) + expect(accessibility.querySelector('img')?.getAttribute('src')).toBe(application.icon) + const recording = within(requirements).getByRole('group', { name: dictionary.screenRecording }) + expect(within(recording).getByText(application.name)).toBeTruthy() + expect(recording.textContent).toContain(dictionary.screenRecordingPermission.replace('{application}', application.name)) + expect(within(requirements).getByText(dictionary.permissionHint)).toBeTruthy() + f.publish({ system: { ...f.state.getSnapshot().system, error: 'offline' } }) + expect(screen.queryByText(application.name)).toBeNull() + expect(within(requirements).getByText(dictionary.permissionHintUnknown)).toBeTruthy() +}) + +it('names the authorization target even when its icon is unavailable', () => { + const f = fixture() + render() + f.publish({ ...observe({ ...value, system: { ...value.system, + launchApplication: { name: 'iTerm.app', path: '/Applications/iTerm.app' } } }) }) + const accessibility = screen.getByRole('group', { name: en.accessibility }) + expect(within(accessibility).getByText('iTerm.app')).toBeTruthy() + expect(accessibility.querySelector('img')).toBeNull() +}) + +it('waits for the first current observation instead of opening setup with a stale error', () => { + const f = fixture() + render() + f.publish({ system: { ...f.state.getSnapshot().system, busy: true, error: 'provider is reloading' }, busy: true }) + expect(screen.queryByRole('dialog')).toBeNull() + f.publish({ ...observe(value) }) + expect(screen.getByRole('dialog', { name: en.setupTitle })).toBeTruthy() + expect(screen.queryByRole('alert')).toBeNull() +}) + + +it('omits setup instructions once both system permissions are granted', () => { + const f = fixture() + render() + f.publish({ ...observe({ ...value, system: { ...value.system, accessibility: true, screenRecording: true } }) }) + expect(screen.getAllByText(en.granted)).toHaveLength(2) + expect(screen.queryByText(en.permissionHintUnknown)).toBeNull() +}) + + +it('keeps all four requirements visible before the first permission response', () => { + const f = fixture() + render() + const requirements = screen.getByRole('region', { name: en.title }) + expect(within(requirements).getAllByRole('group')).toHaveLength(4) + expect(screen.queryByRole('alert')).toBeNull() + expect(screen.queryByRole('button', { name: en.request })).toBeNull() + for (const permission of [en.accessibility, en.screenRecording]) expect(within(screen.getByRole('group', { name: permission })).queryByRole('button')).toBeNull() +}) + +it('fills application names, system permissions, and icons as their observations arrive', () => { + const f = fixture() + render() + const requirements = screen.getByRole('region', { name: en.title }) + const access = screen.getByRole('region', { name: en.applicationsTitle }) + const app = { id: 'a'.repeat(64) as CuaApplicationId, name: 'Calculator', executablePath: '/Applications/Calculator.app' } + f.publish({ applications: { value: { ...value, allowedApps: [app] }, busy: false, error: null } }) + expect(within(access).getByText(app.name)).toBeTruthy() + expect(within(access).getByRole('button', { name: en.revoke }).hasAttribute('disabled')).toBe(false) + expect(screen.queryByText(en.granted)).toBeNull() + const identity = within(access).getByTitle(app.executablePath) + expect(identity.querySelector('img')).toBeNull() + const placeholder = identity.querySelector('[aria-hidden]')! + expect(placeholder).toBeTruthy() + f.publish({ system: { value: value.system, busy: false, error: null } }) + expect(within(requirements).getByText(en.granted)).toBeTruthy() + expect(within(requirements).getByRole('button', { name: en.request }).hasAttribute('disabled')).toBe(false) + expect(within(access).getByTitle(app.executablePath)).toBe(identity) + expect(identity.querySelector('[aria-hidden]')).toBe(placeholder) + f.publish({ icons: { value: { [app.executablePath]: 'data:image/png;base64,aWNvbg==' }, busy: false, error: null } }) + expect(identity.querySelector('img')?.getAttribute('src')).toBe('data:image/png;base64,aWNvbg==') + expect(identity.querySelector('[aria-hidden]')).toBeNull() + expect(screen.getByRole('region', { name: en.title })).toBe(requirements) +}) + +it.each(['applications', 'icons'] as const)('contains a failed %s query in the application section', (key) => { + const f = fixture() + f.publish({ ...observe(value), [key]: { value: null, busy: false, error: 'offline' } }) + render() + const requirements = screen.getByRole('region', { name: en.title }) + const access = screen.getByRole('region', { name: en.applicationsTitle }) + expect(within(requirements).getByText(en.granted)).toBeTruthy() + expect(within(requirements).queryByRole('alert')).toBeNull() + expect(within(access).getByRole('alert').textContent).toContain('offline') + fireEvent.click(within(access).getByRole('button', { name: en.retry })) + expect(f.props.refresh).toHaveBeenCalledTimes(2) +}) + +it('finishes activation guidance without waiting for application icons', () => { + const f = fixture(), onDismiss = vi.fn() + render() + f.publish({ system: { value: { ...value.system, accessibility: true }, busy: false, error: null } }) + expect(onDismiss).toHaveBeenCalledOnce() + expect(screen.queryByRole('dialog')).toBeNull() + expect(f.state.getSnapshot().icons.busy).toBe(true) +}) + +it.each([en, zh])('saves persistent PTC only from its own row and retains failures for retry', async (dictionary) => { + const f = fixture() + render() + const row = screen.getByRole('group', { name: dictionary.persistentTitle }) + expect(f.props.setPersistentPtc).not.toHaveBeenCalled() + const write = Promise.withResolvers() + f.props.setPersistentPtc.mockReturnValueOnce(write.promise) + fireEvent.click(within(row).getByRole('button', { name: dictionary.persistentConfirm })) + expect(within(row).getByRole('button').hasAttribute('disabled')).toBe(true) + await act(async () => { write.resolve(false); await write.promise }) + expect(within(row).getByRole('alert').textContent).toBe(dictionary.persistentNotSaved) + fireEvent.click(within(row).getByRole('button')) + await waitFor(() => { expect(within(row).queryByRole('alert')).toBeNull() }) + f.publish({ runtime: { value: { persistent: true, writable: true }, busy: false, error: null } }) + expect(within(row).getByRole('button', { name: dictionary.persistentSet }).hasAttribute('disabled')).toBe(true) + expect(f.props.setPtcDefault).not.toHaveBeenCalled() +}) + +it('loads the runtime setting independently and reports unavailable and read-only configurations', () => { + const f = fixture() + f.publish({ runtime: { value: null, busy: true, error: null } }) + render() + const row = screen.getByRole('group', { name: en.persistentTitle }) + expect(within(row).queryByRole('button')).toBeNull() + expect(screen.getByRole('button', { name: en.defaultSet })).toBeTruthy() + f.publish({ runtime: { value: null, busy: false, error: 'offline' } }) + expect(within(row).getByRole('alert').textContent).toBe(en.persistentUnavailable) + f.publish({ runtime: { value: { persistent: false, writable: false }, busy: false, error: null } }) + expect(within(row).getByRole('alert').textContent).toBe(en.persistentReadOnly) + expect(within(row).getByRole('button').hasAttribute('disabled')).toBe(true) + expect(f.props.setPersistentPtc).not.toHaveBeenCalled() +}) + + +it('retains the previous single control while a stale query error is being refreshed', () => { + const f = fixture() + f.publish(observe(value)) + render() + f.publish({ system: { value: value.system, busy: true, error: 'stale error' }, busy: true }) + expect(screen.getByRole('button', { name: en.request }).hasAttribute('disabled')).toBe(true) + expect(screen.getByRole('button', { name: en.granted }).hasAttribute('disabled')).toBe(true) + expect(screen.queryByRole('alert')).toBeNull() +}) diff --git a/packages/experimental/cua/tsconfig.client.json b/packages/experimental/cua/tsconfig.client.json new file mode 100644 index 0000000000..03db4207e8 --- /dev/null +++ b/packages/experimental/cua/tsconfig.client.json @@ -0,0 +1,56 @@ +{ + "extends": "../../../tsconfig.base.client.json", + "compilerOptions": { + "rootDir": "src", + "outDir": "lib/types", + "tsBuildInfoFile": "lib/tsconfig.client.tsbuildinfo" + }, + "include": [ + "src/client", + "src/css-modules.d.ts", + "src/access-types.ts", + "src/application-types.ts", + "src/remote-types.ts" + ], + "references": [ + { + "path": "../../../vendor/cordis" + }, + { + "path": "../../client/locale/tsconfig.client.json" + }, + { + "path": "../../client/store" + }, + { + "path": "../../client/ui-primitives" + }, + { + "path": "../../client/ui-renderer" + }, + { + "path": "../../client/ui-slots" + }, + { + "path": "../../client/ui-plugin-manager/tsconfig.client.json" + }, + { + "path": "../../client/ui-settings/tsconfig.client.json" + }, + { + "path": "../../client/ui-layout" + }, + { + "path": "../../api/remotes/tsconfig.client.json" + }, + { + "path": "../../api/gateway/tsconfig.client.json" + }, + { + "path": "../../typert/protocol" + }, + { + "path": "../../util/brand" + } + ] +} diff --git a/packages/experimental/cua/tsconfig.host.json b/packages/experimental/cua/tsconfig.host.json new file mode 100644 index 0000000000..ebeaed4918 --- /dev/null +++ b/packages/experimental/cua/tsconfig.host.json @@ -0,0 +1,73 @@ +{ + "extends": "../../../tsconfig.base.json", + "compilerOptions": { + "rootDir": "src", + "outDir": "lib/types", + "tsBuildInfoFile": "lib/tsconfig.host.tsbuildinfo" + }, + "include": [ + "src/*.ts" + ], + "references": [ + { + "path": "../../../vendor/cordis" + }, + { + "path": "../../../vendor/schemastery" + }, + { + "path": "../../computer-use/computer-use" + }, + { + "path": "../../core/tools" + }, + { + "path": "../../core/system-prompt" + }, + { + "path": "../../core/agent" + }, + { + "path": "../../subprocess/subprocess" + }, + { + "path": "../../util/values" + }, + { + "path": "../../util/brand" + }, + { + "path": "../../util/atomic-write" + }, + { + "path": "../../ptc-runtime/ptc-runtime" + }, + { + "path": "../../typert/protocol" + }, + { + "path": "../../settings/settings" + }, + { + "path": "../../interaction/user-questions" + }, + { + "path": "../../../vendor/loader" + }, + { + "path": "../../attachment/attachment" + }, + { + "path": "../../llm/llm" + }, + { + "path": "../../core/scope" + }, + { + "path": "../../preset/agent-preset-registry" + } + ], + "exclude": [ + "src/css-modules.d.ts" + ] +} diff --git a/packages/experimental/cua/tsconfig.json b/packages/experimental/cua/tsconfig.json index a73c2cfb2e..2eca820546 100644 --- a/packages/experimental/cua/tsconfig.json +++ b/packages/experimental/cua/tsconfig.json @@ -1,60 +1,11 @@ { - "extends": "../../../tsconfig.base.json", - "compilerOptions": { - "rootDir": "src", - "outDir": "lib/types" - }, - "include": [ - "src" - ], + "files": [], "references": [ { - "path": "../../../vendor/cordis" + "path": "./tsconfig.host.json" }, { - "path": "../../../vendor/schemastery" - }, - { - "path": "../../computer-use/computer-use" - }, - { - "path": "../../core/tools" - }, - { - "path": "../../core/system-prompt" - }, - { - "path": "../../core/agent" - }, - { - "path": "../../subprocess/subprocess" - }, - { - "path": "../../util/values" - }, - { - "path": "../../util/brand" - }, - { - "path": "../../util/atomic-write" - }, - { - "path": "../../ptc-runtime/ptc-runtime" - }, - { - "path": "../../settings/settings" - }, - { - "path": "../../interaction/user-questions" - }, - { - "path": "../../../vendor/loader" - }, - { - "path": "../../attachment/attachment" - }, - { - "path": "../../llm/llm" + "path": "./tsconfig.client.json" } ] } diff --git a/packages/experimental/cua/tsdown.config.ts b/packages/experimental/cua/tsdown.config.ts index ccc53ca43d..5dac622abb 100644 --- a/packages/experimental/cua/tsdown.config.ts +++ b/packages/experimental/cua/tsdown.config.ts @@ -1,10 +1,13 @@ -import { defineConfig } from 'tsdown' +import type { UserConfig } from 'tsdown' +import { clientBundle } from '../../client/tsdown.client.ts' -export default defineConfig(Object.entries({ - index: 'lib/types/index.js', api: 'lib/types/api.js', bridge: 'lib/types/bridge.js', +const runtime: UserConfig[] = Object.entries({ + 'remote-types': 'lib/types/remote-types.js', api: 'lib/types/api.js', bridge: 'lib/types/bridge.js', 'native-client': 'lib/types/native-client.js', protocol: 'lib/types/protocol.js', }).map(([name, path]) => ({ entry: { [name]: path }, outDir: 'lib', format: ['esm'], platform: 'node', target: 'es2024', fixedExtension: false, dts: false, clean: false, -}))) +})) + +export default clientBundle(import.meta.dirname, ['lib/types/index.js'], { hostPhase: true, companions: runtime }) diff --git a/packages/extensions/cordis-client-runner/src/client/slot-catalog.ts b/packages/extensions/cordis-client-runner/src/client/slot-catalog.ts index 8998562805..aacb4a45e7 100644 --- a/packages/extensions/cordis-client-runner/src/client/slot-catalog.ts +++ b/packages/extensions/cordis-client-runner/src/client/slot-catalog.ts @@ -2281,12 +2281,13 @@ export const CLIENT_SLOT_API: readonly ClientSlotEntry[] = [ 'useSessionAccess: (sessionId: SessionId, action: RemoteEndpoint) => boolean', 'useWorkspaceAccess: (workspaceId: WorkspaceId, action: RemoteEndpoint) => boolean', ], - keyDomain: 'open: any string the owner dispatches (no compile-time key set), already taken: @deepseek-ai/dsh-experimental-voice-input-bundle', + keyDomain: 'open: any string the owner dispatches (no compile-time key set), already taken: @deepseek-ai/dsh-cua-bundle, @deepseek-ai/dsh-experimental-voice-input-bundle', hookContext: '', slotInject: '', declaredBy: 'an entry in \'main\' (client-ui-plugin-manager), so it exists while that entry is mounted', occupants: [ 'experimental-client-ui-voice-input VoiceSetupPrompt key \'@deepseek-ai/dsh-experimental-voice-input-bundle\'', + 'cua PermissionSetup key \'@deepseek-ai/dsh-cua-bundle\'', ], replaceRisk: 'shadows-shipped-ui', example: 'return {\n inject: [\'slots\'],\n apply(ctx) {\n ctx.slots.inject(\'plugins.bundle.activation\', () => ctx.slots.register(\n { name: \'plugins.bundle.activation\', key: \'\' },\n () => React.createElement(\'div\', null, \'hello\'),\n ))\n },\n}', @@ -2323,13 +2324,14 @@ export const CLIENT_SLOT_API: readonly ClientSlotEntry[] = [ 'useSessionAccess: (sessionId: SessionId, action: RemoteEndpoint) => boolean', 'useWorkspaceAccess: (workspaceId: WorkspaceId, action: RemoteEndpoint) => boolean', ], - keyDomain: 'open: any string the owner dispatches (no compile-time key set), already taken: @deepseek-ai/dsh-experimental-cot-translation-bundle, @deepseek-ai/dsh-experimental-voice-input-bundle', + keyDomain: 'open: any string the owner dispatches (no compile-time key set), already taken: @deepseek-ai/dsh-cua-bundle, @deepseek-ai/dsh-experimental-cot-translation-bundle, @deepseek-ai/dsh-experimental-voice-input-bundle', hookContext: '', slotInject: '', declaredBy: 'an entry in \'main\' (client-ui-plugin-manager), so it exists while that entry is mounted', occupants: [ 'experimental-client-ui-cot-translation TranslationSettings key \'@deepseek-ai/dsh-experimental-cot-translation-bundle\'', 'experimental-client-ui-voice-input VoicePreparation key \'@deepseek-ai/dsh-experimental-voice-input-bundle\'', + 'cua PermissionPanel key \'@deepseek-ai/dsh-cua-bundle\'', ], replaceRisk: 'shadows-shipped-ui', example: 'return {\n inject: [\'slots\'],\n apply(ctx) {\n ctx.slots.inject(\'plugins.bundle.config\', () => ctx.slots.register(\n { name: \'plugins.bundle.config\', key: \'\' },\n () => React.createElement(\'div\', null, \'hello\'),\n ))\n },\n}', @@ -3399,6 +3401,7 @@ export const CLIENT_SLOT_API: readonly ClientSlotEntry[] = [ 'client-ui-workspace SessionRenameDialog id \'workspace.session-rename\'', 'client-ui-workspace SessionArchiveConfirmDialog id \'workspace.session-archive\'', 'client-ui-workspace RowActionToast id \'workspace.row-toast\'', + 'cua PermissionToast id \'cua-permission-outcome\'', ], replaceRisk: 'none', example: 'return {\n inject: [\'slots\'],\n apply(ctx) {\n ctx.slots.inject(\'shell.overlay\', () => ctx.slots.register(\n { name: \'shell.overlay\', id: \'my-entry\', order: 100, label: \'My entry\' },\n () => React.createElement(\'div\', null, \'hello\'),\n ))\n },\n}', diff --git a/packages/extensions/tool-cordis/src/api-catalog.ts b/packages/extensions/tool-cordis/src/api-catalog.ts index 6aa344b3b8..e2541c5eb6 100644 --- a/packages/extensions/tool-cordis/src/api-catalog.ts +++ b/packages/extensions/tool-cordis/src/api-catalog.ts @@ -922,6 +922,67 @@ export const SERVICE_API: readonly ServiceApiEntry[] = [ }, ], }, + { + key: 'cuaAccess', + summary: 'Browser controls for provider-owned native permission and application decisions.', + description: 'Browser controls for provider-owned native permission and application decisions.', + methods: [ + { + signature: '@Remote ptcConfiguration(): { persistent: boolean; writable: boolean }', + description: 'Read PTC execution state separately from native permission checks.', + parameters: [], + returns: 'Whether the shared runtime retains memory and has an editable profile entry.', + }, + { + signature: '@Remote async enablePersistentPtc(): Promise', + description: 'Persist in-memory PTC execution through the existing profile editor and Loader lifecycle.', + parameters: [], + returns: 'Completion after the shared runtime reload; already-persistent runtimes are left intact.', + }, + { + signature: '@Remote applications(): CuaApplicationState', + description: 'Read application decisions without waiting for the native helper.', + parameters: [], + returns: 'Saved grants and refused applications with their persistence owner.', + }, + { + signature: '@Remote async systemStatus(): Promise', + description: 'Inspect system privileges independently of application decisions and icons.', + parameters: [], + returns: 'Fresh native preflight and launch metadata.', + }, + { + signature: '@Remote async applicationIcons(): Promise>', + description: 'Load presentation icons separately from application decisions.', + parameters: [], + returns: 'Optional native PNG icons keyed by executable path for saved and blocked applications.', + }, + { + signature: '@Remote async requestPermission(permission: CuaSystemPermission): Promise', + description: 'Request a system privilege explicitly selected by the user.', + parameters: [{ name: 'permission', description: 'macOS privilege to request.' }], + returns: 'Preflight state after requesting; a false value never implies consent.', + }, + { + signature: '@Remote async openSettings(permission: CuaSystemPermission): Promise', + description: 'Open the macOS privacy pane selected by the user.', + parameters: [{ name: 'permission', description: 'Corresponding system privilege.' }], + returns: 'Completion after macOS accepted the open request.', + }, + { + signature: '@Remote revoke(id: CuaApplicationId): Promise', + description: 'Remove saved and session access and join affected operations.', + parameters: [{ name: 'id', description: 'Application identity displayed on the permission page.' }], + returns: 'Completion after persistence and cancellation; dispatched input cannot be undone.', + }, + { + signature: '@Remote async retryApproval(id: CuaApplicationId): Promise', + description: 'Clear this application\'s refusals across sessions so each session may ask for access again.', + parameters: [{ name: 'id', description: 'Refused or revoked application displayed on the permission page.' }], + returns: 'Completion after clearing refusals; no permission is granted and no action is replayed.', + }, + ], + }, { key: 'deepseekAccount', summary: 'Account operations; only Host consumers can obtain a request credential.', @@ -5400,6 +5461,30 @@ export const TYPE_API: readonly TypeApiEntry[] = [ name: 'CronScheduleRecord', declaration: 'export interface CronScheduleRecord {\n readonly id: ScheduleId;\n readonly kind: \'cron\';\n readonly title: string;\n readonly prompt: string;\n readonly expression: string;\n readonly timeZone: string;\n readonly scheduledAt: string;\n}', }, + { + name: 'CuaApplicationGrant', + declaration: 'export interface CuaApplicationGrant {\n id: CuaApplicationId;\n name: string;\n executablePath: string;\n}', + }, + { + name: 'CuaApplicationId', + declaration: 'export type CuaApplicationId = Branded<\'CuaApplicationId\'>;', + }, + { + name: 'CuaApplicationState', + declaration: 'export interface CuaApplicationState {\n settingsNamespace: string | null;\n allowedApps: readonly CuaApplicationGrant[];\n blockedApps: readonly CuaApplicationGrant[];\n persistentAvailable: boolean;\n}', + }, + { + name: 'CuaPermissionProcess', + declaration: 'export interface CuaPermissionProcess {\n pid: number;\n path?: string | undefined;\n name?: string | undefined;\n}', + }, + { + name: 'CuaSystemPermission', + declaration: 'export type CuaSystemPermission = \'accessibility\' | \'screenRecording\';', + }, + { + name: 'CuaSystemPermissionState', + declaration: 'export interface CuaSystemPermissionState {\n accessibility: boolean;\n screenRecording: boolean;\n helper: CuaPermissionProcess;\n parent: CuaPermissionProcess;\n launchApplication?: {\n name: string;\n path: string;\n icon?: string | undefined;\n } | undefined;\n}', + }, { name: 'DailyInput', declaration: 'export interface DailyInput {\n readonly time: string;\n readonly time_zone: string;\n}', diff --git a/packages/preset/agent-preset/skills/cordis-composition-reference/references/packages.md b/packages/preset/agent-preset/skills/cordis-composition-reference/references/packages.md index cd38b17726..f6690231e2 100644 --- a/packages/preset/agent-preset/skills/cordis-composition-reference/references/packages.md +++ b/packages/preset/agent-preset/skills/cordis-composition-reference/references/packages.md @@ -184,6 +184,7 @@ Every package below exports a Cordis plugin that a bundle patch can name in a Lo | Package | Config | Description | |---|---|---| +| `@deepseek-ai/dsh-cua` | no | Shared Computer Use application objects, authorization, and logged tool execution | | `@deepseek-ai/dsh-experimental-agent-team` | yes | Implicit-root Agent Teams roster, direct peer messaging, and shared task DAG | | `@deepseek-ai/dsh-experimental-api-speech-to-text` | yes | Authenticated experimental speech transcription for browser clients | | `@deepseek-ai/dsh-experimental-auto-review` | no | Per-tool LLM authorization review for the DeepSeek Harness Auto permission preset | diff --git a/packages/skill/skill-office/tests/render-tools.e2e.ts b/packages/skill/skill-office/tests/render-tools.e2e.ts index f2dc0cfbf2..126290b3c2 100644 --- a/packages/skill/skill-office/tests/render-tools.e2e.ts +++ b/packages/skill/skill-office/tests/render-tools.e2e.ts @@ -37,17 +37,22 @@ async function sessionEvents(cwd: string) { } it.skipIf(!process.env.DEEPSEEK_API_KEY || !runtime).each([{ format: 'docx', mode: 'native' }, { format: 'pdf', mode: 'native' }, { format: 'docx', mode: 'ptc' }])('loads the operation skill and renders $format through the shell in $mode mode', async ({ format, mode }) => { + let original: Buffer | undefined await runLoaderSmoke({ label: 'Office CLI rendering', tempDirPrefix: 'dsh-office-render-', ...launch, env: { ...launch.env, DSH_TOOLS_MODE: mode }, binArgs: ['--profile', 'headless', '--patch', 'office.patch.json', - `Export page 1 of input.${format} as PNG into preview and check that the exported image is readable. Report briefly in English.`], + // Source-content verification can require an authoring skill; this case covers image-only inspection. + `Export page 1 of input.${format} as PNG into preview. Check only that the exported image is readable; ` + + 'do not inspect the source document content or metadata, and keep the source file unchanged. Report briefly in English.'], async prepare(cwd) { await execa(python!, ['-c', "from docx import Document\nd=Document()\nd.add_heading('Quarterly revenue',0)\nd.add_paragraph('Q1: 12; Q2: 18; Total: 30')\nd.save('input.docx')"], { cwd }) if (format === 'pdf') await execa(python!, ['-c', "from PIL import Image, ImageDraw\nim=Image.new('RGB',(300,200),'white')\nImageDraw.Draw(im).rectangle((30,30,150,100),fill='black')\nim.save('input.pdf','PDF')"], { cwd }) await prepareOffice(cwd) + original = await readFile(join(cwd, `input.${format}`)) }, async inspect(cwd) { + expect(await readFile(join(cwd, `input.${format}`))).toEqual(original) const events = await sessionEvents(cwd) const calls = events.flatMap((event) => { if (event.type === 'tool/call') return [event.data] @@ -63,17 +68,22 @@ it.skipIf(!process.env.DEEPSEEK_API_KEY || !runtime).each([{ format: 'docx', mod const pixels = await execa(python!, ['-c', "from pathlib import Path\nfrom PIL import Image\nfiles=list(Path('preview').glob('*.png'))\nassert files\nfor f in files:\n im=Image.open(f).convert('RGBA')\n visible=Image.alpha_composite(Image.new('RGBA',im.size,'white'),im).convert('RGB')\n print('content' if any(a!=b for a,b in visible.getextrema()) else 'blank')"], { cwd }) expect(pixels.exitCode).toBe(0) if (format === 'pdf') expect(pixels.stdout.trim()).toBe('content') + const final = events.filter(e => e.type === 'assistant/message').at(-1)?.data.message.content + .filter(b => b.type === 'text').map(b => b.text).join(' ') + const diagnostic = JSON.stringify({ + format, mode, callOrder: calls.map(call => call.name), skills, commands, pixels: pixels.stdout, final, + }, null, 2) const authoring = calls.findIndex(call => call.name === 'skill' && /office-(docx|pptx|xlsx)/u.test(call.arguments)) if (pixels.stdout.includes('blank')) { if (authoring !== -1) { const rendered = calls.findIndex(call => (call.name === 'bash' || call.name === 'pwsh') && /\srender\b/u.test(call.arguments)) expect(authoring, 'content checks after a blank preview may need the format skill').toBeGreaterThan(rendered) } - const final = events.filter(e => e.type === 'assistant/message').at(-1)?.data.message.content - .filter(b => b.type === 'text').map(b => b.text).join(' ') - expect(final).toMatch(/visual (?:inspection|check)[^.!?]{0,100}(?:not completed|incomplete|could not be completed|unverified)/iu) + expect(final, diagnostic).toMatch( + /visual (?:inspection|check)[^.!?]{0,100}(?:not completed|incomplete|could not be completed|unverified)/iu, + ) } else { - expect(authoring, JSON.stringify(skills)).toBe(-1) + expect(authoring, diagnostic).toBe(-1) } expect(calls.some(e => e.name === 'read_image')).toBe(true) }, diff --git a/packages/util/package-manifest/README.i18n.yaml b/packages/util/package-manifest/README.i18n.yaml index 2ce3905fcf..bb73ae8f0f 100644 --- a/packages/util/package-manifest/README.i18n.yaml +++ b/packages/util/package-manifest/README.i18n.yaml @@ -15,8 +15,8 @@ en: d152484eb41ac6b4 zh: 1db86285abf6c499 /deepseek-ai-dsh-package-manifest/use-this-package: - en: f275964a8845888a - zh: b284e09aacea079b + en: 0f5e861b0c0c5ee2 + zh: 766eaa36b794a22b /deepseek-ai-dsh-package-manifest/understand-the-implementation: en: 41ec449dee927f04 zh: ec1b583bd9025d84 diff --git a/packages/util/package-manifest/README.md b/packages/util/package-manifest/README.md index d50312947e..a1d5e7f423 100644 --- a/packages/util/package-manifest/README.md +++ b/packages/util/package-manifest/README.md @@ -43,7 +43,7 @@ const manifest: DshPackageManifest = { } ``` -`DshPackageManifest` describes the package.json fields used by DSH, with required `name` and `version`; it is not an exhaustive npm schema. Local profile readers use `Partial` because profiles need no published version. `DshManifest` describes only public author fields under `dsh`. `DshBundleManifest.patch` is one patch file path or an ordered list of them, each relative to the package root; the launcher applies a list in order as one bundle layer. TypeScript checks the example and erases `import type`; these interfaces do not parse JSON or write a file. +`DshPackageManifest` describes the package.json fields used by DSH, with required `name` and `version`; it includes dependency, peer, and optional dependency declarations but is not an exhaustive npm schema. Local profile readers use `Partial` because profiles need no published version. `DshManifest` describes only public author fields under `dsh`. `DshBundleManifest.patch` is one patch file path or an ordered list of them, each relative to the package root; the launcher applies a list in order as one bundle layer. TypeScript checks the example and erases `import type`; these interfaces do not parse JSON or write a file. The following metadata fields are optional. Omitting them leaves the format version or compatible host versions undeclared; readers do not infer defaults. diff --git a/packages/util/package-manifest/README.zh.md b/packages/util/package-manifest/README.zh.md index 9a535a094a..d676ff9bf7 100644 --- a/packages/util/package-manifest/README.zh.md +++ b/packages/util/package-manifest/README.zh.md @@ -43,7 +43,7 @@ const manifest: DshPackageManifest = { } ``` -`DshPackageManifest` 描述 DSH 使用的 package.json 字段，其中 `name` 和 `version` 必填；它不是完整的 npm schema（模式）。本地 profile 读取方使用 `Partial`，因为 profile 无需发布版本。`DshManifest` 仅描述 `dsh` 下的公共作者字段。`DshBundleManifest.patch` 是一个 patch 文件路径，或一个有序的路径列表，均相对于包根目录；launcher 按列表顺序把它们作为同一个组合包层应用。TypeScript 检查示例并删除 `import type`；这些接口不解析 JSON，也不写入文件。 +`DshPackageManifest` 描述 DSH 使用的 package.json 字段，其中 `name` 和 `version` 必填；它包含普通依赖、peer 依赖及可选依赖声明，未覆盖 npm 的所有字段。本地 profile 读取方使用 `Partial`，因为 profile 无需发布版本。`DshManifest` 仅描述 `dsh` 下的公共作者字段。`DshBundleManifest.patch` 是一个 patch 文件路径，或一个有序的路径列表，均相对于包根目录；launcher 按列表顺序把它们作为同一个组合包层应用。TypeScript 检查示例并删除 `import type`；这些接口不解析 JSON，也不写入文件。 以下元数据字段均可选。省略时，格式版本或兼容的宿主版本保持未声明状态；读取方不推断默认值。 diff --git a/packages/util/package-manifest/src/types.ts b/packages/util/package-manifest/src/types.ts index e5289ad1e2..27de17acfe 100644 --- a/packages/util/package-manifest/src/types.ts +++ b/packages/util/package-manifest/src/types.ts @@ -18,6 +18,8 @@ export interface DshPackageManifest { private?: boolean /** Packages installed alongside this package. */ dependencies?: Record + /** Packages that may be absent when installation or platform checks skip them. */ + optionalDependencies?: Record /** Compatible versions of packages supplied by the consuming project. */ peerDependencies?: Record /** Runtime requirements; DSH compatibility is declarative until a reader enforces it. */ diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 8ca82ea3af..f4487f549b 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -36,6 +36,9 @@ importers: '@deepseek-ai/dsh-cua': specifier: workspace:* version: link:packages/experimental/cua + '@deepseek-ai/dsh-cua-bundle': + specifier: workspace:* + version: link:packages/experimental/cua-bundle '@deepseek-ai/dsh-experimental-browser-use-chrome-devtools-mcp': specifier: workspace:* version: link:packages/experimental/browser-use-chrome-devtools-mcp @@ -309,6 +312,9 @@ importers: '@deepseek-ai/dsh-cordis-client-runner': specifier: workspace:* version: link:../../packages/extensions/cordis-client-runner + '@deepseek-ai/dsh-cua-bundle': + specifier: workspace:* + version: link:../../packages/experimental/cua-bundle '@deepseek-ai/dsh-experimental-agent-team-profile': specifier: workspace:* version: link:../../packages/experimental/agent-team-profile @@ -8541,12 +8547,12 @@ importers: '@deepseek-ai/dsh-atomic-write': specifier: workspace:* version: link:../../util/atomic-write - '@deepseek-ai/dsh-brand': + '@deepseek-ai/dsh-computer-use': specifier: workspace:* - version: link:../../util/brand - '@deepseek-ai/dsh-util-values': + version: link:../../computer-use/computer-use + '@deepseek-ai/dsh-typert-protocol': specifier: workspace:* - version: link:../../util/values + version: link:../../typert/protocol '@deepseek-ai/schemastery': specifier: link:../../../vendor/schemastery version: link:../../../vendor/schemastery @@ -8575,6 +8581,15 @@ importers: '@deepseek-ai/dsh-agent-loop-testkit': specifier: workspace:* version: link:../../test-support/agent-loop-testkit + '@deepseek-ai/dsh-agent-preset-registry': + specifier: workspace:* + version: link:../../preset/agent-preset-registry + '@deepseek-ai/dsh-api-gateway': + specifier: workspace:* + version: link:../../api/gateway + '@deepseek-ai/dsh-api-remotes': + specifier: workspace:* + version: link:../../api/remotes '@deepseek-ai/dsh-app-boot': specifier: workspace:* version: link:../../boot/app-boot @@ -8584,9 +8599,36 @@ importers: '@deepseek-ai/dsh-attachment-local': specifier: workspace:* version: link:../../attachment/attachment-local - '@deepseek-ai/dsh-computer-use': + '@deepseek-ai/dsh-brand': specifier: workspace:* - version: link:../../computer-use/computer-use + version: link:../../util/brand + '@deepseek-ai/dsh-client-locale': + specifier: workspace:* + version: link:../../client/locale + '@deepseek-ai/dsh-client-store': + specifier: workspace:* + version: link:../../client/store + '@deepseek-ai/dsh-client-test-runtime': + specifier: workspace:* + version: link:../../test-support/client-runtime + '@deepseek-ai/dsh-client-ui-layout': + specifier: workspace:* + version: link:../../client/ui-layout + '@deepseek-ai/dsh-client-ui-plugin-manager': + specifier: workspace:* + version: link:../../client/ui-plugin-manager + '@deepseek-ai/dsh-client-ui-primitives': + specifier: workspace:* + version: link:../../client/ui-primitives + '@deepseek-ai/dsh-client-ui-renderer': + specifier: workspace:* + version: link:../../client/ui-renderer + '@deepseek-ai/dsh-client-ui-settings': + specifier: workspace:* + version: link:../../client/ui-settings + '@deepseek-ai/dsh-client-ui-slots': + specifier: workspace:* + version: link:../../client/ui-slots '@deepseek-ai/dsh-config-editor': specifier: workspace:* version: link:../../boot/config-editor @@ -8602,6 +8644,9 @@ importers: '@deepseek-ai/dsh-llm-deepseek': specifier: workspace:* version: link:../../llm/llm-deepseek + '@deepseek-ai/dsh-loader-smoke': + specifier: workspace:* + version: link:../../test-support/loader-smoke '@deepseek-ai/dsh-ptc-runtime': specifier: workspace:* version: link:../../ptc-runtime/ptc-runtime @@ -8614,6 +8659,9 @@ importers: '@deepseek-ai/dsh-sandbox-policy': specifier: workspace:* version: link:../../sandbox/sandbox-policy + '@deepseek-ai/dsh-scope': + specifier: workspace:* + version: link:../../core/scope '@deepseek-ai/dsh-session': specifier: workspace:* version: link:../../core/session @@ -8638,13 +8686,51 @@ importers: '@deepseek-ai/dsh-user-questions': specifier: workspace:* version: link:../../interaction/user-questions + '@deepseek-ai/dsh-util-values': + specifier: workspace:* + version: link:../../util/values '@deepseek-ai/dsh-working-directory': specifier: workspace:* version: link:../../session/working-directory + '@testing-library/react': + specifier: ^16.1.0 + version: 16.3.3(@testing-library/dom@10.4.2)(@types/react-dom@18.3.7(@types/react@18.3.31))(@types/react@18.3.31)(react-dom@18.3.1(react@18.3.1))(react@18.3.1) + react: + specifier: ^18.2.0 + version: 18.3.1 + react-dom: + specifier: ^18.2.0 + version: 18.3.1(react@18.3.1) sharp: specifier: ^0.35.3 version: 0.35.5(@types/node@22.20.1) + packages/experimental/cua-bundle: + dependencies: + '@deepseek-ai/dsh-computer-use': + specifier: workspace:* + version: link:../../computer-use/computer-use + '@deepseek-ai/dsh-cua': + specifier: workspace:* + version: link:../cua + devDependencies: + '@deepseek-ai/cordis': + specifier: workspace:~ + version: link:../../../vendor/cordis + '@deepseek-ai/cordis-plugin-include': + specifier: workspace:~ + version: link:../../../vendor/include + '@deepseek-ai/cordis-plugin-loader': + specifier: workspace:~ + version: link:../../../vendor/loader + js-yaml: + specifier: ^4.2.0 + version: 4.3.2 + optionalDependencies: + '@deepseek-ai/dsh-experimental-cua-macos': + specifier: workspace:* + version: link:../cua-macos + packages/experimental/cua-macos: dependencies: '@deepseek-ai/dsh-atomic-write': diff --git a/scripts/check-workspace-constraints.ts b/scripts/check-workspace-constraints.ts index c02b46ed3b..108a345794 100644 --- a/scripts/check-workspace-constraints.ts +++ b/scripts/check-workspace-constraints.ts @@ -181,7 +181,7 @@ const packageFileExtras: Readonly> = { // The embedded OAR implementation retains its upstream license. '@deepseek-ai/dsh-experimental-subagent-oar': ['LICENSE-OAR'], // CUA publishes its client API and attributed native source for local macOS builds. - '@deepseek-ai/dsh-cua': ['lib/api.js', 'lib/bridge.js', 'lib/native-client.js', 'lib/protocol.js'], + '@deepseek-ai/dsh-cua': ['lib/remote-types.js', 'lib/api.js', 'lib/bridge.js', 'lib/native-client.js', 'lib/protocol.js'], '@deepseek-ai/dsh-experimental-cua-macos': ['native/bin', 'native/Package.swift', 'native/Sources', 'native/tests/swift', 'native/tests/test_swift.py', 'native/LICENSE.cua-driver', 'native/upstream.json', 'native/README.md', 'native/README.zh.md', 'native/protocol.schema.json'], '@deepseek-ai/dsh-webhook-github': ['examples/github-review/cordis.yml', 'examples/github-review/github-ready-review-rule.mjs'], // Owned Worker bundles import this public bootstrap before their business entry. diff --git a/scripts/doc-standard.spec.ts b/scripts/doc-standard.spec.ts index 3ad1e0ac77..dc6d976d34 100644 --- a/scripts/doc-standard.spec.ts +++ b/scripts/doc-standard.spec.ts @@ -55,7 +55,6 @@ const KIND_TEMPLATES: Readonly> = { * library; the check re-derives the entry shape so a stale entry fails loud. */ const PACKAGE_LIBRARIES: Readonly> = { - 'packages/experimental/cua': 'Shared CUA objects and installer consumed by platform providers; no plugin entry.', 'packages/experimental/browser-use-runtime': 'Provider-owned browser resource management and MCP integration helpers; no plugin entry.', 'packages/boot/app-boot': 'Boot library the app bins import; plain helper exports.', 'packages/boot/cmdline': 'Command-line library the app bins import; plain module exports.', diff --git a/scripts/experimental-package-policy.ts b/scripts/experimental-package-policy.ts index 466acaed2a..44fbd52ebe 100644 --- a/scripts/experimental-package-policy.ts +++ b/scripts/experimental-package-policy.ts @@ -1,8 +1,9 @@ /** Experimental package identities, publication policy, and retained npm names. */ -/** Existing npm names retained by capabilities now housed under experimental/. */ +/** Explicit npm identities retained across experimental classification and package-role moves. */ export const EXPERIMENTAL_PACKAGE_NAME_EXCEPTIONS: Readonly> = { 'packages/experimental/cua': '@deepseek-ai/dsh-cua', + 'packages/experimental/cua-bundle': '@deepseek-ai/dsh-cua-bundle', 'packages/experimental/tool-session-query': '@deepseek-ai/dsh-tool-session-query', 'packages/experimental/session-title-all-prompts-llm': '@deepseek-ai/dsh-session-title-all-prompts-llm', 'packages/experimental/tool-terminal': '@deepseek-ai/dsh-tool-terminal', diff --git a/scripts/gen-cordis-catalog.ts b/scripts/gen-cordis-catalog.ts index 44c305f342..9398831552 100644 --- a/scripts/gen-cordis-catalog.ts +++ b/scripts/gen-cordis-catalog.ts @@ -79,6 +79,7 @@ export const SERVICE_PAGE: Record = { ptcRuntime: 'ptc-runtime.md', browserUse: 'browser-use.md', computerUse: 'computer-use.md', + cuaAccess: 'computer-use.md', commands: 'commands.md', compaction: 'compaction.md', cordisInspect: 'extensions.md', @@ -331,6 +332,12 @@ export const LINK_MAP: Readonly> = { InspectOptions: 'boot.md', BrowserUseProviderName: 'browser-use.md', ComputerUseProviderName: 'computer-use.md', + CuaApplicationId: 'computer-use.md', + CuaApplicationGrant: 'computer-use.md', + CuaPermissionProcess: 'computer-use.md', + CuaApplicationState: 'computer-use.md', + CuaSystemPermission: 'computer-use.md', + CuaSystemPermissionState: 'computer-use.md', RenderedDocumentBytes: 'office-to-pdf.md', OfficeToPdfRequest: 'office-to-pdf.md', OfficeToPdfResult: 'office-to-pdf.md', diff --git a/scripts/gen-doc-graphs.ts b/scripts/gen-doc-graphs.ts index 7b4b9df647..94d7ff7c34 100644 --- a/scripts/gen-doc-graphs.ts +++ b/scripts/gen-doc-graphs.ts @@ -223,6 +223,11 @@ const SERVICE_ROLES: ServiceRole[] = [ consumers: ['experimental-computer-use-cua-driver-mcp', 'experimental-computer-use-cua-driver-native'], note: 'One provider-owned name per service instance. Each provider also owns its model tools; the service has no common action API, runtime selection, or Session workflow lock.', }, + { + key: 'cuaAccess', pkg: 'cua', title: 'Computer Use permission controls', + mode: 'core', consumers: ['cua'], + note: 'The native provider owns application consent and cancellation; the optional Client page queries and changes those decisions through its Remote.', + }, { key: 'officeToPdf', pkg: 'office-to-pdf', title: 'Office to PDF conversion', mode: 'core', consumers: ['client-ui-sidebar-documentpreview'], diff --git a/scripts/package-dependency-policy.ts b/scripts/package-dependency-policy.ts index ef15d1bbf7..95e498cdfc 100644 --- a/scripts/package-dependency-policy.ts +++ b/scripts/package-dependency-policy.ts @@ -51,6 +51,8 @@ const DUPLICATE_SAFE_PACKAGES: readonly string[] = [ * prominent heading in the pull request description. */ const SAFE_HOST_DEPENDENCY_EXPORTS = { + '@deepseek-ai/dsh-atomic-write': ['withFileLock'], + '@deepseek-ai/dsh-computer-use/brand': ['ComputerUseProviderName'], '@deepseek-ai/dsh-credentials': ['credentialKey'], '@deepseek-ai/dsh-deque': ['Deque'], '@deepseek-ai/dsh-llm': ['callConfigEquals'], @@ -61,6 +63,9 @@ const SAFE_HOST_DEPENDENCY_EXPORTS = { /** Runtime exports that require every consumer to resolve the provider's shared peer instance. */ const PEER_REQUIRED_HOST_EXPORTS = { + // CUA checks UserQuestionError and preserves AttachmentError from image admission. + '@deepseek-ai/dsh-attachment': ['admitEncodedImages'], + '@deepseek-ai/dsh-user-questions': ['UserQuestionError'], // `instanceof` checks need the translator's own class instance. '@deepseek-ai/dsh-experimental-translator': ['TranslationError'], '@deepseek-ai/dsh-scope': ['carrierKeyOf', 'createScope', 'scopeOf', 'scopeTarget'], diff --git a/scripts/release/families.spec.ts b/scripts/release/families.spec.ts index 4a6a12c0a0..5ead068208 100644 --- a/scripts/release/families.spec.ts +++ b/scripts/release/families.spec.ts @@ -66,6 +66,7 @@ describe('release families', () => { '@deepseek-ai/dsh-experimental-computer-use-cua-driver-mcp', '@deepseek-ai/dsh-experimental-computer-use-cua-driver-native', '@deepseek-ai/dsh-experimental-cot-translation-bundle', + '@deepseek-ai/dsh-cua-bundle', '@deepseek-ai/dsh-experimental-cua-macos', '@deepseek-ai/dsh-cua', '@deepseek-ai/dsh-hook-protocol', diff --git a/scripts/release/verify-packed-install.spec.ts b/scripts/release/verify-packed-install.spec.ts index b40f21ae92..4eb45db00d 100644 --- a/scripts/release/verify-packed-install.spec.ts +++ b/scripts/release/verify-packed-install.spec.ts @@ -38,7 +38,8 @@ const path = require('node:path') const consumer = process.env.INIT_CWD || process.cwd() const cache = path.join(consumer, '.npm-cache', '_cacache') const files = fs.existsSync(cache) ? fs.readdirSync(cache, { recursive: true }).filter(name => fs.statSync(path.join(cache, name)).isFile()) : [] -fs.writeFileSync(process.env.CACHE_EVIDENCE, JSON.stringify({ consumer, files })) +fs.writeFileSync(process.env.CACHE_EVIDENCE, JSON.stringify({ consumer, files, + installed: ['matching', 'wrong-os', 'wrong-cpu'].filter(name => fs.existsSync(path.join(consumer, 'node_modules', name))) })) ` writeFileSync(join(source, 'probe.cjs'), probe + (outcome === 'install failure' ? '\nprocess.exit(37)\n' : '')) writeFileSync(join(source, 'lib/bin.js'), outcome === 'entry failure' ? 'process.exit(37)\n' : 'console.log("0.0.1")\n') @@ -51,6 +52,21 @@ fs.writeFileSync(process.env.CACHE_EVIDENCE, JSON.stringify({ consumer, files }) expect(archive.error).toBeUndefined() expect(archive.signal).toBeNull() expect(archive.status, archive.stderr).toBe(0) + for (const [name, platform] of [ + ['matching', { os: [process.platform], cpu: ['any'] }], + ['wrong-os', { os: [`!${process.platform}`] }], + ['wrong-cpu', { cpu: [process.arch === 'arm64' ? 'x64' : 'arm64'] }], + ] as const) { + const packageDirectory = join(directory, name, 'package') + mkdirSync(packageDirectory, { recursive: true }) + writeFileSync(join(packageDirectory, 'package.json'), JSON.stringify({ name, version: '0.0.1', ...platform })) + const packedPlatform = spawnSync('tar', ['-czf', join(packed, `${name}.tgz`), '-C', join(directory, name), 'package'], { + encoding: 'utf8', timeout: 30_000, + }) + expect(packedPlatform.error).toBeUndefined() + expect(packedPlatform.signal).toBeNull() + expect(packedPlatform.status, packedPlatform.stderr).toBe(0) + } const environment = Object.fromEntries(Object.entries(process.env).filter(([key]) => !/^npm_config_/i.test(key))) Object.assign(environment, { npm_config_cache: ambient, @@ -68,7 +84,8 @@ fs.writeFileSync(process.env.CACHE_EVIDENCE, JSON.stringify({ consumer, files }) expect(result.error).toBeUndefined() expect(result.signal).toBeNull() expect(result.status, result.stdout + result.stderr).toBe(outcome === 'success' ? 0 : 1) - const observation = JSON.parse(readFileSync(evidence, 'utf8')) as { consumer: string; files: string[] } + const observation = JSON.parse(readFileSync(evidence, 'utf8')) as { consumer: string; files: string[]; installed: string[] } + expect(observation.installed).toEqual(['matching']) expect(observation.files.length, 'npm must write tarball content into the consumer-private cache').toBeGreaterThan(0) expect(observation.consumer).toContain('dsh-packed-dsh-') expect(existsSync(observation.consumer), 'the verifier finally must remove its consumer and cache').toBe(false) diff --git a/scripts/release/verify-packed-install.ts b/scripts/release/verify-packed-install.ts index 82e209cf49..08ab4860c0 100644 --- a/scripts/release/verify-packed-install.ts +++ b/scripts/release/verify-packed-install.ts @@ -24,7 +24,7 @@ import { pathToFileURL } from 'node:url' import { parseArgs } from 'node:util' import { releaseFamily } from './families.ts' import { capture, isEntry } from './process.ts' -import { packedIdentity } from './tarball.ts' +import { packedManifest } from './tarball.ts' import { verifyInstalledProductIsolation } from './installed-product-isolation.ts' /** @@ -45,30 +45,46 @@ function consumerEnvironment(consumerRoot: string): NodeJS.ProcessEnv { return environment } +function acceptsPlatform(value: unknown, current: string): boolean { + if (value === undefined) return true + if (!Array.isArray(value) || !value.every((tag): tag is string => typeof tag === 'string')) { + throw new Error('packed platform constraints must be arrays of strings') + } + let accepted = value.every(tag => tag.startsWith('!')) + for (const tag of value) { + if (tag === `!${current}`) return false + if (tag === current || (tag === 'any' && value.length === 1)) accepted = true + } + return accepted +} + /** - * Every packed tarball in the given directories, as `file:` dependency entries. + * Packed tarballs and their compatibility with the verification host, as `file:` dependency entries. * * The directories are read by their contents rather than a pack order file: a * directory here can hold tarballs packed only to satisfy a cross-sequence * dependency, which no release order describes. * @param directories - absolute directories holding packed tarballs. - * @returns Package name to tarball file URL, and the version each carries. + * @returns Package name to tarball URL, version, and host compatibility. */ -function packedDependencies(directories: readonly string[]): Map { - const dependencies = new Map() +function packedDependencies(directories: readonly string[]): Map { + const dependencies = new Map() for (const directory of directories) { const tarballs = readdirSync(directory).filter(name => name.endsWith('.tgz')).sort() if (tarballs.length === 0) throw new Error(`${directory} holds no packed tarball`) for (const filename of tarballs) { const tarball = join(directory, filename) - const { name, version } = packedIdentity(tarball) - dependencies.set(name, { url: pathToFileURL(tarball).href, version }) + const manifest = packedManifest(tarball) + const { name, version } = manifest + if (typeof name !== 'string' || typeof version !== 'string') throw new Error(`${tarball} manifest lacks name/version`) + const compatible = acceptsPlatform(manifest.os, process.platform) && acceptsPlatform(manifest.cpu, process.arch) + dependencies.set(name, { url: pathToFileURL(tarball).href, version, compatible }) } } return dependencies } -/** Install every tarball under `--from` and drive the `--family` entry. */ +/** Install platform-compatible tarballs under `--from` and drive the `--family` entry. */ function main(): void { const { values } = parseArgs({ options: { family: { type: 'string' }, from: { type: 'string', multiple: true } }, @@ -96,7 +112,11 @@ function main(): void { name: `dsh-packed-install-${family.id}`, version: '0.0.0', private: true, - dependencies: Object.fromEntries([...packed].map(([name, entryPacked]) => [name, entryPacked.url])), + dependencies: Object.fromEntries([...packed].filter(([, entryPacked]) => entryPacked.compatible) + .map(([name, entryPacked]) => [name, entryPacked.url])), + // Keep foreign-platform metadata local without turning those tarballs into required installations. + optionalDependencies: Object.fromEntries([...packed].filter(([, entryPacked]) => !entryPacked.compatible) + .map(([name, entryPacked]) => [name, entryPacked.url])), }, null, 2)}\n`) const environment = consumerEnvironment(consumerRoot) diff --git a/scripts/verify-cordis-config.spec.ts b/scripts/verify-cordis-config.spec.ts index ec0385af10..8c7209e86d 100644 --- a/scripts/verify-cordis-config.spec.ts +++ b/scripts/verify-cordis-config.spec.ts @@ -80,11 +80,14 @@ describe('workspace Bundle discovery and product dependency closures', () => { } const self = { file, name: '@deepseek-ai/dsh-subagent-example' } expect(bundlePluginDependencyErrors(manifestPath, manifest, [self])).toEqual([]) + expect(bundlePluginDependencyErrors(manifestPath, { + ...manifest, optionalDependencies: { '@deepseek-ai/dsh-platform-plugin': 'workspace:*' }, + }, [{ file, name: '@deepseek-ai/dsh-platform-plugin' }])).toEqual([]) expect(bundlePluginDependencyErrors(manifestPath, manifest, [ self, { file, name: '@deepseek-ai/dsh-missing-plugin' }, ])).toEqual([ - `${file}: @deepseek-ai/dsh-missing-plugin must be declared in ${manifestPath} dependencies`, + `${file}: @deepseek-ai/dsh-missing-plugin must be declared in ${manifestPath} dependencies or optionalDependencies`, ]) }) }) diff --git a/scripts/verify-cordis-config.ts b/scripts/verify-cordis-config.ts index 76e9bdba87..3de9914cdf 100644 --- a/scripts/verify-cordis-config.ts +++ b/scripts/verify-cordis-config.ts @@ -396,8 +396,8 @@ export function bundlePluginDependencyErrors( return missingPluginDependencies( // A Bundle may mount its own package (for example, its provider or runtime row). references.filter(reference => packageNameFromSpecifier(reference.name) !== manifest.name), - manifest.dependencies ?? {}, - `${manifestPath} dependencies`, + { ...manifest.dependencies, ...manifest.optionalDependencies }, + `${manifestPath} dependencies or optionalDependencies`, ) } diff --git a/snapshots/session/cua-native/consent-fixture.mjs b/snapshots/session/cua-native/consent-fixture.mjs deleted file mode 100644 index e0b0fe035e..0000000000 --- a/snapshots/session/cua-native/consent-fixture.mjs +++ /dev/null @@ -1,14 +0,0 @@ -/** Scripted human consent for the native application replay. */ -import { join } from 'node:path' -import { startFixtureHost } from './fixture-host.mjs' -export const name = 'cua-consent-fixture' -export const inject = ['userQuestions'] - -/** Answer through the existing human-question service while native dispatch stays real. */ -export async function apply(ctx) { - const close = await startFixtureHost(join(process.cwd(), 'native-helper.mjs')) - ctx.effect(() => close) - ctx.on('user-questions/request', async request => ({ answers: request.questions.map(question => ({ - id: question.id, selected: ['Allow for this task'], - })) })) -} diff --git a/snapshots/session/cua-native/cordis.snapshot.yml b/snapshots/session/cua-native/cordis.snapshot.yml index f6708af93d..958f41ca53 100644 --- a/snapshots/session/cua-native/cordis.snapshot.yml +++ b/snapshots/session/cua-native/cordis.snapshot.yml @@ -50,11 +50,13 @@ - id: cua-native name: '@deepseek-ai/dsh-experimental-cua-macos' config: + agentPreset: false helperPath: !!js process.cwd() + '/native-helper.mjs' + allowedApps: [{"id": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", "name": "Fixture", "executablePath": "/fixture/app"}] - id: user-questions name: '@deepseek-ai/dsh-user-questions' - insert: - - id: cua-consent-fixture - name: './consent-fixture.mjs' + - id: cua-permission-fixture + name: './permission-fixture.mjs' diff --git a/snapshots/session/cua-native/cordis.yml b/snapshots/session/cua-native/cordis.yml index b60d7872d4..c25eadc41d 100644 --- a/snapshots/session/cua-native/cordis.yml +++ b/snapshots/session/cua-native/cordis.yml @@ -32,11 +32,13 @@ - id: cua-native name: '@deepseek-ai/dsh-experimental-cua-macos' config: + agentPreset: false helperPath: !!js process.cwd() + '/native-helper.mjs' + allowedApps: [{"id": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", "name": "Fixture", "executablePath": "/fixture/app"}] - id: user-questions name: '@deepseek-ai/dsh-user-questions' - insert: - - id: cua-consent-fixture - name: './consent-fixture.mjs' + - id: cua-permission-fixture + name: './permission-fixture.mjs' diff --git a/snapshots/session/cua-native/permission-fixture.mjs b/snapshots/session/cua-native/permission-fixture.mjs new file mode 100644 index 0000000000..e1580ceedc --- /dev/null +++ b/snapshots/session/cua-native/permission-fixture.mjs @@ -0,0 +1,23 @@ +/** Scripted human answers and a permission-page revocation in the recorded CUA scenario. */ +import { join } from 'node:path' +import { startFixtureHost } from './fixture-host.mjs' +export const name = 'cua-permission-fixture' +export const inject = ['userQuestions', 'cuaAccess', 'tools'] + +/** Supply external human decisions while retaining the real permission policy and tool pipeline. */ +export async function apply(ctx) { + const close = await startFixtureHost(join(process.cwd(), 'native-helper.mjs')) + ctx.effect(() => close) + ctx.on('user-questions/request', async (request) => ({ answers: request.questions.map(question => ({ + id: question.id, + selected: [question.question.includes('Denied Fixture') ? 'Deny' : 'Allow for this session'], + })) })) + ctx.on('tools/post-execute', async (execution, _result, next) => { + const decision = await next() + if (execution.name === 'cua' && execution.arguments.method === 'getApp' + && execution.arguments.params.selector === 'Revocable Fixture') { + await ctx.cuaAccess.revoke('c'.repeat(64)) + } + return decision + }) +} diff --git a/snapshots/session/cua-native/replay.override.json b/snapshots/session/cua-native/replay.override.json index 643fa5098e..2fefba26f5 100644 --- a/snapshots/session/cua-native/replay.override.json +++ b/snapshots/session/cua-native/replay.override.json @@ -32,6 +32,105 @@ } ] }, + { + "kind": "chunks", + "chunks": [ + { + "type": "block-start", + "index": 0, + "blockType": "tool-call" + }, + { + "type": "block-end", + "index": 0, + "block": { + "type": "tool-call", + "id": "cua-unauthorized-getState", + "name": "cua", + "arguments": "{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\"}}" + } + }, + { + "type": "usage", + "usage": { + "inputTokens": 3, + "outputTokens": 3 + } + }, + { + "type": "finish", + "reason": { + "kind": "tool-calls" + } + } + ] + }, + { + "kind": "chunks", + "chunks": [ + { + "type": "block-start", + "index": 0, + "blockType": "tool-call" + }, + { + "type": "block-end", + "index": 0, + "block": { + "type": "tool-call", + "id": "cua-unauthorized-screenshot", + "name": "cua", + "arguments": "{\"method\":\"screenshot\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}" + } + }, + { + "type": "usage", + "usage": { + "inputTokens": 3, + "outputTokens": 3 + } + }, + { + "type": "finish", + "reason": { + "kind": "tool-calls" + } + } + ] + }, + { + "kind": "chunks", + "chunks": [ + { + "type": "block-start", + "index": 0, + "blockType": "tool-call" + }, + { + "type": "block-end", + "index": 0, + "block": { + "type": "tool-call", + "id": "cua-unauthorized-click", + "name": "cua", + "arguments": "{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"snapshotId\":\"fixture-snapshot-1\",\"elementIndex\":0}}" + } + }, + { + "type": "usage", + "usage": { + "inputTokens": 3, + "outputTokens": 3 + } + }, + { + "type": "finish", + "reason": { + "kind": "tool-calls" + } + } + ] + }, { "kind": "chunks", "chunks": [ @@ -1616,6 +1715,105 @@ } ] }, + { + "kind": "chunks", + "chunks": [ + { + "type": "block-start", + "index": 0, + "blockType": "tool-call" + }, + { + "type": "block-end", + "index": 0, + "block": { + "type": "tool-call", + "id": "cua-consent-denied", + "name": "cua", + "arguments": "{\"method\":\"getApp\",\"params\":{\"selector\":\"Denied Fixture\"}}" + } + }, + { + "type": "usage", + "usage": { + "inputTokens": 3, + "outputTokens": 3 + } + }, + { + "type": "finish", + "reason": { + "kind": "tool-calls" + } + } + ] + }, + { + "kind": "chunks", + "chunks": [ + { + "type": "block-start", + "index": 0, + "blockType": "tool-call" + }, + { + "type": "block-end", + "index": 0, + "block": { + "type": "tool-call", + "id": "cua-consent-revocable", + "name": "cua", + "arguments": "{\"method\":\"getApp\",\"params\":{\"selector\":\"Revocable Fixture\"}}" + } + }, + { + "type": "usage", + "usage": { + "inputTokens": 3, + "outputTokens": 3 + } + }, + { + "type": "finish", + "reason": { + "kind": "tool-calls" + } + } + ] + }, + { + "kind": "chunks", + "chunks": [ + { + "type": "block-start", + "index": 0, + "blockType": "tool-call" + }, + { + "type": "block-end", + "index": 0, + "block": { + "type": "tool-call", + "id": "cua-consent-revoked", + "name": "cua", + "arguments": "{\"method\":\"getState\",\"params\":{\"appHandle\":\"revoked-app\"}}" + } + }, + { + "type": "usage", + "usage": { + "inputTokens": 3, + "outputTokens": 3 + } + }, + { + "type": "finish", + "reason": { + "kind": "tool-calls" + } + } + ] + }, { "kind": "chunks", "chunks": [ diff --git a/snapshots/session/cua-native/session.v4.jsonl b/snapshots/session/cua-native/session.v4.jsonl index c20ab1a7bd..e7587237ab 100644 --- a/snapshots/session/cua-native/session.v4.jsonl +++ b/snapshots/session/cua-native/session.v4.jsonl @@ -2,261 +2,291 @@ {"type":"permission/preset","data":{"preset":"danger-full-access"}} {"type":"sandbox/mode","data":{"mode":"danger-full-access"}} {"type":"approval/policy","data":{"policy":"never"}} -{"type":"agent/inbox/spliced","data":{"target":"next-turn","start":0,"inserted":[{"content":[{"type":"text","text":"Inspect CUA capabilities, select the Fixture application, increment its counter once, then read fresh state and capture the window. Also inspect the AXless canvas window: capture it, dispatch one coordinate click at [20,20], and read its fresh empty AX state."}],"source":{"kind":"user"},"role":"user","id":"{{message:1}}"}]}} +{"type":"agent/inbox/spliced","data":{"target":"next-turn","start":0,"inserted":[{"content":[{"type":"text","text":"Inspect CUA capabilities, select the Fixture application, increment its counter once, then read fresh state and capture the window. Also inspect the AXless canvas window: capture it, dispatch one coordinate click at [20,20], and read its fresh empty AX state. Finally verify a denied application selection and refusal of a handle after application access is revoked."}],"source":{"kind":"user"},"role":"user","id":"{{message:1}}"}]}} {"type":"turn/start","data":{"turn":1}} {"type":"agent/inbox/spliced","data":{"target":"next-turn","start":0,"removedCount":1,"inserted":[]}} {"type":"step/start","data":{"turn":1,"step":1}} {"type":"system/message","data":{"turn":1,"step":1,"message":{"role":"system","content":[{"type":"text","text":"{{system}}"}],"source":{"kind":"system-prompt"},"id":"{{message:2}}"}},"surfaceOp":"append"} -{"type":"user/message","data":{"content":[{"type":"text","text":"Inspect CUA capabilities, select the Fixture application, increment its counter once, then read fresh state and capture the window. Also inspect the AXless canvas window: capture it, dispatch one coordinate click at [20,20], and read its fresh empty AX state."}],"source":{"kind":"user"},"role":"user","id":"{{message:1}}"},"surfaceOp":"append"} +{"type":"user/message","data":{"content":[{"type":"text","text":"Inspect CUA capabilities, select the Fixture application, increment its counter once, then read fresh state and capture the window. Also inspect the AXless canvas window: capture it, dispatch one coordinate click at [20,20], and read its fresh empty AX state. Finally verify a denied application selection and refusal of a handle after application access is revoked."}],"source":{"kind":"user"},"role":"user","id":"{{message:1}}"},"surfaceOp":"append"} {"type":"user/message","data":{"content":[{"type":"text","text":"Current runtime context. This snapshot supersedes earlier runtime-context snapshots.\n\nCurrent working directory: \"{{cwd}}\".\n\nCurrent DSH file policy: danger-full-access. The DSH file sandbox does not restrict file modifications by available operations.\n\nApproval prompts are disabled in this session: actions that require approval are rejected automatically — do not request sandbox escalation (do not set `sandbox_permissions`)."}],"source":{"kind":"runtime-context","form":"snapshot","sections":[{"name":"working-directory:current","text":"Current working directory: \"{{cwd}}\"."},{"name":"sandbox:policy","text":"Current DSH file policy: danger-full-access. The DSH file sandbox does not restrict file modifications by available operations."},{"name":"approval:policy","text":"Approval prompts are disabled in this session: actions that require approval are rejected automatically — do not request sandbox escalation (do not set `sandbox_permissions`)."}]},"role":"user","id":"{{message:3}}"},"surfaceOp":"append"} {"type":"request/header","data":{"header":{"config":{"provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"tools":"{{tools}}"},"reason":"initial"}} {"type":"request/context","data":{"provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"}} {"type":"session/title","data":{"title":"Inspect CUA capabilities, select the","messageSeqs":[8],"source":{"kind":"fallback"}}} -{"type":"assistant/message","data":{"turn":1,"step":1,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-1","name":"cua","arguments":"{\"method\":\"getCapabilities\",\"params\":{}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:4}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947554024,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947554025,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-1","name":"cua","arguments":"{\"method\":\"getCapabilities\",\"params\":{}}"}}},{"type":"chunk","time":1791947554025,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947554025,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} +{"type":"assistant/message","data":{"turn":1,"step":1,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-1","name":"cua","arguments":"{\"method\":\"getCapabilities\",\"params\":{}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:4}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947688973,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947688973,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-1","name":"cua","arguments":"{\"method\":\"getCapabilities\",\"params\":{}}"}}},{"type":"chunk","time":1791947688973,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947688973,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} {"type":"tool/call","data":{"turn":1,"step":1,"callId":"cua-native-1","name":"cua","arguments":"{\"method\":\"getCapabilities\",\"params\":{}}"}} {"type":"tool/result","data":{"turn":1,"step":1,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-1"},"toolCallId":"cua-native-1","content":[{"type":"text","text":"{\"platform\":\"macos\",\"sessionLocked\":false,\"permissions\":{\"accessibility\":true,\"screenRecording\":true},\"methods\":[\"getCapabilities\",\"listApps\",\"getApp\",\"activate\",\"show\",\"listWindows\",\"getState\",\"hover\",\"click\",\"setValue\",\"typeText\",\"paste\",\"pressKey\",\"scroll\",\"drag\",\"screenshot\"],\"coordinateSpace\":\"screenshot-pixels\",\"browser\":false}"}],"isError":false,"id":"{{message:5}}"}},"sourceEventSeqs":[14],"surfaceOp":"append"} {"type":"step/end","data":{"turn":1,"step":1}} {"type":"step/start","data":{"turn":1,"step":2}} -{"type":"assistant/message","data":{"turn":1,"step":2,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-2","name":"cua","arguments":"{\"method\":\"getApp\",\"params\":{\"selector\":\"test.fixture\"}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:6}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947554613,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947554614,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-2","name":"cua","arguments":"{\"method\":\"getApp\",\"params\":{\"selector\":\"test.fixture\"}}"}}},{"type":"chunk","time":1791947554614,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947554614,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} -{"type":"tool/call","data":{"turn":1,"step":2,"callId":"cua-native-2","name":"cua","arguments":"{\"method\":\"getApp\",\"params\":{\"selector\":\"test.fixture\"}}"}} -{"type":"tool/result","data":{"turn":1,"step":2,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-2"},"toolCallId":"cua-native-2","content":[{"type":"text","text":"{\"appHandle\":\"fixture-app\",\"pid\":9,\"name\":\"Fixture\",\"bundleId\":\"test.fixture\"}"}],"isError":false,"id":"{{message:7}}"}},"sourceEventSeqs":[19],"surfaceOp":"append"} +{"type":"assistant/message","data":{"turn":1,"step":2,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-unauthorized-getState","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\"}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:6}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947689573,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947689573,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-unauthorized-getState","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\"}}"}}},{"type":"chunk","time":1791947689573,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947689573,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} +{"type":"tool/call","data":{"turn":1,"step":2,"callId":"cua-unauthorized-getState","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\"}}"}} +{"type":"tool/result","data":{"turn":1,"step":2,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-unauthorized-getState"},"toolCallId":"cua-unauthorized-getState","content":[{"type":"text","text":"Error: APP_ACCESS_REQUIRED: Select the application with getApp and obtain user consent before reading or controlling it"}],"isError":true,"id":"{{message:7}}"}},"sourceEventSeqs":[19],"surfaceOp":"append"} {"type":"step/end","data":{"turn":1,"step":2}} {"type":"step/start","data":{"turn":1,"step":3}} -{"type":"assistant/message","data":{"turn":1,"step":3,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-3","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7,\"includeMenus\":true,\"details\":true}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:8}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947554635,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947554635,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-3","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7,\"includeMenus\":true,\"details\":true}}"}}},{"type":"chunk","time":1791947554635,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947554635,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} -{"type":"tool/call","data":{"turn":1,"step":3,"callId":"cua-native-3","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7,\"includeMenus\":true,\"details\":true}}"}} -{"type":"tool/result","data":{"turn":1,"step":3,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-3"},"toolCallId":"cua-native-3","content":[{"type":"text","text":"{\"appHandle\":\"fixture-app\",\"pid\":9,\"windowId\":7,\"snapshotId\":\"fixture-snapshot-1\",\"tree\":\"AX scope=window offsets=UTF-16 bounds=global-points\\n- [0] AXButton \\\"Increment\\\" id=\\\"increment\\\" actions=[\\\"AXPress\\\"] bounds=[60,80,100,20]\\n- [1] AXTextField valueWritable=true selectionWritable=true value=\\\"\\\" placeholder=\\\"Fixture text\\\"\\n- AXStaticText \\\"Count: 0\\\"\\n- [2] AXMenuBarItem \\\"File\\\"\\n - [3] AXMenuItem \\\"Fixture command\\\"\",\"windows\":[{\"windowId\":7,\"pid\":9,\"appName\":\"Fixture\",\"title\":\"CUA fixture\",\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"isOnScreen\":true,\"onCurrentSpace\":true}],\"truncated\":false}"}],"isError":false,"id":"{{message:9}}"}},"sourceEventSeqs":[24],"surfaceOp":"append"} +{"type":"assistant/message","data":{"turn":1,"step":3,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-unauthorized-screenshot","name":"cua","arguments":"{\"method\":\"screenshot\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:8}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947689596,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947689596,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-unauthorized-screenshot","name":"cua","arguments":"{\"method\":\"screenshot\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}}},{"type":"chunk","time":1791947689596,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947689596,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} +{"type":"tool/call","data":{"turn":1,"step":3,"callId":"cua-unauthorized-screenshot","name":"cua","arguments":"{\"method\":\"screenshot\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}} +{"type":"tool/result","data":{"turn":1,"step":3,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-unauthorized-screenshot"},"toolCallId":"cua-unauthorized-screenshot","content":[{"type":"text","text":"Error: APP_ACCESS_REQUIRED: Select the application with getApp and obtain user consent before reading or controlling it"}],"isError":true,"id":"{{message:9}}"}},"sourceEventSeqs":[24],"surfaceOp":"append"} {"type":"step/end","data":{"turn":1,"step":3}} {"type":"step/start","data":{"turn":1,"step":4}} -{"type":"assistant/message","data":{"turn":1,"step":4,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-4","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"snapshotId\":\"fixture-snapshot-1\",\"elementIndex\":0,\"holdMs\":800}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:10}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947554651,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947554651,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-4","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"snapshotId\":\"fixture-snapshot-1\",\"elementIndex\":0,\"holdMs\":800}}"}}},{"type":"chunk","time":1791947554651,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947554651,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} -{"type":"tool/call","data":{"turn":1,"step":4,"callId":"cua-native-4","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"snapshotId\":\"fixture-snapshot-1\",\"elementIndex\":0,\"holdMs\":800}}"}} -{"type":"tool/result","data":{"turn":1,"step":4,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-4"},"toolCallId":"cua-native-4","content":[{"type":"text","text":"{\"dispatched\":true,\"verified\":false}"}],"isError":false,"id":"{{message:11}}"}},"sourceEventSeqs":[29],"surfaceOp":"append"} +{"type":"assistant/message","data":{"turn":1,"step":4,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-unauthorized-click","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"snapshotId\":\"fixture-snapshot-1\",\"elementIndex\":0}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:10}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947689615,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947689615,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-unauthorized-click","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"snapshotId\":\"fixture-snapshot-1\",\"elementIndex\":0}}"}}},{"type":"chunk","time":1791947689615,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947689615,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} +{"type":"tool/call","data":{"turn":1,"step":4,"callId":"cua-unauthorized-click","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"snapshotId\":\"fixture-snapshot-1\",\"elementIndex\":0}}"}} +{"type":"tool/result","data":{"turn":1,"step":4,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-unauthorized-click"},"toolCallId":"cua-unauthorized-click","content":[{"type":"text","text":"Error: APP_ACCESS_REQUIRED: Select the application with getApp and obtain user consent before reading or controlling it"}],"isError":true,"id":"{{message:11}}"}},"sourceEventSeqs":[29],"surfaceOp":"append"} {"type":"step/end","data":{"turn":1,"step":4}} {"type":"step/start","data":{"turn":1,"step":5}} -{"type":"assistant/message","data":{"turn":1,"step":5,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-5","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:12}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947554667,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947554667,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-5","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}}},{"type":"chunk","time":1791947554667,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947554667,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} -{"type":"tool/call","data":{"turn":1,"step":5,"callId":"cua-native-5","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}} -{"type":"tool/result","data":{"turn":1,"step":5,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-5"},"toolCallId":"cua-native-5","content":[{"type":"text","text":"{\"appHandle\":\"fixture-app\",\"pid\":9,\"windowId\":7,\"snapshotId\":\"fixture-snapshot-2\",\"tree\":\"AX scope=window offsets=UTF-16\\n- [0] AXButton \\\"Increment\\\"\\n- [1] AXTextField valueWritable=true selectionWritable=true value=\\\"\\\" placeholder=\\\"Fixture text\\\"\\n- AXStaticText \\\"Count: 1\\\"\\n- [2] AXMenuBarItem \\\"File\\\"\\nMenu contents omitted; request includeMenus=true to expand.\",\"windows\":[{\"windowId\":7,\"pid\":9,\"appName\":\"Fixture\",\"title\":\"CUA fixture\",\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"isOnScreen\":true,\"onCurrentSpace\":true}],\"truncated\":false}"}],"isError":false,"id":"{{message:13}}"}},"sourceEventSeqs":[34],"surfaceOp":"append"} +{"type":"assistant/message","data":{"turn":1,"step":5,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-2","name":"cua","arguments":"{\"method\":\"getApp\",\"params\":{\"selector\":\"test.fixture\"}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:12}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947689634,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947689634,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-2","name":"cua","arguments":"{\"method\":\"getApp\",\"params\":{\"selector\":\"test.fixture\"}}"}}},{"type":"chunk","time":1791947689634,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947689634,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} +{"type":"tool/call","data":{"turn":1,"step":5,"callId":"cua-native-2","name":"cua","arguments":"{\"method\":\"getApp\",\"params\":{\"selector\":\"test.fixture\"}}"}} +{"type":"tool/result","data":{"turn":1,"step":5,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-2"},"toolCallId":"cua-native-2","content":[{"type":"text","text":"{\"appHandle\":\"fixture-app\",\"pid\":9,\"name\":\"Fixture\",\"bundleId\":\"test.fixture\"}"}],"isError":false,"id":"{{message:13}}"}},"sourceEventSeqs":[34],"surfaceOp":"append"} {"type":"step/end","data":{"turn":1,"step":5}} {"type":"step/start","data":{"turn":1,"step":6}} -{"type":"assistant/message","data":{"turn":1,"step":6,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-6","name":"cua","arguments":"{\"method\":\"screenshot\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:14}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947554684,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947554684,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-6","name":"cua","arguments":"{\"method\":\"screenshot\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}}},{"type":"chunk","time":1791947554684,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947554684,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} -{"type":"tool/call","data":{"turn":1,"step":6,"callId":"cua-native-6","name":"cua","arguments":"{\"method\":\"screenshot\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}} -{"type":"tool/result","data":{"turn":1,"step":6,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-6"},"toolCallId":"cua-native-6","content":[{"type":"text","text":"{\"pixelWidth\":1,\"pixelHeight\":1,\"windowId\":7,\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"coordinateSpace\":\"screenshot-pixels\"}"},{"type":"image","attachment":{"attachmentId":"sha256:999f1d1527ee7e79266f16add5430fff76b1225d742464a5b1ff1f02971bb8ee","mediaType":"image/png","width":1,"height":1,"bytes":90}}],"isError":false,"id":"{{message:15}}"}},"sourceEventSeqs":[39],"surfaceOp":"append"} +{"type":"assistant/message","data":{"turn":1,"step":6,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-3","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7,\"includeMenus\":true,\"details\":true}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:14}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947689657,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947689657,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-3","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7,\"includeMenus\":true,\"details\":true}}"}}},{"type":"chunk","time":1791947689657,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947689657,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} +{"type":"tool/call","data":{"turn":1,"step":6,"callId":"cua-native-3","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7,\"includeMenus\":true,\"details\":true}}"}} +{"type":"tool/result","data":{"turn":1,"step":6,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-3"},"toolCallId":"cua-native-3","content":[{"type":"text","text":"{\"appHandle\":\"fixture-app\",\"pid\":9,\"windowId\":7,\"snapshotId\":\"fixture-snapshot-1\",\"tree\":\"AX scope=window offsets=UTF-16 bounds=global-points\\n- [0] AXButton \\\"Increment\\\" id=\\\"increment\\\" actions=[\\\"AXPress\\\"] bounds=[60,80,100,20]\\n- [1] AXTextField valueWritable=true selectionWritable=true value=\\\"\\\" placeholder=\\\"Fixture text\\\"\\n- AXStaticText \\\"Count: 0\\\"\\n- [2] AXMenuBarItem \\\"File\\\"\\n - [3] AXMenuItem \\\"Fixture command\\\"\",\"windows\":[{\"windowId\":7,\"pid\":9,\"appName\":\"Fixture\",\"title\":\"CUA fixture\",\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"isOnScreen\":true,\"onCurrentSpace\":true}],\"truncated\":false}"}],"isError":false,"id":"{{message:15}}"}},"sourceEventSeqs":[39],"surfaceOp":"append"} {"type":"step/end","data":{"turn":1,"step":6}} {"type":"step/start","data":{"turn":1,"step":7}} -{"type":"assistant/message","data":{"turn":1,"step":7,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-7","name":"cua","arguments":"{\"method\":\"listWindows\",\"params\":{\"appHandle\":\"fixture-app\"}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:16}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947554733,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947554733,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-7","name":"cua","arguments":"{\"method\":\"listWindows\",\"params\":{\"appHandle\":\"fixture-app\"}}"}}},{"type":"chunk","time":1791947554733,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947554733,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} -{"type":"tool/call","data":{"turn":1,"step":7,"callId":"cua-native-7","name":"cua","arguments":"{\"method\":\"listWindows\",\"params\":{\"appHandle\":\"fixture-app\"}}"}} -{"type":"tool/result","data":{"turn":1,"step":7,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-7"},"toolCallId":"cua-native-7","content":[{"type":"text","text":"[{\"windowId\":7,\"pid\":9,\"appName\":\"Fixture\",\"title\":\"CUA fixture\",\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"isOnScreen\":true,\"onCurrentSpace\":true},{\"windowId\":8,\"pid\":9,\"appName\":\"Fixture\",\"title\":\"AXless canvas\",\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"isOnScreen\":true,\"onCurrentSpace\":true}]"}],"isError":false,"id":"{{message:17}}"}},"sourceEventSeqs":[44],"surfaceOp":"append"} +{"type":"assistant/message","data":{"turn":1,"step":7,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-4","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"snapshotId\":\"fixture-snapshot-1\",\"elementIndex\":0,\"holdMs\":800}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:16}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947689683,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947689683,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-4","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"snapshotId\":\"fixture-snapshot-1\",\"elementIndex\":0,\"holdMs\":800}}"}}},{"type":"chunk","time":1791947689683,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947689683,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} +{"type":"tool/call","data":{"turn":1,"step":7,"callId":"cua-native-4","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"snapshotId\":\"fixture-snapshot-1\",\"elementIndex\":0,\"holdMs\":800}}"}} +{"type":"tool/result","data":{"turn":1,"step":7,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-4"},"toolCallId":"cua-native-4","content":[{"type":"text","text":"{\"dispatched\":true,\"verified\":false}"}],"isError":false,"id":"{{message:17}}"}},"sourceEventSeqs":[44],"surfaceOp":"append"} {"type":"step/end","data":{"turn":1,"step":7}} {"type":"step/start","data":{"turn":1,"step":8}} -{"type":"assistant/message","data":{"turn":1,"step":8,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-8","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":8}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:18}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947554749,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947554749,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-8","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":8}}"}}},{"type":"chunk","time":1791947554749,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947554749,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} -{"type":"tool/call","data":{"turn":1,"step":8,"callId":"cua-native-8","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":8}}"}} -{"type":"tool/result","data":{"turn":1,"step":8,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-8"},"toolCallId":"cua-native-8","content":[{"type":"text","text":"{\"appHandle\":\"fixture-app\",\"pid\":9,\"windowId\":8,\"snapshotId\":\"fixture-snapshot-3\",\"tree\":\"\",\"windows\":[{\"windowId\":8,\"pid\":9,\"appName\":\"Fixture\",\"title\":\"AXless canvas\",\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"isOnScreen\":true,\"onCurrentSpace\":true}],\"truncated\":false}"}],"isError":false,"id":"{{message:19}}"}},"sourceEventSeqs":[49],"surfaceOp":"append"} +{"type":"assistant/message","data":{"turn":1,"step":8,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-5","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:18}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947689706,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947689706,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-5","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}}},{"type":"chunk","time":1791947689706,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947689706,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} +{"type":"tool/call","data":{"turn":1,"step":8,"callId":"cua-native-5","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}} +{"type":"tool/result","data":{"turn":1,"step":8,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-5"},"toolCallId":"cua-native-5","content":[{"type":"text","text":"{\"appHandle\":\"fixture-app\",\"pid\":9,\"windowId\":7,\"snapshotId\":\"fixture-snapshot-2\",\"tree\":\"AX scope=window offsets=UTF-16\\n- [0] AXButton \\\"Increment\\\"\\n- [1] AXTextField valueWritable=true selectionWritable=true value=\\\"\\\" placeholder=\\\"Fixture text\\\"\\n- AXStaticText \\\"Count: 1\\\"\\n- [2] AXMenuBarItem \\\"File\\\"\\nMenu contents omitted; request includeMenus=true to expand.\",\"windows\":[{\"windowId\":7,\"pid\":9,\"appName\":\"Fixture\",\"title\":\"CUA fixture\",\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"isOnScreen\":true,\"onCurrentSpace\":true}],\"truncated\":false}"}],"isError":false,"id":"{{message:19}}"}},"sourceEventSeqs":[49],"surfaceOp":"append"} {"type":"step/end","data":{"turn":1,"step":8}} {"type":"step/start","data":{"turn":1,"step":9}} -{"type":"assistant/message","data":{"turn":1,"step":9,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-9","name":"cua","arguments":"{\"method\":\"screenshot\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":8}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:20}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947554765,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947554765,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-9","name":"cua","arguments":"{\"method\":\"screenshot\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":8}}"}}},{"type":"chunk","time":1791947554765,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947554765,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} -{"type":"tool/call","data":{"turn":1,"step":9,"callId":"cua-native-9","name":"cua","arguments":"{\"method\":\"screenshot\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":8}}"}} -{"type":"tool/result","data":{"turn":1,"step":9,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-9"},"toolCallId":"cua-native-9","content":[{"type":"text","text":"{\"pixelWidth\":1,\"pixelHeight\":1,\"windowId\":8,\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"coordinateSpace\":\"screenshot-pixels\"}"},{"type":"image","attachment":{"attachmentId":"sha256:999f1d1527ee7e79266f16add5430fff76b1225d742464a5b1ff1f02971bb8ee","mediaType":"image/png","width":1,"height":1,"bytes":90}}],"isError":false,"id":"{{message:21}}"}},"sourceEventSeqs":[54],"surfaceOp":"append"} +{"type":"assistant/message","data":{"turn":1,"step":9,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-6","name":"cua","arguments":"{\"method\":\"screenshot\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:20}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947689729,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947689729,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-6","name":"cua","arguments":"{\"method\":\"screenshot\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}}},{"type":"chunk","time":1791947689729,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947689729,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} +{"type":"tool/call","data":{"turn":1,"step":9,"callId":"cua-native-6","name":"cua","arguments":"{\"method\":\"screenshot\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}} +{"type":"tool/result","data":{"turn":1,"step":9,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-6"},"toolCallId":"cua-native-6","content":[{"type":"text","text":"{\"pixelWidth\":1,\"pixelHeight\":1,\"windowId\":7,\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"coordinateSpace\":\"screenshot-pixels\"}"},{"type":"image","attachment":{"attachmentId":"sha256:999f1d1527ee7e79266f16add5430fff76b1225d742464a5b1ff1f02971bb8ee","mediaType":"image/png","width":1,"height":1,"bytes":90}}],"isError":false,"id":"{{message:21}}"}},"sourceEventSeqs":[54],"surfaceOp":"append"} {"type":"step/end","data":{"turn":1,"step":9}} {"type":"step/start","data":{"turn":1,"step":10}} -{"type":"assistant/message","data":{"turn":1,"step":10,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-10","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":8,\"snapshotId\":\"fixture-snapshot-3\",\"x\":0,\"y\":0}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:22}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947554791,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947554791,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-10","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":8,\"snapshotId\":\"fixture-snapshot-3\",\"x\":0,\"y\":0}}"}}},{"type":"chunk","time":1791947554791,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947554791,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} -{"type":"tool/call","data":{"turn":1,"step":10,"callId":"cua-native-10","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":8,\"snapshotId\":\"fixture-snapshot-3\",\"x\":0,\"y\":0}}"}} -{"type":"tool/result","data":{"turn":1,"step":10,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-10"},"toolCallId":"cua-native-10","content":[{"type":"text","text":"{\"dispatched\":true,\"verified\":false}"}],"isError":false,"id":"{{message:23}}"}},"sourceEventSeqs":[59],"surfaceOp":"append"} +{"type":"assistant/message","data":{"turn":1,"step":10,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-7","name":"cua","arguments":"{\"method\":\"listWindows\",\"params\":{\"appHandle\":\"fixture-app\"}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:22}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947689792,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947689792,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-7","name":"cua","arguments":"{\"method\":\"listWindows\",\"params\":{\"appHandle\":\"fixture-app\"}}"}}},{"type":"chunk","time":1791947689792,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947689792,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} +{"type":"tool/call","data":{"turn":1,"step":10,"callId":"cua-native-7","name":"cua","arguments":"{\"method\":\"listWindows\",\"params\":{\"appHandle\":\"fixture-app\"}}"}} +{"type":"tool/result","data":{"turn":1,"step":10,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-7"},"toolCallId":"cua-native-7","content":[{"type":"text","text":"[{\"windowId\":7,\"pid\":9,\"appName\":\"Fixture\",\"title\":\"CUA fixture\",\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"isOnScreen\":true,\"onCurrentSpace\":true},{\"windowId\":8,\"pid\":9,\"appName\":\"Fixture\",\"title\":\"AXless canvas\",\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"isOnScreen\":true,\"onCurrentSpace\":true}]"}],"isError":false,"id":"{{message:23}}"}},"sourceEventSeqs":[59],"surfaceOp":"append"} {"type":"step/end","data":{"turn":1,"step":10}} {"type":"step/start","data":{"turn":1,"step":11}} -{"type":"assistant/message","data":{"turn":1,"step":11,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-11","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":8}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:24}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947554807,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947554807,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-11","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":8}}"}}},{"type":"chunk","time":1791947554807,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947554807,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} -{"type":"tool/call","data":{"turn":1,"step":11,"callId":"cua-native-11","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":8}}"}} -{"type":"tool/result","data":{"turn":1,"step":11,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-11"},"toolCallId":"cua-native-11","content":[{"type":"text","text":"{\"appHandle\":\"fixture-app\",\"pid\":9,\"windowId\":8,\"snapshotId\":\"fixture-snapshot-4\",\"tree\":\"\",\"windows\":[{\"windowId\":8,\"pid\":9,\"appName\":\"Fixture\",\"title\":\"AXless canvas\",\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"isOnScreen\":true,\"onCurrentSpace\":true}],\"truncated\":false}"}],"isError":false,"id":"{{message:25}}"}},"sourceEventSeqs":[64],"surfaceOp":"append"} +{"type":"assistant/message","data":{"turn":1,"step":11,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-8","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":8}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:24}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947689834,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947689834,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-8","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":8}}"}}},{"type":"chunk","time":1791947689834,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947689834,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} +{"type":"tool/call","data":{"turn":1,"step":11,"callId":"cua-native-8","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":8}}"}} +{"type":"tool/result","data":{"turn":1,"step":11,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-8"},"toolCallId":"cua-native-8","content":[{"type":"text","text":"{\"appHandle\":\"fixture-app\",\"pid\":9,\"windowId\":8,\"snapshotId\":\"fixture-snapshot-3\",\"tree\":\"\",\"windows\":[{\"windowId\":8,\"pid\":9,\"appName\":\"Fixture\",\"title\":\"AXless canvas\",\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"isOnScreen\":true,\"onCurrentSpace\":true}],\"truncated\":false}"}],"isError":false,"id":"{{message:25}}"}},"sourceEventSeqs":[64],"surfaceOp":"append"} {"type":"step/end","data":{"turn":1,"step":11}} {"type":"step/start","data":{"turn":1,"step":12}} -{"type":"assistant/message","data":{"turn":1,"step":12,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-show","name":"cua","arguments":"{\"method\":\"show\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:26}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947554823,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947554823,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-show","name":"cua","arguments":"{\"method\":\"show\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}}},{"type":"chunk","time":1791947554823,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947554823,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} -{"type":"tool/call","data":{"turn":1,"step":12,"callId":"cua-native-show","name":"cua","arguments":"{\"method\":\"show\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}} -{"type":"tool/result","data":{"turn":1,"step":12,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-show"},"toolCallId":"cua-native-show","content":[{"type":"text","text":"{\"dispatched\":true,\"verified\":false,\"windowId\":7,\"foregroundChanged\":false}"}],"isError":false,"id":"{{message:27}}"}},"sourceEventSeqs":[69],"surfaceOp":"append"} +{"type":"assistant/message","data":{"turn":1,"step":12,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-9","name":"cua","arguments":"{\"method\":\"screenshot\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":8}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:26}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947689861,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947689861,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-9","name":"cua","arguments":"{\"method\":\"screenshot\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":8}}"}}},{"type":"chunk","time":1791947689861,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947689861,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} +{"type":"tool/call","data":{"turn":1,"step":12,"callId":"cua-native-9","name":"cua","arguments":"{\"method\":\"screenshot\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":8}}"}} +{"type":"tool/result","data":{"turn":1,"step":12,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-9"},"toolCallId":"cua-native-9","content":[{"type":"text","text":"{\"pixelWidth\":1,\"pixelHeight\":1,\"windowId\":8,\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"coordinateSpace\":\"screenshot-pixels\"}"},{"type":"image","attachment":{"attachmentId":"sha256:999f1d1527ee7e79266f16add5430fff76b1225d742464a5b1ff1f02971bb8ee","mediaType":"image/png","width":1,"height":1,"bytes":90}}],"isError":false,"id":"{{message:27}}"}},"sourceEventSeqs":[69],"surfaceOp":"append"} {"type":"step/end","data":{"turn":1,"step":12}} {"type":"step/start","data":{"turn":1,"step":13}} -{"type":"assistant/message","data":{"turn":1,"step":13,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-activate-first","name":"cua","arguments":"{\"method\":\"activate\",\"params\":{\"appHandle\":\"fixture-app\"}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:28}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947554839,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947554839,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-activate-first","name":"cua","arguments":"{\"method\":\"activate\",\"params\":{\"appHandle\":\"fixture-app\"}}"}}},{"type":"chunk","time":1791947554839,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947554839,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} -{"type":"tool/call","data":{"turn":1,"step":13,"callId":"cua-native-activate-first","name":"cua","arguments":"{\"method\":\"activate\",\"params\":{\"appHandle\":\"fixture-app\"}}"}} -{"type":"tool/result","data":{"turn":1,"step":13,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-activate-first"},"toolCallId":"cua-native-activate-first","content":[{"type":"text","text":"{\"dispatched\":true,\"verified\":false,\"foreground\":{\"requestId\":\"cua-native-activate-first\",\"operation\":\"activate\",\"targetPid\":9,\"targetWindowId\":null,\"before\":{\"frontPid\":42,\"focusedWindowId\":7,\"frontWindowId\":99,\"frontWindowPid\":42},\"after\":{\"frontPid\":9,\"focusedWindowId\":7,\"frontWindowId\":7,\"frontWindowPid\":9},\"activationRequested\":true,\"raiseRequested\":false}}"}],"isError":false,"id":"{{message:29}}"}},"sourceEventSeqs":[74],"surfaceOp":"append"} +{"type":"assistant/message","data":{"turn":1,"step":13,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-10","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":8,\"snapshotId\":\"fixture-snapshot-3\",\"x\":0,\"y\":0}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:28}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947689896,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947689896,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-10","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":8,\"snapshotId\":\"fixture-snapshot-3\",\"x\":0,\"y\":0}}"}}},{"type":"chunk","time":1791947689896,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947689896,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} +{"type":"tool/call","data":{"turn":1,"step":13,"callId":"cua-native-10","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":8,\"snapshotId\":\"fixture-snapshot-3\",\"x\":0,\"y\":0}}"}} +{"type":"tool/result","data":{"turn":1,"step":13,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-10"},"toolCallId":"cua-native-10","content":[{"type":"text","text":"{\"dispatched\":true,\"verified\":false}"}],"isError":false,"id":"{{message:29}}"}},"sourceEventSeqs":[74],"surfaceOp":"append"} {"type":"step/end","data":{"turn":1,"step":13}} {"type":"step/start","data":{"turn":1,"step":14}} -{"type":"assistant/message","data":{"turn":1,"step":14,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-activate-repeat","name":"cua","arguments":"{\"method\":\"activate\",\"params\":{\"appHandle\":\"fixture-app\"}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:30}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947554856,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947554856,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-activate-repeat","name":"cua","arguments":"{\"method\":\"activate\",\"params\":{\"appHandle\":\"fixture-app\"}}"}}},{"type":"chunk","time":1791947554856,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947554856,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} -{"type":"tool/call","data":{"turn":1,"step":14,"callId":"cua-native-activate-repeat","name":"cua","arguments":"{\"method\":\"activate\",\"params\":{\"appHandle\":\"fixture-app\"}}"}} -{"type":"tool/result","data":{"turn":1,"step":14,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-activate-repeat"},"toolCallId":"cua-native-activate-repeat","content":[{"type":"text","text":"{\"dispatched\":true,\"verified\":false,\"foreground\":{\"requestId\":\"cua-native-activate-repeat\",\"operation\":\"activate\",\"targetPid\":9,\"targetWindowId\":null,\"before\":{\"frontPid\":9,\"focusedWindowId\":7,\"frontWindowId\":7,\"frontWindowPid\":9},\"after\":{\"frontPid\":9,\"focusedWindowId\":7,\"frontWindowId\":7,\"frontWindowPid\":9},\"activationRequested\":false,\"raiseRequested\":false}}"}],"isError":false,"id":"{{message:31}}"}},"sourceEventSeqs":[79],"surfaceOp":"append"} +{"type":"assistant/message","data":{"turn":1,"step":14,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-11","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":8}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:30}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947689918,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947689918,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-11","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":8}}"}}},{"type":"chunk","time":1791947689918,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947689918,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} +{"type":"tool/call","data":{"turn":1,"step":14,"callId":"cua-native-11","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":8}}"}} +{"type":"tool/result","data":{"turn":1,"step":14,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-11"},"toolCallId":"cua-native-11","content":[{"type":"text","text":"{\"appHandle\":\"fixture-app\",\"pid\":9,\"windowId\":8,\"snapshotId\":\"fixture-snapshot-4\",\"tree\":\"\",\"windows\":[{\"windowId\":8,\"pid\":9,\"appName\":\"Fixture\",\"title\":\"AXless canvas\",\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"isOnScreen\":true,\"onCurrentSpace\":true}],\"truncated\":false}"}],"isError":false,"id":"{{message:31}}"}},"sourceEventSeqs":[79],"surfaceOp":"append"} {"type":"step/end","data":{"turn":1,"step":14}} {"type":"step/start","data":{"turn":1,"step":15}} -{"type":"assistant/message","data":{"turn":1,"step":15,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-transition","name":"cua","arguments":"{\"method\":\"pressKey\",\"params\":{\"appHandle\":\"fixture-app\",\"key\":\"Return\"}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:32}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947554872,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947554872,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-transition","name":"cua","arguments":"{\"method\":\"pressKey\",\"params\":{\"appHandle\":\"fixture-app\",\"key\":\"Return\"}}"}}},{"type":"chunk","time":1791947554872,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947554872,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} -{"type":"tool/call","data":{"turn":1,"step":15,"callId":"cua-native-transition","name":"cua","arguments":"{\"method\":\"pressKey\",\"params\":{\"appHandle\":\"fixture-app\",\"key\":\"Return\"}}"}} -{"type":"tool/result","data":{"turn":1,"step":15,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-transition"},"toolCallId":"cua-native-transition","content":[{"type":"text","text":"{\"dispatched\":true,\"verified\":false}"}],"isError":false,"id":"{{message:33}}"}},"sourceEventSeqs":[84],"surfaceOp":"append"} +{"type":"assistant/message","data":{"turn":1,"step":15,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-show","name":"cua","arguments":"{\"method\":\"show\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:32}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947689940,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947689940,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-show","name":"cua","arguments":"{\"method\":\"show\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}}},{"type":"chunk","time":1791947689940,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947689940,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} +{"type":"tool/call","data":{"turn":1,"step":15,"callId":"cua-native-show","name":"cua","arguments":"{\"method\":\"show\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}} +{"type":"tool/result","data":{"turn":1,"step":15,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-show"},"toolCallId":"cua-native-show","content":[{"type":"text","text":"{\"dispatched\":true,\"verified\":false,\"windowId\":7,\"foregroundChanged\":false}"}],"isError":false,"id":"{{message:33}}"}},"sourceEventSeqs":[84],"surfaceOp":"append"} {"type":"step/end","data":{"turn":1,"step":15}} {"type":"step/start","data":{"turn":1,"step":16}} -{"type":"assistant/message","data":{"turn":1,"step":16,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-stale-read","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:34}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947554888,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947554888,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-stale-read","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}}},{"type":"chunk","time":1791947554888,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947554888,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} -{"type":"tool/call","data":{"turn":1,"step":16,"callId":"cua-native-stale-read","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}} -{"type":"tool/result","data":{"turn":1,"step":16,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-stale-read"},"toolCallId":"cua-native-stale-read","content":[{"type":"text","text":"Error: STALE_SNAPSHOT: Accessibility tree changed during observation; read fresh getAXState"}],"isError":true,"id":"{{message:35}}"}},"sourceEventSeqs":[89],"surfaceOp":"append"} +{"type":"assistant/message","data":{"turn":1,"step":16,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-activate-first","name":"cua","arguments":"{\"method\":\"activate\",\"params\":{\"appHandle\":\"fixture-app\"}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:34}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947689961,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947689961,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-activate-first","name":"cua","arguments":"{\"method\":\"activate\",\"params\":{\"appHandle\":\"fixture-app\"}}"}}},{"type":"chunk","time":1791947689961,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947689961,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} +{"type":"tool/call","data":{"turn":1,"step":16,"callId":"cua-native-activate-first","name":"cua","arguments":"{\"method\":\"activate\",\"params\":{\"appHandle\":\"fixture-app\"}}"}} +{"type":"tool/result","data":{"turn":1,"step":16,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-activate-first"},"toolCallId":"cua-native-activate-first","content":[{"type":"text","text":"{\"dispatched\":true,\"verified\":false,\"foreground\":{\"requestId\":\"cua-native-activate-first\",\"operation\":\"activate\",\"targetPid\":9,\"targetWindowId\":null,\"before\":{\"frontPid\":42,\"focusedWindowId\":7,\"frontWindowId\":99,\"frontWindowPid\":42},\"after\":{\"frontPid\":9,\"focusedWindowId\":7,\"frontWindowId\":7,\"frontWindowPid\":9},\"activationRequested\":true,\"raiseRequested\":false}}"}],"isError":false,"id":"{{message:35}}"}},"sourceEventSeqs":[89],"surfaceOp":"append"} {"type":"step/end","data":{"turn":1,"step":16}} {"type":"step/start","data":{"turn":1,"step":17}} -{"type":"assistant/message","data":{"turn":1,"step":17,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-stale-index","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"snapshotId\":\"fixture-snapshot-2\",\"elementIndex\":0}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:36}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947554904,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947554904,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-stale-index","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"snapshotId\":\"fixture-snapshot-2\",\"elementIndex\":0}}"}}},{"type":"chunk","time":1791947554904,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947554904,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} -{"type":"tool/call","data":{"turn":1,"step":17,"callId":"cua-native-stale-index","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"snapshotId\":\"fixture-snapshot-2\",\"elementIndex\":0}}"}} -{"type":"tool/result","data":{"turn":1,"step":17,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-stale-index"},"toolCallId":"cua-native-stale-index","content":[{"type":"text","text":"Error: STALE_SNAPSHOT: Read fresh state before acting"}],"isError":true,"id":"{{message:37}}"}},"sourceEventSeqs":[94],"surfaceOp":"append"} +{"type":"assistant/message","data":{"turn":1,"step":17,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-activate-repeat","name":"cua","arguments":"{\"method\":\"activate\",\"params\":{\"appHandle\":\"fixture-app\"}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:36}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947689983,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947689983,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-activate-repeat","name":"cua","arguments":"{\"method\":\"activate\",\"params\":{\"appHandle\":\"fixture-app\"}}"}}},{"type":"chunk","time":1791947689983,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947689983,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} +{"type":"tool/call","data":{"turn":1,"step":17,"callId":"cua-native-activate-repeat","name":"cua","arguments":"{\"method\":\"activate\",\"params\":{\"appHandle\":\"fixture-app\"}}"}} +{"type":"tool/result","data":{"turn":1,"step":17,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-activate-repeat"},"toolCallId":"cua-native-activate-repeat","content":[{"type":"text","text":"{\"dispatched\":true,\"verified\":false,\"foreground\":{\"requestId\":\"cua-native-activate-repeat\",\"operation\":\"activate\",\"targetPid\":9,\"targetWindowId\":null,\"before\":{\"frontPid\":9,\"focusedWindowId\":7,\"frontWindowId\":7,\"frontWindowPid\":9},\"after\":{\"frontPid\":9,\"focusedWindowId\":7,\"frontWindowId\":7,\"frontWindowPid\":9},\"activationRequested\":false,\"raiseRequested\":false}}"}],"isError":false,"id":"{{message:37}}"}},"sourceEventSeqs":[94],"surfaceOp":"append"} {"type":"step/end","data":{"turn":1,"step":17}} {"type":"step/start","data":{"turn":1,"step":18}} -{"type":"assistant/message","data":{"turn":1,"step":18,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-other-window","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":8,\"snapshotId\":\"fixture-snapshot-4\",\"x\":0,\"y\":0,\"holdMs\":800}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:38}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947554920,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947554920,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-other-window","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":8,\"snapshotId\":\"fixture-snapshot-4\",\"x\":0,\"y\":0,\"holdMs\":800}}"}}},{"type":"chunk","time":1791947554920,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947554920,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} -{"type":"tool/call","data":{"turn":1,"step":18,"callId":"cua-native-other-window","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":8,\"snapshotId\":\"fixture-snapshot-4\",\"x\":0,\"y\":0,\"holdMs\":800}}"}} -{"type":"tool/result","data":{"turn":1,"step":18,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-other-window"},"toolCallId":"cua-native-other-window","content":[{"type":"text","text":"{\"dispatched\":true,\"verified\":false}"}],"isError":false,"id":"{{message:39}}"}},"sourceEventSeqs":[99],"surfaceOp":"append"} +{"type":"assistant/message","data":{"turn":1,"step":18,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-transition","name":"cua","arguments":"{\"method\":\"pressKey\",\"params\":{\"appHandle\":\"fixture-app\",\"key\":\"Return\"}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:38}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947690002,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947690002,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-transition","name":"cua","arguments":"{\"method\":\"pressKey\",\"params\":{\"appHandle\":\"fixture-app\",\"key\":\"Return\"}}"}}},{"type":"chunk","time":1791947690002,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947690002,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} +{"type":"tool/call","data":{"turn":1,"step":18,"callId":"cua-native-transition","name":"cua","arguments":"{\"method\":\"pressKey\",\"params\":{\"appHandle\":\"fixture-app\",\"key\":\"Return\"}}"}} +{"type":"tool/result","data":{"turn":1,"step":18,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-transition"},"toolCallId":"cua-native-transition","content":[{"type":"text","text":"{\"dispatched\":true,\"verified\":false}"}],"isError":false,"id":"{{message:39}}"}},"sourceEventSeqs":[99],"surfaceOp":"append"} {"type":"step/end","data":{"turn":1,"step":18}} {"type":"step/start","data":{"turn":1,"step":19}} -{"type":"assistant/message","data":{"turn":1,"step":19,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-fresh-read","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:40}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947554936,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947554936,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-fresh-read","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}}},{"type":"chunk","time":1791947554936,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947554936,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} -{"type":"tool/call","data":{"turn":1,"step":19,"callId":"cua-native-fresh-read","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}} -{"type":"tool/result","data":{"turn":1,"step":19,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-fresh-read"},"toolCallId":"cua-native-fresh-read","content":[{"type":"text","text":"{\"appHandle\":\"fixture-app\",\"pid\":9,\"windowId\":7,\"snapshotId\":\"fixture-snapshot-5\",\"tree\":\"AX scope=window offsets=UTF-16\\n- [0] AXButton \\\"Increment\\\"\\n- [1] AXTextField valueWritable=true selectionWritable=true value=\\\"\\\" placeholder=\\\"Fixture text\\\"\\n- AXStaticText \\\"Count: 1\\\"\\n- [2] AXMenuBarItem \\\"File\\\"\\nMenu contents omitted; request includeMenus=true to expand.\",\"windows\":[{\"windowId\":7,\"pid\":9,\"appName\":\"Fixture\",\"title\":\"CUA fixture\",\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"isOnScreen\":true,\"onCurrentSpace\":true}],\"truncated\":false}"}],"isError":false,"id":"{{message:41}}"}},"sourceEventSeqs":[104],"surfaceOp":"append"} +{"type":"assistant/message","data":{"turn":1,"step":19,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-stale-read","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:40}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947690019,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947690019,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-stale-read","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}}},{"type":"chunk","time":1791947690019,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947690019,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} +{"type":"tool/call","data":{"turn":1,"step":19,"callId":"cua-native-stale-read","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}} +{"type":"tool/result","data":{"turn":1,"step":19,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-stale-read"},"toolCallId":"cua-native-stale-read","content":[{"type":"text","text":"Error: STALE_SNAPSHOT: Accessibility tree changed during observation; read fresh getAXState"}],"isError":true,"id":"{{message:41}}"}},"sourceEventSeqs":[104],"surfaceOp":"append"} {"type":"step/end","data":{"turn":1,"step":19}} {"type":"step/start","data":{"turn":1,"step":20}} -{"type":"assistant/message","data":{"turn":1,"step":20,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-fresh-click","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"snapshotId\":\"fixture-snapshot-5\",\"elementIndex\":0}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:42}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947554954,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947554954,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-fresh-click","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"snapshotId\":\"fixture-snapshot-5\",\"elementIndex\":0}}"}}},{"type":"chunk","time":1791947554954,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947554954,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} -{"type":"tool/call","data":{"turn":1,"step":20,"callId":"cua-native-fresh-click","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"snapshotId\":\"fixture-snapshot-5\",\"elementIndex\":0}}"}} -{"type":"tool/result","data":{"turn":1,"step":20,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-fresh-click"},"toolCallId":"cua-native-fresh-click","content":[{"type":"text","text":"{\"dispatched\":true,\"verified\":false}"}],"isError":false,"id":"{{message:43}}"}},"sourceEventSeqs":[109],"surfaceOp":"append"} +{"type":"assistant/message","data":{"turn":1,"step":20,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-stale-index","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"snapshotId\":\"fixture-snapshot-2\",\"elementIndex\":0}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:42}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947690040,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947690040,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-stale-index","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"snapshotId\":\"fixture-snapshot-2\",\"elementIndex\":0}}"}}},{"type":"chunk","time":1791947690040,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947690040,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} +{"type":"tool/call","data":{"turn":1,"step":20,"callId":"cua-native-stale-index","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"snapshotId\":\"fixture-snapshot-2\",\"elementIndex\":0}}"}} +{"type":"tool/result","data":{"turn":1,"step":20,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-stale-index"},"toolCallId":"cua-native-stale-index","content":[{"type":"text","text":"Error: STALE_SNAPSHOT: Read fresh state before acting"}],"isError":true,"id":"{{message:43}}"}},"sourceEventSeqs":[109],"surfaceOp":"append"} {"type":"step/end","data":{"turn":1,"step":20}} {"type":"step/start","data":{"turn":1,"step":21}} -{"type":"assistant/message","data":{"turn":1,"step":21,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-verify","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:44}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947554970,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947554970,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-verify","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}}},{"type":"chunk","time":1791947554970,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947554970,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} -{"type":"tool/call","data":{"turn":1,"step":21,"callId":"cua-native-verify","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}} -{"type":"tool/result","data":{"turn":1,"step":21,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-verify"},"toolCallId":"cua-native-verify","content":[{"type":"text","text":"{\"appHandle\":\"fixture-app\",\"pid\":9,\"windowId\":7,\"snapshotId\":\"fixture-snapshot-6\",\"tree\":\"AX scope=window offsets=UTF-16\\n- [0] AXButton \\\"Increment\\\"\\n- [1] AXTextField valueWritable=true selectionWritable=true value=\\\"\\\" placeholder=\\\"Fixture text\\\"\\n- AXStaticText \\\"Count: 2\\\"\\n- [2] AXMenuBarItem \\\"File\\\"\\nMenu contents omitted; request includeMenus=true to expand.\",\"windows\":[{\"windowId\":7,\"pid\":9,\"appName\":\"Fixture\",\"title\":\"CUA fixture\",\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"isOnScreen\":true,\"onCurrentSpace\":true}],\"truncated\":false}"}],"isError":false,"id":"{{message:45}}"}},"sourceEventSeqs":[114],"surfaceOp":"append"} +{"type":"assistant/message","data":{"turn":1,"step":21,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-other-window","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":8,\"snapshotId\":\"fixture-snapshot-4\",\"x\":0,\"y\":0,\"holdMs\":800}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:44}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947690064,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947690064,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-other-window","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":8,\"snapshotId\":\"fixture-snapshot-4\",\"x\":0,\"y\":0,\"holdMs\":800}}"}}},{"type":"chunk","time":1791947690064,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947690064,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} +{"type":"tool/call","data":{"turn":1,"step":21,"callId":"cua-native-other-window","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":8,\"snapshotId\":\"fixture-snapshot-4\",\"x\":0,\"y\":0,\"holdMs\":800}}"}} +{"type":"tool/result","data":{"turn":1,"step":21,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-other-window"},"toolCallId":"cua-native-other-window","content":[{"type":"text","text":"{\"dispatched\":true,\"verified\":false}"}],"isError":false,"id":"{{message:45}}"}},"sourceEventSeqs":[114],"surfaceOp":"append"} {"type":"step/end","data":{"turn":1,"step":21}} {"type":"step/start","data":{"turn":1,"step":22}} -{"type":"assistant/message","data":{"turn":1,"step":22,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-geometry-observe","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":9}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:46}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947554987,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947554987,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-geometry-observe","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":9}}"}}},{"type":"chunk","time":1791947554987,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947554987,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} -{"type":"tool/call","data":{"turn":1,"step":22,"callId":"cua-native-geometry-observe","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":9}}"}} -{"type":"tool/result","data":{"turn":1,"step":22,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-geometry-observe"},"toolCallId":"cua-native-geometry-observe","content":[{"type":"text","text":"{\"appHandle\":\"fixture-app\",\"pid\":9,\"windowId\":9,\"snapshotId\":\"fixture-snapshot-7\",\"tree\":\"AX scope=window offsets=UTF-16\\n- [0] AXButton \\\"Increment\\\"\\n- [1] AXTextField valueWritable=true selectionWritable=true value=\\\"\\\" placeholder=\\\"Fixture text\\\"\\n- AXStaticText \\\"Count: 2\\\"\\n- [2] AXMenuBarItem \\\"File\\\"\\nMenu contents omitted; request includeMenus=true to expand.\",\"windows\":[{\"windowId\":9,\"pid\":9,\"appName\":\"Fixture\",\"title\":\"Transformed window\",\"bounds\":{\"x\":16,\"y\":512,\"width\":131,\"height\":145},\"isOnScreen\":true,\"onCurrentSpace\":true}],\"truncated\":false}"}],"isError":false,"id":"{{message:47}}"}},"sourceEventSeqs":[119],"surfaceOp":"append"} +{"type":"assistant/message","data":{"turn":1,"step":22,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-fresh-read","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:46}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947690087,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947690087,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-fresh-read","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}}},{"type":"chunk","time":1791947690087,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947690087,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} +{"type":"tool/call","data":{"turn":1,"step":22,"callId":"cua-native-fresh-read","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}} +{"type":"tool/result","data":{"turn":1,"step":22,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-fresh-read"},"toolCallId":"cua-native-fresh-read","content":[{"type":"text","text":"{\"appHandle\":\"fixture-app\",\"pid\":9,\"windowId\":7,\"snapshotId\":\"fixture-snapshot-5\",\"tree\":\"AX scope=window offsets=UTF-16\\n- [0] AXButton \\\"Increment\\\"\\n- [1] AXTextField valueWritable=true selectionWritable=true value=\\\"\\\" placeholder=\\\"Fixture text\\\"\\n- AXStaticText \\\"Count: 1\\\"\\n- [2] AXMenuBarItem \\\"File\\\"\\nMenu contents omitted; request includeMenus=true to expand.\",\"windows\":[{\"windowId\":7,\"pid\":9,\"appName\":\"Fixture\",\"title\":\"CUA fixture\",\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"isOnScreen\":true,\"onCurrentSpace\":true}],\"truncated\":false}"}],"isError":false,"id":"{{message:47}}"}},"sourceEventSeqs":[119],"surfaceOp":"append"} {"type":"step/end","data":{"turn":1,"step":22}} {"type":"step/start","data":{"turn":1,"step":23}} -{"type":"assistant/message","data":{"turn":1,"step":23,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-geometry-capture","name":"cua","arguments":"{\"method\":\"screenshot\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":9}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:48}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947555003,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947555003,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-geometry-capture","name":"cua","arguments":"{\"method\":\"screenshot\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":9}}"}}},{"type":"chunk","time":1791947555003,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947555003,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} -{"type":"tool/call","data":{"turn":1,"step":23,"callId":"cua-native-geometry-capture","name":"cua","arguments":"{\"method\":\"screenshot\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":9}}"}} -{"type":"tool/result","data":{"turn":1,"step":23,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-geometry-capture"},"toolCallId":"cua-native-geometry-capture","content":[{"type":"text","text":"Error: WINDOW_CHANGED: Window display and accessibility bounds disagree. Read fresh state; if unchanged, use AX actions or explicitly restore the window before capture or coordinate input"}],"isError":true,"id":"{{message:49}}"}},"sourceEventSeqs":[124],"surfaceOp":"append"} +{"type":"assistant/message","data":{"turn":1,"step":23,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-fresh-click","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"snapshotId\":\"fixture-snapshot-5\",\"elementIndex\":0}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:48}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947690111,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947690111,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-fresh-click","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"snapshotId\":\"fixture-snapshot-5\",\"elementIndex\":0}}"}}},{"type":"chunk","time":1791947690111,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947690111,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} +{"type":"tool/call","data":{"turn":1,"step":23,"callId":"cua-native-fresh-click","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"snapshotId\":\"fixture-snapshot-5\",\"elementIndex\":0}}"}} +{"type":"tool/result","data":{"turn":1,"step":23,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-fresh-click"},"toolCallId":"cua-native-fresh-click","content":[{"type":"text","text":"{\"dispatched\":true,\"verified\":false}"}],"isError":false,"id":"{{message:49}}"}},"sourceEventSeqs":[124],"surfaceOp":"append"} {"type":"step/end","data":{"turn":1,"step":23}} {"type":"step/start","data":{"turn":1,"step":24}} -{"type":"assistant/message","data":{"turn":1,"step":24,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-geometry-click","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":9,\"snapshotId\":\"fixture-snapshot-7\",\"x\":20,\"y\":20,\"relative\":true}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:50}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947555020,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947555020,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-geometry-click","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":9,\"snapshotId\":\"fixture-snapshot-7\",\"x\":20,\"y\":20,\"relative\":true}}"}}},{"type":"chunk","time":1791947555020,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947555020,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} -{"type":"tool/call","data":{"turn":1,"step":24,"callId":"cua-native-geometry-click","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":9,\"snapshotId\":\"fixture-snapshot-7\",\"x\":20,\"y\":20,\"relative\":true}}"}} -{"type":"tool/result","data":{"turn":1,"step":24,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-geometry-click"},"toolCallId":"cua-native-geometry-click","content":[{"type":"text","text":"Error: WINDOW_CHANGED: Window display and accessibility bounds disagree. Read fresh state; if unchanged, use AX actions or explicitly restore the window before capture or coordinate input"}],"isError":true,"id":"{{message:51}}"}},"sourceEventSeqs":[129],"surfaceOp":"append"} +{"type":"assistant/message","data":{"turn":1,"step":24,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-verify","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:50}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947690141,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947690141,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-verify","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}}},{"type":"chunk","time":1791947690141,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947690141,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} +{"type":"tool/call","data":{"turn":1,"step":24,"callId":"cua-native-verify","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}} +{"type":"tool/result","data":{"turn":1,"step":24,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-verify"},"toolCallId":"cua-native-verify","content":[{"type":"text","text":"{\"appHandle\":\"fixture-app\",\"pid\":9,\"windowId\":7,\"snapshotId\":\"fixture-snapshot-6\",\"tree\":\"AX scope=window offsets=UTF-16\\n- [0] AXButton \\\"Increment\\\"\\n- [1] AXTextField valueWritable=true selectionWritable=true value=\\\"\\\" placeholder=\\\"Fixture text\\\"\\n- AXStaticText \\\"Count: 2\\\"\\n- [2] AXMenuBarItem \\\"File\\\"\\nMenu contents omitted; request includeMenus=true to expand.\",\"windows\":[{\"windowId\":7,\"pid\":9,\"appName\":\"Fixture\",\"title\":\"CUA fixture\",\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"isOnScreen\":true,\"onCurrentSpace\":true}],\"truncated\":false}"}],"isError":false,"id":"{{message:51}}"}},"sourceEventSeqs":[129],"surfaceOp":"append"} {"type":"step/end","data":{"turn":1,"step":24}} {"type":"step/start","data":{"turn":1,"step":25}} -{"type":"assistant/message","data":{"turn":1,"step":25,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-geometry-scroll","name":"cua","arguments":"{\"method\":\"scroll\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":9,\"x\":20,\"y\":20,\"deltaLinesX\":0,\"deltaLinesY\":-1,\"relative\":true}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:52}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947555037,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947555037,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-geometry-scroll","name":"cua","arguments":"{\"method\":\"scroll\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":9,\"x\":20,\"y\":20,\"deltaLinesX\":0,\"deltaLinesY\":-1,\"relative\":true}}"}}},{"type":"chunk","time":1791947555037,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947555037,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} -{"type":"tool/call","data":{"turn":1,"step":25,"callId":"cua-native-geometry-scroll","name":"cua","arguments":"{\"method\":\"scroll\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":9,\"x\":20,\"y\":20,\"deltaLinesX\":0,\"deltaLinesY\":-1,\"relative\":true}}"}} -{"type":"tool/result","data":{"turn":1,"step":25,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-geometry-scroll"},"toolCallId":"cua-native-geometry-scroll","content":[{"type":"text","text":"Error: WINDOW_CHANGED: Window display and accessibility bounds disagree. Read fresh state; if unchanged, use AX actions or explicitly restore the window before capture or coordinate input"}],"isError":true,"id":"{{message:53}}"}},"sourceEventSeqs":[134],"surfaceOp":"append"} +{"type":"assistant/message","data":{"turn":1,"step":25,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-geometry-observe","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":9}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:52}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947690162,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947690162,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-geometry-observe","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":9}}"}}},{"type":"chunk","time":1791947690162,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947690162,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} +{"type":"tool/call","data":{"turn":1,"step":25,"callId":"cua-native-geometry-observe","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":9}}"}} +{"type":"tool/result","data":{"turn":1,"step":25,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-geometry-observe"},"toolCallId":"cua-native-geometry-observe","content":[{"type":"text","text":"{\"appHandle\":\"fixture-app\",\"pid\":9,\"windowId\":9,\"snapshotId\":\"fixture-snapshot-7\",\"tree\":\"AX scope=window offsets=UTF-16\\n- [0] AXButton \\\"Increment\\\"\\n- [1] AXTextField valueWritable=true selectionWritable=true value=\\\"\\\" placeholder=\\\"Fixture text\\\"\\n- AXStaticText \\\"Count: 2\\\"\\n- [2] AXMenuBarItem \\\"File\\\"\\nMenu contents omitted; request includeMenus=true to expand.\",\"windows\":[{\"windowId\":9,\"pid\":9,\"appName\":\"Fixture\",\"title\":\"Transformed window\",\"bounds\":{\"x\":16,\"y\":512,\"width\":131,\"height\":145},\"isOnScreen\":true,\"onCurrentSpace\":true}],\"truncated\":false}"}],"isError":false,"id":"{{message:53}}"}},"sourceEventSeqs":[134],"surfaceOp":"append"} {"type":"step/end","data":{"turn":1,"step":25}} {"type":"step/start","data":{"turn":1,"step":26}} -{"type":"assistant/message","data":{"turn":1,"step":26,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-geometry-drag","name":"cua","arguments":"{\"method\":\"drag\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":9,\"fromX\":20,\"fromY\":20,\"toX\":30,\"toY\":30,\"relative\":true}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:54}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947555054,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947555054,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-geometry-drag","name":"cua","arguments":"{\"method\":\"drag\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":9,\"fromX\":20,\"fromY\":20,\"toX\":30,\"toY\":30,\"relative\":true}}"}}},{"type":"chunk","time":1791947555054,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947555054,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} -{"type":"tool/call","data":{"turn":1,"step":26,"callId":"cua-native-geometry-drag","name":"cua","arguments":"{\"method\":\"drag\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":9,\"fromX\":20,\"fromY\":20,\"toX\":30,\"toY\":30,\"relative\":true}}"}} -{"type":"tool/result","data":{"turn":1,"step":26,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-geometry-drag"},"toolCallId":"cua-native-geometry-drag","content":[{"type":"text","text":"Error: WINDOW_CHANGED: Window display and accessibility bounds disagree. Read fresh state; if unchanged, use AX actions or explicitly restore the window before capture or coordinate input"}],"isError":true,"id":"{{message:55}}"}},"sourceEventSeqs":[139],"surfaceOp":"append"} +{"type":"assistant/message","data":{"turn":1,"step":26,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-geometry-capture","name":"cua","arguments":"{\"method\":\"screenshot\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":9}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:54}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947690186,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947690186,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-geometry-capture","name":"cua","arguments":"{\"method\":\"screenshot\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":9}}"}}},{"type":"chunk","time":1791947690186,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947690186,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} +{"type":"tool/call","data":{"turn":1,"step":26,"callId":"cua-native-geometry-capture","name":"cua","arguments":"{\"method\":\"screenshot\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":9}}"}} +{"type":"tool/result","data":{"turn":1,"step":26,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-geometry-capture"},"toolCallId":"cua-native-geometry-capture","content":[{"type":"text","text":"Error: WINDOW_CHANGED: Window display and accessibility bounds disagree. Read fresh state; if unchanged, use AX actions or explicitly restore the window before capture or coordinate input"}],"isError":true,"id":"{{message:55}}"}},"sourceEventSeqs":[139],"surfaceOp":"append"} {"type":"step/end","data":{"turn":1,"step":26}} {"type":"step/start","data":{"turn":1,"step":27}} -{"type":"assistant/message","data":{"turn":1,"step":27,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-geometry-ax","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"snapshotId\":\"fixture-snapshot-7\",\"elementIndex\":0}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:56}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947555071,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947555071,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-geometry-ax","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"snapshotId\":\"fixture-snapshot-7\",\"elementIndex\":0}}"}}},{"type":"chunk","time":1791947555071,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947555071,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} -{"type":"tool/call","data":{"turn":1,"step":27,"callId":"cua-native-geometry-ax","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"snapshotId\":\"fixture-snapshot-7\",\"elementIndex\":0}}"}} -{"type":"tool/result","data":{"turn":1,"step":27,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-geometry-ax"},"toolCallId":"cua-native-geometry-ax","content":[{"type":"text","text":"{\"dispatched\":true,\"verified\":false}"}],"isError":false,"id":"{{message:57}}"}},"sourceEventSeqs":[144],"surfaceOp":"append"} +{"type":"assistant/message","data":{"turn":1,"step":27,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-geometry-click","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":9,\"snapshotId\":\"fixture-snapshot-7\",\"x\":20,\"y\":20,\"relative\":true}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:56}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947690221,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947690221,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-geometry-click","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":9,\"snapshotId\":\"fixture-snapshot-7\",\"x\":20,\"y\":20,\"relative\":true}}"}}},{"type":"chunk","time":1791947690221,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947690221,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} +{"type":"tool/call","data":{"turn":1,"step":27,"callId":"cua-native-geometry-click","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":9,\"snapshotId\":\"fixture-snapshot-7\",\"x\":20,\"y\":20,\"relative\":true}}"}} +{"type":"tool/result","data":{"turn":1,"step":27,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-geometry-click"},"toolCallId":"cua-native-geometry-click","content":[{"type":"text","text":"Error: WINDOW_CHANGED: Window display and accessibility bounds disagree. Read fresh state; if unchanged, use AX actions or explicitly restore the window before capture or coordinate input"}],"isError":true,"id":"{{message:57}}"}},"sourceEventSeqs":[144],"surfaceOp":"append"} {"type":"step/end","data":{"turn":1,"step":27}} {"type":"step/start","data":{"turn":1,"step":28}} -{"type":"assistant/message","data":{"turn":1,"step":28,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-geometry-verify","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":9}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:58}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947555087,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947555087,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-geometry-verify","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":9}}"}}},{"type":"chunk","time":1791947555087,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947555087,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} -{"type":"tool/call","data":{"turn":1,"step":28,"callId":"cua-native-geometry-verify","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":9}}"}} -{"type":"tool/result","data":{"turn":1,"step":28,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-geometry-verify"},"toolCallId":"cua-native-geometry-verify","content":[{"type":"text","text":"{\"appHandle\":\"fixture-app\",\"pid\":9,\"windowId\":9,\"snapshotId\":\"fixture-snapshot-8\",\"tree\":\"AX scope=window offsets=UTF-16\\n- [0] AXButton \\\"Increment\\\"\\n- [1] AXTextField valueWritable=true selectionWritable=true value=\\\"\\\" placeholder=\\\"Fixture text\\\"\\n- AXStaticText \\\"Count: 3\\\"\\n- [2] AXMenuBarItem \\\"File\\\"\\nMenu contents omitted; request includeMenus=true to expand.\",\"windows\":[{\"windowId\":9,\"pid\":9,\"appName\":\"Fixture\",\"title\":\"Transformed window\",\"bounds\":{\"x\":16,\"y\":512,\"width\":131,\"height\":145},\"isOnScreen\":true,\"onCurrentSpace\":true}],\"truncated\":false}"}],"isError":false,"id":"{{message:59}}"}},"sourceEventSeqs":[149],"surfaceOp":"append"} +{"type":"assistant/message","data":{"turn":1,"step":28,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-geometry-scroll","name":"cua","arguments":"{\"method\":\"scroll\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":9,\"x\":20,\"y\":20,\"deltaLinesX\":0,\"deltaLinesY\":-1,\"relative\":true}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:58}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947690248,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947690248,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-geometry-scroll","name":"cua","arguments":"{\"method\":\"scroll\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":9,\"x\":20,\"y\":20,\"deltaLinesX\":0,\"deltaLinesY\":-1,\"relative\":true}}"}}},{"type":"chunk","time":1791947690248,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947690248,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} +{"type":"tool/call","data":{"turn":1,"step":28,"callId":"cua-native-geometry-scroll","name":"cua","arguments":"{\"method\":\"scroll\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":9,\"x\":20,\"y\":20,\"deltaLinesX\":0,\"deltaLinesY\":-1,\"relative\":true}}"}} +{"type":"tool/result","data":{"turn":1,"step":28,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-geometry-scroll"},"toolCallId":"cua-native-geometry-scroll","content":[{"type":"text","text":"Error: WINDOW_CHANGED: Window display and accessibility bounds disagree. Read fresh state; if unchanged, use AX actions or explicitly restore the window before capture or coordinate input"}],"isError":true,"id":"{{message:59}}"}},"sourceEventSeqs":[149],"surfaceOp":"append"} {"type":"step/end","data":{"turn":1,"step":28}} {"type":"step/start","data":{"turn":1,"step":29}} -{"type":"assistant/message","data":{"turn":1,"step":29,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-capture-busy-arm","name":"cua","arguments":"{\"method\":\"pressKey\",\"params\":{\"appHandle\":\"fixture-app\",\"key\":\"F11\"}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:60}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947555103,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947555103,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-capture-busy-arm","name":"cua","arguments":"{\"method\":\"pressKey\",\"params\":{\"appHandle\":\"fixture-app\",\"key\":\"F11\"}}"}}},{"type":"chunk","time":1791947555103,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947555103,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} -{"type":"tool/call","data":{"turn":1,"step":29,"callId":"cua-capture-busy-arm","name":"cua","arguments":"{\"method\":\"pressKey\",\"params\":{\"appHandle\":\"fixture-app\",\"key\":\"F11\"}}"}} -{"type":"tool/result","data":{"turn":1,"step":29,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-capture-busy-arm"},"toolCallId":"cua-capture-busy-arm","content":[{"type":"text","text":"{\"dispatched\":true,\"verified\":false}"}],"isError":false,"id":"{{message:61}}"}},"sourceEventSeqs":[154],"surfaceOp":"append"} +{"type":"assistant/message","data":{"turn":1,"step":29,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-geometry-drag","name":"cua","arguments":"{\"method\":\"drag\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":9,\"fromX\":20,\"fromY\":20,\"toX\":30,\"toY\":30,\"relative\":true}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:60}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947690267,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947690267,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-geometry-drag","name":"cua","arguments":"{\"method\":\"drag\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":9,\"fromX\":20,\"fromY\":20,\"toX\":30,\"toY\":30,\"relative\":true}}"}}},{"type":"chunk","time":1791947690267,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947690267,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} +{"type":"tool/call","data":{"turn":1,"step":29,"callId":"cua-native-geometry-drag","name":"cua","arguments":"{\"method\":\"drag\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":9,\"fromX\":20,\"fromY\":20,\"toX\":30,\"toY\":30,\"relative\":true}}"}} +{"type":"tool/result","data":{"turn":1,"step":29,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-geometry-drag"},"toolCallId":"cua-native-geometry-drag","content":[{"type":"text","text":"Error: WINDOW_CHANGED: Window display and accessibility bounds disagree. Read fresh state; if unchanged, use AX actions or explicitly restore the window before capture or coordinate input"}],"isError":true,"id":"{{message:61}}"}},"sourceEventSeqs":[154],"surfaceOp":"append"} {"type":"step/end","data":{"turn":1,"step":29}} {"type":"step/start","data":{"turn":1,"step":30}} -{"type":"assistant/message","data":{"turn":1,"step":30,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-capture-busy","name":"cua","arguments":"{\"method\":\"screenshot\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:62}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947555120,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947555120,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-capture-busy","name":"cua","arguments":"{\"method\":\"screenshot\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}}},{"type":"chunk","time":1791947555120,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947555120,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} -{"type":"tool/call","data":{"turn":1,"step":30,"callId":"cua-capture-busy","name":"cua","arguments":"{\"method\":\"screenshot\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}} -{"type":"tool/result","data":{"turn":1,"step":30,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-capture-busy"},"toolCallId":"cua-capture-busy","content":[{"type":"text","text":"Error: CAPTURE_BUSY: A previous system capture is still pending; other native operations remain available\nScreenshot diagnostics:\n[cua-screenshot] {\"requestId\":\"cua-capture-busy\",\"stage\":\"helper.failed\",\"elapsedMs\":1,\"errorCode\":\"CAPTURE_BUSY\"}"}],"isError":true,"id":"{{message:63}}"}},"sourceEventSeqs":[159],"surfaceOp":"append"} +{"type":"assistant/message","data":{"turn":1,"step":30,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-geometry-ax","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"snapshotId\":\"fixture-snapshot-7\",\"elementIndex\":0}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:62}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947690294,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947690294,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-geometry-ax","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"snapshotId\":\"fixture-snapshot-7\",\"elementIndex\":0}}"}}},{"type":"chunk","time":1791947690294,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947690294,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} +{"type":"tool/call","data":{"turn":1,"step":30,"callId":"cua-native-geometry-ax","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"snapshotId\":\"fixture-snapshot-7\",\"elementIndex\":0}}"}} +{"type":"tool/result","data":{"turn":1,"step":30,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-geometry-ax"},"toolCallId":"cua-native-geometry-ax","content":[{"type":"text","text":"{\"dispatched\":true,\"verified\":false}"}],"isError":false,"id":"{{message:63}}"}},"sourceEventSeqs":[159],"surfaceOp":"append"} {"type":"step/end","data":{"turn":1,"step":30}} {"type":"step/start","data":{"turn":1,"step":31}} -{"type":"assistant/message","data":{"turn":1,"step":31,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-capture-busy-ax","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:64}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947555141,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947555141,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-capture-busy-ax","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}}},{"type":"chunk","time":1791947555141,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947555141,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} -{"type":"tool/call","data":{"turn":1,"step":31,"callId":"cua-capture-busy-ax","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}} -{"type":"tool/result","data":{"turn":1,"step":31,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-capture-busy-ax"},"toolCallId":"cua-capture-busy-ax","content":[{"type":"text","text":"{\"appHandle\":\"fixture-app\",\"pid\":9,\"windowId\":7,\"snapshotId\":\"fixture-snapshot-9\",\"tree\":\"AX scope=window offsets=UTF-16\\n- [0] AXButton \\\"Increment\\\"\\n- [1] AXTextField valueWritable=true selectionWritable=true value=\\\"\\\" placeholder=\\\"Fixture text\\\"\\n- AXStaticText \\\"Count: 3\\\"\\n- [2] AXMenuBarItem \\\"File\\\"\\nMenu contents omitted; request includeMenus=true to expand.\",\"windows\":[{\"windowId\":7,\"pid\":9,\"appName\":\"Fixture\",\"title\":\"CUA fixture\",\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"isOnScreen\":true,\"onCurrentSpace\":true}],\"truncated\":false}"}],"isError":false,"id":"{{message:65}}"}},"sourceEventSeqs":[164],"surfaceOp":"append"} +{"type":"assistant/message","data":{"turn":1,"step":31,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-geometry-verify","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":9}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:64}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947690314,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947690314,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-geometry-verify","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":9}}"}}},{"type":"chunk","time":1791947690314,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947690314,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} +{"type":"tool/call","data":{"turn":1,"step":31,"callId":"cua-native-geometry-verify","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":9}}"}} +{"type":"tool/result","data":{"turn":1,"step":31,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-geometry-verify"},"toolCallId":"cua-native-geometry-verify","content":[{"type":"text","text":"{\"appHandle\":\"fixture-app\",\"pid\":9,\"windowId\":9,\"snapshotId\":\"fixture-snapshot-8\",\"tree\":\"AX scope=window offsets=UTF-16\\n- [0] AXButton \\\"Increment\\\"\\n- [1] AXTextField valueWritable=true selectionWritable=true value=\\\"\\\" placeholder=\\\"Fixture text\\\"\\n- AXStaticText \\\"Count: 3\\\"\\n- [2] AXMenuBarItem \\\"File\\\"\\nMenu contents omitted; request includeMenus=true to expand.\",\"windows\":[{\"windowId\":9,\"pid\":9,\"appName\":\"Fixture\",\"title\":\"Transformed window\",\"bounds\":{\"x\":16,\"y\":512,\"width\":131,\"height\":145},\"isOnScreen\":true,\"onCurrentSpace\":true}],\"truncated\":false}"}],"isError":false,"id":"{{message:65}}"}},"sourceEventSeqs":[164],"surfaceOp":"append"} {"type":"step/end","data":{"turn":1,"step":31}} {"type":"step/start","data":{"turn":1,"step":32}} -{"type":"assistant/message","data":{"turn":1,"step":32,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-capture-callback","name":"cua","arguments":"{\"method\":\"pressKey\",\"params\":{\"appHandle\":\"fixture-app\",\"key\":\"F10\"}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:66}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947555157,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947555157,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-capture-callback","name":"cua","arguments":"{\"method\":\"pressKey\",\"params\":{\"appHandle\":\"fixture-app\",\"key\":\"F10\"}}"}}},{"type":"chunk","time":1791947555157,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947555157,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} -{"type":"tool/call","data":{"turn":1,"step":32,"callId":"cua-capture-callback","name":"cua","arguments":"{\"method\":\"pressKey\",\"params\":{\"appHandle\":\"fixture-app\",\"key\":\"F10\"}}"}} -{"type":"tool/result","data":{"turn":1,"step":32,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-capture-callback"},"toolCallId":"cua-capture-callback","content":[{"type":"text","text":"{\"dispatched\":true,\"verified\":false}"}],"isError":false,"id":"{{message:67}}"}},"sourceEventSeqs":[169],"surfaceOp":"append"} +{"type":"assistant/message","data":{"turn":1,"step":32,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-capture-busy-arm","name":"cua","arguments":"{\"method\":\"pressKey\",\"params\":{\"appHandle\":\"fixture-app\",\"key\":\"F11\"}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:66}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947690340,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947690340,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-capture-busy-arm","name":"cua","arguments":"{\"method\":\"pressKey\",\"params\":{\"appHandle\":\"fixture-app\",\"key\":\"F11\"}}"}}},{"type":"chunk","time":1791947690340,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947690340,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} +{"type":"tool/call","data":{"turn":1,"step":32,"callId":"cua-capture-busy-arm","name":"cua","arguments":"{\"method\":\"pressKey\",\"params\":{\"appHandle\":\"fixture-app\",\"key\":\"F11\"}}"}} +{"type":"tool/result","data":{"turn":1,"step":32,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-capture-busy-arm"},"toolCallId":"cua-capture-busy-arm","content":[{"type":"text","text":"{\"dispatched\":true,\"verified\":false}"}],"isError":false,"id":"{{message:67}}"}},"sourceEventSeqs":[169],"surfaceOp":"append"} {"type":"step/end","data":{"turn":1,"step":32}} {"type":"step/start","data":{"turn":1,"step":33}} -{"type":"assistant/message","data":{"turn":1,"step":33,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-capture-recovered","name":"cua","arguments":"{\"method\":\"screenshot\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:68}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947555173,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947555173,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-capture-recovered","name":"cua","arguments":"{\"method\":\"screenshot\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}}},{"type":"chunk","time":1791947555173,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947555173,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} -{"type":"tool/call","data":{"turn":1,"step":33,"callId":"cua-capture-recovered","name":"cua","arguments":"{\"method\":\"screenshot\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}} -{"type":"tool/result","data":{"turn":1,"step":33,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-capture-recovered"},"toolCallId":"cua-capture-recovered","content":[{"type":"text","text":"{\"pixelWidth\":1,\"pixelHeight\":1,\"windowId\":7,\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"coordinateSpace\":\"screenshot-pixels\"}"},{"type":"image","attachment":{"attachmentId":"sha256:999f1d1527ee7e79266f16add5430fff76b1225d742464a5b1ff1f02971bb8ee","mediaType":"image/png","width":1,"height":1,"bytes":90}}],"isError":false,"id":"{{message:69}}"}},"sourceEventSeqs":[174],"surfaceOp":"append"} +{"type":"assistant/message","data":{"turn":1,"step":33,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-capture-busy","name":"cua","arguments":"{\"method\":\"screenshot\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:68}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947690365,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947690365,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-capture-busy","name":"cua","arguments":"{\"method\":\"screenshot\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}}},{"type":"chunk","time":1791947690365,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947690365,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} +{"type":"tool/call","data":{"turn":1,"step":33,"callId":"cua-capture-busy","name":"cua","arguments":"{\"method\":\"screenshot\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}} +{"type":"tool/result","data":{"turn":1,"step":33,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-capture-busy"},"toolCallId":"cua-capture-busy","content":[{"type":"text","text":"Error: CAPTURE_BUSY: A previous system capture is still pending; other native operations remain available\nScreenshot diagnostics:\n[cua-screenshot] {\"requestId\":\"cua-capture-busy\",\"stage\":\"helper.failed\",\"elapsedMs\":1,\"errorCode\":\"CAPTURE_BUSY\"}"}],"isError":true,"id":"{{message:69}}"}},"sourceEventSeqs":[174],"surfaceOp":"append"} {"type":"step/end","data":{"turn":1,"step":33}} {"type":"step/start","data":{"turn":1,"step":34}} -{"type":"assistant/message","data":{"turn":1,"step":34,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-hover-enter","name":"cua","arguments":"{\"method\":\"hover\",\"params\":{\"appHandle\":\"fixture-app\",\"snapshotId\":\"fixture-snapshot-9\",\"elementIndex\":0}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:70}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947555199,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947555199,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-hover-enter","name":"cua","arguments":"{\"method\":\"hover\",\"params\":{\"appHandle\":\"fixture-app\",\"snapshotId\":\"fixture-snapshot-9\",\"elementIndex\":0}}"}}},{"type":"chunk","time":1791947555199,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947555199,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} -{"type":"tool/call","data":{"turn":1,"step":34,"callId":"cua-hover-enter","name":"cua","arguments":"{\"method\":\"hover\",\"params\":{\"appHandle\":\"fixture-app\",\"snapshotId\":\"fixture-snapshot-9\",\"elementIndex\":0}}"}} -{"type":"tool/result","data":{"turn":1,"step":34,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-hover-enter"},"toolCallId":"cua-hover-enter","content":[{"type":"text","text":"{\"dispatched\":true,\"verified\":false,\"hoverRoute\":\"accessibility\",\"systemPointerMoved\":false}"}],"isError":false,"id":"{{message:71}}"}},"sourceEventSeqs":[179],"surfaceOp":"append"} +{"type":"assistant/message","data":{"turn":1,"step":34,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-capture-busy-ax","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:70}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947690394,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947690394,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-capture-busy-ax","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}}},{"type":"chunk","time":1791947690394,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947690394,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} +{"type":"tool/call","data":{"turn":1,"step":34,"callId":"cua-capture-busy-ax","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}} +{"type":"tool/result","data":{"turn":1,"step":34,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-capture-busy-ax"},"toolCallId":"cua-capture-busy-ax","content":[{"type":"text","text":"{\"appHandle\":\"fixture-app\",\"pid\":9,\"windowId\":7,\"snapshotId\":\"fixture-snapshot-9\",\"tree\":\"AX scope=window offsets=UTF-16\\n- [0] AXButton \\\"Increment\\\"\\n- [1] AXTextField valueWritable=true selectionWritable=true value=\\\"\\\" placeholder=\\\"Fixture text\\\"\\n- AXStaticText \\\"Count: 3\\\"\\n- [2] AXMenuBarItem \\\"File\\\"\\nMenu contents omitted; request includeMenus=true to expand.\",\"windows\":[{\"windowId\":7,\"pid\":9,\"appName\":\"Fixture\",\"title\":\"CUA fixture\",\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"isOnScreen\":true,\"onCurrentSpace\":true}],\"truncated\":false}"}],"isError":false,"id":"{{message:71}}"}},"sourceEventSeqs":[179],"surfaceOp":"append"} {"type":"step/end","data":{"turn":1,"step":34}} {"type":"step/start","data":{"turn":1,"step":35}} -{"type":"assistant/message","data":{"turn":1,"step":35,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-hover-observe","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:72}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947555215,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947555215,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-hover-observe","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}}},{"type":"chunk","time":1791947555215,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947555216,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} -{"type":"tool/call","data":{"turn":1,"step":35,"callId":"cua-hover-observe","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}} -{"type":"tool/result","data":{"turn":1,"step":35,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-hover-observe"},"toolCallId":"cua-hover-observe","content":[{"type":"text","text":"{\"appHandle\":\"fixture-app\",\"pid\":9,\"windowId\":7,\"snapshotId\":\"fixture-snapshot-10\",\"tree\":\"AX scope=window offsets=UTF-16\\n- [0] AXButton \\\"Increment\\\"\\n- [1] AXTextField valueWritable=true selectionWritable=true value=\\\"\\\" placeholder=\\\"Fixture text\\\"\\n- AXStaticText \\\"Count: 3\\\"\\n- AXStaticText \\\"Hover information\\\"\\n- [2] AXMenuBarItem \\\"File\\\"\\nMenu contents omitted; request includeMenus=true to expand.\",\"windows\":[{\"windowId\":7,\"pid\":9,\"appName\":\"Fixture\",\"title\":\"CUA fixture\",\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"isOnScreen\":true,\"onCurrentSpace\":true}],\"truncated\":false}"}],"isError":false,"id":"{{message:73}}"}},"sourceEventSeqs":[184],"surfaceOp":"append"} +{"type":"assistant/message","data":{"turn":1,"step":35,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-capture-callback","name":"cua","arguments":"{\"method\":\"pressKey\",\"params\":{\"appHandle\":\"fixture-app\",\"key\":\"F10\"}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:72}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947690414,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947690414,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-capture-callback","name":"cua","arguments":"{\"method\":\"pressKey\",\"params\":{\"appHandle\":\"fixture-app\",\"key\":\"F10\"}}"}}},{"type":"chunk","time":1791947690414,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947690414,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} +{"type":"tool/call","data":{"turn":1,"step":35,"callId":"cua-capture-callback","name":"cua","arguments":"{\"method\":\"pressKey\",\"params\":{\"appHandle\":\"fixture-app\",\"key\":\"F10\"}}"}} +{"type":"tool/result","data":{"turn":1,"step":35,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-capture-callback"},"toolCallId":"cua-capture-callback","content":[{"type":"text","text":"{\"dispatched\":true,\"verified\":false}"}],"isError":false,"id":"{{message:73}}"}},"sourceEventSeqs":[184],"surfaceOp":"append"} {"type":"step/end","data":{"turn":1,"step":35}} {"type":"step/start","data":{"turn":1,"step":36}} -{"type":"assistant/message","data":{"turn":1,"step":36,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-hover-capture","name":"cua","arguments":"{\"method\":\"screenshot\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:74}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947555231,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947555231,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-hover-capture","name":"cua","arguments":"{\"method\":\"screenshot\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}}},{"type":"chunk","time":1791947555231,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947555231,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} -{"type":"tool/call","data":{"turn":1,"step":36,"callId":"cua-hover-capture","name":"cua","arguments":"{\"method\":\"screenshot\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}} -{"type":"tool/result","data":{"turn":1,"step":36,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-hover-capture"},"toolCallId":"cua-hover-capture","content":[{"type":"text","text":"{\"pixelWidth\":1,\"pixelHeight\":1,\"windowId\":7,\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"coordinateSpace\":\"screenshot-pixels\"}"},{"type":"image","attachment":{"attachmentId":"sha256:999f1d1527ee7e79266f16add5430fff76b1225d742464a5b1ff1f02971bb8ee","mediaType":"image/png","width":1,"height":1,"bytes":90}}],"isError":false,"id":"{{message:75}}"}},"sourceEventSeqs":[189],"surfaceOp":"append"} +{"type":"assistant/message","data":{"turn":1,"step":36,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-capture-recovered","name":"cua","arguments":"{\"method\":\"screenshot\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:74}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947690434,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947690434,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-capture-recovered","name":"cua","arguments":"{\"method\":\"screenshot\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}}},{"type":"chunk","time":1791947690434,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947690434,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} +{"type":"tool/call","data":{"turn":1,"step":36,"callId":"cua-capture-recovered","name":"cua","arguments":"{\"method\":\"screenshot\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}} +{"type":"tool/result","data":{"turn":1,"step":36,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-capture-recovered"},"toolCallId":"cua-capture-recovered","content":[{"type":"text","text":"{\"pixelWidth\":1,\"pixelHeight\":1,\"windowId\":7,\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"coordinateSpace\":\"screenshot-pixels\"}"},{"type":"image","attachment":{"attachmentId":"sha256:999f1d1527ee7e79266f16add5430fff76b1225d742464a5b1ff1f02971bb8ee","mediaType":"image/png","width":1,"height":1,"bytes":90}}],"isError":false,"id":"{{message:75}}"}},"sourceEventSeqs":[189],"surfaceOp":"append"} {"type":"step/end","data":{"turn":1,"step":36}} {"type":"step/start","data":{"turn":1,"step":37}} -{"type":"assistant/message","data":{"turn":1,"step":37,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-hover-popup","name":"cua","arguments":"{\"method\":\"hover\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7,\"x\":100,\"y\":200,\"relative\":true}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:76}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947555256,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947555256,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-hover-popup","name":"cua","arguments":"{\"method\":\"hover\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7,\"x\":100,\"y\":200,\"relative\":true}}"}}},{"type":"chunk","time":1791947555256,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947555256,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} -{"type":"tool/call","data":{"turn":1,"step":37,"callId":"cua-hover-popup","name":"cua","arguments":"{\"method\":\"hover\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7,\"x\":100,\"y\":200,\"relative\":true}}"}} -{"type":"tool/result","data":{"turn":1,"step":37,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-hover-popup"},"toolCallId":"cua-hover-popup","content":[{"type":"text","text":"{\"dispatched\":true,\"verified\":false,\"hoverRoute\":\"accessibility\",\"systemPointerMoved\":false}"}],"isError":false,"id":"{{message:77}}"}},"sourceEventSeqs":[194],"surfaceOp":"append"} +{"type":"assistant/message","data":{"turn":1,"step":37,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-hover-enter","name":"cua","arguments":"{\"method\":\"hover\",\"params\":{\"appHandle\":\"fixture-app\",\"snapshotId\":\"fixture-snapshot-9\",\"elementIndex\":0}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:76}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947690466,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947690466,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-hover-enter","name":"cua","arguments":"{\"method\":\"hover\",\"params\":{\"appHandle\":\"fixture-app\",\"snapshotId\":\"fixture-snapshot-9\",\"elementIndex\":0}}"}}},{"type":"chunk","time":1791947690466,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947690466,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} +{"type":"tool/call","data":{"turn":1,"step":37,"callId":"cua-hover-enter","name":"cua","arguments":"{\"method\":\"hover\",\"params\":{\"appHandle\":\"fixture-app\",\"snapshotId\":\"fixture-snapshot-9\",\"elementIndex\":0}}"}} +{"type":"tool/result","data":{"turn":1,"step":37,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-hover-enter"},"toolCallId":"cua-hover-enter","content":[{"type":"text","text":"{\"dispatched\":true,\"verified\":false,\"hoverRoute\":\"accessibility\",\"systemPointerMoved\":false}"}],"isError":false,"id":"{{message:77}}"}},"sourceEventSeqs":[194],"surfaceOp":"append"} {"type":"step/end","data":{"turn":1,"step":37}} {"type":"step/start","data":{"turn":1,"step":38}} -{"type":"assistant/message","data":{"turn":1,"step":38,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-hover-click","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7,\"snapshotId\":\"fixture-snapshot-10\",\"x\":100,\"y\":200,\"relative\":true}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:78}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947555273,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947555273,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-hover-click","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7,\"snapshotId\":\"fixture-snapshot-10\",\"x\":100,\"y\":200,\"relative\":true}}"}}},{"type":"chunk","time":1791947555273,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947555273,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} -{"type":"tool/call","data":{"turn":1,"step":38,"callId":"cua-hover-click","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7,\"snapshotId\":\"fixture-snapshot-10\",\"x\":100,\"y\":200,\"relative\":true}}"}} -{"type":"tool/result","data":{"turn":1,"step":38,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-hover-click"},"toolCallId":"cua-hover-click","content":[{"type":"text","text":"{\"dispatched\":true,\"verified\":false}"}],"isError":false,"id":"{{message:79}}"}},"sourceEventSeqs":[199],"surfaceOp":"append"} +{"type":"assistant/message","data":{"turn":1,"step":38,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-hover-observe","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:78}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947690495,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947690495,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-hover-observe","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}}},{"type":"chunk","time":1791947690495,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947690495,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} +{"type":"tool/call","data":{"turn":1,"step":38,"callId":"cua-hover-observe","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}} +{"type":"tool/result","data":{"turn":1,"step":38,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-hover-observe"},"toolCallId":"cua-hover-observe","content":[{"type":"text","text":"{\"appHandle\":\"fixture-app\",\"pid\":9,\"windowId\":7,\"snapshotId\":\"fixture-snapshot-10\",\"tree\":\"AX scope=window offsets=UTF-16\\n- [0] AXButton \\\"Increment\\\"\\n- [1] AXTextField valueWritable=true selectionWritable=true value=\\\"\\\" placeholder=\\\"Fixture text\\\"\\n- AXStaticText \\\"Count: 3\\\"\\n- AXStaticText \\\"Hover information\\\"\\n- [2] AXMenuBarItem \\\"File\\\"\\nMenu contents omitted; request includeMenus=true to expand.\",\"windows\":[{\"windowId\":7,\"pid\":9,\"appName\":\"Fixture\",\"title\":\"CUA fixture\",\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"isOnScreen\":true,\"onCurrentSpace\":true}],\"truncated\":false}"}],"isError":false,"id":"{{message:79}}"}},"sourceEventSeqs":[199],"surfaceOp":"append"} {"type":"step/end","data":{"turn":1,"step":38}} {"type":"step/start","data":{"turn":1,"step":39}} -{"type":"assistant/message","data":{"turn":1,"step":39,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-hover-exit","name":"cua","arguments":"{\"method\":\"hover\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7,\"x\":0,\"y\":200,\"relative\":true}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:80}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947555291,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947555291,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-hover-exit","name":"cua","arguments":"{\"method\":\"hover\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7,\"x\":0,\"y\":200,\"relative\":true}}"}}},{"type":"chunk","time":1791947555291,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947555291,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} -{"type":"tool/call","data":{"turn":1,"step":39,"callId":"cua-hover-exit","name":"cua","arguments":"{\"method\":\"hover\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7,\"x\":0,\"y\":200,\"relative\":true}}"}} -{"type":"tool/result","data":{"turn":1,"step":39,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-hover-exit"},"toolCallId":"cua-hover-exit","content":[{"type":"text","text":"{\"dispatched\":true,\"verified\":false,\"hoverRoute\":\"accessibility\",\"systemPointerMoved\":false}"}],"isError":false,"id":"{{message:81}}"}},"sourceEventSeqs":[204],"surfaceOp":"append"} +{"type":"assistant/message","data":{"turn":1,"step":39,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-hover-capture","name":"cua","arguments":"{\"method\":\"screenshot\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:80}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947690517,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947690517,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-hover-capture","name":"cua","arguments":"{\"method\":\"screenshot\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}}},{"type":"chunk","time":1791947690517,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947690517,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} +{"type":"tool/call","data":{"turn":1,"step":39,"callId":"cua-hover-capture","name":"cua","arguments":"{\"method\":\"screenshot\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}} +{"type":"tool/result","data":{"turn":1,"step":39,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-hover-capture"},"toolCallId":"cua-hover-capture","content":[{"type":"text","text":"{\"pixelWidth\":1,\"pixelHeight\":1,\"windowId\":7,\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"coordinateSpace\":\"screenshot-pixels\"}"},{"type":"image","attachment":{"attachmentId":"sha256:999f1d1527ee7e79266f16add5430fff76b1225d742464a5b1ff1f02971bb8ee","mediaType":"image/png","width":1,"height":1,"bytes":90}}],"isError":false,"id":"{{message:81}}"}},"sourceEventSeqs":[204],"surfaceOp":"append"} {"type":"step/end","data":{"turn":1,"step":39}} {"type":"step/start","data":{"turn":1,"step":40}} -{"type":"assistant/message","data":{"turn":1,"step":40,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-hover-verify","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:82}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947555307,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947555307,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-hover-verify","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}}},{"type":"chunk","time":1791947555307,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947555307,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} -{"type":"tool/call","data":{"turn":1,"step":40,"callId":"cua-hover-verify","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}} -{"type":"tool/result","data":{"turn":1,"step":40,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-hover-verify"},"toolCallId":"cua-hover-verify","content":[{"type":"text","text":"{\"appHandle\":\"fixture-app\",\"pid\":9,\"windowId\":7,\"snapshotId\":\"fixture-snapshot-11\",\"tree\":\"AX scope=window offsets=UTF-16\\n- [0] AXButton \\\"Increment\\\"\\n- [1] AXTextField valueWritable=true selectionWritable=true value=\\\"\\\" placeholder=\\\"Fixture text\\\"\\n- AXStaticText \\\"Count: 4\\\"\\n- [2] AXMenuBarItem \\\"File\\\"\\nMenu contents omitted; request includeMenus=true to expand.\",\"windows\":[{\"windowId\":7,\"pid\":9,\"appName\":\"Fixture\",\"title\":\"CUA fixture\",\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"isOnScreen\":true,\"onCurrentSpace\":true}],\"truncated\":false}"}],"isError":false,"id":"{{message:83}}"}},"sourceEventSeqs":[209],"surfaceOp":"append"} +{"type":"assistant/message","data":{"turn":1,"step":40,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-hover-popup","name":"cua","arguments":"{\"method\":\"hover\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7,\"x\":100,\"y\":200,\"relative\":true}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:82}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947690565,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947690565,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-hover-popup","name":"cua","arguments":"{\"method\":\"hover\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7,\"x\":100,\"y\":200,\"relative\":true}}"}}},{"type":"chunk","time":1791947690565,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947690565,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} +{"type":"tool/call","data":{"turn":1,"step":40,"callId":"cua-hover-popup","name":"cua","arguments":"{\"method\":\"hover\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7,\"x\":100,\"y\":200,\"relative\":true}}"}} +{"type":"tool/result","data":{"turn":1,"step":40,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-hover-popup"},"toolCallId":"cua-hover-popup","content":[{"type":"text","text":"{\"dispatched\":true,\"verified\":false,\"hoverRoute\":\"accessibility\",\"systemPointerMoved\":false}"}],"isError":false,"id":"{{message:83}}"}},"sourceEventSeqs":[209],"surfaceOp":"append"} {"type":"step/end","data":{"turn":1,"step":40}} {"type":"step/start","data":{"turn":1,"step":41}} -{"type":"assistant/message","data":{"turn":1,"step":41,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-paste-background","name":"cua","arguments":"{\"method\":\"pressKey\",\"params\":{\"appHandle\":\"fixture-app\",\"key\":\"F9\"}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:84}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947555323,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947555323,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-paste-background","name":"cua","arguments":"{\"method\":\"pressKey\",\"params\":{\"appHandle\":\"fixture-app\",\"key\":\"F9\"}}"}}},{"type":"chunk","time":1791947555323,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947555323,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} -{"type":"tool/call","data":{"turn":1,"step":41,"callId":"cua-paste-background","name":"cua","arguments":"{\"method\":\"pressKey\",\"params\":{\"appHandle\":\"fixture-app\",\"key\":\"F9\"}}"}} -{"type":"tool/result","data":{"turn":1,"step":41,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-paste-background"},"toolCallId":"cua-paste-background","content":[{"type":"text","text":"{\"dispatched\":true,\"verified\":false}"}],"isError":false,"id":"{{message:85}}"}},"sourceEventSeqs":[214],"surfaceOp":"append"} +{"type":"assistant/message","data":{"turn":1,"step":41,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-hover-click","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7,\"snapshotId\":\"fixture-snapshot-10\",\"x\":100,\"y\":200,\"relative\":true}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:84}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947690590,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947690590,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-hover-click","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7,\"snapshotId\":\"fixture-snapshot-10\",\"x\":100,\"y\":200,\"relative\":true}}"}}},{"type":"chunk","time":1791947690590,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947690590,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} +{"type":"tool/call","data":{"turn":1,"step":41,"callId":"cua-hover-click","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7,\"snapshotId\":\"fixture-snapshot-10\",\"x\":100,\"y\":200,\"relative\":true}}"}} +{"type":"tool/result","data":{"turn":1,"step":41,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-hover-click"},"toolCallId":"cua-hover-click","content":[{"type":"text","text":"{\"dispatched\":true,\"verified\":false}"}],"isError":false,"id":"{{message:85}}"}},"sourceEventSeqs":[214],"surfaceOp":"append"} {"type":"step/end","data":{"turn":1,"step":41}} {"type":"step/start","data":{"turn":1,"step":42}} -{"type":"assistant/message","data":{"turn":1,"step":42,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-paste-unread","name":"cua","arguments":"{\"method\":\"paste\",\"params\":{\"appHandle\":\"fixture-app\",\"text\":\"fixture paste\"}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:86}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947555339,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947555339,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-paste-unread","name":"cua","arguments":"{\"method\":\"paste\",\"params\":{\"appHandle\":\"fixture-app\",\"text\":\"fixture paste\"}}"}}},{"type":"chunk","time":1791947555339,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947555339,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} -{"type":"tool/call","data":{"turn":1,"step":42,"callId":"cua-paste-unread","name":"cua","arguments":"{\"method\":\"paste\",\"params\":{\"appHandle\":\"fixture-app\",\"text\":\"fixture paste\"}}"}} -{"type":"tool/result","data":{"turn":1,"step":42,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-paste-unread"},"toolCallId":"cua-paste-unread","content":[{"type":"text","text":"Error: PASTE_NOT_CONSUMED: Paste command was dispatched, but clipboard data was not requested within the read deadline. Bring the target app to the foreground and focus its text input. Inspect the contents before deciding whether to retry; a delayed paste may still run."}],"isError":true,"id":"{{message:87}}"}},"sourceEventSeqs":[219],"surfaceOp":"append"} +{"type":"assistant/message","data":{"turn":1,"step":42,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-hover-exit","name":"cua","arguments":"{\"method\":\"hover\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7,\"x\":0,\"y\":200,\"relative\":true}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:86}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947690613,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947690613,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-hover-exit","name":"cua","arguments":"{\"method\":\"hover\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7,\"x\":0,\"y\":200,\"relative\":true}}"}}},{"type":"chunk","time":1791947690613,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947690613,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} +{"type":"tool/call","data":{"turn":1,"step":42,"callId":"cua-hover-exit","name":"cua","arguments":"{\"method\":\"hover\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7,\"x\":0,\"y\":200,\"relative\":true}}"}} +{"type":"tool/result","data":{"turn":1,"step":42,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-hover-exit"},"toolCallId":"cua-hover-exit","content":[{"type":"text","text":"{\"dispatched\":true,\"verified\":false,\"hoverRoute\":\"accessibility\",\"systemPointerMoved\":false}"}],"isError":false,"id":"{{message:87}}"}},"sourceEventSeqs":[219],"surfaceOp":"append"} {"type":"step/end","data":{"turn":1,"step":42}} {"type":"step/start","data":{"turn":1,"step":43}} -{"type":"assistant/message","data":{"turn":1,"step":43,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-paste-inspect","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:88}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947555356,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947555356,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-paste-inspect","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}}},{"type":"chunk","time":1791947555356,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947555356,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} -{"type":"tool/call","data":{"turn":1,"step":43,"callId":"cua-paste-inspect","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}} -{"type":"tool/result","data":{"turn":1,"step":43,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-paste-inspect"},"toolCallId":"cua-paste-inspect","content":[{"type":"text","text":"{\"appHandle\":\"fixture-app\",\"pid\":9,\"windowId\":7,\"snapshotId\":\"fixture-snapshot-12\",\"tree\":\"AX scope=window offsets=UTF-16\\n- [0] AXButton \\\"Increment\\\"\\n- [1] AXTextField valueWritable=true selectionWritable=true value=\\\"\\\" placeholder=\\\"Fixture text\\\"\\n- AXStaticText \\\"Count: 4\\\"\\n- [2] AXMenuBarItem \\\"File\\\"\\nMenu contents omitted; request includeMenus=true to expand.\",\"windows\":[{\"windowId\":7,\"pid\":9,\"appName\":\"Fixture\",\"title\":\"CUA fixture\",\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"isOnScreen\":true,\"onCurrentSpace\":true}],\"truncated\":false}"}],"isError":false,"id":"{{message:89}}"}},"sourceEventSeqs":[224],"surfaceOp":"append"} +{"type":"assistant/message","data":{"turn":1,"step":43,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-hover-verify","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:88}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947690633,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947690633,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-hover-verify","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}}},{"type":"chunk","time":1791947690633,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947690633,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} +{"type":"tool/call","data":{"turn":1,"step":43,"callId":"cua-hover-verify","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}} +{"type":"tool/result","data":{"turn":1,"step":43,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-hover-verify"},"toolCallId":"cua-hover-verify","content":[{"type":"text","text":"{\"appHandle\":\"fixture-app\",\"pid\":9,\"windowId\":7,\"snapshotId\":\"fixture-snapshot-11\",\"tree\":\"AX scope=window offsets=UTF-16\\n- [0] AXButton \\\"Increment\\\"\\n- [1] AXTextField valueWritable=true selectionWritable=true value=\\\"\\\" placeholder=\\\"Fixture text\\\"\\n- AXStaticText \\\"Count: 4\\\"\\n- [2] AXMenuBarItem \\\"File\\\"\\nMenu contents omitted; request includeMenus=true to expand.\",\"windows\":[{\"windowId\":7,\"pid\":9,\"appName\":\"Fixture\",\"title\":\"CUA fixture\",\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"isOnScreen\":true,\"onCurrentSpace\":true}],\"truncated\":false}"}],"isError":false,"id":"{{message:89}}"}},"sourceEventSeqs":[224],"surfaceOp":"append"} {"type":"step/end","data":{"turn":1,"step":43}} {"type":"step/start","data":{"turn":1,"step":44}} -{"type":"assistant/message","data":{"turn":1,"step":44,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-paste-activate","name":"cua","arguments":"{\"method\":\"activate\",\"params\":{\"appHandle\":\"fixture-app\"}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:90}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947555372,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947555372,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-paste-activate","name":"cua","arguments":"{\"method\":\"activate\",\"params\":{\"appHandle\":\"fixture-app\"}}"}}},{"type":"chunk","time":1791947555372,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947555372,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} -{"type":"tool/call","data":{"turn":1,"step":44,"callId":"cua-paste-activate","name":"cua","arguments":"{\"method\":\"activate\",\"params\":{\"appHandle\":\"fixture-app\"}}"}} -{"type":"tool/result","data":{"turn":1,"step":44,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-paste-activate"},"toolCallId":"cua-paste-activate","content":[{"type":"text","text":"{\"dispatched\":true,\"verified\":false,\"foreground\":{\"requestId\":\"cua-paste-activate\",\"operation\":\"activate\",\"targetPid\":9,\"targetWindowId\":null,\"before\":{\"frontPid\":42,\"focusedWindowId\":7,\"frontWindowId\":99,\"frontWindowPid\":42},\"after\":{\"frontPid\":9,\"focusedWindowId\":7,\"frontWindowId\":7,\"frontWindowPid\":9},\"activationRequested\":true,\"raiseRequested\":false}}"}],"isError":false,"id":"{{message:91}}"}},"sourceEventSeqs":[229],"surfaceOp":"append"} +{"type":"assistant/message","data":{"turn":1,"step":44,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-paste-background","name":"cua","arguments":"{\"method\":\"pressKey\",\"params\":{\"appHandle\":\"fixture-app\",\"key\":\"F9\"}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:90}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947690653,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947690653,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-paste-background","name":"cua","arguments":"{\"method\":\"pressKey\",\"params\":{\"appHandle\":\"fixture-app\",\"key\":\"F9\"}}"}}},{"type":"chunk","time":1791947690653,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947690653,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} +{"type":"tool/call","data":{"turn":1,"step":44,"callId":"cua-paste-background","name":"cua","arguments":"{\"method\":\"pressKey\",\"params\":{\"appHandle\":\"fixture-app\",\"key\":\"F9\"}}"}} +{"type":"tool/result","data":{"turn":1,"step":44,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-paste-background"},"toolCallId":"cua-paste-background","content":[{"type":"text","text":"{\"dispatched\":true,\"verified\":false}"}],"isError":false,"id":"{{message:91}}"}},"sourceEventSeqs":[229],"surfaceOp":"append"} {"type":"step/end","data":{"turn":1,"step":44}} {"type":"step/start","data":{"turn":1,"step":45}} -{"type":"assistant/message","data":{"turn":1,"step":45,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-paste-focus","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:92}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947555388,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947555388,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-paste-focus","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}}},{"type":"chunk","time":1791947555388,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947555388,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} -{"type":"tool/call","data":{"turn":1,"step":45,"callId":"cua-paste-focus","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}} -{"type":"tool/result","data":{"turn":1,"step":45,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-paste-focus"},"toolCallId":"cua-paste-focus","content":[{"type":"text","text":"{\"appHandle\":\"fixture-app\",\"pid\":9,\"windowId\":7,\"snapshotId\":\"fixture-snapshot-13\",\"tree\":\"AX scope=window offsets=UTF-16\\n- [0] AXButton \\\"Increment\\\"\\n- [1] AXTextField valueWritable=true selectionWritable=true value=\\\"\\\" placeholder=\\\"Fixture text\\\"\\n- AXStaticText \\\"Count: 4\\\"\\n- [2] AXMenuBarItem \\\"File\\\"\\nMenu contents omitted; request includeMenus=true to expand.\",\"windows\":[{\"windowId\":7,\"pid\":9,\"appName\":\"Fixture\",\"title\":\"CUA fixture\",\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"isOnScreen\":true,\"onCurrentSpace\":true}],\"truncated\":false}"}],"isError":false,"id":"{{message:93}}"}},"sourceEventSeqs":[234],"surfaceOp":"append"} +{"type":"assistant/message","data":{"turn":1,"step":45,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-paste-unread","name":"cua","arguments":"{\"method\":\"paste\",\"params\":{\"appHandle\":\"fixture-app\",\"text\":\"fixture paste\"}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:92}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947690672,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947690672,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-paste-unread","name":"cua","arguments":"{\"method\":\"paste\",\"params\":{\"appHandle\":\"fixture-app\",\"text\":\"fixture paste\"}}"}}},{"type":"chunk","time":1791947690672,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947690672,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} +{"type":"tool/call","data":{"turn":1,"step":45,"callId":"cua-paste-unread","name":"cua","arguments":"{\"method\":\"paste\",\"params\":{\"appHandle\":\"fixture-app\",\"text\":\"fixture paste\"}}"}} +{"type":"tool/result","data":{"turn":1,"step":45,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-paste-unread"},"toolCallId":"cua-paste-unread","content":[{"type":"text","text":"Error: PASTE_NOT_CONSUMED: Paste command was dispatched, but clipboard data was not requested within the read deadline. Bring the target app to the foreground and focus its text input. Inspect the contents before deciding whether to retry; a delayed paste may still run."}],"isError":true,"id":"{{message:93}}"}},"sourceEventSeqs":[234],"surfaceOp":"append"} {"type":"step/end","data":{"turn":1,"step":45}} {"type":"step/start","data":{"turn":1,"step":46}} -{"type":"assistant/message","data":{"turn":1,"step":46,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-paste-consumed","name":"cua","arguments":"{\"method\":\"paste\",\"params\":{\"appHandle\":\"fixture-app\",\"text\":\"fixture paste\"}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:94}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947555404,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947555404,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-paste-consumed","name":"cua","arguments":"{\"method\":\"paste\",\"params\":{\"appHandle\":\"fixture-app\",\"text\":\"fixture paste\"}}"}}},{"type":"chunk","time":1791947555404,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947555404,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} -{"type":"tool/call","data":{"turn":1,"step":46,"callId":"cua-paste-consumed","name":"cua","arguments":"{\"method\":\"paste\",\"params\":{\"appHandle\":\"fixture-app\",\"text\":\"fixture paste\"}}"}} -{"type":"tool/result","data":{"turn":1,"step":46,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-paste-consumed"},"toolCallId":"cua-paste-consumed","content":[{"type":"text","text":"{\"dispatched\":true,\"verified\":false}"}],"isError":false,"id":"{{message:95}}"}},"sourceEventSeqs":[239],"surfaceOp":"append"} +{"type":"assistant/message","data":{"turn":1,"step":46,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-paste-inspect","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:94}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947690696,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947690696,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-paste-inspect","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}}},{"type":"chunk","time":1791947690696,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947690696,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} +{"type":"tool/call","data":{"turn":1,"step":46,"callId":"cua-paste-inspect","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}} +{"type":"tool/result","data":{"turn":1,"step":46,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-paste-inspect"},"toolCallId":"cua-paste-inspect","content":[{"type":"text","text":"{\"appHandle\":\"fixture-app\",\"pid\":9,\"windowId\":7,\"snapshotId\":\"fixture-snapshot-12\",\"tree\":\"AX scope=window offsets=UTF-16\\n- [0] AXButton \\\"Increment\\\"\\n- [1] AXTextField valueWritable=true selectionWritable=true value=\\\"\\\" placeholder=\\\"Fixture text\\\"\\n- AXStaticText \\\"Count: 4\\\"\\n- [2] AXMenuBarItem \\\"File\\\"\\nMenu contents omitted; request includeMenus=true to expand.\",\"windows\":[{\"windowId\":7,\"pid\":9,\"appName\":\"Fixture\",\"title\":\"CUA fixture\",\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"isOnScreen\":true,\"onCurrentSpace\":true}],\"truncated\":false}"}],"isError":false,"id":"{{message:95}}"}},"sourceEventSeqs":[239],"surfaceOp":"append"} {"type":"step/end","data":{"turn":1,"step":46}} {"type":"step/start","data":{"turn":1,"step":47}} -{"type":"assistant/message","data":{"turn":1,"step":47,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-paste-verify","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:96}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947555421,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947555421,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-paste-verify","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}}},{"type":"chunk","time":1791947555421,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947555421,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} -{"type":"tool/call","data":{"turn":1,"step":47,"callId":"cua-paste-verify","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}} -{"type":"tool/result","data":{"turn":1,"step":47,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-paste-verify"},"toolCallId":"cua-paste-verify","content":[{"type":"text","text":"{\"appHandle\":\"fixture-app\",\"pid\":9,\"windowId\":7,\"snapshotId\":\"fixture-snapshot-14\",\"tree\":\"AX scope=window offsets=UTF-16\\n- [0] AXButton \\\"Increment\\\"\\n- [1] AXTextField valueWritable=true selectionWritable=true value=\\\"fixture paste\\\" placeholder=\\\"Fixture text\\\"\\n- AXStaticText \\\"Count: 4\\\"\\n- [2] AXMenuBarItem \\\"File\\\"\\nMenu contents omitted; request includeMenus=true to expand.\",\"windows\":[{\"windowId\":7,\"pid\":9,\"appName\":\"Fixture\",\"title\":\"CUA fixture\",\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"isOnScreen\":true,\"onCurrentSpace\":true}],\"truncated\":false}"}],"isError":false,"id":"{{message:97}}"}},"sourceEventSeqs":[244],"surfaceOp":"append"} +{"type":"assistant/message","data":{"turn":1,"step":47,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-paste-activate","name":"cua","arguments":"{\"method\":\"activate\",\"params\":{\"appHandle\":\"fixture-app\"}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:96}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947690714,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947690714,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-paste-activate","name":"cua","arguments":"{\"method\":\"activate\",\"params\":{\"appHandle\":\"fixture-app\"}}"}}},{"type":"chunk","time":1791947690714,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947690714,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} +{"type":"tool/call","data":{"turn":1,"step":47,"callId":"cua-paste-activate","name":"cua","arguments":"{\"method\":\"activate\",\"params\":{\"appHandle\":\"fixture-app\"}}"}} +{"type":"tool/result","data":{"turn":1,"step":47,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-paste-activate"},"toolCallId":"cua-paste-activate","content":[{"type":"text","text":"{\"dispatched\":true,\"verified\":false,\"foreground\":{\"requestId\":\"cua-paste-activate\",\"operation\":\"activate\",\"targetPid\":9,\"targetWindowId\":null,\"before\":{\"frontPid\":42,\"focusedWindowId\":7,\"frontWindowId\":99,\"frontWindowPid\":42},\"after\":{\"frontPid\":9,\"focusedWindowId\":7,\"frontWindowId\":7,\"frontWindowPid\":9},\"activationRequested\":true,\"raiseRequested\":false}}"}],"isError":false,"id":"{{message:97}}"}},"sourceEventSeqs":[244],"surfaceOp":"append"} {"type":"step/end","data":{"turn":1,"step":47}} {"type":"step/start","data":{"turn":1,"step":48}} -{"type":"assistant/message","data":{"turn":1,"step":48,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-debug-arm","name":"cua","arguments":"{\"method\":\"pressKey\",\"params\":{\"appHandle\":\"fixture-app\",\"key\":\"F12\"}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:98}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947555437,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947555437,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-debug-arm","name":"cua","arguments":"{\"method\":\"pressKey\",\"params\":{\"appHandle\":\"fixture-app\",\"key\":\"F12\"}}"}}},{"type":"chunk","time":1791947555437,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947555437,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} -{"type":"tool/call","data":{"turn":1,"step":48,"callId":"cua-debug-arm","name":"cua","arguments":"{\"method\":\"pressKey\",\"params\":{\"appHandle\":\"fixture-app\",\"key\":\"F12\"}}"}} -{"type":"tool/result","data":{"turn":1,"step":48,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-debug-arm"},"toolCallId":"cua-debug-arm","content":[{"type":"text","text":"{\"dispatched\":true,\"verified\":false}"}],"isError":false,"id":"{{message:99}}"}},"sourceEventSeqs":[249],"surfaceOp":"append"} +{"type":"assistant/message","data":{"turn":1,"step":48,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-paste-focus","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:98}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947690734,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947690734,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-paste-focus","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}}},{"type":"chunk","time":1791947690734,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947690734,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} +{"type":"tool/call","data":{"turn":1,"step":48,"callId":"cua-paste-focus","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}} +{"type":"tool/result","data":{"turn":1,"step":48,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-paste-focus"},"toolCallId":"cua-paste-focus","content":[{"type":"text","text":"{\"appHandle\":\"fixture-app\",\"pid\":9,\"windowId\":7,\"snapshotId\":\"fixture-snapshot-13\",\"tree\":\"AX scope=window offsets=UTF-16\\n- [0] AXButton \\\"Increment\\\"\\n- [1] AXTextField valueWritable=true selectionWritable=true value=\\\"\\\" placeholder=\\\"Fixture text\\\"\\n- AXStaticText \\\"Count: 4\\\"\\n- [2] AXMenuBarItem \\\"File\\\"\\nMenu contents omitted; request includeMenus=true to expand.\",\"windows\":[{\"windowId\":7,\"pid\":9,\"appName\":\"Fixture\",\"title\":\"CUA fixture\",\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"isOnScreen\":true,\"onCurrentSpace\":true}],\"truncated\":false}"}],"isError":false,"id":"{{message:99}}"}},"sourceEventSeqs":[249],"surfaceOp":"append"} {"type":"step/end","data":{"turn":1,"step":48}} {"type":"step/start","data":{"turn":1,"step":49}} -{"type":"assistant/message","data":{"turn":1,"step":49,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-debug-screenshot","name":"cua","arguments":"{\"method\":\"screenshot\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:100}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947555453,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947555453,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-debug-screenshot","name":"cua","arguments":"{\"method\":\"screenshot\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}}},{"type":"chunk","time":1791947555453,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947555453,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} -{"type":"tool/call","data":{"turn":1,"step":49,"callId":"cua-debug-screenshot","name":"cua","arguments":"{\"method\":\"screenshot\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}} -{"type":"tool/result","data":{"turn":1,"step":49,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-debug-screenshot"},"toolCallId":"cua-debug-screenshot","content":[{"type":"text","text":"Error: NATIVE_PROTOCOL: Helper returned an invalid response; acquire fresh application handles\nScreenshot diagnostics:\n[cua-screenshot] {\"requestId\":\"cua-debug-screenshot\",\"stage\":\"capture-image.begin\",\"elapsedMs\":1}"}],"isError":true,"id":"{{message:101}}"}},"sourceEventSeqs":[254],"surfaceOp":"append"} +{"type":"assistant/message","data":{"turn":1,"step":49,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-paste-consumed","name":"cua","arguments":"{\"method\":\"paste\",\"params\":{\"appHandle\":\"fixture-app\",\"text\":\"fixture paste\"}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:100}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947690757,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947690757,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-paste-consumed","name":"cua","arguments":"{\"method\":\"paste\",\"params\":{\"appHandle\":\"fixture-app\",\"text\":\"fixture paste\"}}"}}},{"type":"chunk","time":1791947690757,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947690757,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} +{"type":"tool/call","data":{"turn":1,"step":49,"callId":"cua-paste-consumed","name":"cua","arguments":"{\"method\":\"paste\",\"params\":{\"appHandle\":\"fixture-app\",\"text\":\"fixture paste\"}}"}} +{"type":"tool/result","data":{"turn":1,"step":49,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-paste-consumed"},"toolCallId":"cua-paste-consumed","content":[{"type":"text","text":"{\"dispatched\":true,\"verified\":false}"}],"isError":false,"id":"{{message:101}}"}},"sourceEventSeqs":[254],"surfaceOp":"append"} {"type":"step/end","data":{"turn":1,"step":49}} {"type":"step/start","data":{"turn":1,"step":50}} -{"type":"assistant/message","data":{"turn":1,"step":50,"message":{"role":"assistant","content":[{"type":"text","text":"Observed Count: 4 after hovering, observing and clicking the revealed control. Moving away dismissed the hover information. Geometry conflicts rejected capture and coordinate gestures; the advertised AX button action remained available. No automatic activation or input retry was requested."}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:102}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947555473,"chunk":{"type":"block-start","index":0,"blockType":"text"}},{"type":"chunk","time":1791947555473,"chunk":{"type":"block-end","index":0,"block":{"type":"text","text":"Observed Count: 4 after hovering, observing and clicking the revealed control. Moving away dismissed the hover information. Geometry conflicts rejected capture and coordinate gestures; the advertised AX button action remained available. No automatic activation or input retry was requested."}}},{"type":"chunk","time":1791947555473,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947555473,"chunk":{"type":"finish","reason":{"kind":"stop"}}}]},"surfaceOp":"append"} +{"type":"assistant/message","data":{"turn":1,"step":50,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-paste-verify","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:102}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947690778,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947690778,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-paste-verify","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}}},{"type":"chunk","time":1791947690778,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947690778,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} +{"type":"tool/call","data":{"turn":1,"step":50,"callId":"cua-paste-verify","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}} +{"type":"tool/result","data":{"turn":1,"step":50,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-paste-verify"},"toolCallId":"cua-paste-verify","content":[{"type":"text","text":"{\"appHandle\":\"fixture-app\",\"pid\":9,\"windowId\":7,\"snapshotId\":\"fixture-snapshot-14\",\"tree\":\"AX scope=window offsets=UTF-16\\n- [0] AXButton \\\"Increment\\\"\\n- [1] AXTextField valueWritable=true selectionWritable=true value=\\\"fixture paste\\\" placeholder=\\\"Fixture text\\\"\\n- AXStaticText \\\"Count: 4\\\"\\n- [2] AXMenuBarItem \\\"File\\\"\\nMenu contents omitted; request includeMenus=true to expand.\",\"windows\":[{\"windowId\":7,\"pid\":9,\"appName\":\"Fixture\",\"title\":\"CUA fixture\",\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"isOnScreen\":true,\"onCurrentSpace\":true}],\"truncated\":false}"}],"isError":false,"id":"{{message:103}}"}},"sourceEventSeqs":[259],"surfaceOp":"append"} {"type":"step/end","data":{"turn":1,"step":50}} +{"type":"step/start","data":{"turn":1,"step":51}} +{"type":"assistant/message","data":{"turn":1,"step":51,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-debug-arm","name":"cua","arguments":"{\"method\":\"pressKey\",\"params\":{\"appHandle\":\"fixture-app\",\"key\":\"F12\"}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:104}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947690801,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947690801,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-debug-arm","name":"cua","arguments":"{\"method\":\"pressKey\",\"params\":{\"appHandle\":\"fixture-app\",\"key\":\"F12\"}}"}}},{"type":"chunk","time":1791947690801,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947690801,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} +{"type":"tool/call","data":{"turn":1,"step":51,"callId":"cua-debug-arm","name":"cua","arguments":"{\"method\":\"pressKey\",\"params\":{\"appHandle\":\"fixture-app\",\"key\":\"F12\"}}"}} +{"type":"tool/result","data":{"turn":1,"step":51,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-debug-arm"},"toolCallId":"cua-debug-arm","content":[{"type":"text","text":"{\"dispatched\":true,\"verified\":false}"}],"isError":false,"id":"{{message:105}}"}},"sourceEventSeqs":[264],"surfaceOp":"append"} +{"type":"step/end","data":{"turn":1,"step":51}} +{"type":"step/start","data":{"turn":1,"step":52}} +{"type":"assistant/message","data":{"turn":1,"step":52,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-debug-screenshot","name":"cua","arguments":"{\"method\":\"screenshot\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:106}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947690821,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947690821,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-debug-screenshot","name":"cua","arguments":"{\"method\":\"screenshot\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}}},{"type":"chunk","time":1791947690821,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947690821,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} +{"type":"tool/call","data":{"turn":1,"step":52,"callId":"cua-debug-screenshot","name":"cua","arguments":"{\"method\":\"screenshot\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}} +{"type":"tool/result","data":{"turn":1,"step":52,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-debug-screenshot"},"toolCallId":"cua-debug-screenshot","content":[{"type":"text","text":"Error: NATIVE_PROTOCOL: Helper returned an invalid response; acquire fresh application handles\nScreenshot diagnostics:\n[cua-screenshot] {\"requestId\":\"cua-debug-screenshot\",\"stage\":\"capture-image.begin\",\"elapsedMs\":1}"}],"isError":true,"id":"{{message:107}}"}},"sourceEventSeqs":[269],"surfaceOp":"append"} +{"type":"step/end","data":{"turn":1,"step":52}} +{"type":"step/start","data":{"turn":1,"step":53}} +{"type":"assistant/message","data":{"turn":1,"step":53,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-consent-denied","name":"cua","arguments":"{\"method\":\"getApp\",\"params\":{\"selector\":\"Denied Fixture\"}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:108}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947690852,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947690852,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-consent-denied","name":"cua","arguments":"{\"method\":\"getApp\",\"params\":{\"selector\":\"Denied Fixture\"}}"}}},{"type":"chunk","time":1791947690852,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947690852,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} +{"type":"tool/call","data":{"turn":1,"step":53,"callId":"cua-consent-denied","name":"cua","arguments":"{\"method\":\"getApp\",\"params\":{\"selector\":\"Denied Fixture\"}}"}} +{"type":"tool/result","data":{"turn":1,"step":53,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-consent-denied"},"toolCallId":"cua-consent-denied","content":[{"type":"text","text":"Error: APP_ACCESS_DENIED: The user denied access to Denied Fixture. Do not retry or use another tool to bypass this decision."}],"isError":true,"id":"{{message:109}}"}},"sourceEventSeqs":[274],"surfaceOp":"append"} +{"type":"step/end","data":{"turn":1,"step":53}} +{"type":"step/start","data":{"turn":1,"step":54}} +{"type":"assistant/message","data":{"turn":1,"step":54,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-consent-revocable","name":"cua","arguments":"{\"method\":\"getApp\",\"params\":{\"selector\":\"Revocable Fixture\"}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:110}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947691022,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947691022,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-consent-revocable","name":"cua","arguments":"{\"method\":\"getApp\",\"params\":{\"selector\":\"Revocable Fixture\"}}"}}},{"type":"chunk","time":1791947691022,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947691022,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} +{"type":"tool/call","data":{"turn":1,"step":54,"callId":"cua-consent-revocable","name":"cua","arguments":"{\"method\":\"getApp\",\"params\":{\"selector\":\"Revocable Fixture\"}}"}} +{"type":"tool/result","data":{"turn":1,"step":54,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-consent-revocable"},"toolCallId":"cua-consent-revocable","content":[{"type":"text","text":"{\"appHandle\":\"revoked-app\",\"pid\":17,\"name\":\"Revocable Fixture\"}"}],"isError":false,"id":"{{message:111}}"}},"sourceEventSeqs":[279],"surfaceOp":"append"} +{"type":"step/end","data":{"turn":1,"step":54}} +{"type":"step/start","data":{"turn":1,"step":55}} +{"type":"assistant/message","data":{"turn":1,"step":55,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-consent-revoked","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"revoked-app\"}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:112}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947691192,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947691192,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-consent-revoked","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"revoked-app\"}}"}}},{"type":"chunk","time":1791947691192,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947691192,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"} +{"type":"tool/call","data":{"turn":1,"step":55,"callId":"cua-consent-revoked","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"revoked-app\"}}"}} +{"type":"tool/result","data":{"turn":1,"step":55,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-consent-revoked"},"toolCallId":"cua-consent-revoked","content":[{"type":"text","text":"Error: APP_ACCESS_REVOKED: Application access was revoked; stop using this application"}],"isError":true,"id":"{{message:113}}"}},"sourceEventSeqs":[284],"surfaceOp":"append"} +{"type":"step/end","data":{"turn":1,"step":55}} +{"type":"step/start","data":{"turn":1,"step":56}} +{"type":"assistant/message","data":{"turn":1,"step":56,"message":{"role":"assistant","content":[{"type":"text","text":"Observed Count: 4 after hovering, observing and clicking the revealed control. Moving away dismissed the hover information. Geometry conflicts rejected capture and coordinate gestures; the advertised AX button action remained available. No automatic activation or input retry was requested."}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:114}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947691212,"chunk":{"type":"block-start","index":0,"blockType":"text"}},{"type":"chunk","time":1791947691212,"chunk":{"type":"block-end","index":0,"block":{"type":"text","text":"Observed Count: 4 after hovering, observing and clicking the revealed control. Moving away dismissed the hover information. Geometry conflicts rejected capture and coordinate gestures; the advertised AX button action remained available. No automatic activation or input retry was requested."}}},{"type":"chunk","time":1791947691212,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947691212,"chunk":{"type":"finish","reason":{"kind":"stop"}}}]},"surfaceOp":"append"} +{"type":"step/end","data":{"turn":1,"step":56}} {"type":"turn/end","data":{"turn":1,"reason":{"kind":"completed"}}} diff --git a/snapshots/session/cua-native/system-prompt.expected.md b/snapshots/session/cua-native/system-prompt.expected.md index 885e9058aa..d2daf65630 100644 --- a/snapshots/session/cua-native/system-prompt.expected.md +++ b/snapshots/session/cua-native/system-prompt.expected.md @@ -29,7 +29,7 @@ Use the workflow tool ONLY when the user explicitly asks for a workflow or for l Start independent delegations with `subagent` or `subagent_fork` together in one assistant message and continue useful work while they run. -Use cua with method getCapabilities and params {} to inspect permissions and sessionLocked. If sessionLocked is true or an operation reports SESSION_LOCKED, stop UI attempts and ask the user to unlock macOS. A null sessionLocked means no usable console session was verified. After unlocking, read fresh state; screenshots from a locked desktop are not current observations. Use listApps or cua.getState to discover running applications, and findApps(query) to search installed applications by name, bundle ID or absolute path. getApp requests user consent before launching a stopped application in the background, or before observations and input on a running one. If access is denied or revoked, stop using that application until the user explicitly grants access again. Never switch tools to bypass a refusal. Keep the returned appHandle; every app operation requires it. listWindows returns window IDs and point bounds. Raw getState returns AX text and snapshotId. Menu-bar entries are shown by default; includeMenus=true expands their contents. details=true adds identifiers, global-point bounds and complete action names. Both options also apply to getAXState. Each read replaces the indices for that app/window scope. Use its element indices only with that snapshotId. Raw screenshot and coordinate operations require a windowId. Coordinates are pixels in the latest returned screenshot, measured from its top-left. Set relative:true for integer thousandths from 0 to 1000 instead. The Host converts both modes to native window points; AX detail bounds remain global macOS points. A verified window can expose no AX elements. Its empty snapshot permits screenshots and coordinate input; element-index actions require an element from a fresh AX observation. +Use cua with method getCapabilities and params {} to inspect permissions and sessionLocked. If sessionLocked is true or an operation reports SESSION_LOCKED, stop UI attempts and ask the user to unlock macOS. A null sessionLocked means no usable console session was verified. After unlocking, read fresh state; screenshots from a locked desktop are not current observations. Use listApps or cua.getState to discover running applications, and findApps(query) to search installed applications by name, bundle ID or absolute path. getApp requests user consent before launching a stopped application in the background, or before observations and input on a running one. If access is denied or revoked, stop using that application until the user explicitly permits another authorization request. Never switch tools to bypass a refusal. Keep the returned appHandle; every app operation requires it. listWindows returns window IDs and point bounds. Raw getState returns AX text and snapshotId. Menu-bar entries are shown by default; includeMenus=true expands their contents. details=true adds identifiers, global-point bounds and complete action names. Both options also apply to getAXState. Each read replaces the indices for that app/window scope. Use its element indices only with that snapshotId. Raw screenshot and coordinate operations require a windowId. Coordinates are pixels in the latest returned screenshot, measured from its top-left. Set relative:true for integer thousandths from 0 to 1000 instead. The Host converts both modes to native window points; AX detail bounds remain global macOS points. A verified window can expose no AX elements. Its empty snapshot permits screenshots and coordinate input; element-index actions require an element from a fresh AX observation. Initialize the object API once in persistent PTC. Reuse its variables in later calls; put temporary declarations inside { ... } blocks to avoid redeclaration errors. All object methods are async. getAXState returns text, not a structured state object. Its text separates values and placeholders, and reports subroles, help, URLs, disabled/focused/selected/expanded state and UTF-16 selection ranges when available. Writability is true, false or unknown on relevant controls; omitted state fields do not assert false. selection=unavailable means the reported range exceeds the observed text. Compact output omits ordinary Press/Pick (click), ShowMenu (right click) and ScrollToVisible actions; details=true lists every available action. Use the displayed action name with performSecondaryAction; ambiguous aliases require the complete name from details=true. Truncation reports the reached provider limit. Narrow the window or increase that provider limit before reading again. @@ -66,7 +66,7 @@ App and Window methods: Raw operation tools use { method, params }, including params: {} for parameterless methods. getApp selects without activating the app. Keep applications in the background for observation and routine input. Do not activate or raise an app during initialization, before each action, or merely because another app is foreground. Prefer advertised AX actions. If a window is hidden or minimized, try app.show() or window.show() and read fresh state. show requests unhide and unminimize without activation, raising or switching Spaces; it chooses the explicit, focused, main or sole AX window. A null windowId means no window was selected for restoration. foregroundChanged compares the front app and top window before and after the request; null means unavailable, and delayed focus changes remain possible. If restoration is unsupported or the window remains unusable, including another Space or Stage Manager, explain the needed foreground step, use app.activate() or an advertised raise action, and read fresh state. Completing the task takes priority when foreground access is necessary. Avoid repeated activation when the user switches away; continue background-capable steps and report what requires foreground access. Applications may still change focus in response to input. -Await operations sequentially. After input, read fresh state to verify the effect. A successful action means dispatched, not verified completion. Cancellation may leave partial effects; never retry uncertain input without observing. STALE_APP requires getApp again; STALE_SNAPSHOT requires getState again. Absolute coordinate input requires a fresh screenshot of that window. Relative input can use a successful AX observation or screenshot. SCREENSHOT_REQUIRED means capture a fresh screenshot before using pixel coordinates. For click and hover, coordinate targets must be [x, y], for example hover([650, 400], {relative:true}). A bare number such as hover(650) is always an AX element index. The relative option applies only to coordinate targets, not AX indices. Window resizing since the latest observation rejects coordinate input; movement alone preserves window-local coordinates. WINDOW_CHANGED requires fresh state before any retry; use an advertised AX action or explicitly restore the window before screenshot or coordinate input, then observe again. PERMISSION_DENIED requires the user to grant macOS Accessibility or Screen Recording access in System Settings. After granting access, observe fresh state before continuing; do not automatically replay prior input. +Await operations sequentially. After input, read fresh state to verify the effect. A successful action means dispatched, not verified completion. Cancellation may leave partial effects; never retry uncertain input without observing. STALE_APP requires getApp again; STALE_SNAPSHOT requires getState again. Absolute coordinate input requires a fresh screenshot of that window. Relative input can use a successful AX observation or screenshot. SCREENSHOT_REQUIRED means capture a fresh screenshot before using pixel coordinates. For click and hover, coordinate targets must be [x, y], for example hover([650, 400], {relative:true}). A bare number such as hover(650) is always an AX element index. The relative option applies only to coordinate targets, not AX indices. Window resizing since the latest observation rejects coordinate input; movement alone preserves window-local coordinates. WINDOW_CHANGED requires fresh state before any retry; use an advertised AX action or explicitly restore the window before screenshot or coordinate input, then observe again. PERMISSION_DENIED requires the user to grant the required macOS system permissions for macOS Accessibility or Screen Recording. After granting access, observe fresh state before continuing; do not automatically replay prior input. Hover first uses advertised AX alternate UI without moving the system cursor, preserving revealed descendants until the pointer target leaves their region. When unavailable, it moves the visible system cursor without pressing a button; that route requires the target point to be unobscured in the foreground app. FOREGROUND_REQUIRED requires explicit activation or raising and a fresh observation before retrying. Returning, AX reads and screenshots do not restore the previous point. Wait for the application's hover delay, observe, then hover into the revealed controls before clicking. Use successive hover calls for intermediate points. The result reports hoverRoute and systemPointerMoved. Other input or application changes can interrupt the interaction; app objects do not reserve applications between calls. diff --git a/snapshots/session/cua-native/workspace/native-helper.mjs b/snapshots/session/cua-native/workspace/native-helper.mjs index c4510416c3..e36445ca94 100755 --- a/snapshots/session/cua-native/workspace/native-helper.mjs +++ b/snapshots/session/cua-native/workspace/native-helper.mjs @@ -125,6 +125,12 @@ function createApplicationState(socket, record, application, captureBroker, conn function refuse(code, message) { throw Object.assign(new Error(message), { code }) } function call(requestId, method, params) { record('call', { requestId, method, params }) + if (method === 'applicationIcons') return Object.fromEntries(params.paths.map(path => [path, `data:image/png;base64,${png}`])) + if (method === 'openPermissionSettings') return { opened: true } + if (method === 'permissionStatus' || method === 'requestPermission') { + const state = JSON.parse(readFileSync(new URL('./permissions.json', import.meta.url), 'utf8')) + return { ...state, helper: { pid: process.pid }, parent: { pid: process.ppid } } + } if (method === 'getCapabilities') return { platform: 'macos', sessionLocked: false, permissions: { accessibility: true, screenRecording: true }, methods, coordinateSpace: 'window-points', browser: false } if (method === 'listApps') return [{ pid: app.pid, name: app.name, bundleId: app.bundleId }] if (method === 'getApp') { diff --git a/snapshots/session/cua-ptc/consent-fixture.mjs b/snapshots/session/cua-ptc/consent-fixture.mjs index 3e6dd4064c..85e83bcf69 100644 --- a/snapshots/session/cua-ptc/consent-fixture.mjs +++ b/snapshots/session/cua-ptc/consent-fixture.mjs @@ -15,6 +15,6 @@ export async function apply(ctx) { try { await close() } finally { await unlink(helper) } }) ctx.on('user-questions/request', async request => ({ answers: request.questions.map(question => ({ - id: question.id, selected: ['Allow for this task'], + id: question.id, selected: ['Allow for this session'], })) })) } diff --git a/snapshots/session/cua-ptc/cordis.snapshot.yml b/snapshots/session/cua-ptc/cordis.snapshot.yml index c2f9b23c4b..e42a3ce149 100644 --- a/snapshots/session/cua-ptc/cordis.snapshot.yml +++ b/snapshots/session/cua-ptc/cordis.snapshot.yml @@ -50,6 +50,7 @@ - id: cua-native name: '@deepseek-ai/dsh-experimental-cua-macos' config: + agentPreset: false helperPath: !!js process.cwd() + '/native-helper.mjs' - id: user-questions diff --git a/snapshots/session/cua-ptc/cordis.yml b/snapshots/session/cua-ptc/cordis.yml index 60a3be9c9e..ac932bb223 100644 --- a/snapshots/session/cua-ptc/cordis.yml +++ b/snapshots/session/cua-ptc/cordis.yml @@ -32,6 +32,7 @@ - id: cua-native name: '@deepseek-ai/dsh-experimental-cua-macos' config: + agentPreset: false helperPath: !!js process.cwd() + '/native-helper.mjs' - id: user-questions diff --git a/snapshots/session/cua-ptc/system-prompt.expected.md b/snapshots/session/cua-ptc/system-prompt.expected.md index 5bc067211a..cf5222fd7f 100644 --- a/snapshots/session/cua-ptc/system-prompt.expected.md +++ b/snapshots/session/cua-ptc/system-prompt.expected.md @@ -31,7 +31,7 @@ Use the workflow tool ONLY when the user explicitly asks for a workflow or for l Start independent delegations with `subagent` or `subagent_fork` together in one assistant message and continue useful work while they run. -Use cua with method getCapabilities and params {} to inspect permissions and sessionLocked. If sessionLocked is true or an operation reports SESSION_LOCKED, stop UI attempts and ask the user to unlock macOS. A null sessionLocked means no usable console session was verified. After unlocking, read fresh state; screenshots from a locked desktop are not current observations. Use listApps or cua.getState to discover running applications, and findApps(query) to search installed applications by name, bundle ID or absolute path. getApp requests user consent before launching a stopped application in the background, or before observations and input on a running one. If access is denied or revoked, stop using that application until the user explicitly grants access again. Never switch tools to bypass a refusal. Keep the returned appHandle; every app operation requires it. listWindows returns window IDs and point bounds. Raw getState returns AX text and snapshotId. Menu-bar entries are shown by default; includeMenus=true expands their contents. details=true adds identifiers, global-point bounds and complete action names. Both options also apply to getAXState. Each read replaces the indices for that app/window scope. Use its element indices only with that snapshotId. Raw screenshot and coordinate operations require a windowId. Coordinates are pixels in the latest returned screenshot, measured from its top-left. Set relative:true for integer thousandths from 0 to 1000 instead. The Host converts both modes to native window points; AX detail bounds remain global macOS points. A verified window can expose no AX elements. Its empty snapshot permits screenshots and coordinate input; element-index actions require an element from a fresh AX observation. +Use cua with method getCapabilities and params {} to inspect permissions and sessionLocked. If sessionLocked is true or an operation reports SESSION_LOCKED, stop UI attempts and ask the user to unlock macOS. A null sessionLocked means no usable console session was verified. After unlocking, read fresh state; screenshots from a locked desktop are not current observations. Use listApps or cua.getState to discover running applications, and findApps(query) to search installed applications by name, bundle ID or absolute path. getApp requests user consent before launching a stopped application in the background, or before observations and input on a running one. If access is denied or revoked, stop using that application until the user explicitly permits another authorization request. Never switch tools to bypass a refusal. Keep the returned appHandle; every app operation requires it. listWindows returns window IDs and point bounds. Raw getState returns AX text and snapshotId. Menu-bar entries are shown by default; includeMenus=true expands their contents. details=true adds identifiers, global-point bounds and complete action names. Both options also apply to getAXState. Each read replaces the indices for that app/window scope. Use its element indices only with that snapshotId. Raw screenshot and coordinate operations require a windowId. Coordinates are pixels in the latest returned screenshot, measured from its top-left. Set relative:true for integer thousandths from 0 to 1000 instead. The Host converts both modes to native window points; AX detail bounds remain global macOS points. A verified window can expose no AX elements. Its empty snapshot permits screenshots and coordinate input; element-index actions require an element from a fresh AX observation. Initialize the object API once in persistent PTC. Reuse its variables in later calls; put temporary declarations inside { ... } blocks to avoid redeclaration errors. All object methods are async. getAXState returns text, not a structured state object. Its text separates values and placeholders, and reports subroles, help, URLs, disabled/focused/selected/expanded state and UTF-16 selection ranges when available. Writability is true, false or unknown on relevant controls; omitted state fields do not assert false. selection=unavailable means the reported range exceeds the observed text. Compact output omits ordinary Press/Pick (click), ShowMenu (right click) and ScrollToVisible actions; details=true lists every available action. Use the displayed action name with performSecondaryAction; ambiguous aliases require the complete name from details=true. Truncation reports the reached provider limit. Narrow the window or increase that provider limit before reading again. @@ -68,7 +68,7 @@ App and Window methods: Raw operation tools use { method, params }, including params: {} for parameterless methods. getApp selects without activating the app. Keep applications in the background for observation and routine input. Do not activate or raise an app during initialization, before each action, or merely because another app is foreground. Prefer advertised AX actions. If a window is hidden or minimized, try app.show() or window.show() and read fresh state. show requests unhide and unminimize without activation, raising or switching Spaces; it chooses the explicit, focused, main or sole AX window. A null windowId means no window was selected for restoration. foregroundChanged compares the front app and top window before and after the request; null means unavailable, and delayed focus changes remain possible. If restoration is unsupported or the window remains unusable, including another Space or Stage Manager, explain the needed foreground step, use app.activate() or an advertised raise action, and read fresh state. Completing the task takes priority when foreground access is necessary. Avoid repeated activation when the user switches away; continue background-capable steps and report what requires foreground access. Applications may still change focus in response to input. -Await operations sequentially. After input, read fresh state to verify the effect. A successful action means dispatched, not verified completion. Cancellation may leave partial effects; never retry uncertain input without observing. STALE_APP requires getApp again; STALE_SNAPSHOT requires getState again. Absolute coordinate input requires a fresh screenshot of that window. Relative input can use a successful AX observation or screenshot. SCREENSHOT_REQUIRED means capture a fresh screenshot before using pixel coordinates. For click and hover, coordinate targets must be [x, y], for example hover([650, 400], {relative:true}). A bare number such as hover(650) is always an AX element index. The relative option applies only to coordinate targets, not AX indices. Window resizing since the latest observation rejects coordinate input; movement alone preserves window-local coordinates. WINDOW_CHANGED requires fresh state before any retry; use an advertised AX action or explicitly restore the window before screenshot or coordinate input, then observe again. PERMISSION_DENIED requires the user to grant macOS Accessibility or Screen Recording access in System Settings. After granting access, observe fresh state before continuing; do not automatically replay prior input. +Await operations sequentially. After input, read fresh state to verify the effect. A successful action means dispatched, not verified completion. Cancellation may leave partial effects; never retry uncertain input without observing. STALE_APP requires getApp again; STALE_SNAPSHOT requires getState again. Absolute coordinate input requires a fresh screenshot of that window. Relative input can use a successful AX observation or screenshot. SCREENSHOT_REQUIRED means capture a fresh screenshot before using pixel coordinates. For click and hover, coordinate targets must be [x, y], for example hover([650, 400], {relative:true}). A bare number such as hover(650) is always an AX element index. The relative option applies only to coordinate targets, not AX indices. Window resizing since the latest observation rejects coordinate input; movement alone preserves window-local coordinates. WINDOW_CHANGED requires fresh state before any retry; use an advertised AX action or explicitly restore the window before screenshot or coordinate input, then observe again. PERMISSION_DENIED requires the user to grant the required macOS system permissions for macOS Accessibility or Screen Recording. After granting access, observe fresh state before continuing; do not automatically replay prior input. Hover first uses advertised AX alternate UI without moving the system cursor, preserving revealed descendants until the pointer target leaves their region. When unavailable, it moves the visible system cursor without pressing a button; that route requires the target point to be unobscured in the foreground app. FOREGROUND_REQUIRED requires explicit activation or raising and a fresh observation before retrying. Returning, AX reads and screenshots do not restore the previous point. Wait for the application's hover delay, observe, then hover into the revealed controls before clicking. Use successive hover calls for intermediate points. The result reports hoverRoute and systemPointerMoved. Other input or application changes can interrupt the interaction; app objects do not reserve applications between calls. diff --git a/tsconfig.base.json b/tsconfig.base.json index e39c17aa4b..cffc7e7341 100644 --- a/tsconfig.base.json +++ b/tsconfig.base.json @@ -303,6 +303,8 @@ "@deepseek-ai/dsh-experimental-client-ui-agent-team/client": ["./packages/experimental/client-ui-agent-team/src/client/index.ts"], "@deepseek-ai/dsh-client-ui-settings-plugins": ["./packages/client/ui-settings-plugins/src"], "@deepseek-ai/dsh-client-ui-settings-shell": ["./packages/client/ui-settings-shell/src"], + "@deepseek-ai/dsh-cua/client": ["./packages/experimental/cua/src/client/index.ts"], + "@deepseek-ai/dsh-cua/types": ["./packages/experimental/cua/src/remote-types.ts"], "@deepseek-ai/dsh-client-ui-settings-agent-loop": ["./packages/client/ui-settings-agent-loop/src"], "@deepseek-ai/dsh-client-ui-settings-session-log": ["./packages/client/ui-settings-session-log/src"], "@deepseek-ai/dsh-client-ui-settings-subagent": ["./packages/client/ui-settings-subagent/src"], @@ -399,6 +401,7 @@ "@deepseek-ai/dsh-config-editor": ["./packages/boot/config-editor/src"], "@deepseek-ai/dsh-credentials": ["./packages/credentials/credentials/src"], "@deepseek-ai/dsh-credentials-local": ["./packages/credentials/credentials-local/src"], + "@deepseek-ai/dsh-cua-bundle": ["./packages/experimental/cua-bundle/src"], "@deepseek-ai/dsh-deepseek-llm-api-extensions": ["./packages/llm/deepseek-llm-api-extensions/src"], "@deepseek-ai/dsh-deque": ["./packages/util/deque/src"], "@deepseek-ai/dsh-file-reference": ["./packages/context/file-reference/src"], diff --git a/tsconfig.client.json b/tsconfig.client.json index b2f0423c7c..c335a29e83 100644 --- a/tsconfig.client.json +++ b/tsconfig.client.json @@ -131,6 +131,7 @@ { "path": "./packages/client/ui-settings-plugins" }, { "path": "./packages/client/ui-settings-shell" }, { "path": "./packages/client/ui-settings-agent-loop" }, + { "path": "./packages/experimental/cua/tsconfig.client.json" }, { "path": "./packages/client/ui-settings-session-log" }, { "path": "./packages/client/ui-settings-subagent" }, { "path": "./packages/client/ui-settings-web-search" }, diff --git a/tsconfig.host.json b/tsconfig.host.json index 3d4f8fbaa0..2e84f9cf02 100644 --- a/tsconfig.host.json +++ b/tsconfig.host.json @@ -74,6 +74,8 @@ "apps/web/tests/plugin-config.e2e.ts", "apps/web/tests/input-plugin-extensions.e2e.ts", "apps/web/tests/plugin-manager.e2e.ts", + "apps/web/tests/cua-optin.e2e.ts", + "apps/web/tests/cua-permissions.e2e.ts", "apps/web/tests/schedule-header.e2e.ts", "apps/web/tests/worktree-opt-in.e2e.ts", "apps/web/tests/session-inspector.e2e.ts", @@ -274,7 +276,8 @@ { "path": "./packages/computer-use/computer-use" }, { "path": "./packages/experimental/computer-use-cua-driver-mcp" }, { "path": "./packages/experimental/computer-use-cua-driver-native" }, - { "path": "./packages/experimental/cua" }, + { "path": "./packages/experimental/cua/tsconfig.host.json" }, + { "path": "./packages/experimental/cua-bundle" }, { "path": "./packages/experimental/cua-macos" }, { "path": "./packages/client/ui-sidebar-documentpreview/tsconfig.host.json" }, { "path": "./packages/client/ui-sidebar-browser/tsconfig.host.json" }, diff --git a/vitest.config.ts b/vitest.config.ts index 9c717c0ced..56d206808f 100644 --- a/vitest.config.ts +++ b/vitest.config.ts @@ -331,6 +331,9 @@ export default defineConfig({ // The speech entry also imports generated Remote definitions; voice-input.e2e.ts // exercises the built entry, while source tests cover mountVoiceInput. 'packages/experimental/client-ui-voice-input/src/client/index.ts', + // The permission entry binds generated Remote code; cua-permissions.e2e.ts covers + // its built entry, while source tests exercise mountCuaPermissions and its UI. + 'packages/experimental/cua/src/client/index.ts', // The mods band entry imports the bridge's generated Remote contribution, which exists only in lib; // the Web snapshot exercises the built entry, while source tests cover mountModsBand. 'packages/experimental/client-ui-claude-code-mods/src/client/index.ts',