diff --git a/.agents/notes/implemented/architecture/2026-09-17-experimental-cua-object-api.i18n.yaml b/.agents/notes/implemented/architecture/2026-09-17-experimental-cua-object-api.i18n.yaml index 865e51fdec..7383dc7a71 100644 --- a/.agents/notes/implemented/architecture/2026-09-17-experimental-cua-object-api.i18n.yaml +++ b/.agents/notes/implemented/architecture/2026-09-17-experimental-cua-object-api.i18n.yaml @@ -9,11 +9,11 @@ en: a22ddcfb339e503d zh: a1f4fadb46fce1e7 /agent-note-native-cua-objects-independent-of-evaluator-ownership/decision: - en: 6ce9add827f6f245 - zh: e90f7fd13d058c87 + en: dce5ac6347fe50a8 + zh: 84df50be936e16dc /agent-note-native-cua-objects-independent-of-evaluator-ownership/alternatives-considered: en: 62dfcec71ed2a058 zh: 0f3c6edfe19fef1f /agent-note-native-cua-objects-independent-of-evaluator-ownership/consequences: - en: b6a6be9b8eb89f24 - zh: 5f0b1fb28699d4ab + en: 528c644d5810e60b + zh: 9649dfc3b1172a6d diff --git a/.agents/notes/implemented/architecture/2026-09-17-experimental-cua-object-api.md b/.agents/notes/implemented/architecture/2026-09-17-experimental-cua-object-api.md index a2c87838b4..6b67f2b33d 100644 --- a/.agents/notes/implemented/architecture/2026-09-17-experimental-cua-object-api.md +++ b/.agents/notes/implemented/architecture/2026-09-17-experimental-cua-object-api.md @@ -14,7 +14,7 @@ The experimental CUA package owns a native-app subset of the Codex-style object The facade accepts an operation dispatcher. Its tool adapter resolves the current tool binding for each operation, preserving normal authorization, logging and image admission. The execution host owns variable persistence and completion serialization. CUA defines no persistence directive, worker or run_code replacement. -A live Agent owns one serial native helper. App handles bind process identity and kernel start time; replacing the process or helper never revives old handles. AX snapshots are scoped to an app and optional window, so independent window observations remain usable. Coordinate actions select an owned window and use window-local points. Keyboard input remains app-scoped because macOS delivers it to the app's current focus. +Native process ownership follows the [shared helper decision](2026-10-09-cua-shared-native-helper.md), with isolated Agent connections. App handles bind process identity and kernel start time; replacing the process or helper never revives old handles. AX snapshots are scoped to an app and optional window, so independent window observations remain usable. Coordinate actions select an owned window and use window-local points. Keyboard input remains app-scoped because macOS delivers it to the app's current focus. The operation schema owns shared data declarations. Host-only limits and launch continuations remain separate from model parameters, and generated native field/enum tables preserve independent native admission without hand-maintained copies. Native code continues to own system calls, permissions and cancellation. AX formatting consumes immutable observations; the walker retains sole ownership of indices and AX references. @@ -30,4 +30,4 @@ A CUA-specific persistent evaluator duplicates execution lifecycle policy and pr The package remains opt-in and macOS-only. Observations do not lock the desktop, and input success records dispatch rather than task completion. Callers must verify effects and must not retry uncertain input automatically. Native fixture tests verify actual window state separately from protocol and model-transcript tests. Persistent execution, binary distribution, permission onboarding and real-model task success remain separately owned work. -Source builds retain a content-addressed executable per compiled generation. On macOS 26.6.2, byte-identical release helpers captured successfully from new paths but timed out through ScreenCaptureKit at a reused publication path with older helpers still running. This observation does not establish the operating system’s internal cause; retaining generations avoids replacing a live helper’s file. The package payload remains separately published for distribution. +Source builds retain a content-addressed executable per compiled generation. The [shared helper decision](2026-10-09-cua-shared-native-helper.md) owns capture-client identity and cross-Host lifetime. Retained source generations avoid replacing a live executable, while the package payload remains separately published for distribution. diff --git a/.agents/notes/implemented/architecture/2026-09-17-experimental-cua-object-api.zh.md b/.agents/notes/implemented/architecture/2026-09-17-experimental-cua-object-api.zh.md index 971a373de5..5645b47bdb 100644 --- a/.agents/notes/implemented/architecture/2026-09-17-experimental-cua-object-api.zh.md +++ b/.agents/notes/implemented/architecture/2026-09-17-experimental-cua-object-api.zh.md @@ -14,7 +14,7 @@ Computer-use 程序需要一致的应用和窗口操作。将这些操作与 CUA 对象接口接收操作分派器。工具适配模块每次操作都获取当前工具绑定，保留标准授权、日志和图片接入。执行宿主负责变量持久化和最终表达式值的序列化。CUA 不定义持久化指令、执行 worker 或 run_code 替代实现。 -每个活动 Agent 拥有一个串行原生辅助进程。应用句柄绑定进程身份及内核启动时间，替换进程或辅助进程不会使旧句柄重新有效。AX 快照按应用和可选窗口保存，不同窗口的观察可以分别使用。坐标操作选择属于该应用的窗口，并使用窗口内 point。键盘输入保持应用范围，因为 macOS 将其送往应用的当前焦点。 +原生进程归属遵循[共享 helper 决策](2026-10-09-cua-shared-native-helper.zh.md)，Agent 连接保持隔离。应用句柄绑定进程身份及内核启动时间，替换进程或辅助进程不会使旧句柄重新有效。AX 快照按应用和可选窗口保存，不同窗口的观察可以分别使用。坐标操作选择属于该应用的窗口，并使用窗口内 point。键盘输入保持应用范围，因为 macOS 将其送往应用的当前焦点。 操作 schema 负责共享数据声明。Host 专属限制和启动续接与模型参数分开，生成的原生字段、枚举表保留独立原生校验，同时避免手工维护副本。系统调用、权限和取消仍由原生代码负责。AX 格式化只消费不可变观察数据，遍历器独自负责索引与 AX 引用。 @@ -30,4 +30,4 @@ CUA 专用持久执行器会重复实现执行生命周期规则，使 PTC 无 本包需要显式启用，且只提供 macOS 实现。观察不会锁定桌面，输入成功只记录分派完成，不能证明任务完成。调用方必须验证效果，不能自动重试结果不确定的输入。原生测试通过实际窗口状态验证效果，与协议和模型消息测试分别进行。持久执行、二进制分发、权限引导及真实模型任务成功率是分别负责的工作。 -源码构建按编译产物的内容摘要保留可执行文件。在 macOS 26.6.2 上，同一份 release 二进制从新路径运行时截图成功，从仍有旧 helper 运行的重复发布路径运行时，ScreenCaptureKit 截图超时。这项观察不能确定操作系统内部原因；保留各代产物可以避免替换运行中 helper 使用的文件。包内分发产物另行发布。 +源码构建按编译产物的内容摘要保留可执行文件。[共享 helper 决策](2026-10-09-cua-shared-native-helper.zh.md)负责捕获客户端身份与跨 Host 生命期。源码各代副本避免替换运行中的可执行文件，包内产物仍单独发布。 diff --git a/.agents/notes/implemented/architecture/2026-09-26-cua-application-consent.i18n.yaml b/.agents/notes/implemented/architecture/2026-09-26-cua-application-consent.i18n.yaml index cacdc69bdb..a5e3035bd0 100644 --- a/.agents/notes/implemented/architecture/2026-09-26-cua-application-consent.i18n.yaml +++ b/.agents/notes/implemented/architecture/2026-09-26-cua-application-consent.i18n.yaml @@ -9,11 +9,11 @@ en: b32a7ce6483b9b48 zh: ec0938ad032b99d8 /agent-note-application-consent-for-native-computer-use/decision: - en: 9234627787165031 - zh: 6fbb56110887a957 + en: 92247ff599c1103f + zh: 26845420a9711c83 /agent-note-application-consent-for-native-computer-use/alternatives-considered: - en: 05fd6823e5c2e18f - zh: 998d7a2030aa0b91 + en: 4693dcd217c4fedb + zh: 1dc9bf9067314006 /agent-note-application-consent-for-native-computer-use/consequences: - en: b329612bb75b4fbe - zh: 70ec1552843d2832 + en: 0146a69ab2086edc + zh: 4257bc1ab4432c3f diff --git a/.agents/notes/implemented/architecture/2026-09-26-cua-application-consent.md b/.agents/notes/implemented/architecture/2026-09-26-cua-application-consent.md index 56009e8ce1..3fb91170e5 100644 --- a/.agents/notes/implemented/architecture/2026-09-26-cua-application-consent.md +++ b/.agents/notes/implemented/architecture/2026-09-26-cua-application-consent.md @@ -12,13 +12,13 @@ macOS privileges authorize an execution host, not a user's choice of target appl The CUA executor owns application consent for every object-API and native-tool dispatch. It uses the existing User Questions service for human decisions and the provider's existing profile configuration for saved grants. General tool approval and operating-system privileges remain separate requirements. Arbitrary persistent PTC code and separately authorized shell tools retain their own execution authority. -A saved identity combines the canonical executable path with a verified designated signing requirement, represented by a SHA-256 digest. The path prevents authorization from following a copied application, and the signing requirement distinguishes replacements while allowing matching signed updates. Unverifiable and ad-hoc applications receive task access only, tied to a process instance and helper lifetime. Labels do not establish identity. +A saved identity combines the canonical executable path with a verified designated signing requirement, represented by a SHA-256 digest. The path prevents authorization from following a copied application, and the signing requirement distinguishes replacements while allowing matching signed updates. Unverifiable and ad-hoc applications receive task access only, tied to a process instance and Agent connection lifetime. Labels do not establish identity. Cold application selection prepares an executable identity and handle without launching. The same consent decision guards the private launch continuation, and the helper checks the executable again before launch and binds the resulting process afterward. Signed selections retain the normal designated-requirement identity; an unverifiable executable uses a content identity for this pending selection and cannot create a saved grant. Preparing metadata before asking avoids starting an application the user refuses. Selections of the same application within one task are serialized, so a pending question does not produce competing decisions. Saved-grant additions read the current configuration inside one mutation queue. Refusals prevent another prompt within the task; an explicitly restored saved grant takes precedence at dispatch. -Removing a saved grant invalidates native helpers retaining the application's handles and cancels active operations; Agent/provider disposal awaits helper cleanup. Reauthorization uses a fresh helper, so old app and window snapshots cannot become valid again. The [native object decision](2026-09-17-experimental-cua-object-api.md) continues to own process identity, coordinate semantics, and separation from the evaluator. +Removing a saved grant invalidates Agent connections retaining the application's handles and cancels active operations; Agent/provider disposal awaits native connection cleanup. Reauthorization uses a fresh connection, so old app and window snapshots cannot become valid again. The [native object decision](2026-09-17-experimental-cua-object-api.md) continues to own process identity, coordinate semantics, and separation from the evaluator. ## Alternatives considered @@ -28,11 +28,11 @@ Removing a saved grant invalidates native helpers retaining the application's ha **Add an approval database or Session control stream.** Profile settings already own user configuration, and User Questions already owns interactive answers. -**Keep the helper after revocation.** Its application lookup can return an earlier handle and retain window snapshots. Invalidating the helper uses existing process ownership and transport cleanup; preserving unrelated handles would require a native per-application retirement protocol. +**Keep the native context after revocation.** Its application lookup can return an earlier handle and retain window snapshots. Closing the Agent connection discards its native context through existing transport cleanup; preserving unrelated handles would require a native per-application retirement protocol. ## Consequences -Revoking one application also invalidates other application handles in the same helper. Those applications require fresh selection; saved grants remain reusable, while task grants expire. Other tasks with independent helpers remain unaffected unless they also hold access to the revoked application. +Revoking one application also invalidates other application handles on the same Agent connection. Those applications require fresh selection; saved grants remain reusable, while task grants expire. Other tasks with independent connections remain unaffected unless they also hold access to the revoked application. Application decisions and failure results are exercised through policy regressions and the recorded CUA scenario. Desktop binary distribution, signing, and packaged-host permissions remain outside the source-launch workflow. diff --git a/.agents/notes/implemented/architecture/2026-09-26-cua-application-consent.zh.md b/.agents/notes/implemented/architecture/2026-09-26-cua-application-consent.zh.md index 7200b153b3..73d4640db7 100644 --- a/.agents/notes/implemented/architecture/2026-09-26-cua-application-consent.zh.md +++ b/.agents/notes/implemented/architecture/2026-09-26-cua-application-consent.zh.md @@ -12,13 +12,13 @@ macOS 权限授权执行宿主，不能表达用户允许操作哪些目标应 CUA 执行器在每次对象 API 和原生工具调用时检查应用授权。人工决定复用 User Questions 服务，长期许可保存在 provider 的现有 profile 配置中。通用工具审批和操作系统权限仍是独立要求。任意 persistent PTC 代码和另行授权的 shell 工具保留各自的执行权限。 -长期身份由规范化可执行文件路径和经过验证的签名指定要求组成，以 SHA-256 摘要表示。路径阻止许可跟随应用副本，签名要求区分替换程序，并允许符合相同签名要求的更新。签名不可验证或使用 ad-hoc 签名的应用只能获得任务授权，许可绑定进程实例和 helper 生命周期。显示名称不用于确定身份。 +长期身份由规范化可执行文件路径和经过验证的签名指定要求组成，以 SHA-256 摘要表示。路径阻止许可跟随应用副本，签名要求区分替换程序，并允许符合相同签名要求的更新。签名不可验证或使用 ad-hoc 签名的应用只能获得任务授权，许可绑定进程实例和 Agent 连接生命周期。显示名称不用于确定身份。 选择尚未运行的应用时，先准备可执行文件身份和句柄，不立即启动。同一授权决定保护内部启动续接，helper 在启动前再次核对可执行文件，并在启动后绑定实际进程。已签名应用沿用签名指定要求身份；签名不可验证的程序使用本次待启动选择的内容身份，不能生成长期许可。先准备元数据再询问，可以避免启动用户拒绝的应用。 同一任务对同一应用的选择串行执行，避免未完成的问题产生相互冲突的决定。长期许可的新增在同一写入队列内读取当前配置。拒绝会阻止任务内的再次询问；用户显式恢复的长期许可在执行检查时优先于旧拒绝记录。 -删除长期许可会使保留目标应用句柄的原生 helper 失效，并取消活动操作；Agent 或提供方销毁时等待组件清理。重新授权使用新 helper，因此旧应用对象和窗口快照不能重新生效。[原生对象决策](2026-09-17-experimental-cua-object-api.zh.md)继续负责进程身份、坐标语义和与执行器分离的设计。 +删除长期许可会使保留目标应用句柄的 Agent 连接失效，并取消活动操作；Agent 或提供方销毁时等待对应的原生连接清理。重新授权使用新连接，因此旧应用对象和窗口快照不能重新生效。[原生对象决策](2026-09-17-experimental-cua-object-api.zh.md)继续负责进程身份、坐标语义和与执行器分离的设计。 ## 考虑过的替代方案 @@ -28,11 +28,11 @@ CUA 执行器在每次对象 API 和原生工具调用时检查应用授权。 **新增审批数据库或 Session 控制流。** Profile settings 已负责用户配置，User Questions 已负责交互回答。 -**撤销后保留 helper。** 原生应用查找可能返回旧句柄并保留窗口快照。使 helper 失效可以复用现有进程归属和传输清理；保留其他应用的句柄则需要新增按应用销毁状态的原生协议。 +**撤销后保留原生状态。** 原生应用查找可能返回旧句柄并保留窗口快照。关闭 Agent 连接可以通过现有传输清理释放其原生状态；保留其他应用的句柄则需要新增按应用销毁状态的原生协议。 ## 影响 -撤销一个应用也会使同一 helper 中其他应用的句柄失效。这些应用需要重新选择；长期许可仍可复用，任务许可随之失效。其他任务使用独立 helper，除非也持有被撤销应用的许可，否则不受影响。 +撤销一个应用也会使同一 Agent 连接中其他应用的句柄失效。这些应用需要重新选择；长期许可仍可复用，任务许可随之失效。其他任务使用独立连接，除非也持有被撤销应用的许可，否则不受影响。 授权决定和失败结果由策略回归测试及 CUA 录制会话验证。桌面安装包的二进制分发、签名和打包宿主权限不属于源码启动流程。 diff --git a/.agents/notes/implemented/architecture/2026-10-09-cua-shared-native-helper.i18n.yaml b/.agents/notes/implemented/architecture/2026-10-09-cua-shared-native-helper.i18n.yaml new file mode 100644 index 0000000000..8a1d618d93 --- /dev/null +++ b/.agents/notes/implemented/architecture/2026-10-09-cua-shared-native-helper.i18n.yaml @@ -0,0 +1,19 @@ +# Bilingual-pair consistency record for 2026-10-09-cua-shared-native-helper.md (docs/i18n/README.md): per heading +# section, a hash of its English and Chinese blocks outside code blocks and generated regions. +# After editing either side, bring the other along and re-record with: +# pnpm run verify-translation-pairing --write .agents/notes/implemented/architecture/2026-10-09-cua-shared-native-helper.md +/agent-note-shared-native-cua-process-with-agent-scoped-state: + en: 65056aee19bf6b22 + zh: 9823fa0c33497ff9 +/agent-note-shared-native-cua-process-with-agent-scoped-state/problem: + en: 386864a282e5ad58 + zh: a4ea75ec70eb7778 +/agent-note-shared-native-cua-process-with-agent-scoped-state/decision: + en: ebf231a6aad109bb + zh: 53b553e679569da1 +/agent-note-shared-native-cua-process-with-agent-scoped-state/alternatives-considered: + en: bd3b8760b5f447df + zh: 3ad546ef7494c42c +/agent-note-shared-native-cua-process-with-agent-scoped-state/consequences: + en: 518a4abf2c4c95be + zh: 72bf2b9817975368 diff --git a/.agents/notes/implemented/architecture/2026-10-09-cua-shared-native-helper.md b/.agents/notes/implemented/architecture/2026-10-09-cua-shared-native-helper.md new file mode 100644 index 0000000000..28807abcfc --- /dev/null +++ b/.agents/notes/implemented/architecture/2026-10-09-cua-shared-native-helper.md @@ -0,0 +1,25 @@ +# Agent Note: Shared native CUA process with Agent-scoped state + +Status: implemented + +English | [中文](2026-10-09-cua-shared-native-helper.zh.md) + +## Problem + +On macOS 26.6.2, replayd derives capture-client identity from the executable path. Two live helpers at the same path repeatedly replace each other's connections even when their screenshot requests are sequential. Byte-identical executables at distinct real paths complete sequential and concurrent captures; two symlinks to one executable reproduce the failure. A per-request lock cannot isolate retained system connections. + +## Decision + +One user shares a native service per executable content digest across Agents and Hosts. A version lock elects the service; an additional canonical-executable-path lock prevents different builds published at one path from running together. Locks remain held until process exit. The managed startup command hands service lifetime to its client connections, and the last connection's completed cleanup ends the service. The diagnostic stdin/stdout entry proxies to this same service. + +Each Agent connection owns its application handles, AX snapshots, limits and cancellation state. An app handle grants no exclusive use of the application. A FIFO orders complete desktop operations, including clipboard transactions, while response writes are independent. Agent disposal retires only its connection. The [object API decision](2026-09-17-experimental-cua-object-api.md) retains evaluator independence and coordinate semantics; the [consent decision](2026-09-26-cua-application-consent.md) retains application authorization. + +Cancellation and timeout first stop one request and wait for acknowledgement. A timed-out screenshot retains the single system-RPC slot until the real callback completes; new screenshots fail with `CAPTURE_BUSY` instead of accumulating requests. Late pixels are discarded, while other AX and input operations remain available. Ending an await or creating a wrapper does not prove that the system connection has recovered. An operation or cleanup that cannot stop can invalidate the process, making every old handle unusable. Uncertain input is never replayed automatically. + +## Alternatives considered + +Separate executable copies per Agent avoid the observed identity collision but multiply system capture clients. One shared service preserves a single capture client while retaining Agent state isolation. An in-process Swift binding changes crash containment without removing the system RPC. Sharing a process does not establish application-level ownership across a multi-step interaction; that scheduling policy remains separate. + +## Consequences + +Two Hosts can race to start the service and receive the same instance. A may retain its app object while B screenshots and changes the same application; A reads fresh AX afterward. An Agent closing cannot retire another Agent's handles. Recovery must distinguish request, snapshot, handle, capture-backend and process failure. A lost system callback occupies a bounded capture slot; reliable in-process backend reset requires separate platform evidence. The historical DeepSeek failures have matching RPC errors, but their competing-client identity was not recovered from retained logs. diff --git a/.agents/notes/implemented/architecture/2026-10-09-cua-shared-native-helper.zh.md b/.agents/notes/implemented/architecture/2026-10-09-cua-shared-native-helper.zh.md new file mode 100644 index 0000000000..3674b6cea1 --- /dev/null +++ b/.agents/notes/implemented/architecture/2026-10-09-cua-shared-native-helper.zh.md @@ -0,0 +1,25 @@ +# Agent Note：共享原生 CUA 进程并隔离 Agent 状态 + +Status: implemented + +[English](2026-10-09-cua-shared-native-helper.md) | 中文 + +## 问题 + +在 macOS 26.6.2 上，replayd 按可执行文件路径生成捕获客户端标识。同一路径的两个活动 helper 即使依次截图，也会反复替换对方的连接。相同字节的可执行文件位于不同实际路径时，交替和并发截图均成功；两个符号链接指向同一文件仍会失败。逐请求加锁不能隔离仍存活的系统连接。 + +## 决策 + +同一用户的多个 Agent 和 Host 按可执行文件内容摘要共享原生服务。版本锁选出服务实例，规范化可执行路径上的另一把锁阻止发布在同一路径的不同构建同时运行。进程退出才释放锁。受管启动命令将服务生命期交给客户端连接，最后一条连接清理完成后服务退出。诊断用 stdin/stdout 入口也代理到同一个服务。 + +每条 Agent 连接独立持有应用句柄、AX 快照、资源限制和取消状态。持有 app 句柄不代表独占应用。FIFO 队列串行执行包含剪贴板事务的完整桌面操作，响应写入独立进行。Agent 销毁只释放自己的连接。[对象接口决策](2026-09-17-experimental-cua-object-api.zh.md)继续负责执行器独立性及坐标语义，[授权决策](2026-09-26-cua-application-consent.zh.md)继续负责应用许可。 + +取消或超时先停止单次请求并等待确认。截图超时后，唯一的系统 RPC 槽位保留到真实回调结束；新截图返回 `CAPTURE_BUSY`，不继续堆积请求。迟到图片被丢弃，其他 AX 与输入操作仍可执行。结束 await 或创建包装对象不能证明系统连接已恢复。操作或清理无法停止时，才可能使进程失效，届时所有旧句柄均不可用。结果不确定的输入不会自动重放。 + +## 考虑过的替代方案 + +每个 Agent 使用独立的可执行文件副本可以避开已观察到的标识冲突，但会增加系统捕获客户端数量。共享服务保留一个捕获客户端，同时隔离 Agent 状态。进程内 Swift 绑定会改变崩溃隔离方式，不能消除系统 RPC。共享进程本身不提供跨多步交互的应用占用，相关调度规则需要单独设计。 + +## 后果 + +两个 Host 同时启动服务时可连接同一个实例。A 可以保留 app 对象，B 对同一应用截图并修改，之后 A 重新读取 AX。一个 Agent 关闭不会释放另一个 Agent 的句柄。恢复需要区分请求、快照、句柄、截图后端和进程失效。系统回调丢失会占用有界的捕获槽位，可靠的进程内后端重置需要另外取得平台验证。历史 DeepSeek 失败具有相同 RPC 错误，但保留日志未能确定当时另一个冲突客户端的身份。 diff --git a/packages/computer-use/cua/README.i18n.yaml b/packages/computer-use/cua/README.i18n.yaml index 1a3b19a54d..eb986905d5 100644 --- a/packages/computer-use/cua/README.i18n.yaml +++ b/packages/computer-use/cua/README.i18n.yaml @@ -15,14 +15,14 @@ en: d152484eb41ac6b4 zh: 7b30b973a849247d /deepseek-ai-dsh-cua/use-this-package: - en: 8678a408b3e5eaea - zh: aa1a498bc0bc50d5 + en: 851d4fc9ce8988cd + zh: 581a3c45f3c0ffea /deepseek-ai-dsh-cua/use-this-package/connect-an-external-execution-environment: - en: 9c5f4b4fe7147455 - zh: eca65a4ffd02edb4 + en: 8c3ee662123e7bb8 + zh: f92124b776260c1b /deepseek-ai-dsh-cua/understand-the-implementation: - en: 0ab772a0dce859f6 - zh: 7851201c70372a25 + en: 40978d9a4963a5b6 + zh: b4641bd2a739a987 /deepseek-ai-dsh-cua/further-exploration: en: b1047318879cd6f4 zh: e4ce4a5ad2a9c4dc @@ -45,8 +45,8 @@ en: a2c9acb5a4320dda zh: e8d1586af60215c4 /deepseek-ai-dsh-cua/known-limitations-and-deferred-work: - en: f7667917fd32b939 - zh: 2360f14900ef735b + en: c4f43ec8daf7537a + zh: 246280e9142fca01 /deepseek-ai-dsh-cua/known-limitations-and-deferred-work/dev-note: en: b836f01bd0bceb4a zh: 8122fa08b2aa95fa diff --git a/packages/computer-use/cua/README.md b/packages/computer-use/cua/README.md index 07e1ec14ed..74c2d1e318 100644 --- a/packages/computer-use/cua/README.md +++ b/packages/computer-use/cua/README.md @@ -42,6 +42,8 @@ The dump command checks profile composition without a model request. For a real | `helperPath` | package-owned executable | Optional absolute native executable override. | | `buildTimeoutMs` | `300000` | Maximum native preparation time. | | `timeoutMs` | `30000` | Per-operation deadline; cancellation can leave partial input. | +| `connectTimeoutMs` | `5000` | Deadline for connecting to or starting the shared helper. | +| `screenshotTimeoutMs` | `5000` | Screenshot deadline including queue time, capped by `timeoutMs`. | | `axTimeoutMs` | `3000` | Deadline per Accessibility system call, configured independently of the complete operation. | | `maxRequestBytes` / `maxResponseBytes` | `1048576` / `33554432` | UTF-8 request and response limits. | | `maxClipboardBytes` | `33554432` | Maximum original clipboard bytes preserved during a paste; oversized contents reject before replacement. | @@ -56,7 +58,7 @@ Keypad names (`KP_0` through `KP_9`, `KP_Enter`, and keypad operators) retain th Agent disposal and plugin unload cancel and await the complete tool call, including image admission and attachment storage. An image write already in progress must settle before disposal returns; a canceled call cannot return a successful screenshot. -Application consent is enforced before AX reads, window-title reads, screenshots, activation, and input through the CUA tool. User Questions offers task access or denial, plus saved access when the application has a verifiable signature and the composition provides settings persistence. Task access is bound to the selected process handle; saved access is bound to its canonical executable path and designated code-signing requirement. Denial prevents repeated prompts in that task. Removing a saved `allowedApps` entry cancels affected operations and invalidates their helper handles; reauthorization requires fresh selection. Other handles in the same helper also expire, while saved grants remain reusable. Agent/provider disposal joins helper cleanup. See the [application consent decision](../../../.agents/notes/implemented/architecture/2026-09-26-cua-application-consent.md). These checks govern CUA calls; persistent PTC and separately authorized shell tools retain their own execution authority. +Application consent is enforced before AX reads, window-title reads, screenshots, activation, and input through the CUA tool. User Questions offers task access or denial, plus saved access when the application has a verifiable signature and the composition provides settings persistence. Task access is bound to the selected process handle; saved access is bound to its canonical executable path and designated code-signing requirement. Denial prevents repeated prompts in that task. Removing a saved `allowedApps` entry cancels affected operations and invalidates their helper handles; reauthorization requires fresh selection. Other handles on the same Agent connection also expire, while saved grants remain reusable. Agent/provider disposal joins cleanup of the corresponding native connection. See the [application consent decision](../../../.agents/notes/implemented/architecture/2026-09-26-cua-application-consent.md). These checks govern CUA calls; persistent PTC and separately authorized shell tools retain their own execution authority. Raw `getState` exposes validated state fields and the complete text for the requested disclosure options. The native helper also returns structured element diagnostics for native checks; the tool projection omits that duplicate array from model text and structured tool values. @@ -66,7 +68,7 @@ Values and placeholders are separate. Named, stateful or actionable groups and s Observations emit by default and also return their value. In persistent PTC, use `void await app.getAXState()` for one emitted copy, or `emit:false` and one explicit print. Capturing a screenshot after an emitted tree can fail independently; retain the successful AX output. -Screenshot diagnostics use `[cua-screenshot]` records keyed by the tool request ID. Native records measure elapsed milliseconds from request receipt around window/AX geometry checks, ScreenCaptureKit enumeration and capture, PNG/Base64 encoding, and response writing. Host debug logs add dispatch, reply and cleanup completion timings. The helper stderr buffer retains at most 8192 bytes; available complete records for a failed screenshot are appended to its error and therefore remain in exported Session traces. Truncation is marked, and late stderr may be absent from successful-call diagnostics. Records contain stages, timings and error codes, not titles, AX text or image bytes. Diagnostic sink failures do not replace capture results. +Screenshot diagnostics use `[cua-screenshot]` records keyed by the tool request ID. Native phase records return over the originating Agent connection and cover window/AX geometry checks, system capture, PNG/Base64 encoding and response writing. Host debug logs add dispatch, reply and cleanup timings. Each request retains at most 8192 bytes of complete diagnostic records, with explicit truncation; screenshot failures append these records to the error for Session export. Records contain stages, timings and error codes, not window titles, AX text or image bytes. Native preparation serializes compiler checks and publication with the shared file lock. It updates the package payload and runs a content-addressed copy under native/.build/prepared, preserving executable files used by earlier source-runtime processes. Identical builds reuse the same file; corrupted retained copies fail validation. Cancellation and preparation failure finish managed process cleanup before releasing the lock. @@ -97,7 +99,7 @@ Retained objects resolve the current tool binding on every call. The external ru AppKit sheets are selectable by their window IDs and remain visible in their document window’s tree. Keyboard input follows a system Open/Save panel only when the application’s focused sheet identifies one Apple panel-service window and that service still focuses the same panel. Unrelated or ambiguous remote recipients fail before dispatch. -Indexed clicks support a single right/middle click or a single/double left click. Click and drag options accept a button and modifier keys; a modified click uses pointer input. Drags hold that button and carry its modifiers through every movement and release. AX text marks disabled controls, `valueWritable=true`, and available `selected` state. The helper uses an advertised accessibility action for unmodified clicks when available; pointer delivery resolves the element’s current bounds and verifies its owning window. Actions report dispatch, not application success. A visual pointer or click marker can be rendered independently of input; AX actions do not require moving the system cursor. Window-directed mouse input can still change application focus, selection or dialogs. Read state after input. `STALE_APP` requires acquiring the application again; `STALE_SNAPSHOT` requires a fresh AX read. Transport timeout/cancellation terminates that Agent’s helper and invalidates its handles. Never retry uncertain input automatically. Closing an Agent releases its helper and AX references, without undoing changes to applications. +Indexed clicks support a single right/middle click or a single/double left click. Click and drag options accept a button and modifier keys; a modified click uses pointer input. Drags hold that button and carry its modifiers through every movement and release. AX text marks disabled controls, `valueWritable=true`, and available `selected` state. The helper uses an advertised accessibility action for unmodified clicks when available; pointer delivery resolves the element’s current bounds and verifies its owning window. Actions report dispatch, not application success. A visual pointer or click marker can be rendered independently of input; AX actions do not require moving the system cursor. Window-directed mouse input can still change application focus, selection or dialogs. Read state after input. `STALE_APP` requires acquiring the application again; `STALE_SNAPSHOT` requires a fresh AX read. Request timeout/cancellation waits for native acknowledgement and preserves usable handles. Missing cleanup acknowledgement closes that Agent’s connection. A screenshot RPC still awaiting its system callback occupies the single capture slot; further screenshots return `CAPTURE_BUSY`, while AX and input remain available. A late capture result is discarded, and its real completion releases that slot. Only an unresponsive operation or failed cleanup can invalidate the shared process and all its handles. Never retry uncertain input automatically. Closing an Agent releases its connection and AX references, without undoing changes to applications. ----- @@ -107,7 +109,7 @@ Indexed clicks support a single right/middle click or a single/double left click
Dispatch and resource ownership -The object facade translates app/window methods into JSON operations. The bridge routes them through the normal tool pipeline, preserving authorization, nested logs and image admission. A separate serial helper belongs to each Agent. Native handles bind a process identity to its kernel start time, and snapshots retain native AX references only inside that helper. The same operation protocol can serve a future platform provider without adding platform logic to the object API. +The object facade translates app/window methods into JSON operations. The bridge routes them through the normal tool pipeline, preserving authorization, nested logs and image admission. Agents and Hosts belonging to one user share a helper for the same native build. Each connection retains independent handles, snapshots, configuration and cancellation state; desktop operations use a global queue. The managed launcher transfers service lifetime to these connections, and the helper exits after the last connection finishes cleanup. Process locks cover both build identity and the real executable path, preventing duplicate startup and path collisions after a running executable is replaced. Native handles bind a process identity to its kernel start time, and snapshots retain native AX references only inside that helper. The same operation protocol can serve a future platform provider without adding platform logic to the object API. [Zod operation definitions](src/protocol.ts) own model parameters and separately declare Host resource fields and private continuations. Public data types derive from those definitions. `pnpm exec tsx packages/computer-use/cua/native/generate-protocol.ts` generates the native JSON Schema and Swift field/enum admission tables; the package test lane checks freshness and rejects Host-only fields at model admission. Native target validation, resource limits and operation behavior remain enforced at execution. @@ -191,6 +193,8 @@ Tool results append to Session history. Changing plugin configuration or tool pr ## Known Limitations and Deferred Work +The queue orders individual native requests. Holding an app object does not reserve its application across a Session; application-level ownership spanning several requests remains separate work. Capture RPCs that never return occupy the single capture slot; creating new app objects does not reset that system connection. + - Only the macOS native implementation is provided. Browser tabs, AX diffs, unassociated system panels and accessory-only apps are unsupported. diff --git a/packages/computer-use/cua/README.zh.md b/packages/computer-use/cua/README.zh.md index e0f161f146..c634fa1491 100644 --- a/packages/computer-use/cua/README.zh.md +++ b/packages/computer-use/cua/README.zh.md @@ -42,6 +42,8 @@ pnpm dsh headless --patch packages/computer-use/cua/examples/macos.cordis.yml -- | `helperPath` | 包内可执行文件 | 可选的原生可执行文件绝对路径覆盖。 | | `buildTimeoutMs` | `300000` | 原生组件准备的最长时间。 | | `timeoutMs` | `30000` | 单次操作截止时间，取消后可能已发生部分输入。 | +| `connectTimeoutMs` | `5000` | 连接或启动共享 helper 的截止时间。 | +| `screenshotTimeoutMs` | `5000` | 截图总期限，包含排队时间，且不超过 `timeoutMs`。 | | `axTimeoutMs` | `3000` | 每次 Accessibility 系统调用的截止时间，与完整操作截止独立配置。 | | `maxRequestBytes` / `maxResponseBytes` | `1048576` / `33554432` | UTF-8 请求与响应字节限制。 | | `maxClipboardBytes` | `33554432` | 粘贴时保存原剪贴板内容的字节上限；超限会在替换前拒绝。 | @@ -56,7 +58,7 @@ pnpm dsh headless --patch packages/computer-use/cua/examples/macos.cordis.yml -- Agent 销毁和插件卸载会取消并等待完整工具调用，包括图像接入检查和附件存储。已经开始的图像写入必须结束后，销毁才会返回；已取消的调用不会返回成功截图。 -CUA 工具在读取 AX、窗口标题、截图、激活和输入之前检查应用授权。User Questions 提供任务级允许和拒绝选项；应用签名可验证且组合支持配置持久化时，也可保存授权。任务授权绑定所选进程句柄，长期授权绑定真实可执行路径和指定代码签名要求。拒绝后，同一任务不再反复询问。删除保存的 `allowedApps` 条目会取消相关操作并使原生组件中的句柄失效；重新授权需要重新选择应用。同一组件中的其他句柄也会失效，已保存的授权仍可复用。Agent 或提供方销毁时等待组件清理。设计理由见[应用授权决策](../../../.agents/notes/implemented/architecture/2026-09-26-cua-application-consent.zh.md)。这些检查约束 CUA 调用；持久 PTC 和另行授权的 shell 工具保留各自的执行权限。 +CUA 工具在读取 AX、窗口标题、截图、激活和输入之前检查应用授权。User Questions 提供任务级允许和拒绝选项；应用签名可验证且组合支持配置持久化时，也可保存授权。任务授权绑定所选进程句柄，长期授权绑定真实可执行路径和指定代码签名要求。拒绝后，同一任务不再反复询问。删除保存的 `allowedApps` 条目会取消相关操作并使原生组件中的句柄失效；重新授权需要重新选择应用。同一 Agent 连接中的其他句柄也会失效，已保存的授权仍可复用。Agent 或提供方销毁时等待对应连接的原生状态清理。设计理由见[应用授权决策](../../../.agents/notes/implemented/architecture/2026-09-26-cua-application-consent.zh.md)。这些检查约束 CUA 调用；持久 PTC 和另行授权的 shell 工具保留各自的执行权限。 原始 `getState` 操作返回经过校验的状态字段，以及按本次披露选项生成的完整文本。原生 helper 还返回供原生检查使用的结构化元素诊断数组；工具投影会从模型文本和结构化工具值中移除这份重复数组。 @@ -66,7 +68,7 @@ AX 观察默认只展示菜单栏入口。使用 `app.getAXState({ includeMenus: 观察方法默认自动输出，同时返回结果。在 persistent PTC 中使用 `void await app.getAXState()` 输出一次，或指定 `emit:false` 后自行打印一次。AX 输出后的截图可能单独失败，已成功取得的 AX 输出仍保留。 -截图诊断使用带工具请求 ID 的 `[cua-screenshot]` 记录。原生日志记录从收到请求开始的累计毫秒数，覆盖窗口及 AX 几何检查、ScreenCaptureKit 窗口枚举与截图、PNG/Base64 编码和响应写出。Host 调试日志补充请求发送、响应到达和清理完成的耗时。helper 的 stderr 缓冲最多保留 8192 字节；截图失败时，可用的本次完整记录会附在错误中，因此能在导出的 Session trace 中查看。缓冲截断会明确标注，成功调用的诊断可能缺少迟到的 stderr 记录。日志只含阶段、耗时和错误码，不含窗口标题、AX 正文或图片内容。诊断写入失败不会覆盖截图结果。 +截图诊断使用带工具请求 ID 的 `[cua-screenshot]` 记录。原生阶段记录经本次 Agent 连接返回，覆盖窗口及 AX 几何检查、系统截图、PNG/Base64 编码和响应写出。Host 调试日志补充发送、响应和清理耗时。每次请求最多保留 8192 字节完整诊断记录，超限会明确标记；截图失败时附在错误中，保留到 Session trace。记录只含阶段、耗时和错误码，不含窗口标题、AX 正文或图片内容。 原生准备复用文件锁，串行检查编译和发布。构建更新包内产物，并从 native/.build/prepared 下按内容摘要区分的副本运行，保留旧源码进程使用的可执行文件。相同构建复用同一文件；保留的副本损坏时拒绝使用。取消或准备失败后，先完成受管进程清理再释放锁。 @@ -97,7 +99,7 @@ void await app.getScreenshot() AppKit sheet 可以通过其窗口 ID 单独选择，也会出现在所属文档窗口的控件树中。只有应用当前获得焦点的 sheet 指向唯一的 Apple 面板服务窗口，且该服务仍聚焦于同一面板时，键盘输入才转交系统打开或保存面板。无法确认或存在歧义的远端接收方会在输入前报错。 -元素索引支持右键或中键单击以及左键单击、双击。点击和拖动选项可指定按键与修饰键；带修饰键的点击使用指针输入。拖动期间保持指定按键，并在移动和释放事件中携带修饰键。AX 文本标明禁用控件、`valueWritable=true` 和可读取的 `selected` 状态。不带修饰键的点击优先使用元素声明的辅助功能动作；需要发送鼠标事件时，会重新读取元素位置并核实所属窗口。动作返回只表示输入已分派。可视指针或点击标记可以独立绘制；AX 动作无需移动系统指针。窗口定向鼠标输入仍可能改变应用焦点、选区或对话框。输入后需要读取状态确认结果。`STALE_APP` 需要重新获取应用，`STALE_SNAPSHOT` 需要重新读取 AX。通信超时或取消会终止该 Agent 的辅助进程，使已有句柄失效。不能自动重试结果不确定的输入。关闭 Agent 会释放辅助进程和 AX 引用，但不会撤销应用中的修改。 +元素索引支持右键或中键单击以及左键单击、双击。点击和拖动选项可指定按键与修饰键；带修饰键的点击使用指针输入。拖动期间保持指定按键，并在移动和释放事件中携带修饰键。AX 文本标明禁用控件、`valueWritable=true` 和可读取的 `selected` 状态。不带修饰键的点击优先使用元素声明的辅助功能动作；需要发送鼠标事件时，会重新读取元素位置并核实所属窗口。动作返回只表示输入已分派。可视指针或点击标记可以独立绘制；AX 动作无需移动系统指针。窗口定向鼠标输入仍可能改变应用焦点、选区或对话框。输入后需要读取状态确认结果。`STALE_APP` 需要重新获取应用，`STALE_SNAPSHOT` 需要重新读取 AX。请求超时或取消会等待原生停止确认，保留仍可用的句柄；无法确认清理完成时关闭该 Agent 的连接。未返回系统回调的截图 RPC 占用唯一捕获槽位，新截图立即返回 `CAPTURE_BUSY`，AX 和输入仍可使用。迟到图片被丢弃，真实回调结束后释放槽位。只有操作无法停止或清理失败时，才可能使共享进程及所有旧句柄失效。不能自动重试结果不确定的输入。关闭 Agent 会释放其连接和 AX 引用，但不会撤销应用中的修改。 ----- @@ -107,7 +109,7 @@ AppKit sheet 可以通过其窗口 ID 单独选择，也会出现在所属文档
操作分派与资源归属 -对象接口将应用和窗口方法转换为 JSON 操作。适配模块通过标准工具链保留授权、嵌套日志和图片接入。每个 Agent 拥有一个串行处理请求的辅助进程。原生句柄绑定进程身份及内核启动时间，快照中的 AX 引用只保存在辅助进程内。未来平台可以实现相同操作协议，无需在对象接口中加入平台逻辑。 +对象接口将应用和窗口方法转换为 JSON 操作。适配模块通过标准工具链保留授权、嵌套日志和图片接入。同一用户的相同 helper 构建由多个 Agent 和 Host 共享。每条连接独立保存句柄、快照、配置和取消状态，桌面操作通过全局队列执行。受管启动器将服务生命期交给这些连接，最后一条连接清理完成后服务退出。进程锁覆盖构建身份及真实可执行路径，防止重复启动或覆盖运行版本后产生路径冲突。原生句柄绑定进程身份及内核启动时间，快照中的 AX 引用只保存在辅助进程内。未来平台可以实现相同操作协议，无需在对象接口中加入平台逻辑。 [Zod 操作定义](src/protocol.ts)负责模型参数，并单独声明 Host 资源字段和内部续接操作。公共数据类型从这些定义推导。`pnpm exec tsx packages/computer-use/cua/native/generate-protocol.ts` 生成原生 JSON Schema 和 Swift 字段、枚举校验表；包测试检查产物是否同步，并验证模型不能提交 Host 专属字段。原生目标验证、资源上限和操作行为仍在执行处检查。 @@ -191,6 +193,8 @@ Use selectText only on controls advertising selectionWritable. It selects a uniq ## 已知限制与后续工作 +队列只排序单次原生请求。持有 app 对象不会为 Session 保留应用，跨多个请求的应用级占用需要单独设计。一直不返回的系统截图 RPC 会占用唯一捕获槽位，重新创建 app 对象不会重置该系统连接。 + - 只提供 macOS 原生实现。不支持浏览器标签、启动应用、AX 差分、无法确认应用归属的系统面板，以及仅后台辅助类型的应用。 diff --git a/packages/computer-use/cua/native/README.txt b/packages/computer-use/cua/native/README.txt index e07792bcff..9b6a26aca8 100644 --- a/packages/computer-use/cua/native/README.txt +++ b/packages/computer-use/cua/native/README.txt @@ -2,13 +2,15 @@ This macOS 14+ JSON-lines helper uses Swift 6, AppKit, Accessibility, CoreGraphics and ScreenCaptureKit. It adapts selected MIT-licensed Cua Driver behavior without loading an external driver. `upstream.json` records the fixed upstream version, adapted Swift files and local modules; `LICENSE.cua-driver` preserves the MIT notice. -Build with `swift build --package-path packages/computer-use/cua/native --configuration release`. The binary is `.build/release/dsh-cua-native` beneath this directory. The DSH CUA plugin owns its process lifetime. EOF drains an accepted final request, releases retained AX references and exits. The `.build/` directory is not committed. +Build with `swift build --package-path packages/computer-use/cua/native --configuration release`. The binary is `.build/release/dsh-cua-native` beneath this directory. The Host runs `--ensure-shared` as a managed startup command; the serving process then belongs to its connected clients and exits after the last connection finishes cleanup. The `.build/` directory is not committed. -Each stdin line is `{ "id": "opaque request id", "method": "method name", "params": {} }`. Each stdout line contains the same ID and exactly one of `result` or `error: {code, message}`. Malformed envelopes return `INVALID_REQUEST` with a null ID. Requests execute serially. Diagnostics must not be written to stdout. `protocol.schema.json` defines request fields. +The shared service uses an owner-only Unix socket under `/tmp/dsh-cua-`, keyed by the executable SHA-256. A version lock elects one process; an additional real-executable-path lock excludes a different build at the same path. The first frame is protocol-1 `hello` with the runtime key and this connection’s byte, AX, operation, screenshot, event-spacing and cleanup limits. `ready` reports the verified runtime key, PID and instance ID. Every connection owns an independent AXWorker. The stdin/stdout entry is a diagnostic proxy to this service. -Screenshot phase records are written to stderr with the `[cua-screenshot]` prefix. Each JSON record contains requestId, stage, elapsedMs since native request receipt, and an optional errorCode. A begin record without its matching end identifies an unfinished phase; the Host reply record distinguishes completed responses whose final stderr record arrived late. Window titles, AX text and image bytes are excluded. The Host retains a bounded tail and attaches available matching records to screenshot failures. +Operation frames remain `{ "id": "opaque request id", "method": "method name", "params": {} }`. The matching reply contains exactly one of `result` or `error: {code, message}`; malformed envelopes return `INVALID_REQUEST` with a null ID. A connection sends at most one operation at a time. `{ "type": "cancel", "id": "request id" }` cancels that request; a late cancel produces no extra reply. `protocol.schema.json` defines operation fields. -AppKit runs on the main event loop. Blocking AX calls and synchronous input use an actor on a dedicated serial executor; standard-input reading and response encoding/writing use separate queues. The coordinator admits one request through completion and output. The host passes its byte limits, per-call AX deadline and shutdown grace through `--max-request-bytes`, `--max-response-bytes`, `--ax-timeout-ms` and `--shutdown-grace-ms`. Unicode text event spacing is set by `--text-event-interval-ms` (0..1000 ms, default 0.25); it does not change chord timing. The provider configures AX timeout separately as axTimeoutMs (default 3000 ms). The AX timeout bounds each individual system call; the host still owns the complete operation deadline. Cancellation stops further input between events and releases only this operation’s held keys/buttons when possible; forced process termination cannot guarantee release. +Screenshot diagnostics use `type: "diagnostic"` frames on the originating connection. Records contain requestId, stage, elapsedMs since receipt and optional errorCode, without window titles, AX text or image bytes. The stdin/stdout proxy forwards them to stderr with `[cua-screenshot]`; operation replies remain on stdout. The Host retains a bounded diagnostic buffer and includes it in screenshot failures. + +AppKit runs on the main event loop. Blocking AX and input use each connection’s serial actor; all connections share a FIFO for complete desktop operations. Socket writes do not hold that operation slot. A request cancellation waits for native work to stop; it does not cancel the connection’s lifetime latch. Screenshot timeout ends only the request. An unfinished system capture retains the single capture slot until its real callback arrives, rejects new captures with CAPTURE_BUSY and leaves AX/input available. Only an operation or cleanup that cannot stop can invalidate the shared process. The proxy accepts `--max-request-bytes`, `--max-response-bytes`, `--ax-timeout-ms`, `--timeout-ms`, `--screenshot-timeout-ms`, `--shutdown-grace-ms` and `--text-event-interval-ms`; the Host sets the same limits in hello. An AX deadline bounds an individual system call. Cancellation can follow partial input, and uncertain input is never replayed automatically. Successful `activate` and `performSecondaryAction` AXRaise results include `foreground` alongside `{dispatched: true, verified: false}`. It records `requestId`, `operation`, `targetPid`, nullable `targetWindowId`, `activationRequested`, `raiseRequested`, and immediate `before`/`after` observations. `frontPid` comes from WindowServer; `focusedWindowId` is the target app's AX-focused window; `frontWindowId` and `frontWindowPid` identify the first visible normal window. Unknown values are null. Observations are sequential, not atomic or proof of causality. Tool-owned requests use the logged tool call ID. Failed operations retain the existing error response and may have partial effects. diff --git a/packages/computer-use/cua/native/Sources/CuaNative/Main.swift b/packages/computer-use/cua/native/Sources/CuaNative/Main.swift index b608e0cd32..174da12742 100644 --- a/packages/computer-use/cua/native/Sources/CuaNative/Main.swift +++ b/packages/computer-use/cua/native/Sources/CuaNative/Main.swift @@ -1,4 +1,4 @@ -// Executable entry: AppKit owns the main run loop, while protocol and AX work stay off it. +// Only server mode owns AppKit; launcher and stdio clients connect to its socket. import AppKit import Foundation import Darwin @@ -6,14 +6,30 @@ import Darwin @main enum CuaNativeMain { @MainActor static func main() { do { - let configuration = try HelperConfiguration(arguments: Array(CommandLine.arguments.dropFirst())) - let app = NSApplication.shared - app.setActivationPolicy(.prohibited) - let coordinator = RequestCoordinator(configuration: configuration) - coordinator.start() - withExtendedLifetime(coordinator) { app.run() } + let arguments = Array(CommandLine.arguments.dropFirst()) + if arguments.first == "--ensure-shared" { + try SharedHelperLauncher.ensure(SharedHelperOptions(arguments: arguments)) + return + } + if arguments.first == "--serve" { + guard setsid() >= 0 else { throw SharedSocket.failure("Creating shared helper session") } + let options = try SharedHelperOptions(arguments: arguments) + let app = NSApplication.shared + app.setActivationPolicy(.prohibited) + guard let server = try SharedHelperServer(options: options) else { return } + try server.start() + withExtendedLifetime(server) { app.run() } + return + } + let configuration = try HelperConfiguration(arguments: arguments) + try SharedHelperProxy.run(configuration: configuration) } catch { let diagnostic = (error as? NativeFailure)?.message ?? error.localizedDescription + if CommandLine.arguments.dropFirst().first == "--serve" { + let failure = JSONValue.object(["type": .string("error"), "code": .string((error as? NativeFailure)?.code ?? "NATIVE_STARTUP"), + "message": .string(diagnostic)]) + if let data = try? JSONEncoder().encode(failure) { try? FileHandle.standardOutput.write(contentsOf: data + Data([10])) } + } if let data = "dsh-cua-native: \(diagnostic)\n".data(using: .utf8) { try? FileHandle.standardError.write(contentsOf: data) } exit(1) } diff --git a/packages/computer-use/cua/native/Sources/CuaNative/RequestCoordinator.swift b/packages/computer-use/cua/native/Sources/CuaNative/RequestCoordinator.swift index 95b7acd803..2edeb1d4d9 100644 --- a/packages/computer-use/cua/native/Sources/CuaNative/RequestCoordinator.swift +++ b/packages/computer-use/cua/native/Sources/CuaNative/RequestCoordinator.swift @@ -1,4 +1,4 @@ -// One active request remains reserved through every await and the final stdout write. +// Each connection retains one request through reply delivery; desktop work uses the shared scheduler. import Foundation import AppKit import Dispatch @@ -8,112 +8,112 @@ import Darwin private let configuration: HelperConfiguration private let cancellation: CancellationLatch private let worker: AXWorker - private let capture = WindowCapture() + private let capture: WindowCapture private let clipboard = ClipboardPaste() private let input: JSONLInput private let writer: JSONLWriter - private let signalQueue = DispatchQueue(label: "dsh.cua.signals") - private var signalSources: [DispatchSourceSignal] = [] + private let channel: SharedSocket + private let scheduler: RequestScheduler + private let onClose: @MainActor @Sendable () -> Void + private let onFatal: @Sendable (NativeFailure, Double) -> Void + private var deadline: NativeDeadline? private var active: UUID? private var operation: Task? private var stopping = false + private var requestControl: NativeRequestControl? + private var activeRequestID: String? + private var replying = false + private var nextFrame: Data? + private var nextCancelled = false - init(configuration: HelperConfiguration) { + init(configuration: HelperConfiguration, capture: WindowCapture, channel: SharedSocket, + cancellation: CancellationLatch, scheduler: RequestScheduler, writer: JSONLWriter, + onClose: @escaping @MainActor @Sendable () -> Void, + onFatal: @escaping @Sendable (NativeFailure, Double) -> Void) { self.configuration = configuration - let cancellation = CancellationLatch() + self.capture = capture self.cancellation = cancellation + self.channel = channel + self.scheduler = scheduler + self.onClose = onClose + self.onFatal = onFatal worker = AXWorker(cancellation: cancellation, axTimeout: configuration.axTimeoutSeconds, textEventInterval: configuration.textEventIntervalSeconds) - input = JSONLInput(limit: configuration.maxRequestBytes, cancellation: cancellation) - writer = JSONLWriter(limit: configuration.maxResponseBytes) + input = JSONLInput(limit: configuration.maxRequestBytes, cancellation: cancellation, channel: channel) + self.writer = writer } func start() { - signal(SIGPIPE, SIG_IGN) let latch = cancellation - let grace = configuration.shutdownGraceSeconds - for number in [SIGTERM, SIGINT] { - signal(number, SIG_IGN) - let source = DispatchSource.makeSignalSource(signal: number, queue: signalQueue) - source.setEventHandler { [weak self] in - latch.cancel() - // This queue remains responsive even if an AppKit system call stalls the main actor. - DispatchQueue.global(qos: .utility).asyncAfter(deadline: .now() + grace) { _exit(128 + number) } - Task { @MainActor in self?.shutdown(status: 128 + number) } - } - source.resume() - signalSources.append(source) - } input.start(onFrame: { [weak self] frame in - Task { @MainActor in self?.receive(frame) } + DispatchQueue.main.async { self?.receive(frame) } }, onEOF: { [weak self] in - Task { @MainActor in self?.shutdown(status: 0) } + latch.cancel() + DispatchQueue.main.async { self?.shutdown() } }, onFailure: { [weak self] failure in - Task { @MainActor in self?.transportFailed(failure) } + latch.cancel() + DispatchQueue.main.async { self?.transportFailed(failure) } }) } private func receive(_ data: Data) { guard !stopping else { return } + if let control = try? JSONDecoder().decode(JSONValue.self, from: data).object, + Set(control.keys) == ["type", "id"], control["type"] == .string("cancel"), let id = control["id"]?.string { + if id == activeRequestID { cancelRequest(NativeFailure(code: "NATIVE_CANCELLED", message: "Native operation cancelled")) } + else if let nextFrame, (try? NativeRequest(data: nextFrame).id) == id { nextCancelled = true } + return + } + if active != nil, replying, nextFrame == nil { nextFrame = data; return } guard active == nil else { transportFailed(NativeFailure(code: "NATIVE_PROTOCOL", message: "Concurrent helper request admission")); return } let token = UUID() active = token + let parsed = try? NativeRequest(data: data) + activeRequestID = parsed?.id + let control = NativeRequestControl() + requestControl = control + let trace: ScreenshotTrace? + if let parsed, parsed.method == "screenshot" { + let writer = self.writer + trace = ScreenshotTrace(requestID: parsed.id, report: { writer.diagnostic($0) }) + trace?.mark("helper.received") + } else { trace = nil } + let seconds = parsed?.method == "screenshot" + ? min(configuration.timeoutSeconds, configuration.screenshotTimeoutSeconds) : configuration.timeoutSeconds + let grace = configuration.shutdownGraceSeconds + let onFatal = self.onFatal + let channel = self.channel + deadline = NativeDeadline(seconds: seconds) { + trace?.mark("helper.deadline", errorCode: "NATIVE_TIMEOUT") + control.cancel(NativeFailure(code: "NATIVE_TIMEOUT", message: "Native operation exceeded its deadline"), grace: grace, + unresponsive: { + onFatal(NativeFailure(code: "NATIVE_UNRESPONSIVE", message: "Native operation did not stop after cancellation"), grace) + }, blockedReply: { channel.shutdown() }) + } operation = Task { [weak self] in guard let self else { return } + defer { control.finish() } var requestID: JSONValue = .null - var trace: ScreenshotTrace? let response: JSONValue do { let request = try NativeRequest(data: data) requestID = .string(request.id) - if request.method == "screenshot" { - trace = ScreenshotTrace(requestID: request.id) - trace?.mark("helper.received") - } - let params = try RequestParameters(request) - let requiresCatalog = ["getApp", "listApps", "findApps", "typeText", "pressKey"].contains(request.method) - let catalog = requiresCatalog ? ApplicationCatalog.snapshot() : ApplicationCatalogSnapshot(applications: [], frontPID: nil) - let result: JSONValue - if request.method == "findApps" { - result = .array(try ApplicationDiscovery.applications(matching: params.string("query"), exact: false, - limit: params.integer("maxAppMatches", in: 1...1000), cancellation: self.cancellation).map { $0.json(catalog: catalog) }) - } else { - var installed: InstalledApplication? - if request.method == "getApp" { - let selector = try params.string("selector") - if !catalog.applications.contains(where: { $0.matches(selector) }) && Int32(selector) == nil { - installed = try ApplicationDiscovery.applications(matching: selector, exact: true, limit: 1, - cancellation: self.cancellation).first - } - } - trace?.mark("worker.begin") - let outcome = try await self.worker.execute(request.method, params: params, catalog: catalog, installed: installed, trace: trace) - trace?.mark("worker.end") - switch outcome { - case .result(let value): result = value - case .capture(let target): - let image = try await self.capture.capture(target, cancellation: self.cancellation, trace: trace) - result = try await self.worker.finishCapture(target, image: image, trace: trace) - case .foreground(let plan): result = try await self.performForeground(plan, requestID: request.id) - case .paste(let plan): - try await self.clipboard.perform(text: plan.text, format: plan.format, maxBytes: plan.maxBytes, - checkCancellation: { try self.cancellation.check() }) { requireClipboard in - try await self.worker.performPaste(plan, beforeDispatch: requireClipboard) - } - result = AXWorker.actionResult - case .launch(let plan): - let pid = try await ApplicationDiscovery.launch(plan.application, cancellation: self.cancellation) - result = try await self.worker.finishLaunch(plan, pid: pid) - } + let result = try await self.scheduler.perform(cancellation: self.cancellation) { + try await self.execute(request, trace: trace) } + if let failure = control.failure { throw failure } response = .object(["id": requestID, "result": result]) - } catch let failure as NativeFailure { + } catch let error as NativeFailure { + let failure = control.failure ?? error trace?.mark("helper.failed", errorCode: failure.code) response = .object(["id": requestID, "error": .object(["code": .string(failure.code), "message": .string(failure.message)])]) } catch { - trace?.mark("helper.failed", errorCode: "NATIVE_ERROR") + let failure = control.failure ?? NativeFailure(code: requestID == .null ? "INVALID_REQUEST" : "NATIVE_ERROR", message: error.localizedDescription) + trace?.mark("helper.failed", errorCode: failure.code) response = .object(["id": requestID, "error": .object([ - "code": .string(requestID == .null ? "INVALID_REQUEST" : "NATIVE_ERROR"), "message": .string(error.localizedDescription)])]) + "code": .string(failure.code), "message": .string(failure.message)])]) } + control.finishOperation() guard !self.stopping, self.active == token else { return } + self.replying = true do { try await self.writer.write(response, trace: trace) } catch let failure as NativeFailure { if failure.code == "NATIVE_RESPONSE_LIMIT" { @@ -128,10 +128,71 @@ import Darwin } catch { self.transportFailed(NativeFailure(code: "NATIVE_IO", message: error.localizedDescription)); return } guard !self.stopping, self.active == token else { return } + self.deadline?.cancel() + self.deadline = nil self.active = nil + self.activeRequestID = nil + self.requestControl = nil self.operation = nil - self.input.next() + self.replying = false + if let next = self.nextFrame { + self.nextFrame = nil + let cancelled = self.nextCancelled + self.nextCancelled = false + self.receive(next) + if cancelled { self.cancelRequest(NativeFailure(code: "NATIVE_CANCELLED", message: "Native operation cancelled")) } + } } + if let operation { control.attach(operation) } + } + + private func cancelRequest(_ failure: NativeFailure) { + let grace = configuration.shutdownGraceSeconds + let onFatal = self.onFatal + let channel = self.channel + requestControl?.cancel(failure, grace: grace, unresponsive: { + onFatal(NativeFailure(code: "NATIVE_UNRESPONSIVE", message: "Native operation did not stop after cancellation"), grace) + }, blockedReply: { channel.shutdown() }) + } + private func execute(_ request: NativeRequest, trace: ScreenshotTrace?) async throws -> JSONValue { + let params = try RequestParameters(request) + if request.method == "screenshot" { try capture.requireAvailable() } + let requiresCatalog = ["getApp", "listApps", "findApps", "typeText", "pressKey"].contains(request.method) + let catalog = requiresCatalog ? ApplicationCatalog.snapshot() : ApplicationCatalogSnapshot(applications: [], frontPID: nil) + let result: JSONValue + if request.method == "findApps" { + result = .array(try ApplicationDiscovery.applications(matching: params.string("query"), exact: false, + limit: params.integer("maxAppMatches", in: 1...1000), cancellation: cancellation).map { $0.json(catalog: catalog) }) + } else { + var installed: InstalledApplication? + if request.method == "getApp" { + let selector = try params.string("selector") + if !catalog.applications.contains(where: { $0.matches(selector) }) && Int32(selector) == nil { + installed = try ApplicationDiscovery.applications(matching: selector, exact: true, limit: 1, + cancellation: cancellation).first + } + } + trace?.mark("worker.begin") + let outcome = try await worker.execute(request.method, params: params, catalog: catalog, installed: installed, trace: trace) + trace?.mark("worker.end") + switch outcome { + case .result(let value): result = value + case .capture(let target): + let image = try await capture.capture(target, cancellation: cancellation, trace: trace) + result = try await worker.finishCapture(target, image: image, trace: trace) + case .foreground(let plan): result = try await performForeground(plan, requestID: request.id) + case .paste(let plan): + try await clipboard.perform(text: plan.text, format: plan.format, maxBytes: plan.maxBytes, + checkCancellation: { try cancellation.check() }) { requireClipboard in + try await worker.performPaste(plan, beforeDispatch: requireClipboard) + } + result = AXWorker.actionResult + case .launch(let plan): + let pid = try await ApplicationDiscovery.launch(plan.application, cancellation: cancellation) + result = try await worker.finishLaunch(plan, pid: pid) + } + } + return result } private func performForeground(_ plan: ForegroundPlan, requestID: String) async throws -> JSONValue { let before = try await worker.foregroundObservation(plan) @@ -154,23 +215,30 @@ import Darwin } private func transportFailed(_ failure: NativeFailure) { if let bytes = "\(failure.code): \(failure.message)\n".data(using: .utf8) { try? FileHandle.standardError.write(contentsOf: bytes) } - shutdown(status: 1) + shutdown() } - private func shutdown(status: Int32) { + func close() { shutdown() } + + private func shutdown() { guard !stopping else { return } stopping = true cancellation.cancel() + deadline?.cancel() + deadline = nil + cancelRequest(NativeFailure(code: "NATIVE_CANCELLED", message: "Native client disconnected")) + input.stop() do { try clipboard.cancel() } catch { fputs("Clipboard restoration failed during shutdown: \(error.localizedDescription)\n", stderr) } active = nil - operation?.cancel() - input.stop() - let grace = configuration.shutdownGraceSeconds - signalQueue.asyncAfter(deadline: .now() + grace) { _exit(status == 0 ? 1 : status) } - Task { [worker] in + let pending = operation + operation = nil + pending?.cancel() + Task { [self] in + await pending?.value + deadline?.cancel() + deadline = nil await worker.shutdown() - // Worker shutdown runs after every earlier synchronous AX/input segment. - exit(status) + onClose() } } } diff --git a/packages/computer-use/cua/native/Sources/CuaNative/RequestScheduler.swift b/packages/computer-use/cua/native/Sources/CuaNative/RequestScheduler.swift new file mode 100644 index 0000000000..a434913e06 --- /dev/null +++ b/packages/computer-use/cua/native/Sources/CuaNative/RequestScheduler.swift @@ -0,0 +1,101 @@ +// Desktop operations hold one process-wide slot; socket writes do not hold it. +import Foundation +import Dispatch + +@MainActor final class RequestScheduler { + private struct Waiting { + let id: UUID + let continuation: CheckedContinuation + } + private var owner: UUID? + private var waiting: [Waiting] = [] + + func perform(cancellation: CancellationLatch, + operation: @MainActor () async throws -> JSONValue) async throws -> JSONValue { + let id = UUID() + try await withTaskCancellationHandler { + try await withCheckedThrowingContinuation { (continuation: CheckedContinuation) in + if cancellation.isCancelled || Task.isCancelled { + continuation.resume(throwing: NativeFailure(code: "NATIVE_CANCELLED", message: "Queued operation cancelled")) + } else if owner == nil { + owner = id + continuation.resume() + } else { + waiting.append(Waiting(id: id, continuation: continuation)) + } + } + } onCancel: { + Task { @MainActor [weak self] in self?.cancel(id) } + } + defer { release(id) } + try cancellation.check() + return try await operation() + } + + private func cancel(_ id: UUID) { + guard let index = waiting.firstIndex(where: { $0.id == id }) else { return } + waiting.remove(at: index).continuation.resume(throwing: NativeFailure(code: "NATIVE_CANCELLED", message: "Queued operation cancelled")) + } + + private func release(_ id: UUID) { + precondition(owner == id) + if waiting.isEmpty { owner = nil } + else { + let next = waiting.removeFirst() + owner = next.id + next.continuation.resume() + } + } +} + +final class NativeDeadline: @unchecked Sendable { + private let lock = NSLock() + private var pending = true + + init(seconds: Double, expired: @escaping @Sendable () -> Void) { + DispatchQueue.global(qos: .utility).asyncAfter(deadline: .now() + seconds) { [self] in + if take() { expired() } + } + } + func cancel() { _ = take() } + private func take() -> Bool { + lock.withLock { + guard pending else { return false } + pending = false + return true + } + } +} + +final class NativeRequestControl: @unchecked Sendable { + private let lock = NSLock() + private var task: Task? + private var reason: NativeFailure? + private var operationDone = false + private var completed = false + + var failure: NativeFailure? { lock.withLock { reason } } + func attach(_ task: Task) { + let cancel = lock.withLock { self.task = task; return reason != nil } + if cancel { task.cancel() } + } + func finishOperation() { lock.withLock { operationDone = true } } + func finish() { lock.withLock { completed = true; task = nil } } + + func cancel(_ failure: NativeFailure, grace: Double, + unresponsive: @escaping @Sendable () -> Void, blockedReply: @escaping @Sendable () -> Void) { + let cancellation: (Bool, Task?) = lock.withLock { + guard !completed, reason == nil else { return (false, nil) } + reason = failure + return (true, task) + } + guard cancellation.0 else { return } + cancellation.1?.cancel() + _ = NativeDeadline(seconds: grace) { [self] in + let state = lock.withLock { (completed, operationDone) } + guard !state.0 else { return } + if state.1 { blockedReply() } + else { unresponsive() } + } + } +} diff --git a/packages/computer-use/cua/native/Sources/CuaNative/Runtime.swift b/packages/computer-use/cua/native/Sources/CuaNative/Runtime.swift index e71629e53f..e2f7fb5bc3 100644 --- a/packages/computer-use/cua/native/Sources/CuaNative/Runtime.swift +++ b/packages/computer-use/cua/native/Sources/CuaNative/Runtime.swift @@ -9,7 +9,7 @@ final class CancellationLatch: @unchecked Sendable { private var cancelled = false func cancel() { lock.withLock { cancelled = true } } func check() throws { - if lock.withLock({ cancelled }) { throw NativeFailure(code: "NATIVE_CANCELLED", message: "Helper operation cancelled") } + if Task.isCancelled || lock.withLock({ cancelled }) { throw NativeFailure(code: "NATIVE_CANCELLED", message: "Helper operation cancelled") } } var isCancelled: Bool { lock.withLock { cancelled } } } @@ -20,6 +20,8 @@ struct HelperConfiguration: Sendable { var axTimeoutSeconds: Float = 30 var shutdownGraceSeconds: Double = 3 var textEventIntervalSeconds: Double = 0.00025 + var timeoutSeconds: Double = 30 + var screenshotTimeoutSeconds: Double = 5 init(arguments: [String]) throws { var index = 0 while index < arguments.count { @@ -41,20 +43,50 @@ struct HelperConfiguration: Sendable { case "--shutdown-grace-ms": guard let value = Double(text), value.isFinite, value > 0 else { throw RequestParameters.invalid("Shutdown grace must be positive") } shutdownGraceSeconds = value / 1000 + case "--timeout-ms", "--screenshot-timeout-ms": + guard let value = Double(text), value.isFinite, value > 0 else { throw RequestParameters.invalid("Operation timeout must be positive") } + if key == "--timeout-ms" { timeoutSeconds = value / 1000 } + else { screenshotTimeoutSeconds = value / 1000 } default: throw RequestParameters.invalid("Unknown helper option \(key)") } index += 2 } } + + init(hello: Data, runtimeKey: String) throws { + let decoded = try JSONDecoder().decode(JSONValue.self, from: hello) + guard let fields = decoded.object, Set(fields.keys) == ["type", "protocol", "runtimeKey", "config"], + fields["type"] == .string("hello"), fields["protocol"] == .number(1), + fields["runtimeKey"] == .string(runtimeKey), let config = fields["config"]?.object, + Set(config.keys) == ["maxRequestBytes", "maxResponseBytes", "axTimeoutMs", "textEventIntervalMs", "screenshotTimeoutMs", "timeoutMs", "graceMs"] else { + throw NativeFailure(code: "NATIVE_PROTOCOL", message: "Expected shared helper protocol 1 hello for this runtime") + } + func number(_ name: String) throws -> String { + guard let number = config[name]?.number, number.isFinite else { throw RequestParameters.invalid("\(name) must be finite") } + if ["maxRequestBytes", "maxResponseBytes"].contains(name) { + guard let integer = Int(exactly: number) else { throw RequestParameters.invalid("\(name) must be an integer") } + return String(integer) + } + return String(number) + } + try self.init(arguments: [ + "--max-request-bytes", number("maxRequestBytes"), "--max-response-bytes", number("maxResponseBytes"), + "--ax-timeout-ms", number("axTimeoutMs"), "--text-event-interval-ms", number("textEventIntervalMs"), + "--screenshot-timeout-ms", number("screenshotTimeoutMs"), "--timeout-ms", number("timeoutMs"), + "--shutdown-grace-ms", number("graceMs"), + ]) + } } final class JSONLInput: Sendable { // Buffer and reads belong exclusively to queue; stop state uses the shared latch. - private let queue = DispatchQueue(label: "dsh.cua.stdin") - private let permit = DispatchSemaphore(value: 0) + private let queue = DispatchQueue(label: "dsh.cua.socket-input") private let cancellation: CancellationLatch private let limit: Int - init(limit: Int, cancellation: CancellationLatch) { self.limit = limit; self.cancellation = cancellation } + private let channel: SharedSocket + init(limit: Int, cancellation: CancellationLatch, channel: SharedSocket) { + self.limit = limit; self.cancellation = cancellation; self.channel = channel + } func start(onFrame: @escaping @Sendable (Data) -> Void, onEOF: @escaping @Sendable () -> Void, onFailure: @escaping @Sendable (NativeFailure) -> Void) { queue.async { @@ -66,52 +98,70 @@ final class JSONLInput: Sendable { buffer.removeSubrange(...newline) guard frame.count + 1 <= self.limit else { throw NativeFailure(code: "NATIVE_REQUEST_LIMIT", message: "Request exceeded helper byte limit") } onFrame(frame) - self.permit.wait() continue } guard buffer.count <= self.limit else { throw NativeFailure(code: "NATIVE_REQUEST_LIMIT", message: "Request exceeded helper byte limit") } - // Foundation's counted read can wait for EOF/full capacity on a pipe. - // POSIX read returns the available request bytes while stdin stays open. + // POSIX read returns available socket bytes without waiting for full capacity. var chunk = [UInt8](repeating: 0, count: min(65536, self.limit + 1 - buffer.count)) let count = chunk.withUnsafeMutableBytes { bytes in - Darwin.read(STDIN_FILENO, bytes.baseAddress, bytes.count) + Darwin.read(self.channel.descriptor, bytes.baseAddress, bytes.count) } if count < 0 && errno == EINTR { continue } - guard count >= 0 else { throw NativeFailure(code: "NATIVE_IO", message: "Reading helper stdin failed (errno \(errno))") } + guard count >= 0 else { throw NativeFailure(code: "NATIVE_IO", message: "Reading helper socket failed (errno \(errno))") } if count == 0 { - if !buffer.isEmpty { onFrame(buffer); self.permit.wait() } + if !buffer.isEmpty { onFrame(buffer) } onEOF() return } buffer.append(contentsOf: chunk.prefix(count)) } } catch let error as NativeFailure { onFailure(error) } - catch { onFailure(NativeFailure(code: "NATIVE_IO", message: "Reading helper stdin failed: \(error.localizedDescription)")) } + catch { onFailure(NativeFailure(code: "NATIVE_IO", message: "Reading helper socket failed: \(error.localizedDescription)")) } } } - func next() { permit.signal() } - func stop() { permit.signal(); try? FileHandle.standardInput.close() } + func stop() { channel.stopReading() } } final class JSONLWriter: Sendable { - private let queue = DispatchQueue(label: "dsh.cua.stdout") + private let queue = DispatchQueue(label: "dsh.cua.socket-output") let limit: Int - init(limit: Int) { self.limit = limit } + private let channel: SharedSocket + init(limit: Int, channel: SharedSocket) { self.limit = limit; self.channel = channel } + + func diagnostic(_ value: JSONValue) { + queue.async { [self] in + // The operation reply owns transport failures; diagnostics remain best effort. + try? send(value) + } + } + + private func send(_ value: JSONValue) throws { + let encoder = JSONEncoder() + encoder.outputFormatting = [.sortedKeys, .withoutEscapingSlashes] + var frame = try encoder.encode(value) + frame.append(10) + guard frame.count <= limit else { throw NativeFailure(code: "NATIVE_RESPONSE_LIMIT", message: "Reply exceeded helper byte limit") } + try channel.write(frame) + } func write(_ response: JSONValue, trace: ScreenshotTrace? = nil) async throws { let limit = self.limit try await withCheckedThrowingContinuation { (continuation: CheckedContinuation) in queue.async { do { - trace?.mark("response.encode.begin") + func mark(_ stage: String) { + guard let trace else { return } + try? self.send(trace.diagnosticValue(stage)) + } + mark("response.encode.begin") let encoder = JSONEncoder() encoder.outputFormatting = [.sortedKeys, .withoutEscapingSlashes] var frame = try encoder.encode(response) frame.append(10) guard frame.count <= limit else { throw NativeFailure(code: "NATIVE_RESPONSE_LIMIT", message: "Reply exceeded helper byte limit") } - trace?.mark("response.encode.end") - trace?.mark("response.write.begin") - try FileHandle.standardOutput.write(contentsOf: frame) - trace?.mark("response.write.end") + mark("response.encode.end") + mark("response.write.begin") + try self.channel.write(frame) + mark("response.write.end") continuation.resume() } catch let error as NativeFailure { continuation.resume(throwing: error) } catch { continuation.resume(throwing: NativeFailure(code: "NATIVE_IO", message: "Encoding or writing helper reply failed")) } diff --git a/packages/computer-use/cua/native/Sources/CuaNative/ScreenshotTrace.swift b/packages/computer-use/cua/native/Sources/CuaNative/ScreenshotTrace.swift index 34f37fa028..a65c108be9 100644 --- a/packages/computer-use/cua/native/Sources/CuaNative/ScreenshotTrace.swift +++ b/packages/computer-use/cua/native/Sources/CuaNative/ScreenshotTrace.swift @@ -1,12 +1,13 @@ -// Screenshot phase timing on stderr; request data and captured content stay off this channel. +// Screenshot phase records omit request contents, window titles and image data. import Foundation import Dispatch struct ScreenshotTrace: Sendable { let requestID: String private let started = DispatchTime.now().uptimeNanoseconds + private let report: (@Sendable (JSONValue) -> Void)? - init(requestID: String) { self.requestID = requestID } + init(requestID: String, report: (@Sendable (JSONValue) -> Void)? = nil) { self.requestID = requestID; self.report = report } func encoded(_ stage: String, errorCode: String? = nil) throws -> Data { struct Entry: Encodable { @@ -23,7 +24,20 @@ struct ScreenshotTrace: Sendable { return line } + func diagnosticValue(_ stage: String, errorCode: String? = nil) -> JSONValue { + var fields: [String: JSONValue] = [ + "type": .string("diagnostic"), "requestId": .string(requestID), "stage": .string(stage), + "elapsedMs": .number(Double(DispatchTime.now().uptimeNanoseconds - started) / 1_000_000), + ] + if let errorCode { fields["errorCode"] = .string(errorCode) } + return .object(fields) + } + func mark(_ stage: String, errorCode: String? = nil) { + if let report { + report(diagnosticValue(stage, errorCode: errorCode)) + return + } // A diagnostic failure must not replace the capture result or its original error. if let line = try? encoded(stage, errorCode: errorCode) { try? FileHandle.standardError.write(contentsOf: line) diff --git a/packages/computer-use/cua/native/Sources/CuaNative/SharedHelperProxy.swift b/packages/computer-use/cua/native/Sources/CuaNative/SharedHelperProxy.swift new file mode 100644 index 0000000000..1c094a6373 --- /dev/null +++ b/packages/computer-use/cua/native/Sources/CuaNative/SharedHelperProxy.swift @@ -0,0 +1,124 @@ +// Stdio diagnostic clients use the same daemon and never create a second capture backend. +import Foundation +import CryptoKit +import Darwin + +enum SharedHelperProxy { + static func run(configuration: HelperConfiguration) throws { + guard let executable = NativePlatform.executablePath(getpid()) else { throw SharedSocket.failure("Resolving helper executable") } + let digest = SHA256.hash(data: try Data(contentsOf: URL(fileURLWithPath: executable), options: .mappedIfSafe)) + .map { String(format: "%02x", $0) }.joined() + let options = try SharedHelperOptions(arguments: ["--ensure-shared", "--runtime-key", digest, + "--runtime-directory", "/tmp/dsh-cua-\(getuid())", "--startup-timeout-ms", String(configuration.timeoutSeconds * 1000)]) + let channel = try connect(options) + defer { channel.shutdown() } + let hello = JSONValue.object(["type": .string("hello"), "protocol": .number(1), "runtimeKey": .string(digest), "config": .object([ + "maxRequestBytes": .number(Double(configuration.maxRequestBytes)), "maxResponseBytes": .number(Double(configuration.maxResponseBytes)), + "axTimeoutMs": .number(Double(configuration.axTimeoutSeconds) * 1000), "textEventIntervalMs": .number(configuration.textEventIntervalSeconds * 1000), + "screenshotTimeoutMs": .number(configuration.screenshotTimeoutSeconds * 1000), "timeoutMs": .number(configuration.timeoutSeconds * 1000), + "graceMs": .number(configuration.shutdownGraceSeconds * 1000), + ])]) + try channel.write(JSONEncoder().encode(hello) + Data([10])) + let frames = Frames(channel: channel, limit: configuration.maxResponseBytes) + guard let handshake = try frames.next(timeout: configuration.timeoutSeconds), + try JSONDecoder().decode(JSONValue.self, from: handshake)["type"] == .string("ready") else { + throw NativeFailure(code: "NATIVE_PROTOCOL", message: "Shared helper rejected stdio client hello") + } + var buffer = Data() + while true { + if let newline = buffer.firstIndex(of: 10) { + let frame = Data(buffer[..= 0 else { throw SharedSocket.failure("Reading helper stdin") } + if count == 0 { break } + buffer.append(contentsOf: bytes.prefix(count)) + } + if !buffer.isEmpty { try exchange(buffer, channel: channel, frames: frames, configuration: configuration) } + Darwin.shutdown(channel.descriptor, SHUT_WR) + while let frame = try frames.next(timeout: configuration.shutdownGraceSeconds) { + guard try diagnostic(frame) else { throw NativeFailure(code: "NATIVE_PROTOCOL", message: "Unexpected response during stdio cleanup") } + } + } + + private static func exchange(_ data: Data, channel: SharedSocket, frames: Frames, configuration: HelperConfiguration) throws { + guard data.count + 1 <= configuration.maxRequestBytes else { throw NativeFailure(code: "NATIVE_REQUEST_LIMIT", message: "Request exceeded helper byte limit") } + try channel.write(data + Data([10])) + while let frame = try frames.next(timeout: configuration.timeoutSeconds) { + if try diagnostic(frame) { continue } + try FileHandle.standardOutput.write(contentsOf: frame + Data([10])) + return + } + throw NativeFailure(code: "NATIVE_EOF", message: "Shared helper closed before its operation reply") + } + + private static func diagnostic(_ data: Data) throws -> Bool { + guard var fields = try JSONDecoder().decode(JSONValue.self, from: data).object, fields["type"] == .string("diagnostic") else { return false } + fields.removeValue(forKey: "type") + let value = try JSONEncoder().encode(JSONValue.object(fields)) + try FileHandle.standardError.write(contentsOf: Data("[cua-screenshot] ".utf8) + value + Data([10])) + return true + } + + private static func connect(_ options: SharedHelperOptions) throws -> SharedSocket { + let started = DispatchTime.now().uptimeNanoseconds + repeat { + try SharedHelperLauncher.ensure(options, publishReady: false) + let descriptor = socket(AF_UNIX, SOCK_STREAM, 0) + guard descriptor >= 0 else { throw SharedSocket.failure("Creating stdio client socket") } + let channel = SharedSocket(descriptor: descriptor) + var address = sockaddr_un() + address.sun_family = sa_family_t(AF_UNIX) + address.sun_len = UInt8(MemoryLayout.size) + withUnsafeMutableBytes(of: &address.sun_path) { $0.copyBytes(from: Array(options.socketPath.utf8) + [0]) } + let connected = withUnsafePointer(to: &address) { pointer in + pointer.withMemoryRebound(to: sockaddr.self, capacity: 1) { Darwin.connect(descriptor, $0, socklen_t(MemoryLayout.size)) } + } + if connected == 0 { return channel } + let error = errno + guard error == ENOENT || error == ECONNREFUSED else { throw SharedSocket.failure("Connecting stdio client") } + usleep(10_000) + } while Double(DispatchTime.now().uptimeNanoseconds - started) / 1_000_000_000 < options.startupTimeout + throw NativeFailure(code: "NATIVE_STARTUP_TIMEOUT", message: "Shared helper did not accept the stdio client") + } + + private final class Frames { + private let channel: SharedSocket + private let limit: Int + private var buffer = Data() + init(channel: SharedSocket, limit: Int) { self.channel = channel; self.limit = limit } + func next(timeout: Double) throws -> Data? { + let started = DispatchTime.now().uptimeNanoseconds + while true { + if let newline = buffer.firstIndex(of: 10) { + let frame = Data(buffer[.. 0 else { throw NativeFailure(code: "NATIVE_TIMEOUT", message: "Shared helper response timed out") } + var item = pollfd(fd: channel.descriptor, events: Int16(POLLIN), revents: 0) + let ready = poll(&item, 1, Int32(min(remaining * 1000, Double(Int32.max)).rounded(.up))) + if ready < 0 && errno == EINTR { continue } + guard ready > 0 else { throw NativeFailure(code: "NATIVE_TIMEOUT", message: "Shared helper response timed out") } + guard buffer.count <= limit else { throw NativeFailure(code: "NATIVE_RESPONSE_LIMIT", message: "Shared helper reply exceeded its byte limit") } + var bytes = [UInt8](repeating: 0, count: min(65536, limit + 1 - buffer.count)) + let count = bytes.withUnsafeMutableBytes { Darwin.read(channel.descriptor, $0.baseAddress, $0.count) } + if count < 0 && errno == EINTR { continue } + guard count >= 0 else { throw SharedSocket.failure("Reading shared helper reply") } + if count == 0 { + guard buffer.isEmpty else { throw NativeFailure(code: "NATIVE_PROTOCOL", message: "Shared helper ended inside a reply") } + return nil + } + buffer.append(contentsOf: bytes.prefix(count)) + } + } + } +} diff --git a/packages/computer-use/cua/native/Sources/CuaNative/SharedHelperRuntime.swift b/packages/computer-use/cua/native/Sources/CuaNative/SharedHelperRuntime.swift new file mode 100644 index 0000000000..f90f1d3c41 --- /dev/null +++ b/packages/computer-use/cua/native/Sources/CuaNative/SharedHelperRuntime.swift @@ -0,0 +1,242 @@ +// Version and executable-path locks outlive every client and release only at process exit. +import Foundation +import Darwin +import CryptoKit + +struct SharedHelperOptions: Sendable { + let runtimeKey: String + let directory: String + let startupTimeout: Double + var socketPath: String { "\(directory)/\(runtimeKey).sock" } + var readyPath: String { "\(directory)/\(runtimeKey).ready" } + var versionLockPath: String { "\(directory)/\(runtimeKey).lock" } + + init(arguments: [String]) throws { + guard let mode = arguments.first, ["--ensure-shared", "--serve"].contains(mode) else { + throw RequestParameters.invalid("Expected shared helper launch mode") + } + var fields: [String: String] = [:] + var index = 1 + while index < arguments.count { + let key = arguments[index] + guard index + 1 < arguments.count, ["--runtime-key", "--runtime-directory", "--startup-timeout-ms"].contains(key), fields[key] == nil else { + throw RequestParameters.invalid("Invalid shared helper option \(key)") + } + fields[key] = arguments[index + 1] + index += 2 + } + guard let key = fields["--runtime-key"], key.utf8.count == 64, + key.utf8.allSatisfy({ (48...57).contains($0) || (97...102).contains($0) }), + let directory = fields["--runtime-directory"], directory.hasPrefix("/"), + let text = fields["--startup-timeout-ms"], let timeout = Double(text), timeout.isFinite, + timeout > 0, timeout <= Double(Int32.max) else { + throw RequestParameters.invalid("Shared helper requires SHA-256 runtime key, absolute runtime directory and positive startup timeout") + } + runtimeKey = key + self.directory = directory + startupTimeout = timeout / 1000 + guard socketPath.utf8.count < MemoryLayout.size(ofValue: sockaddr_un().sun_path) else { + throw RequestParameters.invalid("Shared helper Unix socket path is too long") + } + } + + func prepareDirectory() throws { + if mkdir(directory, 0o700) != 0 && errno != EEXIST { throw SharedSocket.failure("Creating runtime directory") } + var info = stat() + guard lstat(directory, &info) == 0, info.st_mode & S_IFMT == S_IFDIR, + info.st_uid == getuid(), info.st_mode & 0o777 == 0o700 else { + throw NativeFailure(code: "NATIVE_RUNTIME_DIRECTORY", message: "Shared helper directory must be a real 0700 directory owned by the current user") + } + } + + func executableLockPath() throws -> String { + guard let executable = NativePlatform.executablePath(getpid()) else { + throw NativeFailure(code: "NATIVE_IO", message: "Cannot identify helper executable") + } + let path = URL(fileURLWithPath: executable).resolvingSymlinksInPath().path + let digest = SHA256.hash(data: Data(path.utf8)).map { String(format: "%02x", $0) }.joined() + return "\(directory)/path-\(digest).lock" + } + + func ready(pid: Int32, instanceID: String) -> JSONValue { + .object(["type": .string("ready"), "protocol": .number(1), "runtimeKey": .string(runtimeKey), + "pid": .number(Double(pid)), "instanceId": .string(instanceID)]) + } + + func existingReady() throws -> Data { + let until = DispatchTime.now().uptimeNanoseconds + UInt64(startupTimeout * 1_000_000_000) + repeat { + if let data = try? Data(contentsOf: URL(fileURLWithPath: readyPath)), + let ready = try? JSONDecoder().decode(JSONValue.self, from: data), + ready["type"] == .string("ready"), ready["runtimeKey"] == .string(runtimeKey), + let number = ready["pid"].number, let pid = Int32(exactly: number), pid > 0, + kill(pid, 0) == 0, SharedSocket.isOwnedSocket(socketPath) { + var line = data + if line.last != 10 { line.append(10) } + return line + } + usleep(10_000) + } while DispatchTime.now().uptimeNanoseconds < until + throw NativeFailure(code: "NATIVE_STARTUP_TIMEOUT", message: "Existing shared helper did not become ready") + } +} + +final class SharedHelperLock { + private let descriptor: Int32 + init?(path: String) throws { + let fd = open(path, O_CREAT | O_RDWR | O_CLOEXEC | O_NOFOLLOW, 0o600) + guard fd >= 0 else { throw SharedSocket.failure("Opening helper lock") } + var info = stat() + guard fstat(fd, &info) == 0, info.st_mode & S_IFMT == S_IFREG, info.st_uid == getuid(), info.st_mode & 0o777 == 0o600 else { + close(fd) + throw NativeFailure(code: "NATIVE_RUNTIME_DIRECTORY", message: "Helper lock must be a regular 0600 file owned by the current user") + } + if flock(fd, LOCK_EX | LOCK_NB) != 0 { + let error = errno + close(fd) + if error == EWOULDBLOCK { return nil } + throw NativeFailure(code: "NATIVE_IO", message: "Locking shared helper failed (errno \(error))") + } + descriptor = fd + } + deinit { close(descriptor) } +} + +final class SharedSocket: @unchecked Sendable { + let descriptor: Int32 + private let lock = NSLock() + private var stopped = false + init(descriptor: Int32) { + self.descriptor = descriptor + var enabled: Int32 = 1 + _ = setsockopt(descriptor, SOL_SOCKET, SO_NOSIGPIPE, &enabled, socklen_t(MemoryLayout.size)) + _ = fcntl(descriptor, F_SETFD, FD_CLOEXEC) + } + deinit { close(descriptor) } + func stopReading() { _ = Darwin.shutdown(descriptor, SHUT_RD) } + func shutdown() { + lock.withLock { + guard !stopped else { return } + stopped = true + _ = Darwin.shutdown(descriptor, SHUT_RDWR) + } + } + func write(_ data: Data) throws { + try data.withUnsafeBytes { bytes in + var offset = 0 + while offset < bytes.count { + let count = Darwin.write(descriptor, bytes.baseAddress!.advanced(by: offset), bytes.count - offset) + if count < 0 && errno == EINTR { continue } + guard count > 0 else { throw Self.failure("Writing helper socket") } + offset += count + } + } + } + func readHello(timeout: Double) throws -> Data { + let started = DispatchTime.now().uptimeNanoseconds + var data = Data() + while data.count < 16_384 { + let remaining = timeout - Double(DispatchTime.now().uptimeNanoseconds - started) / 1_000_000_000 + guard remaining > 0 else { throw NativeFailure(code: "NATIVE_STARTUP_TIMEOUT", message: "Shared helper hello timed out") } + var item = pollfd(fd: descriptor, events: Int16(POLLIN), revents: 0) + let result = poll(&item, 1, Int32(min(remaining * 1000, Double(Int32.max)).rounded(.up))) + if result < 0 && errno == EINTR { continue } + guard result > 0 else { throw NativeFailure(code: "NATIVE_STARTUP_TIMEOUT", message: "Shared helper hello timed out") } + var byte: UInt8 = 0 + let count = Darwin.read(descriptor, &byte, 1) + if count < 0 && errno == EINTR { continue } + guard count == 1 else { throw Self.failure("Reading shared helper hello") } + if byte == 10 { return data } + data.append(byte) + } + throw NativeFailure(code: "NATIVE_REQUEST_LIMIT", message: "Shared helper hello exceeded 16384 bytes") + } + static func isOwnedSocket(_ path: String) -> Bool { + var info = stat() + return lstat(path, &info) == 0 && info.st_mode & S_IFMT == S_IFSOCK && info.st_uid == getuid() && info.st_mode & 0o777 == 0o600 + } + static func failure(_ action: String) -> NativeFailure { + NativeFailure(code: "NATIVE_IO", message: "\(action) failed (errno \(errno))") + } + static func listen(path: String) throws -> SharedSocket { + var previous = stat() + if lstat(path, &previous) == 0 { + guard isOwnedSocket(path) else { throw NativeFailure(code: "NATIVE_RUNTIME_DIRECTORY", message: "Refusing to replace a non-owned Unix socket") } + guard unlink(path) == 0 else { throw failure("Removing stale helper socket") } + } else if errno != ENOENT { throw failure("Inspecting helper socket") } + let descriptor = socket(AF_UNIX, SOCK_STREAM, 0) + guard descriptor >= 0 else { throw failure("Creating helper socket") } + let channel = SharedSocket(descriptor: descriptor) + var address = sockaddr_un() + address.sun_family = sa_family_t(AF_UNIX) + address.sun_len = UInt8(MemoryLayout.size) + let bytes = Array(path.utf8) + [0] + withUnsafeMutableBytes(of: &address.sun_path) { target in target.copyBytes(from: bytes) } + let bound = withUnsafePointer(to: &address) { pointer in + pointer.withMemoryRebound(to: sockaddr.self, capacity: 1) { Darwin.bind(descriptor, $0, socklen_t(MemoryLayout.size)) } + } + guard bound == 0, chmod(path, 0o600) == 0, Darwin.listen(descriptor, 64) == 0 else { throw failure("Listening on helper socket") } + return channel + } +} + +enum SharedHelperLauncher { + static func ensure(_ options: SharedHelperOptions, publishReady: Bool = true) throws { + try options.prepareDirectory() + if let lock = try SharedHelperLock(path: options.versionLockPath) { + // The serving process acquires its own lock after exec. + withExtendedLifetime(lock) {} + } else { + let ready = try options.existingReady() + if publishReady { try FileHandle.standardOutput.write(contentsOf: ready) } + return + } + guard let executable = NativePlatform.executablePath(getpid()) else { throw SharedSocket.failure("Resolving helper executable") } + var pipes: [Int32] = [0, 0] + guard pipe(&pipes) == 0 else { throw SharedSocket.failure("Creating helper readiness pipe") } + let reader = SharedSocket(descriptor: pipes[0]) + var writeDescriptor = pipes[1] + defer { if writeDescriptor >= 0 { close(writeDescriptor) } } + var actions: posix_spawn_file_actions_t? + let initialized = posix_spawn_file_actions_init(&actions) + guard initialized == 0 else { throw NativeFailure(code: "NATIVE_STARTUP", message: "Creating helper spawn actions failed (errno \(initialized))") } + defer { posix_spawn_file_actions_destroy(&actions) } + let actionResults = [ + posix_spawn_file_actions_addopen(&actions, STDIN_FILENO, "/dev/null", O_RDONLY, 0), + posix_spawn_file_actions_addopen(&actions, STDERR_FILENO, "/dev/null", O_WRONLY, 0), + posix_spawn_file_actions_adddup2(&actions, pipes[1], STDOUT_FILENO), + posix_spawn_file_actions_addclose(&actions, pipes[0]), + posix_spawn_file_actions_addclose(&actions, pipes[1]), + ] + if let failed = actionResults.first(where: { $0 != 0 }) { + throw NativeFailure(code: "NATIVE_STARTUP", message: "Configuring helper spawn actions failed (errno \(failed))") + } + let arguments = [executable, "--serve", "--runtime-key", options.runtimeKey, "--runtime-directory", options.directory, + "--startup-timeout-ms", String(options.startupTimeout * 1000)] + let argv = arguments.map { strdup($0) } + [nil] + let environment = ProcessInfo.processInfo.environment.map { strdup("\($0.key)=\($0.value)") } + [nil] + defer { argv.forEach { free($0) }; environment.forEach { free($0) } } + var pid: pid_t = 0 + let status = argv.withUnsafeBufferPointer { args in + environment.withUnsafeBufferPointer { env in + posix_spawn(&pid, executable, &actions, nil, UnsafeMutablePointer(mutating: args.baseAddress!), UnsafeMutablePointer(mutating: env.baseAddress!)) + } + } + close(pipes[1]) + writeDescriptor = -1 + guard status == 0 else { throw NativeFailure(code: "NATIVE_STARTUP", message: "Starting shared helper failed (errno \(status))") } + do { + let line = try reader.readHello(timeout: options.startupTimeout) + let response = try JSONDecoder().decode(JSONValue.self, from: line) + guard response["type"] == .string("ready") else { + throw NativeFailure(code: response["code"].string ?? "NATIVE_STARTUP", message: response["message"].string ?? "Shared helper startup failed") + } + if publishReady { try FileHandle.standardOutput.write(contentsOf: line + Data([10])) } + } catch { + // This PID is the child this launcher created; unrelated owners are never signalled. + kill(pid, SIGKILL) + while waitpid(pid, nil, 0) < 0 && errno == EINTR {} + throw error + } + } +} diff --git a/packages/computer-use/cua/native/Sources/CuaNative/SharedHelperServer.swift b/packages/computer-use/cua/native/Sources/CuaNative/SharedHelperServer.swift new file mode 100644 index 0000000000..d6fca2420b --- /dev/null +++ b/packages/computer-use/cua/native/Sources/CuaNative/SharedHelperServer.swift @@ -0,0 +1,188 @@ +// One daemon owns ScreenCaptureKit; each socket owns its AX identities and cancellation. +import Foundation +import AppKit +import Dispatch +import Darwin + +final class SharedHelperLifetime: @unchecked Sendable { + private let lock = NSLock() + private var clients: [UUID: (SharedSocket, CancellationLatch, Double)] = [:] + private var stopping = false + private var status: Int32 = 0 + private let startupTimeout: Double + + init(startupTimeout: Double) { self.startupTimeout = startupTimeout } + + func add(_ id: UUID, channel: SharedSocket, cancellation: CancellationLatch) -> Bool { + lock.withLock { + guard !stopping else { return false } + clients[id] = (channel, cancellation, startupTimeout) + return true + } + } + func remove(_ id: UUID) { _ = lock.withLock { clients.removeValue(forKey: id) } } + func configure(_ id: UUID, grace: Double) { + lock.withLock { + guard let client = clients[id] else { return } + clients[id] = (client.0, client.1, grace) + } + } + var shutdownGrace: Double { lock.withLock { clients.values.map { $0.2 }.max() ?? startupTimeout } } + func exitStatusIfEmpty() -> Int32? { + lock.withLock { + guard clients.isEmpty else { return nil } + stopping = true + return status + } + } + var isStopping: Bool { lock.withLock { stopping } } + func abort(grace: Double, status: Int32 = 1) { + let owned: [(SharedSocket, CancellationLatch, Double)]? = lock.withLock { + guard !stopping else { return nil } + stopping = true + self.status = status + return Array(clients.values) + } + guard let owned else { return } + for (channel, cancellation, _) in owned { cancellation.cancel(); channel.shutdown() } + DispatchQueue.global(qos: .utility).asyncAfter(deadline: .now() + grace) { _exit(status) } + } +} + +final class NativeSignals { + private let queue = DispatchQueue(label: "dsh.cua.signals") + private var sources: [DispatchSourceSignal] = [] + + init(onSignal: @escaping @Sendable (Int32) -> Void) { + signal(SIGPIPE, SIG_IGN) + for number in [SIGTERM, SIGINT] { + signal(number, SIG_IGN) + let source = DispatchSource.makeSignalSource(signal: number, queue: queue) + // Construction outside MainActor prevents Swift 6 from isolating the dispatch closure. + source.setEventHandler(handler: { @Sendable in onSignal(number) }) + source.resume() + sources.append(source) + } + } + deinit { sources.forEach { $0.cancel() } } +} + +@MainActor final class SharedHelperServer { + private let options: SharedHelperOptions + private let versionLock: SharedHelperLock + private let executableLock: SharedHelperLock + private let listener: SharedSocket + private let lifetime: SharedHelperLifetime + private let scheduler = RequestScheduler() + private let capture = WindowCapture() + private let instanceID = UUID().uuidString + private var connections: [UUID: RequestCoordinator] = [:] + private var startupDeadline: NativeDeadline? + private var signals: NativeSignals? + + init?(options: SharedHelperOptions) throws { + self.options = options + lifetime = SharedHelperLifetime(startupTimeout: options.startupTimeout) + try options.prepareDirectory() + guard let versionLock = try SharedHelperLock(path: options.versionLockPath) else { + try FileHandle.standardOutput.write(contentsOf: options.existingReady()) + return nil + } + self.versionLock = versionLock + guard let executableLock = try SharedHelperLock(path: options.executableLockPath()) else { + throw NativeFailure(code: "NATIVE_RUNTIME_CONFLICT", message: "Another helper version is using this executable path; close its clients before switching versions") + } + self.executableLock = executableLock + if unlink(options.readyPath) != 0 && errno != ENOENT { throw SharedSocket.failure("Removing stale helper readiness record") } + listener = try SharedSocket.listen(path: options.socketPath) + } + + func start() throws { + let lifetime = self.lifetime + signals = NativeSignals { [weak self] number in + lifetime.abort(grace: lifetime.shutdownGrace, status: 128 + number) + Task { @MainActor in self?.closeConnections(); self?.exitIfEmpty() } + } + startupDeadline = NativeDeadline(seconds: options.startupTimeout) { [weak self] in + Task { @MainActor in self?.exitIfEmpty() } + } + let ready = options.ready(pid: getpid(), instanceID: instanceID) + let data = try JSONEncoder().encode(ready) + try data.write(to: URL(fileURLWithPath: options.readyPath), options: .atomic) + guard chmod(options.readyPath, 0o600) == 0 else { throw SharedSocket.failure("Protecting helper readiness record") } + try FileHandle.standardOutput.write(contentsOf: data + Data([10])) + try FileHandle.standardOutput.close() + let listener = self.listener + DispatchQueue.global(qos: .utility).async { [weak self] in + while !lifetime.isStopping { + let descriptor = accept(listener.descriptor, nil, nil) + if descriptor < 0 && errno == EINTR { continue } + guard descriptor >= 0 else { + if !lifetime.isStopping { lifetime.abort(grace: lifetime.shutdownGrace) } + return + } + let channel = SharedSocket(descriptor: descriptor) + var uid: uid_t = 0 + var gid: gid_t = 0 + guard getpeereid(descriptor, &uid, &gid) == 0, uid == getuid() else { channel.shutdown(); continue } + let id = UUID() + let cancellation = CancellationLatch() + guard lifetime.add(id, channel: channel, cancellation: cancellation) else { channel.shutdown(); return } + Task { @MainActor in self?.admit(id, channel: channel, cancellation: cancellation) } + } + } + } + + private func admit(_ id: UUID, channel: SharedSocket, cancellation: CancellationLatch) { + let options = self.options + DispatchQueue.global(qos: .utility).async { [weak self] in + do { + let configuration = try HelperConfiguration(hello: channel.readHello(timeout: options.startupTimeout), runtimeKey: options.runtimeKey) + Task { @MainActor in await self?.connected(id, channel: channel, cancellation: cancellation, configuration: configuration) } + } catch { + let failure = error as? NativeFailure ?? NativeFailure(code: "NATIVE_PROTOCOL", message: "Invalid shared helper hello") + let response = JSONValue.object(["type": .string("error"), "code": .string(failure.code), "message": .string(failure.message)]) + if let data = try? JSONEncoder().encode(response) { try? channel.write(data + Data([10])) } + channel.shutdown() + Task { @MainActor in self?.disconnected(id) } + } + } + } + + private func connected(_ id: UUID, channel: SharedSocket, cancellation: CancellationLatch, configuration: HelperConfiguration) async { + guard !lifetime.isStopping else { channel.shutdown(); disconnected(id); return } + lifetime.configure(id, grace: configuration.shutdownGraceSeconds) + let writer = JSONLWriter(limit: configuration.maxResponseBytes, channel: channel) + do { try await writer.write(options.ready(pid: getpid(), instanceID: instanceID)) } + catch { channel.shutdown(); disconnected(id); return } + let lifetime = self.lifetime + let coordinator = RequestCoordinator(configuration: configuration, capture: capture, channel: channel, cancellation: cancellation, + scheduler: scheduler, writer: writer, onClose: { [weak self] in + channel.shutdown() + self?.disconnected(id) + }, onFatal: { [weak self] _, grace in + lifetime.abort(grace: grace) + Task { @MainActor in self?.closeConnections() } + }) + connections[id] = coordinator + startupDeadline?.cancel() + coordinator.start() + } + + private func disconnected(_ id: UUID) { + connections.removeValue(forKey: id) + lifetime.remove(id) + exitIfEmpty() + } + + private func closeConnections() { for coordinator in connections.values { coordinator.close() } } + + private func exitIfEmpty() { + guard let status = lifetime.exitStatusIfEmpty() else { return } + listener.shutdown() + unlink(options.socketPath) + unlink(options.readyPath) + // The lock descriptors stay open until process exit, including AX/input cleanup. + withExtendedLifetime((versionLock, executableLock)) { exit(status) } + } +} diff --git a/packages/computer-use/cua/native/Sources/CuaNative/WindowCapture.swift b/packages/computer-use/cua/native/Sources/CuaNative/WindowCapture.swift index 1f1d2dfd88..9dd36fdcb9 100644 --- a/packages/computer-use/cua/native/Sources/CuaNative/WindowCapture.swift +++ b/packages/computer-use/cua/native/Sources/CuaNative/WindowCapture.swift @@ -7,65 +7,93 @@ import UniformTypeIdentifiers struct CapturedPNG: Sendable { let bytes: Data; let width: Int; let height: Int } -private actor CaptureCompletion { +private final class CaptureCompletion: @unchecked Sendable { + private let lock = NSLock() private var continuation: CheckedContinuation? private var result: Result? - nonisolated func install(_ continuation: CheckedContinuation) { - Task { await installed(continuation) } + private var cancelled = false + var isCancelled: Bool { lock.withLock { cancelled } } + + func install(_ continuation: CheckedContinuation) { + let result: Result? = lock.withLock { + if let result = self.result { return result } + self.continuation = continuation + return nil + } + if let result { Self.resume(continuation, result) } } - nonisolated func finish(_ result: Result) { - Task { await finished(result) } + func finish(_ result: Result) { + let pending: CheckedContinuation? = lock.withLock { + guard self.result == nil else { return nil } + self.result = result + let pending = continuation + continuation = nil + return pending + } + if let pending { Self.resume(pending, result) } } - private func installed(_ continuation: CheckedContinuation) { - if let result { Self.resume(continuation, result); return } - self.continuation = continuation - } - private func finished(_ result: Result) { - guard self.result == nil else { return } - self.result = result - let continuation = self.continuation - self.continuation = nil - if let continuation { Self.resume(continuation, result) } + func cancel() { + lock.withLock { cancelled = true } + finish(.failure(NativeFailure(code: "NATIVE_CANCELLED", message: "Capture cancelled; late frames are discarded"))) } private static func resume(_ continuation: CheckedContinuation, _ result: Result) { switch result { case .success(let image): continuation.resume(returning: image); case .failure(let error): continuation.resume(throwing: error) } } } -struct WindowCapture: Sendable { +final class WindowCapture: @unchecked Sendable { private let encoder = DispatchQueue(label: "dsh.cua.png") + private let lock = NSLock() + private var active: UUID? + + func requireAvailable() throws { + if lock.withLock({ active != nil }) { throw Self.busy() } + } + func capture(_ target: CaptureTarget, cancellation: CancellationLatch, trace: ScreenshotTrace? = nil) async throws -> CapturedPNG { + let token = UUID() + guard lock.withLock({ + guard active == nil else { return false } + active = token + return true + }) else { throw Self.busy() } let completion = CaptureCompletion() + let finish: @Sendable (Result) -> Void = { [self] result in + lock.withLock { if active == token { active = nil } } + completion.finish(result) + } let encoder = self.encoder return try await withTaskCancellationHandler { try await withCheckedThrowingContinuation { continuation in completion.install(continuation) do { try cancellation.check() } - catch { completion.finish(.failure(NativeFailure(code: "NATIVE_CANCELLED", message: "Capture cancelled"))); return } + catch { finish(.failure(Self.cancelled())); return } + if completion.isCancelled { finish(.failure(Self.cancelled())); return } trace?.mark("shareable-content.begin") SCShareableContent.getExcludingDesktopWindows(false, onScreenWindowsOnly: false) { content, error in trace?.mark("shareable-content.end") - guard let content else { completion.finish(.failure(Self.captureFailure(error))); return } + if completion.isCancelled { finish(.failure(Self.cancelled())); return } + guard let content else { finish(.failure(Self.captureFailure(error))); return } guard let window = content.windows.first(where: { $0.windowID == target.window.windowID && $0.owningApplication?.processID == target.process.pid }) else { - completion.finish(.failure(NativeFailure(code: "WINDOW_UNAVAILABLE", message: "Selected window is not available for capture"))); return + finish(.failure(NativeFailure(code: "WINDOW_UNAVAILABLE", message: "Selected window is not available for capture"))); return } do { try cancellation.check(); try NativePlatform.verify(target.process) } - catch let error as NativeFailure { completion.finish(.failure(error)); return } - catch { completion.finish(.failure(Self.captureFailure(error))); return } + catch let error as NativeFailure { finish(.failure(error)); return } + catch { finish(.failure(Self.captureFailure(error))); return } trace?.mark("capture-config.begin") let filter = SCContentFilter(desktopIndependentWindow: window) let expected = target.window.bounds guard window.frame.origin.x == expected.x, window.frame.origin.y == expected.y, window.frame.width == expected.width, window.frame.height == expected.height, filter.contentRect.width == expected.width, filter.contentRect.height == expected.height else { - completion.finish(.failure(NativeFailure(code: "WINDOW_CHANGED", message: "Capture surface does not match the selected window point geometry"))); return + finish(.failure(NativeFailure(code: "WINDOW_CHANGED", message: "Capture surface does not match the selected window point geometry"))); return } let configuration = SCStreamConfiguration() let scale = Double(filter.pointPixelScale) let width = (filter.contentRect.width * scale).rounded(.up) let height = (filter.contentRect.height * scale).rounded(.up) guard let pixelsWide = Int(exactly: width), let pixelsHigh = Int(exactly: height), pixelsWide > 0, pixelsHigh > 0 else { - completion.finish(.failure(NativeFailure(code: "NATIVE_ERROR", message: "Capture reported invalid pixel dimensions"))); return + finish(.failure(NativeFailure(code: "NATIVE_ERROR", message: "Capture reported invalid pixel dimensions"))); return } configuration.width = pixelsWide configuration.height = pixelsHigh @@ -76,30 +104,40 @@ struct WindowCapture: Sendable { trace?.mark("capture-image.begin") SCScreenshotManager.captureImage(contentFilter: filter, configuration: configuration) { image, error in trace?.mark("capture-image.end") - guard let image else { completion.finish(.failure(Self.captureFailure(error))); return } + if completion.isCancelled { finish(.failure(Self.cancelled())); return } + guard let image else { finish(.failure(Self.captureFailure(error))); return } guard image.width == pixelsWide, image.height == pixelsHigh else { - completion.finish(.failure(NativeFailure(code: "WINDOW_CHANGED", message: "Capture pixel dimensions changed from the selected window geometry"))); return + finish(.failure(NativeFailure(code: "WINDOW_CHANGED", message: "Capture pixel dimensions changed from the selected window geometry"))); return } trace?.mark("png-encode.queued") encoder.async { do { + if completion.isCancelled { finish(.failure(Self.cancelled())); return } try cancellation.check() trace?.mark("png-encode.begin") let png = try autoreleasepool { try Self.encode(image) } trace?.mark("png-encode.end") + if completion.isCancelled { finish(.failure(Self.cancelled())); return } try cancellation.check() - completion.finish(.success(png)) - } catch let error as NativeFailure { completion.finish(.failure(error)) } - catch { completion.finish(.failure(Self.captureFailure(error))) } + finish(.success(png)) + } catch let error as NativeFailure { finish(.failure(error)) } + catch { finish(.failure(Self.captureFailure(error))) } } } } } } onCancel: { trace?.mark("capture.cancelled") - completion.finish(.failure(NativeFailure(code: "NATIVE_CANCELLED", message: "Capture cancelled; late frames are discarded"))) + // The caller returns now, but the slot belongs to the outstanding system callback. + completion.cancel() } } + private static func cancelled() -> NativeFailure { + NativeFailure(code: "NATIVE_CANCELLED", message: "Capture cancelled; late frames are discarded") + } + private static func busy() -> NativeFailure { + NativeFailure(code: "CAPTURE_BUSY", message: "A previous system capture is still pending; other native operations remain available") + } private static func encode(_ image: CGImage) throws -> CapturedPNG { let data = NSMutableData() guard let destination = CGImageDestinationCreateWithData(data as CFMutableData, UTType.png.identifier as CFString, 1, nil) else { diff --git a/packages/computer-use/cua/native/tests/shared_helper_e2e.py b/packages/computer-use/cua/native/tests/shared_helper_e2e.py new file mode 100644 index 0000000000..6a9afdd5e5 --- /dev/null +++ b/packages/computer-use/cua/native/tests/shared_helper_e2e.py @@ -0,0 +1,222 @@ +"""Shared helper process lifetime, scoped handles and request-local screenshot deadlines. + +Uses an owned copy of the selected helper and an owned AppKit fixture. Existing +Accessibility/Screen Recording grants are required unless --without-desktop is set. +""" +import argparse +import concurrent.futures +import fcntl +import hashlib +import json +import os +from pathlib import Path +import select +import shutil +import signal +import socket +import subprocess +import tempfile +import time + +from native_test_support import BINARY, ROOT, line, stop + + +CONFIG = {"maxRequestBytes": 1024 * 1024, "maxResponseBytes": 32 * 1024 * 1024, + "axTimeoutMs": 3000, "textEventIntervalMs": 0.25, "screenshotTimeoutMs": 5000, + "timeoutMs": 30000, "graceMs": 3000} + + +class Connection: + def __init__(self, path, key, **overrides): + self.socket = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) + self.socket.settimeout(35) + self.socket.connect(str(path)) + self.reader = self.socket.makefile("rb") + self.sequence = 0 + self.diagnostics = [] + self.send({"type": "hello", "protocol": 1, "runtimeKey": key, "config": {**CONFIG, **overrides}}) + self.ready = self.read() + assert self.ready["type"] == "ready", self.ready + + def send(self, value): + self.socket.sendall(json.dumps(value, separators=(",", ":")).encode() + b"\n") + + def read(self): + while True: + data = self.reader.readline() + if not data: + raise EOFError("shared helper disconnected") + value = json.loads(data) + if value.get("type") == "diagnostic": + self.diagnostics.append(value) + else: + return value + + def call(self, method, params, error=None): + self.sequence += 1 + identifier = str(self.sequence) + self.send({"id": identifier, "method": method, "params": params}) + reply = self.read() + assert reply["id"] == identifier, reply + if error: + assert reply.get("error", {}).get("code") == error, reply + return reply["error"] + assert "result" in reply, reply + return reply["result"] + + def close(self): + if self.socket.fileno() < 0: + return + try: + self.socket.shutdown(socket.SHUT_WR) + while self.reader.readline(): + pass + finally: + self.reader.close() + self.socket.close() + + +def wait_released(path, pid, timeout=10): + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + try: + with open(path, "r+b") as lock: + fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB) + try: + os.kill(pid, 0) + except ProcessLookupError: + return + except BlockingIOError: + pass + time.sleep(0.02) + raise AssertionError(f"Owned shared helper {pid} or its version lock did not exit") + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--helper", type=Path, default=BINARY) + parser.add_argument("--without-desktop", action="store_true") + args = parser.parse_args() + with tempfile.TemporaryDirectory(prefix="cua-", dir="/tmp") as temporary: + root = Path(temporary) + helper = root / "helper" + shutil.copy2(args.helper.resolve(strict=True), helper) + key = hashlib.sha256(helper.read_bytes()).hexdigest() + runtime = root / "r" + socket_path = runtime / (key + ".sock") + lock_path = runtime / (key + ".lock") + base = [str(helper), "--runtime-key", key, "--runtime-directory", str(runtime), "--startup-timeout-ms", "10000"] + + def launch(): + result = subprocess.run([base[0], "--ensure-shared", *base[1:]], check=True, + capture_output=True, text=True, timeout=15, start_new_session=True) + return json.loads(result.stdout) + + clients = [] + fixture = None + owner_pid = None + try: + with concurrent.futures.ThreadPoolExecutor(max_workers=2) as workers: + ready = list(workers.map(lambda _: launch(), range(2))) + assert ready[0] == ready[1], ready + owner_pid = ready[0]["pid"] + a = Connection(socket_path, key) + b = Connection(socket_path, key) + clients.extend([a, b]) + assert a.ready == b.ready == ready[0] + for client in clients: + client.call("protocol-unsupported", {}, error="UNSUPPORTED") + # Coalesced frames retain socket order, including cancellation before execution starts. + a.socket.sendall(b'{"id":"coalesced","method":"listApps","params":{}}\n' + b'{"type":"cancel","id":"coalesced"}\n') + coalesced = a.read() + assert coalesced["id"] == "coalesced", coalesced + assert "result" in coalesced or coalesced.get("error", {}).get("code") == "NATIVE_CANCELLED", coalesced + a.call("protocol-after-cancel", {}, error="UNSUPPORTED") + if not args.without_desktop: + executable = root / "fixture" + state = root / "state.json" + subprocess.run(["xcrun", "swiftc", str(ROOT / "tests/Fixture.swift"), "-o", str(executable)], check=True) + fixture = subprocess.Popen([str(executable), str(state)], stdout=subprocess.PIPE, text=True) + assert line(fixture).strip() == "READY" + capabilities = a.call("getCapabilities", {}) + assert capabilities["permissions"] == {"accessibility": True, "screenRecording": True}, capabilities + app_a = a.call("getApp", {"selector": str(fixture.pid)})["appHandle"] + app_b = b.call("getApp", {"selector": str(fixture.pid)})["appHandle"] + assert app_a != app_b + b.call("listWindows", {"appHandle": app_a}, error="STALE_APP") + settle_until = time.monotonic() + 5 + while True: + windows = a.call("listWindows", {"appHandle": app_a}) + primary = next(value for value in windows if value["title"] == "DSH CUA primary fixture") + size = [primary["bounds"]["width"], primary["bounds"]["height"]] + if size == json.loads(state.read_text())["primaryWindowSize"]: + break + assert time.monotonic() < settle_until, "Fixture window geometry did not settle" + time.sleep(0.025) + window = primary["windowId"] + observed = a.call("getState", {"appHandle": app_a, "windowId": window, "maxElements": 300, "maxDepth": 20}) + b.call("click", {"appHandle": app_b, "snapshotId": observed["snapshotId"], "elementIndex": 0}, error="STALE_SNAPSHOT") + + def capture(pair): + client, handle = pair + result = client.call("screenshot", {"appHandle": handle, "windowId": window}) + assert result["mimeType"] == "image/png" and result["pixelWidth"] > 0, result + return result["pixelWidth"], result["pixelHeight"] + + for _ in range(3): + with concurrent.futures.ThreadPoolExecutor(max_workers=2) as workers: + sizes = list(workers.map(capture, [(a, app_a), (b, app_b)])) + assert sizes[0] == sizes[1], sizes + timed = Connection(socket_path, key, screenshotTimeoutMs=1) + clients.append(timed) + timed_handle = timed.call("getApp", {"selector": str(fixture.pid)})["appHandle"] + timed.call("screenshot", {"appHandle": timed_handle, "windowId": window}, error="NATIVE_TIMEOUT") + assert timed.call("listWindows", {"appHandle": timed_handle}) + assert b.call("listWindows", {"appHandle": app_b}) + timed.close() + clients.remove(timed) + a.close() + clients.remove(a) + b.call("protocol-still-open", {}, error="UNSUPPORTED") + assert b.ready["pid"] == owner_pid + b.close() + clients.remove(b) + wait_released(lock_path, owner_pid) + owner_pid = None + + daemon = subprocess.Popen([base[0], "--serve", *base[1:]], stdout=subprocess.PIPE, stderr=subprocess.PIPE) + try: + assert select.select([daemon.stdout], [], [], 15)[0], "Daemon did not report ready" + signalled_ready = json.loads(daemon.stdout.readline()) + assert signalled_ready["type"] == "ready", signalled_ready + daemon.send_signal(signal.SIGTERM) + status = daemon.wait(timeout=15) + assert status == 128 + signal.SIGTERM, (status, daemon.stderr.read()) + wait_released(lock_path, daemon.pid) + finally: + stop(daemon) + daemon.stdout.close() + daemon.stderr.close() + print("PASS: concurrent launch, shared PID, client isolation, cleanup/lock release and SIGTERM") + finally: + errors = [] + for client in reversed(clients): + try: + client.close() + except Exception as error: + errors.append(error) + if fixture is not None: + stop(fixture) + fixture.stdout.close() + if owner_pid is not None: + try: + wait_released(lock_path, owner_pid) + except Exception as error: + errors.append(error) + if errors: + raise RuntimeError(f"Shared-helper cleanup failed: {errors}") + + +if __name__ == "__main__": + main() diff --git a/packages/computer-use/cua/native/tests/swift/SharedHelperTests.swift b/packages/computer-use/cua/native/tests/swift/SharedHelperTests.swift new file mode 100644 index 0000000000..1e11aa31e2 --- /dev/null +++ b/packages/computer-use/cua/native/tests/swift/SharedHelperTests.swift @@ -0,0 +1,87 @@ +// Shared helper admission preserves connection settings and private runtime ownership. +import Foundation +import Darwin +import Testing +@testable import CuaNative + +private let runtimeKey = String(repeating: "a", count: 64) + +private func hello(overrides: [String: JSONValue] = [:]) throws -> Data { + var config: [String: JSONValue] = [ + "maxRequestBytes": .number(4096), "maxResponseBytes": .number(8192), "axTimeoutMs": .number(250), + "textEventIntervalMs": .number(0.5), "screenshotTimeoutMs": .number(2000), + "timeoutMs": .number(8000), "graceMs": .number(500), + ] + config.merge(overrides) { _, value in value } + return try JSONEncoder().encode(JSONValue.object([ + "type": .string("hello"), "protocol": .number(1), "runtimeKey": .string(runtimeKey), "config": .object(config), + ])) +} + +@Test func admitsPerConnectionConfiguration() throws { + let configuration = try HelperConfiguration(hello: hello(), runtimeKey: runtimeKey) + #expect(configuration.maxRequestBytes == 4096) + #expect(configuration.maxResponseBytes == 8192) + #expect(configuration.axTimeoutSeconds == 0.25) + #expect(configuration.textEventIntervalSeconds == 0.0005) + #expect(configuration.screenshotTimeoutSeconds == 2) + #expect(configuration.timeoutSeconds == 8) + #expect(configuration.shutdownGraceSeconds == 0.5) + #expect(throws: NativeFailure.self) { + try HelperConfiguration(hello: hello(), runtimeKey: String(repeating: "b", count: 64)) + } +} + +@Test(arguments: [ + ("maxRequestBytes", JSONValue.number(1023)), ("maxResponseBytes", .number(1.5)), + ("axTimeoutMs", .number(0)), ("textEventIntervalMs", .number(-1)), + ("screenshotTimeoutMs", .number(0)), ("timeoutMs", .string("8000")), ("graceMs", .number(-1)), + ("unexpected", .bool(true)), +]) +func refusesInvalidSharedConfiguration(_ name: String, _ value: JSONValue) throws { + #expect(throws: NativeFailure.self) { try HelperConfiguration(hello: hello(overrides: [name: value]), runtimeKey: runtimeKey) } +} + +@Test func rejectsSharedHelloFieldsAndProtocolVersions() throws { + let base = try JSONDecoder().decode(JSONValue.self, from: hello()) + for (name, value) in [("protocol", JSONValue.number(2)), ("type", .string("request")), ("extra", .null)] { + var fields = try #require(base.object) + fields[name] = value + let data = try JSONEncoder().encode(JSONValue.object(fields)) + #expect(throws: NativeFailure.self) { try HelperConfiguration(hello: data, runtimeKey: runtimeKey) } + } +} + +@Test func versionLocksStayExclusiveUntilOwnerRelease() throws { + let directory = "/tmp/cua-\(UUID().uuidString.prefix(12))" + let options = try SharedHelperOptions(arguments: ["--ensure-shared", "--runtime-key", runtimeKey, + "--runtime-directory", directory, "--startup-timeout-ms", "1000"]) + try options.prepareDirectory() + defer { try? FileManager.default.removeItem(atPath: directory) } + var first = try SharedHelperLock(path: options.versionLockPath) + #expect(first != nil) + try withExtendedLifetime(first) { #expect(try SharedHelperLock(path: options.versionLockPath) == nil) } + first = nil + let successor = try SharedHelperLock(path: options.versionLockPath) + #expect(successor != nil) +} + +@Test func refusesSymlinkRuntimeDirectories() throws { + let directory = "/tmp/cua-\(UUID().uuidString.prefix(12))" + guard symlink("/tmp", directory) == 0 else { throw SharedSocket.failure("Creating fixture symlink") } + defer { unlink(directory) } + let options = try SharedHelperOptions(arguments: ["--ensure-shared", "--runtime-key", runtimeKey, + "--runtime-directory", directory, "--startup-timeout-ms", "1000"]) + #expect(throws: NativeFailure.self) { try options.prepareDirectory() } +} + +@Test func validatesSharedLaunchIdentityAndSocketPath() throws { + for arguments in [ + ["--ensure-shared", "--runtime-key", "invalid", "--runtime-directory", "/tmp/cua", "--startup-timeout-ms", "1000"], + ["--serve", "--runtime-key", runtimeKey, "--runtime-directory", "relative", "--startup-timeout-ms", "1000"], + ["--serve", "--runtime-key", runtimeKey, "--runtime-directory", "/tmp/cua", "--startup-timeout-ms", "0"], + ["--serve", "--runtime-key", runtimeKey, "--runtime-directory", "/" + String(repeating: "x", count: 100), "--startup-timeout-ms", "1000"], + ] { + #expect(throws: NativeFailure.self) { try SharedHelperOptions(arguments: arguments) } + } +} diff --git a/packages/computer-use/cua/native/upstream.json b/packages/computer-use/cua/native/upstream.json index a3a670c7b8..03b40dfe4a 100644 --- a/packages/computer-use/cua/native/upstream.json +++ b/packages/computer-use/cua/native/upstream.json @@ -13,7 +13,7 @@ { "local": "Sources/CuaNative/WindowPointerInput.swift", "upstream": "libs/cua-driver/rust/crates/platform-macos/src/input/mouse.rs", - "changes": "Reimplement the exposed window-directed click, drag and line-scroll paths in Swift. Preserve routing fields and gesture timing, post each event once, and release this operation’s held button on cooperative cancellation. Preserve fractional movement delays for short or unevenly divided drags. Support middle-button and modifier-bearing clicks and configurable-button drags." + "changes": "Reimplement the exposed window-directed click, drag and line-scroll paths in Swift. Preserve routing fields and gesture timing, post each event once, and release this operation\u2019s held button on cooperative cancellation. Preserve fractional movement delays for short or unevenly divided drags. Support middle-button and modifier-bearing clicks and configurable-button drags." }, { "local": "Sources/CuaNative/PIDEventPoster.swift", @@ -39,6 +39,7 @@ "localFiles": [ "Sources/CuaNative/AXTextFormatter.swift", "Sources/CuaNative/AXWorker.swift", + "Sources/CuaNative/ApplicationAuthorization.swift", "Sources/CuaNative/ApplicationDiscovery.swift", "Sources/CuaNative/ClipboardPaste.swift", "Sources/CuaNative/Main.swift", @@ -46,8 +47,12 @@ "Sources/CuaNative/Protocol.swift", "Sources/CuaNative/RequestAdmission.swift", "Sources/CuaNative/RequestCoordinator.swift", + "Sources/CuaNative/RequestScheduler.swift", "Sources/CuaNative/Runtime.swift", "Sources/CuaNative/ScreenshotTrace.swift", + "Sources/CuaNative/SharedHelperProxy.swift", + "Sources/CuaNative/SharedHelperRuntime.swift", + "Sources/CuaNative/SharedHelperServer.swift", "Sources/CuaNative/TextSelection.swift", "Sources/CuaNative/WindowCapture.swift", "Sources/CuaNative/WindowGeometry.swift" diff --git a/packages/computer-use/cua/src/index.ts b/packages/computer-use/cua/src/index.ts index 6d547b0fd5..97df99da5b 100644 --- a/packages/computer-use/cua/src/index.ts +++ b/packages/computer-use/cua/src/index.ts @@ -1,4 +1,4 @@ -/** macOS computer use over a managed native helper. */ +/** macOS computer use through Agent-scoped connections to a shared native helper. */ import { isAbsolute, join, resolve } from 'node:path' import { tmpdir } from 'node:os' import { withFileLock } from '@deepseek-ai/dsh-atomic-write' @@ -25,7 +25,7 @@ import type {} from '@deepseek-ai/cordis-plugin-loader' /** Loader plugin identity. */ export const name = 'cua' -/** Services required for logged tools and owned native process lifetime. */ +/** Services required for logged tools and native launcher ownership. */ export const inject = ['computerUse', 'tools', 'systemPrompt', 'subprocess', 'agents'] /** Native executable and per-operation resource bounds. */ @@ -41,6 +41,10 @@ export interface Config { buildTimeoutMs: number /** Deadline for one native operation, including capture and AX traversal. */ timeoutMs: number + /** Maximum time to connect to or start the shared native helper. */ + connectTimeoutMs: number + /** Screenshot deadline including time queued behind other native operations. */ + screenshotTimeoutMs: number /** Deadline for each individual Accessibility system call. */ axTimeoutMs: number /** Maximum bytes retained for one helper response frame. */ @@ -57,7 +61,7 @@ export interface Config { dragSteps: number /** Delay in milliseconds after each Unicode key-down or key-up event. */ textEventIntervalMs: number - /** Managed process termination grace in milliseconds. */ + /** Agent connection cleanup and native launcher termination grace in milliseconds. */ graceMs: number /** Maximum nodes admitted by a native accessibility snapshot. */ maxElements: number @@ -74,6 +78,8 @@ export const Config = Schema.object({ helperPath: Schema.string(), buildTimeoutMs: Schema.number().step(1).min(1).max(2 ** 31 - 1).default(300_000), timeoutMs: Schema.number().step(1).min(1).max(2 ** 31 - 1).default(30_000), + connectTimeoutMs: Schema.number().step(1).min(1).max(2 ** 31 - 1).default(5000), + screenshotTimeoutMs: Schema.number().step(1).min(1).max(2 ** 31 - 1).default(5000), axTimeoutMs: Schema.number().step(1).min(1).max(2 ** 31 - 1).default(3000), maxAppMatches: Schema.number().step(1).min(1).max(1000).default(20), maxClipboardBytes: Schema.number().step(1).min(1024).default(32 * 1024 * 1024), @@ -167,6 +173,7 @@ export async function apply(ctx: Context, config: Config): Promise { function mount(ctx: Context, config: Config, helperPath: string): void { const native: NativeClientConfig = { helperPath, cwd: process.cwd(), timeoutMs: config.timeoutMs, axTimeoutMs: config.axTimeoutMs, + connectTimeoutMs: config.connectTimeoutMs, screenshotTimeoutMs: config.screenshotTimeoutMs, maxRequestBytes: config.maxRequestBytes, maxResponseBytes: config.maxResponseBytes, graceMs: config.graceMs, textEventIntervalMs: config.textEventIntervalMs, } diff --git a/packages/computer-use/cua/src/native-client.ts b/packages/computer-use/cua/src/native-client.ts index 8153c3a7dc..019ba815ca 100644 --- a/packages/computer-use/cua/src/native-client.ts +++ b/packages/computer-use/cua/src/native-client.ts @@ -1,9 +1,11 @@ -/** Byte-bounded JSON-line transport with linear frame assembly for one owned helper. */ +/** Agent-scoped JSONL requests over a connection to the shared native helper. */ import { randomUUID } from 'node:crypto' -import type { SubprocessHandle, SubprocessRuntime } from '@deepseek-ai/dsh-subprocess' +import assert from 'node:assert/strict' +import type { SubprocessRuntime } from '@deepseek-ai/dsh-subprocess' import type { ToolExecution } from '@deepseek-ai/dsh-tools' import type { JsonValue } from '@deepseek-ai/dsh-util-values' import { z } from 'zod' +import { createSharedNativeChannel, type NativeChannel } from './shared-native.ts' const Reply = z.union([ z.object({ id: z.string(), result: z.json() }).strict(), @@ -14,6 +16,7 @@ const screenshotPrefix = '[cua-screenshot] ' const ScreenshotDiagnostic = z.object({ requestId: z.string(), stage: z.string(), elapsedMs: z.number().finite().nonnegative(), errorCode: z.string().optional(), }) +const WireReply = z.union([Reply, ScreenshotDiagnostic.extend({ type: z.literal('diagnostic') })]) /** Native refusal or transport failure with a stable machine-readable code. */ export class NativeError extends Error { @@ -33,6 +36,8 @@ export interface NativeClientConfig { helperPath: string cwd: string timeoutMs: number + connectTimeoutMs: number + screenshotTimeoutMs: number axTimeoutMs: number maxRequestBytes: number maxResponseBytes: number @@ -40,14 +45,17 @@ export interface NativeClientConfig { textEventIntervalMs: number } -/** Serial helper calls retain native handles until helper disposal or replacement. */ +/** One Agent owns this connection and its native handles; other Agents use independent connections. */ export class NativeClient { - private readonly handle: SubprocessHandle + private readonly handle: NativeChannel private readonly config: NativeClientConfig private readonly chunks: Buffer[] = [] private bufferedBytes = 0 private failure: NativeError | undefined private disposal: Promise | undefined + private diagnostics: string[] = [] + private diagnosticBytes = 0 + private discardedDiagnostics = false private active: { id: string; resolve: (value: JsonValue) => void; reject: (error: NativeError) => void } | undefined /** @@ -58,33 +66,23 @@ export class NativeClient { constructor(owner: { subprocess: Pick }, config: NativeClientConfig, private readonly reportDiagnostic?: (message: string) => void) { this.config = config - this.handle = owner.subprocess.spawn({ - argv: [ - config.helperPath, - '--max-request-bytes', String(config.maxRequestBytes), - '--max-response-bytes', String(config.maxResponseBytes), - '--ax-timeout-ms', String(config.axTimeoutMs), - '--shutdown-grace-ms', String(config.graceMs), - '--text-event-interval-ms', String(config.textEventIntervalMs), - ], cwd: config.cwd, graceMs: config.graceMs, - stdio: { stdin: 'pipe', stdout: 'pipe', stderr: { maxBytes: 8192 } }, - }) - this.handle.stdout?.on('data', (chunk: Buffer) => { this.receive(chunk) }) - this.handle.stdout?.on('error', (error: Error) => { this.fail(new NativeError('NATIVE_IO', 'Helper stdout failed', { cause: error })) }) - this.handle.stdout?.on('end', () => { this.fail(new NativeError('NATIVE_EOF', 'Helper output ended; acquire fresh application handles')) }) - this.handle.stdout?.on('close', () => { this.fail(new NativeError('NATIVE_EOF', 'Helper output closed; acquire fresh application handles')) }) - this.handle.stdin?.on('error', (error: Error) => { this.fail(new NativeError('NATIVE_IO', 'Helper stdin failed', { cause: error })) }) + this.handle = createSharedNativeChannel(owner, config) + this.handle.stdout.on('data', (chunk: Buffer) => { this.receive(chunk) }) + this.handle.stdout.on('error', (error: Error) => { this.fail(new NativeError('NATIVE_IO', `Native connection failed: ${error.message}`, { cause: error })) }) + this.handle.stdout.on('end', () => { this.fail(new NativeError('NATIVE_EOF', 'Native connection ended; acquire fresh application handles')) }) + this.handle.stdout.on('close', () => { this.fail(new NativeError('NATIVE_EOF', 'Native connection closed; acquire fresh application handles')) }) + this.handle.stdin.on('error', (error: Error) => { this.fail(new NativeError('NATIVE_IO', 'Writing native requests failed', { cause: error })) }) void this.handle.done.then( - () => { this.fail(new NativeError('NATIVE_EXITED', 'Helper exited; acquire fresh application handles')) }, - (error: unknown) => { this.fail(new NativeError('NATIVE_EXITED', 'Helper process failed', { cause: error })) }, + () => { this.fail(new NativeError('NATIVE_EXITED', 'Native connection closed; acquire fresh application handles')) }, + (error: unknown) => { this.fail(new NativeError('NATIVE_EXITED', 'Native connection failed', { cause: error })) }, ) } /** - * Send one operation. Native refusals preserve the helper; transport failures invalidate it. + * Send one operation. Native refusals preserve this connection; transport failures invalidate its handles. * @param method - Helper operation name. * @param params - JSON operation parameters. - * @param signal - Cancellation kills this helper and invalidates its retained handles. + * @param signal - Cancellation stops this request; the connection survives when native cleanup acknowledges it. * @param callId - Logged tool invocation ID, when this request belongs to a tool execution. * @returns The validated helper result, after failure cleanup when needed. */ @@ -97,9 +95,13 @@ export class NativeClient { if (this.active) throw new NativeError('NATIVE_BUSY', 'Await native operations sequentially') const id = callId ?? randomUUID() const capture = method === 'screenshot' - const diagnosticOffset = capture ? this.handle.collected.stderr?.readFrom(0).nextOffset ?? 0 : 0 + this.diagnostics = [] + this.diagnosticBytes = 0 + this.discardedDiagnostics = false const started = performance.now() let nativeFailure: NativeError | undefined + let interruption: NativeError | undefined + let cancellationTimer: ReturnType | undefined const frame = `${JSON.stringify({ id, method, params })}\n` if (Buffer.byteLength(frame) > this.config.maxRequestBytes) { throw new NativeError('NATIVE_REQUEST_LIMIT', 'Request exceeded the configured byte limit before dispatch') @@ -107,38 +109,54 @@ export class NativeClient { const outcome = Promise.withResolvers() this.active = { id, resolve: outcome.resolve, reject: (error) => { nativeFailure = error; outcome.reject(error) } } if (capture) this.diagnostic(screenshotPrefix + JSON.stringify({ requestId: id, stage: 'host.dispatch', elapsedMs: 0 })) - const abort = (): void => { this.fail(new NativeError('NATIVE_CANCELLED', 'Operation cancelled; inspect fresh state before retrying input')) } + const interrupt = (error: NativeError): void => { + if (interruption || this.active?.id !== id) return + interruption = error + cancellationTimer = setTimeout(() => { + this.fail(new NativeError('NATIVE_CLEANUP', 'Native operation did not acknowledge cancellation; acquire fresh application handles', { cause: error })) + }, this.config.graceMs) + try { + this.handle.stdin.write(`${JSON.stringify({ type: 'cancel', id })}\n`, (failure) => { + if (failure) this.fail(new NativeError('NATIVE_IO', 'Sending native cancellation failed', { cause: failure })) + }) + } catch (failure) { + this.fail(new NativeError('NATIVE_IO', 'Sending native cancellation failed', { cause: failure })) + } + } + const abort = (): void => { interrupt(new NativeError('NATIVE_CANCELLED', 'Operation cancelled; inspect fresh state before retrying input')) } signal.addEventListener('abort', abort, { once: true }) + const timeoutMs = capture ? Math.min(this.config.timeoutMs, this.config.screenshotTimeoutMs) : this.config.timeoutMs const timer = setTimeout(() => { - this.fail(new NativeError('NATIVE_TIMEOUT', `Operation exceeded ${this.config.timeoutMs}ms`)) - }, this.config.timeoutMs) + interrupt(new NativeError('NATIVE_TIMEOUT', `Operation exceeded ${timeoutMs}ms`)) + }, timeoutMs) try { - if (!this.handle.stdin || !this.handle.stdout) { - this.fail(new NativeError('NATIVE_IO', 'Helper did not expose the requested protocol pipes')) - } else { - try { - this.handle.stdin.write(frame, (error) => { - if (error) this.fail(new NativeError('NATIVE_IO', 'Writing a helper request failed', { cause: error })) - }) - } catch (error) { - this.fail(new NativeError('NATIVE_IO', 'Writing a helper request failed', { cause: error })) - } + try { + this.handle.stdin.write(frame, (error) => { + if (error) this.fail(new NativeError('NATIVE_IO', 'Writing a helper request failed', { cause: error })) + }) + } catch (error) { + this.fail(new NativeError('NATIVE_IO', 'Writing a helper request failed', { cause: error })) } const result = await outcome.promise if (this.failure) throw this.failure + if (interruption) throw interruption if (capture) this.diagnostic(screenshotPrefix + JSON.stringify({ requestId: id, stage: 'host.reply', elapsedMs: performance.now() - started, })) return result + } catch (error) { + if (interruption && !this.failure) throw interruption + throw error } finally { clearTimeout(timer) + clearTimeout(cancellationTimer) signal.removeEventListener('abort', abort) try { if (this.failure !== undefined) await this.dispose() } finally { if (capture) { - const diagnostics = this.screenshotDiagnostics(id, diagnosticOffset) - const failure = nativeFailure ?? this.failure + const diagnostics = this.screenshotDiagnostics() + const failure = this.failure ?? interruption ?? nativeFailure if (failure && diagnostics) failure.message += `\nScreenshot diagnostics:\n${diagnostics}` if (diagnostics) this.diagnostic(diagnostics) this.diagnostic(screenshotPrefix + JSON.stringify({ @@ -149,26 +167,25 @@ export class NativeClient { } } - private screenshotDiagnostics(id: string, offset: number): string { - const output = this.handle.collected.stderr?.readFrom(offset) - if (!output) return '' - const records: string[] = [] - for (const line of output.text.split('\n')) { - if (!line.startsWith(screenshotPrefix)) continue - let value: unknown - try { value = JSON.parse(line.slice(screenshotPrefix.length)) } - catch (error) { - // A retained tail may begin or end inside one diagnostic record. - void error - continue - } - const parsed = ScreenshotDiagnostic.safeParse(value) - if (parsed.success && parsed.data.requestId === id) records.push(screenshotPrefix + JSON.stringify(parsed.data)) - } - if (output.lossy) records.unshift('[Earlier screenshot diagnostic bytes were discarded from the bounded stderr tail.]') + private screenshotDiagnostics(): string { + const records = [...this.diagnostics] + if (this.discardedDiagnostics) records.unshift('[Earlier screenshot diagnostic records were discarded from the bounded diagnostic buffer.]') return records.join('\n') } + private retainDiagnostic(value: z.infer): void { + if (value.requestId !== this.active?.id) return + const line = screenshotPrefix + JSON.stringify(ScreenshotDiagnostic.parse(value)) + this.diagnostics.push(line) + this.diagnosticBytes += Buffer.byteLength(line) + while (this.diagnosticBytes > 8192) { + const discarded = this.diagnostics.shift() + assert(discarded !== undefined, 'Diagnostic bytes must belong to retained records') + this.diagnosticBytes -= Buffer.byteLength(discarded) + this.discardedDiagnostics = true + } + } + private diagnostic(message: string): void { try { this.reportDiagnostic?.(message) } catch (error) { @@ -177,22 +194,21 @@ export class NativeClient { } } - /** Whether this helper has lost its retained native state. */ + /** Whether this Agent's connection has lost its retained native state. */ get disposed(): boolean { return this.failure !== undefined } - /** Invalidate retained handles and request termination; the owner must await dispose before releasing the helper. */ + /** Invalidate this Agent's handles and close its connection; await dispose for native cleanup. */ invalidate(): void { this.fail(new NativeError('NATIVE_DISPOSED', 'Helper disposed')) } - /** Stop pending work and await the entire managed process range before releasing streams. */ + /** Stop this Agent's pending work and await its native connection cleanup. */ dispose(): Promise { this.invalidate() this.disposal ??= (async () => { try { - const [, range] = await Promise.allSettled([this.handle.done, this.handle.waitForExit()]) - if (range.status === 'rejected') throw range.reason + await this.handle.waitForClose() } finally { - this.handle.stdout?.destroy() - this.handle.stdin?.destroy() + this.handle.stdout.destroy() + this.handle.stdin.destroy() } })() return this.disposal @@ -205,29 +221,40 @@ export class NativeClient { this.bufferedBytes = 0 this.active?.reject(error) this.active = undefined - this.handle.terminate() + this.handle.close() } private receive(chunk: Buffer): void { if (this.failure) return - if (this.bufferedBytes + chunk.length > this.config.maxResponseBytes) { - this.fail(new NativeError('NATIVE_RESPONSE_LIMIT', 'Helper response exceeded the configured byte limit')) - return + let offset = 0 + while (offset < chunk.length && !this.failure) { + const newline = chunk.indexOf(10, offset) + const end = newline < 0 ? chunk.length : newline + 1 + const piece = chunk.subarray(offset, end) + if (this.bufferedBytes + piece.length > this.config.maxResponseBytes) { + this.fail(new NativeError('NATIVE_RESPONSE_LIMIT', 'Helper response exceeded the configured byte limit')) + return + } + this.chunks.push(piece) + this.bufferedBytes += piece.length + if (newline >= 0) this.receiveFrame() + offset = end } - if (chunk.length === 0) return - this.chunks.push(chunk) - this.bufferedBytes += chunk.length - const newline = chunk.indexOf(10) - if (newline < 0) return + } + + private receiveFrame(): void { try { - if (newline !== chunk.length - 1) throw new Error('One request must produce exactly one response frame') const frame = Buffer.concat(this.chunks, this.bufferedBytes) const line = new TextDecoder('utf-8', { fatal: true }).decode(frame.subarray(0, frame.length - 1)) - const reply = Reply.parse(JSON.parse(line)) - const active = this.active - if (!active || reply.id !== active.id) throw new Error('Helper reply has no matching request') + const reply = WireReply.parse(JSON.parse(line)) this.chunks.length = 0 this.bufferedBytes = 0 + if ('type' in reply) { + this.retainDiagnostic(reply) + return + } + const active = this.active + if (!active || reply.id !== active.id) throw new Error('Helper reply has no matching request') this.active = undefined if ('error' in reply) { const error = new NativeError(reply.error.code, reply.error.message) diff --git a/packages/computer-use/cua/src/shared-native.ts b/packages/computer-use/cua/src/shared-native.ts new file mode 100644 index 0000000000..b3af0cce73 --- /dev/null +++ b/packages/computer-use/cua/src/shared-native.ts @@ -0,0 +1,294 @@ +/** Per-Agent byte connections to the per-user native screenshot process. */ +import { createHash } from 'node:crypto' +import { lstat, mkdir, readFile, realpath } from 'node:fs/promises' +import { Socket } from 'node:net' +import { userInfo } from 'node:os' +import { join } from 'node:path' +import { PassThrough, Writable, type Readable } from 'node:stream' +import { setTimeout as delay } from 'node:timers/promises' +import type { SubprocessHandle, SubprocessOutcome, SubprocessRuntime } from '@deepseek-ai/dsh-subprocess' +import { z } from 'zod' +import type { NativeClientConfig } from './native-client.ts' + +const Ready = z.object({ + type: z.literal('ready'), protocol: z.literal(1), runtimeKey: z.string(), + pid: z.number().int().positive(), instanceId: z.string().min(1), +}).strict() +const Refusal = z.object({ type: z.literal('error'), code: z.string().min(1), message: z.string() }).strict() +const handshakeByteLimit = 8192 +type LauncherExit = [PromiseSettledResult, PromiseSettledResult] +interface RuntimeLocation { helperPath: string; runtimeKey: string; directory: string } + +/** One Agent's ordered native protocol stream and its cleanup completion. */ +export interface NativeChannel { + readonly stdin: Writable + readonly stdout: Readable + readonly done: Promise + /** Stop writes and release this connection's native state without terminating other clients. */ + close(): void + /** @returns Completion after this socket and its managed startup process have closed. */ + waitForClose(): Promise +} + +function transportError(code: string, message: string, cause?: unknown): Error & { code: string } { + return Object.assign(new Error(message, { cause }), { code }) +} + +function asError(error: unknown): Error { + return error instanceof Error ? error : transportError('NATIVE_CONNECT', 'Native connection failed', error) +} + +async function untilAbort(operation: Promise, signal: AbortSignal): Promise { + const cancelled = Promise.withResolvers() + const abort = (): void => { cancelled.reject(signal.reason) } + if (signal.aborted) abort() + else signal.addEventListener('abort', abort, { once: true }) + try { return await Promise.race([operation, cancelled.promise]) } + finally { signal.removeEventListener('abort', abort) } +} + +class SharedNativeChannel implements NativeChannel { + readonly stdin: Writable + readonly stdout = new PassThrough() + readonly done: Promise + private readonly ready = Promise.withResolvers() + private readonly closed: PromiseWithResolvers = Promise.withResolvers() + private readonly startup = new AbortController() + private readonly startupTimer: ReturnType + private launcher: SubprocessHandle | undefined + private launcherExit: Promise | undefined + private launcherFinished = false + private socket: Socket | undefined + private socketClosed: Promise | undefined + private connected = false + private closing = false + private cleanupTimer: ReturnType | undefined + private cleanupFailure: Error | undefined + + constructor(private readonly owner: { subprocess: Pick }, private readonly config: NativeClientConfig) { + this.stdin = new Writable({ + write: (chunk: Buffer, encoding, callback) => { + void this.ready.promise.then((socket) => { + if (this.closing) callback(transportError('NATIVE_CLOSED', 'Native connection is closing')) + else { + try { socket.write(chunk, encoding, callback) } + catch (error) { callback(asError(error)) } + } + }, callback) + }, + final: (callback) => { this.close(); callback() }, + destroy: (error, callback) => { this.close(); callback(error) }, + }) + // Consumers can attach after construction; connection failures still settle done. + this.stdin.on('error', () => {}) + this.stdout.on('error', () => {}) + void this.ready.promise.catch(() => {}) + void this.closed.promise.catch(() => {}) + this.startupTimer = setTimeout(() => { + this.startup.abort(transportError('NATIVE_CONNECT_TIMEOUT', `Native connection exceeded ${config.connectTimeoutMs}ms`)) + }, config.connectTimeoutMs) + this.done = this.run() + void this.done.catch(() => {}) + } + + close(): void { + if (this.closing) return + this.closing = true + clearTimeout(this.startupTimer) + if (!this.connected) { + this.startup.abort(transportError('NATIVE_CLOSED', 'Native connection closed before it became ready')) + this.socket?.destroy() + if (!this.launcherFinished) this.launcher?.terminate() + } else { + this.socket?.end() + this.cleanupTimer = setTimeout(() => { + this.cleanupFailure = transportError('NATIVE_CLEANUP', `Native connection did not finish cleanup within ${this.config.graceMs}ms`) + this.socket?.destroy(this.cleanupFailure) + }, this.config.graceMs) + } + } + + waitForClose(): Promise { return this.closed.promise } + + private async run(): Promise { + let failure: Error | undefined + try { + const socket = await this.connect(await this.prepare()) + this.startup.signal.throwIfAborted() + clearTimeout(this.startupTimer) + this.connected = true + this.ready.resolve(socket) + await this.socketClosed + } catch (error) { + failure = asError(this.startup.signal.aborted ? this.startup.signal.reason : error) + this.ready.reject(failure) + if (!this.closing) this.stdout.destroy(failure) + } finally { + clearTimeout(this.startupTimer) + clearTimeout(this.cleanupTimer) + this.closing = true + this.socket?.destroy() + try { await this.cleanup() } + catch (error) { this.cleanupFailure ??= asError(error) } + this.stdin.destroy() + if (this.cleanupFailure) { + this.stdout.destroy(this.cleanupFailure) + this.closed.reject(this.cleanupFailure) + } else { + this.stdout.end() + this.closed.resolve() + } + } + if (this.cleanupFailure) throw this.cleanupFailure + if (failure) throw failure + } + + private async prepare(): Promise { + const helperPath = await realpath(this.config.helperPath) + const runtimeKey = createHash('sha256').update(await readFile(helperPath, { signal: this.startup.signal })).digest('hex') + const uid = userInfo().uid + const directory = `/tmp/dsh-cua-${uid}` + try { await mkdir(directory, { mode: 0o700 }) } + catch (error) { + if (!(error instanceof Error && 'code' in error && error.code === 'EEXIST')) throw error + } + const stat = await lstat(directory) + if (!stat.isDirectory() || stat.uid !== uid || (stat.mode & 0o777) !== 0o700) { + throw transportError('NATIVE_RUNTIME_DIRECTORY', 'Native runtime directory must be a real, owner-only directory owned by the current user') + } + return { helperPath, runtimeKey, directory } + } + + private async ensureShared({ helperPath, runtimeKey, directory }: RuntimeLocation): Promise { + this.startup.signal.throwIfAborted() + this.launcherFinished = false + this.launcher = this.owner.subprocess.spawn({ + argv: [helperPath, '--ensure-shared', '--runtime-key', runtimeKey, '--runtime-directory', directory, + '--startup-timeout-ms', String(this.config.connectTimeoutMs)], + cwd: this.config.cwd, graceMs: this.config.graceMs, signal: this.startup.signal, + stdio: { stdin: 'ignore', stdout: { maxBytes: 8192 }, stderr: { maxBytes: 8192 } }, + }) + this.launcherExit = Promise.allSettled([this.launcher.done, this.launcher.waitForExit()]) + const [outcome, range] = await untilAbort(this.launcherExit, this.startup.signal) + this.launcherFinished = true + if (range.status === 'rejected') throw range.reason + if (!range.value) throw transportError('NATIVE_CLEANUP', 'Native launcher process range did not close') + if (outcome.status === 'rejected') throw outcome.reason + if (outcome.value.exitCode !== 0 || outcome.value.signal !== null) { + const stderr = this.launcher.collected.stderr?.readFrom(0).text ?? '' + throw transportError('NATIVE_STARTUP', `Native launcher failed (exit ${outcome.value.exitCode}, signal ${outcome.value.signal})${stderr ? `: ${stderr}` : ''}`) + } + } + + private async connect(location: RuntimeLocation): Promise { + for (;;) { + await this.ensureShared(location) + const socket = await this.connectOnce(join(location.directory, `${location.runtimeKey}.sock`), location.runtimeKey) + if (socket) return socket + // A server can exit with its last client before admitting this connection. + await delay(25, undefined, { signal: this.startup.signal }) + } + } + + private async connectOnce(path: string, runtimeKey: string): Promise { + this.startup.signal.throwIfAborted() + const socket = new Socket() + this.socket = socket + const handshake: PromiseWithResolvers = Promise.withResolvers() + const ended: PromiseWithResolvers = Promise.withResolvers() + this.socketClosed = ended.promise + void ended.promise.catch(() => {}) + let peerEnded = false + let socketError: Error | undefined + let bytes = Buffer.alloc(0) + let accepted = false + let refused = false + const abort = (): void => { socket.destroy(asError(this.startup.signal.reason)) } + this.startup.signal.addEventListener('abort', abort, { once: true }) + socket.on('error', (error: Error) => { socketError = error; handshake.reject(error) }) + socket.on('end', () => { peerEnded = true }) + socket.on('close', () => { + this.startup.signal.removeEventListener('abort', abort) + const error = socketError ?? (peerEnded ? undefined : transportError('NATIVE_EOF', 'Native socket closed without cleanup acknowledgement')) + handshake.reject(error ?? transportError('NATIVE_EOF', 'Native socket ended before its handshake')) + if (error) ended.reject(error) + else ended.resolve() + }) + socket.once('connect', () => { + const { maxRequestBytes, maxResponseBytes, axTimeoutMs, textEventIntervalMs, screenshotTimeoutMs, timeoutMs, graceMs } = this.config + socket.write(JSON.stringify({ type: 'hello', protocol: 1, runtimeKey, + config: { maxRequestBytes, maxResponseBytes, axTimeoutMs, textEventIntervalMs, screenshotTimeoutMs, timeoutMs, graceMs } }) + '\n') + }) + const receive = (chunk: Buffer): void => { + const newline = chunk.indexOf(10) + const length = newline < 0 ? chunk.length : newline + 1 + if (bytes.length + length > handshakeByteLimit) { + refused = true + socket.destroy(transportError('NATIVE_PROTOCOL', 'Native handshake exceeded its byte limit')) + return + } + bytes = Buffer.concat([bytes, chunk.subarray(0, length)]) + if (newline < 0) return + try { + const value: unknown = JSON.parse(new TextDecoder('utf-8', { fatal: true }).decode(bytes.subarray(0, -1))) + const refusal = Refusal.safeParse(value) + if (refusal.success) { + refused = true + socket.destroy(transportError(refusal.data.code, refusal.data.message)) + return + } + const ready = Ready.parse(value) + if (ready.runtimeKey !== runtimeKey) throw transportError('NATIVE_PROTOCOL', 'Native handshake returned a different runtime identity') + accepted = true + socket.removeListener('data', receive) + this.startup.signal.removeEventListener('abort', abort) + if (chunk.length > length) this.stdout.write(chunk.subarray(length)) + socket.pipe(this.stdout, { end: false }) + handshake.resolve() + } catch (error) { + refused = true + socket.destroy(transportError('NATIVE_PROTOCOL', 'Native helper refused or returned an invalid handshake', error)) + } + } + socket.on('data', receive) + socket.connect(path) + try { + await untilAbort(handshake.promise, this.startup.signal) + return socket + } catch (error) { + socket.destroy() + await ended.promise.catch(() => {}) + if (!refused && !accepted && error instanceof Error && 'code' in error + && ['ENOENT', 'ECONNREFUSED', 'ECONNRESET', 'NATIVE_EOF'].includes(String(error.code))) return undefined + throw error + } finally { + if (!accepted) socket.removeListener('data', receive) + } + } + + private async cleanup(): Promise { + if (!this.launcherFinished) this.launcher?.terminate() + const controller = new AbortController() + const timer = setTimeout(() => { + controller.abort(transportError('NATIVE_CLEANUP', `Native startup cleanup exceeded ${this.config.graceMs}ms`)) + }, this.config.graceMs) + try { + const [, launcher] = await untilAbort(Promise.allSettled([this.socketClosed, this.launcherExit] as const), controller.signal) + if (launcher.status === 'fulfilled' && launcher.value) { + const range = launcher.value[1] + if (range.status === 'rejected') throw range.reason + if (!range.value) throw transportError('NATIVE_CLEANUP', 'Native launcher process range did not close') + } + } finally { clearTimeout(timer) } + } +} + +/** + * Prepare or join the native process; writes wait for identity-checked readiness. + * @param owner - Provider owning the short-lived native startup command. + * @param config - Native runtime identity, operation limits and connection deadlines. + * @returns One independently disposable Agent connection, without operation replay. + */ +export function createSharedNativeChannel(owner: { subprocess: Pick }, config: NativeClientConfig): NativeChannel { + return new SharedNativeChannel(owner, config) +} diff --git a/packages/computer-use/cua/tests/application-access.spec.ts b/packages/computer-use/cua/tests/application-access.spec.ts index dcdcbe6454..31b423eed9 100644 --- a/packages/computer-use/cua/tests/application-access.spec.ts +++ b/packages/computer-use/cua/tests/application-access.spec.ts @@ -28,7 +28,7 @@ vi.mock('../src/native-client.ts', async original => ({ }, })) -const nativeConfig = { helperPath: '/fixture', cwd: '/', timeoutMs: 1000, axTimeoutMs: 100, +const nativeConfig = { helperPath: '/fixture', cwd: '/', timeoutMs: 1000, connectTimeoutMs: 5000, screenshotTimeoutMs: 5000, axTimeoutMs: 100, maxRequestBytes: 4096, maxResponseBytes: 4096, graceMs: 100, textEventIntervalMs: 1 } const identity = ApplicationAuthorizationResult.parse({ id: 'a'.repeat(64), name: 'Test application', executablePath: '/Applications/Fixture.app/Contents/MacOS/Fixture', persistable: true }) const handle = 'fixture-app' as CuaAppHandle diff --git a/packages/computer-use/cua/tests/large-image-ptc.spec.ts b/packages/computer-use/cua/tests/large-image-ptc.spec.ts index 6cc3b58f82..0eb18106ba 100644 --- a/packages/computer-use/cua/tests/large-image-ptc.spec.ts +++ b/packages/computer-use/cua/tests/large-image-ptc.spec.ts @@ -1,5 +1,5 @@ /** Valid large PNG admission through Loader, MCP projection, disk attachments and a real Node PTC process. */ -import { chmod, mkdtemp, readFile, rm, writeFile } from 'node:fs/promises' +import { mkdtemp, readFile, rm, writeFile } from 'node:fs/promises' import { tmpdir } from 'node:os' import { join } from 'node:path' import { pathToFileURL } from 'node:url' @@ -29,6 +29,7 @@ import WorkingDirectory from '@deepseek-ai/dsh-working-directory' import sharp from 'sharp' import { expect, it, onTestFinished } from 'vitest' import * as NativeProvider from '../src/index.ts' +import { installNativeFixture } from './native-fixture.ts' const WIDTH = 2048 const HEIGHT = 1024 @@ -53,26 +54,11 @@ it.skipIf(process.platform === 'win32')('delivers a valid PNG larger than 4 MiB await writeFile(sourcePath, source) const encodedLength = source.toString('base64').length - const helperPath = join(root, 'large-image-helper.mjs') - await writeFile(helperPath, `#!/usr/bin/env node -import { readFileSync } from 'node:fs'; -import { createInterface } from 'node:readline'; -let acquired = false; -createInterface({ input: process.stdin }).on('line', line => { - const { id, method, params } = JSON.parse(line); - let result; - if (method === 'getApp' && params.selector === '9') { - acquired = true; - result = { appHandle: 'large-image-app', pid: 9, name: 'Generated screenshot', authorization: {"id": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", "name": "Fixture", "executablePath": "/fixture/app", "persistable": true} }; - } else if (method === 'screenshot' && acquired && params.appHandle === 'large-image-app' && params.windowId === 7) { - result = { mimeType: 'image/png', data: readFileSync(new URL('./generated-screenshot.png', import.meta.url)).toString('base64'), - pixelWidth: ${WIDTH}, pixelHeight: ${HEIGHT}, windowId: 7, coordinateSpace: 'window-points', - bounds: { x: 0, y: 0, width: 1024, height: 512 } }; - } else throw new Error('Unexpected fixture request'); - process.stdout.write(JSON.stringify({ id, result }) + '\\n'); -}); -`) - await chmod(helperPath, 0o700) + const helperPath = await installNativeFixture(root) + await writeFile(join(root, 'fixture-screenshot.json'), JSON.stringify({ + file: './generated-screenshot.png', pixelWidth: WIDTH, pixelHeight: HEIGHT, + bounds: { x: 0, y: 0, width: 1024, height: 512 }, + })) const modules = { llm: LlmRuntime, session: SessionStore, projection: SessionProjectionRegistry, systemPrompt: SystemPrompt, tools: ToolRuntime, agents: AgentRegistry, agentLoop: AgentLoop, diff --git a/packages/computer-use/cua/tests/loader-composition.spec.ts b/packages/computer-use/cua/tests/loader-composition.spec.ts index f106a021d2..9499cc5f1b 100644 --- a/packages/computer-use/cua/tests/loader-composition.spec.ts +++ b/packages/computer-use/cua/tests/loader-composition.spec.ts @@ -1,5 +1,5 @@ /** Real Loader, AgentLoop and helper-process transport; only desktop and model responses are fixtures. */ -import { chmod, copyFile, mkdtemp, readFile, rm, writeFile } from 'node:fs/promises' +import { mkdtemp, readFile, rm, writeFile } from 'node:fs/promises' import { tmpdir } from 'node:os' import { join } from 'node:path' import { pathToFileURL } from 'node:url' @@ -20,6 +20,8 @@ import ToolRuntime from '@deepseek-ai/dsh-tools' import SubprocessLocal from '@deepseek-ai/dsh-subprocess-local' import { z } from 'zod' import * as NativeProvider from '../src/index.ts' +import { createCuaApi } from '../src/api.ts' +import { installNativeFixture } from './native-fixture.ts' const png = 'iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAIAAACQd1PeAAAACXBIWXMAAAPoAAAD6AG1e1JrAAAADElEQVQImWNgZGIGAAAOAAeCcsnOAAAAAElFTkSuQmCC' const calls = [ @@ -60,13 +62,11 @@ const AuditEvent = z.object({ }) // The fixture uses a POSIX executable shebang; stream/protocol unit tests run on every host. -it.skipIf(process.platform === 'win32')('loads the native provider, records verified fixture effects and images, and joins helpers on agent disposal', async () => { +it.skipIf(process.platform === 'win32')('shares one application between retained Agent objects while isolating native handles and releasing each connection', async () => { const root = await mkdtemp(join(tmpdir(), 'dsh-native-composition-')) const context = new Context() onTestFinished(async () => { await context.fiber.dispose(); await rm(root, { recursive: true, force: true }) }) - const helperPath = join(root, 'native-helper.mjs') - await copyFile(new URL('../../../../snapshots/session/cua-native/workspace/native-helper.mjs', import.meta.url), helperPath) - await chmod(helperPath, 0o700) + const helperPath = await installNativeFixture(root) await writeFile(join(root, 'observe-helper'), '') const readEvents = async () => (await readFile(join(root, 'helper-events.jsonl'), 'utf8')).trim().split('\n').map(line => AuditEvent.parse(JSON.parse(line))) const modules = { @@ -148,7 +148,34 @@ it.skipIf(process.platform === 'win32')('loads the native provider, records veri } const directCalls = (await readEvents()).filter(event => event.event === 'call').slice(-3) expect(directCalls.map(event => event.requestId)).toEqual(['programmatic-window', 'activate-first', 'activate-repeat']) + + let objectCall = 0 + const objectApi = (owner: typeof agent) => createCuaApi(async (operation) => { + const result = await context.tools.execute({ agent: owner, signal: new AbortController().signal, + callId: ToolCallId(`retained-object-${++objectCall}`), name: 'cua', arguments: operation }) + if (result.isError) throw new Error(JSON.stringify(result.content)) + return z.object({ structuredContent: z.json() }).parse(result.value).structuredContent + }) + const retainedApp = await objectApi(agent).getApp('test.fixture') + const retainedWindow = await retainedApp.getWindow(7) + expect(await retainedWindow.getAXState({ emit: false })).toContain('Count: 1') + const peer = await context.agents.create({ sessionId: SessionId('native-loader-peer'), agentOptions: { provider: 'native-fixture', model: 'vision' } }) + const peerApp = await objectApi(peer.agent).getApp('test.fixture') + const peerWindow = await peerApp.getWindow(7) + expect(peerApp.appHandle).not.toBe(retainedApp.appHandle) + expect(Buffer.from(await peerWindow.getScreenshot({ emit: false })).toString('base64')).toBe(png) + await peerWindow.click(0) + expect(await retainedWindow.getAXState({ emit: false })).toContain('Count: 2') + const sharedEvents = await readEvents() + expect(sharedEvents.filter(event => event.event === 'start')).toHaveLength(1) + expect(sharedEvents.filter(event => event.event === 'connect')).toHaveLength(2) + expect(new Set(sharedEvents.map(event => event.pid)).size).toBe(1) await handle.dispose() + expect((await readEvents()).filter(event => event.event === 'exit')).toHaveLength(0) + expect((await readEvents()).filter(event => event.event === 'disconnect')).toHaveLength(1) + await peerWindow.click(0) + expect(await peerWindow.getAXState({ emit: false })).toContain('Count: 3') + await peer.dispose() expect((await readEvents()).filter(event => event.event === 'exit')).toHaveLength(1) const next = await context.agents.create({ sessionId: SessionId('native-loader-next'), agentOptions: { provider: 'native-fixture', model: 'vision' } }) diff --git a/packages/computer-use/cua/tests/native-client.spec.ts b/packages/computer-use/cua/tests/native-client.spec.ts index 2a95a73bb4..7bb8c698b5 100644 --- a/packages/computer-use/cua/tests/native-client.spec.ts +++ b/packages/computer-use/cua/tests/native-client.spec.ts @@ -1,47 +1,52 @@ -/** Controlled process streams exercise the platform-independent native protocol client. */ +/** Controlled connection streams exercise the platform-independent native protocol client. */ import { PassThrough, Writable } from 'node:stream' import { ToolCallId } from '@deepseek-ai/dsh-llm' -import type { SubprocessHandle, SubprocessOutcome, SubprocessOutputReader } from '@deepseek-ai/dsh-subprocess' +import type { NativeChannel } from '../src/shared-native.ts' import { describe, expect, it, onTestFinished, vi } from 'vitest' import { z } from 'zod' import { NativeClient, NativeError } from '../src/native-client.ts' -const Request = z.object({ id: z.string(), method: z.string(), params: z.record(z.string(), z.json()) }) +const shared = vi.hoisted(() => ({ create: vi.fn() })) +vi.mock('../src/shared-native.ts', () => ({ createSharedNativeChannel: shared.create })) + +const Request = z.object({ id: z.string(), method: z.string(), params: z.record(z.string(), z.json()) }).strict() +const Cancel = z.object({ type: z.literal('cancel'), id: z.string() }).strict() +const Frame = z.union([Request, Cancel]) function fixture(options: { maxRequestBytes?: number maxResponseBytes?: number timeoutMs?: number - stdin?: boolean - stdout?: boolean + screenshotTimeoutMs?: number + graceMs?: number writeError?: Error - stderr?: SubprocessOutputReader + cancelWriteError?: Error reportDiagnostic?: (message: string) => void } = {}) { const requests: z.infer[] = [] + const cancellations: z.infer[] = [] const stdout = new PassThrough() - const done = Promise.withResolvers() - const gone = Promise.withResolvers() + const done: PromiseWithResolvers = Promise.withResolvers() + const gone: PromiseWithResolvers = Promise.withResolvers() const stdin = new Writable({ write(chunk, _encoding, callback) { - requests.push(Request.parse(JSON.parse(String(chunk)))) - callback(options.writeError) + const frame = Frame.parse(JSON.parse(String(chunk))) + if ('type' in frame) { cancellations.push(frame); callback(options.cancelWriteError) } + else { requests.push(frame); callback(options.writeError) } } }) - const terminated: PromiseWithResolvers = Promise.withResolvers() - const terminate = vi.fn(() => { terminated.resolve() }) - const waitForExit = vi.fn(() => gone.promise) - const handle: SubprocessHandle = { - stdin: options.stdin === false ? undefined : stdin, - stdout: options.stdout === false ? undefined : stdout, - stderr: undefined, control: undefined, collected: options.stderr === undefined ? {} : { stderr: options.stderr }, - done: done.promise, terminate, waitForExit, - } - const spawn = vi.fn(() => handle) - const client = new NativeClient({ subprocess: { spawn } }, { + const closed: PromiseWithResolvers = Promise.withResolvers() + const close = vi.fn(() => { closed.resolve() }) + const waitForClose = vi.fn(() => gone.promise) + const channel: NativeChannel = { stdin, stdout, done: done.promise, close, waitForClose } + shared.create.mockClear().mockReturnValueOnce(channel) + const owner = { subprocess: { spawn: vi.fn() } } + const config = { helperPath: '/fixture/helper', cwd: '/fixture', axTimeoutMs: 321, timeoutMs: options.timeoutMs ?? 60_000, + connectTimeoutMs: 5000, screenshotTimeoutMs: options.screenshotTimeoutMs ?? 5000, maxRequestBytes: options.maxRequestBytes ?? 1024, - maxResponseBytes: options.maxResponseBytes ?? 1024, graceMs: 10, textEventIntervalMs: 0.5, - }, options.reportDiagnostic) - function finish() { done.resolve({ exitCode: 0, signal: null }); gone.resolve(true) } + maxResponseBytes: options.maxResponseBytes ?? 1024, graceMs: options.graceMs ?? 10, textEventIntervalMs: 0.5, + } + const client = new NativeClient(owner, config, options.reportDiagnostic) + function finish() { done.resolve(); gone.resolve() } function reply(value: unknown) { stdout.write(Buffer.from(JSON.stringify(value) + '\n')) } function id(index = requests.length - 1) { const request = requests[index] @@ -54,16 +59,14 @@ function fixture(options: { stdout.destroy() stdin.destroy() }) - return { client, requests, stdout, stdin, spawn, terminate, terminated: terminated.promise, waitForExit, done, gone, finish, reply, id } + return { client, requests, cancellations, stdout, stdin, owner, config, connect: shared.create, + close, closed: closed.promise, waitForClose, done, gone, finish, reply, id } } describe('native helper transport', () => { - it('passes host byte limits and deadlines into the helper process', () => { + it('passes transport limits and connection deadlines to the shared channel', () => { const f = fixture({ maxRequestBytes: 4096, maxResponseBytes: 8192, timeoutMs: 1234 }) - expect(f.spawn).toHaveBeenCalledWith(expect.objectContaining({ argv: [ - '/fixture/helper', '--max-request-bytes', '4096', '--max-response-bytes', '8192', - '--ax-timeout-ms', '321', '--shutdown-grace-ms', '10', '--text-event-interval-ms', '0.5', - ] })) + expect(f.connect).toHaveBeenCalledExactlyOnceWith(f.owner, f.config) }) it('uses the logged tool invocation ID for the native request and its matching reply', async () => { @@ -74,7 +77,7 @@ describe('native helper transport', () => { await expect(pending).resolves.toEqual({ dispatched: true, verified: false }) }) - it('accepts split UTF-8 replies and preserves native refusal codes without replacing the helper', async () => { + it('accepts split UTF-8 replies and preserves native refusal codes without closing the connection', async () => { const f = fixture() const first = f.client.request('listApps', {}, new AbortController().signal) const bytes = Buffer.from(JSON.stringify({ id: f.id(), result: [{ name: '编辑器' }] }) + '\n') @@ -85,14 +88,14 @@ describe('native helper transport', () => { f.reply({ id: f.id(), error: { code: 'STALE_SNAPSHOT', message: 'Refresh observation' } }) await rejected expect(f.client.disposed).toBe(false) - expect(f.terminate).not.toHaveBeenCalled() + expect(f.close).not.toHaveBeenCalled() const third = f.client.request('getCapabilities', {}, new AbortController().signal) f.reply({ id: f.id(), result: null }) expect(await third).toBeNull() - expect(f.spawn).toHaveBeenCalledTimes(1) + expect(f.connect).toHaveBeenCalledTimes(1) }) - it.each(['malformed', 'wrong-id', 'both', 'neither', 'empty-code', 'extra-field', 'invalid-utf8', 'extra-frame'] as const)('invalidates the helper on a %s response before releasing the caller', async (kind) => { + it.each(['malformed', 'wrong-id', 'both', 'neither', 'empty-code', 'extra-field', 'invalid-utf8', 'extra-frame', 'duplicate-reply'] as const)('invalidates the connection on a %s response before releasing the caller', async (kind) => { const f = fixture() const pending = f.client.request('listApps', {}, new AbortController().signal) const rejected = expect(pending).rejects.toMatchObject({ code: 'NATIVE_PROTOCOL' }) @@ -103,9 +106,10 @@ describe('native helper transport', () => { else if (kind === 'empty-code') f.reply({ id: f.id(), error: { code: '', message: 'bad' } }) else if (kind === 'extra-field') f.reply({ id: f.id(), result: [], extra: true }) else if (kind === 'invalid-utf8') f.stdout.write(Buffer.concat([Buffer.from(`{"id":"${f.id()}","result":"`), Buffer.from([0xff]), Buffer.from('"}\n')])) - else f.stdout.write(JSON.stringify({ id: f.id(), result: [] }) + '\n{}\n') + else if (kind === 'extra-frame') f.stdout.write(JSON.stringify({ id: f.id(), result: [] }) + '\n{}\n') + else f.stdout.write((JSON.stringify({ id: f.id(), result: [] }) + '\n').repeat(2)) expect(f.client.disposed).toBe(true) - expect(f.terminate).toHaveBeenCalledTimes(1) + expect(f.close).toHaveBeenCalledTimes(1) f.finish() await rejected await expect(f.client.request('listApps', {}, new AbortController().signal)).rejects.toMatchObject({ code: 'NATIVE_CLOSED' }) @@ -160,7 +164,7 @@ describe('native helper transport', () => { await expect(f.client.request('typeText', { text: text.slice(1) + 'é' }, new AbortController().signal)).rejects.toMatchObject({ code: 'NATIVE_REQUEST_LIMIT' }) expect(f.requests).toHaveLength(1) expect(f.client.disposed).toBe(false) - expect(f.terminate).not.toHaveBeenCalled() + expect(f.close).not.toHaveBeenCalled() const next = f.client.request('listApps', {}, new AbortController().signal) f.reply({ id: f.id(), result: [] }) await expect(next).resolves.toEqual([]) @@ -174,36 +178,40 @@ describe('native helper transport', () => { f.stdout.write('late output') f.finish() await rejected - expect(f.terminate).toHaveBeenCalledTimes(1) + expect(f.close).toHaveBeenCalledTimes(1) }) - it('preserves a producer response-limit diagnostic and awaits helper invalidation', async () => { + it('preserves a producer response-limit diagnostic and awaits connection invalidation', async () => { const f = fixture() const pending = f.client.request('screenshot', {}, new AbortController().signal) const rejected = expect(pending).rejects.toMatchObject({ code: 'NATIVE_RESPONSE_LIMIT' }) f.reply({ id: f.id(), error: { code: 'NATIVE_RESPONSE_LIMIT', message: 'Reply exceeded helper byte limit' } }) expect(f.client.disposed).toBe(true) - expect(f.terminate).toHaveBeenCalledTimes(1) + expect(f.close).toHaveBeenCalledTimes(1) f.finish() await rejected }) - it('cancellation waits for the complete managed process range', async () => { + it('waits for cancellation acknowledgement and keeps acquired handles on the same connection', async () => { const f = fixture() const abort = new AbortController() - const pending = f.client.request('getState', {}, abort.signal) + const pending = f.client.request('screenshot', { appHandle: 'retained-app' }, abort.signal) let settled = false const observed = pending.then(() => { settled = true }, (error: unknown) => { settled = true; expect(error).toMatchObject({ code: 'NATIVE_CANCELLED' }) }) abort.abort() - f.done.resolve({ exitCode: 0, signal: null }) await Promise.resolve() - await Promise.resolve() - expect(f.terminate).toHaveBeenCalledTimes(1) + expect(f.cancellations).toEqual([{ type: 'cancel', id: f.id() }]) + expect(f.close).not.toHaveBeenCalled() expect(settled).toBe(false) - f.gone.resolve(true) + await expect(f.client.request('getState', {}, new AbortController().signal)).rejects.toMatchObject({ code: 'NATIVE_BUSY' }) + f.reply({ id: f.id(), error: { code: 'NATIVE_CANCELLED', message: 'Capture stopped' } }) await observed - expect(f.stdout.destroyed).toBe(true) - expect(f.stdin.destroyed).toBe(true) + expect(f.client.disposed).toBe(false) + const next = f.client.request('getState', { appHandle: 'retained-app' }, new AbortController().signal) + f.reply({ id: f.id(), result: { snapshotId: 'fresh' } }) + await expect(next).resolves.toEqual({ snapshotId: 'fresh' }) + expect(f.connect).toHaveBeenCalledTimes(1) + expect(f.stdout.destroyed).toBe(false) }) it('rejects a pre-cancelled operation without writing or invalidating acquired handles', async () => { @@ -222,32 +230,93 @@ describe('native helper transport', () => { expect(await first).toEqual({ snapshotId: 'one' }) }) - it('cancellation after a reply but before result delivery still invalidates the helper', async () => { + it('preserves a completed reply when cancellation arrives before its promise continuation', async () => { const f = fixture() const abort = new AbortController() const pending = f.client.request('click', {}, abort.signal) - const rejected = expect(pending).rejects.toMatchObject({ code: 'NATIVE_CANCELLED' }) f.reply({ id: f.id(), result: { dispatched: true } }) abort.abort() - f.finish() - await rejected + await expect(pending).resolves.toEqual({ dispatched: true }) + expect(f.cancellations).toEqual([]) + expect(f.client.disposed).toBe(false) }) - it('expires the configured operation deadline and waits for shutdown', async () => { + it('accepts a successful reply as cancellation acknowledgement when the reply was already in flight', async () => { + const f = fixture() + const abort = new AbortController() + const pending = f.client.request('screenshot', {}, abort.signal) + const rejected = expect(pending).rejects.toMatchObject({ code: 'NATIVE_CANCELLED' }) + abort.abort() + f.reply({ id: f.id(), result: { data: 'late-image' } }) + await rejected + expect(f.client.disposed).toBe(false) + expect(f.close).not.toHaveBeenCalled() + }) + + it('expires the operation deadline without closing an acknowledged connection', async () => { vi.useFakeTimers() try { const f = fixture({ timeoutMs: 25 }) const pending = f.client.request('getState', {}, new AbortController().signal) const rejected = expect(pending).rejects.toMatchObject({ code: 'NATIVE_TIMEOUT' }) await vi.advanceTimersByTimeAsync(25) - expect(f.client.disposed).toBe(true) - f.finish() + expect(f.cancellations).toEqual([{ type: 'cancel', id: f.id() }]) + expect(f.client.disposed).toBe(false) + f.reply({ id: f.id(), error: { code: 'NATIVE_CANCELLED', message: 'Request cancelled' } }) await rejected + expect(f.close).not.toHaveBeenCalled() expect(vi.getTimerCount()).toBe(0) } finally { vi.useRealTimers() } }) - it.each(['stdout-error', 'stdin-error', 'stdout-end', 'stdout-close', 'process-exit', 'process-rejection'] as const)('contains %s while an operation is pending', async (kind) => { + it('closes only after the cancellation grace expires without an acknowledgement', async () => { + vi.useFakeTimers() + try { + const f = fixture({ timeoutMs: 25, graceMs: 10 }) + const pending = f.client.request('screenshot', {}, new AbortController().signal) + const rejected = expect(pending).rejects.toMatchObject({ code: 'NATIVE_CLEANUP', cause: { code: 'NATIVE_TIMEOUT' } }) + await vi.advanceTimersByTimeAsync(25) + expect(f.close).not.toHaveBeenCalled() + await vi.advanceTimersByTimeAsync(10) + expect(f.close).toHaveBeenCalledTimes(1) + f.finish() + await rejected + expect(f.client.disposed).toBe(true) + expect(vi.getTimerCount()).toBe(0) + } finally { vi.useRealTimers() } + }) + + it('sends cancellation once and preserves the first interruption reason', async () => { + vi.useFakeTimers() + try { + const f = fixture({ timeoutMs: 25 }) + const abort = new AbortController() + const pending = f.client.request('screenshot', {}, abort.signal) + const rejected = expect(pending).rejects.toMatchObject({ code: 'NATIVE_TIMEOUT' }) + await vi.advanceTimersByTimeAsync(25) + abort.abort() + expect(f.cancellations).toEqual([{ type: 'cancel', id: f.id() }]) + f.reply({ id: f.id(), result: { data: 'late-image' } }) + await rejected + expect(f.client.disposed).toBe(false) + expect(vi.getTimerCount()).toBe(0) + } finally { vi.useRealTimers() } + }) + + it.each(['throw', 'callback'] as const)('closes the connection when sending cancellation fails through %s', async (kind) => { + const failure = new Error('cancel write failed') + const f = fixture(kind === 'callback' ? { cancelWriteError: failure } : {}) + const abort = new AbortController() + const pending = f.client.request('screenshot', {}, abort.signal) + const rejected = expect(pending).rejects.toMatchObject({ code: 'NATIVE_IO', cause: failure }) + if (kind === 'throw') vi.spyOn(f.stdin, 'write').mockImplementation(() => { throw failure }) + abort.abort() + await f.closed + f.finish() + await rejected + }) + + it.each(['stdout-error', 'stdin-error', 'stdout-end', 'stdout-close', 'connection-exit', 'connection-rejection'] as const)('contains %s while an operation is pending', async (kind) => { const f = fixture() const pending = f.client.request('getState', {}, new AbortController().signal) const code = kind.endsWith('-error') ? 'NATIVE_IO' : kind.startsWith('stdout-') ? 'NATIVE_EOF' : 'NATIVE_EXITED' @@ -256,12 +325,12 @@ describe('native helper transport', () => { else if (kind === 'stdin-error') f.stdin.emit('error', new Error('write failed')) else if (kind === 'stdout-end') f.stdout.emit('end') else if (kind === 'stdout-close') f.stdout.emit('close') - else if (kind === 'process-exit') f.done.resolve({ exitCode: 1, signal: null }) - else f.done.reject(new Error('spawn failed')) - f.gone.resolve(true) - if (!kind.startsWith('process-')) f.done.resolve({ exitCode: 1, signal: null }) + else if (kind === 'connection-exit') f.done.resolve() + else f.done.reject(new Error('connection failed')) + f.gone.resolve() + if (!kind.startsWith('connection-')) f.done.resolve() await rejected - expect(f.terminate).toHaveBeenCalledTimes(1) + expect(f.close).toHaveBeenCalledTimes(1) }) it.each(['throw', 'callback'] as const)('invalidates uncertain dispatch after a write %s failure', async (kind) => { @@ -270,20 +339,11 @@ describe('native helper transport', () => { if (kind === 'throw') vi.spyOn(f.stdin, 'write').mockImplementation(() => { throw failure }) const pending = f.client.request('click', {}, new AbortController().signal) const rejected = expect(pending).rejects.toMatchObject({ code: 'NATIVE_IO', cause: failure }) - await f.terminated + await f.closed f.finish() await rejected expect(f.client.disposed).toBe(true) - expect(f.terminate).toHaveBeenCalledTimes(1) - }) - - it.each([{ stdin: false }, { stdout: false }, { stdin: false, stdout: false }])('cleans up an incomplete process handle %j', async (options) => { - const f = fixture(options) - const pending = f.client.request('listApps', {}, new AbortController().signal) - const rejected = expect(pending).rejects.toMatchObject({ code: 'NATIVE_IO' }) - f.finish() - await rejected - expect(f.requests).toEqual([]) + expect(f.close).toHaveBeenCalledTimes(1) }) it('rejects unsolicited replies and shares one quiescent disposal across callers', async () => { @@ -292,28 +352,24 @@ describe('native helper transport', () => { const first = f.client.dispose() const second = f.client.dispose() expect(second).toBe(first) - expect(f.waitForExit).toHaveBeenCalledTimes(1) + expect(f.waitForClose).toHaveBeenCalledTimes(1) f.finish() await first expect(() => { f.stdout.emit('error', new Error('late read error')) f.stdin.emit('error', new Error('late write error')) }).not.toThrow() - expect(f.terminate).toHaveBeenCalledTimes(1) + expect(f.close).toHaveBeenCalledTimes(1) }) - it('rejects pending work on disposal and releases streams even when range observation fails', async () => { + it('rejects pending work on disposal and releases streams even when connection cleanup fails', async () => { const f = fixture() const pending = f.client.request('getState', {}, new AbortController().signal) - const failure = new Error('cannot observe range') + const failure = new Error('cannot close connection') const rejected = expect(pending).rejects.toBe(failure) const disposing = f.client.dispose() const disposalRejected = expect(disposing).rejects.toBe(failure) f.gone.reject(failure) - await Promise.resolve() - await Promise.resolve() - expect(f.stdout.destroyed).toBe(false) - f.done.resolve({ exitCode: 0, signal: null }) await disposalRejected await rejected expect(f.stdout.destroyed).toBe(true) @@ -326,55 +382,87 @@ describe('native helper transport', () => { expect(error.message).toBe('PERMISSION_DENIED: Enable Accessibility') }) - it('retains the last screenshot phase after timeout and helper cleanup', async () => { + it('uses the shorter screenshot deadline while leaving ordinary operations their configured time', async () => { + vi.useFakeTimers() + try { + const f = fixture({ timeoutMs: 100, screenshotTimeoutMs: 25 }) + const ordinary = f.client.request('getState', {}, new AbortController().signal) + await vi.advanceTimersByTimeAsync(25) + expect(f.client.disposed).toBe(false) + f.reply({ id: f.id(), result: {} }) + await ordinary + const screenshot = f.client.request('screenshot', {}, new AbortController().signal) + const rejected = expect(screenshot).rejects.toMatchObject({ code: 'NATIVE_TIMEOUT', message: 'NATIVE_TIMEOUT: Operation exceeded 25ms' }) + await vi.advanceTimersByTimeAsync(25) + f.reply({ id: f.id(), error: { code: 'NATIVE_CANCELLED', message: 'Capture stopped' } }) + await rejected + expect(f.client.disposed).toBe(false) + const next = f.client.request('getState', { appHandle: 'retained-app' }, new AbortController().signal) + f.reply({ id: f.id(), result: { snapshotId: 'after-timeout' } }) + await expect(next).resolves.toEqual({ snapshotId: 'after-timeout' }) + expect(f.close).not.toHaveBeenCalled() + expect(vi.getTimerCount()).toBe(0) + } finally { vi.useRealTimers() } + }) + + it('retains the last screenshot phase after timeout and request cancellation', async () => { vi.useFakeTimers() try { - let text = '' const reportDiagnostic = vi.fn() - const f = fixture({ timeoutMs: 25, reportDiagnostic, - stderr: { readFrom: offset => ({ text: text.slice(offset), nextOffset: text.length, lossy: false }) }, - }) + const f = fixture({ timeoutMs: 25, reportDiagnostic }) const pending = f.client.request('screenshot', {}, new AbortController().signal, ToolCallId('capture-timeout')) const rejected = expect(pending).rejects.toMatchObject({ code: 'NATIVE_TIMEOUT', message: expect.stringContaining('"stage":"capture-image.begin"'), }) + f.reply({ type: 'diagnostic', requestId: 'capture-timeout', stage: 'capture-image.begin', elapsedMs: 2 }) await vi.advanceTimersByTimeAsync(25) - text = '[cua-screenshot] {"requestId":"capture-timeout","stage":"capture-image.begin","elapsedMs":2}\n' - f.finish() + f.reply({ id: f.id(), error: { code: 'NATIVE_CANCELLED', message: 'Capture stopped' } }) await rejected + expect(f.client.disposed).toBe(false) expect(reportDiagnostic.mock.calls.flat().join('\n')).toContain('host.settled') expect(vi.getTimerCount()).toBe(0) } finally { vi.useRealTimers() } }) - it('filters screenshot diagnostics by request and omits unrelated stderr and extra fields', async () => { - let text = 'earlier stderr\n' - const readFrom = vi.fn((offset: number) => ({ text: text.slice(offset), nextOffset: text.length, lossy: false })) - const f = fixture({ stderr: { readFrom } }) + it('accepts diagnostic and result frames in one chunk and omits unrelated requests and extra fields', async () => { + const f = fixture() const pending = f.client.request('screenshot', {}, new AbortController().signal, ToolCallId('current')) - text += 'private window title\n[cua-screenshot] incomplete\n' - + '[cua-screenshot] {"requestId":"previous","stage":"old.begin","elapsedMs":1}\n' - + '[cua-screenshot] {"requestId":"current","stage":"bad","elapsedMs":"invalid"}\n' - + '[cua-screenshot] {"requestId":"current","stage":"ax-geometry.begin","elapsedMs":3,"title":"private"}\n' - f.reply({ id: 'current', error: { code: 'WINDOW_CHANGED', message: 'Window changed' } }) + f.stdout.write([ + { type: 'diagnostic', requestId: 'previous', stage: 'old.begin', elapsedMs: 1 }, + { type: 'diagnostic', requestId: 'current', stage: 'ax-geometry.begin', elapsedMs: 3, title: 'private' }, + { id: 'current', error: { code: 'WINDOW_CHANGED', message: 'Window changed' } }, + ].map(frame => JSON.stringify(frame) + '\n').join('')) await expect(pending).rejects.toMatchObject({ code: 'WINDOW_CHANGED', message: 'WINDOW_CHANGED: Window changed\nScreenshot diagnostics:\n' + '[cua-screenshot] {"requestId":"current","stage":"ax-geometry.begin","elapsedMs":3}', }) - expect(readFrom).toHaveBeenLastCalledWith('earlier stderr\n'.length) expect(f.client.disposed).toBe(false) }) - it('marks truncated screenshot diagnostics and keeps successful results out of model text', async () => { + it('rejects malformed diagnostic frames through the same protocol failure path', async () => { + const f = fixture() + const pending = f.client.request('screenshot', {}, new AbortController().signal) + const rejected = expect(pending).rejects.toMatchObject({ code: 'NATIVE_PROTOCOL' }) + f.reply({ type: 'diagnostic', requestId: f.id(), stage: 'capture-image.begin', elapsedMs: 'invalid' }) + f.finish() + await rejected + expect(f.close).toHaveBeenCalledTimes(1) + }) + + it('bounds retained screenshot diagnostics and keeps successful results out of model text', async () => { const reportDiagnostic = vi.fn() - const readFrom = vi.fn() - .mockReturnValueOnce({ text: '', nextOffset: 10, lossy: false }) - .mockReturnValue({ text: '[cua-screenshot] {"requestId":"capture","stage":"png-encode.end","elapsedMs":4}\n', nextOffset: 200, lossy: true }) - const f = fixture({ reportDiagnostic, stderr: { readFrom } }) + const f = fixture({ reportDiagnostic }) const pending = f.client.request('screenshot', {}, new AbortController().signal, ToolCallId('capture')) + for (let index = 0; index < 100; index++) { + f.reply({ type: 'diagnostic', requestId: 'capture', stage: `phase-${index}`, elapsedMs: index }) + } f.reply({ id: 'capture', result: { data: 'image' } }) await expect(pending).resolves.toEqual({ data: 'image' }) - expect(reportDiagnostic.mock.calls.flat().join('\n')).toContain('Earlier screenshot diagnostic bytes were discarded') + const retained = reportDiagnostic.mock.calls.map(call => call[0]).find(value => value.includes('discarded')) + expect(retained).toContain('Earlier screenshot diagnostic records were discarded') + expect(retained).not.toContain('"stage":"phase-0"') + expect(retained).toContain('"stage":"phase-99"') + expect(Buffer.byteLength(retained!.split('\n').slice(1).join(''))).toBeLessThanOrEqual(8192) expect(reportDiagnostic.mock.calls.flat().join('\n')).toContain('host.reply') }) diff --git a/packages/computer-use/cua/tests/native-fixture.ts b/packages/computer-use/cua/tests/native-fixture.ts new file mode 100644 index 0000000000..3dd1072fd0 --- /dev/null +++ b/packages/computer-use/cua/tests/native-fixture.ts @@ -0,0 +1,15 @@ +/** Per-test executable identity for the shared native protocol fixture. */ +import { readFile, writeFile } from 'node:fs/promises' +import { join } from 'node:path' + +/** + * Copy the native fixture with an identity unique to the caller's temporary directory. + * @param root - Existing test-owned temporary directory. + * @returns Executable path for the provider configuration. + */ +export async function installNativeFixture(root: string): Promise { + const helperPath = join(root, 'native-helper.mjs') + const source = await readFile(new URL('../../../../snapshots/session/cua-native/workspace/native-helper.mjs', import.meta.url), 'utf8') + await writeFile(helperPath, `${source}\n// Fixture directory: ${JSON.stringify(root)}\n`, { mode: 0o700 }) + return helperPath +} diff --git a/packages/computer-use/cua/tests/persistent-bridge.built.e2e.ts b/packages/computer-use/cua/tests/persistent-bridge.built.e2e.ts index 1a8bf2a777..e291b9cb86 100644 --- a/packages/computer-use/cua/tests/persistent-bridge.built.e2e.ts +++ b/packages/computer-use/cua/tests/persistent-bridge.built.e2e.ts @@ -1,6 +1,6 @@ /** Built bridge imports and retained objects from a project outside the Harness installation. */ import { existsSync } from 'node:fs' -import { chmod, copyFile, mkdtemp, rm } from 'node:fs/promises' +import { mkdtemp, rm } from 'node:fs/promises' import { tmpdir } from 'node:os' import { dirname, join } from 'node:path' import { fileURLToPath } from 'node:url' @@ -18,6 +18,7 @@ import { SessionId } from '@deepseek-ai/dsh-session' import Subprocess from '@deepseek-ai/dsh-subprocess-local' import { expect, it } from 'vitest' import * as Cua from '../src/index.ts' +import { installNativeFixture } from './native-fixture.ts' import { Request } from '../src/protocol.ts' const bridge = fileURLToPath(new URL('../lib/bridge.js', import.meta.url)) @@ -36,9 +37,7 @@ it.skipIf(process.platform === 'win32' || !existsSync(bridge) || !existsSync(boo const cwd = await mkdtemp(join(tmpdir(), 'dsh-cua-external-project-')) const ctx = new Context() try { - const helperPath = join(cwd, 'native-helper.mjs') - await copyFile(new URL('../../../../snapshots/session/cua-native/workspace/native-helper.mjs', import.meta.url), helperPath) - await chmod(helperPath, 0o700) + const helperPath = await installNativeFixture(cwd) await mountAgentLoopTestDependencies(ctx) await ctx.plugin(FileSystem) await ctx.plugin(Subprocess) diff --git a/packages/computer-use/cua/tests/provider-lifecycle.spec.ts b/packages/computer-use/cua/tests/provider-lifecycle.spec.ts index 4d23495e2d..dec37bd68b 100644 --- a/packages/computer-use/cua/tests/provider-lifecycle.spec.ts +++ b/packages/computer-use/cua/tests/provider-lifecycle.spec.ts @@ -49,7 +49,7 @@ vi.mock('../src/native-client.ts', async importOriginal => ({ })) const config = { - helperPath: resolve('fixture-helper'), buildTimeoutMs: 300_000, timeoutMs: 10_000, axTimeoutMs: 3000, maxRequestBytes: 1024, maxResponseBytes: 4096, maxClipboardBytes: 8192, maxAppMatches: 20, + helperPath: resolve('fixture-helper'), buildTimeoutMs: 300_000, timeoutMs: 10_000, connectTimeoutMs: 5000, screenshotTimeoutMs: 5000, axTimeoutMs: 3000, maxRequestBytes: 1024, maxResponseBytes: 4096, maxClipboardBytes: 8192, maxAppMatches: 20, graceMs: 10, textEventIntervalMs: 8, maxElements: 64, maxDepth: 8, dragDurationMs: 200, dragSteps: 10, } satisfies Omit diff --git a/packages/computer-use/cua/tests/shared-native.spec.ts b/packages/computer-use/cua/tests/shared-native.spec.ts new file mode 100644 index 0000000000..38160e92e9 --- /dev/null +++ b/packages/computer-use/cua/tests/shared-native.spec.ts @@ -0,0 +1,442 @@ +/** Real Unix sockets exercise connection isolation and native startup ownership. */ +import { createHash, randomUUID } from 'node:crypto' +import { chmod, mkdir, mkdtemp, realpath, rm, symlink, writeFile } from 'node:fs/promises' +import { createServer, Socket } from 'node:net' +import { userInfo } from 'node:os' +import { join } from 'node:path' +import type { SubprocessHandle, SubprocessOutcome, SubprocessRuntime } from '@deepseek-ai/dsh-subprocess' +import { describe, expect, it, onTestFinished, vi } from 'vitest' +import type { NativeClientConfig } from '../src/native-client.ts' +import { createSharedNativeChannel, type NativeChannel } from '../src/shared-native.ts' + +const runtime = vi.hoisted(() => ({ source: '', directory: '', foreignOwner: false })) +vi.mock('node:fs/promises', async (original) => { + const actual = await original() + return { + ...actual, + mkdir: (path: string, options: import('node:fs').MakeDirectoryOptions) => actual.mkdir(path === runtime.source ? runtime.directory : path, options), + lstat: async (path: string) => { + const result = await actual.lstat(path === runtime.source ? runtime.directory : path) + if (path === runtime.source && runtime.foreignOwner) result.uid += 1 + return result + }, + } +}) +vi.mock('node:path', async (original) => { + const actual = await original() + return { ...actual, join: (...paths: string[]) => actual.join(...paths.map(path => path === runtime.source ? runtime.directory : path)) } +}) + +type Peer = (socket: Socket, ready: Buffer) => void + +function launcherFixture(): SubprocessHandle { + return { + stdin: undefined, stdout: undefined, stderr: undefined, control: undefined, collected: {}, + done: Promise.resolve({ exitCode: 0, signal: null }), waitForExit: vi.fn(async () => true), terminate: vi.fn(), + } +} + +async function fixture(options: { + ready?: Peer + close?: (socket: Socket) => void + root?: 'missing' | 'symlink' | 'file' | 'public' | 'foreign' + launch?: () => SubprocessHandle + connectTimeoutMs?: number + graceMs?: number + listen?: boolean +} = {}) { + const root = await mkdtemp('/tmp/cua-') + const sockets = new Set() + const channels: NativeChannel[] = [] + const server = createServer({ allowHalfOpen: true }) + runtime.source = `/tmp/dsh-cua-${userInfo().uid}` + runtime.directory = join(root, 'r') + runtime.foreignOwner = options.root === 'foreign' + onTestFinished(async () => { + for (const channel of channels) channel.close() + for (const socket of sockets) socket.destroy() + await Promise.all(channels.map(channel => channel.waitForClose().catch(() => {}))) + await new Promise((resolve) => { server.close(() => resolve()) }) + runtime.source = '' + runtime.directory = '' + runtime.foreignOwner = false + await rm(root, { recursive: true, force: true }) + }) + if (options.root === 'symlink') { + const target = join(root, 'target') + await mkdir(target, { mode: 0o700 }) + await symlink(target, runtime.directory) + } else if (options.root === 'file') await writeFile(runtime.directory, 'not a directory') + else if (options.root !== 'missing') await mkdir(runtime.directory, { mode: 0o700 }) + if (options.root === 'public') await chmod(runtime.directory, 0o755) + const helperPath = join(root, 'helper') + const binary = randomUUID() + await writeFile(helperPath, binary) + const runtimeKey = createHash('sha256').update(binary).digest('hex') + const socketPath = join(runtime.directory, `${runtimeKey}.sock`) + const hellos: unknown[] = [] + const operations: string[] = [] + const helloWaiters: (() => void)[] = [] + server.on('connection', (socket) => { + sockets.add(socket) + socket.on('error', () => {}) + socket.on('close', () => { sockets.delete(socket) }) + socket.on('end', () => { + if (options.close) options.close(socket) + else socket.end() + }) + let pending = '' + let hello = true + let sequence = 0 + socket.on('data', (chunk) => { + pending += String(chunk) + for (;;) { + const newline = pending.indexOf('\n') + if (newline < 0) return + const line = pending.slice(0, newline) + pending = pending.slice(newline + 1) + if (hello) { + hello = false + hellos.push(JSON.parse(line)) + for (const waiter of helloWaiters.splice(0)) waiter() + const ready = Buffer.from(JSON.stringify({ type: 'ready', protocol: 1, runtimeKey, pid: process.pid, instanceId: randomUUID() }) + '\n') + if (options.ready) options.ready(socket, ready) + else socket.write(ready) + } else { + operations.push(line) + sequence += 1 + socket.write(JSON.stringify({ sequence, request: line }) + '\n') + } + } + }) + }) + async function listen(): Promise { + await new Promise((resolve, reject) => { + server.once('error', reject) + server.listen(socketPath, () => { server.removeListener('error', reject); resolve() }) + }) + } + if (options.listen !== false && !['file', 'missing'].includes(options.root ?? '')) await listen() + const handles: SubprocessHandle[] = [] + const spawned: PromiseWithResolvers = Promise.withResolvers() + const spawn = vi.fn(() => { + const handle = options.launch ? options.launch() : launcherFixture() + handles.push(handle) + spawned.resolve() + return handle + }) + const config: NativeClientConfig = { + helperPath, cwd: root, connectTimeoutMs: options.connectTimeoutMs ?? 5000, screenshotTimeoutMs: 5000, + timeoutMs: 30_000, graceMs: options.graceMs ?? 1000, axTimeoutMs: 3000, + maxRequestBytes: 65536, maxResponseBytes: 1048576, textEventIntervalMs: 0.5, + } + function open(): NativeChannel { + const channel = createSharedNativeChannel({ subprocess: { spawn } }, config) + channels.push(channel) + return channel + } + async function waitHello(count: number): Promise { + while (hellos.length < count) await new Promise((resolve) => { helloWaiters.push(resolve) }) + } + return { open, config, spawn, spawned: spawned.promise, handles, hellos, waitHello, operations, sockets, socketPath, listen } +} + +function write(channel: NativeChannel, text: string): Promise { + return new Promise((resolve, reject) => { channel.stdin.write(text, (error) => { if (error) reject(error); else resolve() }) }) +} + +function nextBytes(channel: NativeChannel, lines = 1): Promise { + return new Promise((resolve, reject) => { + const chunks: Buffer[] = [] + let receivedLines = 0 + const error = (value: Error): void => { channel.stdout.removeListener('data', data); reject(value) } + const data = (value: Buffer): void => { + chunks.push(value) + for (const byte of value) if (byte === 10) receivedLines += 1 + if (receivedLines < lines) return + channel.stdout.removeListener('data', data) + channel.stdout.removeListener('error', error) + resolve(Buffer.concat(chunks)) + } + channel.stdout.on('data', data) + channel.stdout.once('error', error) + }) +} + +describe.skipIf(process.platform === 'win32')('shared native Unix connection', () => { + it('keeps concurrent Agent scopes independent and leaves other sockets alive when one closes', async () => { + const f = await fixture() + const first = f.open() + const second = f.open() + const firstReply = nextBytes(first) + const secondReply = nextBytes(second) + await Promise.all([write(first, 'one\n'), write(second, 'two\n')]) + expect(JSON.parse(String(await firstReply))).toEqual({ sequence: 1, request: 'one' }) + expect(JSON.parse(String(await secondReply))).toEqual({ sequence: 1, request: 'two' }) + first.close() + await first.waitForClose() + const remaining = nextBytes(second) + await write(second, 'three\n') + expect(JSON.parse(String(await remaining))).toEqual({ sequence: 2, request: 'three' }) + expect(f.spawn).toHaveBeenCalledTimes(2) + for (const handle of f.handles) expect(handle.terminate).not.toHaveBeenCalled() + expect(f.hellos).toEqual([expect.objectContaining({ type: 'hello', protocol: 1, config: { + axTimeoutMs: 3000, timeoutMs: 30000, screenshotTimeoutMs: 5000, graceMs: 1000, + maxRequestBytes: 65536, maxResponseBytes: 1048576, textEventIntervalMs: 0.5, + } }), expect.anything()]) + }) + + it('waits for both launcher outcome and managed range release before sending any bytes', async () => { + const outcome = Promise.withResolvers() + const range = Promise.withResolvers() + const f = await fixture({ launch: () => ({ ...launcherFixture(), done: outcome.promise, waitForExit: () => range.promise }) }) + const channel = f.open() + const reply = nextBytes(channel) + const pending = write(channel, 'after-ready\n') + await f.spawned + outcome.resolve({ exitCode: 0, signal: null }) + expect(f.hellos).toHaveLength(0) + expect(f.operations).toHaveLength(0) + range.resolve(true) + await pending + expect(String(await reply)).toContain('after-ready') + expect(f.spawn.mock.calls[0]?.[0].argv).toEqual([ + await realpath(f.config.helperPath), '--ensure-shared', '--runtime-key', expect.stringMatching(/^[0-9a-f]{64}$/), + '--runtime-directory', runtime.source, '--startup-timeout-ms', '5000', + ]) + }) + + it('removes the ready frame while preserving response bytes from its final chunk', async () => { + const result = Buffer.from('{"diagnostic":{"stage":"hello"}}\n{"id":"one","result":true}\n') + const f = await fixture({ ready: (socket, ready) => { + socket.write(ready.subarray(0, 7), () => { + setImmediate(() => socket.write(Buffer.concat([ready.subarray(7), result]))) + }) + } }) + const channel = f.open() + expect(await nextBytes(channel, 2)).toEqual(result) + channel.close() + await channel.waitForClose() + }) + + it.each(['protocol', 'identity', 'json', 'utf8', 'oversized', 'refusal', 'refusal-retry-code'] as const)('rejects %s handshakes before writing operations', async (kind) => { + const f = await fixture({ ready: (socket, ready) => { + if (kind === 'protocol') socket.write(String(ready).replace('"protocol":1', '"protocol":2')) + else if (kind === 'identity') socket.write(String(ready).replace(/"runtimeKey":"[^"]*"/, '"runtimeKey":"other"')) + else if (kind === 'json') socket.write('{broken}\n') + else if (kind === 'utf8') socket.write(Buffer.from([0xff, 10])) + else if (kind === 'oversized') socket.write('x'.repeat(8193)) + else if (kind === 'refusal') socket.write('{"type":"error","code":"CONFIG_REFUSED","message":"invalid limits"}\n') + else socket.write('{"type":"error","code":"NATIVE_EOF","message":"explicit refusal"}\n') + } }) + const channel = f.open() + const failedWrite = expect(write(channel, 'never\n')).rejects.toBeInstanceOf(Error) + await expect(channel.done).rejects.toMatchObject({ code: kind === 'refusal' ? 'CONFIG_REFUSED' : kind === 'refusal-retry-code' ? 'NATIVE_EOF' : 'NATIVE_PROTOCOL' }) + await failedWrite + await channel.waitForClose() + expect(f.operations).toHaveLength(0) + expect(f.spawn).toHaveBeenCalledTimes(1) + }) + + it.each(['symlink', 'file', 'public', 'foreign'] as const)('rejects a %s runtime directory without launching native code', async (root) => { + const f = await fixture({ root, listen: false }) + const channel = f.open() + await expect(channel.done).rejects.toMatchObject({ code: 'NATIVE_RUNTIME_DIRECTORY' }) + await channel.waitForClose() + expect(f.spawn).not.toHaveBeenCalled() + }) + + it('creates a private runtime directory before starting the launcher', async () => { + const outcome = Promise.withResolvers() + const f = await fixture({ root: 'missing', launch: () => ({ ...launcherFixture(), done: outcome.promise }) }) + const channel = f.open() + const reply = nextBytes(channel) + const pending = write(channel, 'created\n') + await f.spawned + await f.listen() + outcome.resolve({ exitCode: 0, signal: null }) + await pending + expect(String(await reply)).toContain('created') + }) + + it('returns a failed launcher outcome with its retained stderr and releases the managed range', async () => { + const f = await fixture({ launch: () => ({ ...launcherFixture(), + done: Promise.resolve({ exitCode: 12, signal: null }), + collected: { stderr: { readFrom: () => ({ text: 'launcher refused private root', nextOffset: 29, lossy: false }) } }, + }) }) + const channel = f.open() + await expect(channel.done).rejects.toThrow('launcher refused private root') + await channel.waitForClose() + expect(f.hellos).toHaveLength(0) + }) + + it.each(['unobservable', 'not-empty'] as const)('reports %s startup ranges as cleanup failures', async (kind) => { + const failure = new Error('native range is no longer observable') + const f = await fixture({ launch: () => ({ ...launcherFixture(), waitForExit: async () => { + if (kind === 'unobservable') throw failure + return false + } }) }) + const channel = f.open() + await expect(channel.done).rejects.toThrow(kind === 'unobservable' ? failure.message : 'process range did not close') + await expect(channel.waitForClose()).rejects.toThrow(kind === 'unobservable' ? failure.message : 'process range did not close') + expect(f.hellos).toHaveLength(0) + }) + + it('preserves spawn failures after an empty managed range has settled', async () => { + const failure = new Error('native executable failed to start') + const f = await fixture({ launch: () => ({ ...launcherFixture(), done: Promise.reject(failure) }) }) + const channel = f.open() + await expect(channel.done).rejects.toBe(failure) + await channel.waitForClose() + expect(f.hellos).toHaveLength(0) + }) + + it('bounds a peer that accepts the socket but never completes its handshake', async () => { + const f = await fixture({ ready: () => {} }) + vi.useFakeTimers() + try { + const channel = f.open() + await f.waitHello(1) + await vi.advanceTimersByTimeAsync(f.config.connectTimeoutMs) + await expect(channel.done).rejects.toMatchObject({ code: 'NATIVE_CONNECT_TIMEOUT' }) + await channel.waitForClose() + expect(f.operations).toHaveLength(0) + } finally { vi.useRealTimers() } + }) + + it('waits for the server cleanup acknowledgement after half-closing the socket', async () => { + const eof = Promise.withResolvers() + const f = await fixture({ close: (socket) => { eof.resolve(socket) } }) + const channel = f.open() + const reply = nextBytes(channel) + await write(channel, 'last\n') + await reply + let closed = false + void channel.waitForClose().then(() => { closed = true }) + channel.close() + const peer = await eof.promise + expect(closed).toBe(false) + peer.end() + await channel.waitForClose() + expect(closed).toBe(true) + }) + + it('reports cleanup timeout without terminating the shared process', async () => { + const f = await fixture({ close: () => {}, graceMs: 20 }) + const channel = f.open() + const reply = nextBytes(channel) + await write(channel, 'last\n') + await reply + channel.close() + await expect(channel.waitForClose()).rejects.toMatchObject({ code: 'NATIVE_CLEANUP' }) + await expect(channel.done).rejects.toMatchObject({ code: 'NATIVE_CLEANUP' }) + expect(f.handles[0]?.terminate).not.toHaveBeenCalled() + }) + + it('bounds launcher retries while the listener remains absent', async () => { + const f = await fixture({ listen: false, connectTimeoutMs: 100 }) + vi.useFakeTimers() + try { + const channel = f.open() + await f.spawned + await vi.advanceTimersByTimeAsync(f.config.connectTimeoutMs) + await expect(channel.done).rejects.toMatchObject({ code: 'NATIVE_CONNECT_TIMEOUT' }) + await channel.waitForClose() + expect(f.spawn).toHaveBeenCalled() + expect(f.hellos).toHaveLength(0) + } finally { vi.useRealTimers() } + }) + + it('cancels startup before spawning when closed immediately', async () => { + const f = await fixture() + const channel = f.open() + channel.close() + await channel.waitForClose() + await expect(channel.done).rejects.toBeInstanceOf(Error) + expect(f.spawn).not.toHaveBeenCalled() + }) + + it('terminates only the unfinished startup command and waits for its managed range on cancellation', async () => { + const outcome = Promise.withResolvers() + const range = Promise.withResolvers() + const stopped: PromiseWithResolvers = Promise.withResolvers() + const f = await fixture({ launch: () => ({ ...launcherFixture(), done: outcome.promise, + waitForExit: () => range.promise, + terminate: () => { stopped.resolve(); outcome.resolve({ exitCode: null, signal: 'SIGTERM' }) }, + }) }) + const channel = f.open() + await f.spawned + channel.close() + await stopped.promise + let closed = false + void channel.waitForClose().then(() => { closed = true }) + expect(closed).toBe(false) + range.resolve(true) + await channel.waitForClose() + expect(closed).toBe(true) + }) + + it('starts a replacement server when the previous listener exits between ensure and connect', async () => { + const replaced: PromiseWithResolvers = Promise.withResolvers() + const outcome = Promise.withResolvers() + let launches = 0 + const f = await fixture({ listen: false, launch: () => { + launches += 1 + if (launches === 1) return launcherFixture() + replaced.resolve() + return { ...launcherFixture(), done: outcome.promise } + } }) + const channel = f.open() + const reply = nextBytes(channel) + const pending = write(channel, 'replacement\n') + await replaced.promise + await f.listen() + outcome.resolve({ exitCode: 0, signal: null }) + await pending + expect(String(await reply)).toContain('replacement') + expect(f.spawn).toHaveBeenCalledTimes(2) + expect(f.operations).toEqual(['replacement']) + }) + + it('does not restart or replay operations after a ready peer disconnects', async () => { + const f = await fixture() + const channel = f.open() + const reply = nextBytes(channel) + await write(channel, 'once\n') + await reply + for (const socket of f.sockets) socket.end() + await channel.done + await channel.waitForClose() + expect(f.spawn).toHaveBeenCalledTimes(1) + expect(f.operations).toEqual(['once']) + }) + + it.each(['eof', 'reset'] as const)('retries a pre-ready %s without sending or replaying an operation', async (kind) => { + let attempts = 0 + const f = await fixture({ ready: (socket, ready) => { + attempts += 1 + if (attempts === 1) socket.end() + else socket.write(ready) + } }) + const originalEmit = Socket.prototype.emit + let injected = false + // Unix kernels differ in reporting close as EOF or ECONNRESET; inject only that error event. + const emit = kind === 'reset' ? vi.spyOn(Socket.prototype, 'emit').mockImplementation(function (this: Socket, event: string | symbol, ...args: unknown[]) { + if (!injected && event === 'end') { + injected = true + originalEmit.call(this, 'error', Object.assign(new Error('peer reset during admission'), { code: 'ECONNRESET' })) + } + return originalEmit.call(this, event, ...args) + }) : undefined + try { + const channel = f.open() + const reply = nextBytes(channel) + await write(channel, 'after-admission\n') + expect(JSON.parse(String(await reply))).toEqual({ sequence: 1, request: 'after-admission' }) + expect(f.spawn).toHaveBeenCalledTimes(2) + expect(f.operations).toEqual(['after-admission']) + expect(attempts).toBe(2) + if (kind === 'reset') expect(injected).toBe(true) + } finally { emit?.mockRestore() } + }) +}) diff --git a/snapshots/session/cua-native/workspace/native-helper.mjs b/snapshots/session/cua-native/workspace/native-helper.mjs index d70e9e6f0e..36d1d3cb74 100755 --- a/snapshots/session/cua-native/workspace/native-helper.mjs +++ b/snapshots/session/cua-native/workspace/native-helper.mjs @@ -1,118 +1,207 @@ #!/usr/bin/env node -/** Deterministic JSONL native substitute; this fixture never reads or controls the desktop. */ -import { appendFileSync, existsSync, writeSync } from 'node:fs' +/** Deterministic shared native substitute; this fixture never reads or controls the desktop. */ +import { spawn } from 'node:child_process' +import { randomUUID } from 'node:crypto' +import { appendFileSync, existsSync, mkdirSync, readFileSync, unlinkSync } from 'node:fs' +import { createServer } from 'node:net' +import { join } from 'node:path' import { createInterface } from 'node:readline' +import { fileURLToPath } from 'node:url' -const observe = existsSync(new URL('./observe-helper', import.meta.url)) -const record = (event, data = {}) => { - if (observe) appendFileSync(new URL('./helper-events.jsonl', import.meta.url), JSON.stringify({ event, pid: process.pid, ...data }) + '\n') +const args = process.argv.slice(2) +const argument = name => args[args.indexOf(name) + 1] +const runtimeKey = argument('--runtime-key') +const runtimeDirectory = argument('--runtime-directory') +if (!runtimeKey || !runtimeDirectory) throw new Error('Shared fixture needs its runtime identity and directory') +const socketPath = join(runtimeDirectory, `${runtimeKey}.sock`) + +if (args.includes('--ensure-shared')) { + const child = spawn(process.execPath, [fileURLToPath(import.meta.url), '--serve', ...args.filter(arg => arg !== '--ensure-shared')], { + detached: true, stdio: ['ignore', 'ignore', 'ignore', 'ipc'], + }) + let timedOut = false + const timer = setTimeout(() => { timedOut = true; child.kill() }, Number(argument('--startup-timeout-ms'))) + child.once('error', () => process.exit(1)) + child.once('exit', code => process.exit(!timedOut && code === 0 ? 0 : 1)) + child.once('message', () => { + clearTimeout(timer) + if (child.connected) child.disconnect() + child.unref() + process.stdout.write(JSON.stringify({ type: 'ready', runtimeKey }) + '\n') + }) +} else { + serve() } -record('start') -process.once('exit', () => record('exit')) -process.once('SIGTERM', () => process.exit(0)) -const png = 'iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAIAAACQd1PeAAAACXBIWXMAAAPoAAAD6AG1e1JrAAAADElEQVQImWNgZGIGAAAOAAeCcsnOAAAAAElFTkSuQmCC' -const app = { appHandle: 'fixture-app', pid: 9, name: 'Fixture', bundleId: 'test.fixture' } -const bounds = { x: 40, y: 60, width: 320, height: 200 } -const window = { windowId: 7, pid: 9, appName: 'Fixture', title: 'CUA fixture', bounds, isOnScreen: true, onCurrentSpace: true } -const axlessWindow = { ...window, windowId: 8, title: 'AXless canvas' } -const transformedWindow = { ...window, windowId: 9, title: 'Transformed window', bounds: { x: 16, y: 512, width: 131, height: 145 } } -const snapshots = new Map() -const methods = ['getCapabilities', 'listApps', 'getApp', 'activate', 'show', 'listWindows', 'getState', 'click', 'setValue', 'typeText', 'pressKey', 'scroll', 'drag', 'screenshot'] -let acquired = false -let generation = 0 -let count = 0 -let value = '' -let observationChanged = false -let malformedScreenshot = false -let visualCount = 0 -let frontPid = 42 -let frontWindowId = 99 -function refuse(code, message) { throw Object.assign(new Error(message), { code }) } -function call(requestId, method, params) { - record('call', { requestId, method, params }) - if (method === 'getCapabilities') return { platform: 'macos', sessionLocked: false, permissions: { accessibility: true, screenRecording: true }, methods, coordinateSpace: 'window-points', browser: false } - if (method === 'listApps') return [{ pid: app.pid, name: app.name, bundleId: app.bundleId }] - if (method === 'getApp') { - if (params.selector === 'Denied Fixture' || params.selector === 'Revocable Fixture') { - const denied = params.selector === 'Denied Fixture' - return { appHandle: denied ? 'denied-app' : 'revoked-app', pid: 17, name: params.selector, - authorization: { id: (denied ? 'b' : 'c').repeat(64), name: params.selector, - executablePath: denied ? '/fixture/denied' : '/fixture/revocable', persistable: false } } + +function serve() { + mkdirSync(runtimeDirectory, { recursive: true, mode: 0o700 }) + const observe = existsSync(new URL('./observe-helper', import.meta.url)) + const record = (event, data = {}) => { + if (observe) appendFileSync(new URL('./helper-events.jsonl', import.meta.url), JSON.stringify({ event, pid: process.pid, ...data }) + '\n') + } + const instanceId = randomUUID() + const sockets = new Set() + const application = { count: 0, value: '', visualCount: 0, frontPid: 42, frontWindowId: 99 } + let started = false + let hadClient = false + let connection = 0 + const startupDeadline = setTimeout(() => process.exit(1), Number(argument('--startup-timeout-ms'))) + const server = createServer({ allowHalfOpen: true }, socket => { + sockets.add(socket) + let connected = false + const connectionId = ++connection + const input = createInterface({ input: socket }) + const scopedRecord = (event, data = {}) => record(event, { connection: connectionId, ...data }) + const call = createApplicationState(socket, scopedRecord, application, connectionId) + input.on('line', line => { + const request = JSON.parse(line) + if (!connected) { + if (request.type !== 'hello' || request.protocol !== 1 || request.runtimeKey !== runtimeKey) { + socket.destroy() + return + } + connected = true + hadClient = true + clearTimeout(startupDeadline) + scopedRecord('connect') + socket.write(JSON.stringify({ type: 'ready', protocol: 1, runtimeKey, pid: process.pid, instanceId }) + '\n') + return + } + if (request.type === 'cancel') { + if (typeof request.id !== 'string' || Object.keys(request).length !== 2) socket.destroy() + return + } + let response + try { + const result = call(request.id, request.method, request.params) + if (result === undefined) return + response = { id: request.id, result } + } catch (error) { + response = { id: request.id, error: { code: error.code ?? 'FIXTURE_ERROR', message: error.message } } + } + socket.write(JSON.stringify(response) + '\n') + }) + const release = () => { + if (!sockets.delete(socket)) return + if (connected) scopedRecord('disconnect') + if (sockets.size === 0 && hadClient) process.exit(0) + socket.end() } - if (!['Fixture', 'test.fixture', '9'].includes(params.selector)) refuse('APP_NOT_FOUND', 'Select the running Fixture application') - acquired = true - return { ...app, authorization: {"id": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", "name": "Fixture", "executablePath": "/fixture/app", "persistable": true} } - } - if (!acquired || params.appHandle !== app.appHandle) refuse('APP_NOT_FOUND', 'Acquire an application in this helper first') - if (method === 'show') return { dispatched: true, verified: false, windowId: params.windowId ?? window.windowId, foregroundChanged: false } - if (method === 'activate') { - const before = { frontPid, focusedWindowId: window.windowId, frontWindowId, frontWindowPid: frontPid } - const activationRequested = frontPid !== app.pid - if (activationRequested) { - frontPid = app.pid - frontWindowId = window.windowId + socket.once('end', release) + socket.once('close', release) + socket.on('error', () => socket.destroy()) + }) + server.once('error', error => { + if (error.code !== 'EADDRINUSE') process.exit(1) + process.send({ ready: true }, () => process.exit(0)) + }) + server.listen(socketPath, () => { + started = true + record('start') + process.send?.({ ready: true }) + }) + process.once('SIGTERM', () => process.exit(0)) + process.once('exit', () => { + if (!started) return + record('exit') + try { unlinkSync(socketPath) } catch (error) { if (error.code !== 'ENOENT') throw error } + }) +} + +function createApplicationState(socket, record, application, connectionId) { + const png = 'iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAIAAACQd1PeAAAACXBIWXMAAAPoAAAD6AG1e1JrAAAADElEQVQImWNgZGIGAAAOAAeCcsnOAAAAAElFTkSuQmCC' + const scope = connectionId === 1 ? 'fixture' : `fixture-${connectionId}` + const app = { appHandle: `${scope}-app`, pid: 9, name: 'Fixture', bundleId: 'test.fixture' } + const bounds = { x: 40, y: 60, width: 320, height: 200 } + const window = { windowId: 7, pid: 9, appName: 'Fixture', title: 'CUA fixture', bounds, isOnScreen: true, onCurrentSpace: true } + const axlessWindow = { ...window, windowId: 8, title: 'AXless canvas' } + const transformedWindow = { ...window, windowId: 9, title: 'Transformed window', bounds: { x: 16, y: 512, width: 131, height: 145 } } + const screenshotConfig = new URL('./fixture-screenshot.json', import.meta.url) + const customScreenshot = existsSync(screenshotConfig) ? JSON.parse(readFileSync(screenshotConfig, 'utf8')) : undefined + const snapshots = new Map() + const methods = ['getCapabilities', 'listApps', 'getApp', 'activate', 'show', 'listWindows', 'getState', 'click', 'setValue', 'typeText', 'pressKey', 'scroll', 'drag', 'screenshot'] + let acquired = false + let generation = 0 + let observationChanged = false + let malformedScreenshot = false + function refuse(code, message) { throw Object.assign(new Error(message), { code }) } + function call(requestId, method, params) { + record('call', { requestId, method, params }) + if (method === 'getCapabilities') return { platform: 'macos', sessionLocked: false, permissions: { accessibility: true, screenRecording: true }, methods, coordinateSpace: 'window-points', browser: false } + if (method === 'listApps') return [{ pid: app.pid, name: app.name, bundleId: app.bundleId }] + if (method === 'getApp') { + if (params.selector === 'Denied Fixture' || params.selector === 'Revocable Fixture') { + const denied = params.selector === 'Denied Fixture' + return { appHandle: denied ? 'denied-app' : 'revoked-app', pid: 17, name: params.selector, + authorization: { id: (denied ? 'b' : 'c').repeat(64), name: params.selector, + executablePath: denied ? '/fixture/denied' : '/fixture/revocable', persistable: false } } + } + if (!['Fixture', 'test.fixture', '9'].includes(params.selector)) refuse('APP_NOT_FOUND', 'Select the running Fixture application') + acquired = true + return { ...app, authorization: {"id": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", "name": "Fixture", "executablePath": "/fixture/app", "persistable": true} } } - const after = { frontPid, focusedWindowId: window.windowId, frontWindowId, frontWindowPid: frontPid } - return { dispatched: true, verified: false, foreground: { - requestId, operation: 'activate', targetPid: app.pid, targetWindowId: null, - before, after, activationRequested, raiseRequested: false, - } } - } - if (method === 'listWindows') return [window, axlessWindow] - if (method === 'getState') { - snapshots.delete(params.windowId ?? null) - if (observationChanged) { - observationChanged = false - refuse('STALE_SNAPSHOT', 'Accessibility tree changed during observation; read fresh getAXState') + if (!acquired || params.appHandle !== app.appHandle) refuse('APP_NOT_FOUND', 'Acquire an application in this helper first') + if (method === 'show') return { dispatched: true, verified: false, windowId: params.windowId ?? window.windowId, foregroundChanged: false } + if (method === 'activate') { + const before = { frontPid: application.frontPid, focusedWindowId: window.windowId, frontWindowId: application.frontWindowId, frontWindowPid: application.frontPid } + const activationRequested = application.frontPid !== app.pid + if (activationRequested) { + application.frontPid = app.pid + application.frontWindowId = window.windowId + } + const after = { frontPid: application.frontPid, focusedWindowId: window.windowId, frontWindowId: application.frontWindowId, frontWindowPid: application.frontPid } + return { dispatched: true, verified: false, foreground: { + requestId, operation: 'activate', targetPid: app.pid, targetWindowId: null, + before, after, activationRequested, raiseRequested: false, + } } } - generation++ - const axless = params.windowId === 8 - const snapshotId = `fixture-snapshot-${generation}` - snapshots.set(params.windowId ?? null, { snapshotId, windowId: params.windowId ?? null, axless }) - if (axless) return { appHandle: app.appHandle, pid: 9, windowId: 8, snapshotId, - tree: '', elements: [], windows: [axlessWindow], truncated: false } - return { appHandle: app.appHandle, pid: 9, windowId: params.windowId ?? null, - snapshotId: `fixture-snapshot-${generation}`, tree: `AX scope=${params.windowId === undefined ? 'application' : 'window'} offsets=UTF-16${params.details ? ' bounds=global-points' : ''}\n- [0] AXButton "Increment"${params.details ? ' id="increment" actions=["AXPress"] bounds=[60,80,100,20]' : ''}\n- [1] AXTextField valueWritable=true selectionWritable=true value=${JSON.stringify(value)} placeholder="Fixture text"\n- AXStaticText "Count: ${count}"\n- [2] AXMenuBarItem "File"${params.includeMenus ? '\n - [3] AXMenuItem "Fixture command"' : '\nMenu contents omitted; request includeMenus=true to expand.'}`, - elements: [{ index: 0, role: 'AXButton', title: 'Increment', value: null, description: null, - identifier: null, bounds: [60, 80, 100, 20], actions: ['AXPress'], enabled: true }], - windows: [params.windowId === 9 ? transformedWindow : window], truncated: false } - } - if (params.windowId === 9 && ['screenshot', 'click', 'scroll', 'drag'].includes(method) && params.elementIndex === undefined) { - refuse('WINDOW_CHANGED', 'Window display and accessibility bounds disagree. Read fresh state; if unchanged, use AX actions or explicitly restore the window before capture or coordinate input') - } - if (method === 'screenshot' && malformedScreenshot) { - writeSync(2, '[cua-screenshot] ' + JSON.stringify({ requestId, stage: 'capture-image.begin', elapsedMs: 1 }) + '\n') - writeSync(1, 'invalid screenshot response\n') - return - } - if (method === 'screenshot') return { windowId: params.windowId, mimeType: 'image/png', data: png, pixelWidth: 1, pixelHeight: 1, bounds, coordinateSpace: 'window-points' } - if (method === 'click' || method === 'setValue') { - const observed = [...snapshots.values()].find(snapshot => snapshot.snapshotId === params.snapshotId) - if (!observed) refuse('STALE_SNAPSHOT', 'Read fresh state before acting') - if (params.windowId !== undefined && observed.windowId !== null && observed.windowId !== params.windowId) refuse('STALE_SNAPSHOT', 'Snapshot belongs to another window') - if (observed.axless && params.elementIndex !== undefined) refuse('INVALID_ARGUMENT', 'Element index is absent from the selected snapshot') - if (method === 'click' && params.windowId === 8) visualCount++ - else if (method === 'click') count++ - else value = params.value - record('state', { count, value, visualCount }) + if (method === 'listWindows') return [window, axlessWindow] + if (method === 'getState') { + snapshots.delete(params.windowId ?? null) + if (observationChanged) { + observationChanged = false + refuse('STALE_SNAPSHOT', 'Accessibility tree changed during observation; read fresh getAXState') + } + generation++ + const axless = params.windowId === 8 + const snapshotId = `${scope}-snapshot-${generation}` + snapshots.set(params.windowId ?? null, { snapshotId, windowId: params.windowId ?? null, axless }) + if (axless) return { appHandle: app.appHandle, pid: 9, windowId: 8, snapshotId, + tree: '', elements: [], windows: [axlessWindow], truncated: false } + return { appHandle: app.appHandle, pid: 9, windowId: params.windowId ?? null, + snapshotId: `${scope}-snapshot-${generation}`, tree: `AX scope=${params.windowId === undefined ? 'application' : 'window'} offsets=UTF-16${params.details ? ' bounds=global-points' : ''}\n- [0] AXButton "Increment"${params.details ? ' id="increment" actions=["AXPress"] bounds=[60,80,100,20]' : ''}\n- [1] AXTextField valueWritable=true selectionWritable=true value=${JSON.stringify(application.value)} placeholder="Fixture text"\n- AXStaticText "Count: ${application.count}"\n- [2] AXMenuBarItem "File"${params.includeMenus ? '\n - [3] AXMenuItem "Fixture command"' : '\nMenu contents omitted; request includeMenus=true to expand.'}`, + elements: [{ index: 0, role: 'AXButton', title: 'Increment', value: null, description: null, + identifier: null, bounds: [60, 80, 100, 20], actions: ['AXPress'], enabled: true }], + windows: [params.windowId === 9 ? transformedWindow : window], truncated: false } + } + if (params.windowId === 9 && ['screenshot', 'click', 'scroll', 'drag'].includes(method) && params.elementIndex === undefined) { + refuse('WINDOW_CHANGED', 'Window display and accessibility bounds disagree. Read fresh state; if unchanged, use AX actions or explicitly restore the window before capture or coordinate input') + } + if (method === 'screenshot' && malformedScreenshot) { + socket.write(JSON.stringify({ type: 'diagnostic', requestId, stage: 'capture-image.begin', elapsedMs: 1 }) + '\ninvalid screenshot response\n') + return + } + if (method === 'screenshot') return customScreenshot + ? { pixelWidth: customScreenshot.pixelWidth, pixelHeight: customScreenshot.pixelHeight, bounds: customScreenshot.bounds, + windowId: params.windowId, mimeType: 'image/png', coordinateSpace: 'window-points', + data: readFileSync(new URL(customScreenshot.file, import.meta.url)).toString('base64') } + : { windowId: params.windowId, mimeType: 'image/png', data: png, pixelWidth: 1, pixelHeight: 1, bounds, coordinateSpace: 'window-points' } + if (method === 'click' || method === 'setValue') { + const observed = [...snapshots.values()].find(snapshot => snapshot.snapshotId === params.snapshotId) + if (!observed) refuse('STALE_SNAPSHOT', 'Read fresh state before acting') + if (params.windowId !== undefined && observed.windowId !== null && observed.windowId !== params.windowId) refuse('STALE_SNAPSHOT', 'Snapshot belongs to another window') + if (observed.axless && params.elementIndex !== undefined) refuse('INVALID_ARGUMENT', 'Element index is absent from the selected snapshot') + if (method === 'click' && params.windowId === 8) application.visualCount++ + else if (method === 'click') application.count++ + else application.value = params.value + record('state', { count: application.count, value: application.value, visualCount: application.visualCount }) + return { dispatched: true, verified: false } + } + if (method === 'pressKey' && params.key === 'Return') observationChanged = true + if (method === 'pressKey' && params.key === 'F12') malformedScreenshot = true + if (method === 'typeText') application.value += params.text + else if (!['pressKey', 'scroll', 'drag'].includes(method)) refuse('UNSUPPORTED_OPERATION', method) return { dispatched: true, verified: false } } - if (method === 'pressKey' && params.key === 'Return') observationChanged = true - if (method === 'pressKey' && params.key === 'F12') malformedScreenshot = true - if (method === 'typeText') value += params.text - else if (!['pressKey', 'scroll', 'drag'].includes(method)) refuse('UNSUPPORTED_OPERATION', method) - return { dispatched: true, verified: false } + return call } -const input = createInterface({ input: process.stdin }) -input.once('close', () => process.exit(0)) -input.on('line', (line) => { - const request = JSON.parse(line) - let response - try { - const result = call(request.id, request.method, request.params) - if (result === undefined) return - response = { id: request.id, result } - } - catch (error) { response = { id: request.id, error: { code: error.code ?? 'FIXTURE_ERROR', message: error.message } } } - process.stdout.write(JSON.stringify(response) + '\n') -})