diff --git a/.agents/notes/implemented/architecture/2026-09-17-experimental-cua-object-api.i18n.yaml b/.agents/notes/implemented/architecture/2026-09-17-experimental-cua-object-api.i18n.yaml index f61dcaca9c..6bf6cbc019 100644 --- a/.agents/notes/implemented/architecture/2026-09-17-experimental-cua-object-api.i18n.yaml +++ b/.agents/notes/implemented/architecture/2026-09-17-experimental-cua-object-api.i18n.yaml @@ -9,8 +9,8 @@ en: a22ddcfb339e503d zh: a1f4fadb46fce1e7 /agent-note-native-cua-objects-independent-of-evaluator-ownership/decision: - en: fecd529e76af4100 - zh: d8df0c4c9a4e2c9c + en: 6ce9add827f6f245 + zh: e90f7fd13d058c87 /agent-note-native-cua-objects-independent-of-evaluator-ownership/alternatives-considered: en: 62dfcec71ed2a058 zh: 0f3c6edfe19fef1f diff --git a/.agents/notes/implemented/architecture/2026-09-17-experimental-cua-object-api.md b/.agents/notes/implemented/architecture/2026-09-17-experimental-cua-object-api.md index 9b3ae36a02..6592c6ac63 100644 --- a/.agents/notes/implemented/architecture/2026-09-17-experimental-cua-object-api.md +++ b/.agents/notes/implemented/architecture/2026-09-17-experimental-cua-object-api.md @@ -16,6 +16,10 @@ The facade accepts an operation dispatcher. Its tool adapter resolves the curren A live Agent owns one serial native helper. App handles bind process identity and kernel start time; replacing the process or helper never revives old handles. AX snapshots are scoped to an app and optional window, so independent window observations remain usable. Coordinate actions select an owned window and use window-local points. Keyboard input remains app-scoped because macOS delivers it to the app's current focus. +The operation schema owns shared data declarations. Host-only limits and launch continuations remain separate from model parameters, and generated native field/enum tables preserve independent native admission without hand-maintained copies. Native code continues to own system calls, permissions and cancellation. AX formatting consumes immutable observations; the walker retains sole ownership of indices and AX references. + +Clipboard paste is distinct from Unicode key input: Return and Tab key events can trigger actions instead of inserting clipboard content. Pasteboard access runs on the AppKit main actor and retains promised data until a reader requests it. AX command completion alone does not establish that the application has fetched the data. Restoration checks ownership before replacing contents; the caller still verifies the application result. + ## Alternatives considered Depending on the complete upstream driver would simplify upgrades but does not meet the requirement for locally owned implementation. Copying browser and other platform code would add unsupported scope. The JSON operation protocol allows another native implementation without changing object consumers. diff --git a/.agents/notes/implemented/architecture/2026-09-17-experimental-cua-object-api.zh.md b/.agents/notes/implemented/architecture/2026-09-17-experimental-cua-object-api.zh.md index 85b82763fa..6e82353a7a 100644 --- a/.agents/notes/implemented/architecture/2026-09-17-experimental-cua-object-api.zh.md +++ b/.agents/notes/implemented/architecture/2026-09-17-experimental-cua-object-api.zh.md @@ -16,6 +16,10 @@ Computer-use 程序需要一致的应用和窗口操作。将这些操作与 CUA 每个活动 Agent 拥有一个串行原生辅助进程。应用句柄绑定进程身份及内核启动时间，替换进程或辅助进程不会使旧句柄重新有效。AX 快照按应用和可选窗口保存，不同窗口的观察可以分别使用。坐标操作选择属于该应用的窗口，并使用窗口内 point。键盘输入保持应用范围，因为 macOS 将其送往应用的当前焦点。 +操作 schema 负责共享数据声明。Host 专属限制和启动续接与模型参数分开，生成的原生字段、枚举表保留独立原生校验，同时避免手工维护副本。系统调用、权限和取消仍由原生代码负责。AX 格式化只消费不可变观察数据，遍历器独自负责索引与 AX 引用。 + +剪贴板粘贴与 Unicode 按键输入分别实现，因为 Return 和 Tab 按键可能触发动作，而不是插入剪贴板内容。剪贴板访问使用 AppKit 主线程，保留延迟提供的数据直到读取方请求。AX 命令完成本身不能证明应用已取得数据。恢复前检查剪贴板归属；调用方仍需验证应用中的结果。 + ## 考虑过的替代方案 依赖完整上游驱动可以简化升级，但不符合本地维护实现的要求。复制浏览器及其他平台代码会引入尚不支持的范围。JSON 操作协议允许替换原生实现，而不改变对象调用方。 diff --git a/.agents/notes/implemented/architecture/2026-09-26-cua-application-consent.i18n.yaml b/.agents/notes/implemented/architecture/2026-09-26-cua-application-consent.i18n.yaml index b2f620de49..cacdc69bdb 100644 --- a/.agents/notes/implemented/architecture/2026-09-26-cua-application-consent.i18n.yaml +++ b/.agents/notes/implemented/architecture/2026-09-26-cua-application-consent.i18n.yaml @@ -9,8 +9,8 @@ en: b32a7ce6483b9b48 zh: ec0938ad032b99d8 /agent-note-application-consent-for-native-computer-use/decision: - en: ce0086958b9eaf33 - zh: ef8936d60714437d + en: 9234627787165031 + zh: 6fbb56110887a957 /agent-note-application-consent-for-native-computer-use/alternatives-considered: en: 05fd6823e5c2e18f zh: 998d7a2030aa0b91 diff --git a/.agents/notes/implemented/architecture/2026-09-26-cua-application-consent.md b/.agents/notes/implemented/architecture/2026-09-26-cua-application-consent.md index a5db903cb9..56009e8ce1 100644 --- a/.agents/notes/implemented/architecture/2026-09-26-cua-application-consent.md +++ b/.agents/notes/implemented/architecture/2026-09-26-cua-application-consent.md @@ -14,6 +14,8 @@ The CUA executor owns application consent for every object-API and native-tool d A saved identity combines the canonical executable path with a verified designated signing requirement, represented by a SHA-256 digest. The path prevents authorization from following a copied application, and the signing requirement distinguishes replacements while allowing matching signed updates. Unverifiable and ad-hoc applications receive task access only, tied to a process instance and helper lifetime. Labels do not establish identity. +Cold application selection prepares an executable identity and handle without launching. The same consent decision guards the private launch continuation, and the helper checks the executable again before launch and binds the resulting process afterward. Signed selections retain the normal designated-requirement identity; an unverifiable executable uses a content identity for this pending selection and cannot create a saved grant. Preparing metadata before asking avoids starting an application the user refuses. + Selections of the same application within one task are serialized, so a pending question does not produce competing decisions. Saved-grant additions read the current configuration inside one mutation queue. Refusals prevent another prompt within the task; an explicitly restored saved grant takes precedence at dispatch. Removing a saved grant invalidates native helpers retaining the application's handles and cancels active operations; Agent/provider disposal awaits helper cleanup. Reauthorization uses a fresh helper, so old app and window snapshots cannot become valid again. The [native object decision](2026-09-17-experimental-cua-object-api.md) continues to own process identity, coordinate semantics, and separation from the evaluator. diff --git a/.agents/notes/implemented/architecture/2026-09-26-cua-application-consent.zh.md b/.agents/notes/implemented/architecture/2026-09-26-cua-application-consent.zh.md index caa19aa311..7200b153b3 100644 --- a/.agents/notes/implemented/architecture/2026-09-26-cua-application-consent.zh.md +++ b/.agents/notes/implemented/architecture/2026-09-26-cua-application-consent.zh.md @@ -14,6 +14,8 @@ CUA 执行器在每次对象 API 和原生工具调用时检查应用授权。 长期身份由规范化可执行文件路径和经过验证的签名指定要求组成，以 SHA-256 摘要表示。路径阻止许可跟随应用副本，签名要求区分替换程序，并允许符合相同签名要求的更新。签名不可验证或使用 ad-hoc 签名的应用只能获得任务授权，许可绑定进程实例和 helper 生命周期。显示名称不用于确定身份。 +选择尚未运行的应用时，先准备可执行文件身份和句柄，不立即启动。同一授权决定保护内部启动续接，helper 在启动前再次核对可执行文件，并在启动后绑定实际进程。已签名应用沿用签名指定要求身份；签名不可验证的程序使用本次待启动选择的内容身份，不能生成长期许可。先准备元数据再询问，可以避免启动用户拒绝的应用。 + 同一任务对同一应用的选择串行执行，避免未完成的问题产生相互冲突的决定。长期许可的新增在同一写入队列内读取当前配置。拒绝会阻止任务内的再次询问；用户显式恢复的长期许可在执行检查时优先于旧拒绝记录。 删除长期许可会使保留目标应用句柄的原生 helper 失效，并取消活动操作；Agent 或提供方销毁时等待组件清理。重新授权使用新 helper，因此旧应用对象和窗口快照不能重新生效。[原生对象决策](2026-09-17-experimental-cua-object-api.zh.md)继续负责进程身份、坐标语义和与执行器分离的设计。 diff --git a/packages/computer-use/cua/README.i18n.yaml b/packages/computer-use/cua/README.i18n.yaml index fc529946ab..d51a2dd57b 100644 --- a/packages/computer-use/cua/README.i18n.yaml +++ b/packages/computer-use/cua/README.i18n.yaml @@ -18,11 +18,11 @@ en: 867e827e2d69c25f zh: e4eb59c7c8028846 /deepseek-ai-dsh-cua/use-this-package/connect-an-external-execution-environment: - en: f16132600c14bd68 - zh: 35ea7ae88c0e851d + en: a9a65a2953478d19 + zh: ef06dc75607e1724 /deepseek-ai-dsh-cua/understand-the-implementation: - en: ee9fd0ddf93d496d - zh: 38d8d5f8b58e6f50 + en: 0ab772a0dce859f6 + zh: 7851201c70372a25 /deepseek-ai-dsh-cua/further-exploration: en: b1047318879cd6f4 zh: e4ce4a5ad2a9c4dc diff --git a/packages/computer-use/cua/README.md b/packages/computer-use/cua/README.md index 5af2ed0c22..1f503091c0 100644 --- a/packages/computer-use/cua/README.md +++ b/packages/computer-use/cua/README.md @@ -77,7 +77,7 @@ Use app methods directly for single-window tasks. `getApp` reads initial AX stat `getWindow` emits its initial AX tree. An owned window without an AX surface returns an empty snapshot and remains available for screenshots and coordinate input. Window objects retain separate snapshots, so reading one window does not invalidate another. Re-reading the same scope invalidates old indices. An AX node removed during observation rejects the read with `STALE_SNAPSHOT` and publishes no partial tree; read again without repeating the preceding input. App-scoped screenshots and coordinates reuse the latest AX observation: its focused on-screen, nonzero-size window, otherwise the sole such window. They do not query current focus again; read `app.getAXState()` to follow a focus change. Explicit window objects retain their selected target. Keyboard methods stay on `app` because input follows the app’s current focus. -`paste(text, { format: "text" | "html" })` temporarily publishes the requested clipboard format and invokes the application’s enabled standard Paste menu command. It restores every captured original format while it still owns the clipboard, preserving a later copy. Paste operations share a cross-process lock and cancellation joins helper cleanup. A forced process exit can prevent restoration. Unsupported Paste commands fail without typing fallback; `typeText` remains separate key-event input. +`paste(text, { format: "text" | "html" })` temporarily publishes the requested clipboard format and invokes the application’s enabled standard Paste menu command. It keeps the temporary format until a pasteboard reader requests its data, then restores every captured original format while it still owns the clipboard, preserving a later copy. Paste operations share a cross-process lock and cancellation joins helper cleanup. A forced process exit can prevent restoration. Unsupported Paste commands fail without typing fallback; `typeText` remains separate key-event input. `selectText` selects a unique literal occurrence or places the cursor before/after it through writable `AXSelectedTextRange`. Adjacent prefix/suffix disambiguate repeats. Selection preserves the remaining text and formatting, and verifies the resulting UTF-16 range; unsupported, missing, ambiguous or changed text fails without keyboard fallback. AX text includes `selectionWritable`, selection ranges and selected text when available. @@ -101,6 +101,10 @@ Indexed clicks support a single right/middle click or a single/double left click The object facade translates app/window methods into JSON operations. The bridge routes them through the normal tool pipeline, preserving authorization, nested logs and image admission. A separate serial helper belongs to each Agent. Native handles bind a process identity to its kernel start time, and snapshots retain native AX references only inside that helper. The same operation protocol can serve a future platform provider without adding platform logic to the object API. +[Zod operation definitions](src/protocol.ts) own model parameters and separately declare Host resource fields and private continuations. Public data types derive from those definitions. `pnpm exec tsx packages/computer-use/cua/native/generate-protocol.ts` generates the native JSON Schema and Swift field/enum admission tables; the package test lane checks freshness and rejects Host-only fields at model admission. Native target validation, resource limits and operation behavior remain enforced at execution. + +[AX text formatting](native/Sources/CuaNative/AXTextFormatter.swift) consumes already-read node data. The AX walker alone assigns indices and retains native references, so formatting owns no second snapshot or target registry. + Native request IDs match the logged tool call IDs. Successful activation and window-raise results retain foreground request details and immediate before/after focus observations in the raw tool result. An already-frontmost app skips redundant activation; a window raise is skipped only when the exact window is also focused, main, unminimized and first among visible normal windows. Unknown observations retain the requested operation. These observations do not establish who caused an unrelated focus change or verify completion. Node roles and metadata share one AX batch per observation. Group and scroll-area nodes contribute only children; unused metadata errors do not reject them. Unsupported batching retains individual attribute reads. AX traversal visits each native identity once, including proxy references that compare equal, so cyclic accessibility children cannot repeat whole subtrees. The [source manifest](native/upstream.json) records adapted upstream behavior and local Swift modules. AppKit runs on the main event loop; a separate serial executor owns AX references and synchronous input. A request retains exclusive execution through capture and response delivery. No invariant companion is published: there is no independently projected registry or durable evaluator state to reconcile. Protocol validation, target identity and resource cleanup are checked by their owning implementations. diff --git a/packages/computer-use/cua/README.zh.md b/packages/computer-use/cua/README.zh.md index f8e7b44bbb..5e3d0c6aa6 100644 --- a/packages/computer-use/cua/README.zh.md +++ b/packages/computer-use/cua/README.zh.md @@ -77,7 +77,7 @@ void await app.getScreenshot() `getWindow` 会输出初始 AX 树。已确认归属但没有 AX 界面的窗口返回空快照，仍可截图和发送坐标输入。不同窗口分别保留快照，读取一个窗口不会使另一个窗口失效；重新读取同一范围会使旧索引失效。观察期间节点被移除时，该次读取返回 `STALE_SNAPSHOT`，不发布部分树；可以重新读取，但不要重复此前的输入。应用级截图和坐标操作复用最近一次 AX 观察：优先使用其中在屏幕上且尺寸非零的焦点窗口，否则要求只有一个符合条件的窗口。它们不会重新查询即时焦点；需要跟随焦点变化时，先读取 `app.getAXState()`。显式窗口对象保持选定的目标。键盘方法保留在 `app` 上，因为输入跟随应用的当前焦点。 -`paste(text, { format: "text" | "html" })` 临时提供指定格式的剪贴板内容，并调用应用已启用的标准 Paste 菜单命令。只在仍持有剪贴板时恢复保存的全部原格式，保留期间产生的新复制内容。粘贴操作共用跨进程锁，取消会等待辅助进程清理。强制终止进程可能阻止恢复。应用不支持 Paste 命令时明确失败，不自动改成逐字输入；`typeText` 仍单独提供按键事件输入。 +`paste(text, { format: "text" | "html" })` 临时提供指定格式的剪贴板内容，并调用应用已启用的标准 Paste 菜单命令。临时格式保留到剪贴板读取方请求其数据，然后只在仍持有剪贴板时恢复保存的全部原格式，保留期间产生的新复制内容。粘贴操作共用跨进程锁，取消会等待辅助进程清理。强制终止进程可能阻止恢复。应用不支持 Paste 命令时明确失败，不自动改成逐字输入；`typeText` 仍单独提供按键事件输入。 `selectText` 通过可写的 `AXSelectedTextRange` 选中唯一的字面文本，或把光标放在它的前后。紧邻的前后文用于区分重复文本。选区操作保留其余文字和格式，并核对实际 UTF-16 范围；控件不支持、文本不存在、匹配不唯一或文字变化时明确失败，不自动改用键盘输入。AX 文本包含可用的 `selectionWritable`、选区范围和选中文字。 @@ -101,6 +101,10 @@ AppKit sheet 可以通过其窗口 ID 单独选择，也会出现在所属文档 对象接口将应用和窗口方法转换为 JSON 操作。适配模块通过标准工具链保留授权、嵌套日志和图片接入。每个 Agent 拥有一个串行处理请求的辅助进程。原生句柄绑定进程身份及内核启动时间，快照中的 AX 引用只保存在辅助进程内。未来平台可以实现相同操作协议，无需在对象接口中加入平台逻辑。 +[Zod 操作定义](src/protocol.ts)负责模型参数，并单独声明 Host 资源字段和内部续接操作。公共数据类型从这些定义推导。`pnpm exec tsx packages/computer-use/cua/native/generate-protocol.ts` 生成原生 JSON Schema 和 Swift 字段、枚举校验表；包测试检查产物是否同步，并验证模型不能提交 Host 专属字段。原生目标验证、资源上限和操作行为仍在执行处检查。 + +[AX 文本格式化](native/Sources/CuaNative/AXTextFormatter.swift)只接收已读取的节点数据。AX 遍历器独自分配索引并保存原生引用，格式化不维护另一套快照或目标注册表。 + 原生请求 ID 与工具日志中的调用 ID 一致。成功的激活和窗口抬起操作会在原始工具结果中保留前台操作信息，以及操作前后的即时焦点观察。应用已在前台时跳过重复激活；窗口抬起还要求该窗口正是焦点主窗口、未最小化且位于可见普通窗口的最前方，才会跳过。无法确认时保留原请求。这些观察不能确认其他焦点切换的发起者，也不代表操作已经完成。 每次观察在同一批 AX 请求中读取节点角色和元数据。组节点和滚动区域节点只提供子节点，未使用的元数据错误不会导致它们被拒绝。不支持批量读取时保留逐个属性读取。AX 遍历对每个原生身份只访问一次，包括表示同一元素的不同代理引用，避免循环子节点重复展开整个子树。[源码清单](native/upstream.json)记录改写的上游行为和本地 Swift 模块。AppKit 使用主事件循环，独立串行执行器持有 AX 引用并发送同步输入。一条请求在截图和结果发送期间仍独占执行。本包不发布 invariant 入口，因为没有需要对照的独立注册表投影或持久执行器状态。协议验证、目标身份和资源清理由对应实现负责。 diff --git a/packages/computer-use/cua/native/README.txt b/packages/computer-use/cua/native/README.txt index 8f7447b957..5e29d3d6a6 100644 --- a/packages/computer-use/cua/native/README.txt +++ b/packages/computer-use/cua/native/README.txt @@ -57,3 +57,5 @@ System file-panel keyboard routing resolves the remote recipient through the sel Keyboard modifiers in this native request schema accept command/control/option/shift/fn and cmd/ctrl/alt aliases. The TypeScript raw-tool schema accepts only canonical names; the object API normalizes key-chord aliases before dispatch. Application selection verifies the executable path and code-signing requirement before returning an authorization identity to the Host. Unsigned applications receive a process-lifetime identity. The Host obtains human consent before using that handle for observation or input. + +Protocol declarations are generated from src/protocol.ts. Run `pnpm exec tsx packages/computer-use/cua/native/generate-protocol.ts` from the repository root; package tests verify generated JSON and Swift field/enum tables. Model-submittable parameters exclude Host resource bounds and the private launch continuation. diff --git a/packages/computer-use/cua/native/Sources/CuaNative/AXAccess.swift b/packages/computer-use/cua/native/Sources/CuaNative/AXAccess.swift index 2d7f61de41..1285619d41 100644 --- a/packages/computer-use/cua/native/Sources/CuaNative/AXAccess.swift +++ b/packages/computer-use/cua/native/Sources/CuaNative/AXAccess.swift @@ -401,20 +401,12 @@ final class AXAccess { return } let index: Int? = addressable ? indexed.count : nil - var line = String(repeating: " ", count: level) + "- " - if let index { indexed[index] = element; line += "[\(index)] " } - line += role - if enabled == false { line += " disabled" } - if valueWritable { line += " valueWritable=true" } - if selectionWritable { line += " selectionWritable=true" } - if let selected { line += " selected=\(selected)" } - if let title, !title.isEmpty { line += " \(String(reflecting: title))" } - if let value, !value.isEmpty { line += " = \(String(reflecting: value))" } - if let description, !description.isEmpty { line += " (\(description))" } - if let selectedRange { line += " selection=[\(selectedRange.location), \(selectedRange.length)]" } - if let selectedText, !selectedText.isEmpty { line += " selectedText=\(String(reflecting: selectedText))" } - if !actionNames.isEmpty { line += " actions=[\(actionNames.joined(separator: ", "))]" } - lines.append(line) + if let index { indexed[index] = element } + let node = AXTextNode(index: index, role: role, title: title, value: value, description: description, + actions: actionNames, enabled: enabled, valueWritable: valueWritable, + selectionWritable: selectionWritable, selected: selected, selectedRange: selectedRange, + selectedText: selectedText) + lines.append(AXTextFormatter.line(node, level: level)) if let index { let frame: WindowBounds? if let position = try observed("AXPosition") { frame = try bounds(position: position, size: observed("AXSize")) } @@ -434,7 +426,7 @@ final class AXAccess { for child in try childValues(observed("AXChildren"), attribute: "AXChildren") { try walk(child, level: level + 1, depth: depth + 1) } } for root in roots { try walk(root, level: 0, depth: 0) } - if truncated { lines.append("Accessibility tree truncated; observe another window or use screenshot coordinates for unlisted elements.") } + if truncated { lines.append(AXTextFormatter.truncated) } return AXProjection(tree: lines.joined(separator: "\n"), elements: elements, indexed: indexed, focusedWindowID: focusedID, truncated: truncated) } diff --git a/packages/computer-use/cua/native/Sources/CuaNative/AXTextFormatter.swift b/packages/computer-use/cua/native/Sources/CuaNative/AXTextFormatter.swift new file mode 100644 index 0000000000..894b23d3a8 --- /dev/null +++ b/packages/computer-use/cua/native/Sources/CuaNative/AXTextFormatter.swift @@ -0,0 +1,37 @@ +// Pure model-facing text projection; the AX walker owns native references, indexing and traversal. +import Foundation + +struct AXTextNode { + let index: Int? + let role: String + let title: String? + let value: String? + let description: String? + let actions: [String] + let enabled: Bool? + let valueWritable: Bool + let selectionWritable: Bool + let selected: Bool? + let selectedRange: CFRange? + let selectedText: String? +} + +enum AXTextFormatter { + static let truncated = "Accessibility tree truncated; observe another window or use screenshot coordinates for unlisted elements." + static func line(_ node: AXTextNode, level: Int) -> String { + var line = String(repeating: " ", count: level) + "- " + if let index = node.index { line += "[\(index)] " } + line += node.role + if node.enabled == false { line += " disabled" } + if node.valueWritable { line += " valueWritable=true" } + if node.selectionWritable { line += " selectionWritable=true" } + if let selected = node.selected { line += " selected=\(selected)" } + if let title = node.title, !title.isEmpty { line += " \(String(reflecting: title))" } + if let value = node.value, !value.isEmpty { line += " = \(String(reflecting: value))" } + if let description = node.description, !description.isEmpty { line += " (\(description))" } + if let range = node.selectedRange { line += " selection=[\(range.location), \(range.length)]" } + if let text = node.selectedText, !text.isEmpty { line += " selectedText=\(String(reflecting: text))" } + if !node.actions.isEmpty { line += " actions=[\(node.actions.joined(separator: ", "))]" } + return line + } +} diff --git a/packages/computer-use/cua/native/Sources/CuaNative/AXWorker.swift b/packages/computer-use/cua/native/Sources/CuaNative/AXWorker.swift index 61b4aa7366..911d2e1515 100644 --- a/packages/computer-use/cua/native/Sources/CuaNative/AXWorker.swift +++ b/packages/computer-use/cua/native/Sources/CuaNative/AXWorker.swift @@ -180,7 +180,7 @@ actor AXWorker { return .result(.object(["platform": .string("macos"), "permissions": .object([ "accessibility": .bool(AXIsProcessTrusted()), "screenRecording": .bool(CGPreflightScreenCaptureAccess())]), "sessionLocked": locked.map(JSONValue.bool) ?? .null, - "methods": .array(RequestParameters.fields.keys.filter { $0 != "launchApp" }.sorted().map(JSONValue.string)), "coordinateSpace": .string("window-points"), "browser": .bool(false)])) + "methods": .array(NativeProtocol.modelMethods.map(JSONValue.string)), "coordinateSpace": .string("window-points"), "browser": .bool(false)])) } if method == "listApps" { return .result(.array(catalog.applications.filter { $0.regular && $0.name != nil }.map(\.json))) } try NativePlatform.requireUnlocked(locked) diff --git a/packages/computer-use/cua/native/Sources/CuaNative/NativeProtocol.generated.swift b/packages/computer-use/cua/native/Sources/CuaNative/NativeProtocol.generated.swift new file mode 100644 index 0000000000..79fbbc9aa5 --- /dev/null +++ b/packages/computer-use/cua/native/Sources/CuaNative/NativeProtocol.generated.swift @@ -0,0 +1,31 @@ +// Generated by native/generate-protocol.ts from src/protocol.ts. Do not edit. +enum NativeProtocol { + static let modelMethods: [String] = ["activate", "click", "drag", "findApps", "getApp", "getCapabilities", "getState", "listApps", "listWindows", "paste", "performSecondaryAction", "pressKey", "screenshot", "scroll", "selectText", "setValue", "typeText"] + static let fields: [String: Set] = [ + "activate": ["appHandle"], + "click": ["appHandle", "clickCount", "elementIndex", "modifiers", "mouseButton", "snapshotId", "windowId", "x", "y"], + "drag": ["appHandle", "durationMs", "fromX", "fromY", "modifiers", "mouseButton", "snapshotId", "steps", "toX", "toY", "windowId"], + "findApps": ["maxAppMatches", "query"], + "getApp": ["selector"], + "getCapabilities": [], + "getState": ["appHandle", "maxDepth", "maxElements", "windowId"], + "launchApp": ["appHandle"], + "listApps": [], + "listWindows": ["appHandle"], + "paste": ["appHandle", "format", "maxClipboardBytes", "maxDepth", "maxElements", "text"], + "performSecondaryAction": ["action", "appHandle", "elementIndex", "snapshotId"], + "pressKey": ["appHandle", "key", "modifiers"], + "screenshot": ["appHandle", "windowId"], + "scroll": ["appHandle", "deltaLinesX", "deltaLinesY", "snapshotId", "windowId", "x", "y"], + "selectText": ["appHandle", "elementIndex", "prefix", "selectionType", "snapshotId", "suffix", "text"], + "setValue": ["appHandle", "elementIndex", "snapshotId", "value"], + "typeText": ["appHandle", "text"], + ] + static let stringEnums: [String: [String: Set]] = [ + "click": ["modifiers": ["command", "control", "fn", "option", "shift"], "mouseButton": ["left", "middle", "right"]], + "drag": ["modifiers": ["command", "control", "fn", "option", "shift"], "mouseButton": ["left", "middle", "right"]], + "paste": ["format": ["html", "text"]], + "pressKey": ["modifiers": ["command", "control", "fn", "option", "shift"]], + "selectText": ["selectionType": ["cursor_after", "cursor_before", "text"]], + ] +} diff --git a/packages/computer-use/cua/native/Sources/CuaNative/RequestAdmission.swift b/packages/computer-use/cua/native/Sources/CuaNative/RequestAdmission.swift index 367d3e62ec..81bd59e0f4 100644 --- a/packages/computer-use/cua/native/Sources/CuaNative/RequestAdmission.swift +++ b/packages/computer-use/cua/native/Sources/CuaNative/RequestAdmission.swift @@ -2,26 +2,22 @@ import Foundation struct RequestParameters: Sendable { - static let fields: [String: Set] = [ - "getCapabilities": [], "listApps": [], "findApps": ["query", "maxAppMatches"], "getApp": ["selector"], - "launchApp": ["appHandle"], - "listWindows": ["appHandle"], "activate": ["appHandle"], - "getState": ["appHandle", "windowId", "maxElements", "maxDepth"], - "click": ["appHandle", "snapshotId", "elementIndex", "windowId", "x", "y", "mouseButton", "clickCount", "modifiers"], - "setValue": ["appHandle", "snapshotId", "elementIndex", "value"], - "selectText": ["appHandle", "snapshotId", "elementIndex", "text", "prefix", "suffix", "selectionType"], - "performSecondaryAction": ["appHandle", "snapshotId", "elementIndex", "action"], - "paste": ["appHandle", "text", "format", "maxClipboardBytes", "maxElements", "maxDepth"], - "typeText": ["appHandle", "text"], "pressKey": ["appHandle", "key", "modifiers"], - "scroll": ["appHandle", "windowId", "snapshotId", "x", "y", "deltaLinesX", "deltaLinesY"], - "drag": ["appHandle", "windowId", "snapshotId", "fromX", "fromY", "toX", "toY", "durationMs", "steps", "mouseButton", "modifiers"], - "screenshot": ["appHandle", "windowId"], - ] + static let fields = NativeProtocol.fields let values: [String: JSONValue] init(_ request: NativeRequest) throws { guard let values = request.params.object else { throw Self.invalid("params must be an object") } guard let allowed = Self.fields[request.method] else { throw NativeFailure(code: "UNSUPPORTED", message: "Unsupported native operation: \(request.method)") } guard Set(values.keys).isSubset(of: allowed) else { throw Self.invalid("Unknown operation parameter") } + for (field, allowed) in NativeProtocol.stringEnums[request.method] ?? [:] { + guard let value = values[field] else { continue } + switch value { + case .string(let text): + guard allowed.contains(text) else { throw Self.invalid("Unsupported value for \(field)") } + case .array(let items): + guard items.allSatisfy({ $0.string.map(allowed.contains) ?? false }) else { throw Self.invalid("Unsupported value for \(field)") } + default: throw Self.invalid("Invalid enum value for \(field)") + } + } self.values = values } static func invalid(_ message: String) -> NativeFailure { NativeFailure(code: "INVALID_ARGUMENT", message: message) } @@ -48,9 +44,8 @@ struct RequestParameters: Sendable { func modifiers() throws -> [String] { guard let value = values["modifiers"] else { return [] } guard case .array(let items) = value else { throw Self.invalid("modifiers must be an array") } - let allowed: Set = ["command", "cmd", "shift", "option", "alt", "control", "ctrl", "fn"] return try items.map { item in - guard let name = item.string, allowed.contains(name) else { throw Self.invalid("Unsupported key modifier") } + guard let name = item.string else { throw Self.invalid("Unsupported key modifier") } return name } } diff --git a/packages/computer-use/cua/native/generate-protocol.ts b/packages/computer-use/cua/native/generate-protocol.ts new file mode 100644 index 0000000000..105e124a80 --- /dev/null +++ b/packages/computer-use/cua/native/generate-protocol.ts @@ -0,0 +1,91 @@ +/** Derive native field/enum admission and the wire schema from model and Host-only Zod definitions. */ +import { readFile, writeFile } from 'node:fs/promises' +import { z } from 'zod' +import { NativeOnlyOperations, NativeResourceParameters, Request } from '../src/protocol.ts' + +type Schema = ReturnType + +function object(schema: Schema | boolean | undefined): Schema { + if (schema === undefined || typeof schema === 'boolean') throw new Error('CUA protocol requires an object schema at this position') + return schema +} +function variants(schema: Schema): Schema[] { return schema.anyOf ?? schema.oneOf ?? [schema] } + +/** + * Render deterministic native artifacts; platform actions and lifecycle remain handwritten. + * @returns Native JSON Schema and Swift field/enum tables, including private continuations. + */ +export function renderNativeProtocol(): { json: string; swift: string } { + const resources = new Map(Object.entries(NativeResourceParameters)) + const modelMethods = Request.options.map(schema => schema.shape.method.value).sort() + const schemas = [...Request.options, ...NativeOnlyOperations].map(schema => z.toJSONSchema(schema)) + const fields = new Map>() + const enums = new Map>>() + for (const schema of schemas) { + const method = object(schema.properties?.method).const + if (typeof method !== 'string') throw new Error('CUA method must be one string literal') + const params = object(schema.properties?.params) + const extension = resources.get(method) + if (extension !== undefined) { + const extra = z.toJSONSchema(extension) + if (params.type !== 'object') throw new Error(`Host fields require object parameters: ${method}`) + params.properties = { ...params.properties, ...extra.properties } + params.required = [...new Set([...(params.required ?? []), ...(extra.required ?? [])])] + } + schema.properties = { id: { type: 'string' }, ...schema.properties } + schema.required = ['id', 'method', 'params'] + delete schema.$schema + const names = new Set() + const values = new Map>() + for (const variant of variants(params)) { + for (const [name, raw] of Object.entries(variant.properties ?? {})) { + names.add(name) + const property = object(raw) + const item = property.type === 'array' && !Array.isArray(property.items) ? object(property.items) : property + const allowed = item.enum?.filter((value): value is string => typeof value === 'string') + if (allowed?.length) { + const union = values.get(name) ?? new Set() + for (const value of allowed) union.add(value) + values.set(name, union) + } + } + } + fields.set(method, names) + if (values.size) enums.set(method, values) + } + const quoted = (values: Iterable) => [...values].sort().map(value => JSON.stringify(value)).join(', ') + const swift = [ + '// Generated by native/generate-protocol.ts from src/protocol.ts. Do not edit.', + 'enum NativeProtocol {', + ` static let modelMethods: [String] = [${quoted(modelMethods)}]`, + ' static let fields: [String: Set] = [', + ...[...fields].sort(([a], [b]) => a.localeCompare(b)).map(([method, names]) => ` ${JSON.stringify(method)}: [${quoted(names)}],`), + ' ]', + ' static let stringEnums: [String: [String: Set]] = [', + ...[...enums].sort(([a], [b]) => a.localeCompare(b)).map(([method, properties]) => + ` ${JSON.stringify(method)}: [${[...properties].sort(([a], [b]) => a.localeCompare(b)) + .map(([name, values]) => `${JSON.stringify(name)}: [${quoted(values)}]`).join(', ')}],`), + ' ]', + '}', + '', + ].join('\n') + return { swift, json: JSON.stringify({ + $schema: 'https://json-schema.org/draft/2020-12/schema', + $comment: 'Generated by native/generate-protocol.ts from src/protocol.ts. Do not edit.', + title: 'dsh-cua native request', + description: 'Host resource fields and private continuations are not model tool parameters. App handles and snapshots belong to one helper; coordinates are window-local macOS points.', + oneOf: schemas, + }, null, 2) + '\n' } +} + +if (import.meta.main) { + const generated = renderNativeProtocol() + for (const [file, content] of [ + ['protocol.schema.json', generated.json], ['Sources/CuaNative/NativeProtocol.generated.swift', generated.swift], + ]) { + const path = new URL(file, import.meta.url) + if (process.argv.includes('--check')) { + if (await readFile(path, 'utf8') !== content) throw new Error(`Stale generated CUA protocol: ${file}`) + } else await writeFile(path, content) + } +} diff --git a/packages/computer-use/cua/native/protocol.schema.json b/packages/computer-use/cua/native/protocol.schema.json index d76e16d7a3..a818389d35 100644 --- a/packages/computer-use/cua/native/protocol.schema.json +++ b/packages/computer-use/cua/native/protocol.schema.json @@ -1,132 +1,206 @@ { "$schema": "https://json-schema.org/draft/2020-12/schema", + "$comment": "Generated by native/generate-protocol.ts from src/protocol.ts. Do not edit.", "title": "dsh-cua native request", - "description": "App handles identify one acquired process lifetime in one helper. Window observations retain separate snapshot generations. Window coordinates are top-left local macOS points. Scroll deltas use whole lines; positive Y reveals above and positive X reveals left.", + "description": "Host resource fields and private continuations are not model tool parameters. App handles and snapshots belong to one helper; coordinates are window-local macOS points.", "oneOf": [ { "type": "object", - "additionalProperties": false, "properties": { "id": { "type": "string" }, "method": { + "type": "string", "const": "getCapabilities" }, "params": { "type": "object", - "additionalProperties": false, "properties": {}, - "required": [] + "additionalProperties": false } }, "required": [ "id", "method", "params" - ] + ], + "additionalProperties": false }, { "type": "object", - "additionalProperties": false, "properties": { "id": { "type": "string" }, "method": { + "type": "string", "const": "listApps" }, "params": { "type": "object", - "additionalProperties": false, "properties": {}, - "required": [] + "additionalProperties": false } }, "required": [ "id", "method", "params" - ] + ], + "additionalProperties": false }, { "type": "object", - "additionalProperties": false, "properties": { "id": { "type": "string" }, "method": { + "type": "string", + "const": "findApps" + }, + "params": { + "type": "object", + "properties": { + "query": { + "type": "string", + "minLength": 1, + "pattern": "\\S" + }, + "maxAppMatches": { + "type": "integer", + "minimum": 1, + "maximum": 1000 + } + }, + "required": [ + "query", + "maxAppMatches" + ], + "additionalProperties": false + } + }, + "required": [ + "id", + "method", + "params" + ], + "additionalProperties": false + }, + { + "type": "object", + "properties": { + "id": { + "type": "string" + }, + "method": { + "type": "string", "const": "getApp" }, "params": { "type": "object", - "additionalProperties": false, "properties": { "selector": { - "type": "string" + "type": "string", + "minLength": 1 } }, "required": [ "selector" - ] + ], + "additionalProperties": false } }, "required": [ "id", "method", "params" - ] + ], + "additionalProperties": false }, { "type": "object", - "additionalProperties": false, "properties": { "id": { "type": "string" }, "method": { - "const": "listWindows" + "type": "string", + "const": "activate" }, "params": { "type": "object", - "additionalProperties": false, "properties": { "appHandle": { - "type": "string" + "type": "string", + "minLength": 1 } }, "required": [ "appHandle" - ] + ], + "additionalProperties": false } }, "required": [ "id", "method", "params" - ] + ], + "additionalProperties": false }, { "type": "object", - "additionalProperties": false, "properties": { "id": { "type": "string" }, "method": { + "type": "string", + "const": "listWindows" + }, + "params": { + "type": "object", + "properties": { + "appHandle": { + "type": "string", + "minLength": 1 + } + }, + "required": [ + "appHandle" + ], + "additionalProperties": false + } + }, + "required": [ + "id", + "method", + "params" + ], + "additionalProperties": false + }, + { + "type": "object", + "properties": { + "id": { + "type": "string" + }, + "method": { + "type": "string", "const": "getState" }, "params": { "type": "object", - "additionalProperties": false, "properties": { "appHandle": { - "type": "string" + "type": "string", + "minLength": 1 }, "windowId": { "type": "integer", - "minimum": 1, + "exclusiveMinimum": 0, "maximum": 4294967295 }, "maxElements": { @@ -144,71 +218,152 @@ "appHandle", "maxElements", "maxDepth" - ] + ], + "additionalProperties": false } }, "required": [ "id", "method", "params" - ] + ], + "additionalProperties": false }, { "type": "object", - "additionalProperties": false, "properties": { "id": { "type": "string" }, "method": { + "type": "string", "const": "click" }, "params": { - "type": "object", - "additionalProperties": false, - "properties": { - "appHandle": { - "type": "string" + "anyOf": [ + { + "type": "object", + "properties": { + "appHandle": { + "type": "string", + "minLength": 1 + }, + "snapshotId": { + "type": "string", + "minLength": 1 + }, + "elementIndex": { + "type": "integer", + "minimum": 0, + "maximum": 9007199254740991 + }, + "mouseButton": { + "type": "string", + "enum": [ + "left", + "right", + "middle" + ] + }, + "modifiers": { + "type": "array", + "items": { + "type": "string", + "enum": [ + "command", + "control", + "option", + "shift", + "fn" + ] + } + }, + "clickCount": { + "anyOf": [ + { + "type": "number", + "const": 1 + }, + { + "type": "number", + "const": 2 + } + ] + } + }, + "required": [ + "appHandle", + "snapshotId", + "elementIndex" + ], + "additionalProperties": false }, - "snapshotId": { - "type": "string" - }, - "elementIndex": { - "type": "integer", - "minimum": 0 - }, - "mouseButton": { - "type": "string", - "enum": [ - "left", - "right", - "middle" - ] - }, - "clickCount": { - "enum": [ - 1, - 2 - ] - }, - "modifiers": { - "type": "array", - "items": { - "type": "string", - "enum": [ - "command", - "control", - "option", - "shift", - "fn" - ] - } + { + "type": "object", + "properties": { + "appHandle": { + "type": "string", + "minLength": 1 + }, + "windowId": { + "type": "integer", + "exclusiveMinimum": 0, + "maximum": 4294967295 + }, + "snapshotId": { + "type": "string", + "minLength": 1 + }, + "x": { + "type": "number", + "minimum": 0 + }, + "y": { + "type": "number", + "minimum": 0 + }, + "mouseButton": { + "type": "string", + "enum": [ + "left", + "right", + "middle" + ] + }, + "modifiers": { + "type": "array", + "items": { + "type": "string", + "enum": [ + "command", + "control", + "option", + "shift", + "fn" + ] + } + }, + "clickCount": { + "anyOf": [ + { + "type": "number", + "const": 1 + }, + { + "type": "number", + "const": 2 + } + ] + } + }, + "required": [ + "appHandle", + "windowId", + "x", + "y" + ], + "additionalProperties": false } - }, - "required": [ - "appHandle", - "snapshotId", - "elementIndex" ] } }, @@ -216,104 +371,34 @@ "id", "method", "params" - ] + ], + "additionalProperties": false }, { "type": "object", - "additionalProperties": false, - "properties": { - "id": { - "type": "string" - }, - "method": { - "const": "click" - }, - "params": { - "type": "object", - "additionalProperties": false, - "properties": { - "appHandle": { - "type": "string" - }, - "windowId": { - "type": "integer", - "minimum": 1, - "maximum": 4294967295 - }, - "snapshotId": { - "type": "string" - }, - "x": { - "type": "number" - }, - "y": { - "type": "number" - }, - "mouseButton": { - "type": "string", - "enum": [ - "left", - "right", - "middle" - ] - }, - "clickCount": { - "enum": [ - 1, - 2 - ] - }, - "modifiers": { - "type": "array", - "items": { - "type": "string", - "enum": [ - "command", - "control", - "option", - "shift", - "fn" - ] - } - } - }, - "required": [ - "appHandle", - "windowId", - "x", - "y" - ] - } - }, - "required": [ - "id", - "method", - "params" - ] - }, - { - "type": "object", - "additionalProperties": false, "properties": { "id": { "type": "string" }, "method": { + "type": "string", "const": "setValue" }, "params": { "type": "object", - "additionalProperties": false, "properties": { "appHandle": { - "type": "string" + "type": "string", + "minLength": 1 }, "snapshotId": { - "type": "string" + "type": "string", + "minLength": 1 }, "elementIndex": { "type": "integer", - "minimum": 0 + "minimum": 0, + "maximum": 9007199254740991 }, "value": { "type": "string" @@ -324,394 +409,42 @@ "snapshotId", "elementIndex", "value" - ] - } - }, - "required": [ - "id", - "method", - "params" - ] - }, - { - "type": "object", - "additionalProperties": false, - "properties": { - "id": { - "type": "string" - }, - "method": { - "const": "typeText" - }, - "params": { - "type": "object", - "additionalProperties": false, - "properties": { - "appHandle": { - "type": "string" - }, - "text": { - "type": "string" - } - }, - "required": [ - "appHandle", - "text" - ] - } - }, - "required": [ - "id", - "method", - "params" - ] - }, - { - "type": "object", - "additionalProperties": false, - "properties": { - "id": { - "type": "string" - }, - "method": { - "const": "pressKey" - }, - "params": { - "type": "object", - "additionalProperties": false, - "properties": { - "appHandle": { - "type": "string" - }, - "key": { - "type": "string" - }, - "modifiers": { - "type": "array", - "items": { - "enum": [ - "command", - "cmd", - "shift", - "option", - "alt", - "control", - "ctrl", - "fn" - ] - } - } - }, - "required": [ - "appHandle", - "key" - ] - } - }, - "required": [ - "id", - "method", - "params" - ] - }, - { - "type": "object", - "additionalProperties": false, - "properties": { - "id": { - "type": "string" - }, - "method": { - "const": "scroll" - }, - "params": { - "type": "object", - "additionalProperties": false, - "properties": { - "appHandle": { - "type": "string" - }, - "windowId": { - "type": "integer", - "minimum": 1, - "maximum": 4294967295 - }, - "snapshotId": { - "type": "string" - }, - "x": { - "type": "number" - }, - "y": { - "type": "number" - }, - "deltaLinesX": { - "type": "integer", - "minimum": -10, - "maximum": 10 - }, - "deltaLinesY": { - "type": "integer", - "minimum": -10, - "maximum": 10 - } - }, - "required": [ - "appHandle", - "windowId", - "x", - "y" ], - "anyOf": [ - { - "required": [ - "deltaLinesX" - ], - "properties": { - "deltaLinesX": { - "not": { - "const": 0 - } - } - } - }, - { - "required": [ - "deltaLinesY" - ], - "properties": { - "deltaLinesY": { - "not": { - "const": 0 - } - } - } - } - ] + "additionalProperties": false } }, "required": [ "id", "method", "params" - ] + ], + "additionalProperties": false }, { "type": "object", - "additionalProperties": false, - "properties": { - "id": { - "type": "string" - }, - "method": { - "const": "drag" - }, - "params": { - "type": "object", - "additionalProperties": false, - "properties": { - "appHandle": { - "type": "string" - }, - "windowId": { - "type": "integer", - "minimum": 1, - "maximum": 4294967295 - }, - "snapshotId": { - "type": "string" - }, - "fromX": { - "type": "number" - }, - "fromY": { - "type": "number" - }, - "toX": { - "type": "number" - }, - "toY": { - "type": "number" - }, - "durationMs": { - "type": "integer", - "minimum": 1, - "maximum": 5000 - }, - "steps": { - "type": "integer", - "minimum": 1, - "maximum": 240 - }, - "mouseButton": { - "type": "string", - "enum": [ - "left", - "right", - "middle" - ] - }, - "modifiers": { - "type": "array", - "items": { - "type": "string", - "enum": [ - "command", - "control", - "option", - "shift", - "fn" - ] - } - } - }, - "required": [ - "appHandle", - "windowId", - "fromX", - "fromY", - "toX", - "toY", - "durationMs", - "steps" - ] - } - }, - "required": [ - "id", - "method", - "params" - ] - }, - { - "type": "object", - "additionalProperties": false, - "properties": { - "id": { - "type": "string" - }, - "method": { - "const": "screenshot" - }, - "params": { - "type": "object", - "additionalProperties": false, - "properties": { - "appHandle": { - "type": "string" - }, - "windowId": { - "type": "integer", - "minimum": 1, - "maximum": 4294967295 - } - }, - "required": [ - "appHandle", - "windowId" - ] - } - }, - "required": [ - "id", - "method", - "params" - ] - }, - { - "type": "object", - "additionalProperties": false, - "properties": { - "id": { - "type": "string" - }, - "method": { - "const": "performSecondaryAction" - }, - "params": { - "type": "object", - "additionalProperties": false, - "properties": { - "appHandle": { - "type": "string" - }, - "snapshotId": { - "type": "string" - }, - "elementIndex": { - "type": "integer", - "minimum": 0 - }, - "action": { - "type": "string" - } - }, - "required": [ - "appHandle", - "snapshotId", - "elementIndex", - "action" - ] - } - }, - "required": [ - "id", - "method", - "params" - ] - }, - { - "type": "object", - "additionalProperties": false, - "properties": { - "id": { - "type": "string" - }, - "method": { - "const": "activate" - }, - "params": { - "type": "object", - "additionalProperties": false, - "properties": { - "appHandle": { - "type": "string" - } - }, - "required": [ - "appHandle" - ] - } - }, - "required": [ - "id", - "method", - "params" - ] - }, - { - "type": "object", - "additionalProperties": false, "properties": { "id": { "type": "string" }, "method": { + "type": "string", "const": "selectText" }, "params": { "type": "object", - "additionalProperties": false, "properties": { "appHandle": { - "type": "string" + "type": "string", + "minLength": 1 }, "snapshotId": { - "type": "string" + "type": "string", + "minLength": 1 }, "elementIndex": { "type": "integer", - "minimum": 0 + "minimum": 0, + "maximum": 9007199254740991 }, "text": { "type": "string", @@ -737,31 +470,115 @@ "snapshotId", "elementIndex", "text" - ] + ], + "additionalProperties": false } }, "required": [ "id", "method", "params" - ] + ], + "additionalProperties": false }, { "type": "object", - "additionalProperties": false, "properties": { "id": { "type": "string" }, "method": { + "type": "string", + "const": "performSecondaryAction" + }, + "params": { + "type": "object", + "properties": { + "appHandle": { + "type": "string", + "minLength": 1 + }, + "snapshotId": { + "type": "string", + "minLength": 1 + }, + "elementIndex": { + "type": "integer", + "minimum": 0, + "maximum": 9007199254740991 + }, + "action": { + "type": "string", + "minLength": 1 + } + }, + "required": [ + "appHandle", + "snapshotId", + "elementIndex", + "action" + ], + "additionalProperties": false + } + }, + "required": [ + "id", + "method", + "params" + ], + "additionalProperties": false + }, + { + "type": "object", + "properties": { + "id": { + "type": "string" + }, + "method": { + "type": "string", + "const": "typeText" + }, + "params": { + "type": "object", + "properties": { + "appHandle": { + "type": "string", + "minLength": 1 + }, + "text": { + "type": "string" + } + }, + "required": [ + "appHandle", + "text" + ], + "additionalProperties": false + } + }, + "required": [ + "id", + "method", + "params" + ], + "additionalProperties": false + }, + { + "type": "object", + "properties": { + "id": { + "type": "string" + }, + "method": { + "type": "string", "const": "paste" }, "params": { "type": "object", - "additionalProperties": false, "properties": { "appHandle": { - "type": "string" + "type": "string", + "minLength": 1 }, "text": { "type": "string" @@ -775,7 +592,8 @@ }, "maxClipboardBytes": { "type": "integer", - "minimum": 1024 + "minimum": 1024, + "maximum": 9007199254740991 }, "maxElements": { "type": "integer", @@ -794,64 +612,272 @@ "maxClipboardBytes", "maxElements", "maxDepth" - ] + ], + "additionalProperties": false } }, "required": [ "id", "method", "params" - ] + ], + "additionalProperties": false }, { "type": "object", - "additionalProperties": false, "properties": { "id": { "type": "string" }, "method": { - "const": "findApps" + "type": "string", + "const": "pressKey" }, "params": { "type": "object", - "additionalProperties": false, "properties": { - "query": { + "appHandle": { "type": "string", "minLength": 1 }, - "maxAppMatches": { - "type": "integer", - "minimum": 1, - "maximum": 1000 + "key": { + "type": "string", + "minLength": 1 + }, + "modifiers": { + "type": "array", + "items": { + "type": "string", + "enum": [ + "command", + "control", + "option", + "shift", + "fn" + ] + } } }, "required": [ - "query", - "maxAppMatches" - ] + "appHandle", + "key" + ], + "additionalProperties": false } }, "required": [ "id", "method", "params" - ] + ], + "additionalProperties": false }, { "type": "object", - "additionalProperties": false, "properties": { "id": { "type": "string" }, "method": { + "type": "string", + "const": "scroll" + }, + "params": { + "type": "object", + "properties": { + "appHandle": { + "type": "string", + "minLength": 1 + }, + "windowId": { + "type": "integer", + "exclusiveMinimum": 0, + "maximum": 4294967295 + }, + "snapshotId": { + "type": "string", + "minLength": 1 + }, + "x": { + "type": "number", + "minimum": 0 + }, + "y": { + "type": "number", + "minimum": 0 + }, + "deltaLinesX": { + "type": "integer", + "minimum": -10, + "maximum": 10 + }, + "deltaLinesY": { + "type": "integer", + "minimum": -10, + "maximum": 10 + } + }, + "required": [ + "appHandle", + "windowId", + "x", + "y", + "deltaLinesX", + "deltaLinesY" + ], + "additionalProperties": false + } + }, + "required": [ + "id", + "method", + "params" + ], + "additionalProperties": false + }, + { + "type": "object", + "properties": { + "id": { + "type": "string" + }, + "method": { + "type": "string", + "const": "drag" + }, + "params": { + "type": "object", + "properties": { + "appHandle": { + "type": "string", + "minLength": 1 + }, + "windowId": { + "type": "integer", + "exclusiveMinimum": 0, + "maximum": 4294967295 + }, + "mouseButton": { + "type": "string", + "enum": [ + "left", + "right", + "middle" + ] + }, + "modifiers": { + "type": "array", + "items": { + "type": "string", + "enum": [ + "command", + "control", + "option", + "shift", + "fn" + ] + } + }, + "snapshotId": { + "type": "string", + "minLength": 1 + }, + "fromX": { + "type": "number", + "minimum": 0 + }, + "fromY": { + "type": "number", + "minimum": 0 + }, + "toX": { + "type": "number", + "minimum": 0 + }, + "toY": { + "type": "number", + "minimum": 0 + }, + "durationMs": { + "type": "integer", + "minimum": 1, + "maximum": 5000 + }, + "steps": { + "type": "integer", + "minimum": 1, + "maximum": 240 + } + }, + "required": [ + "appHandle", + "windowId", + "fromX", + "fromY", + "toX", + "toY", + "durationMs", + "steps" + ], + "additionalProperties": false + } + }, + "required": [ + "id", + "method", + "params" + ], + "additionalProperties": false + }, + { + "type": "object", + "properties": { + "id": { + "type": "string" + }, + "method": { + "type": "string", + "const": "screenshot" + }, + "params": { + "type": "object", + "properties": { + "appHandle": { + "type": "string", + "minLength": 1 + }, + "windowId": { + "type": "integer", + "exclusiveMinimum": 0, + "maximum": 4294967295 + } + }, + "required": [ + "appHandle", + "windowId" + ], + "additionalProperties": false + } + }, + "required": [ + "id", + "method", + "params" + ], + "additionalProperties": false + }, + { + "type": "object", + "properties": { + "id": { + "type": "string" + }, + "method": { + "type": "string", "const": "launchApp" }, "params": { "type": "object", - "additionalProperties": false, "properties": { "appHandle": { "type": "string", @@ -860,7 +886,8 @@ }, "required": [ "appHandle" - ] + ], + "additionalProperties": false } }, "required": [ @@ -868,7 +895,7 @@ "method", "params" ], - "description": "Host-only continuation after application consent; not exposed by the model tool." + "additionalProperties": false } ] } diff --git a/packages/computer-use/cua/native/tests/capabilities_e2e.py b/packages/computer-use/cua/native/tests/capabilities_e2e.py index b7cc23e4ac..6b03ae5dda 100644 --- a/packages/computer-use/cua/native/tests/capabilities_e2e.py +++ b/packages/computer-use/cua/native/tests/capabilities_e2e.py @@ -89,8 +89,9 @@ def main(): call("selectText", {**target, "text": "target", "prefix": "第二😀 "}) selected = wait_for_state(state_file, lambda value: value["selection"] == [17, 6]) assert selected["text"] == "第一😀 target，第二😀 target.", selected - call("paste", {**owner, "text": "替换", "format": "text", "maxClipboardBytes": 33554432, "maxElements": 2000, "maxDepth": 25}) - state = wait_for_state(state_file, lambda value: value["text"] == "第一😀 target，第二😀 替换.") + call("paste", {**owner, "text": "替换\n第二行\t保留", "format": "text", "maxClipboardBytes": 33554432, "maxElements": 2000, "maxDepth": 25}) + state = wait_for_state(state_file, lambda value: value["text"] == "第一😀 target，第二😀 替换\n第二行\t保留.") + assert state["pasteAttempts"] == [True], state observed = call("getState", {**owner, "maxElements": 2000, "maxDepth": 25}) field = next(item for item in observed["elements"] if item["role"] == "AXTextArea") target = {**owner, "snapshotId": observed["snapshotId"], "elementIndex": field["index"]} diff --git a/packages/computer-use/cua/native/tests/swift/AXTextFormatterTests.swift b/packages/computer-use/cua/native/tests/swift/AXTextFormatterTests.swift new file mode 100644 index 0000000000..549af45f56 --- /dev/null +++ b/packages/computer-use/cua/native/tests/swift/AXTextFormatterTests.swift @@ -0,0 +1,23 @@ +// Formatting fixtures pin model-visible text independently of accessibility APIs. +import Foundation +import Testing +@testable import CuaNative + +@Test func formatsEditableTextAndSelectionWithoutAssigningAnotherIndex() { + let node = AXTextNode(index: 7, role: "AXTextArea", title: "Editor", value: "a😀 target", description: "Document", + actions: [], enabled: true, valueWritable: true, selectionWritable: true, selected: nil, + selectedRange: CFRange(location: 4, length: 6), selectedText: "target") + #expect(AXTextFormatter.line(node, level: 1) + == " - [7] AXTextArea valueWritable=true selectionWritable=true \"Editor\" = \"a😀 target\" (Document) selection=[4, 6] selectedText=\"target\"") +} + +@Test func formatsDisabledAndSelectedControlsWithoutInventingCapabilities() { + let node = AXTextNode(index: nil, role: "AXButton", title: "Save", value: nil, description: nil, + actions: ["AXPress"], enabled: false, valueWritable: false, selectionWritable: false, + selected: false, selectedRange: nil, selectedText: nil) + #expect(AXTextFormatter.line(node, level: 0) == "- AXButton disabled selected=false \"Save\" actions=[AXPress]") + let empty = AXTextNode(index: nil, role: "AXWindow", title: "", value: "", description: "", + actions: [], enabled: nil, valueWritable: false, selectionWritable: false, + selected: nil, selectedRange: nil, selectedText: "") + #expect(AXTextFormatter.line(empty, level: 0) == "- AXWindow") +} diff --git a/packages/computer-use/cua/native/tests/swift/InputMappingTests.swift b/packages/computer-use/cua/native/tests/swift/InputMappingTests.swift index 25ca984d5a..4d70f75138 100644 --- a/packages/computer-use/cua/native/tests/swift/InputMappingTests.swift +++ b/packages/computer-use/cua/native/tests/swift/InputMappingTests.swift @@ -25,3 +25,10 @@ func keypadKeysRetainTheirPhysicalIdentity(_ key: String, _ code: UInt16) throws #expect(WindowPointerInput.up(button) == .otherMouseUp) #expect(throws: NativeFailure.self) { try WindowPointerInput.button("unsupported") } } + +@Test func everyDeclaredPointerButtonAndModifierHasANativeMapping() throws { + let buttons = try #require(NativeProtocol.stringEnums["click"]?["mouseButton"]) + let modifiers = try #require(NativeProtocol.stringEnums["click"]?["modifiers"]) + for button in buttons { _ = try WindowPointerInput.button(button) } + for modifier in modifiers { _ = try KeyboardInput.modifierFlags([modifier]) } +} diff --git a/packages/computer-use/cua/native/upstream.json b/packages/computer-use/cua/native/upstream.json index 0f322402f0..59b52d4e75 100644 --- a/packages/computer-use/cua/native/upstream.json +++ b/packages/computer-use/cua/native/upstream.json @@ -37,10 +37,12 @@ } ], "localFiles": [ + "Sources/CuaNative/AXTextFormatter.swift", "Sources/CuaNative/AXWorker.swift", "Sources/CuaNative/ApplicationDiscovery.swift", "Sources/CuaNative/ClipboardPaste.swift", "Sources/CuaNative/Main.swift", + "Sources/CuaNative/NativeProtocol.generated.swift", "Sources/CuaNative/Protocol.swift", "Sources/CuaNative/RequestAdmission.swift", "Sources/CuaNative/RequestCoordinator.swift", diff --git a/packages/computer-use/cua/src/api.ts b/packages/computer-use/cua/src/api.ts index 7c897b712f..2c532b3c7c 100644 --- a/packages/computer-use/cua/src/api.ts +++ b/packages/computer-use/cua/src/api.ts @@ -2,7 +2,7 @@ import { Buffer } from 'node:buffer' import type { JsonValue } from '@deepseek-ai/dsh-util-values' import { ActionResult, AppResult, AppsResult, CapabilitiesResult, FoundAppsResult, Request, ScreenshotResult, StateResult, WindowsResult } from './protocol.ts' -import type { CuaApi, CuaApp, CuaAppHandle, CuaDispatch, CuaObservationOptions, CuaObservationSink, CuaSurface, CuaWindow } from './types.ts' +import type { CuaApi, CuaApp, CuaAppHandle, CuaDispatch, CuaObservationOptions, CuaObservationSink, CuaOperation, CuaSurface, CuaWindow } from './types.ts' export type * from './types.ts' function keyChord(chord: string): { key: string; modifiers: string[] } { @@ -33,11 +33,8 @@ function keyChord(chord: string): { key: string; modifiers: string[] } { * @returns Objects independent of variable storage and evaluator lifecycle. */ export function createCuaApi(dispatch: CuaDispatch, sink?: CuaObservationSink): CuaApi { - const call = (method: string, params: Record) => { - Request.parse({ method, params }) - return dispatch({ method, params }) - } - const action = async (method: string, params: Record) => ActionResult.parse(await call(method, params)) + const call = (method: CuaOperation['method'], params: Record) => dispatch(Request.parse({ method, params })) + const action = async (method: CuaOperation['method'], params: Record) => ActionResult.parse(await call(method, params)) async function emit(value: object, options?: CuaObservationOptions): Promise { if (options?.emit !== false) await sink?.text(JSON.stringify(value)) } diff --git a/packages/computer-use/cua/src/index.ts b/packages/computer-use/cua/src/index.ts index d7666d7aef..162a63ef4c 100644 --- a/packages/computer-use/cua/src/index.ts +++ b/packages/computer-use/cua/src/index.ts @@ -252,7 +252,7 @@ function mount(ctx: Context, config: Config, helperPath: string): void { ? await withFileLock(join(tmpdir(), 'dsh-cua-system-clipboard'), requestNative, { waitMs: config.timeoutMs, signal }) : await requestNative() if (method === 'getApp') { - const prepared = AppResult.extend({ pid: z.number().int().positive().nullable() }).parse(result) + const prepared = AppResult.extend({ pid: AppResult.shape.pid.nullable() }).parse(result) const { authorization } = z.object({ authorization: ApplicationAuthorizationResult }).parse(result) await access.select(agent, client, prepared.appHandle, authorization, signal) signal.throwIfAborted() diff --git a/packages/computer-use/cua/src/protocol.ts b/packages/computer-use/cua/src/protocol.ts index 449c352401..91a9ce60f1 100644 --- a/packages/computer-use/cua/src/protocol.ts +++ b/packages/computer-use/cua/src/protocol.ts @@ -3,7 +3,8 @@ import { Buffer } from 'node:buffer' import { z } from 'zod' import type { CuaAppHandle, CuaSnapshotId } from './types.ts' -const id = z.number().int().positive() +const id = z.number().int().positive().max(2 ** 32 - 1) +const pid = z.number().int().positive().max(2 ** 31 - 1) const point = z.number().nonnegative() const app = { appHandle: z.string().min(1) } const pointer = { mouseButton: z.enum(['left', 'right', 'middle']).optional(), @@ -23,7 +24,7 @@ const imageBase64 = z.string().min(1).refine(value => Buffer.from(value, 'base64 message: 'Expected canonical base64 image data', }) -/** Complete native request schema; resource bounds are supplied by the host. */ +/** Model-submittable operation parameters; Host resource fields and private continuations are separate. */ export const Request = z.discriminatedUnion('method', [ operation('getCapabilities', {}), operation('listApps', {}), operation('findApps', { query: z.string().min(1).regex(/\S/) }), operation('getApp', { selector: z.string().min(1) }), @@ -46,24 +47,36 @@ export const Request = z.discriminatedUnion('method', [ operation('screenshot', window), ]) +/** Resource parameters supplied by the Host, never accepted from model tool arguments. */ +export const NativeResourceParameters = { + getState: z.object({ maxElements: z.number().int().min(1).max(10000), maxDepth: z.number().int().min(1).max(100) }).strict(), + drag: z.object({ durationMs: z.number().int().min(1).max(5000), steps: z.number().int().min(1).max(240) }).strict(), + paste: z.object({ maxClipboardBytes: z.number().int().min(1024), + maxElements: z.number().int().min(1).max(10000), maxDepth: z.number().int().min(1).max(100), + }).strict(), + findApps: z.object({ maxAppMatches: z.number().int().min(1).max(1000) }).strict(), +} +/** Executor-only continuation after application consent; absent from the model Request union. */ +export const NativeOnlyOperations = [operation('launchApp', app)] as const + /** Process identity returned only after a successful app selection. */ export const AppResult = z.object({ - appHandle: z.string().min(1).transform(value => value as CuaAppHandle), pid: id, name: z.string(), bundleId: z.string().optional(), + appHandle: z.string().min(1).transform(value => value as CuaAppHandle), pid, name: z.string(), bundleId: z.string().optional(), }) /** Running application records from the helper. */ -export const AppsResult = z.array(z.object({ pid: id, name: z.string(), bundleId: z.string().optional() })) +export const AppsResult = z.array(z.object({ pid, name: z.string(), bundleId: z.string().optional() })) /** Installed application matches keep running and stopped records distinct. */ export const FoundAppsResult = z.array(z.intersection(z.object({ id: z.string().min(1), displayName: z.string(), path: z.string().min(1) }), - z.discriminatedUnion('isRunning', [z.object({ isRunning: z.literal(true), pid: id }), z.object({ isRunning: z.literal(false) })]))) + z.discriminatedUnion('isRunning', [z.object({ isRunning: z.literal(true), pid }), z.object({ isRunning: z.literal(false) })]))) /** Current windows and their global point bounds. */ export const WindowsResult = z.array(z.object({ - windowId: id, pid: id, appName: z.string(), title: z.string(), bounds, + windowId: id, pid, appName: z.string(), title: z.string(), bounds, isOnScreen: z.boolean(), onCurrentSpace: z.boolean().nullable(), })) /** Observed state with window and process identity for stale-target rejection. */ export const StateResult = z.object({ appHandle: z.string().min(1).transform(value => value as CuaAppHandle), - pid: id, windowId: id.nullable(), + pid, windowId: id.nullable(), focusedWindowId: id.nullable().optional(), snapshotId: z.string().min(1).transform(value => value as CuaSnapshotId), tree: z.string(), windows: WindowsResult, truncated: z.boolean(), diff --git a/packages/computer-use/cua/src/types.ts b/packages/computer-use/cua/src/types.ts index 2ba48abf7b..4f3569c64b 100644 --- a/packages/computer-use/cua/src/types.ts +++ b/packages/computer-use/cua/src/types.ts @@ -1,13 +1,16 @@ /** Native application objects independent of evaluator ownership and persistence. */ import type { Branded } from '@deepseek-ai/dsh-brand' import type { JsonValue } from '@deepseek-ai/dsh-util-values' +import type { z } from 'zod' +import type { ActionResult, CapabilitiesResult, FoundAppsResult, Request, WindowsResult } from './protocol.ts' -/** Helper-issued application identity, bound to one process lifetime. */ +/** Helper-issued application selection; public app objects bind it to one process lifetime. */ export type CuaAppHandle = Branded<'CuaAppHandle'> /** Helper-issued accessibility observation identity. */ export type CuaSnapshotId = Branded<'CuaSnapshotId'> /** One operation sent through an authorized and logged host dispatcher. */ -export interface CuaOperation { method: string; params: Record } +export type CuaOperation = z.output +type CuaParameters = Extract['params'] /** Dispatch exactly once; a failed action must not be retried implicitly. */ export type CuaDispatch = (operation: CuaOperation) => Promise /** Optional output owned by the caller; persistence and image admission are external. */ @@ -27,48 +30,32 @@ export interface CuaObservationSink { export interface CuaObservationOptions { emit?: boolean } /** Diffing is not implemented; all reads return a complete tree. */ export interface CuaAxOptions extends CuaObservationOptions { disableDiffing?: boolean } -/** Available permissions and operations reported by the selected native implementation. */ -export interface CuaCapabilities { - platform: string - permissions: { accessibility: boolean; screenRecording: boolean } - /** Null means the helper could not verify a usable console session. */ - sessionLocked: boolean | null - methods: string[] - coordinateSpace: 'window-points' - browser: false -} +/** Reported permissions and operations; null sessionLocked means no usable console session was verified. */ +export type CuaCapabilities = z.output /** App discovery result accepted by getApp. */ -export interface CuaAppInfo { id: string; displayName: string; isRunning: true; pid: number } +export type CuaAppInfo = Pick, 'id' | 'displayName' | 'isRunning' | 'pid'> /** An installed application match; running matches include a process identifier. */ -export type CuaFoundAppInfo = { id: string; displayName: string; path: string } & ({ isRunning: true; pid: number } | { isRunning: false }) +export type CuaFoundAppInfo = z.output[number] /** Window bounds in global macOS points. Local operations use an origin of [0,0]. */ -export interface CuaBounds { x: number; y: number; width: number; height: number } -/** Observed window metadata; identifiers are valid only while the window exists. */ -export interface CuaWindowInfo { - windowId: number - pid: number - appName: string - title: string - bounds: CuaBounds - isOnScreen: boolean - onCurrentSpace: boolean | null -} +export type CuaBounds = CuaWindowInfo['bounds'] +/** Observed window metadata; identifiers are valid only while the window exists, and null Space membership is unknown. */ +export type CuaWindowInfo = z.output[number] /** Successful input dispatch; application outcome still requires observation. */ -export interface CuaActionResult { dispatched: true; verified: false } +export type CuaActionResult = z.output /** Modifiers attached to a window-directed pointer gesture. */ -export type CuaPointerModifier = 'command' | 'control' | 'option' | 'shift' | 'fn' +export type CuaPointerModifier = NonNullable['modifiers']>[number] /** Buttons supported by window-directed clicks and drags. */ -export type CuaMouseButton = 'left' | 'right' | 'middle' +export type CuaMouseButton = NonNullable['mouseButton']> /** Left clicks support one or two presses; right and middle clicks support one. */ -export interface CuaClickOptions { mouseButton?: CuaMouseButton; clickCount?: 1 | 2; modifiers?: readonly CuaPointerModifier[] } +export type CuaClickOptions = Pick, 'mouseButton' | 'clickCount' | 'modifiers'> /** One same-window drag; omitted button is left and omitted modifiers are empty. */ -export interface CuaDragOptions { mouseButton?: CuaMouseButton; modifiers?: readonly CuaPointerModifier[] } +export type CuaDragOptions = Pick, 'mouseButton' | 'modifiers'> /** Clipboard representation delivered through the application's Paste command. */ -export interface CuaPasteOptions { format?: 'text' | 'html' } +export type CuaPasteOptions = Pick, 'format'> /** Adjacent literal context disambiguates repeated text; cursor modes collapse the matched range. */ -export interface CuaSelectTextOptions { prefix?: string; suffix?: string; selectionType?: 'text' | 'cursor_before' | 'cursor_after' } +export type CuaSelectTextOptions = Pick, 'prefix' | 'suffix' | 'selectionType'> /** Positive deltas reveal content above/left; each axis is an integer from -10 to 10. */ -export interface CuaScrollOptions { x: number; y: number; deltaLinesX: number; deltaLinesY: number } +export type CuaScrollOptions = Pick, 'x' | 'y' | 'deltaLinesX' | 'deltaLinesY'> /** An observed AX tree and independently captured PNG; capture is not atomic. */ export interface CuaObservation { state: string; screenshot: Uint8Array } /** diff --git a/packages/computer-use/cua/tests/protocol-generation.spec.ts b/packages/computer-use/cua/tests/protocol-generation.spec.ts new file mode 100644 index 0000000000..996bde8923 --- /dev/null +++ b/packages/computer-use/cua/tests/protocol-generation.spec.ts @@ -0,0 +1,23 @@ +/** Native admission artifacts follow the model schema without exposing Host-only operations. */ +import { readFile } from 'node:fs/promises' +import { describe, expect, it } from 'vitest' +import { renderNativeProtocol } from '../native/generate-protocol.ts' +import { Request } from '../src/protocol.ts' + +describe('native protocol generation', () => { + it('keeps native JSON Schema and Swift admission tables current', async () => { + const generated = renderNativeProtocol() + expect(await readFile(new URL('../native/protocol.schema.json', import.meta.url), 'utf8')).toBe(generated.json) + expect(await readFile(new URL('../native/Sources/CuaNative/NativeProtocol.generated.swift', import.meta.url), 'utf8')).toBe(generated.swift) + }) + + it.each([ + { method: 'getState', params: { appHandle: 'app', maxElements: 1 } }, + { method: 'drag', params: { appHandle: 'app', windowId: 1, fromX: 1, fromY: 1, toX: 2, toY: 2, steps: 1 } }, + { method: 'paste', params: { appHandle: 'app', text: 'text', maxClipboardBytes: 1024 } }, + { method: 'findApps', params: { query: 'Editor', maxAppMatches: 1 } }, + { method: 'launchApp', params: { appHandle: 'pending' } }, + ])('rejects Host-only request data at model admission: $method', (request) => { + expect(Request.safeParse(request).success).toBe(false) + }) +})