fix(cua): bound clipboard consumption waits with actionable failure

This commit is contained in:
creatixchu committed 2026-10-14 11:08:00 +08:00
1 parent 0cca141695
commit 0f83b644c8
22 files changed
+486 -33

No files matched your search

@@ -15,8 +15,8 @@
en: d152484eb41ac6b4
zh: 7b30b973a849247d
/deepseek-ai-dsh-experimental-cua-macos/use-this-package:
en: b80fbe57355aefad
zh: 6ae84f80ad3f7829
en: e2e1797ad6b73f25
zh: 35b7a06e471e433c
/deepseek-ai-dsh-experimental-cua-macos/use-this-package/connect-an-external-execution-environment:
en: e7909c9b43e44fbc
zh: 98949877d4ea3a86
@@ -47,6 +47,7 @@ The dump command checks profile composition without a model request. For a real
| `axTimeoutMs` | `3000` | Deadline per Accessibility system call, configured independently of the complete operation. |
| `maxRequestBytes` / `maxResponseBytes` | `1048576` / `33554432` | UTF-8 request and response limits. |
| `maxClipboardBytes` | `33554432` | Maximum original clipboard bytes preserved during a paste; oversized contents reject before replacement. |
| `pasteReadTimeoutMs` | `1000` | Clipboard read deadline after the Paste command returns; expiry reports `PASTE_NOT_CONSUMED`. |
| `maxAppMatches` | `20` | Maximum installed-app search matches; broader queries reject and require a narrower name. |
| `maxElements` / `maxDepth` | `2000` / `25` | Accessibility traversal limits. |
| `dragDurationMs` / `dragSteps` | `400` / `20` | Duration and movement segments for a drag. |
@@ -47,6 +47,7 @@ pnpm dsh headless --patch packages/experimental/cua-macos/examples/macos.cordis.
| `axTimeoutMs` | `3000` | 每次 Accessibility 系统调用的截止时间,与完整操作截止独立配置。 |
| `maxRequestBytes` / `maxResponseBytes` | `1048576` / `33554432` | UTF-8 请求与响应字节限制。 |
| `maxClipboardBytes` | `33554432` | 粘贴时保存原剪贴板内容的字节上限;超限会在替换前拒绝。 |
| `pasteReadTimeoutMs` | `1000` | Paste 命令返回后等待剪贴板读取的上限;到期返回 `PASTE_NOT_CONSUMED`。 |
| `maxAppMatches` | `20` | 已安装应用搜索的结果上限;范围过大的查询会被拒绝,需要缩小名称范围。 |
| `maxElements` / `maxDepth` | `2000` / `25` | 辅助功能树遍历限制。 |
| `dragDurationMs` / `dragSteps` | `400` / `20` | 拖动时长和移动分段数。 |
@@ -21,11 +21,11 @@
en: 3202c567a6e1a654
zh: 143eebd0096a0487
/dsh-cua-native-helper/observation-and-input:
en: aa1bb051117c029b
zh: 32b66ee62ec3eae6
en: f2db99ae46a21c61
zh: 48c6752bd71dbf27
/dsh-cua-native-helper/verification:
en: ad55e01846bde8a7
zh: 906dfec56544b9f5
en: 2afe5ceb0a3c4965
zh: 86bf7187ddae489f
/dsh-cua-native-helper/further-exploration:
en: 265e1927952ce6d7
zh: 7c35d9fd98b6e6c8
@@ -85,7 +85,7 @@ Request modifier arrays accept command/control/option/shift/fn. The object API n
`selectText` requires a fresh element snapshot and writable AXSelectedTextRange. It matches one literal occurrence, narrowed by immediately adjacent prefix/suffix, and selects text or places the cursor before/after it. Matching and reported ranges use UTF-16 offsets. Ambiguous or missing text fails; the helper rechecks the value and selected range without rewriting content. AX text includes available writable, disabled, selected and selection-range information; AXAccess owns indices and native references while AXTextFormatter renders the observed values.
`paste` publishes one plain-text or HTML clipboard format, dispatches the application's enabled standard Paste menu command, and waits for a clipboard data request. It then restores the preserved formats only while its ownership marker and change count still match. The Host serializes its pastes across helpers with a file lock. maxClipboardBytes bounds both input and preserved contents; unsupported menus and oversized data fail before dispatch. A later copy is preserved. Cancellation restores owned clipboard data when possible; forced termination cannot guarantee restoration, and a data request does not prove the target completed editing.
`paste` publishes one plain-text or HTML clipboard format, dispatches the application's enabled standard Paste menu command, and waits for a clipboard data request. It then restores the preserved formats only while its ownership marker and change count still match. The Host serializes its pastes across helpers with a file lock. maxClipboardBytes bounds both input and preserved contents; unsupported menus and oversized data fail before dispatch. A later copy is preserved. Cancellation restores owned clipboard data when possible; forced termination cannot guarantee restoration, and a data request does not prove the target completed editing. After dispatch, `pasteReadTimeoutMs` bounds waiting for clipboard consumption. Expiry returns `PASTE_NOT_CONSUMED`, closes the promised data provider, and restores the original clipboard only while still owned. The failure does not prove the app will never process its queued Paste command; inspect the target before deciding whether to retry.
`hover` accepts an observed element index or a window-local point and never presses a mouse button. Disabled controls may be hovered. No request completion, AX read or screenshot restores the previous point; subsequent hover calls can enter revealed content. Click sends its button events at the target without an off-window priming click. Individual requests are serialized, but no application is reserved between calls. Other input or application changes can interrupt the interaction.
@@ -117,6 +117,8 @@ For an independently isolated Electron/Chromium check, run `python3 packages/exp
Run `python3 packages/experimental/cua-macos/native/tests/capabilities_e2e.py --helper /absolute/helper --artifacts /private/evidence` to verify a stopped application's discovery and launch, Unicode selection, multiline plain/HTML paste, middle/modifier gestures and keypad identities through an isolated AppKit receiver. It requires existing Accessibility/Screen Recording grants and records the receiver's actual state separately from dispatch results.
Run `python3 packages/experimental/cua-macos/native/tests/paste_e2e.py --artifacts /private/evidence` to verify that an owned AppKit app accepts a background Paste action without reading its data, reports `PASTE_NOT_CONSUMED`, then consumes the same paste in the foreground. The test requires an unlocked Mac and Accessibility permission, activates only its own two applications, and records received actions and text independently of helper replies.
-----
## Further Exploration
@@ -85,7 +85,7 @@ AppKit 在主事件循环上运行。阻塞式 AX 和输入使用每条连接的
`selectText` 要求新近元素快照和可写 AXSelectedTextRange。它匹配唯一的字面文本,可用紧邻的 prefix/suffix 缩小范围,然后选中文本或将光标放在前后。匹配和返回范围采用 UTF-16 偏移。歧义或缺失文本会失败,helper 不改写内容,并重新核实值和选区。AX 文本包含可用的可写性、禁用状态、选中状态和选区信息;AXAccess 持有索引和原生引用,AXTextFormatter 渲染已观察的值。
`paste` 提供一种纯文本或 HTML 剪贴板格式,调用应用已启用的标准 Paste 菜单命令,并等待剪贴板数据请求。随后仅在所有权标记和修改计数仍匹配时恢复原有格式。Host 通过文件锁串行化不同 helper 的粘贴。maxClipboardBytes 同时限制输入和保留内容,不支持的菜单和过大的数据在分派前失败。后续复制会被保留。取消会尽可能恢复仍持有的剪贴板数据,强制结束不保证恢复,数据请求也不证明目标已完成编辑。
`paste` 提供一种纯文本或 HTML 剪贴板格式,调用应用已启用的标准 Paste 菜单命令,并等待剪贴板数据请求。随后仅在所有权标记和修改计数仍匹配时恢复原有格式。Host 通过文件锁串行化不同 helper 的粘贴。maxClipboardBytes 同时限制输入和保留内容,不支持的菜单和过大的数据在分派前失败。后续复制会被保留。取消会尽可能恢复仍持有的剪贴板数据,强制结束不保证恢复,数据请求也不证明目标已完成编辑。 分派完成后,`pasteReadTimeoutMs` 限制等待剪贴板读取的时间。到期返回 `PASTE_NOT_CONSUMED`,关闭延迟数据提供方,并仅在仍持有剪贴板时恢复原内容。该失败不能证明应用以后不会处理已排队的 Paste 命令;决定是否重试前先检查目标内容。
`hover` 接受已观察的元素索引或窗口内坐标,不按下鼠标按钮,也可用于禁用控件。调用结束、AX 读取和截图都不恢复原位置,后续 hover 调用可进入新出现的内容。点击只在目标位置发送按键事件,不先在窗口外点击。单次请求串行执行,不在调用之间保留应用。其他输入或应用变化可能中断交互。
@@ -117,6 +117,9 @@ Blender 键盘检查使用 `python3 packages/experimental/cua-macos/native/tests
运行 `python3 packages/experimental/cua-macos/native/tests/capabilities_e2e.py --helper /absolute/helper --artifacts /private/evidence`,通过隔离的 AppKit 接收方验证未运行应用的发现和启动、Unicode 选区、多行纯文本/HTML 粘贴、中键/修饰键手势及数字键盘身份。需要已有辅助功能/屏幕录制权限,接收方实际状态与分派结果分开记录。
运行 `python3 packages/experimental/cua-macos/native/tests/paste_e2e.py --artifacts /private/evidence`,验证自有 AppKit 应用在后台接收 Paste 动作但不读取数据时返回 `PASTE_NOT_CONSUMED`,以及前台读取同一次粘贴后成功。测试要求 Mac 已解锁并已授予辅助功能权限,只激活自有的两个应用,并独立记录收到的动作和文本以核对 helper 回复。
-----
## 延伸阅读
@@ -52,6 +52,7 @@ struct PastePlan: Sendable {
let text: String
let format: String
let maxBytes: Int
let readTimeout: Duration
let maxElements: Int
let maxDepth: Int
}
@@ -319,6 +320,7 @@ actor AXWorker {
case "paste":
return .paste(PastePlan(handle: handle, text: try params.string("text"), format: try params.optionalString("format") ?? "text",
maxBytes: try params.integer("maxClipboardBytes", in: 1024...Int.max),
readTimeout: .milliseconds(try params.integer("pasteReadTimeoutMs", in: 1...2147483647)),
maxElements: try params.integer("maxElements", in: 1...10000), maxDepth: try params.integer("maxDepth", in: 1...100)))
case "typeText", "pressKey":
let recipient = try ax().keyboardRecipient(identity, catalog: catalog)
@@ -3,7 +3,7 @@ import AppKit
import Foundation
private final class PasteDataProvider: NSObject, NSPasteboardItemDataProvider, @unchecked Sendable {
private enum Delivery { case pending, supplied, failed }
private enum Delivery { case pending, supplied, failed, closed }
private let data: Data
private let type: NSPasteboard.PasteboardType
// AppKit can request promised data on another thread while the AX worker waits.
@@ -11,20 +11,30 @@ private final class PasteDataProvider: NSObject, NSPasteboardItemDataProvider, @
private var delivery: Delivery = .pending
init(data: Data, type: NSPasteboard.PasteboardType) { self.data = data; self.type = type }
func pasteboard(_ pasteboard: NSPasteboard?, item: NSPasteboardItem, provideDataForType type: NSPasteboard.PasteboardType) {
let supplied = type == self.type && item.setData(data, forType: type)
condition.lock()
defer { condition.unlock() }
guard case .pending = delivery else { return }
let supplied = type == self.type && item.setData(data, forType: type)
delivery = supplied ? .supplied : .failed
condition.broadcast()
condition.unlock()
}
@MainActor func wait(check: () throws -> Void) async throws {
func close() { condition.withLock { delivery = .closed } }
@MainActor func wait(timeout: Duration, check: () throws -> Void) async throws {
let clock = ContinuousClock()
let deadline = clock.now.advanced(by: timeout)
while true {
try check()
let state = condition.withLock { delivery }
let state = condition.withLock {
if case .pending = delivery, clock.now >= deadline { delivery = .closed }
return delivery
}
switch state {
case .supplied: return
case .failed: throw NativeFailure(code: "CLIPBOARD_UNAVAILABLE", message: "Requested paste data could not be supplied")
case .pending: try await Task.sleep(for: .milliseconds(25))
case .closed: throw NativeFailure(code: "PASTE_NOT_CONSUMED", message:
"Paste command was dispatched, but clipboard data was not requested within the read deadline. " +
"Bring the target app to the foreground and focus its text input. Inspect the contents before deciding " +
"whether to retry; a delayed paste may still run.")
case .pending: try await clock.sleep(until: min(deadline, clock.now.advanced(by: .milliseconds(25))))
}
}
}
@@ -39,7 +49,7 @@ private final class PasteDataProvider: NSObject, NSPasteboardItemDataProvider, @
init(pasteboard: NSPasteboard = .general) { self.pasteboard = pasteboard }
func perform(text: String, format: String, maxBytes: Int, checkCancellation: () throws -> Void = {},
func perform(text: String, format: String, maxBytes: Int, readTimeout: Duration, checkCancellation: () throws -> Void = {},
operation: (@escaping @MainActor @Sendable () throws -> Void) async throws -> Void) async throws {
try checkCancellation()
let type: NSPasteboard.PasteboardType
@@ -55,6 +65,7 @@ private final class PasteDataProvider: NSObject, NSPasteboardItemDataProvider, @
let owner = Data(UUID().uuidString.utf8)
let item = NSPasteboardItem()
let provider = PasteDataProvider(data: data, type: type)
defer { provider.close() }
guard item.setDataProvider(provider, forTypes: [type]), item.setData(owner, forType: Self.ownerType) else {
throw NativeFailure(code: "CLIPBOARD_UNAVAILABLE", message: "Could not prepare paste contents")
}
@@ -79,7 +90,7 @@ private final class PasteDataProvider: NSObject, NSPasteboardItemDataProvider, @
do {
try requireOwnership()
try await operation(requireOwnership)
try await provider.wait { try checkCancellation(); try requireOwnership() }
try await provider.wait(timeout: readTimeout) { try checkCancellation(); try requireOwnership() }
} catch {
if owned() { try restore(saved) }
throw error
@@ -13,7 +13,7 @@ enum NativeProtocol {
"launchApp": ["appHandle"],
"listApps": [],
"listWindows": ["appHandle"],
"paste": ["appHandle", "format", "maxClipboardBytes", "maxDepth", "maxElements", "text"],
"paste": ["appHandle", "format", "maxClipboardBytes", "maxDepth", "maxElements", "pasteReadTimeoutMs", "text"],
"performSecondaryAction": ["action", "appHandle", "elementIndex", "snapshotId"],
"pressKey": ["appHandle", "key", "modifiers"],
"screenshot": ["appHandle", "windowId"],
@@ -189,7 +189,7 @@ import Darwin
try WindowPointerInput.hover(in: route, at: point, checkCancellation: { try latch.check() })
result = AXWorker.hoverResult(route: "system-pointer", pointerMoved: true)
case .paste(let plan):
try await clipboard.perform(text: plan.text, format: plan.format, maxBytes: plan.maxBytes,
try await clipboard.perform(text: plan.text, format: plan.format, maxBytes: plan.maxBytes, readTimeout: plan.readTimeout,
checkCancellation: { try cancellation.check() }) { requireClipboard in
try await worker.performPaste(plan, beforeDispatch: requireClipboard)
}
@@ -718,6 +718,11 @@
"minimum": 1024,
"maximum": 9007199254740991
},
"pasteReadTimeoutMs": {
"type": "integer",
"minimum": 1,
"maximum": 2147483647
},
"maxElements": {
"type": "integer",
"minimum": 1,
@@ -733,6 +738,7 @@
"appHandle",
"text",
"maxClipboardBytes",
"pasteReadTimeoutMs",
"maxElements",
"maxDepth"
],
@@ -0,0 +1,56 @@
// A foreground-sensitive Paste receiver records AX command delivery independently of clipboard consumption.
import AppKit
final class PasteFixture: NSObject, NSApplicationDelegate, NSTextViewDelegate {
let statePath: String
let label: String
var window: NSWindow!
var text: NSTextView!
var pasteCalls: [Bool] = []
init(statePath: String, label: String) { self.statePath = statePath; self.label = label }
func applicationDidFinishLaunching(_ notification: Notification) {
let menu = NSMenu()
let editItem = NSMenuItem()
let edit = NSMenu(title: "Edit")
edit.autoenablesItems = false
let paste = NSMenuItem(title: "Paste", action: #selector(pasteText(_:)), keyEquivalent: "v")
paste.target = self
edit.addItem(paste)
editItem.submenu = edit
menu.addItem(editItem)
NSApp.mainMenu = menu
window = NSWindow(contentRect: NSRect(x: 140, y: 160, width: 460, height: 220),
styleMask: [.titled, .closable], backing: .buffered, defer: false)
window.title = label
window.animationBehavior = .none
text = NSTextView(frame: NSRect(x: 20, y: 20, width: 420, height: 180))
text.isRichText = false
text.isEditable = true
text.delegate = self
window.contentView!.addSubview(text)
window.makeKeyAndOrderFront(nil)
window.makeFirstResponder(text)
NSApp.activate(ignoringOtherApps: true)
persist()
print("READY"); fflush(stdout)
}
@objc func pasteText(_ sender: Any?) {
pasteCalls.append(NSApp.isActive)
if NSApp.isActive { text.paste(sender) }
persist()
}
func applicationDidBecomeActive(_ notification: Notification) { persist() }
func applicationDidResignActive(_ notification: Notification) { persist() }
func textDidChange(_ notification: Notification) { persist() }
func persist() {
guard let window, let text else { return }
let state: [String: Any] = ["active": NSApp.isActive, "textFocused": window.firstResponder === text,
"text": text.string, "pasteCalls": pasteCalls]
try! JSONSerialization.data(withJSONObject: state).write(to: URL(fileURLWithPath: statePath), options: .atomic)
}
}
let app = NSApplication.shared
app.setActivationPolicy(.regular)
let fixture = PasteFixture(statePath: CommandLine.arguments[1], label: CommandLine.arguments[2])
app.delegate = fixture
app.run()
@@ -90,7 +90,7 @@ def main():
call("selectText", {**target, "text": "target", "prefix": "第二😀 "})
selected = wait_for_state(state_file, lambda value: value["selection"] == [17, 6])
assert selected["text"] == "第一😀 target,第二😀 target.", selected
call("paste", {**owner, "text": "替换\n第二行\t保留", "format": "text", "maxClipboardBytes": 33554432, "maxElements": 2000, "maxDepth": 25})
call("paste", {**owner, "text": "替换\n第二行\t保留", "format": "text", "maxClipboardBytes": 33554432, "pasteReadTimeoutMs": 1000, "maxElements": 2000, "maxDepth": 25})
state = wait_for_state(state_file, lambda value: value["text"] == "第一😀 target,第二😀 替换\n第二行\t保留.")
assert state["pasteAttempts"] == [True], state
observed = call("getState", {**owner, "maxElements": 2000, "maxDepth": 25})
@@ -98,7 +98,7 @@ def main():
target = {**owner, "snapshotId": observed["snapshotId"], "elementIndex": field["index"]}
call("selectText", {**target, "text": state["text"]})
html = '<h1>Heading</h1><p><b>Bold</b> and <a href="https://example.com/">link</a></p>'
call("paste", {**owner, "text": html, "format": "html", "maxClipboardBytes": 33554432, "maxElements": 2000, "maxDepth": 25})
call("paste", {**owner, "text": html, "format": "html", "maxClipboardBytes": 33554432, "pasteReadTimeoutMs": 1000, "maxElements": 2000, "maxDepth": 25})
state = wait_for_state(state_file, lambda value: any(run.get("link") == "https://example.com/" for run in value["runs"]))
assert any(run["bold"] and "Bold" in run["text"] for run in state["runs"]), state
pad = state["pad"]
@@ -0,0 +1,75 @@
"""Verify bounded unread-paste failure and foreground success against owned AppKit apps."""
import argparse
import json
from pathlib import Path
import subprocess
import tempfile
import time
from native_test_support import BINARY, ROOT, NativeHelper, line, run_cleanup, stop
def wait_state(path, predicate):
deadline = time.monotonic() + 10
latest = None
while time.monotonic() < deadline:
if path.exists():
latest = json.loads(path.read_text())
if predicate(latest):
return latest
time.sleep(0.025)
raise AssertionError(f"Fixture state did not settle: {latest}")
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--helper", type=Path, default=BINARY)
parser.add_argument("--artifacts", type=Path)
args = parser.parse_args()
transcript = []
processes = []
helper = None
with tempfile.TemporaryDirectory(prefix="dsh-cua-paste-") as directory:
root = Path(directory)
binary = root / "paste-fixture"
subprocess.run(["swiftc", str(ROOT / "tests/PasteFixture.swift"), "-o", str(binary)], check=True)
try:
for label in ["target", "foreground"]:
process = subprocess.Popen([str(binary), str(root / f"{label}.json"), f"DSH paste {label}"],
stdout=subprocess.PIPE, text=True)
processes.append(process)
assert line(process).strip() == "READY"
helper = NativeHelper(transcript, args.helper)
app = helper.call("getApp", {"selector": str(processes[0].pid)})
blocker = helper.call("getApp", {"selector": str(processes[1].pid)})
owner = {"appHandle": app["appHandle"]}
params = {**owner, "text": "test", "format": "text", "pasteReadTimeoutMs": 1000,
"maxClipboardBytes": 33554432, "maxElements": 2000, "maxDepth": 25}
helper.call("activate", {"appHandle": blocker["appHandle"]})
wait_state(root / "target.json", lambda state: not state["active"] and state["textFocused"])
started = time.monotonic()
error = helper.call("paste", params, "PASTE_NOT_CONSUMED")
elapsed = time.monotonic() - started
assert "foreground" in error["message"] and "focus" in error["message"], error
background = wait_state(root / "target.json", lambda state: state["pasteCalls"] == [False])
assert background["text"] == "" and not background["active"], background
helper.call("activate", owner)
wait_state(root / "target.json", lambda state: state["active"] and state["textFocused"])
result = helper.call("paste", params)
assert result == {"dispatched": True, "verified": False}, result
foreground = wait_state(root / "target.json", lambda state: state["text"] == "test")
assert foreground["pasteCalls"] == [False, True] and foreground["active"], foreground
if args.artifacts:
args.artifacts.mkdir(parents=True, exist_ok=True)
(args.artifacts / "results.json").write_text(json.dumps({"background": background,
"foreground": foreground, "backgroundElapsedSeconds": elapsed, "error": error}, indent=2) + "\n")
print(f"PASS: background PASTE_NOT_CONSUMED in {elapsed:.3f}s; foreground paste inserted test")
finally:
if args.artifacts:
args.artifacts.mkdir(parents=True, exist_ok=True)
(args.artifacts / "transcript.json").write_text(json.dumps(transcript, indent=2) + "\n")
run_cleanup(*([helper.stop] if helper else []), *[lambda process=process: stop(process) for process in reversed(processes)])
if __name__ == "__main__":
main()
@@ -12,7 +12,7 @@ import Testing
#expect(item.setString("original", forType: .string))
#expect(item.setData(Data([1, 2, 3]), forType: custom))
#expect(board.writeObjects([item]))
try await ClipboardPaste(pasteboard: board).perform(text: "line 1\nline 2\tvalue", format: "text", maxBytes: 1024) { current in
try await ClipboardPaste(pasteboard: board).perform(text: "line 1\nline 2\tvalue", format: "text", maxBytes: 1024, readTimeout: .seconds(1)) { current in
try current()
#expect(board.string(forType: .string) == "line 1\nline 2\tvalue")
}
@@ -24,7 +24,7 @@ import Testing
let board = NSPasteboard.withUniqueName()
defer { board.releaseGlobally() }
let html = "<h1>Title</h1><p><b>Bold</b> <a href=\"https://example.com/\">link</a></p>"
try await ClipboardPaste(pasteboard: board).perform(text: html, format: "html", maxBytes: 1024) { current in
try await ClipboardPaste(pasteboard: board).perform(text: html, format: "html", maxBytes: 1024, readTimeout: .seconds(1)) { current in
try current()
#expect(board.data(forType: .html) == Data(html.utf8))
#expect(board.string(forType: .string) == nil)
@@ -37,7 +37,7 @@ import Testing
defer { board.releaseGlobally() }
#expect(board.setString("original", forType: .string))
do {
try await ClipboardPaste(pasteboard: board).perform(text: "paste", format: "text", maxBytes: 1024) { current in
try await ClipboardPaste(pasteboard: board).perform(text: "paste", format: "text", maxBytes: 1024, readTimeout: .seconds(1)) { current in
board.clearContents()
#expect(board.setString("new user copy", forType: .string))
try current()
@@ -52,7 +52,7 @@ import Testing
defer { board.releaseGlobally() }
#expect(board.setString("original", forType: .string))
do {
try await ClipboardPaste(pasteboard: board).perform(text: "paste", format: "text", maxBytes: 1024) { _ in
try await ClipboardPaste(pasteboard: board).perform(text: "paste", format: "text", maxBytes: 1024, readTimeout: .seconds(1)) { _ in
throw NativeFailure(code: "NATIVE_CANCELLED", message: "fixture cancellation")
}
Issue.record("Cancelled operation succeeded")
@@ -62,7 +62,7 @@ import Testing
let large = String(repeating: "x", count: 2048)
#expect(board.setString(large, forType: .string))
do {
try await ClipboardPaste(pasteboard: board).perform(text: "paste", format: "text", maxBytes: 1024) { _ in
try await ClipboardPaste(pasteboard: board).perform(text: "paste", format: "text", maxBytes: 1024, readTimeout: .seconds(1)) { _ in
Issue.record("Oversized clipboard reached dispatch")
}
Issue.record("Oversized clipboard was accepted")
@@ -74,10 +74,63 @@ import Testing
let board = NSPasteboard.withUniqueName()
defer { board.releaseGlobally() }
#expect(board.setString("original", forType: .string))
var dispatched = false
do {
try await ClipboardPaste(pasteboard: board).perform(text: "paste", format: "text", maxBytes: 1024,
checkCancellation: { throw NativeFailure(code: "NATIVE_CANCELLED", message: "delivery cancelled") }) { _ in }
try await ClipboardPaste(pasteboard: board).perform(text: "paste", format: "text", maxBytes: 1024, readTimeout: .seconds(1),
checkCancellation: { if dispatched { throw NativeFailure(code: "NATIVE_CANCELLED", message: "delivery cancelled") } }) { _ in
dispatched = true
}
Issue.record("Unread clipboard data completed without cancellation")
} catch let error as NativeFailure { #expect(error.code == "NATIVE_CANCELLED") }
#expect(board.string(forType: .string) == "original")
}
@Test @MainActor func clipboardPasteReportsUnreadDataAfterDispatchAndRestoresOriginalContents() async throws {
let board = NSPasteboard.withUniqueName()
defer { board.releaseGlobally() }
#expect(board.setString("original", forType: .string))
var dispatched = false
do {
try await ClipboardPaste(pasteboard: board).perform(text: "paste", format: "text", maxBytes: 1024,
readTimeout: .zero) { current in
try current()
dispatched = true
}
Issue.record("Unread paste completed successfully")
} catch let error as NativeFailure {
#expect(error.code == "PASTE_NOT_CONSUMED")
#expect(error.message.contains("foreground"))
#expect(error.message.contains("focus"))
#expect(error.message.contains("Inspect"))
}
#expect(dispatched)
#expect(board.string(forType: .string) == "original")
}
@Test @MainActor func clipboardPasteAcceptsAReadAfterDispatchBeforeTheDeadline() async throws {
let board = NSPasteboard.withUniqueName()
defer { board.releaseGlobally() }
#expect(board.setString("original", forType: .string))
var dispatched = false
var consumed: String?
try await ClipboardPaste(pasteboard: board).perform(text: "paste", format: "text", maxBytes: 1024,
readTimeout: .seconds(1), checkCancellation: {
if dispatched { consumed = board.string(forType: .string) }
}) { _ in dispatched = true }
#expect(consumed == "paste")
#expect(board.string(forType: .string) == "original")
}
@Test @MainActor func clipboardPasteKeepsAnUnsupportedMenuFailureDistinctFromUnreadData() async throws {
let board = NSPasteboard.withUniqueName()
defer { board.releaseGlobally() }
#expect(board.setString("original", forType: .string))
do {
try await ClipboardPaste(pasteboard: board).perform(text: "paste", format: "text", maxBytes: 1024,
readTimeout: .zero) { _ in
throw NativeFailure(code: "UNSUPPORTED", message: "No standard Paste command is available")
}
Issue.record("Unsupported Paste command completed successfully")
} catch let error as NativeFailure { #expect(error.code == "UNSUPPORTED") }
#expect(board.string(forType: .string) == "original")
}
@@ -15,6 +15,7 @@ describe('native protocol generation', () => {
{ method: 'getState', params: { appHandle: 'app', maxElements: 1 } },
{ method: 'drag', params: { appHandle: 'app', windowId: 1, fromX: 1, fromY: 1, toX: 2, toY: 2, steps: 1 } },
{ method: 'paste', params: { appHandle: 'app', text: 'text', maxClipboardBytes: 1024 } },
{ method: 'paste', params: { appHandle: 'app', text: 'text', pasteReadTimeoutMs: 1000 } },
{ method: 'findApps', params: { query: 'Editor', maxAppMatches: 1 } },
{ method: 'launchApp', params: { appHandle: 'pending' } },
])('rejects Host-only request data at model admission: $method', (request) => {
@@ -55,7 +55,7 @@ vi.mock('@deepseek-ai/dsh-cua/native-client', async importOriginal => ({
}))
const config = {
helperPath: resolve('fixture-helper'), buildTimeoutMs: 300_000, timeoutMs: 10_000, connectTimeoutMs: 5000, screenshotTimeoutMs: 5000, axTimeoutMs: 3000, maxRequestBytes: 1024, maxResponseBytes: 4096, maxClipboardBytes: 8192, maxAppMatches: 20,
helperPath: resolve('fixture-helper'), buildTimeoutMs: 300_000, timeoutMs: 10_000, connectTimeoutMs: 5000, screenshotTimeoutMs: 5000, axTimeoutMs: 3000, maxRequestBytes: 1024, maxResponseBytes: 4096, maxClipboardBytes: 8192, pasteReadTimeoutMs: 1000, maxAppMatches: 20,
graceMs: 10, textEventIntervalMs: 8, maxElements: 64, maxDepth: 8, dragDurationMs: 200, dragSteps: 10,
} satisfies Omit<NativeProvider.Config, 'allowedApps'>
@@ -337,7 +337,7 @@ it('supplies configured resource bounds and clears registrations on unload', asy
await f.execute('drag', { appHandle: 'app-one', windowId: 7, fromX: 1, fromY: 2, toX: 3, toY: 4, relative: true })
expect(native.request).toHaveBeenLastCalledWith('drag', expect.objectContaining({ durationMs: 200, steps: 10 }), expect.any(AbortSignal), ToolCallId('lifecycle-3'))
await f.execute('paste', { appHandle: 'app-one', text: 'hello', format: 'text' })
expect(native.request).toHaveBeenLastCalledWith('paste', expect.objectContaining({ maxClipboardBytes: 8192, maxElements: 64, maxDepth: 8 }), expect.any(AbortSignal), ToolCallId('lifecycle-4'))
expect(native.request).toHaveBeenLastCalledWith('paste', expect.objectContaining({ maxClipboardBytes: 8192, pasteReadTimeoutMs: 1000, maxElements: 64, maxDepth: 8 }), expect.any(AbortSignal), ToolCallId('lifecycle-4'))
await f.execute('findApps', { query: 'Fixture' })
expect(native.request).toHaveBeenLastCalledWith('findApps', { query: 'Fixture', maxAppMatches: 20 }, expect.any(AbortSignal), ToolCallId('lifecycle-5'))
await f.provider.dispose()
+5 -1
View File
@@ -52,6 +52,8 @@ export interface CuaConfig {
maxAppMatches: number
/** Maximum bytes preserved from the system clipboard during one paste. */
maxClipboardBytes: number
/** Maximum wait for clipboard data consumption after the Paste command returns. */
pasteReadTimeoutMs: number
/** Maximum UTF-8 bytes admitted for one helper request frame. */
maxRequestBytes: number
/** Duration in milliseconds of one same-window drag. */
@@ -82,6 +84,7 @@ export const CuaConfig = Schema.object({
axTimeoutMs: Schema.number().step(1).min(1).max(2 ** 31 - 1).default(3000),
maxAppMatches: Schema.number().step(1).min(1).max(1000).default(20),
maxClipboardBytes: Schema.number().step(1).min(1024).default(32 * 1024 * 1024),
pasteReadTimeoutMs: Schema.number().step(1).min(1).max(2 ** 31 - 1).default(1000),
maxRequestBytes: Schema.number().step(1).min(1024).default(1024 * 1024),
dragDurationMs: Schema.number().step(1).min(1).max(5000).default(400),
dragSteps: Schema.number().step(1).min(1).max(240).default(20),
@@ -224,7 +227,8 @@ export function installCua(ctx: Context, config: CuaConfig, backend: CuaBackend)
const nativeParams = method === 'findApps'
? { ...params, maxAppMatches: config.maxAppMatches }
: method === 'paste'
? { ...params, maxClipboardBytes: config.maxClipboardBytes, maxElements: config.maxElements, maxDepth: config.maxDepth }
? { ...params, maxClipboardBytes: config.maxClipboardBytes, pasteReadTimeoutMs: config.pasteReadTimeoutMs,
maxElements: config.maxElements, maxDepth: config.maxDepth }
: method === 'getState'
? { ...params, maxElements: config.maxElements, maxDepth: config.maxDepth }
: method === 'drag' ? { ...params, durationMs: config.dragDurationMs, steps: config.dragSteps }
+1 -1
View File
@@ -72,7 +72,7 @@ export const Request = z.discriminatedUnion('method', [
export const NativeResourceParameters = {
getState: z.object({ maxElements: z.number().int().min(1).max(10000), maxDepth: z.number().int().min(1).max(100) }).strict(),
drag: z.object({ durationMs: z.number().int().min(1).max(5000), steps: z.number().int().min(1).max(240) }).strict(),
paste: z.object({ maxClipboardBytes: z.number().int().min(1024),
paste: z.object({ maxClipboardBytes: z.number().int().min(1024), pasteReadTimeoutMs: z.number().int().min(1).max(2 ** 31 - 1),
maxElements: z.number().int().min(1).max(10000), maxDepth: z.number().int().min(1).max(100),
}).strict(),
findApps: z.object({ maxAppMatches: z.number().int().min(1).max(1000) }).strict(),
+1
View File
@@ -202,6 +202,7 @@ export interface CuaApp extends CuaSurface {
typeText(text: string): Promise<CuaActionResult>
/**
* Paste text or HTML through the application's standard Paste menu command.
* PASTE_NOT_CONSUMED means no clipboard read within the configured deadline after dispatch; inspect before retrying.
* @param text - Clipboard contents in the selected format; defaults to plain text.
* @param options - Clipboard format. Original contents are restored unless another writer changes them.
* @returns Dispatch evidence; unsupported Paste commands fail without typing fallback.
@@ -1385,6 +1385,237 @@
}
]
},
{
"kind": "chunks",
"chunks": [
{
"type": "block-start",
"index": 0,
"blockType": "tool-call"
},
{
"type": "block-end",
"index": 0,
"block": {
"type": "tool-call",
"id": "cua-paste-background",
"name": "cua",
"arguments": "{\"method\":\"pressKey\",\"params\":{\"appHandle\":\"fixture-app\",\"key\":\"F9\"}}"
}
},
{
"type": "usage",
"usage": {
"inputTokens": 3,
"outputTokens": 3
}
},
{
"type": "finish",
"reason": {
"kind": "tool-calls"
}
}
]
},
{
"kind": "chunks",
"chunks": [
{
"type": "block-start",
"index": 0,
"blockType": "tool-call"
},
{
"type": "block-end",
"index": 0,
"block": {
"type": "tool-call",
"id": "cua-paste-unread",
"name": "cua",
"arguments": "{\"method\":\"paste\",\"params\":{\"appHandle\":\"fixture-app\",\"text\":\"fixture paste\"}}"
}
},
{
"type": "usage",
"usage": {
"inputTokens": 3,
"outputTokens": 3
}
},
{
"type": "finish",
"reason": {
"kind": "tool-calls"
}
}
]
},
{
"kind": "chunks",
"chunks": [
{
"type": "block-start",
"index": 0,
"blockType": "tool-call"
},
{
"type": "block-end",
"index": 0,
"block": {
"type": "tool-call",
"id": "cua-paste-inspect",
"name": "cua",
"arguments": "{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"
}
},
{
"type": "usage",
"usage": {
"inputTokens": 3,
"outputTokens": 3
}
},
{
"type": "finish",
"reason": {
"kind": "tool-calls"
}
}
]
},
{
"kind": "chunks",
"chunks": [
{
"type": "block-start",
"index": 0,
"blockType": "tool-call"
},
{
"type": "block-end",
"index": 0,
"block": {
"type": "tool-call",
"id": "cua-paste-activate",
"name": "cua",
"arguments": "{\"method\":\"activate\",\"params\":{\"appHandle\":\"fixture-app\"}}"
}
},
{
"type": "usage",
"usage": {
"inputTokens": 3,
"outputTokens": 3
}
},
{
"type": "finish",
"reason": {
"kind": "tool-calls"
}
}
]
},
{
"kind": "chunks",
"chunks": [
{
"type": "block-start",
"index": 0,
"blockType": "tool-call"
},
{
"type": "block-end",
"index": 0,
"block": {
"type": "tool-call",
"id": "cua-paste-focus",
"name": "cua",
"arguments": "{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"
}
},
{
"type": "usage",
"usage": {
"inputTokens": 3,
"outputTokens": 3
}
},
{
"type": "finish",
"reason": {
"kind": "tool-calls"
}
}
]
},
{
"kind": "chunks",
"chunks": [
{
"type": "block-start",
"index": 0,
"blockType": "tool-call"
},
{
"type": "block-end",
"index": 0,
"block": {
"type": "tool-call",
"id": "cua-paste-consumed",
"name": "cua",
"arguments": "{\"method\":\"paste\",\"params\":{\"appHandle\":\"fixture-app\",\"text\":\"fixture paste\"}}"
}
},
{
"type": "usage",
"usage": {
"inputTokens": 3,
"outputTokens": 3
}
},
{
"type": "finish",
"reason": {
"kind": "tool-calls"
}
}
]
},
{
"kind": "chunks",
"chunks": [
{
"type": "block-start",
"index": 0,
"blockType": "tool-call"
},
{
"type": "block-end",
"index": 0,
"block": {
"type": "tool-call",
"id": "cua-paste-verify",
"name": "cua",
"arguments": "{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"
}
},
{
"type": "usage",
"usage": {
"inputTokens": 3,
"outputTokens": 3
}
},
{
"type": "finish",
"reason": {
"kind": "tool-calls"
}
}
]
},
{
"kind": "chunks",
"chunks": [
@@ -117,7 +117,7 @@ function createApplicationState(socket, record, application, captureBroker, conn
const screenshotConfig = new URL('./fixture-screenshot.json', import.meta.url)
const customScreenshot = existsSync(screenshotConfig) ? JSON.parse(readFileSync(screenshotConfig, 'utf8')) : undefined
const snapshots = new Map()
const methods = ['getCapabilities', 'listApps', 'getApp', 'activate', 'show', 'listWindows', 'getState', 'hover', 'click', 'setValue', 'typeText', 'pressKey', 'scroll', 'drag', 'screenshot']
const methods = ['getCapabilities', 'listApps', 'getApp', 'activate', 'show', 'listWindows', 'getState', 'hover', 'click', 'setValue', 'typeText', 'paste', 'pressKey', 'scroll', 'drag', 'screenshot']
let acquired = false
let generation = 0
let observationChanged = false
@@ -207,6 +207,12 @@ function createApplicationState(socket, record, application, captureBroker, conn
if (method === 'pressKey' && params.key === 'F11') captureBroker.pendingCapture = true
if (method === 'pressKey' && params.key === 'F10') captureBroker.pendingCapture = false
if (method === 'pressKey' && params.key === 'F12') malformedScreenshot = true
if (method === 'pressKey' && params.key === 'F9') { application.frontPid = 42; application.frontWindowId = 99 }
if (method === 'paste') {
if (application.frontPid !== app.pid) refuse('PASTE_NOT_CONSUMED', 'Paste command was dispatched, but clipboard data was not requested within the read deadline. Bring the target app to the foreground and focus its text input. Inspect the contents before deciding whether to retry; a delayed paste may still run.')
application.value += params.text
return { dispatched: true, verified: false }
}
if (method === 'typeText') application.value += params.text
else if (!['pressKey', 'scroll', 'drag'].includes(method)) refuse('UNSUPPORTED_OPERATION', method)
return { dispatched: true, verified: false }