feat(cua-macos): send owner key repeats and explain conflicting holds
This commit is contained in:
creatixchu committed
1 parent
ab9700e6ac
commit
dd5cda398c
16 files changed
+146
-41
No files matched your search
@@ -15,8 +15,8 @@
|
||||
en: 1777e7bf6785f986
|
||||
zh: e3daac1074ebb74d
|
||||
/deepseek-ai-dsh-experimental-cua-macos/use-this-package:
|
||||
en: db5e8ad4d2ea832a
|
||||
zh: 647cf55f4851760e
|
||||
en: 83d4eb62d0d0db2f
|
||||
zh: 3afde1c3b8c25211
|
||||
/deepseek-ai-dsh-experimental-cua-macos/use-this-package/connect-an-external-execution-environment:
|
||||
en: b734a0907747580e
|
||||
zh: 8110c5951b70e100
|
||||
|
||||
@@ -81,7 +81,7 @@ The [configuration catalog](../../../docs/config-catalog.md#deepseek-aidsh-exper
|
||||
|
||||
`pressKey(key, modifiers?)` presses and releases one key. For shortcuts, pass held modifiers separately, as in `pressKey('a', ['command'])`. Pressing a modifier itself, as in `pressKey('shift')`, sets its flag on press and clears it on release. A single `+` or `plus` sends Shift+=. The macOS helper accepts `command`/`cmd`, `shift`, `option`/`alt`, `control`/`ctrl`, and `fn`, and maps them to their physical keys. Key names are platform-defined; keypad names (`KP_0` through `KP_9`, `KP_Enter`, and keypad operators) retain their physical key codes. `typeText` sends literal text, including its line breaks and tabs. Use the App object's `pressKey` for a short press.
|
||||
|
||||
For a long press, call `app.sendKeyEvent('w', 'down')` inside `run_code`. Each successful down remains active across subsequent tool calls in that execution. Choose when to send each up: after W down, D down and D up, W stays down. When the program returns, exits with an error, or the user pauses it, the helper releases the remaining keys at the processes that received their downs. A later execution cannot release an earlier execution's keys.
|
||||
For a long press, call `app.sendKeyEvent('w', 'down')` inside `run_code`. The first down holds the key across subsequent tool calls. For an ordinary key already held by this execution, each additional down sends one key-down marked as a repeat to the original process. Choose the interval between calls. One up releases the key. Choose when to send each up: after W down, D down and D up, W stays down. When the program returns, exits with an error, or the user pauses it, the helper releases the remaining keys at the processes that received their downs. A later execution cannot release an earlier execution's keys.
|
||||
|
||||
Agent disposal and plugin unload cancel and await the complete tool call, including image admission and attachment storage. An image write already in progress must settle before disposal returns; a canceled call cannot return a successful screenshot.
|
||||
|
||||
@@ -224,7 +224,7 @@ Available at app or window level depending on the platform.
|
||||
- typeText(text: string) -> ActionResult — type literal text at the app's current focus; line endings send Return and tabs send Tab.
|
||||
- paste(text: string, options?: { format?: 'text' | 'html' }) -> ActionResult — deliver text or HTML through the app's Paste command, restoring prior clipboard contents unless another writer changes them; a Paste failure is surfaced without substituting typeText, and PASTE_NOT_CONSUMED means inspect before any retry.
|
||||
- pressKey(key: string, modifiers?: string[]) -> ActionResult — press and release one complete keystroke. For example pressKey('Tab', ['shift']) or pressKey('+'). Key and modifier names are platform-defined strings. Pass key and modifiers separately; do not pack them into a chord string.
|
||||
- sendKeyEvent(key: string, direction: 'down' | 'up') -> ActionResult — send 'down' once per key to hold one or more keys across tool calls within one run_code, for example W down, D down, D up, W up. Holding a physical key on a keyboard can produce repeated key-down events after a delay. This method posts one event per call and rejects a second 'down' for a held key. Use the app's response to decide what to do while holding the key and when to send 'up'. When this run_code returns, throws, or the user pauses it, macOS releases only its remaining keys at their original recipients.
|
||||
- sendKeyEvent(key: string, direction: 'down' | 'up') -> ActionResult — send 'down' to hold a key across tool calls in one run_code, for example W down, D down, D up, W up. Holding a key on a physical keyboard can produce repeated key-down events after a delay. To send one such repeat for a non-modifier key this run already holds, send 'down' again. Each call sends one event. Use the app's response to decide whether and when to repeat or release the key. When this run_code returns, throws, or the user pauses it, macOS releases only its remaining keys at their original recipients.
|
||||
|
||||
Raw tools use { method, params }: include appHandle for app calls and windowId for window calls, and use snapshotId with elementId for element routes. The emit option is facade-only: emit is local output control, so raw callers must follow the raw operation schema for admitted params. getAccessibilityTree and getWindow read through raw getState; getWindows maps to listWindows; getScreenshot maps to screenshot; cua.getState maps to listApps; every other named operation uses the same method name.
|
||||
```
|
||||
|
||||
@@ -81,7 +81,7 @@ pnpm dsh headless --patch packages/experimental/cua-macos/examples/macos.cordis.
|
||||
|
||||
`pressKey(key, modifiers?)` 按下并松开一个键。使用快捷键时单独传入修饰键，例如 `pressKey('a', ['command'])`。单独调用 `pressKey('shift')` 时，按下事件带有 Shift 标志，释放事件清除该标志。`+` 和 `plus` 都会发送 Shift 和等号。macOS helper 接受 `command`/`cmd`、`shift`、`option`/`alt`、`control`/`ctrl` 和 `fn`，并映射到对应物理键。按键名由平台定义；数字键盘名称（`KP_0` 到 `KP_9`、`KP_Enter` 和数字键盘运算键）保留独立的物理键码。`typeText` 发送字面文本，包括其中的换行和制表符。短按使用 App 对象上的 `pressKey`。
|
||||
|
||||
长按时，在 `run_code` 中调用 `app.sendKeyEvent('w', 'down')`。按下成功后，在同一次执行中调用其他工具时该键仍保持按下。每个键可分别松开。例如先按 W，再按 D，松开 D 后 W 仍按着。程序正常返回、出错退出或用户暂停时，helper 为本次执行仍按着的键逐一发送松开事件，每个事件都发回原接收进程。后续执行不能释放前一次执行的键。
|
||||
长按时，在 `run_code` 中调用 `app.sendKeyEvent('w', 'down')`。首次按下后，该键在同一次执行的后续工具调用中保持按下。对自己已按住的普通键再发一次 `down`，原接收进程会收到一次带重复标志的按下事件。调用方决定是否重复和间隔多久，松键只需一次 `up`。每个键可分别松开。例如先按 W，再按 D，松开 D 后 W 仍按着。程序正常返回、出错退出或用户暂停时，helper 为本次执行仍按着的键逐一发送松开事件，每个事件都发回原接收进程。后续执行不能释放前一次执行的键。
|
||||
|
||||
Agent 销毁和插件卸载会取消并等待完整工具调用，包括图像接入检查和附件存储。已经开始的图像写入必须结束后，销毁才会返回；已取消的调用不会返回成功截图。
|
||||
|
||||
@@ -224,7 +224,7 @@ Available at app or window level depending on the platform.
|
||||
- typeText(text: string) -> ActionResult — type literal text at the app's current focus; line endings send Return and tabs send Tab.
|
||||
- paste(text: string, options?: { format?: 'text' | 'html' }) -> ActionResult — deliver text or HTML through the app's Paste command, restoring prior clipboard contents unless another writer changes them; a Paste failure is surfaced without substituting typeText, and PASTE_NOT_CONSUMED means inspect before any retry.
|
||||
- pressKey(key: string, modifiers?: string[]) -> ActionResult — press and release one complete keystroke. For example pressKey('Tab', ['shift']) or pressKey('+'). Key and modifier names are platform-defined strings. Pass key and modifiers separately; do not pack them into a chord string.
|
||||
- sendKeyEvent(key: string, direction: 'down' | 'up') -> ActionResult — send 'down' once per key to hold one or more keys across tool calls within one run_code, for example W down, D down, D up, W up. Holding a physical key on a keyboard can produce repeated key-down events after a delay. This method posts one event per call and rejects a second 'down' for a held key. Use the app's response to decide what to do while holding the key and when to send 'up'. When this run_code returns, throws, or the user pauses it, macOS releases only its remaining keys at their original recipients.
|
||||
- sendKeyEvent(key: string, direction: 'down' | 'up') -> ActionResult — send 'down' to hold a key across tool calls in one run_code, for example W down, D down, D up, W up. Holding a key on a physical keyboard can produce repeated key-down events after a delay. To send one such repeat for a non-modifier key this run already holds, send 'down' again. Each call sends one event. Use the app's response to decide whether and when to repeat or release the key. When this run_code returns, throws, or the user pauses it, macOS releases only its remaining keys at their original recipients.
|
||||
|
||||
Raw tools use { method, params }: include appHandle for app calls and windowId for window calls, and use snapshotId with elementId for element routes. The emit option is facade-only: emit is local output control, so raw callers must follow the raw operation schema for admitted params. getAccessibilityTree and getWindow read through raw getState; getWindows maps to listWindows; getScreenshot maps to screenshot; cua.getState maps to listApps; every other named operation uses the same method name.
|
||||
```
|
||||
|
||||
@@ -21,8 +21,8 @@
|
||||
en: 2a69d89b7fc95998
|
||||
zh: 084575bb31fb40dd
|
||||
/dsh-cua-native-helper/observation-and-input:
|
||||
en: 0bbc620165d868fa
|
||||
zh: abe73bd9c7b8cc5d
|
||||
en: c6787eda94ba4833
|
||||
zh: 855e3c6432918775
|
||||
/dsh-cua-native-helper/verification:
|
||||
en: c87945aeec587bf5
|
||||
zh: e5d69e4927e2fed3
|
||||
|
||||
@@ -105,7 +105,7 @@ Capture and pointer delivery read CG bounds before and after an exact-window AX
|
||||
|
||||
`typeText` and `pressKey` target the selected process's current focused control, not an exact window. `pressKey` presses the listed modifiers followed by one key, then releases all pressed keys in reverse order, including on cancellation. Keyboard events go directly to the target PID and bypass the global HID queue. Key names are platform-defined; keypad names such as `KP_0` and `KP_Enter` preserve their physical key codes. Background pointer actions may change that keyboard focus. Text injection sends Unicode characters individually and does not use the clipboard. LF, CR and CRLF send one Return; tabs send Tab. Applications apply their normal editing behavior, including indentation and focus navigation. Each `typeText` down/up event is followed by the configured text interval; `pressKey` keeps its own press and release timing, and dispatch completion does not confirm that the receiving application consumed every event. Cancellation can interrupt text or a gesture after partial input; it cannot undo received events. Very long text can exceed the host's operation deadline. Validate observed application state before deciding how to continue.
|
||||
|
||||
The Host associates directional key events with one `run_code` execution. On down, the helper retains the selected application, recipient PID and process start time, physical key code, and matching up event. Explicit up and cleanup verify the original process identity. On return, error, or user pause, the Host waits for any in-flight native request and asks the helper to release only that execution’s remaining keys. Disconnect cleanup also releases keys owned by the closed connection.
|
||||
The Host associates directional key events with one `run_code` execution. On down, the helper retains the selected application, recipient PID and process start time, physical key code, and matching up event. A repeated down from the owner posts one key-down with `keyboardEventAutorepeat` set and reuses the existing record. Repeats, explicit up and cleanup verify the original process identity. A conflicting down or up reports the key and whether another session or another run holds it. On return, error, or user pause, the Host waits for any in-flight native request and asks the helper to release only that execution’s remaining keys. Disconnect cleanup also releases keys owned by the closed connection.
|
||||
|
||||
The selected helper application needs macOS Accessibility permission for AX state and input, and Screen Recording permission for screenshots. Capability preflight and permission errors do not grant permissions. Capture refuses a window that moves or resizes while the image is being read. Unassociated system panels and accessory-only applications are not selectable.
|
||||
|
||||
|
||||
@@ -105,7 +105,7 @@ AppKit 在主事件循环上运行。阻塞式 socket 接收与握手使用后
|
||||
|
||||
`typeText` 和 `pressKey` 面向所选进程当前获得焦点的控件，不指定窗口。`pressKey` 向目标 PID 依次发送修饰键和一个主键的按下事件，再逆序释放。操作中途取消也会逆序释放已按下的键。键盘事件直接发给目标进程，绕过全局 HID 队列。按键名由平台定义；KP_0 和 KP_Enter 等数字键盘名称保留物理键码。后台鼠标操作可能改变键盘焦点。文本逐个 Unicode 字符输入，不使用剪贴板。LF、CR 和 CRLF 发送一次 Return，制表符发送 Tab。应用保留缩进和焦点导航等正常编辑行为。`typeText` 的每次按下和抬起后等待配置的文本事件间隔；`pressKey` 使用自己的按下与释放节奏，分派完成不证明应用已消费全部事件。取消可能中断部分文本或手势，无法撤销已接收事件。很长的文本可能超过 Host 操作期限。继续输入前应核实应用状态。
|
||||
|
||||
宿主为每次 `run_code` 的 `sendKeyEvent` 附上执行标识。按下时，helper 保存已选应用、接收进程的 PID 与启动时间、物理键码及对应的松开事件。显式松键或执行结束时会核对原进程身份。正常返回、异常退出或用户暂停时，宿主等待在途原生请求结束，再通知 helper 释放该执行持有的其余按键。连接断开时，helper 也会清理该连接持有的按键。
|
||||
宿主为每次 `run_code` 的 `sendKeyEvent` 附上执行标识。按下时，helper 保存已选应用、接收进程的 PID 与启动时间、物理键码及对应的松开事件。持有方再次按下普通键时，helper 复用原记录，发送一次带 `keyboardEventAutorepeat` 标志的按下事件。重复、显式松键和执行结束时都会核对原进程身份。其他执行尝试按下或松开同一个键时，错误会指出键名，并区分其他会话和同一连接内的其他执行。正常返回、异常退出或用户暂停时，宿主等待在途原生请求结束，再通知 helper 释放该执行持有的其余按键。连接断开时，helper 也会清理该连接持有的按键。
|
||||
|
||||
所选 helper 应用需要 macOS 辅助功能权限以读取 AX 和输入，需要屏幕录制权限以截图。能力预检和权限错误不授予权限。截图会拒绝在读取图片期间移动或缩放的窗口。无法确认归属的系统面板及仅后台辅助类型的应用不可选择。
|
||||
|
||||
|
||||
@@ -215,7 +215,12 @@ actor AXWorker {
|
||||
return .result(Self.actionResult)
|
||||
}
|
||||
if method == "sendKeyEvent", try params.string("direction") == "up" {
|
||||
let dispatched = try inputs().0.keyUp(params.string("key"), app: params.string("appHandle"), run: params.string("runId"))
|
||||
let handle = try params.string("appHandle")
|
||||
let dispatched = try inputs().0.keyUp(params.string("key"), app: handle, run: params.string("runId")) {
|
||||
try NativePlatform.requireUnlocked(observeSession())
|
||||
try NativePlatform.requireAccessibility()
|
||||
return try ax().keyboardRecipient(validateIdentity(handle), catalog: catalog)
|
||||
}
|
||||
return .result(.object(["dispatched": .bool(dispatched)]))
|
||||
}
|
||||
let locked = observeSession()
|
||||
@@ -339,8 +344,9 @@ actor AXWorker {
|
||||
readTimeout: .milliseconds(try params.integer("pasteReadTimeoutMs", in: 1...2147483647)),
|
||||
maxElements: try params.integer("maxElements", in: 1...10000), maxDepth: try params.integer("maxDepth", in: 1...100)))
|
||||
case "sendKeyEvent":
|
||||
let recipient = try ax().keyboardRecipient(identity, catalog: catalog)
|
||||
try inputs().0.keyDown(params.string("key"), app: handle, run: params.string("runId"), recipient: recipient)
|
||||
try inputs().0.keyDown(params.string("key"), app: handle, run: params.string("runId")) {
|
||||
try ax().keyboardRecipient(identity, catalog: catalog)
|
||||
}
|
||||
case "typeText", "pressKey":
|
||||
let recipient = try ax().keyboardRecipient(identity, catalog: catalog)
|
||||
try NativePlatform.verify(recipient)
|
||||
|
||||
@@ -26,11 +26,34 @@ final class HeldKeys: @unchecked Sendable {
|
||||
lock.withLock { Dictionary(uniqueKeysWithValues: pressed.filter { $0.recipient.pid == pid }.map { ($0.code, $0.flag) }) }
|
||||
}
|
||||
|
||||
func down(owner: Owner, recipient: ProcessIdentity, code: CGKeyCode, flag: CGEventFlags,
|
||||
private func conflict(_ name: String, holder: Owner, caller: Owner) -> NativeFailure {
|
||||
let source: String
|
||||
if holder.connection != caller.connection { source = "another session" }
|
||||
else if holder.run != caller.run { source = "another run_code execution" }
|
||||
else { source = "a different application handle in this run_code" }
|
||||
return RequestParameters.invalid("Key '\(name)' is held for this target by \(source). No event was sent. Wait for the holder to release it or for its run_code to end.")
|
||||
}
|
||||
|
||||
/// Repeats use the saved recipient even when the selected application's focused panel has changed.
|
||||
func repeatIfHeld(owner: Owner, name: String, code: CGKeyCode, down: CGEvent, post: (CGEvent, Int32) -> Void) throws -> Bool {
|
||||
try lock.withLock {
|
||||
guard let key = pressed.first(where: { $0.owner == owner && $0.code == code }) else { return false }
|
||||
guard down.type == .keyDown else {
|
||||
throw RequestParameters.invalid("Modifier '\(name)' is already held. Send 'up' to release it.")
|
||||
}
|
||||
try NativePlatform.verify(key.recipient)
|
||||
down.flags = flags(for: key.recipient)
|
||||
down.setIntegerValueField(.keyboardEventAutorepeat, value: 1)
|
||||
post(down, key.recipient.pid)
|
||||
return true
|
||||
}
|
||||
}
|
||||
|
||||
func down(owner: Owner, recipient: ProcessIdentity, name: String, code: CGKeyCode, flag: CGEventFlags,
|
||||
down: CGEvent, up: CGEvent, post: (CGEvent, Int32) -> Void) throws {
|
||||
try lock.withLock {
|
||||
guard !pressed.contains(where: { $0.code == code && ($0.owner == owner || $0.recipient.pid == recipient.pid) }) else {
|
||||
throw RequestParameters.invalid("Key is already held for this target")
|
||||
if let key = pressed.first(where: { $0.code == code && $0.recipient.pid == recipient.pid }) {
|
||||
throw conflict(name, holder: key.owner, caller: owner)
|
||||
}
|
||||
pressed.append(Press(owner: owner, recipient: recipient, code: code, flag: flag, up: up))
|
||||
down.flags = flags(for: recipient)
|
||||
@@ -56,13 +79,20 @@ final class HeldKeys: @unchecked Sendable {
|
||||
return true
|
||||
}
|
||||
|
||||
func up(owner: Owner, code: CGKeyCode, post: (CGEvent, Int32) -> Void) throws -> Bool {
|
||||
try lock.withLock {
|
||||
guard let index = pressed.firstIndex(where: { $0.owner == owner && $0.code == code }) else {
|
||||
throw RequestParameters.invalid("This run has not pressed that key for this application")
|
||||
}
|
||||
func up(owner: Owner, name: String, code: CGKeyCode, recipientIfUnheld: () throws -> ProcessIdentity,
|
||||
post: (CGEvent, Int32) -> Void) throws -> Bool {
|
||||
let dispatched = try lock.withLock { () throws -> Bool? in
|
||||
guard let index = pressed.firstIndex(where: { $0.owner == owner && $0.code == code }) else { return nil }
|
||||
return try release(at: index, post: post)
|
||||
}
|
||||
if let dispatched { return dispatched }
|
||||
let recipient = try recipientIfUnheld()
|
||||
throw lock.withLock {
|
||||
if let key = pressed.first(where: { $0.code == code && $0.recipient == recipient }) {
|
||||
return conflict(name, holder: key.owner, caller: owner)
|
||||
}
|
||||
return RequestParameters.invalid("Key '\(name)' is not held for this target. Send 'down' in this run_code before 'up'.")
|
||||
}
|
||||
}
|
||||
|
||||
func release(connection: UUID, run: String?, post: (CGEvent, Int32) -> Void) throws {
|
||||
|
||||
@@ -22,20 +22,24 @@ final class KeyboardInput {
|
||||
self.checkCancellation = checkCancellation
|
||||
}
|
||||
|
||||
func keyDown(_ name: String, app: String, run: String, recipient: ProcessIdentity) throws {
|
||||
let modifier = Self.namedModifier(name)
|
||||
func keyDown(_ name: String, app: String, run: String, recipient: () throws -> ProcessIdentity) throws {
|
||||
let owner = HeldKeys.Owner(connection: connection, run: run, app: app)
|
||||
let code = try Self.keyCode(name)
|
||||
let source = try Self.source()
|
||||
let down = try Self.keyEvent(source: source, code: code, down: true, flags: [])
|
||||
try checkCancellation()
|
||||
if try held.repeatIfHeld(owner: owner, name: name, code: code, down: down, post: post) { return }
|
||||
let target = try recipient()
|
||||
let up = try Self.keyEvent(source: source, code: code, down: false, flags: [])
|
||||
try checkCancellation()
|
||||
try NativePlatform.verify(recipient)
|
||||
try held.down(owner: .init(connection: connection, run: run, app: app), recipient: recipient,
|
||||
code: code, flag: modifier?.flag ?? [], down: down, up: up, post: post)
|
||||
try NativePlatform.verify(target)
|
||||
try held.down(owner: owner, recipient: target, name: name, code: code, flag: Self.namedModifier(name)?.flag ?? [],
|
||||
down: down, up: up, post: post)
|
||||
}
|
||||
|
||||
func keyUp(_ name: String, app: String, run: String) throws -> Bool {
|
||||
try held.up(owner: .init(connection: connection, run: run, app: app), code: Self.keyCode(name), post: post)
|
||||
func keyUp(_ name: String, app: String, run: String, recipientIfUnheld: () throws -> ProcessIdentity) throws -> Bool {
|
||||
try held.up(owner: .init(connection: connection, run: run, app: app), name: name, code: Self.keyCode(name),
|
||||
recipientIfUnheld: recipientIfUnheld, post: post)
|
||||
}
|
||||
|
||||
func releaseHeldKeys(run: String? = nil) throws {
|
||||
@@ -52,7 +56,7 @@ final class KeyboardInput {
|
||||
chord.append(Key(keyCode: code, flag: Self.namedModifier(name)?.flag ?? []))
|
||||
let alreadyHeld = held.snapshot(pid: pid)
|
||||
guard alreadyHeld[code] == nil else {
|
||||
throw RequestParameters.invalid("Release the held key before using it in pressKey")
|
||||
throw RequestParameters.invalid("Key '\(name)' is already held for this target. Use pressKey after the run_code that holds it releases it.")
|
||||
}
|
||||
var selected = Set(alreadyHeld.keys)
|
||||
var flags = alreadyHeld.values.reduce(CGEventFlags()) { $0.union($1) }
|
||||
|
||||
@@ -85,8 +85,11 @@ final class CapabilitiesFixture: NSObject, NSApplicationDelegate, NSWindowDelega
|
||||
guard let self else { return }
|
||||
var value: [String: Any] = ["type": event.type.rawValue, "timestamp": event.timestamp, "flags": event.modifierFlags.rawValue,
|
||||
"shift": event.modifierFlags.contains(.shift), "command": event.modifierFlags.contains(.command)]
|
||||
if [.keyDown, .keyUp].contains(event.type) { value["keyCode"] = event.keyCode; value["characters"] = event.characters ?? "" }
|
||||
else { value["button"] = event.buttonNumber }
|
||||
if [.keyDown, .keyUp].contains(event.type) {
|
||||
value["keyCode"] = event.keyCode
|
||||
value["characters"] = event.characters ?? ""
|
||||
value["repeat"] = event.isARepeat
|
||||
} else { value["button"] = event.buttonNumber }
|
||||
self.events.append(value)
|
||||
self.persist()
|
||||
}
|
||||
|
||||
@@ -150,6 +150,17 @@ def check_keyboard(call, owner, state_file, state):
|
||||
state = wait_for_state(state_file, lambda value: len(value["events"]) >= count + 4)
|
||||
assert [(e["type"], e["keyCode"]) for e in state["events"][count:]] == [(10, 13), (10, 2), (11, 2), (11, 13)]
|
||||
|
||||
for explicit_up in (True, False):
|
||||
count = len(state["events"])
|
||||
key("w", "down")
|
||||
key("w", "down")
|
||||
if explicit_up:
|
||||
key("w", "up")
|
||||
call("releaseHeldKeys", {"runId": "held-input"})
|
||||
state = wait_for_state(state_file, lambda value: len(value["events"]) >= count + 3)
|
||||
repeat = [(e["type"], e["keyCode"], e["repeat"]) for e in state["events"][count:]]
|
||||
assert repeat == [(10, 13, False), (10, 13, True), (11, 13, False)], repeat
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser()
|
||||
@@ -191,7 +202,7 @@ def main():
|
||||
pid = wait_for_state(state_file, lambda value: value["windowId"] > 0)["pid"]
|
||||
if args.keyboard_only:
|
||||
check_keyboard(call, owner, state_file, json.loads(state_file.read_text()))
|
||||
print("PASS: Shift+D shortcut, two-second W+D hold, independent D up and remaining W cleanup")
|
||||
print("PASS: Shift+D shortcut, two-second W+D hold, independent D up, explicit W repeat and one final up")
|
||||
return
|
||||
if not args.click_only:
|
||||
call("activate", owner)
|
||||
|
||||
@@ -72,15 +72,66 @@ func plusAlonePressesShiftEquals(_ alias: String) throws {
|
||||
#expect(pid == target.pid)
|
||||
events.append(event)
|
||||
}, textEventInterval: 0, held: held)
|
||||
try input.keyDown("shift", app: "editor", run: "one", recipient: target)
|
||||
try input.keyDown("w", app: "editor", run: "two", recipient: target)
|
||||
#expect(throws: NativeFailure.self) { try input.keyDown("W", app: "editor", run: "one", recipient: target) }
|
||||
#expect(throws: NativeFailure.self) { try input.keyUp("w", app: "other", run: "two") }
|
||||
try input.keyDown("shift", app: "editor", run: "one", recipient: { target })
|
||||
#expect(throws: NativeFailure.self) { try input.keyDown("shift", app: "editor", run: "one", recipient: { target }) }
|
||||
try input.keyDown("w", app: "editor", run: "two", recipient: { target })
|
||||
#expect(throws: NativeFailure.self) { try input.keyDown("W", app: "editor", run: "one", recipient: { target }) }
|
||||
#expect(throws: NativeFailure.self) { try input.keyUp("w", app: "other", run: "two", recipientIfUnheld: { target }) }
|
||||
#expect(throws: NativeFailure.self) { try input.pressKey("w", modifiers: [], to: target.pid) }
|
||||
try input.pressKey("d", modifiers: ["shift"], to: target.pid)
|
||||
try input.releaseHeldKeys(run: "one")
|
||||
#expect(try input.keyUp("w", app: "editor", run: "two"))
|
||||
#expect(try input.keyUp("w", app: "editor", run: "two", recipientIfUnheld: { target }))
|
||||
try input.releaseHeldKeys(run: "two")
|
||||
#expect(events.map { $0.getIntegerValueField(.keyboardEventKeycode) } == [56, 13, 2, 2, 56, 13])
|
||||
#expect(events.map { $0.flags.contains(.maskShift) } == [true, true, true, true, false, false])
|
||||
}
|
||||
|
||||
@Test(arguments: [true, false])
|
||||
func repeatedDownUsesOriginalRecipientAndOneRelease(_ explicitUp: Bool) throws {
|
||||
let target = try NativePlatform.identity(getpid())
|
||||
var events: [CGEvent] = []
|
||||
let input = KeyboardInput(post: { event, pid in
|
||||
#expect(pid == target.pid)
|
||||
events.append(event)
|
||||
}, textEventInterval: 0)
|
||||
func changedFocus() throws -> ProcessIdentity {
|
||||
throw RequestParameters.invalid("The repeat or release looked up a new recipient")
|
||||
}
|
||||
try input.keyDown("w", app: "editor", run: "one", recipient: { target })
|
||||
try input.keyDown("W", app: "editor", run: "one", recipient: changedFocus)
|
||||
if explicitUp { #expect(try input.keyUp("w", app: "editor", run: "one", recipientIfUnheld: changedFocus)) }
|
||||
try input.releaseHeldKeys(run: "one")
|
||||
#expect(events.map(\.type) == [.keyDown, .keyDown, .keyUp])
|
||||
#expect(events.map { $0.getIntegerValueField(.keyboardEventAutorepeat) } == [0, 1, 0])
|
||||
}
|
||||
|
||||
@Test func foreignSessionCannotRepeatOrReleaseTheOwnersKey() throws {
|
||||
let target = try NativePlatform.identity(getpid())
|
||||
let held = HeldKeys()
|
||||
var events: [CGEvent] = []
|
||||
let a = KeyboardInput(post: { event, _ in events.append(event) }, textEventInterval: 0, held: held)
|
||||
let b = KeyboardInput(post: { _, _ in Issue.record("Non-owner dispatched an event") }, textEventInterval: 0, held: held)
|
||||
try a.keyDown("w", app: "a", run: "same-run-id", recipient: { target })
|
||||
for direction in ["down", "up"] {
|
||||
do {
|
||||
if direction == "down" { try b.keyDown("W", app: "b", run: "same-run-id", recipient: { target }) }
|
||||
else { _ = try b.keyUp("W", app: "b", run: "same-run-id", recipientIfUnheld: { target }) }
|
||||
Issue.record("Non-owner was allowed to send an event")
|
||||
} catch let failure as NativeFailure {
|
||||
#expect(failure.code == "INVALID_ARGUMENT")
|
||||
#expect(failure.message.contains("Key 'W' is held for this target by another session"))
|
||||
#expect(failure.message.contains("Wait for the holder to release it"))
|
||||
}
|
||||
}
|
||||
try b.releaseHeldKeys()
|
||||
try a.keyDown("W", app: "a", run: "same-run-id", recipient: { target })
|
||||
try a.releaseHeldKeys()
|
||||
#expect(events.map(\.type) == [.keyDown, .keyDown, .keyUp])
|
||||
do {
|
||||
_ = try b.keyUp("W", app: "b", run: "same-run-id", recipientIfUnheld: { target })
|
||||
Issue.record("Unheld key was released")
|
||||
} catch let failure as NativeFailure {
|
||||
#expect(failure.code == "INVALID_ARGUMENT")
|
||||
#expect(failure.message.contains("Key 'W' is not held for this target"))
|
||||
}
|
||||
}
|
||||
@@ -14,7 +14,7 @@ export const GUIDANCE = `${PLATFORM_GUIDANCE}
|
||||
|
||||
${COMMON_GUIDANCE.replace(/^- ([\w.]+)\(.*$/gm, (entry, method: string) => {
|
||||
if (method === 'pressKey') return `${entry}
|
||||
- sendKeyEvent(key: string, direction: 'down' | 'up') -> ActionResult — send 'down' once per key to hold one or more keys across tool calls within one run_code, for example W down, D down, D up, W up. Holding a physical key on a keyboard can produce repeated key-down events after a delay. This method posts one event per call and rejects a second 'down' for a held key. Use the app's response to decide what to do while holding the key and when to send 'up'. When this run_code returns, throws, or the user pauses it, macOS releases only its remaining keys at their original recipients.`
|
||||
- sendKeyEvent(key: string, direction: 'down' | 'up') -> ActionResult — send 'down' to hold a key across tool calls in one run_code, for example W down, D down, D up, W up. Holding a key on a physical keyboard can produce repeated key-down events after a delay. To send one such repeat for a non-modifier key this run already holds, send 'down' again. Each call sends one event. Use the app's response to decide whether and when to repeat or release the key. When this run_code returns, throws, or the user pauses it, macOS releases only its remaining keys at their original recipients.`
|
||||
const detail = methodDetails.get(method)
|
||||
return detail ? `${entry} ${detail}` : entry
|
||||
})}`
|
||||
@@ -79,7 +79,7 @@ Available at app or window level depending on the platform.
|
||||
- typeText(text: string) -> ActionResult — type literal text at the app's current focus; line endings send Return and tabs send Tab.
|
||||
- paste(text: string, options?: { format?: 'text' | 'html' }) -> ActionResult — deliver text or HTML through the app's Paste command, restoring prior clipboard contents unless another writer changes them; a Paste failure is surfaced without substituting typeText, and PASTE_NOT_CONSUMED means inspect before any retry.
|
||||
- pressKey(key: string, modifiers?: string[]) -> ActionResult — press and release one complete keystroke. For example pressKey('Tab', ['shift']) or pressKey('+'). Key and modifier names are platform-defined strings. Pass key and modifiers separately; do not pack them into a chord string.
|
||||
- sendKeyEvent(key: string, direction: 'down' | 'up') -> ActionResult — send 'down' once per key to hold one or more keys across tool calls within one run_code, for example W down, D down, D up, W up. Holding a physical key on a keyboard can produce repeated key-down events after a delay. This method posts one event per call and rejects a second 'down' for a held key. Use the app's response to decide what to do while holding the key and when to send 'up'. When this run_code returns, throws, or the user pauses it, macOS releases only its remaining keys at their original recipients.
|
||||
- sendKeyEvent(key: string, direction: 'down' | 'up') -> ActionResult — send 'down' to hold a key across tool calls in one run_code, for example W down, D down, D up, W up. Holding a key on a physical keyboard can produce repeated key-down events after a delay. To send one such repeat for a non-modifier key this run already holds, send 'down' again. Each call sends one event. Use the app's response to decide whether and when to repeat or release the key. When this run_code returns, throws, or the user pauses it, macOS releases only its remaining keys at their original recipients.
|
||||
|
||||
Raw tools use { method, params }: include appHandle for app calls and windowId for window calls, and use snapshotId with elementId for element routes. The emit option is facade-only: emit is local output control, so raw callers must follow the raw operation schema for admitted params. getAccessibilityTree and getWindow read through raw getState; getWindows maps to listWindows; getScreenshot maps to screenshot; cua.getState maps to listApps; every other named operation uses the same method name.
|
||||
|
||||
|
||||
@@ -81,7 +81,7 @@ Available at app or window level depending on the platform.
|
||||
- typeText(text: string) -> ActionResult — type literal text at the app's current focus; line endings send Return and tabs send Tab.
|
||||
- paste(text: string, options?: { format?: 'text' | 'html' }) -> ActionResult — deliver text or HTML through the app's Paste command, restoring prior clipboard contents unless another writer changes them; a Paste failure is surfaced without substituting typeText, and PASTE_NOT_CONSUMED means inspect before any retry.
|
||||
- pressKey(key: string, modifiers?: string[]) -> ActionResult — press and release one complete keystroke. For example pressKey('Tab', ['shift']) or pressKey('+'). Key and modifier names are platform-defined strings. Pass key and modifiers separately; do not pack them into a chord string.
|
||||
- sendKeyEvent(key: string, direction: 'down' | 'up') -> ActionResult — send 'down' once per key to hold one or more keys across tool calls within one run_code, for example W down, D down, D up, W up. Holding a physical key on a keyboard can produce repeated key-down events after a delay. This method posts one event per call and rejects a second 'down' for a held key. Use the app's response to decide what to do while holding the key and when to send 'up'. When this run_code returns, throws, or the user pauses it, macOS releases only its remaining keys at their original recipients.
|
||||
- sendKeyEvent(key: string, direction: 'down' | 'up') -> ActionResult — send 'down' to hold a key across tool calls in one run_code, for example W down, D down, D up, W up. Holding a key on a physical keyboard can produce repeated key-down events after a delay. To send one such repeat for a non-modifier key this run already holds, send 'down' again. Each call sends one event. Use the app's response to decide whether and when to repeat or release the key. When this run_code returns, throws, or the user pauses it, macOS releases only its remaining keys at their original recipients.
|
||||
|
||||
Raw tools use { method, params }: include appHandle for app calls and windowId for window calls, and use snapshotId with elementId for element routes. The emit option is facade-only: emit is local output control, so raw callers must follow the raw operation schema for admitted params. getAccessibilityTree and getWindow read through raw getState; getWindows maps to listWindows; getScreenshot maps to screenshot; cua.getState maps to listApps; every other named operation uses the same method name.
|
||||
|
||||
|
||||
@@ -81,7 +81,7 @@ Available at app or window level depending on the platform.
|
||||
- typeText(text: string) -> ActionResult — type literal text at the app's current focus; line endings send Return and tabs send Tab.
|
||||
- paste(text: string, options?: { format?: 'text' | 'html' }) -> ActionResult — deliver text or HTML through the app's Paste command, restoring prior clipboard contents unless another writer changes them; a Paste failure is surfaced without substituting typeText, and PASTE_NOT_CONSUMED means inspect before any retry.
|
||||
- pressKey(key: string, modifiers?: string[]) -> ActionResult — press and release one complete keystroke. For example pressKey('Tab', ['shift']) or pressKey('+'). Key and modifier names are platform-defined strings. Pass key and modifiers separately; do not pack them into a chord string.
|
||||
- sendKeyEvent(key: string, direction: 'down' | 'up') -> ActionResult — send 'down' once per key to hold one or more keys across tool calls within one run_code, for example W down, D down, D up, W up. Holding a physical key on a keyboard can produce repeated key-down events after a delay. This method posts one event per call and rejects a second 'down' for a held key. Use the app's response to decide what to do while holding the key and when to send 'up'. When this run_code returns, throws, or the user pauses it, macOS releases only its remaining keys at their original recipients.
|
||||
- sendKeyEvent(key: string, direction: 'down' | 'up') -> ActionResult — send 'down' to hold a key across tool calls in one run_code, for example W down, D down, D up, W up. Holding a key on a physical keyboard can produce repeated key-down events after a delay. To send one such repeat for a non-modifier key this run already holds, send 'down' again. Each call sends one event. Use the app's response to decide whether and when to repeat or release the key. When this run_code returns, throws, or the user pauses it, macOS releases only its remaining keys at their original recipients.
|
||||
|
||||
Raw tools use { method, params }: include appHandle for app calls and windowId for window calls, and use snapshotId with elementId for element routes. The emit option is facade-only: emit is local output control, so raw callers must follow the raw operation schema for admitted params. getAccessibilityTree and getWindow read through raw getState; getWindows maps to listWindows; getScreenshot maps to screenshot; cua.getState maps to listApps; every other named operation uses the same method name.
|
||||
|
||||
|
||||
Reference in new issue
Block a user