Update CUA replay fixtures and native examples to the current API
This commit is contained in:
creatixchu committed
1 parent
d3304de65b
commit
c301c68825
19 files changed
+321
-302
No files matched your search
@@ -15,8 +15,8 @@
|
||||
en: 1777e7bf6785f986
|
||||
zh: e3daac1074ebb74d
|
||||
/deepseek-ai-dsh-experimental-cua-macos/use-this-package:
|
||||
en: fc4209dcb76851a8
|
||||
zh: 6df3337ab2a964eb
|
||||
en: 7fdb7c0462275417
|
||||
zh: 0a4a9562f4a82945
|
||||
/deepseek-ai-dsh-experimental-cua-macos/use-this-package/connect-an-external-execution-environment:
|
||||
en: 8bf27cf5a258b1f4
|
||||
zh: dba694e92bc6c091
|
||||
@@ -45,8 +45,8 @@
|
||||
en: a2c9acb5a4320dda
|
||||
zh: e8d1586af60215c4
|
||||
/deepseek-ai-dsh-experimental-cua-macos/known-limitations-and-deferred-work:
|
||||
en: 4fefaf88ea9eefc6
|
||||
zh: d3d18cfbccf8c20f
|
||||
en: dfedb97af676b4f4
|
||||
zh: 1c910e047dbb04ff
|
||||
/deepseek-ai-dsh-experimental-cua-macos/known-limitations-and-deferred-work/dev-note:
|
||||
en: b836f01bd0bceb4a
|
||||
zh: 8122fa08b2aa95fa
|
||||
@@ -88,7 +88,7 @@ Raw `getState` exposes validated state fields and the complete text for the requ
|
||||
|
||||
AX observations show menu-bar entries by default. Use `app.getAccessibilityTree({ includeMenus: true })` to expand menu contents and `app.getAccessibilityTree({ details: true })` for identifiers, global-point bounds and complete action names. These options also apply to raw `getState`. Every read replaces the application's current snapshot; use element IDs from that read. Details do not bypass the configured node and depth limits. Truncation identifies the reached limit.
|
||||
|
||||
Values and placeholders are separate. Named, stateful or actionable groups and scroll areas retain their own rows. Repeated names, descriptions and help text on one node are shown once. Values, placeholders and URLs keep their distinct meanings even when their text matches; body text shared by distinct nodes remains intact. Writability is explicit on relevant controls, including `AXIncrementor`; `unknown` means the system did not report it. Missing optional state fields do not imply false. UTF-16 ranges outside the observed value are marked unavailable. Custom action aliases omit AppKit target/selector diagnostics; ambiguous aliases require a complete name from `details:true`.
|
||||
Values and placeholders are separate. Named, stateful or actionable groups and scroll areas retain their own rows. Repeated names, descriptions and help text on one node are shown once. Values, placeholders and URLs keep their distinct meanings even when their text matches; body text shared by distinct nodes remains intact. Writability is explicit on relevant controls, including `AXIncrementor`; `unknown` means the system did not report it. Missing optional state fields do not imply false. UTF-16 ranges outside the observed value are marked unavailable. Compact tree labels can omit AppKit target and selector details. Pass the exact name returned by `listActions(elementId)` to `performAction`.
|
||||
|
||||
Observations emit by default and also return their value. In persistent PTC, use `void await app.getAccessibilityTree()` for one emitted copy, or `emit:false` and one explicit print. Capturing a screenshot after an emitted tree can fail independently; retain the successful AX output.
|
||||
|
||||
@@ -173,7 +173,7 @@ The plugin contributes one `cua` operation tool with method-specific parameters.
|
||||
##### Operation guidance (verbatim)
|
||||
|
||||
```markdown
|
||||
macOS: accessibility tree operations are available at app or window level. Simulated keyboard input is available at app level. Simulated mouse input is available at window level. getApp also accepts bundle IDs and .app paths, and findApps searches /Applications, /System/Applications, and ~/Applications. Use the Computer Use permission page to grant the helper application Accessibility and Screen Recording permissions in System Settings. Native window and AX bounds use macOS points, while delivered screenshots carry their own pixel dimensions. show does not switch Spaces, and Stage Manager can present reduced window geometry; explain any required foreground step before activate or raise.
|
||||
macOS: accessibility tree operations are available at app or window level. Simulated keyboard input is available at app level. Simulated mouse input is available at window level. Use window coordinates for click and mouseMove on macOS. The helper rejects element-targeted pointer requests carrying the window ID supplied by these object methods. For an element’s default action, use invoke with its latest ID. getApp also accepts bundle IDs and .app paths, and findApps searches /Applications, /System/Applications, and ~/Applications. Use the Computer Use permission page to grant the helper application Accessibility and Screen Recording permissions in System Settings. Native window and AX bounds use macOS points, while delivered screenshots carry their own pixel dimensions. show does not switch Spaces, and Stage Manager can present reduced window geometry; explain any required foreground step before activate or raise.
|
||||
|
||||
Use cua to observe and operate native applications. Always use cua.getCapabilities() to confirm whether a specific API is available on the current platform; its reported capabilities take precedence over availability statements in these instructions. All methods are async; observation options default to { emit: true }, and emit: false suppresses sink output only, leaving returned values unchanged. Actions report dispatch evidence, not verified completion. If sessionLocked is true or an operation reports SESSION_LOCKED, stop UI actions and ask the user to unlock the desktop session; null or unknown means verify a usable unlocked desktop session first. cua.getApp requests user consent; if access is denied or revoked, stop using that application until the user explicitly permits another request, and never bypass a refusal or switch tools to avoid it. Never replay uncertain input without a fresh observation.
|
||||
|
||||
@@ -241,6 +241,7 @@ The queue orders individual native requests. Holding an app object does not rese
|
||||
- The guide closes when its selected settings window leaves the visible-window list, including when a Space switch or minimization removes it. Request permission again after returning to reopen the guide.
|
||||
- Only the macOS native implementation is provided. Windows support is in development; this package remains in `experimental` until that support is complete, then moves to the `computer-use` group. Browser tabs, AX diffs, unassociated system panels and accessory-only apps are unsupported.
|
||||
- Element IDs belong to the application's current accessibility snapshot; a later read replaces them. Screenshots and pointer input require an explicit window object.
|
||||
- On macOS, the window object includes its window ID on element-targeted `click` and `mouseMove` calls. The helper rejects that combination with `INVALID_ARGUMENT`. Use window coordinates for pointer input or `invoke(elementId)` for a control’s default action.
|
||||
- Coordinates use the returned screenshot pixels, including the title bar; `relative: true` selects integers from 0 to 999. Native point conversion is Host-owned. Window movement, application changes and other desktop users can race observations and input.
|
||||
- macOS Stage Manager can reduce background window bounds and screenshots to thumbnails. Full-window observation and coordinate input in this state are not guaranteed; model guidance requires explaining foreground needs and avoiding repeated focus reclamation.
|
||||
- Background input depends on private SkyLight symbols; macOS updates can make them unavailable. Validate each target OS version.
|
||||
|
||||
@@ -88,7 +88,7 @@ CUA 工具在读取 AX、窗口标题、截图、激活和输入前检查应用
|
||||
|
||||
AX 观察默认只展示菜单栏入口。使用 `app.getAccessibilityTree({ includeMenus: true })` 展开菜单内容，使用 `app.getAccessibilityTree({ details: true })` 读取标识符、全局 point 坐标和完整动作名。原始 `getState` 也接受这两个选项。每次读取都会替换应用当前的快照；应使用该次读取返回的元素 ID。详情不会绕过配置的节点数和深度上限；截断提示会指明触发的上限。
|
||||
|
||||
实际值与占位文字分别展示。有名称、状态或可用动作的分组和滚动区域保留独立行。同一节点中重复的名称、描述和帮助只显示一次。实际值、占位文字和 URL 即使文字相同也保留各自含义，不删除不同节点共有的正文。相关控件明确展示可写性，包括 `AXIncrementor`；`unknown` 表示系统未报告。缺少可选状态字段不代表 false。超出当前值长度的 UTF-16 选区标为不可用。自定义动作简称省略 AppKit 的 target/selector 诊断信息；同名动作需要使用 `details:true` 返回的完整名称。
|
||||
实际值与占位文字分别展示。有名称、状态或可用动作的分组和滚动区域保留独立行。同一节点中重复的名称、描述和帮助只显示一次。实际值、占位文字和 URL 即使文字相同也保留各自含义，不删除不同节点共有的正文。相关控件明确展示可写性，包括 `AXIncrementor`；`unknown` 表示系统未报告。缺少可选状态字段不代表 false。超出当前值长度的 UTF-16 选区标为不可用。树中的动作简称可能省略 AppKit 的 target 和 selector 信息。调用 `performAction` 时使用 `listActions(elementId)` 返回的完整名称。
|
||||
|
||||
观察方法默认自动输出，同时返回结果。在 persistent PTC 中使用 `void await app.getAccessibilityTree()` 输出一次，或指定 `emit:false` 后自行打印一次。AX 输出后的截图可能单独失败，已成功取得的 AX 输出仍保留。
|
||||
|
||||
@@ -173,7 +173,7 @@ AppKit sheet 可以通过其窗口 ID 单独选择，也会出现在所属文档
|
||||
##### 操作指导文本（原文）
|
||||
|
||||
```markdown
|
||||
macOS: accessibility tree operations are available at app or window level. Simulated keyboard input is available at app level. Simulated mouse input is available at window level. getApp also accepts bundle IDs and .app paths, and findApps searches /Applications, /System/Applications, and ~/Applications. Use the Computer Use permission page to grant the helper application Accessibility and Screen Recording permissions in System Settings. Native window and AX bounds use macOS points, while delivered screenshots carry their own pixel dimensions. show does not switch Spaces, and Stage Manager can present reduced window geometry; explain any required foreground step before activate or raise.
|
||||
macOS: accessibility tree operations are available at app or window level. Simulated keyboard input is available at app level. Simulated mouse input is available at window level. Use window coordinates for click and mouseMove on macOS. The helper rejects element-targeted pointer requests carrying the window ID supplied by these object methods. For an element’s default action, use invoke with its latest ID. getApp also accepts bundle IDs and .app paths, and findApps searches /Applications, /System/Applications, and ~/Applications. Use the Computer Use permission page to grant the helper application Accessibility and Screen Recording permissions in System Settings. Native window and AX bounds use macOS points, while delivered screenshots carry their own pixel dimensions. show does not switch Spaces, and Stage Manager can present reduced window geometry; explain any required foreground step before activate or raise.
|
||||
|
||||
Use cua to observe and operate native applications. Always use cua.getCapabilities() to confirm whether a specific API is available on the current platform; its reported capabilities take precedence over availability statements in these instructions. All methods are async; observation options default to { emit: true }, and emit: false suppresses sink output only, leaving returned values unchanged. Actions report dispatch evidence, not verified completion. If sessionLocked is true or an operation reports SESSION_LOCKED, stop UI actions and ask the user to unlock the desktop session; null or unknown means verify a usable unlocked desktop session first. cua.getApp requests user consent; if access is denied or revoked, stop using that application until the user explicitly permits another request, and never bypass a refusal or switch tools to avoid it. Never replay uncertain input without a fresh observation.
|
||||
|
||||
@@ -241,6 +241,7 @@ Raw tools use { method, params }: include appHandle for app calls and windowId f
|
||||
- 所选设置窗口离开可见窗口列表时，引导框会关闭，包括切换桌面空间或最小化导致窗口不再可见的情况。返回后再次申请权限可重新打开引导。
|
||||
- 只提供 macOS 原生实现，Windows 支持正在开发中。本包暂放在 `experimental`，待 Windows 支持完成后迁回 `computer-use` 组。不支持浏览器标签、启动应用、AX 差分、无法确认应用归属的系统面板，以及仅后台辅助类型的应用。
|
||||
- 元素 ID 只对应用当前的辅助功能快照有效；之后的读取会替换它们。截图和指针输入都要求显式的窗口对象。
|
||||
- 在 macOS 上，窗口对象调用 `click(elementId)` 或 `mouseMove(elementId)` 时会附带窗口 ID，helper 会以 `INVALID_ARGUMENT` 拒绝该组合。鼠标操作可传窗口坐标，执行控件的默认动作可调用 `invoke(elementId)`。
|
||||
- 坐标使用包含标题栏的返回截图像素；`relative: true` 使用 0～999 的整数，由 Host 换算原生 point。窗口移动、应用变化及其他桌面使用者的操作可能与观察和输入交错。
|
||||
- macOS 台前调度可能将后台窗口的尺寸与截图变为缩略图，当前无法保证这种状态下的完整画面与坐标输入；模型指引要求说明前台需求并避免反复抢回焦点。
|
||||
- 后台输入依赖私有 SkyLight 符号，macOS 升级可能使其不可用，需在目标系统版本验证。
|
||||
|
||||
@@ -16,7 +16,7 @@ import tempfile
|
||||
import time
|
||||
|
||||
from native_test_support import ROOT, NativeHelper, run_cleanup
|
||||
from capabilities_e2e import stop_application
|
||||
from capabilities_e2e import stop_application, wait_for_state
|
||||
|
||||
|
||||
def main():
|
||||
@@ -50,7 +50,9 @@ def main():
|
||||
binary = app_binary('Target', state_path)
|
||||
subprocess.run(['swiftc', str(ROOT / 'tests/AxlessFixture.swift'), '-o', str(binary)], check=True)
|
||||
pending = helper.call('getApp', {'selector': str(binary.parents[2])})
|
||||
fixture = helper.call('launchApp', {'appHandle': pending['appHandle']})['pid']
|
||||
launched = helper.call('launchApp', {'appHandle': pending['appHandle']})
|
||||
assert launched['running'] is True, launched
|
||||
fixture = wait_for_state(state_path, lambda state: state['pid'] > 0)['pid']
|
||||
try:
|
||||
|
||||
def wait_state(predicate, description):
|
||||
@@ -128,7 +130,9 @@ def main():
|
||||
binary = app_binary('Restoration', state_path)
|
||||
shutil.copy2(original_binary, binary)
|
||||
pending = helper.call('getApp', {'selector': str(binary.parents[2])})
|
||||
fixture = helper.call('launchApp', {'appHandle': pending['appHandle']})['pid']
|
||||
launched = helper.call('launchApp', {'appHandle': pending['appHandle']})
|
||||
assert launched['running'] is True, launched
|
||||
fixture = wait_for_state(state_path, lambda state: state['pid'] > 0)['pid']
|
||||
actual = wait_state(lambda state: bool(state.get('windowId')), 'restoration fixture did not initialize')
|
||||
owner = {'appHandle': pending['appHandle']}
|
||||
window_id = actual['windowId']
|
||||
@@ -137,7 +141,9 @@ def main():
|
||||
foreground_binary = app_binary('Foreground', foreground_path)
|
||||
shutil.copy2(binary, foreground_binary)
|
||||
pending_foreground = helper.call('getApp', {'selector': str(foreground_binary.parents[2])})
|
||||
foreground = helper.call('launchApp', {'appHandle': pending_foreground['appHandle']})['pid']
|
||||
launched = helper.call('launchApp', {'appHandle': pending_foreground['appHandle']})
|
||||
assert launched['running'] is True, launched
|
||||
foreground = wait_for_state(foreground_path, lambda state: state['pid'] > 0)['pid']
|
||||
subprocess.run(['open', '-a', str(foreground_binary.parents[2])], check=True)
|
||||
wait_state(lambda state: foreground_path.exists() and json.loads(foreground_path.read_text()).get('active')
|
||||
and not state['active'], 'foreground fixture did not take focus')
|
||||
|
||||
@@ -168,8 +168,8 @@ def exercise(args, artifacts, transcript, summary):
|
||||
"maxElements": 2000, "maxDepth": 25})
|
||||
window_elements = [element for element in observed["elements"] if element["role"] == "AXWindow"]
|
||||
assert len(window_elements) == 1, observed
|
||||
call("performSecondaryAction", {**owner, "snapshotId": observed["snapshotId"],
|
||||
"elementId": window_elements[0]["elementId"], "action": "raise"})
|
||||
call("performAction", {**owner, "snapshotId": observed["snapshotId"],
|
||||
"elementId": window_elements[0]["elementId"], "action": "AXRaise"})
|
||||
observed = call("getState", {**owner, "windowId": window["windowId"],
|
||||
"maxElements": 2000, "maxDepth": 25})
|
||||
window = next(value for value in observed["windows"] if value["windowId"] == window["windowId"])
|
||||
|
||||
@@ -147,12 +147,12 @@ def main():
|
||||
matches = call("findApps", {"query": str(application), "maxAppMatches": 20})
|
||||
assert len(matches) == 1 and matches[0]["isRunning"] is False, matches
|
||||
pending = call("getApp", {"selector": str(application)})
|
||||
assert pending["pid"] is None and not state_file.exists(), pending
|
||||
assert pending["running"] is False and not state_file.exists(), pending
|
||||
owner = {"appHandle": pending["appHandle"]}
|
||||
# The Host consent barrier is separately exercised by provider-lifecycle.spec.ts.
|
||||
app = call("launchApp", owner)
|
||||
pid = app["pid"]
|
||||
wait_for_state(state_file, lambda value: value["pid"] == pid and value["windowId"] > 0)
|
||||
assert app["running"] is True, app
|
||||
pid = wait_for_state(state_file, lambda value: value["windowId"] > 0)["pid"]
|
||||
if not args.click_only:
|
||||
call("activate", owner)
|
||||
state = wait_for_state(state_file, lambda value: value["pid"] == pid and value["active"] and value["keyWindow"])
|
||||
|
||||
@@ -51,7 +51,7 @@ def check_file_panel(call, owner, primary_params, primary, directory, wait_state
|
||||
assert any(e["role"] == "AXSheet" for e in scoped["elements"]), scoped
|
||||
assert "Increment primary" not in scoped["tree"], scoped
|
||||
panel = next(e for e in scoped["elements"] if e["identifier"] == "open-panel")
|
||||
call("performSecondaryAction", {**owner, "snapshotId": scoped["snapshotId"], "elementId": panel["elementId"], "action": "raise"})
|
||||
call("performAction", {**owner, "snapshotId": scoped["snapshotId"], "elementId": panel["elementId"], "action": "AXRaise"})
|
||||
deadline = time.monotonic() + 30
|
||||
while True:
|
||||
try:
|
||||
@@ -188,8 +188,8 @@ def main():
|
||||
call("activate", owner)
|
||||
current = call("getState", primary_params)
|
||||
window_element = next(element for element in current["elements"] if element["role"] == "AXWindow")
|
||||
call("performSecondaryAction", {**owner, "snapshotId": current["snapshotId"],
|
||||
"elementId": window_element["elementId"], "action": "raise"})
|
||||
call("performAction", {**owner, "snapshotId": current["snapshotId"],
|
||||
"elementId": window_element["elementId"], "action": "AXRaise"})
|
||||
call("getState", primary_params)
|
||||
def mouse_move(target):
|
||||
result = call("mouseMove", target)
|
||||
|
||||
@@ -222,11 +222,11 @@ def same_value(left, right):
|
||||
return left == right
|
||||
|
||||
|
||||
def check_observation(value, row_count, fixture_pid, window_id):
|
||||
required = {"appHandle", "snapshotId", "pid", "windowId", "tree", "elements", "windows", "focusedWindowId", "truncated"}
|
||||
def check_observation(value, row_count, app_handle, window_id):
|
||||
required = {"appHandle", "snapshotId", "windowId", "tree", "elements", "windows", "focusedWindowId", "truncated"}
|
||||
if not isinstance(value, dict) or not required.issubset(value):
|
||||
raise AssertionError("Observation is missing required result fields")
|
||||
if value["pid"] != fixture_pid or value["windowId"] != window_id:
|
||||
if value["appHandle"] != app_handle or value["windowId"] != window_id:
|
||||
raise AssertionError("Observation belongs to a different fixture process or window")
|
||||
if value["truncated"] is not False:
|
||||
raise AssertionError("Complete benchmark observation was truncated")
|
||||
@@ -312,7 +312,7 @@ def scenario_run(args, binary, work, artifacts, row_count, deadline, summary, ow
|
||||
for label in ["reference", "candidate"]:
|
||||
params = {**owners[label], "windowId": window_id, **OBSERVATION_LIMITS}
|
||||
result, record = helpers[label].call("getState", params, phase, group)
|
||||
observation = check_observation(result, row_count, fixture.process.pid, window_id)
|
||||
observation = check_observation(result, row_count, owners[label]["appHandle"], window_id)
|
||||
record.update({"observationSHA256": sha256(json_bytes(observation)),
|
||||
"treeUTF8Bytes": len(result["tree"].encode("utf-8")), "elementCount": len(result["elements"]),
|
||||
"windowCount": len(result["windows"]), "staticRows": row_count})
|
||||
|
||||
@@ -94,16 +94,6 @@ private func node(_ role: String = "AXButton") -> AXTextNode {
|
||||
}
|
||||
}
|
||||
|
||||
@Test func actionAliasesResolveOnlyWhenUnambiguous() throws {
|
||||
let first = "Name:Move previous\nTarget:0x123\nSelector:(null)"
|
||||
let other = "Name:Move previous\nTarget:0x456\nSelector:(null)"
|
||||
#expect(try AXWorker.resolveAction("Move previous", names: [first]) == first)
|
||||
#expect(try AXWorker.resolveAction("press", names: ["AXPress"]) == "AXPress")
|
||||
#expect(throws: NativeFailure.self) { try AXWorker.resolveAction("Move previous", names: [first, other]) }
|
||||
#expect(try AXWorker.resolveAction(other, names: [first, other]) == other)
|
||||
#expect(throws: NativeFailure.self) { try AXWorker.resolveAction("missing", names: [first]) }
|
||||
}
|
||||
|
||||
@Test func identifiesTheReachedTraversalLimit() {
|
||||
#expect(AXTextFormatter.truncation(maxElements: nil, maxDepth: 25).contains("(maxDepth=25)"))
|
||||
#expect(AXTextFormatter.truncation(maxElements: 2000, maxDepth: nil).contains("(maxElements=2000)"))
|
||||
|
||||
@@ -1,42 +1,8 @@
|
||||
// A same-app foreground process is insufficient to skip raising a different window.
|
||||
import Testing
|
||||
@testable import CuaNative
|
||||
|
||||
private func plan(window: UInt32? = 7, role: String? = "AXWindow", main: Bool? = true,
|
||||
minimized: Bool? = false) -> ForegroundPlan {
|
||||
ForegroundPlan(appHandle: "app", identity: ProcessIdentity(pid: 42, startedSeconds: 1, startedMicroseconds: 0),
|
||||
snapshotID: "snapshot", index: 0, targetWindowID: window, role: role, main: main, minimized: minimized)
|
||||
}
|
||||
|
||||
@Test func onlySkipsRaisingTheExactFocusedMainFrontWindow() {
|
||||
let current = ForegroundObservation(frontPID: 42, focusedWindowID: 7, frontWindowID: 7, frontWindowPID: 42)
|
||||
#expect(!AXWorker.shouldRaise(plan(), before: current))
|
||||
for requested in [plan(window: nil), plan(window: 8), plan(role: "AXSheet"), plan(main: nil),
|
||||
plan(main: false), plan(minimized: nil), plan(minimized: true)] {
|
||||
#expect(AXWorker.shouldRaise(requested, before: current))
|
||||
}
|
||||
for observed in [
|
||||
ForegroundObservation(frontPID: nil, focusedWindowID: 7, frontWindowID: 7, frontWindowPID: 42),
|
||||
ForegroundObservation(frontPID: 99, focusedWindowID: 7, frontWindowID: 7, frontWindowPID: 42),
|
||||
ForegroundObservation(frontPID: 42, focusedWindowID: nil, frontWindowID: 7, frontWindowPID: 42),
|
||||
ForegroundObservation(frontPID: 42, focusedWindowID: 8, frontWindowID: 7, frontWindowPID: 42),
|
||||
ForegroundObservation(frontPID: 42, focusedWindowID: 7, frontWindowID: nil, frontWindowPID: 42),
|
||||
ForegroundObservation(frontPID: 42, focusedWindowID: 7, frontWindowID: 8, frontWindowPID: 42),
|
||||
ForegroundObservation(frontPID: 42, focusedWindowID: 7, frontWindowID: 7, frontWindowPID: nil),
|
||||
ForegroundObservation(frontPID: 42, focusedWindowID: 7, frontWindowID: 7, frontWindowPID: 99),
|
||||
] {
|
||||
#expect(AXWorker.shouldRaise(plan(), before: observed))
|
||||
}
|
||||
}
|
||||
|
||||
@Test func preservesUnknownForegroundValuesAndDispatchOnlyResult() {
|
||||
let observed = ForegroundObservation(frontPID: nil, focusedWindowID: nil, frontWindowID: nil, frontWindowPID: nil)
|
||||
#expect(observed.json == .object(["frontPid": .null, "focusedWindowId": .null, "frontWindowId": .null, "frontWindowPid": .null]))
|
||||
#expect(AXWorker.actionResult == .object(["dispatched": .bool(true)]))
|
||||
}
|
||||
|
||||
@Test func normalizesOnlyTheObservedAXActionName() {
|
||||
#expect(AXWorker.actionKey("AXRaise") == AXWorker.actionKey("raise"))
|
||||
#expect(AXWorker.actionKey("AXZoomWindow") == AXWorker.actionKey("zoom window"))
|
||||
#expect(AXWorker.actionKey("AXRaise") != AXWorker.actionKey("AXPress"))
|
||||
}
|
||||
@@ -31,3 +31,8 @@ import Testing
|
||||
return false
|
||||
} == ["AXValue"])
|
||||
}
|
||||
|
||||
@Test func preservesDistinctNativeNamesWithTheSameDisplayLabel() throws {
|
||||
let names = ["Name:Move previous\nTarget:0x123\nSelector:(null)", "Name:Move previous\nTarget:0x456\nSelector:(null)"]
|
||||
#expect(try AXAccess.listActionNames(actions: names, role: "AXButton", writableAttributes: []) == names)
|
||||
}
|
||||
@@ -165,8 +165,7 @@ def run_sample(args, label, repeat, length, artifacts, summary, deadline):
|
||||
windows_deadline = min(deadline, time.monotonic() + args.event_timeout)
|
||||
while True:
|
||||
windows, _, _ = helper.call("listWindows", owner)
|
||||
visible = [window for window in windows if window["pid"] == fixture.process.pid and
|
||||
window["isOnScreen"] and window["bounds"]["width"] > 200 and window["bounds"]["height"] > 200]
|
||||
visible = [window for window in windows if window["isOnScreen"] and window["bounds"]["width"] > 200 and window["bounds"]["height"] > 200]
|
||||
if len(visible) == 1:
|
||||
window = visible[0]
|
||||
break
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
import { COMMON_GUIDANCE } from '@deepseek-ai/dsh-cua/guidance'
|
||||
|
||||
/** macOS operation levels, permissions, selectors, and native measurement units. */
|
||||
export const PLATFORM_GUIDANCE = 'macOS: accessibility tree operations are available at app or window level. Simulated keyboard input is available at app level. Simulated mouse input is available at window level. getApp also accepts bundle IDs and .app paths, and findApps searches /Applications, /System/Applications, and ~/Applications. Use the Computer Use permission page to grant the helper application Accessibility and Screen Recording permissions in System Settings. Native window and AX bounds use macOS points, while delivered screenshots carry their own pixel dimensions. show does not switch Spaces, and Stage Manager can present reduced window geometry; explain any required foreground step before activate or raise.'
|
||||
export const PLATFORM_GUIDANCE = 'macOS: accessibility tree operations are available at app or window level. Simulated keyboard input is available at app level. Simulated mouse input is available at window level. Use window coordinates for click and mouseMove on macOS. The helper rejects element-targeted pointer requests carrying the window ID supplied by these object methods. For an element’s default action, use invoke with its latest ID. getApp also accepts bundle IDs and .app paths, and findApps searches /Applications, /System/Applications, and ~/Applications. Use the Computer Use permission page to grant the helper application Accessibility and Screen Recording permissions in System Settings. Native window and AX bounds use macOS points, while delivered screenshots carry their own pixel dimensions. show does not switch Spaces, and Stage Manager can present reduced window geometry; explain any required foreground step before activate or raise.'
|
||||
|
||||
/** Complete model instructions for this provider. */
|
||||
export const GUIDANCE = `${PLATFORM_GUIDANCE}
|
||||
|
||||
@@ -14,7 +14,7 @@
|
||||
{"type":"session/title","data":{"title":"Inspect CUA capabilities, select the","messageSeqs":[8],"source":{"kind":"fallback"}}}
|
||||
{"type":"assistant/message","data":{"turn":1,"step":1,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-1","name":"cua","arguments":"{\"method\":\"getCapabilities\",\"params\":{}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:4}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947688973,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947688973,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-1","name":"cua","arguments":"{\"method\":\"getCapabilities\",\"params\":{}}"}}},{"type":"chunk","time":1791947688973,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947688973,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"}
|
||||
{"type":"tool/call","data":{"turn":1,"step":1,"callId":"cua-native-1","name":"cua","arguments":"{\"method\":\"getCapabilities\",\"params\":{}}"}}
|
||||
{"type":"tool/result","data":{"turn":1,"step":1,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-1"},"toolCallId":"cua-native-1","content":[{"type":"text","text":"{\"platform\":\"macos\",\"sessionLocked\":false,\"permissions\":{\"accessibility\":true,\"screenRecording\":true},\"methods\":[\"getCapabilities\",\"listApps\",\"getApp\",\"activate\",\"show\",\"listWindows\",\"getState\",\"hover\",\"click\",\"setValue\",\"typeText\",\"paste\",\"pressKey\",\"scroll\",\"drag\",\"screenshot\"],\"coordinateSpace\":\"screenshot-pixels\",\"browser\":false}"}],"isError":false,"id":"{{message:5}}"}},"sourceEventSeqs":[14],"surfaceOp":"append"}
|
||||
{"type":"tool/result","data":{"turn":1,"step":1,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-1"},"toolCallId":"cua-native-1","content":[{"type":"text","text":"{\"sessionLocked\":false,\"permissions\":{\"accessibility\":true,\"screenRecording\":true},\"platform\":\"macos\",\"methods\":[\"activate\",\"click\",\"drag\",\"findApps\",\"getApp\",\"getCapabilities\",\"getState\",\"invoke\",\"listActions\",\"listApps\",\"listWindows\",\"mouseMove\",\"paste\",\"performAction\",\"pressKey\",\"screenshot\",\"scroll\",\"selectText\",\"setValue\",\"show\",\"typeText\"],\"appLevelAccessibility\":true,\"keyboard\":[\"app\"],\"browser\":false,\"coordinateSpace\":\"screenshot-pixels\"}"}],"isError":false,"id":"{{message:5}}"}},"sourceEventSeqs":[14],"surfaceOp":"append"}
|
||||
{"type":"step/end","data":{"turn":1,"step":1}}
|
||||
{"type":"step/start","data":{"turn":1,"step":2}}
|
||||
{"type":"assistant/message","data":{"turn":1,"step":2,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-unauthorized-getState","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\"}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:6}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947689573,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947689573,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-unauthorized-getState","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\"}}"}}},{"type":"chunk","time":1791947689573,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947689573,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"}
|
||||
@@ -27,29 +27,29 @@
|
||||
{"type":"tool/result","data":{"turn":1,"step":3,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-unauthorized-screenshot"},"toolCallId":"cua-unauthorized-screenshot","content":[{"type":"text","text":"Error: APP_ACCESS_REQUIRED: Select the application with getApp and obtain user consent before reading or controlling it"}],"isError":true,"id":"{{message:9}}"}},"sourceEventSeqs":[24],"surfaceOp":"append"}
|
||||
{"type":"step/end","data":{"turn":1,"step":3}}
|
||||
{"type":"step/start","data":{"turn":1,"step":4}}
|
||||
{"type":"assistant/message","data":{"turn":1,"step":4,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-unauthorized-click","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"snapshotId\":\"fixture-snapshot-1\",\"elementIndex\":0}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:10}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947689615,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947689615,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-unauthorized-click","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"snapshotId\":\"fixture-snapshot-1\",\"elementIndex\":0}}"}}},{"type":"chunk","time":1791947689615,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947689615,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"}
|
||||
{"type":"tool/call","data":{"turn":1,"step":4,"callId":"cua-unauthorized-click","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"snapshotId\":\"fixture-snapshot-1\",\"elementIndex\":0}}"}}
|
||||
{"type":"assistant/message","data":{"turn":1,"step":4,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-unauthorized-click","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"snapshotId\":\"fixture-snapshot-1\",\"elementId\":\"0\"}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:10}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947689615,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947689615,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-unauthorized-click","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"snapshotId\":\"fixture-snapshot-1\",\"elementId\":\"0\"}}"}}},{"type":"chunk","time":1791947689615,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947689615,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"}
|
||||
{"type":"tool/call","data":{"turn":1,"step":4,"callId":"cua-unauthorized-click","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"snapshotId\":\"fixture-snapshot-1\",\"elementId\":\"0\"}}"}}
|
||||
{"type":"tool/result","data":{"turn":1,"step":4,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-unauthorized-click"},"toolCallId":"cua-unauthorized-click","content":[{"type":"text","text":"Error: APP_ACCESS_REQUIRED: Select the application with getApp and obtain user consent before reading or controlling it"}],"isError":true,"id":"{{message:11}}"}},"sourceEventSeqs":[29],"surfaceOp":"append"}
|
||||
{"type":"step/end","data":{"turn":1,"step":4}}
|
||||
{"type":"step/start","data":{"turn":1,"step":5}}
|
||||
{"type":"assistant/message","data":{"turn":1,"step":5,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-2","name":"cua","arguments":"{\"method\":\"getApp\",\"params\":{\"selector\":\"test.fixture\"}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:12}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947689634,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947689634,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-2","name":"cua","arguments":"{\"method\":\"getApp\",\"params\":{\"selector\":\"test.fixture\"}}"}}},{"type":"chunk","time":1791947689634,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947689634,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"}
|
||||
{"type":"tool/call","data":{"turn":1,"step":5,"callId":"cua-native-2","name":"cua","arguments":"{\"method\":\"getApp\",\"params\":{\"selector\":\"test.fixture\"}}"}}
|
||||
{"type":"tool/result","data":{"turn":1,"step":5,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-2"},"toolCallId":"cua-native-2","content":[{"type":"text","text":"{\"appHandle\":\"fixture-app\",\"pid\":9,\"name\":\"Fixture\",\"bundleId\":\"test.fixture\"}"}],"isError":false,"id":"{{message:13}}"}},"sourceEventSeqs":[34],"surfaceOp":"append"}
|
||||
{"type":"tool/result","data":{"turn":1,"step":5,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-2"},"toolCallId":"cua-native-2","content":[{"type":"text","text":"{\"appHandle\":\"fixture-app\",\"name\":\"Fixture\",\"bundleId\":\"test.fixture\",\"running\":true}"}],"isError":false,"id":"{{message:13}}"}},"sourceEventSeqs":[34],"surfaceOp":"append"}
|
||||
{"type":"step/end","data":{"turn":1,"step":5}}
|
||||
{"type":"step/start","data":{"turn":1,"step":6}}
|
||||
{"type":"assistant/message","data":{"turn":1,"step":6,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-3","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7,\"includeMenus\":true,\"details\":true}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:14}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947689657,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947689657,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-3","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7,\"includeMenus\":true,\"details\":true}}"}}},{"type":"chunk","time":1791947689657,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947689657,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"}
|
||||
{"type":"tool/call","data":{"turn":1,"step":6,"callId":"cua-native-3","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7,\"includeMenus\":true,\"details\":true}}"}}
|
||||
{"type":"tool/result","data":{"turn":1,"step":6,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-3"},"toolCallId":"cua-native-3","content":[{"type":"text","text":"{\"appHandle\":\"fixture-app\",\"pid\":9,\"windowId\":7,\"snapshotId\":\"fixture-snapshot-1\",\"tree\":\"AX scope=window offsets=UTF-16 bounds=global-points\\n- [0] AXButton \\\"Increment\\\" id=\\\"increment\\\" actions=[\\\"AXPress\\\"] bounds=[60,80,100,20]\\n- [1] AXTextField valueWritable=true selectionWritable=true value=\\\"\\\" placeholder=\\\"Fixture text\\\"\\n- AXStaticText \\\"Count: 0\\\"\\n- [2] AXMenuBarItem \\\"File\\\"\\n - [3] AXMenuItem \\\"Fixture command\\\"\",\"windows\":[{\"windowId\":7,\"pid\":9,\"appName\":\"Fixture\",\"title\":\"CUA fixture\",\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"isOnScreen\":true,\"onCurrentSpace\":true}],\"truncated\":false}"}],"isError":false,"id":"{{message:15}}"}},"sourceEventSeqs":[39],"surfaceOp":"append"}
|
||||
{"type":"tool/result","data":{"turn":1,"step":6,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-3"},"toolCallId":"cua-native-3","content":[{"type":"text","text":"{\"appHandle\":\"fixture-app\",\"windowId\":7,\"snapshotId\":\"fixture-snapshot-1\",\"tree\":\"AX scope=window offsets=UTF-16 bounds=global-points\\n- [0] AXButton \\\"Increment\\\" id=\\\"increment\\\" actions=[\\\"AXPress\\\"] bounds=[60,80,100,20]\\n- [1] AXTextField valueWritable=true selectionWritable=true value=\\\"\\\" placeholder=\\\"Fixture text\\\"\\n- AXStaticText \\\"Count: 0\\\"\\n- [2] AXMenuBarItem \\\"File\\\"\\n - [3] AXMenuItem \\\"Fixture command\\\"\",\"windows\":[{\"windowId\":7,\"appName\":\"Fixture\",\"title\":\"CUA fixture\",\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"isOnScreen\":true,\"onCurrentSpace\":true}],\"truncated\":false}"}],"isError":false,"id":"{{message:15}}"}},"sourceEventSeqs":[39],"surfaceOp":"append"}
|
||||
{"type":"step/end","data":{"turn":1,"step":6}}
|
||||
{"type":"step/start","data":{"turn":1,"step":7}}
|
||||
{"type":"assistant/message","data":{"turn":1,"step":7,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-4","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"snapshotId\":\"fixture-snapshot-1\",\"elementIndex\":0,\"holdMs\":800}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:16}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947689683,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947689683,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-4","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"snapshotId\":\"fixture-snapshot-1\",\"elementIndex\":0,\"holdMs\":800}}"}}},{"type":"chunk","time":1791947689683,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947689683,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"}
|
||||
{"type":"tool/call","data":{"turn":1,"step":7,"callId":"cua-native-4","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"snapshotId\":\"fixture-snapshot-1\",\"elementIndex\":0,\"holdMs\":800}}"}}
|
||||
{"type":"tool/result","data":{"turn":1,"step":7,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-4"},"toolCallId":"cua-native-4","content":[{"type":"text","text":"{\"dispatched\":true,\"verified\":false}"}],"isError":false,"id":"{{message:17}}"}},"sourceEventSeqs":[44],"surfaceOp":"append"}
|
||||
{"type":"assistant/message","data":{"turn":1,"step":7,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-4","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"snapshotId\":\"fixture-snapshot-1\",\"elementId\":\"0\",\"holdMs\":800}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:16}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947689683,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947689683,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-4","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"snapshotId\":\"fixture-snapshot-1\",\"elementId\":\"0\",\"holdMs\":800}}"}}},{"type":"chunk","time":1791947689683,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947689683,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"}
|
||||
{"type":"tool/call","data":{"turn":1,"step":7,"callId":"cua-native-4","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"snapshotId\":\"fixture-snapshot-1\",\"elementId\":\"0\",\"holdMs\":800}}"}}
|
||||
{"type":"tool/result","data":{"turn":1,"step":7,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-4"},"toolCallId":"cua-native-4","content":[{"type":"text","text":"{\"dispatched\":true}"}],"isError":false,"id":"{{message:17}}"}},"sourceEventSeqs":[44],"surfaceOp":"append"}
|
||||
{"type":"step/end","data":{"turn":1,"step":7}}
|
||||
{"type":"step/start","data":{"turn":1,"step":8}}
|
||||
{"type":"assistant/message","data":{"turn":1,"step":8,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-5","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:18}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947689706,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947689706,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-5","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}}},{"type":"chunk","time":1791947689706,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947689706,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"}
|
||||
{"type":"tool/call","data":{"turn":1,"step":8,"callId":"cua-native-5","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}}
|
||||
{"type":"tool/result","data":{"turn":1,"step":8,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-5"},"toolCallId":"cua-native-5","content":[{"type":"text","text":"{\"appHandle\":\"fixture-app\",\"pid\":9,\"windowId\":7,\"snapshotId\":\"fixture-snapshot-2\",\"tree\":\"AX scope=window offsets=UTF-16\\n- [0] AXButton \\\"Increment\\\"\\n- [1] AXTextField valueWritable=true selectionWritable=true value=\\\"\\\" placeholder=\\\"Fixture text\\\"\\n- AXStaticText \\\"Count: 1\\\"\\n- [2] AXMenuBarItem \\\"File\\\"\\nMenu contents omitted; request includeMenus=true to expand.\",\"windows\":[{\"windowId\":7,\"pid\":9,\"appName\":\"Fixture\",\"title\":\"CUA fixture\",\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"isOnScreen\":true,\"onCurrentSpace\":true}],\"truncated\":false}"}],"isError":false,"id":"{{message:19}}"}},"sourceEventSeqs":[49],"surfaceOp":"append"}
|
||||
{"type":"tool/result","data":{"turn":1,"step":8,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-5"},"toolCallId":"cua-native-5","content":[{"type":"text","text":"{\"appHandle\":\"fixture-app\",\"windowId\":7,\"snapshotId\":\"fixture-snapshot-2\",\"tree\":\"AX scope=window offsets=UTF-16\\n- [0] AXButton \\\"Increment\\\"\\n- [1] AXTextField valueWritable=true selectionWritable=true value=\\\"\\\" placeholder=\\\"Fixture text\\\"\\n- AXStaticText \\\"Count: 1\\\"\\n- [2] AXMenuBarItem \\\"File\\\"\\nMenu contents omitted; request includeMenus=true to expand.\",\"windows\":[{\"windowId\":7,\"appName\":\"Fixture\",\"title\":\"CUA fixture\",\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"isOnScreen\":true,\"onCurrentSpace\":true}],\"truncated\":false}"}],"isError":false,"id":"{{message:19}}"}},"sourceEventSeqs":[49],"surfaceOp":"append"}
|
||||
{"type":"step/end","data":{"turn":1,"step":8}}
|
||||
{"type":"step/start","data":{"turn":1,"step":9}}
|
||||
{"type":"assistant/message","data":{"turn":1,"step":9,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-6","name":"cua","arguments":"{\"method\":\"screenshot\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:20}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947689729,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947689729,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-6","name":"cua","arguments":"{\"method\":\"screenshot\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}}},{"type":"chunk","time":1791947689729,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947689729,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"}
|
||||
@@ -59,12 +59,12 @@
|
||||
{"type":"step/start","data":{"turn":1,"step":10}}
|
||||
{"type":"assistant/message","data":{"turn":1,"step":10,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-7","name":"cua","arguments":"{\"method\":\"listWindows\",\"params\":{\"appHandle\":\"fixture-app\"}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:22}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947689792,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947689792,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-7","name":"cua","arguments":"{\"method\":\"listWindows\",\"params\":{\"appHandle\":\"fixture-app\"}}"}}},{"type":"chunk","time":1791947689792,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947689792,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"}
|
||||
{"type":"tool/call","data":{"turn":1,"step":10,"callId":"cua-native-7","name":"cua","arguments":"{\"method\":\"listWindows\",\"params\":{\"appHandle\":\"fixture-app\"}}"}}
|
||||
{"type":"tool/result","data":{"turn":1,"step":10,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-7"},"toolCallId":"cua-native-7","content":[{"type":"text","text":"[{\"windowId\":7,\"pid\":9,\"appName\":\"Fixture\",\"title\":\"CUA fixture\",\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"isOnScreen\":true,\"onCurrentSpace\":true},{\"windowId\":8,\"pid\":9,\"appName\":\"Fixture\",\"title\":\"AXless canvas\",\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"isOnScreen\":true,\"onCurrentSpace\":true}]"}],"isError":false,"id":"{{message:23}}"}},"sourceEventSeqs":[59],"surfaceOp":"append"}
|
||||
{"type":"tool/result","data":{"turn":1,"step":10,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-7"},"toolCallId":"cua-native-7","content":[{"type":"text","text":"[{\"windowId\":7,\"appName\":\"Fixture\",\"title\":\"CUA fixture\",\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"isOnScreen\":true,\"onCurrentSpace\":true},{\"windowId\":8,\"appName\":\"Fixture\",\"title\":\"AXless canvas\",\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"isOnScreen\":true,\"onCurrentSpace\":true}]"}],"isError":false,"id":"{{message:23}}"}},"sourceEventSeqs":[59],"surfaceOp":"append"}
|
||||
{"type":"step/end","data":{"turn":1,"step":10}}
|
||||
{"type":"step/start","data":{"turn":1,"step":11}}
|
||||
{"type":"assistant/message","data":{"turn":1,"step":11,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-8","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":8}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:24}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947689834,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947689834,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-8","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":8}}"}}},{"type":"chunk","time":1791947689834,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947689834,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"}
|
||||
{"type":"tool/call","data":{"turn":1,"step":11,"callId":"cua-native-8","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":8}}"}}
|
||||
{"type":"tool/result","data":{"turn":1,"step":11,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-8"},"toolCallId":"cua-native-8","content":[{"type":"text","text":"{\"appHandle\":\"fixture-app\",\"pid\":9,\"windowId\":8,\"snapshotId\":\"fixture-snapshot-3\",\"tree\":\"\",\"windows\":[{\"windowId\":8,\"pid\":9,\"appName\":\"Fixture\",\"title\":\"AXless canvas\",\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"isOnScreen\":true,\"onCurrentSpace\":true}],\"truncated\":false}"}],"isError":false,"id":"{{message:25}}"}},"sourceEventSeqs":[64],"surfaceOp":"append"}
|
||||
{"type":"tool/result","data":{"turn":1,"step":11,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-8"},"toolCallId":"cua-native-8","content":[{"type":"text","text":"{\"appHandle\":\"fixture-app\",\"windowId\":8,\"snapshotId\":\"fixture-snapshot-3\",\"tree\":\"\",\"windows\":[{\"windowId\":8,\"appName\":\"Fixture\",\"title\":\"AXless canvas\",\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"isOnScreen\":true,\"onCurrentSpace\":true}],\"truncated\":false}"}],"isError":false,"id":"{{message:25}}"}},"sourceEventSeqs":[64],"surfaceOp":"append"}
|
||||
{"type":"step/end","data":{"turn":1,"step":11}}
|
||||
{"type":"step/start","data":{"turn":1,"step":12}}
|
||||
{"type":"assistant/message","data":{"turn":1,"step":12,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-9","name":"cua","arguments":"{\"method\":\"screenshot\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":8}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:26}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947689861,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947689861,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-9","name":"cua","arguments":"{\"method\":\"screenshot\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":8}}"}}},{"type":"chunk","time":1791947689861,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947689861,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"}
|
||||
@@ -74,67 +74,67 @@
|
||||
{"type":"step/start","data":{"turn":1,"step":13}}
|
||||
{"type":"assistant/message","data":{"turn":1,"step":13,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-10","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":8,\"snapshotId\":\"fixture-snapshot-3\",\"x\":0,\"y\":0}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:28}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947689896,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947689896,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-10","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":8,\"snapshotId\":\"fixture-snapshot-3\",\"x\":0,\"y\":0}}"}}},{"type":"chunk","time":1791947689896,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947689896,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"}
|
||||
{"type":"tool/call","data":{"turn":1,"step":13,"callId":"cua-native-10","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":8,\"snapshotId\":\"fixture-snapshot-3\",\"x\":0,\"y\":0}}"}}
|
||||
{"type":"tool/result","data":{"turn":1,"step":13,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-10"},"toolCallId":"cua-native-10","content":[{"type":"text","text":"{\"dispatched\":true,\"verified\":false}"}],"isError":false,"id":"{{message:29}}"}},"sourceEventSeqs":[74],"surfaceOp":"append"}
|
||||
{"type":"tool/result","data":{"turn":1,"step":13,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-10"},"toolCallId":"cua-native-10","content":[{"type":"text","text":"{\"dispatched\":true}"}],"isError":false,"id":"{{message:29}}"}},"sourceEventSeqs":[74],"surfaceOp":"append"}
|
||||
{"type":"step/end","data":{"turn":1,"step":13}}
|
||||
{"type":"step/start","data":{"turn":1,"step":14}}
|
||||
{"type":"assistant/message","data":{"turn":1,"step":14,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-11","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":8}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:30}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947689918,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947689918,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-11","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":8}}"}}},{"type":"chunk","time":1791947689918,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947689918,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"}
|
||||
{"type":"tool/call","data":{"turn":1,"step":14,"callId":"cua-native-11","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":8}}"}}
|
||||
{"type":"tool/result","data":{"turn":1,"step":14,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-11"},"toolCallId":"cua-native-11","content":[{"type":"text","text":"{\"appHandle\":\"fixture-app\",\"pid\":9,\"windowId\":8,\"snapshotId\":\"fixture-snapshot-4\",\"tree\":\"\",\"windows\":[{\"windowId\":8,\"pid\":9,\"appName\":\"Fixture\",\"title\":\"AXless canvas\",\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"isOnScreen\":true,\"onCurrentSpace\":true}],\"truncated\":false}"}],"isError":false,"id":"{{message:31}}"}},"sourceEventSeqs":[79],"surfaceOp":"append"}
|
||||
{"type":"tool/result","data":{"turn":1,"step":14,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-11"},"toolCallId":"cua-native-11","content":[{"type":"text","text":"{\"appHandle\":\"fixture-app\",\"windowId\":8,\"snapshotId\":\"fixture-snapshot-4\",\"tree\":\"\",\"windows\":[{\"windowId\":8,\"appName\":\"Fixture\",\"title\":\"AXless canvas\",\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"isOnScreen\":true,\"onCurrentSpace\":true}],\"truncated\":false}"}],"isError":false,"id":"{{message:31}}"}},"sourceEventSeqs":[79],"surfaceOp":"append"}
|
||||
{"type":"step/end","data":{"turn":1,"step":14}}
|
||||
{"type":"step/start","data":{"turn":1,"step":15}}
|
||||
{"type":"assistant/message","data":{"turn":1,"step":15,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-show","name":"cua","arguments":"{\"method\":\"show\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:32}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947689940,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947689940,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-show","name":"cua","arguments":"{\"method\":\"show\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}}},{"type":"chunk","time":1791947689940,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947689940,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"}
|
||||
{"type":"tool/call","data":{"turn":1,"step":15,"callId":"cua-native-show","name":"cua","arguments":"{\"method\":\"show\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}}
|
||||
{"type":"tool/result","data":{"turn":1,"step":15,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-show"},"toolCallId":"cua-native-show","content":[{"type":"text","text":"{\"dispatched\":true,\"verified\":false,\"windowId\":7,\"foregroundChanged\":false}"}],"isError":false,"id":"{{message:33}}"}},"sourceEventSeqs":[84],"surfaceOp":"append"}
|
||||
{"type":"tool/result","data":{"turn":1,"step":15,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-show"},"toolCallId":"cua-native-show","content":[{"type":"text","text":"{\"dispatched\":true,\"windowId\":7,\"foregroundChanged\":false}"}],"isError":false,"id":"{{message:33}}"}},"sourceEventSeqs":[84],"surfaceOp":"append"}
|
||||
{"type":"step/end","data":{"turn":1,"step":15}}
|
||||
{"type":"step/start","data":{"turn":1,"step":16}}
|
||||
{"type":"assistant/message","data":{"turn":1,"step":16,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-activate-first","name":"cua","arguments":"{\"method\":\"activate\",\"params\":{\"appHandle\":\"fixture-app\"}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:34}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947689961,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947689961,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-activate-first","name":"cua","arguments":"{\"method\":\"activate\",\"params\":{\"appHandle\":\"fixture-app\"}}"}}},{"type":"chunk","time":1791947689961,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947689961,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"}
|
||||
{"type":"tool/call","data":{"turn":1,"step":16,"callId":"cua-native-activate-first","name":"cua","arguments":"{\"method\":\"activate\",\"params\":{\"appHandle\":\"fixture-app\"}}"}}
|
||||
{"type":"tool/result","data":{"turn":1,"step":16,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-activate-first"},"toolCallId":"cua-native-activate-first","content":[{"type":"text","text":"{\"dispatched\":true,\"verified\":false,\"foreground\":{\"requestId\":\"cua-native-activate-first\",\"operation\":\"activate\",\"targetPid\":9,\"targetWindowId\":null,\"before\":{\"frontPid\":42,\"focusedWindowId\":7,\"frontWindowId\":99,\"frontWindowPid\":42},\"after\":{\"frontPid\":9,\"focusedWindowId\":7,\"frontWindowId\":7,\"frontWindowPid\":9},\"activationRequested\":true,\"raiseRequested\":false}}"}],"isError":false,"id":"{{message:35}}"}},"sourceEventSeqs":[89],"surfaceOp":"append"}
|
||||
{"type":"tool/result","data":{"turn":1,"step":16,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-activate-first"},"toolCallId":"cua-native-activate-first","content":[{"type":"text","text":"{\"dispatched\":true,\"foreground\":{\"requestId\":\"cua-native-activate-first\",\"operation\":\"activate\",\"targetPid\":9,\"targetWindowId\":null,\"before\":{\"frontPid\":42,\"focusedWindowId\":7,\"frontWindowId\":99,\"frontWindowPid\":42},\"after\":{\"frontPid\":9,\"focusedWindowId\":7,\"frontWindowId\":7,\"frontWindowPid\":9},\"activationRequested\":true,\"raiseRequested\":false}}"}],"isError":false,"id":"{{message:35}}"}},"sourceEventSeqs":[89],"surfaceOp":"append"}
|
||||
{"type":"step/end","data":{"turn":1,"step":16}}
|
||||
{"type":"step/start","data":{"turn":1,"step":17}}
|
||||
{"type":"assistant/message","data":{"turn":1,"step":17,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-activate-repeat","name":"cua","arguments":"{\"method\":\"activate\",\"params\":{\"appHandle\":\"fixture-app\"}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:36}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947689983,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947689983,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-activate-repeat","name":"cua","arguments":"{\"method\":\"activate\",\"params\":{\"appHandle\":\"fixture-app\"}}"}}},{"type":"chunk","time":1791947689983,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947689983,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"}
|
||||
{"type":"tool/call","data":{"turn":1,"step":17,"callId":"cua-native-activate-repeat","name":"cua","arguments":"{\"method\":\"activate\",\"params\":{\"appHandle\":\"fixture-app\"}}"}}
|
||||
{"type":"tool/result","data":{"turn":1,"step":17,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-activate-repeat"},"toolCallId":"cua-native-activate-repeat","content":[{"type":"text","text":"{\"dispatched\":true,\"verified\":false,\"foreground\":{\"requestId\":\"cua-native-activate-repeat\",\"operation\":\"activate\",\"targetPid\":9,\"targetWindowId\":null,\"before\":{\"frontPid\":9,\"focusedWindowId\":7,\"frontWindowId\":7,\"frontWindowPid\":9},\"after\":{\"frontPid\":9,\"focusedWindowId\":7,\"frontWindowId\":7,\"frontWindowPid\":9},\"activationRequested\":false,\"raiseRequested\":false}}"}],"isError":false,"id":"{{message:37}}"}},"sourceEventSeqs":[94],"surfaceOp":"append"}
|
||||
{"type":"tool/result","data":{"turn":1,"step":17,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-activate-repeat"},"toolCallId":"cua-native-activate-repeat","content":[{"type":"text","text":"{\"dispatched\":true,\"foreground\":{\"requestId\":\"cua-native-activate-repeat\",\"operation\":\"activate\",\"targetPid\":9,\"targetWindowId\":null,\"before\":{\"frontPid\":9,\"focusedWindowId\":7,\"frontWindowId\":7,\"frontWindowPid\":9},\"after\":{\"frontPid\":9,\"focusedWindowId\":7,\"frontWindowId\":7,\"frontWindowPid\":9},\"activationRequested\":false,\"raiseRequested\":false}}"}],"isError":false,"id":"{{message:37}}"}},"sourceEventSeqs":[94],"surfaceOp":"append"}
|
||||
{"type":"step/end","data":{"turn":1,"step":17}}
|
||||
{"type":"step/start","data":{"turn":1,"step":18}}
|
||||
{"type":"assistant/message","data":{"turn":1,"step":18,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-transition","name":"cua","arguments":"{\"method\":\"pressKey\",\"params\":{\"appHandle\":\"fixture-app\",\"key\":\"Return\"}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:38}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947690002,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947690002,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-transition","name":"cua","arguments":"{\"method\":\"pressKey\",\"params\":{\"appHandle\":\"fixture-app\",\"key\":\"Return\"}}"}}},{"type":"chunk","time":1791947690002,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947690002,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"}
|
||||
{"type":"tool/call","data":{"turn":1,"step":18,"callId":"cua-native-transition","name":"cua","arguments":"{\"method\":\"pressKey\",\"params\":{\"appHandle\":\"fixture-app\",\"key\":\"Return\"}}"}}
|
||||
{"type":"tool/result","data":{"turn":1,"step":18,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-transition"},"toolCallId":"cua-native-transition","content":[{"type":"text","text":"{\"dispatched\":true,\"verified\":false}"}],"isError":false,"id":"{{message:39}}"}},"sourceEventSeqs":[99],"surfaceOp":"append"}
|
||||
{"type":"tool/result","data":{"turn":1,"step":18,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-transition"},"toolCallId":"cua-native-transition","content":[{"type":"text","text":"{\"dispatched\":true}"}],"isError":false,"id":"{{message:39}}"}},"sourceEventSeqs":[99],"surfaceOp":"append"}
|
||||
{"type":"step/end","data":{"turn":1,"step":18}}
|
||||
{"type":"step/start","data":{"turn":1,"step":19}}
|
||||
{"type":"assistant/message","data":{"turn":1,"step":19,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-stale-read","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:40}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947690019,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947690019,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-stale-read","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}}},{"type":"chunk","time":1791947690019,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947690019,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"}
|
||||
{"type":"tool/call","data":{"turn":1,"step":19,"callId":"cua-native-stale-read","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}}
|
||||
{"type":"tool/result","data":{"turn":1,"step":19,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-stale-read"},"toolCallId":"cua-native-stale-read","content":[{"type":"text","text":"Error: STALE_SNAPSHOT: Accessibility tree changed during observation; read fresh getAXState"}],"isError":true,"id":"{{message:41}}"}},"sourceEventSeqs":[104],"surfaceOp":"append"}
|
||||
{"type":"tool/result","data":{"turn":1,"step":19,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-stale-read"},"toolCallId":"cua-native-stale-read","content":[{"type":"text","text":"Error: STALE_SNAPSHOT: Accessibility tree changed during observation; read fresh getAccessibilityTree"}],"isError":true,"id":"{{message:41}}"}},"sourceEventSeqs":[104],"surfaceOp":"append"}
|
||||
{"type":"step/end","data":{"turn":1,"step":19}}
|
||||
{"type":"step/start","data":{"turn":1,"step":20}}
|
||||
{"type":"assistant/message","data":{"turn":1,"step":20,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-stale-index","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"snapshotId\":\"fixture-snapshot-2\",\"elementIndex\":0}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:42}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947690040,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947690040,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-stale-index","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"snapshotId\":\"fixture-snapshot-2\",\"elementIndex\":0}}"}}},{"type":"chunk","time":1791947690040,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947690040,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"}
|
||||
{"type":"tool/call","data":{"turn":1,"step":20,"callId":"cua-native-stale-index","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"snapshotId\":\"fixture-snapshot-2\",\"elementIndex\":0}}"}}
|
||||
{"type":"assistant/message","data":{"turn":1,"step":20,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-stale-index","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"snapshotId\":\"fixture-snapshot-2\",\"elementId\":\"0\"}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:42}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947690040,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947690040,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-stale-index","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"snapshotId\":\"fixture-snapshot-2\",\"elementId\":\"0\"}}"}}},{"type":"chunk","time":1791947690040,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947690040,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"}
|
||||
{"type":"tool/call","data":{"turn":1,"step":20,"callId":"cua-native-stale-index","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"snapshotId\":\"fixture-snapshot-2\",\"elementId\":\"0\"}}"}}
|
||||
{"type":"tool/result","data":{"turn":1,"step":20,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-stale-index"},"toolCallId":"cua-native-stale-index","content":[{"type":"text","text":"Error: STALE_SNAPSHOT: Read fresh state before acting"}],"isError":true,"id":"{{message:43}}"}},"sourceEventSeqs":[109],"surfaceOp":"append"}
|
||||
{"type":"step/end","data":{"turn":1,"step":20}}
|
||||
{"type":"step/start","data":{"turn":1,"step":21}}
|
||||
{"type":"assistant/message","data":{"turn":1,"step":21,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-other-window","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":8,\"snapshotId\":\"fixture-snapshot-4\",\"x\":0,\"y\":0,\"holdMs\":800}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:44}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947690064,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947690064,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-other-window","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":8,\"snapshotId\":\"fixture-snapshot-4\",\"x\":0,\"y\":0,\"holdMs\":800}}"}}},{"type":"chunk","time":1791947690064,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947690064,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"}
|
||||
{"type":"tool/call","data":{"turn":1,"step":21,"callId":"cua-native-other-window","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":8,\"snapshotId\":\"fixture-snapshot-4\",\"x\":0,\"y\":0,\"holdMs\":800}}"}}
|
||||
{"type":"tool/result","data":{"turn":1,"step":21,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-other-window"},"toolCallId":"cua-native-other-window","content":[{"type":"text","text":"{\"dispatched\":true,\"verified\":false}"}],"isError":false,"id":"{{message:45}}"}},"sourceEventSeqs":[114],"surfaceOp":"append"}
|
||||
{"type":"tool/result","data":{"turn":1,"step":21,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-other-window"},"toolCallId":"cua-native-other-window","content":[{"type":"text","text":"{\"dispatched\":true}"}],"isError":false,"id":"{{message:45}}"}},"sourceEventSeqs":[114],"surfaceOp":"append"}
|
||||
{"type":"step/end","data":{"turn":1,"step":21}}
|
||||
{"type":"step/start","data":{"turn":1,"step":22}}
|
||||
{"type":"assistant/message","data":{"turn":1,"step":22,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-fresh-read","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:46}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947690087,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947690087,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-fresh-read","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}}},{"type":"chunk","time":1791947690087,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947690087,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"}
|
||||
{"type":"tool/call","data":{"turn":1,"step":22,"callId":"cua-native-fresh-read","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}}
|
||||
{"type":"tool/result","data":{"turn":1,"step":22,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-fresh-read"},"toolCallId":"cua-native-fresh-read","content":[{"type":"text","text":"{\"appHandle\":\"fixture-app\",\"pid\":9,\"windowId\":7,\"snapshotId\":\"fixture-snapshot-5\",\"tree\":\"AX scope=window offsets=UTF-16\\n- [0] AXButton \\\"Increment\\\"\\n- [1] AXTextField valueWritable=true selectionWritable=true value=\\\"\\\" placeholder=\\\"Fixture text\\\"\\n- AXStaticText \\\"Count: 1\\\"\\n- [2] AXMenuBarItem \\\"File\\\"\\nMenu contents omitted; request includeMenus=true to expand.\",\"windows\":[{\"windowId\":7,\"pid\":9,\"appName\":\"Fixture\",\"title\":\"CUA fixture\",\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"isOnScreen\":true,\"onCurrentSpace\":true}],\"truncated\":false}"}],"isError":false,"id":"{{message:47}}"}},"sourceEventSeqs":[119],"surfaceOp":"append"}
|
||||
{"type":"tool/result","data":{"turn":1,"step":22,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-fresh-read"},"toolCallId":"cua-native-fresh-read","content":[{"type":"text","text":"{\"appHandle\":\"fixture-app\",\"windowId\":7,\"snapshotId\":\"fixture-snapshot-5\",\"tree\":\"AX scope=window offsets=UTF-16\\n- [0] AXButton \\\"Increment\\\"\\n- [1] AXTextField valueWritable=true selectionWritable=true value=\\\"\\\" placeholder=\\\"Fixture text\\\"\\n- AXStaticText \\\"Count: 1\\\"\\n- [2] AXMenuBarItem \\\"File\\\"\\nMenu contents omitted; request includeMenus=true to expand.\",\"windows\":[{\"windowId\":7,\"appName\":\"Fixture\",\"title\":\"CUA fixture\",\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"isOnScreen\":true,\"onCurrentSpace\":true}],\"truncated\":false}"}],"isError":false,"id":"{{message:47}}"}},"sourceEventSeqs":[119],"surfaceOp":"append"}
|
||||
{"type":"step/end","data":{"turn":1,"step":22}}
|
||||
{"type":"step/start","data":{"turn":1,"step":23}}
|
||||
{"type":"assistant/message","data":{"turn":1,"step":23,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-fresh-click","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"snapshotId\":\"fixture-snapshot-5\",\"elementIndex\":0}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:48}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947690111,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947690111,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-fresh-click","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"snapshotId\":\"fixture-snapshot-5\",\"elementIndex\":0}}"}}},{"type":"chunk","time":1791947690111,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947690111,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"}
|
||||
{"type":"tool/call","data":{"turn":1,"step":23,"callId":"cua-native-fresh-click","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"snapshotId\":\"fixture-snapshot-5\",\"elementIndex\":0}}"}}
|
||||
{"type":"tool/result","data":{"turn":1,"step":23,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-fresh-click"},"toolCallId":"cua-native-fresh-click","content":[{"type":"text","text":"{\"dispatched\":true,\"verified\":false}"}],"isError":false,"id":"{{message:49}}"}},"sourceEventSeqs":[124],"surfaceOp":"append"}
|
||||
{"type":"assistant/message","data":{"turn":1,"step":23,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-fresh-click","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"snapshotId\":\"fixture-snapshot-5\",\"elementId\":\"0\"}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:48}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947690111,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947690111,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-fresh-click","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"snapshotId\":\"fixture-snapshot-5\",\"elementId\":\"0\"}}"}}},{"type":"chunk","time":1791947690111,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947690111,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"}
|
||||
{"type":"tool/call","data":{"turn":1,"step":23,"callId":"cua-native-fresh-click","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"snapshotId\":\"fixture-snapshot-5\",\"elementId\":\"0\"}}"}}
|
||||
{"type":"tool/result","data":{"turn":1,"step":23,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-fresh-click"},"toolCallId":"cua-native-fresh-click","content":[{"type":"text","text":"{\"dispatched\":true}"}],"isError":false,"id":"{{message:49}}"}},"sourceEventSeqs":[124],"surfaceOp":"append"}
|
||||
{"type":"step/end","data":{"turn":1,"step":23}}
|
||||
{"type":"step/start","data":{"turn":1,"step":24}}
|
||||
{"type":"assistant/message","data":{"turn":1,"step":24,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-verify","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:50}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947690141,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947690141,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-verify","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}}},{"type":"chunk","time":1791947690141,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947690141,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"}
|
||||
{"type":"tool/call","data":{"turn":1,"step":24,"callId":"cua-native-verify","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}}
|
||||
{"type":"tool/result","data":{"turn":1,"step":24,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-verify"},"toolCallId":"cua-native-verify","content":[{"type":"text","text":"{\"appHandle\":\"fixture-app\",\"pid\":9,\"windowId\":7,\"snapshotId\":\"fixture-snapshot-6\",\"tree\":\"AX scope=window offsets=UTF-16\\n- [0] AXButton \\\"Increment\\\"\\n- [1] AXTextField valueWritable=true selectionWritable=true value=\\\"\\\" placeholder=\\\"Fixture text\\\"\\n- AXStaticText \\\"Count: 2\\\"\\n- [2] AXMenuBarItem \\\"File\\\"\\nMenu contents omitted; request includeMenus=true to expand.\",\"windows\":[{\"windowId\":7,\"pid\":9,\"appName\":\"Fixture\",\"title\":\"CUA fixture\",\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"isOnScreen\":true,\"onCurrentSpace\":true}],\"truncated\":false}"}],"isError":false,"id":"{{message:51}}"}},"sourceEventSeqs":[129],"surfaceOp":"append"}
|
||||
{"type":"tool/result","data":{"turn":1,"step":24,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-verify"},"toolCallId":"cua-native-verify","content":[{"type":"text","text":"{\"appHandle\":\"fixture-app\",\"windowId\":7,\"snapshotId\":\"fixture-snapshot-6\",\"tree\":\"AX scope=window offsets=UTF-16\\n- [0] AXButton \\\"Increment\\\"\\n- [1] AXTextField valueWritable=true selectionWritable=true value=\\\"\\\" placeholder=\\\"Fixture text\\\"\\n- AXStaticText \\\"Count: 2\\\"\\n- [2] AXMenuBarItem \\\"File\\\"\\nMenu contents omitted; request includeMenus=true to expand.\",\"windows\":[{\"windowId\":7,\"appName\":\"Fixture\",\"title\":\"CUA fixture\",\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"isOnScreen\":true,\"onCurrentSpace\":true}],\"truncated\":false}"}],"isError":false,"id":"{{message:51}}"}},"sourceEventSeqs":[129],"surfaceOp":"append"}
|
||||
{"type":"step/end","data":{"turn":1,"step":24}}
|
||||
{"type":"step/start","data":{"turn":1,"step":25}}
|
||||
{"type":"assistant/message","data":{"turn":1,"step":25,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-geometry-observe","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":9}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:52}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947690162,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947690162,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-geometry-observe","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":9}}"}}},{"type":"chunk","time":1791947690162,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947690162,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"}
|
||||
{"type":"tool/call","data":{"turn":1,"step":25,"callId":"cua-native-geometry-observe","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":9}}"}}
|
||||
{"type":"tool/result","data":{"turn":1,"step":25,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-geometry-observe"},"toolCallId":"cua-native-geometry-observe","content":[{"type":"text","text":"{\"appHandle\":\"fixture-app\",\"pid\":9,\"windowId\":9,\"snapshotId\":\"fixture-snapshot-7\",\"tree\":\"AX scope=window offsets=UTF-16\\n- [0] AXButton \\\"Increment\\\"\\n- [1] AXTextField valueWritable=true selectionWritable=true value=\\\"\\\" placeholder=\\\"Fixture text\\\"\\n- AXStaticText \\\"Count: 2\\\"\\n- [2] AXMenuBarItem \\\"File\\\"\\nMenu contents omitted; request includeMenus=true to expand.\",\"windows\":[{\"windowId\":9,\"pid\":9,\"appName\":\"Fixture\",\"title\":\"Transformed window\",\"bounds\":{\"x\":16,\"y\":512,\"width\":131,\"height\":145},\"isOnScreen\":true,\"onCurrentSpace\":true}],\"truncated\":false}"}],"isError":false,"id":"{{message:53}}"}},"sourceEventSeqs":[134],"surfaceOp":"append"}
|
||||
{"type":"tool/result","data":{"turn":1,"step":25,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-geometry-observe"},"toolCallId":"cua-native-geometry-observe","content":[{"type":"text","text":"{\"appHandle\":\"fixture-app\",\"windowId\":9,\"snapshotId\":\"fixture-snapshot-7\",\"tree\":\"AX scope=window offsets=UTF-16\\n- [0] AXButton \\\"Increment\\\"\\n- [1] AXTextField valueWritable=true selectionWritable=true value=\\\"\\\" placeholder=\\\"Fixture text\\\"\\n- AXStaticText \\\"Count: 2\\\"\\n- [2] AXMenuBarItem \\\"File\\\"\\nMenu contents omitted; request includeMenus=true to expand.\",\"windows\":[{\"windowId\":9,\"appName\":\"Fixture\",\"title\":\"Transformed window\",\"bounds\":{\"x\":16,\"y\":512,\"width\":131,\"height\":145},\"isOnScreen\":true,\"onCurrentSpace\":true}],\"truncated\":false}"}],"isError":false,"id":"{{message:53}}"}},"sourceEventSeqs":[134],"surfaceOp":"append"}
|
||||
{"type":"step/end","data":{"turn":1,"step":25}}
|
||||
{"type":"step/start","data":{"turn":1,"step":26}}
|
||||
{"type":"assistant/message","data":{"turn":1,"step":26,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-geometry-capture","name":"cua","arguments":"{\"method\":\"screenshot\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":9}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:54}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947690186,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947690186,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-geometry-capture","name":"cua","arguments":"{\"method\":\"screenshot\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":9}}"}}},{"type":"chunk","time":1791947690186,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947690186,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"}
|
||||
@@ -157,19 +157,19 @@
|
||||
{"type":"tool/result","data":{"turn":1,"step":29,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-geometry-drag"},"toolCallId":"cua-native-geometry-drag","content":[{"type":"text","text":"Error: WINDOW_CHANGED: Window display and accessibility bounds disagree. Read fresh state; if unchanged, use AX actions or explicitly restore the window before capture or coordinate input"}],"isError":true,"id":"{{message:61}}"}},"sourceEventSeqs":[154],"surfaceOp":"append"}
|
||||
{"type":"step/end","data":{"turn":1,"step":29}}
|
||||
{"type":"step/start","data":{"turn":1,"step":30}}
|
||||
{"type":"assistant/message","data":{"turn":1,"step":30,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-geometry-ax","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"snapshotId\":\"fixture-snapshot-7\",\"elementIndex\":0}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:62}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947690294,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947690294,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-geometry-ax","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"snapshotId\":\"fixture-snapshot-7\",\"elementIndex\":0}}"}}},{"type":"chunk","time":1791947690294,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947690294,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"}
|
||||
{"type":"tool/call","data":{"turn":1,"step":30,"callId":"cua-native-geometry-ax","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"snapshotId\":\"fixture-snapshot-7\",\"elementIndex\":0}}"}}
|
||||
{"type":"tool/result","data":{"turn":1,"step":30,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-geometry-ax"},"toolCallId":"cua-native-geometry-ax","content":[{"type":"text","text":"{\"dispatched\":true,\"verified\":false}"}],"isError":false,"id":"{{message:63}}"}},"sourceEventSeqs":[159],"surfaceOp":"append"}
|
||||
{"type":"assistant/message","data":{"turn":1,"step":30,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-geometry-ax","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"snapshotId\":\"fixture-snapshot-7\",\"elementId\":\"0\"}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:62}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947690294,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947690294,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-geometry-ax","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"snapshotId\":\"fixture-snapshot-7\",\"elementId\":\"0\"}}"}}},{"type":"chunk","time":1791947690294,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947690294,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"}
|
||||
{"type":"tool/call","data":{"turn":1,"step":30,"callId":"cua-native-geometry-ax","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"snapshotId\":\"fixture-snapshot-7\",\"elementId\":\"0\"}}"}}
|
||||
{"type":"tool/result","data":{"turn":1,"step":30,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-geometry-ax"},"toolCallId":"cua-native-geometry-ax","content":[{"type":"text","text":"{\"dispatched\":true}"}],"isError":false,"id":"{{message:63}}"}},"sourceEventSeqs":[159],"surfaceOp":"append"}
|
||||
{"type":"step/end","data":{"turn":1,"step":30}}
|
||||
{"type":"step/start","data":{"turn":1,"step":31}}
|
||||
{"type":"assistant/message","data":{"turn":1,"step":31,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-native-geometry-verify","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":9}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:64}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947690314,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947690314,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-native-geometry-verify","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":9}}"}}},{"type":"chunk","time":1791947690314,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947690314,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"}
|
||||
{"type":"tool/call","data":{"turn":1,"step":31,"callId":"cua-native-geometry-verify","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":9}}"}}
|
||||
{"type":"tool/result","data":{"turn":1,"step":31,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-geometry-verify"},"toolCallId":"cua-native-geometry-verify","content":[{"type":"text","text":"{\"appHandle\":\"fixture-app\",\"pid\":9,\"windowId\":9,\"snapshotId\":\"fixture-snapshot-8\",\"tree\":\"AX scope=window offsets=UTF-16\\n- [0] AXButton \\\"Increment\\\"\\n- [1] AXTextField valueWritable=true selectionWritable=true value=\\\"\\\" placeholder=\\\"Fixture text\\\"\\n- AXStaticText \\\"Count: 3\\\"\\n- [2] AXMenuBarItem \\\"File\\\"\\nMenu contents omitted; request includeMenus=true to expand.\",\"windows\":[{\"windowId\":9,\"pid\":9,\"appName\":\"Fixture\",\"title\":\"Transformed window\",\"bounds\":{\"x\":16,\"y\":512,\"width\":131,\"height\":145},\"isOnScreen\":true,\"onCurrentSpace\":true}],\"truncated\":false}"}],"isError":false,"id":"{{message:65}}"}},"sourceEventSeqs":[164],"surfaceOp":"append"}
|
||||
{"type":"tool/result","data":{"turn":1,"step":31,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-native-geometry-verify"},"toolCallId":"cua-native-geometry-verify","content":[{"type":"text","text":"{\"appHandle\":\"fixture-app\",\"windowId\":9,\"snapshotId\":\"fixture-snapshot-8\",\"tree\":\"AX scope=window offsets=UTF-16\\n- [0] AXButton \\\"Increment\\\"\\n- [1] AXTextField valueWritable=true selectionWritable=true value=\\\"\\\" placeholder=\\\"Fixture text\\\"\\n- AXStaticText \\\"Count: 3\\\"\\n- [2] AXMenuBarItem \\\"File\\\"\\nMenu contents omitted; request includeMenus=true to expand.\",\"windows\":[{\"windowId\":9,\"appName\":\"Fixture\",\"title\":\"Transformed window\",\"bounds\":{\"x\":16,\"y\":512,\"width\":131,\"height\":145},\"isOnScreen\":true,\"onCurrentSpace\":true}],\"truncated\":false}"}],"isError":false,"id":"{{message:65}}"}},"sourceEventSeqs":[164],"surfaceOp":"append"}
|
||||
{"type":"step/end","data":{"turn":1,"step":31}}
|
||||
{"type":"step/start","data":{"turn":1,"step":32}}
|
||||
{"type":"assistant/message","data":{"turn":1,"step":32,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-capture-busy-arm","name":"cua","arguments":"{\"method\":\"pressKey\",\"params\":{\"appHandle\":\"fixture-app\",\"key\":\"F11\"}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:66}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947690340,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947690340,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-capture-busy-arm","name":"cua","arguments":"{\"method\":\"pressKey\",\"params\":{\"appHandle\":\"fixture-app\",\"key\":\"F11\"}}"}}},{"type":"chunk","time":1791947690340,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947690340,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"}
|
||||
{"type":"tool/call","data":{"turn":1,"step":32,"callId":"cua-capture-busy-arm","name":"cua","arguments":"{\"method\":\"pressKey\",\"params\":{\"appHandle\":\"fixture-app\",\"key\":\"F11\"}}"}}
|
||||
{"type":"tool/result","data":{"turn":1,"step":32,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-capture-busy-arm"},"toolCallId":"cua-capture-busy-arm","content":[{"type":"text","text":"{\"dispatched\":true,\"verified\":false}"}],"isError":false,"id":"{{message:67}}"}},"sourceEventSeqs":[169],"surfaceOp":"append"}
|
||||
{"type":"tool/result","data":{"turn":1,"step":32,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-capture-busy-arm"},"toolCallId":"cua-capture-busy-arm","content":[{"type":"text","text":"{\"dispatched\":true}"}],"isError":false,"id":"{{message:67}}"}},"sourceEventSeqs":[169],"surfaceOp":"append"}
|
||||
{"type":"step/end","data":{"turn":1,"step":32}}
|
||||
{"type":"step/start","data":{"turn":1,"step":33}}
|
||||
{"type":"assistant/message","data":{"turn":1,"step":33,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-capture-busy","name":"cua","arguments":"{\"method\":\"screenshot\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:68}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947690365,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947690365,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-capture-busy","name":"cua","arguments":"{\"method\":\"screenshot\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}}},{"type":"chunk","time":1791947690365,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947690365,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"}
|
||||
@@ -179,12 +179,12 @@
|
||||
{"type":"step/start","data":{"turn":1,"step":34}}
|
||||
{"type":"assistant/message","data":{"turn":1,"step":34,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-capture-busy-ax","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:70}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947690394,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947690394,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-capture-busy-ax","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}}},{"type":"chunk","time":1791947690394,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947690394,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"}
|
||||
{"type":"tool/call","data":{"turn":1,"step":34,"callId":"cua-capture-busy-ax","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}}
|
||||
{"type":"tool/result","data":{"turn":1,"step":34,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-capture-busy-ax"},"toolCallId":"cua-capture-busy-ax","content":[{"type":"text","text":"{\"appHandle\":\"fixture-app\",\"pid\":9,\"windowId\":7,\"snapshotId\":\"fixture-snapshot-9\",\"tree\":\"AX scope=window offsets=UTF-16\\n- [0] AXButton \\\"Increment\\\"\\n- [1] AXTextField valueWritable=true selectionWritable=true value=\\\"\\\" placeholder=\\\"Fixture text\\\"\\n- AXStaticText \\\"Count: 3\\\"\\n- [2] AXMenuBarItem \\\"File\\\"\\nMenu contents omitted; request includeMenus=true to expand.\",\"windows\":[{\"windowId\":7,\"pid\":9,\"appName\":\"Fixture\",\"title\":\"CUA fixture\",\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"isOnScreen\":true,\"onCurrentSpace\":true}],\"truncated\":false}"}],"isError":false,"id":"{{message:71}}"}},"sourceEventSeqs":[179],"surfaceOp":"append"}
|
||||
{"type":"tool/result","data":{"turn":1,"step":34,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-capture-busy-ax"},"toolCallId":"cua-capture-busy-ax","content":[{"type":"text","text":"{\"appHandle\":\"fixture-app\",\"windowId\":7,\"snapshotId\":\"fixture-snapshot-9\",\"tree\":\"AX scope=window offsets=UTF-16\\n- [0] AXButton \\\"Increment\\\"\\n- [1] AXTextField valueWritable=true selectionWritable=true value=\\\"\\\" placeholder=\\\"Fixture text\\\"\\n- AXStaticText \\\"Count: 3\\\"\\n- [2] AXMenuBarItem \\\"File\\\"\\nMenu contents omitted; request includeMenus=true to expand.\",\"windows\":[{\"windowId\":7,\"appName\":\"Fixture\",\"title\":\"CUA fixture\",\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"isOnScreen\":true,\"onCurrentSpace\":true}],\"truncated\":false}"}],"isError":false,"id":"{{message:71}}"}},"sourceEventSeqs":[179],"surfaceOp":"append"}
|
||||
{"type":"step/end","data":{"turn":1,"step":34}}
|
||||
{"type":"step/start","data":{"turn":1,"step":35}}
|
||||
{"type":"assistant/message","data":{"turn":1,"step":35,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-capture-callback","name":"cua","arguments":"{\"method\":\"pressKey\",\"params\":{\"appHandle\":\"fixture-app\",\"key\":\"F10\"}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:72}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947690414,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947690414,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-capture-callback","name":"cua","arguments":"{\"method\":\"pressKey\",\"params\":{\"appHandle\":\"fixture-app\",\"key\":\"F10\"}}"}}},{"type":"chunk","time":1791947690414,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947690414,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"}
|
||||
{"type":"tool/call","data":{"turn":1,"step":35,"callId":"cua-capture-callback","name":"cua","arguments":"{\"method\":\"pressKey\",\"params\":{\"appHandle\":\"fixture-app\",\"key\":\"F10\"}}"}}
|
||||
{"type":"tool/result","data":{"turn":1,"step":35,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-capture-callback"},"toolCallId":"cua-capture-callback","content":[{"type":"text","text":"{\"dispatched\":true,\"verified\":false}"}],"isError":false,"id":"{{message:73}}"}},"sourceEventSeqs":[184],"surfaceOp":"append"}
|
||||
{"type":"tool/result","data":{"turn":1,"step":35,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-capture-callback"},"toolCallId":"cua-capture-callback","content":[{"type":"text","text":"{\"dispatched\":true}"}],"isError":false,"id":"{{message:73}}"}},"sourceEventSeqs":[184],"surfaceOp":"append"}
|
||||
{"type":"step/end","data":{"turn":1,"step":35}}
|
||||
{"type":"step/start","data":{"turn":1,"step":36}}
|
||||
{"type":"assistant/message","data":{"turn":1,"step":36,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-capture-recovered","name":"cua","arguments":"{\"method\":\"screenshot\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:74}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947690434,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947690434,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-capture-recovered","name":"cua","arguments":"{\"method\":\"screenshot\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}}},{"type":"chunk","time":1791947690434,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947690434,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"}
|
||||
@@ -192,14 +192,14 @@
|
||||
{"type":"tool/result","data":{"turn":1,"step":36,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-capture-recovered"},"toolCallId":"cua-capture-recovered","content":[{"type":"text","text":"{\"pixelWidth\":1,\"pixelHeight\":1,\"windowId\":7,\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"coordinateSpace\":\"screenshot-pixels\"}"},{"type":"image","attachment":{"attachmentId":"sha256:999f1d1527ee7e79266f16add5430fff76b1225d742464a5b1ff1f02971bb8ee","mediaType":"image/png","width":1,"height":1,"bytes":90}}],"isError":false,"id":"{{message:75}}"}},"sourceEventSeqs":[189],"surfaceOp":"append"}
|
||||
{"type":"step/end","data":{"turn":1,"step":36}}
|
||||
{"type":"step/start","data":{"turn":1,"step":37}}
|
||||
{"type":"assistant/message","data":{"turn":1,"step":37,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-hover-enter","name":"cua","arguments":"{\"method\":\"hover\",\"params\":{\"appHandle\":\"fixture-app\",\"snapshotId\":\"fixture-snapshot-9\",\"elementIndex\":0}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:76}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947690466,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947690466,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-hover-enter","name":"cua","arguments":"{\"method\":\"hover\",\"params\":{\"appHandle\":\"fixture-app\",\"snapshotId\":\"fixture-snapshot-9\",\"elementIndex\":0}}"}}},{"type":"chunk","time":1791947690466,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947690466,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"}
|
||||
{"type":"tool/call","data":{"turn":1,"step":37,"callId":"cua-hover-enter","name":"cua","arguments":"{\"method\":\"hover\",\"params\":{\"appHandle\":\"fixture-app\",\"snapshotId\":\"fixture-snapshot-9\",\"elementIndex\":0}}"}}
|
||||
{"type":"tool/result","data":{"turn":1,"step":37,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-hover-enter"},"toolCallId":"cua-hover-enter","content":[{"type":"text","text":"{\"dispatched\":true,\"verified\":false,\"hoverRoute\":\"accessibility\",\"systemPointerMoved\":false}"}],"isError":false,"id":"{{message:77}}"}},"sourceEventSeqs":[194],"surfaceOp":"append"}
|
||||
{"type":"assistant/message","data":{"turn":1,"step":37,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-hover-enter","name":"cua","arguments":"{\"method\":\"mouseMove\",\"params\":{\"appHandle\":\"fixture-app\",\"snapshotId\":\"fixture-snapshot-9\",\"elementId\":\"0\"}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:76}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947690466,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947690466,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-hover-enter","name":"cua","arguments":"{\"method\":\"mouseMove\",\"params\":{\"appHandle\":\"fixture-app\",\"snapshotId\":\"fixture-snapshot-9\",\"elementId\":\"0\"}}"}}},{"type":"chunk","time":1791947690466,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947690466,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"}
|
||||
{"type":"tool/call","data":{"turn":1,"step":37,"callId":"cua-hover-enter","name":"cua","arguments":"{\"method\":\"mouseMove\",\"params\":{\"appHandle\":\"fixture-app\",\"snapshotId\":\"fixture-snapshot-9\",\"elementId\":\"0\"}}"}}
|
||||
{"type":"tool/result","data":{"turn":1,"step":37,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-hover-enter"},"toolCallId":"cua-hover-enter","content":[{"type":"text","text":"{\"dispatched\":true,\"hoverRoute\":\"accessibility\",\"systemPointerMoved\":false}"}],"isError":false,"id":"{{message:77}}"}},"sourceEventSeqs":[194],"surfaceOp":"append"}
|
||||
{"type":"step/end","data":{"turn":1,"step":37}}
|
||||
{"type":"step/start","data":{"turn":1,"step":38}}
|
||||
{"type":"assistant/message","data":{"turn":1,"step":38,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-hover-observe","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:78}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947690495,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947690495,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-hover-observe","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}}},{"type":"chunk","time":1791947690495,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947690495,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"}
|
||||
{"type":"tool/call","data":{"turn":1,"step":38,"callId":"cua-hover-observe","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}}
|
||||
{"type":"tool/result","data":{"turn":1,"step":38,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-hover-observe"},"toolCallId":"cua-hover-observe","content":[{"type":"text","text":"{\"appHandle\":\"fixture-app\",\"pid\":9,\"windowId\":7,\"snapshotId\":\"fixture-snapshot-10\",\"tree\":\"AX scope=window offsets=UTF-16\\n- [0] AXButton \\\"Increment\\\"\\n- [1] AXTextField valueWritable=true selectionWritable=true value=\\\"\\\" placeholder=\\\"Fixture text\\\"\\n- AXStaticText \\\"Count: 3\\\"\\n- AXStaticText \\\"Hover information\\\"\\n- [2] AXMenuBarItem \\\"File\\\"\\nMenu contents omitted; request includeMenus=true to expand.\",\"windows\":[{\"windowId\":7,\"pid\":9,\"appName\":\"Fixture\",\"title\":\"CUA fixture\",\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"isOnScreen\":true,\"onCurrentSpace\":true}],\"truncated\":false}"}],"isError":false,"id":"{{message:79}}"}},"sourceEventSeqs":[199],"surfaceOp":"append"}
|
||||
{"type":"tool/result","data":{"turn":1,"step":38,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-hover-observe"},"toolCallId":"cua-hover-observe","content":[{"type":"text","text":"{\"appHandle\":\"fixture-app\",\"windowId\":7,\"snapshotId\":\"fixture-snapshot-10\",\"tree\":\"AX scope=window offsets=UTF-16\\n- [0] AXButton \\\"Increment\\\"\\n- [1] AXTextField valueWritable=true selectionWritable=true value=\\\"\\\" placeholder=\\\"Fixture text\\\"\\n- AXStaticText \\\"Count: 3\\\"\\n- AXStaticText \\\"Hover information\\\"\\n- [2] AXMenuBarItem \\\"File\\\"\\nMenu contents omitted; request includeMenus=true to expand.\",\"windows\":[{\"windowId\":7,\"appName\":\"Fixture\",\"title\":\"CUA fixture\",\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"isOnScreen\":true,\"onCurrentSpace\":true}],\"truncated\":false}"}],"isError":false,"id":"{{message:79}}"}},"sourceEventSeqs":[199],"surfaceOp":"append"}
|
||||
{"type":"step/end","data":{"turn":1,"step":38}}
|
||||
{"type":"step/start","data":{"turn":1,"step":39}}
|
||||
{"type":"assistant/message","data":{"turn":1,"step":39,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-hover-capture","name":"cua","arguments":"{\"method\":\"screenshot\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:80}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947690517,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947690517,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-hover-capture","name":"cua","arguments":"{\"method\":\"screenshot\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}}},{"type":"chunk","time":1791947690517,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947690517,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"}
|
||||
@@ -207,29 +207,29 @@
|
||||
{"type":"tool/result","data":{"turn":1,"step":39,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-hover-capture"},"toolCallId":"cua-hover-capture","content":[{"type":"text","text":"{\"pixelWidth\":1,\"pixelHeight\":1,\"windowId\":7,\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"coordinateSpace\":\"screenshot-pixels\"}"},{"type":"image","attachment":{"attachmentId":"sha256:999f1d1527ee7e79266f16add5430fff76b1225d742464a5b1ff1f02971bb8ee","mediaType":"image/png","width":1,"height":1,"bytes":90}}],"isError":false,"id":"{{message:81}}"}},"sourceEventSeqs":[204],"surfaceOp":"append"}
|
||||
{"type":"step/end","data":{"turn":1,"step":39}}
|
||||
{"type":"step/start","data":{"turn":1,"step":40}}
|
||||
{"type":"assistant/message","data":{"turn":1,"step":40,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-hover-popup","name":"cua","arguments":"{\"method\":\"hover\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7,\"x\":100,\"y\":200,\"relative\":true}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:82}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947690565,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947690565,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-hover-popup","name":"cua","arguments":"{\"method\":\"hover\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7,\"x\":100,\"y\":200,\"relative\":true}}"}}},{"type":"chunk","time":1791947690565,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947690565,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"}
|
||||
{"type":"tool/call","data":{"turn":1,"step":40,"callId":"cua-hover-popup","name":"cua","arguments":"{\"method\":\"hover\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7,\"x\":100,\"y\":200,\"relative\":true}}"}}
|
||||
{"type":"tool/result","data":{"turn":1,"step":40,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-hover-popup"},"toolCallId":"cua-hover-popup","content":[{"type":"text","text":"{\"dispatched\":true,\"verified\":false,\"hoverRoute\":\"accessibility\",\"systemPointerMoved\":false}"}],"isError":false,"id":"{{message:83}}"}},"sourceEventSeqs":[209],"surfaceOp":"append"}
|
||||
{"type":"assistant/message","data":{"turn":1,"step":40,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-hover-popup","name":"cua","arguments":"{\"method\":\"mouseMove\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7,\"x\":100,\"y\":200,\"relative\":true}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:82}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947690565,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947690565,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-hover-popup","name":"cua","arguments":"{\"method\":\"mouseMove\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7,\"x\":100,\"y\":200,\"relative\":true}}"}}},{"type":"chunk","time":1791947690565,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947690565,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"}
|
||||
{"type":"tool/call","data":{"turn":1,"step":40,"callId":"cua-hover-popup","name":"cua","arguments":"{\"method\":\"mouseMove\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7,\"x\":100,\"y\":200,\"relative\":true}}"}}
|
||||
{"type":"tool/result","data":{"turn":1,"step":40,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-hover-popup"},"toolCallId":"cua-hover-popup","content":[{"type":"text","text":"{\"dispatched\":true,\"hoverRoute\":\"accessibility\",\"systemPointerMoved\":false}"}],"isError":false,"id":"{{message:83}}"}},"sourceEventSeqs":[209],"surfaceOp":"append"}
|
||||
{"type":"step/end","data":{"turn":1,"step":40}}
|
||||
{"type":"step/start","data":{"turn":1,"step":41}}
|
||||
{"type":"assistant/message","data":{"turn":1,"step":41,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-hover-click","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7,\"snapshotId\":\"fixture-snapshot-10\",\"x\":100,\"y\":200,\"relative\":true}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:84}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947690590,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947690590,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-hover-click","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7,\"snapshotId\":\"fixture-snapshot-10\",\"x\":100,\"y\":200,\"relative\":true}}"}}},{"type":"chunk","time":1791947690590,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947690590,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"}
|
||||
{"type":"tool/call","data":{"turn":1,"step":41,"callId":"cua-hover-click","name":"cua","arguments":"{\"method\":\"click\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7,\"snapshotId\":\"fixture-snapshot-10\",\"x\":100,\"y\":200,\"relative\":true}}"}}
|
||||
{"type":"tool/result","data":{"turn":1,"step":41,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-hover-click"},"toolCallId":"cua-hover-click","content":[{"type":"text","text":"{\"dispatched\":true,\"verified\":false}"}],"isError":false,"id":"{{message:85}}"}},"sourceEventSeqs":[214],"surfaceOp":"append"}
|
||||
{"type":"tool/result","data":{"turn":1,"step":41,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-hover-click"},"toolCallId":"cua-hover-click","content":[{"type":"text","text":"{\"dispatched\":true}"}],"isError":false,"id":"{{message:85}}"}},"sourceEventSeqs":[214],"surfaceOp":"append"}
|
||||
{"type":"step/end","data":{"turn":1,"step":41}}
|
||||
{"type":"step/start","data":{"turn":1,"step":42}}
|
||||
{"type":"assistant/message","data":{"turn":1,"step":42,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-hover-exit","name":"cua","arguments":"{\"method\":\"hover\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7,\"x\":0,\"y\":200,\"relative\":true}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:86}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947690613,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947690613,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-hover-exit","name":"cua","arguments":"{\"method\":\"hover\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7,\"x\":0,\"y\":200,\"relative\":true}}"}}},{"type":"chunk","time":1791947690613,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947690613,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"}
|
||||
{"type":"tool/call","data":{"turn":1,"step":42,"callId":"cua-hover-exit","name":"cua","arguments":"{\"method\":\"hover\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7,\"x\":0,\"y\":200,\"relative\":true}}"}}
|
||||
{"type":"tool/result","data":{"turn":1,"step":42,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-hover-exit"},"toolCallId":"cua-hover-exit","content":[{"type":"text","text":"{\"dispatched\":true,\"verified\":false,\"hoverRoute\":\"accessibility\",\"systemPointerMoved\":false}"}],"isError":false,"id":"{{message:87}}"}},"sourceEventSeqs":[219],"surfaceOp":"append"}
|
||||
{"type":"assistant/message","data":{"turn":1,"step":42,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-hover-exit","name":"cua","arguments":"{\"method\":\"mouseMove\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7,\"x\":0,\"y\":200,\"relative\":true}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:86}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947690613,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947690613,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-hover-exit","name":"cua","arguments":"{\"method\":\"mouseMove\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7,\"x\":0,\"y\":200,\"relative\":true}}"}}},{"type":"chunk","time":1791947690613,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947690613,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"}
|
||||
{"type":"tool/call","data":{"turn":1,"step":42,"callId":"cua-hover-exit","name":"cua","arguments":"{\"method\":\"mouseMove\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7,\"x\":0,\"y\":200,\"relative\":true}}"}}
|
||||
{"type":"tool/result","data":{"turn":1,"step":42,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-hover-exit"},"toolCallId":"cua-hover-exit","content":[{"type":"text","text":"{\"dispatched\":true,\"hoverRoute\":\"accessibility\",\"systemPointerMoved\":false}"}],"isError":false,"id":"{{message:87}}"}},"sourceEventSeqs":[219],"surfaceOp":"append"}
|
||||
{"type":"step/end","data":{"turn":1,"step":42}}
|
||||
{"type":"step/start","data":{"turn":1,"step":43}}
|
||||
{"type":"assistant/message","data":{"turn":1,"step":43,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-hover-verify","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:88}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947690633,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947690633,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-hover-verify","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}}},{"type":"chunk","time":1791947690633,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947690633,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"}
|
||||
{"type":"tool/call","data":{"turn":1,"step":43,"callId":"cua-hover-verify","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}}
|
||||
{"type":"tool/result","data":{"turn":1,"step":43,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-hover-verify"},"toolCallId":"cua-hover-verify","content":[{"type":"text","text":"{\"appHandle\":\"fixture-app\",\"pid\":9,\"windowId\":7,\"snapshotId\":\"fixture-snapshot-11\",\"tree\":\"AX scope=window offsets=UTF-16\\n- [0] AXButton \\\"Increment\\\"\\n- [1] AXTextField valueWritable=true selectionWritable=true value=\\\"\\\" placeholder=\\\"Fixture text\\\"\\n- AXStaticText \\\"Count: 4\\\"\\n- [2] AXMenuBarItem \\\"File\\\"\\nMenu contents omitted; request includeMenus=true to expand.\",\"windows\":[{\"windowId\":7,\"pid\":9,\"appName\":\"Fixture\",\"title\":\"CUA fixture\",\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"isOnScreen\":true,\"onCurrentSpace\":true}],\"truncated\":false}"}],"isError":false,"id":"{{message:89}}"}},"sourceEventSeqs":[224],"surfaceOp":"append"}
|
||||
{"type":"tool/result","data":{"turn":1,"step":43,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-hover-verify"},"toolCallId":"cua-hover-verify","content":[{"type":"text","text":"{\"appHandle\":\"fixture-app\",\"windowId\":7,\"snapshotId\":\"fixture-snapshot-11\",\"tree\":\"AX scope=window offsets=UTF-16\\n- [0] AXButton \\\"Increment\\\"\\n- [1] AXTextField valueWritable=true selectionWritable=true value=\\\"\\\" placeholder=\\\"Fixture text\\\"\\n- AXStaticText \\\"Count: 4\\\"\\n- [2] AXMenuBarItem \\\"File\\\"\\nMenu contents omitted; request includeMenus=true to expand.\",\"windows\":[{\"windowId\":7,\"appName\":\"Fixture\",\"title\":\"CUA fixture\",\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"isOnScreen\":true,\"onCurrentSpace\":true}],\"truncated\":false}"}],"isError":false,"id":"{{message:89}}"}},"sourceEventSeqs":[224],"surfaceOp":"append"}
|
||||
{"type":"step/end","data":{"turn":1,"step":43}}
|
||||
{"type":"step/start","data":{"turn":1,"step":44}}
|
||||
{"type":"assistant/message","data":{"turn":1,"step":44,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-paste-background","name":"cua","arguments":"{\"method\":\"pressKey\",\"params\":{\"appHandle\":\"fixture-app\",\"key\":\"F9\"}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:90}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947690653,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947690653,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-paste-background","name":"cua","arguments":"{\"method\":\"pressKey\",\"params\":{\"appHandle\":\"fixture-app\",\"key\":\"F9\"}}"}}},{"type":"chunk","time":1791947690653,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947690653,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"}
|
||||
{"type":"tool/call","data":{"turn":1,"step":44,"callId":"cua-paste-background","name":"cua","arguments":"{\"method\":\"pressKey\",\"params\":{\"appHandle\":\"fixture-app\",\"key\":\"F9\"}}"}}
|
||||
{"type":"tool/result","data":{"turn":1,"step":44,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-paste-background"},"toolCallId":"cua-paste-background","content":[{"type":"text","text":"{\"dispatched\":true,\"verified\":false}"}],"isError":false,"id":"{{message:91}}"}},"sourceEventSeqs":[229],"surfaceOp":"append"}
|
||||
{"type":"tool/result","data":{"turn":1,"step":44,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-paste-background"},"toolCallId":"cua-paste-background","content":[{"type":"text","text":"{\"dispatched\":true}"}],"isError":false,"id":"{{message:91}}"}},"sourceEventSeqs":[229],"surfaceOp":"append"}
|
||||
{"type":"step/end","data":{"turn":1,"step":44}}
|
||||
{"type":"step/start","data":{"turn":1,"step":45}}
|
||||
{"type":"assistant/message","data":{"turn":1,"step":45,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-paste-unread","name":"cua","arguments":"{\"method\":\"paste\",\"params\":{\"appHandle\":\"fixture-app\",\"text\":\"fixture paste\"}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:92}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947690672,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947690672,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-paste-unread","name":"cua","arguments":"{\"method\":\"paste\",\"params\":{\"appHandle\":\"fixture-app\",\"text\":\"fixture paste\"}}"}}},{"type":"chunk","time":1791947690672,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947690672,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"}
|
||||
@@ -239,32 +239,32 @@
|
||||
{"type":"step/start","data":{"turn":1,"step":46}}
|
||||
{"type":"assistant/message","data":{"turn":1,"step":46,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-paste-inspect","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:94}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947690696,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947690696,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-paste-inspect","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}}},{"type":"chunk","time":1791947690696,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947690696,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"}
|
||||
{"type":"tool/call","data":{"turn":1,"step":46,"callId":"cua-paste-inspect","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}}
|
||||
{"type":"tool/result","data":{"turn":1,"step":46,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-paste-inspect"},"toolCallId":"cua-paste-inspect","content":[{"type":"text","text":"{\"appHandle\":\"fixture-app\",\"pid\":9,\"windowId\":7,\"snapshotId\":\"fixture-snapshot-12\",\"tree\":\"AX scope=window offsets=UTF-16\\n- [0] AXButton \\\"Increment\\\"\\n- [1] AXTextField valueWritable=true selectionWritable=true value=\\\"\\\" placeholder=\\\"Fixture text\\\"\\n- AXStaticText \\\"Count: 4\\\"\\n- [2] AXMenuBarItem \\\"File\\\"\\nMenu contents omitted; request includeMenus=true to expand.\",\"windows\":[{\"windowId\":7,\"pid\":9,\"appName\":\"Fixture\",\"title\":\"CUA fixture\",\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"isOnScreen\":true,\"onCurrentSpace\":true}],\"truncated\":false}"}],"isError":false,"id":"{{message:95}}"}},"sourceEventSeqs":[239],"surfaceOp":"append"}
|
||||
{"type":"tool/result","data":{"turn":1,"step":46,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-paste-inspect"},"toolCallId":"cua-paste-inspect","content":[{"type":"text","text":"{\"appHandle\":\"fixture-app\",\"windowId\":7,\"snapshotId\":\"fixture-snapshot-12\",\"tree\":\"AX scope=window offsets=UTF-16\\n- [0] AXButton \\\"Increment\\\"\\n- [1] AXTextField valueWritable=true selectionWritable=true value=\\\"\\\" placeholder=\\\"Fixture text\\\"\\n- AXStaticText \\\"Count: 4\\\"\\n- [2] AXMenuBarItem \\\"File\\\"\\nMenu contents omitted; request includeMenus=true to expand.\",\"windows\":[{\"windowId\":7,\"appName\":\"Fixture\",\"title\":\"CUA fixture\",\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"isOnScreen\":true,\"onCurrentSpace\":true}],\"truncated\":false}"}],"isError":false,"id":"{{message:95}}"}},"sourceEventSeqs":[239],"surfaceOp":"append"}
|
||||
{"type":"step/end","data":{"turn":1,"step":46}}
|
||||
{"type":"step/start","data":{"turn":1,"step":47}}
|
||||
{"type":"assistant/message","data":{"turn":1,"step":47,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-paste-activate","name":"cua","arguments":"{\"method\":\"activate\",\"params\":{\"appHandle\":\"fixture-app\"}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:96}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947690714,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947690714,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-paste-activate","name":"cua","arguments":"{\"method\":\"activate\",\"params\":{\"appHandle\":\"fixture-app\"}}"}}},{"type":"chunk","time":1791947690714,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947690714,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"}
|
||||
{"type":"tool/call","data":{"turn":1,"step":47,"callId":"cua-paste-activate","name":"cua","arguments":"{\"method\":\"activate\",\"params\":{\"appHandle\":\"fixture-app\"}}"}}
|
||||
{"type":"tool/result","data":{"turn":1,"step":47,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-paste-activate"},"toolCallId":"cua-paste-activate","content":[{"type":"text","text":"{\"dispatched\":true,\"verified\":false,\"foreground\":{\"requestId\":\"cua-paste-activate\",\"operation\":\"activate\",\"targetPid\":9,\"targetWindowId\":null,\"before\":{\"frontPid\":42,\"focusedWindowId\":7,\"frontWindowId\":99,\"frontWindowPid\":42},\"after\":{\"frontPid\":9,\"focusedWindowId\":7,\"frontWindowId\":7,\"frontWindowPid\":9},\"activationRequested\":true,\"raiseRequested\":false}}"}],"isError":false,"id":"{{message:97}}"}},"sourceEventSeqs":[244],"surfaceOp":"append"}
|
||||
{"type":"tool/result","data":{"turn":1,"step":47,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-paste-activate"},"toolCallId":"cua-paste-activate","content":[{"type":"text","text":"{\"dispatched\":true,\"foreground\":{\"requestId\":\"cua-paste-activate\",\"operation\":\"activate\",\"targetPid\":9,\"targetWindowId\":null,\"before\":{\"frontPid\":42,\"focusedWindowId\":7,\"frontWindowId\":99,\"frontWindowPid\":42},\"after\":{\"frontPid\":9,\"focusedWindowId\":7,\"frontWindowId\":7,\"frontWindowPid\":9},\"activationRequested\":true,\"raiseRequested\":false}}"}],"isError":false,"id":"{{message:97}}"}},"sourceEventSeqs":[244],"surfaceOp":"append"}
|
||||
{"type":"step/end","data":{"turn":1,"step":47}}
|
||||
{"type":"step/start","data":{"turn":1,"step":48}}
|
||||
{"type":"assistant/message","data":{"turn":1,"step":48,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-paste-focus","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:98}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947690734,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947690734,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-paste-focus","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}}},{"type":"chunk","time":1791947690734,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947690734,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"}
|
||||
{"type":"tool/call","data":{"turn":1,"step":48,"callId":"cua-paste-focus","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}}
|
||||
{"type":"tool/result","data":{"turn":1,"step":48,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-paste-focus"},"toolCallId":"cua-paste-focus","content":[{"type":"text","text":"{\"appHandle\":\"fixture-app\",\"pid\":9,\"windowId\":7,\"snapshotId\":\"fixture-snapshot-13\",\"tree\":\"AX scope=window offsets=UTF-16\\n- [0] AXButton \\\"Increment\\\"\\n- [1] AXTextField valueWritable=true selectionWritable=true value=\\\"\\\" placeholder=\\\"Fixture text\\\"\\n- AXStaticText \\\"Count: 4\\\"\\n- [2] AXMenuBarItem \\\"File\\\"\\nMenu contents omitted; request includeMenus=true to expand.\",\"windows\":[{\"windowId\":7,\"pid\":9,\"appName\":\"Fixture\",\"title\":\"CUA fixture\",\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"isOnScreen\":true,\"onCurrentSpace\":true}],\"truncated\":false}"}],"isError":false,"id":"{{message:99}}"}},"sourceEventSeqs":[249],"surfaceOp":"append"}
|
||||
{"type":"tool/result","data":{"turn":1,"step":48,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-paste-focus"},"toolCallId":"cua-paste-focus","content":[{"type":"text","text":"{\"appHandle\":\"fixture-app\",\"windowId\":7,\"snapshotId\":\"fixture-snapshot-13\",\"tree\":\"AX scope=window offsets=UTF-16\\n- [0] AXButton \\\"Increment\\\"\\n- [1] AXTextField valueWritable=true selectionWritable=true value=\\\"\\\" placeholder=\\\"Fixture text\\\"\\n- AXStaticText \\\"Count: 4\\\"\\n- [2] AXMenuBarItem \\\"File\\\"\\nMenu contents omitted; request includeMenus=true to expand.\",\"windows\":[{\"windowId\":7,\"appName\":\"Fixture\",\"title\":\"CUA fixture\",\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"isOnScreen\":true,\"onCurrentSpace\":true}],\"truncated\":false}"}],"isError":false,"id":"{{message:99}}"}},"sourceEventSeqs":[249],"surfaceOp":"append"}
|
||||
{"type":"step/end","data":{"turn":1,"step":48}}
|
||||
{"type":"step/start","data":{"turn":1,"step":49}}
|
||||
{"type":"assistant/message","data":{"turn":1,"step":49,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-paste-consumed","name":"cua","arguments":"{\"method\":\"paste\",\"params\":{\"appHandle\":\"fixture-app\",\"text\":\"fixture paste\"}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:100}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947690757,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947690757,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-paste-consumed","name":"cua","arguments":"{\"method\":\"paste\",\"params\":{\"appHandle\":\"fixture-app\",\"text\":\"fixture paste\"}}"}}},{"type":"chunk","time":1791947690757,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947690757,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"}
|
||||
{"type":"tool/call","data":{"turn":1,"step":49,"callId":"cua-paste-consumed","name":"cua","arguments":"{\"method\":\"paste\",\"params\":{\"appHandle\":\"fixture-app\",\"text\":\"fixture paste\"}}"}}
|
||||
{"type":"tool/result","data":{"turn":1,"step":49,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-paste-consumed"},"toolCallId":"cua-paste-consumed","content":[{"type":"text","text":"{\"dispatched\":true,\"verified\":false}"}],"isError":false,"id":"{{message:101}}"}},"sourceEventSeqs":[254],"surfaceOp":"append"}
|
||||
{"type":"tool/result","data":{"turn":1,"step":49,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-paste-consumed"},"toolCallId":"cua-paste-consumed","content":[{"type":"text","text":"{\"dispatched\":true}"}],"isError":false,"id":"{{message:101}}"}},"sourceEventSeqs":[254],"surfaceOp":"append"}
|
||||
{"type":"step/end","data":{"turn":1,"step":49}}
|
||||
{"type":"step/start","data":{"turn":1,"step":50}}
|
||||
{"type":"assistant/message","data":{"turn":1,"step":50,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-paste-verify","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:102}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947690778,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947690778,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-paste-verify","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}}},{"type":"chunk","time":1791947690778,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947690778,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"}
|
||||
{"type":"tool/call","data":{"turn":1,"step":50,"callId":"cua-paste-verify","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}}
|
||||
{"type":"tool/result","data":{"turn":1,"step":50,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-paste-verify"},"toolCallId":"cua-paste-verify","content":[{"type":"text","text":"{\"appHandle\":\"fixture-app\",\"pid\":9,\"windowId\":7,\"snapshotId\":\"fixture-snapshot-14\",\"tree\":\"AX scope=window offsets=UTF-16\\n- [0] AXButton \\\"Increment\\\"\\n- [1] AXTextField valueWritable=true selectionWritable=true value=\\\"fixture paste\\\" placeholder=\\\"Fixture text\\\"\\n- AXStaticText \\\"Count: 4\\\"\\n- [2] AXMenuBarItem \\\"File\\\"\\nMenu contents omitted; request includeMenus=true to expand.\",\"windows\":[{\"windowId\":7,\"pid\":9,\"appName\":\"Fixture\",\"title\":\"CUA fixture\",\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"isOnScreen\":true,\"onCurrentSpace\":true}],\"truncated\":false}"}],"isError":false,"id":"{{message:103}}"}},"sourceEventSeqs":[259],"surfaceOp":"append"}
|
||||
{"type":"tool/result","data":{"turn":1,"step":50,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-paste-verify"},"toolCallId":"cua-paste-verify","content":[{"type":"text","text":"{\"appHandle\":\"fixture-app\",\"windowId\":7,\"snapshotId\":\"fixture-snapshot-14\",\"tree\":\"AX scope=window offsets=UTF-16\\n- [0] AXButton \\\"Increment\\\"\\n- [1] AXTextField valueWritable=true selectionWritable=true value=\\\"fixture paste\\\" placeholder=\\\"Fixture text\\\"\\n- AXStaticText \\\"Count: 4\\\"\\n- [2] AXMenuBarItem \\\"File\\\"\\nMenu contents omitted; request includeMenus=true to expand.\",\"windows\":[{\"windowId\":7,\"appName\":\"Fixture\",\"title\":\"CUA fixture\",\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"isOnScreen\":true,\"onCurrentSpace\":true}],\"truncated\":false}"}],"isError":false,"id":"{{message:103}}"}},"sourceEventSeqs":[259],"surfaceOp":"append"}
|
||||
{"type":"step/end","data":{"turn":1,"step":50}}
|
||||
{"type":"step/start","data":{"turn":1,"step":51}}
|
||||
{"type":"assistant/message","data":{"turn":1,"step":51,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-debug-arm","name":"cua","arguments":"{\"method\":\"pressKey\",\"params\":{\"appHandle\":\"fixture-app\",\"key\":\"F12\"}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:104}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947690801,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947690801,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-debug-arm","name":"cua","arguments":"{\"method\":\"pressKey\",\"params\":{\"appHandle\":\"fixture-app\",\"key\":\"F12\"}}"}}},{"type":"chunk","time":1791947690801,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947690801,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"}
|
||||
{"type":"tool/call","data":{"turn":1,"step":51,"callId":"cua-debug-arm","name":"cua","arguments":"{\"method\":\"pressKey\",\"params\":{\"appHandle\":\"fixture-app\",\"key\":\"F12\"}}"}}
|
||||
{"type":"tool/result","data":{"turn":1,"step":51,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-debug-arm"},"toolCallId":"cua-debug-arm","content":[{"type":"text","text":"{\"dispatched\":true,\"verified\":false}"}],"isError":false,"id":"{{message:105}}"}},"sourceEventSeqs":[264],"surfaceOp":"append"}
|
||||
{"type":"tool/result","data":{"turn":1,"step":51,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-debug-arm"},"toolCallId":"cua-debug-arm","content":[{"type":"text","text":"{\"dispatched\":true}"}],"isError":false,"id":"{{message:105}}"}},"sourceEventSeqs":[264],"surfaceOp":"append"}
|
||||
{"type":"step/end","data":{"turn":1,"step":51}}
|
||||
{"type":"step/start","data":{"turn":1,"step":52}}
|
||||
{"type":"assistant/message","data":{"turn":1,"step":52,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-debug-screenshot","name":"cua","arguments":"{\"method\":\"screenshot\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:106}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947690821,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947690821,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-debug-screenshot","name":"cua","arguments":"{\"method\":\"screenshot\",\"params\":{\"appHandle\":\"fixture-app\",\"windowId\":7}}"}}},{"type":"chunk","time":1791947690821,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947690821,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"}
|
||||
@@ -279,7 +279,7 @@
|
||||
{"type":"step/start","data":{"turn":1,"step":54}}
|
||||
{"type":"assistant/message","data":{"turn":1,"step":54,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-consent-revocable","name":"cua","arguments":"{\"method\":\"getApp\",\"params\":{\"selector\":\"Revocable Fixture\"}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:110}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947691022,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947691022,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-consent-revocable","name":"cua","arguments":"{\"method\":\"getApp\",\"params\":{\"selector\":\"Revocable Fixture\"}}"}}},{"type":"chunk","time":1791947691022,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947691022,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"}
|
||||
{"type":"tool/call","data":{"turn":1,"step":54,"callId":"cua-consent-revocable","name":"cua","arguments":"{\"method\":\"getApp\",\"params\":{\"selector\":\"Revocable Fixture\"}}"}}
|
||||
{"type":"tool/result","data":{"turn":1,"step":54,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-consent-revocable"},"toolCallId":"cua-consent-revocable","content":[{"type":"text","text":"{\"appHandle\":\"revoked-app\",\"pid\":17,\"name\":\"Revocable Fixture\"}"}],"isError":false,"id":"{{message:111}}"}},"sourceEventSeqs":[279],"surfaceOp":"append"}
|
||||
{"type":"tool/result","data":{"turn":1,"step":54,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-consent-revocable"},"toolCallId":"cua-consent-revocable","content":[{"type":"text","text":"{\"appHandle\":\"revoked-app\",\"name\":\"Revocable Fixture\",\"running\":true}"}],"isError":false,"id":"{{message:111}}"}},"sourceEventSeqs":[279],"surfaceOp":"append"}
|
||||
{"type":"step/end","data":{"turn":1,"step":54}}
|
||||
{"type":"step/start","data":{"turn":1,"step":55}}
|
||||
{"type":"assistant/message","data":{"turn":1,"step":55,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-consent-revoked","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"revoked-app\"}}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:112}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791947691192,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791947691192,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-consent-revoked","name":"cua","arguments":"{\"method\":\"getState\",\"params\":{\"appHandle\":\"revoked-app\"}}"}}},{"type":"chunk","time":1791947691192,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791947691192,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"}
|
||||
|
||||
@@ -33,53 +33,53 @@ Start independent delegations with `subagent` or `subagent_fork` together in one
|
||||
|
||||
Bash, formatters, code generators, and scripts can overwrite concurrent edits without conflict detection; coordinate their changes explicitly. The agent integrating the work must review the final diff and run relevant tests.
|
||||
|
||||
Use cua with method getCapabilities and params {} to inspect permissions and sessionLocked. If sessionLocked is true or an operation reports SESSION_LOCKED, stop UI attempts and ask the user to unlock the desktop session. A null sessionLocked means no usable console session was verified. After unlocking, read fresh state; screenshots from a locked desktop are not current observations. Use listApps or cua.getState to discover running applications, and findApps(query) to search installed applications by name or application path. getApp requests user consent before launching a stopped application in the background, or before observations and input on a running one. If access is denied or revoked, stop using that application until the user explicitly permits another authorization request. Never switch tools to bypass a refusal. Keep the returned appHandle; every app operation requires it. listWindows returns window IDs and bounds in the provider’s native units. Raw getState returns AX text and snapshotId. Menu-bar entries are shown by default; includeMenus=true expands their contents. details=true adds identifiers, global native bounds and complete action names. Both options also apply to getAXState. Each read replaces the indices for that app/window scope. Use its element indices only with that snapshotId. Raw screenshot and coordinate operations require a windowId. Coordinates are pixels in the latest returned screenshot, measured from its top-left. Set relative:true for integer thousandths from 0 to 999 instead. The Host converts both modes to native window coordinates; AX detail bounds remain global native coordinates. A verified window can expose no AX elements. Its empty snapshot permits screenshots and coordinate input; element-index actions require an element from a fresh AX observation.
|
||||
macOS: accessibility tree operations are available at app or window level. Simulated keyboard input is available at app level. Simulated mouse input is available at window level. getApp also accepts bundle IDs and .app paths, and findApps searches /Applications, /System/Applications, and ~/Applications. Use the Computer Use permission page to grant the helper application Accessibility and Screen Recording permissions in System Settings. Native window and AX bounds use macOS points, while delivered screenshots carry their own pixel dimensions. show does not switch Spaces, and Stage Manager can present reduced window geometry; explain any required foreground step before activate or raise.
|
||||
|
||||
Initialize the object API once in persistent PTC. Reuse its variables in later calls; put temporary declarations inside { ... } blocks to avoid redeclaration errors. All object methods are async. getAXState returns text, not a structured state object. Its text separates values and placeholders, and reports subroles, help, URLs, disabled/focused/selected/expanded state and UTF-16 selection ranges when available. Writability is true, false or unknown on relevant controls; omitted state fields do not assert false. selection=unavailable means the reported range exceeds the observed text. Compact output omits ordinary Press/Pick (click), ShowMenu (right click) and ScrollToVisible actions; details=true lists every available action. Use the displayed action name with performSecondaryAction; ambiguous aliases require the complete name from details=true. Truncation reports the reached provider limit. Narrow the window or increase that provider limit before reading again.
|
||||
Use cua to observe and operate native applications. Always use cua.getCapabilities() to confirm whether a specific API is available on the current platform; its reported capabilities take precedence over availability statements in these instructions. All methods are async; observation options default to { emit: true }, and emit: false suppresses sink output only, leaving returned values unchanged. Actions report dispatch evidence, not verified completion. If sessionLocked is true or an operation reports SESSION_LOCKED, stop UI actions and ask the user to unlock the desktop session; null or unknown means verify a usable unlocked desktop session first. cua.getApp requests user consent; if access is denied or revoked, stop using that application until the user explicitly permits another request, and never bypass a refusal or switch tools to avoid it. Never replay uncertain input without a fresh observation.
|
||||
|
||||
Use app methods directly for single-window tasks; getApp already reads initial AX state. Call app.getWindows() and await app.getWindow(windowId) only to target a particular window or resolve ambiguous window selection. App screenshots and coordinate actions use the latest AX observation: its focused on-screen, nonzero-size window, otherwise the sole such window. They do not refresh focus automatically. Read app.getAXState() to follow changed focus.
|
||||
In persistent PTC, use the host-provided bridge initialization once and retain the returned objects; cua comes from that bridge and is not a preinstalled global. Use void await for observations that emit, and never stringify screenshot image bytes.
|
||||
|
||||
CuaScreenshot contains { bytes: Uint8Array, mimeType: "image/png" | "image/jpeg" | "image/webp" | "image/gif", pixelWidth: number, pixelHeight: number, windowId: number }. Use mimeType to choose a file extension or declare an image; do not assume PNG.
|
||||
Signatures show resolved return values after await. ActionResult = { dispatched: boolean, foregroundChanged?: boolean | null }; foregroundChanged is omitted when not sampled and null when the sample is unavailable. JsonValue = string | number | boolean | null | JsonValue[] | { [key: string]: JsonValue }; numbers must be finite. App is the application object with readonly app.appHandle: string (opaque) and the methods listed below for app level. Window is the window object with readonly window.windowId: number and the methods listed below for window level. Neither object carries pid.
|
||||
|
||||
Object API signatures (options default to { emit: true }):
|
||||
Setup and discovery:
|
||||
- cua.getCapabilities(options?: { emit?: boolean }) -> { platform: string, permissions: { accessibility: boolean, screenRecording: boolean }, appLevelAccessibility: boolean, keyboard: ('app' | 'window')[], sessionLocked: boolean | null, methods: string[], coordinateSpace: 'screenshot-pixels', browser: false } — inspect reported permissions, sessionLocked, methods, and keyboard scopes without requesting permissions.
|
||||
- cua.listApps(options?: { emit?: boolean }) -> { id: string, displayName: string, isRunning: true, pid: number }[] — enumerate running applications.
|
||||
- cua.findApps(query: string, options?: { emit?: boolean }) -> ({ id: string, displayName: string, path: string } & ({ isRunning: true, pid: number } | { isRunning: false }))[] — search installed applications, including stopped matches; only running matches carry pid.
|
||||
- cua.getState(options?: { emit?: boolean }) -> { apps: { id: string, displayName: string, isRunning: true, pid: number }[], browsers: [] } — summarize running applications; this is not an accessibility observation.
|
||||
- cua.getApp(selector: string) -> App — select an application by name, bundle ID, absolute path, or running PID string, obtain consent, launch in the background if needed, and read initial AX state silently; it does not activate the app.
|
||||
- app.getWindows(options?: { emit?: boolean }) -> { windowId: number, appName: string, title: string, bounds: { x: number, y: number, width: number, height: number }, isOnScreen: boolean, onCurrentSpace: boolean | null }[] — enumerate windows with global native bounds; null onCurrentSpace means unknown Space membership.
|
||||
- app.getWindow(windowId: number) -> Window — select one window and read its initial AX state silently; do this before any window screenshot or coordinate input.
|
||||
- app.activate() -> ActionResult — request explicit foreground activation; use only when foreground access is required.
|
||||
- app.show() -> { dispatched: boolean, windowId: number | null, foregroundChanged: boolean | null } — unhide the app and restore its selected, focused, main, or sole window without activation.
|
||||
- window.show() -> { dispatched: boolean, windowId: number | null, foregroundChanged: boolean | null } — unhide and restore that explicit window without activation.
|
||||
|
||||
- cua.getCapabilities(options?) -> { platform, permissions, sessionLocked, methods, coordinateSpace, browser }
|
||||
- cua.listApps(options?) -> Array<{ id, displayName, isRunning: true, pid }>
|
||||
- cua.findApps(query: string, options?) -> Array<{ id, displayName, path, isRunning, pid? }>
|
||||
- cua.getState(options?) -> { apps, browsers: [] }
|
||||
- cua.getApp(selector: string) -> App
|
||||
- app.activate() -> { dispatched: true, verified: false }
|
||||
- app.getWindows(options?) -> Array<{ windowId, pid, appName, title, bounds, isOnScreen, onCurrentSpace }>
|
||||
- app.getWindow(windowId: number) -> Window
|
||||
- app.typeText(text: string), app.pressKey(chord: string) -> action result
|
||||
- app.paste(text: string, options?: { format?: "text" | "html" }) -> action result
|
||||
## Accessibility tree
|
||||
Available at app or window level depending on the platform.
|
||||
|
||||
App and Window methods:
|
||||
Element ids are opaque strings from the latest successful tree; the object facade keeps one current snapshot per app across its repeated app and window objects, a new read replaces it, and a failed read invalidates the previous one. Repeated strings can appear for different elements, so do not reuse an old id blindly, and element operations do not fall back to pointer input or typing.
|
||||
- getAccessibilityTree(options?: { emit?: boolean, includeMenus?: boolean, details?: boolean, disableDiffing?: boolean }) -> string — read the full tree text; includeMenus expands menu contents and details adds identifiers, native bounds, and complete action names.
|
||||
- listActions(elementId: string) -> string[] — list the actions an element advertises.
|
||||
- invoke(elementId: string) -> ActionResult — invoke the element's default action; dispatched is false when it has none.
|
||||
- performAction(elementId: string, action: string, argument?: JsonValue) -> ActionResult — perform an observed action with an optional direct JSON argument.
|
||||
- setValue(elementId: string, value: JsonValue) -> ActionResult — write a control value.
|
||||
- selectText(elementId: string, text: string, options?: { prefix?: string, suffix?: string, selectionType?: 'text' | 'cursor_before' | 'cursor_after' }) -> ActionResult — select one literal occurrence or place the cursor beside it, using adjacent context to disambiguate.
|
||||
|
||||
- show() -> { dispatched: true, verified: false, windowId: number | null, foregroundChanged: boolean | null }
|
||||
- getAXState(options?: { emit?: boolean, disableDiffing?: boolean, includeMenus?: boolean, details?: boolean }) -> string
|
||||
- getScreenshot(options?: { emit?: boolean }) -> CuaScreenshot
|
||||
- getAXStateAndScreenshot(options?) -> { state: string, screenshot: CuaScreenshot }
|
||||
- click(target: number | [x, y], options?: { mouseButton?: "left" | "right" | "middle", clickCount?: 1 | 2, holdMs?: number, modifiers?: Array<"command" | "control" | "option" | "shift" | "fn">, relative?: boolean }) -> action result
|
||||
- hover(target: number | [x, y], options?: { relative?: boolean }) -> { dispatched, verified, hoverRoute, systemPointerMoved }
|
||||
- setValue(elementIndex: number, value: string) -> action result
|
||||
- selectText(elementIndex: number, text: string, options?: { prefix?: string, suffix?: string, selectionType?: "text" | "cursor_before" | "cursor_after" }) -> action result
|
||||
- performSecondaryAction(elementIndex: number, action: string) -> action result
|
||||
- scroll({ x, y, deltaLinesX, deltaLinesY, relative? }) -> action result
|
||||
- drag(from: [x, y], to: [x, y], options?: { mouseButton?: "left" | "right" | "middle", modifiers?: Array<"command" | "control" | "option" | "shift" | "fn">, relative?: boolean }) -> action result
|
||||
## Simulated mouse
|
||||
Use an explicit window for screenshots and pointer input.
|
||||
- getScreenshot(options?: { emit?: boolean }) -> { bytes: Uint8Array, mimeType: 'image/png' | 'image/jpeg' | 'image/webp' | 'image/gif', pixelWidth: number, pixelHeight: number, windowId: number } — capture the selected window as one encoded image; bytes is a JavaScript byte array, and mimeType identifies its encoding, so do not assume PNG.
|
||||
- mouseMove(target: string | [number, number], options?: { relative?: boolean }) -> { dispatched: boolean, foregroundChanged?: boolean | null, hoverRoute: 'accessibility' | 'system-pointer', systemPointerMoved: boolean } — reveal AX alternate UI or move the system pointer without pressing; inspect hoverRoute and systemPointerMoved, and expect no automatic pointer restoration.
|
||||
- click(target: string | [number, number], options?: { mouseButton?: 'left' | 'right' | 'middle', clickCount?: 1 | 2, modifiers?: ('command' | 'control' | 'option' | 'shift' | 'fn')[], holdMs?: number, relative?: boolean }) -> ActionResult — click an element id or point, or perform a left-button double-click; holdMs controls the press duration of each press.
|
||||
- scroll(options: { x: number, y: number, deltaLinesX: number, deltaLinesY: number, relative?: boolean }) -> ActionResult — scroll at a point; deltas are integers from -10 to 10 with at least one nonzero, and positive values reveal content above or left.
|
||||
- drag(from: [number, number], to: [number, number], options?: { mouseButton?: 'left' | 'right' | 'middle', modifiers?: ('command' | 'control' | 'option' | 'shift' | 'fn')[], relative?: boolean }) -> ActionResult — drag between two points of one window.
|
||||
|
||||
Raw operation tools use { method, params }, including params: {} for parameterless methods. getApp selects without activating the app. Keep applications in the background for observation and actions that the provider supports without foreground access. Do not activate or raise an app during initialization, before each action, or merely because another app is foreground. Prefer advertised AX actions. If a window is hidden or minimized, try app.show() or window.show() and read fresh state. show requests unhide and unminimize without activation or raising; it chooses the explicit, focused, main or sole AX window. A null windowId means no window was selected for restoration. foregroundChanged compares the front app and top window before and after the request; null means unavailable, and delayed focus changes remain possible. If restoration is unsupported or the window remains unusable, explain the needed foreground step, use app.activate() or an advertised raise action, and read fresh state. Completing the task takes priority when foreground access is necessary. Avoid repeated activation when the user switches away; continue background-capable steps and report what requires foreground access. Applications may still change focus in response to input.
|
||||
A string target is an element id; coordinate targets are [x, y] and are pixels in the latest returned screenshot by default. With relative: true they are integer thousandths of the window width and height from 0 to 999, based on the current AX observation or screenshot. Left clicks support one or two presses; right and middle clicks support one. FOREGROUND_REQUIRED means explain the need, then explicitly activate or raise and observe again; activation is never automatic. Read fresh state after actions, and treat WINDOW_CHANGED, STALE_SNAPSHOT, and SCREENSHOT_REQUIRED as requiring the stated fresh observation or capture before any retry.
|
||||
|
||||
Await operations sequentially. After input, read fresh state to verify the effect. A successful action means dispatched, not verified completion. Cancellation may leave partial effects; never retry uncertain input without observing. STALE_APP requires getApp again; STALE_SNAPSHOT requires getState again. Absolute coordinate input requires a fresh screenshot of that window. Relative input can use a successful AX observation or screenshot. SCREENSHOT_REQUIRED means capture a fresh screenshot before using pixel coordinates. For coordinate input, targets must be [x, y], for example click([650, 400], {relative:true}). A bare number such as click(650) is always an AX element index. The relative option applies only to coordinate targets, not AX indices. Window resizing since the latest observation rejects coordinate input; movement alone preserves window-local coordinates. WINDOW_CHANGED requires fresh state before any retry; use an advertised AX action or explicitly restore the window before screenshot or coordinate input, then observe again. PERMISSION_DENIED requires the user to address the reported system access restriction. After granting access, observe fresh state before continuing; do not automatically replay prior input.
|
||||
## Simulated keyboard
|
||||
Available at app or window level depending on the platform.
|
||||
- typeText(text: string) -> ActionResult — type literal text at the app's current focus; line endings send Return and tabs send Tab.
|
||||
- paste(text: string, options?: { format?: 'text' | 'html' }) -> ActionResult — deliver text or HTML through the app's Paste command, restoring prior clipboard contents unless another writer changes them; a Paste failure is surfaced without substituting typeText, and PASTE_NOT_CONSUMED means inspect before any retry.
|
||||
- pressKey(key: string, modifiers?: string[]) -> ActionResult — press and release one complete keystroke. For example pressKey('Tab', ['shift']) or pressKey('+'). Key and modifier names are platform-defined strings. Pass key and modifiers separately; do not pack them into a chord string.
|
||||
|
||||
Hover reveals alternate UI through a provider-supported accessibility action or moves the system pointer without pressing a button. Wait for the application's hover delay and observe before clicking; subsequent observations do not restore the pointer. Use successive hover calls for intermediate points. Other input and application changes can interrupt the interaction. click accepts optional holdMs, a nonnegative integer in milliseconds for each press, including double-click presses. Zero releases without an additional hold. The configured maximum is stated below. Explicit holdMs uses pointer input even for AX targets; omission preserves ordinary clicks. The operation deadline applies, and cancellation or timeout releases the button.
|
||||
|
||||
Use selectText only on controls advertising selectionWritable. It selects a unique literal occurrence, or places the cursor before/after it; prefix and suffix must be immediately adjacent. AMBIGUOUS_TEXT requires more context; TEXT_NOT_FOUND or SELECTION_CHANGED requires a fresh observation before deciding on input. Left clicks support single/double clicks; right and middle clicks support single clicks. Modified clicks use pointer input. Keypad names such as KP_0 and KP_Enter retain their physical key identity. Window objects expose observation and pointer/AX actions; keyboard input remains app-scoped and follows current focus. Scroll deltas are integer lines from -10 to 10; positive values reveal content above or left. Paste delivers plain text or HTML through the system clipboard and the application's paste action. Original clipboard formats are restored if the clipboard still belongs to this paste; a later copy is preserved. Unsupported Paste commands fail without typing fallback. CLIPBOARD_CHANGED means another writer intervened; observe before deciding on further input. Browser applications can be controlled with getApp like other native apps; browser tabs are unavailable. Installed-app search covers the locations documented by the platform provider; use an absolute application path for another location. Launch does not restore a window or document, and cancellation may leave an application already started. The execution host must install the object API and own variable persistence; this plugin does not preload a cua global. In persistent PTC, use the host-provided initialization snippet below once. Its installation-owned file URL works independently of the project directory. Retain app objects between calls. Observation methods emit by default and also return their value. Use void await app.getAXState() to avoid a duplicate REPL completion, or pass emit:false and print the returned text once. End screenshot calls with void await app.getScreenshot() or void await app.getAXStateAndScreenshot() so image bytes are not serialized as the program completion value.
|
||||
|
||||
macOS: getApp also accepts bundle IDs and .app paths. findApps searches /Applications, /System/Applications and ~/Applications. Accessibility and Screen Recording permissions belong to the responsible launching application. Use the Computer Use permission page to request access, open System Settings, and check again. Native window and AX bounds use macOS points. show does not switch Spaces. Stage Manager can present reduced window geometry; explain any required foreground step before activate or raise.
|
||||
|
||||
Hover first uses advertised AX alternate UI without moving the system cursor, preserving revealed descendants until the pointer target leaves their region. The system-pointer fallback requires the target point to be unobscured in the foreground app. FOREGROUND_REQUIRED requires explicit activation or raising and a fresh observation before retrying.
|
||||
|
||||
Paste delivers plain text or HTML through the application's enabled standard Paste menu command. Paste follows the target app's focused text input. Foreground requirements vary by application; for example, ChatGPT/Codex was observed to accept background Paste without reading clipboard data. PASTE_NOT_CONSUMED means the command was dispatched but no clipboard read was observed before the configured deadline. Inspect the target contents, explicitly activate the app and focus the intended input, then observe again before deciding whether to retry; a delayed paste may still run.
|
||||
Raw tools use { method, params }: include appHandle for app calls and windowId for window calls, and use snapshotId with elementId for element routes. Object options such as emit and disableDiffing are facade-only: emit is local output control and neither is a raw RPC parameter, so raw callers must follow the raw operation schema for admitted params. getAccessibilityTree and getWindow read through raw getState; getWindows maps to listWindows; getScreenshot maps to screenshot; cua.getState maps to listApps; every other named operation uses the same method name.
|
||||
|
||||
Maximum click holdMs: 5000 ms.
|
||||
@@ -332,14 +332,19 @@
|
||||
"type": "string",
|
||||
"minLength": 1
|
||||
},
|
||||
"windowId": {
|
||||
"type": "integer",
|
||||
"exclusiveMinimum": 0,
|
||||
"maximum": 4294967295
|
||||
},
|
||||
"snapshotId": {
|
||||
"type": "string",
|
||||
"minLength": 1
|
||||
},
|
||||
"elementId": {
|
||||
"description": "Opaque string defined by the platform provider, valid for at least the lifetime of one snapshot.",
|
||||
"type": "string",
|
||||
"minLength": 1
|
||||
"minLength": 1,
|
||||
"description": "Opaque string defined by the platform provider, valid for at least the lifetime of one snapshot."
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
@@ -370,14 +375,19 @@
|
||||
"type": "string",
|
||||
"minLength": 1
|
||||
},
|
||||
"windowId": {
|
||||
"type": "integer",
|
||||
"exclusiveMinimum": 0,
|
||||
"maximum": 4294967295
|
||||
},
|
||||
"snapshotId": {
|
||||
"type": "string",
|
||||
"minLength": 1
|
||||
},
|
||||
"elementId": {
|
||||
"description": "Opaque string defined by the platform provider, valid for at least the lifetime of one snapshot.",
|
||||
"type": "string",
|
||||
"minLength": 1
|
||||
"minLength": 1,
|
||||
"description": "Opaque string defined by the platform provider, valid for at least the lifetime of one snapshot."
|
||||
},
|
||||
"action": {
|
||||
"type": "string",
|
||||
@@ -416,14 +426,19 @@
|
||||
"type": "string",
|
||||
"minLength": 1
|
||||
},
|
||||
"windowId": {
|
||||
"type": "integer",
|
||||
"exclusiveMinimum": 0,
|
||||
"maximum": 4294967295
|
||||
},
|
||||
"snapshotId": {
|
||||
"type": "string",
|
||||
"minLength": 1
|
||||
},
|
||||
"elementId": {
|
||||
"description": "Opaque string defined by the platform provider, valid for at least the lifetime of one snapshot.",
|
||||
"type": "string",
|
||||
"minLength": 1
|
||||
"minLength": 1,
|
||||
"description": "Opaque string defined by the platform provider, valid for at least the lifetime of one snapshot."
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
@@ -445,7 +460,7 @@
|
||||
"properties": {
|
||||
"method": {
|
||||
"type": "string",
|
||||
"const": "hover"
|
||||
"const": "mouseMove"
|
||||
},
|
||||
"params": {
|
||||
"anyOf": [
|
||||
@@ -456,14 +471,19 @@
|
||||
"type": "string",
|
||||
"minLength": 1
|
||||
},
|
||||
"windowId": {
|
||||
"type": "integer",
|
||||
"exclusiveMinimum": 0,
|
||||
"maximum": 4294967295
|
||||
},
|
||||
"snapshotId": {
|
||||
"type": "string",
|
||||
"minLength": 1
|
||||
},
|
||||
"elementId": {
|
||||
"description": "Opaque string defined by the platform provider, valid for at least the lifetime of one snapshot.",
|
||||
"type": "string",
|
||||
"minLength": 1
|
||||
"minLength": 1,
|
||||
"description": "Opaque string defined by the platform provider, valid for at least the lifetime of one snapshot."
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
@@ -577,14 +597,19 @@
|
||||
"type": "string",
|
||||
"minLength": 1
|
||||
},
|
||||
"windowId": {
|
||||
"type": "integer",
|
||||
"exclusiveMinimum": 0,
|
||||
"maximum": 4294967295
|
||||
},
|
||||
"snapshotId": {
|
||||
"type": "string",
|
||||
"minLength": 1
|
||||
},
|
||||
"elementId": {
|
||||
"description": "Opaque string defined by the platform provider, valid for at least the lifetime of one snapshot.",
|
||||
"type": "string",
|
||||
"minLength": 1
|
||||
"minLength": 1,
|
||||
"description": "Opaque string defined by the platform provider, valid for at least the lifetime of one snapshot."
|
||||
},
|
||||
"mouseButton": {
|
||||
"type": "string",
|
||||
@@ -813,14 +838,19 @@
|
||||
"type": "string",
|
||||
"minLength": 1
|
||||
},
|
||||
"windowId": {
|
||||
"type": "integer",
|
||||
"exclusiveMinimum": 0,
|
||||
"maximum": 4294967295
|
||||
},
|
||||
"snapshotId": {
|
||||
"type": "string",
|
||||
"minLength": 1
|
||||
},
|
||||
"elementId": {
|
||||
"description": "Opaque string defined by the platform provider, valid for at least the lifetime of one snapshot.",
|
||||
"type": "string",
|
||||
"minLength": 1
|
||||
"minLength": 1,
|
||||
"description": "Opaque string defined by the platform provider, valid for at least the lifetime of one snapshot."
|
||||
},
|
||||
"value": {
|
||||
"$ref": "#/$defs/__schema0"
|
||||
@@ -855,14 +885,19 @@
|
||||
"type": "string",
|
||||
"minLength": 1
|
||||
},
|
||||
"windowId": {
|
||||
"type": "integer",
|
||||
"exclusiveMinimum": 0,
|
||||
"maximum": 4294967295
|
||||
},
|
||||
"snapshotId": {
|
||||
"type": "string",
|
||||
"minLength": 1
|
||||
},
|
||||
"elementId": {
|
||||
"description": "Opaque string defined by the platform provider, valid for at least the lifetime of one snapshot.",
|
||||
"type": "string",
|
||||
"minLength": 1
|
||||
"minLength": 1,
|
||||
"description": "Opaque string defined by the platform provider, valid for at least the lifetime of one snapshot."
|
||||
},
|
||||
"text": {
|
||||
"type": "string",
|
||||
@@ -898,49 +933,6 @@
|
||||
],
|
||||
"additionalProperties": false
|
||||
},
|
||||
{
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"method": {
|
||||
"type": "string",
|
||||
"const": "performSecondaryAction"
|
||||
},
|
||||
"params": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"appHandle": {
|
||||
"type": "string",
|
||||
"minLength": 1
|
||||
},
|
||||
"snapshotId": {
|
||||
"type": "string",
|
||||
"minLength": 1
|
||||
},
|
||||
"elementId": {
|
||||
"description": "Opaque string defined by the platform provider, valid for at least the lifetime of one snapshot.",
|
||||
"type": "string",
|
||||
"minLength": 1
|
||||
},
|
||||
"action": {
|
||||
"type": "string",
|
||||
"minLength": 1
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"appHandle",
|
||||
"snapshotId",
|
||||
"elementId",
|
||||
"action"
|
||||
],
|
||||
"additionalProperties": false
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"method",
|
||||
"params"
|
||||
],
|
||||
"additionalProperties": false
|
||||
},
|
||||
{
|
||||
"type": "object",
|
||||
"properties": {
|
||||
@@ -955,6 +947,11 @@
|
||||
"type": "string",
|
||||
"minLength": 1
|
||||
},
|
||||
"windowId": {
|
||||
"type": "integer",
|
||||
"exclusiveMinimum": 0,
|
||||
"maximum": 4294967295
|
||||
},
|
||||
"text": {
|
||||
"type": "string"
|
||||
}
|
||||
@@ -986,6 +983,11 @@
|
||||
"type": "string",
|
||||
"minLength": 1
|
||||
},
|
||||
"windowId": {
|
||||
"type": "integer",
|
||||
"exclusiveMinimum": 0,
|
||||
"maximum": 4294967295
|
||||
},
|
||||
"text": {
|
||||
"type": "string"
|
||||
},
|
||||
@@ -1024,6 +1026,11 @@
|
||||
"type": "string",
|
||||
"minLength": 1
|
||||
},
|
||||
"windowId": {
|
||||
"type": "integer",
|
||||
"exclusiveMinimum": 0,
|
||||
"maximum": 4294967295
|
||||
},
|
||||
"key": {
|
||||
"type": "string",
|
||||
"minLength": 1
|
||||
@@ -1031,14 +1038,7 @@
|
||||
"modifiers": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "string",
|
||||
"enum": [
|
||||
"command",
|
||||
"control",
|
||||
"option",
|
||||
"shift",
|
||||
"fn"
|
||||
]
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
},
|
||||
@@ -1055,6 +1055,51 @@
|
||||
],
|
||||
"additionalProperties": false
|
||||
},
|
||||
{
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"method": {
|
||||
"type": "string",
|
||||
"const": "sendKeyEvent"
|
||||
},
|
||||
"params": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"appHandle": {
|
||||
"type": "string",
|
||||
"minLength": 1
|
||||
},
|
||||
"windowId": {
|
||||
"type": "integer",
|
||||
"exclusiveMinimum": 0,
|
||||
"maximum": 4294967295
|
||||
},
|
||||
"key": {
|
||||
"type": "string",
|
||||
"minLength": 1
|
||||
},
|
||||
"direction": {
|
||||
"type": "string",
|
||||
"enum": [
|
||||
"down",
|
||||
"up"
|
||||
]
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"appHandle",
|
||||
"key",
|
||||
"direction"
|
||||
],
|
||||
"additionalProperties": false
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"method",
|
||||
"params"
|
||||
],
|
||||
"additionalProperties": false
|
||||
},
|
||||
{
|
||||
"type": "object",
|
||||
"properties": {
|
||||
|
||||
@@ -15,9 +15,9 @@
|
||||
{"type":"assistant/message","data":{"turn":1,"step":1,"message":{"role":"assistant","content":[{"type":"tool-call","id":"cua-ptc-native-result","name":"run_code","arguments":"{\"code\":\"const { Buffer } = await import('node:buffer');\\nconst app = await tools.cua({ method: 'getApp', params: { selector: 'test.fixture' } });\\nconst state = await tools.cua({ method: 'getState', params: { appHandle: app.appHandle, windowId: 7 } });\\nconst shot = await tools.cua({ method: 'screenshot', params: { appHandle: app.appHandle, windowId: 7 } });\\nconst bytes = Buffer.from(shot.data, 'base64');\\nreturn { appHandle: app.appHandle, hasIncrement: state.tree.includes('Increment'), pngBytes: bytes.length, signature: bytes.subarray(0, 8).toString('hex'), imageWidth: shot.attachment.width, imageHeight: shot.attachment.height };\",\"description\":\"Read native CUA values and receive one screenshot\"}"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:4}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1791864698727,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}},{"type":"chunk","time":1791864698727,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"cua-ptc-native-result","name":"run_code","arguments":"{\"code\":\"const { Buffer } = await import('node:buffer');\\nconst app = await tools.cua({ method: 'getApp', params: { selector: 'test.fixture' } });\\nconst state = await tools.cua({ method: 'getState', params: { appHandle: app.appHandle, windowId: 7 } });\\nconst shot = await tools.cua({ method: 'screenshot', params: { appHandle: app.appHandle, windowId: 7 } });\\nconst bytes = Buffer.from(shot.data, 'base64');\\nreturn { appHandle: app.appHandle, hasIncrement: state.tree.includes('Increment'), pngBytes: bytes.length, signature: bytes.subarray(0, 8).toString('hex'), imageWidth: shot.attachment.width, imageHeight: shot.attachment.height };\",\"description\":\"Read native CUA values and receive one screenshot\"}"}}},{"type":"chunk","time":1791864698727,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1791864698727,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}]},"surfaceOp":"append"}
|
||||
{"type":"tool/call","data":{"turn":1,"step":1,"callId":"cua-ptc-native-result","name":"run_code","arguments":"{\"code\":\"const { Buffer } = await import('node:buffer');\\nconst app = await tools.cua({ method: 'getApp', params: { selector: 'test.fixture' } });\\nconst state = await tools.cua({ method: 'getState', params: { appHandle: app.appHandle, windowId: 7 } });\\nconst shot = await tools.cua({ method: 'screenshot', params: { appHandle: app.appHandle, windowId: 7 } });\\nconst bytes = Buffer.from(shot.data, 'base64');\\nreturn { appHandle: app.appHandle, hasIncrement: state.tree.includes('Increment'), pngBytes: bytes.length, signature: bytes.subarray(0, 8).toString('hex'), imageWidth: shot.attachment.width, imageHeight: shot.attachment.height };\",\"description\":\"Read native CUA values and receive one screenshot\"}"}}
|
||||
{"type":"tool/ptc-dispatch-start","data":{"rootCallId":"cua-ptc-native-result","parentCallId":"cua-ptc-native-result","subCallId":"cua-ptc-native-result:ptc:1","name":"cua","arguments":{"method":"getApp","params":{"selector":"test.fixture"}}}}
|
||||
{"type":"tool/ptc-dispatch","data":{"rootCallId":"cua-ptc-native-result","parentCallId":"cua-ptc-native-result","subCallId":"cua-ptc-native-result:ptc:1","name":"cua","arguments":{"method":"getApp","params":{"selector":"test.fixture"}},"isError":false,"content":[{"type":"text","text":"{\"appHandle\":\"fixture-app\",\"pid\":9,\"name\":\"Fixture\",\"bundleId\":\"test.fixture\"}"}]}}
|
||||
{"type":"tool/ptc-dispatch","data":{"rootCallId":"cua-ptc-native-result","parentCallId":"cua-ptc-native-result","subCallId":"cua-ptc-native-result:ptc:1","name":"cua","arguments":{"method":"getApp","params":{"selector":"test.fixture"}},"isError":false,"content":[{"type":"text","text":"{\"appHandle\":\"fixture-app\",\"name\":\"Fixture\",\"bundleId\":\"test.fixture\",\"running\":true}"}]}}
|
||||
{"type":"tool/ptc-dispatch-start","data":{"rootCallId":"cua-ptc-native-result","parentCallId":"cua-ptc-native-result","subCallId":"cua-ptc-native-result:ptc:2","name":"cua","arguments":{"method":"getState","params":{"appHandle":"fixture-app","windowId":7}}}}
|
||||
{"type":"tool/ptc-dispatch","data":{"rootCallId":"cua-ptc-native-result","parentCallId":"cua-ptc-native-result","subCallId":"cua-ptc-native-result:ptc:2","name":"cua","arguments":{"method":"getState","params":{"appHandle":"fixture-app","windowId":7}},"isError":false,"content":[{"type":"text","text":"{\"appHandle\":\"fixture-app\",\"pid\":9,\"windowId\":7,\"snapshotId\":\"fixture-snapshot-1\",\"tree\":\"AX scope=window offsets=UTF-16\\n- [0] AXButton \\\"Increment\\\"\\n- [1] AXTextField valueWritable=true selectionWritable=true value=\\\"\\\" placeholder=\\\"Fixture text\\\"\\n- AXStaticText \\\"Count: 0\\\"\\n- [2] AXMenuBarItem \\\"File\\\"\\nMenu contents omitted; request includeMenus=true to expand.\",\"windows\":[{\"windowId\":7,\"pid\":9,\"appName\":\"Fixture\",\"title\":\"CUA fixture\",\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"isOnScreen\":true,\"onCurrentSpace\":true}],\"truncated\":false}"}]}}
|
||||
{"type":"tool/ptc-dispatch","data":{"rootCallId":"cua-ptc-native-result","parentCallId":"cua-ptc-native-result","subCallId":"cua-ptc-native-result:ptc:2","name":"cua","arguments":{"method":"getState","params":{"appHandle":"fixture-app","windowId":7}},"isError":false,"content":[{"type":"text","text":"{\"appHandle\":\"fixture-app\",\"windowId\":7,\"snapshotId\":\"fixture-snapshot-1\",\"tree\":\"AX scope=window offsets=UTF-16\\n- [0] AXButton \\\"Increment\\\"\\n- [1] AXTextField valueWritable=true selectionWritable=true value=\\\"\\\" placeholder=\\\"Fixture text\\\"\\n- AXStaticText \\\"Count: 0\\\"\\n- [2] AXMenuBarItem \\\"File\\\"\\nMenu contents omitted; request includeMenus=true to expand.\",\"windows\":[{\"windowId\":7,\"appName\":\"Fixture\",\"title\":\"CUA fixture\",\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"isOnScreen\":true,\"onCurrentSpace\":true}],\"truncated\":false}"}]}}
|
||||
{"type":"tool/ptc-dispatch-start","data":{"rootCallId":"cua-ptc-native-result","parentCallId":"cua-ptc-native-result","subCallId":"cua-ptc-native-result:ptc:3","name":"cua","arguments":{"method":"screenshot","params":{"appHandle":"fixture-app","windowId":7}}}}
|
||||
{"type":"tool/ptc-dispatch","data":{"rootCallId":"cua-ptc-native-result","parentCallId":"cua-ptc-native-result","subCallId":"cua-ptc-native-result:ptc:3","name":"cua","arguments":{"method":"screenshot","params":{"appHandle":"fixture-app","windowId":7}},"isError":false,"content":[{"type":"text","text":"{\"pixelWidth\":1,\"pixelHeight\":1,\"windowId\":7,\"bounds\":{\"x\":40,\"y\":60,\"width\":320,\"height\":200},\"coordinateSpace\":\"screenshot-pixels\"}"},{"type":"image","attachment":{"attachmentId":"sha256:999f1d1527ee7e79266f16add5430fff76b1225d742464a5b1ff1f02971bb8ee","mediaType":"image/png","width":1,"height":1,"bytes":90}}]}}
|
||||
{"type":"tool/result","data":{"turn":1,"step":1,"message":{"role":"tool","source":{"kind":"tool","callId":"cua-ptc-native-result"},"toolCallId":"cua-ptc-native-result","content":[{"type":"text","text":"{\n \"appHandle\": \"fixture-app\",\n \"hasIncrement\": true,\n \"pngBytes\": 90,\n \"signature\": \"89504e470d0a1a0a\",\n \"imageWidth\": 1,\n \"imageHeight\": 1\n}"}],"isError":false,"id":"{{message:5}}"}},"sourceEventSeqs":[14],"surfaceOp":"append"}
|
||||
|
||||
@@ -35,54 +35,54 @@ Start independent delegations with `subagent` or `subagent_fork` together in one
|
||||
|
||||
Bash, formatters, code generators, and scripts can overwrite concurrent edits without conflict detection; coordinate their changes explicitly. The agent integrating the work must review the final diff and run relevant tests.
|
||||
|
||||
Use cua with method getCapabilities and params {} to inspect permissions and sessionLocked. If sessionLocked is true or an operation reports SESSION_LOCKED, stop UI attempts and ask the user to unlock the desktop session. A null sessionLocked means no usable console session was verified. After unlocking, read fresh state; screenshots from a locked desktop are not current observations. Use listApps or cua.getState to discover running applications, and findApps(query) to search installed applications by name or application path. getApp requests user consent before launching a stopped application in the background, or before observations and input on a running one. If access is denied or revoked, stop using that application until the user explicitly permits another authorization request. Never switch tools to bypass a refusal. Keep the returned appHandle; every app operation requires it. listWindows returns window IDs and bounds in the provider’s native units. Raw getState returns AX text and snapshotId. Menu-bar entries are shown by default; includeMenus=true expands their contents. details=true adds identifiers, global native bounds and complete action names. Both options also apply to getAXState. Each read replaces the indices for that app/window scope. Use its element indices only with that snapshotId. Raw screenshot and coordinate operations require a windowId. Coordinates are pixels in the latest returned screenshot, measured from its top-left. Set relative:true for integer thousandths from 0 to 999 instead. The Host converts both modes to native window coordinates; AX detail bounds remain global native coordinates. A verified window can expose no AX elements. Its empty snapshot permits screenshots and coordinate input; element-index actions require an element from a fresh AX observation.
|
||||
macOS: accessibility tree operations are available at app or window level. Simulated keyboard input is available at app level. Simulated mouse input is available at window level. getApp also accepts bundle IDs and .app paths, and findApps searches /Applications, /System/Applications, and ~/Applications. Use the Computer Use permission page to grant the helper application Accessibility and Screen Recording permissions in System Settings. Native window and AX bounds use macOS points, while delivered screenshots carry their own pixel dimensions. show does not switch Spaces, and Stage Manager can present reduced window geometry; explain any required foreground step before activate or raise.
|
||||
|
||||
Initialize the object API once in persistent PTC. Reuse its variables in later calls; put temporary declarations inside { ... } blocks to avoid redeclaration errors. All object methods are async. getAXState returns text, not a structured state object. Its text separates values and placeholders, and reports subroles, help, URLs, disabled/focused/selected/expanded state and UTF-16 selection ranges when available. Writability is true, false or unknown on relevant controls; omitted state fields do not assert false. selection=unavailable means the reported range exceeds the observed text. Compact output omits ordinary Press/Pick (click), ShowMenu (right click) and ScrollToVisible actions; details=true lists every available action. Use the displayed action name with performSecondaryAction; ambiguous aliases require the complete name from details=true. Truncation reports the reached provider limit. Narrow the window or increase that provider limit before reading again.
|
||||
Use cua to observe and operate native applications. Always use cua.getCapabilities() to confirm whether a specific API is available on the current platform; its reported capabilities take precedence over availability statements in these instructions. All methods are async; observation options default to { emit: true }, and emit: false suppresses sink output only, leaving returned values unchanged. Actions report dispatch evidence, not verified completion. If sessionLocked is true or an operation reports SESSION_LOCKED, stop UI actions and ask the user to unlock the desktop session; null or unknown means verify a usable unlocked desktop session first. cua.getApp requests user consent; if access is denied or revoked, stop using that application until the user explicitly permits another request, and never bypass a refusal or switch tools to avoid it. Never replay uncertain input without a fresh observation.
|
||||
|
||||
Use app methods directly for single-window tasks; getApp already reads initial AX state. Call app.getWindows() and await app.getWindow(windowId) only to target a particular window or resolve ambiguous window selection. App screenshots and coordinate actions use the latest AX observation: its focused on-screen, nonzero-size window, otherwise the sole such window. They do not refresh focus automatically. Read app.getAXState() to follow changed focus.
|
||||
In persistent PTC, use the host-provided bridge initialization once and retain the returned objects; cua comes from that bridge and is not a preinstalled global. Use void await for observations that emit, and never stringify screenshot image bytes.
|
||||
|
||||
CuaScreenshot contains { bytes: Uint8Array, mimeType: "image/png" | "image/jpeg" | "image/webp" | "image/gif", pixelWidth: number, pixelHeight: number, windowId: number }. Use mimeType to choose a file extension or declare an image; do not assume PNG.
|
||||
Signatures show resolved return values after await. ActionResult = { dispatched: boolean, foregroundChanged?: boolean | null }; foregroundChanged is omitted when not sampled and null when the sample is unavailable. JsonValue = string | number | boolean | null | JsonValue[] | { [key: string]: JsonValue }; numbers must be finite. App is the application object with readonly app.appHandle: string (opaque) and the methods listed below for app level. Window is the window object with readonly window.windowId: number and the methods listed below for window level. Neither object carries pid.
|
||||
|
||||
Object API signatures (options default to { emit: true }):
|
||||
Setup and discovery:
|
||||
- cua.getCapabilities(options?: { emit?: boolean }) -> { platform: string, permissions: { accessibility: boolean, screenRecording: boolean }, appLevelAccessibility: boolean, keyboard: ('app' | 'window')[], sessionLocked: boolean | null, methods: string[], coordinateSpace: 'screenshot-pixels', browser: false } — inspect reported permissions, sessionLocked, methods, and keyboard scopes without requesting permissions.
|
||||
- cua.listApps(options?: { emit?: boolean }) -> { id: string, displayName: string, isRunning: true, pid: number }[] — enumerate running applications.
|
||||
- cua.findApps(query: string, options?: { emit?: boolean }) -> ({ id: string, displayName: string, path: string } & ({ isRunning: true, pid: number } | { isRunning: false }))[] — search installed applications, including stopped matches; only running matches carry pid.
|
||||
- cua.getState(options?: { emit?: boolean }) -> { apps: { id: string, displayName: string, isRunning: true, pid: number }[], browsers: [] } — summarize running applications; this is not an accessibility observation.
|
||||
- cua.getApp(selector: string) -> App — select an application by name, bundle ID, absolute path, or running PID string, obtain consent, launch in the background if needed, and read initial AX state silently; it does not activate the app.
|
||||
- app.getWindows(options?: { emit?: boolean }) -> { windowId: number, appName: string, title: string, bounds: { x: number, y: number, width: number, height: number }, isOnScreen: boolean, onCurrentSpace: boolean | null }[] — enumerate windows with global native bounds; null onCurrentSpace means unknown Space membership.
|
||||
- app.getWindow(windowId: number) -> Window — select one window and read its initial AX state silently; do this before any window screenshot or coordinate input.
|
||||
- app.activate() -> ActionResult — request explicit foreground activation; use only when foreground access is required.
|
||||
- app.show() -> { dispatched: boolean, windowId: number | null, foregroundChanged: boolean | null } — unhide the app and restore its selected, focused, main, or sole window without activation.
|
||||
- window.show() -> { dispatched: boolean, windowId: number | null, foregroundChanged: boolean | null } — unhide and restore that explicit window without activation.
|
||||
|
||||
- cua.getCapabilities(options?) -> { platform, permissions, sessionLocked, methods, coordinateSpace, browser }
|
||||
- cua.listApps(options?) -> Array<{ id, displayName, isRunning: true, pid }>
|
||||
- cua.findApps(query: string, options?) -> Array<{ id, displayName, path, isRunning, pid? }>
|
||||
- cua.getState(options?) -> { apps, browsers: [] }
|
||||
- cua.getApp(selector: string) -> App
|
||||
- app.activate() -> { dispatched: true, verified: false }
|
||||
- app.getWindows(options?) -> Array<{ windowId, pid, appName, title, bounds, isOnScreen, onCurrentSpace }>
|
||||
- app.getWindow(windowId: number) -> Window
|
||||
- app.typeText(text: string), app.pressKey(chord: string) -> action result
|
||||
- app.paste(text: string, options?: { format?: "text" | "html" }) -> action result
|
||||
## Accessibility tree
|
||||
Available at app or window level depending on the platform.
|
||||
|
||||
App and Window methods:
|
||||
Element ids are opaque strings from the latest successful tree; the object facade keeps one current snapshot per app across its repeated app and window objects, a new read replaces it, and a failed read invalidates the previous one. Repeated strings can appear for different elements, so do not reuse an old id blindly, and element operations do not fall back to pointer input or typing.
|
||||
- getAccessibilityTree(options?: { emit?: boolean, includeMenus?: boolean, details?: boolean, disableDiffing?: boolean }) -> string — read the full tree text; includeMenus expands menu contents and details adds identifiers, native bounds, and complete action names.
|
||||
- listActions(elementId: string) -> string[] — list the actions an element advertises.
|
||||
- invoke(elementId: string) -> ActionResult — invoke the element's default action; dispatched is false when it has none.
|
||||
- performAction(elementId: string, action: string, argument?: JsonValue) -> ActionResult — perform an observed action with an optional direct JSON argument.
|
||||
- setValue(elementId: string, value: JsonValue) -> ActionResult — write a control value.
|
||||
- selectText(elementId: string, text: string, options?: { prefix?: string, suffix?: string, selectionType?: 'text' | 'cursor_before' | 'cursor_after' }) -> ActionResult — select one literal occurrence or place the cursor beside it, using adjacent context to disambiguate.
|
||||
|
||||
- show() -> { dispatched: true, verified: false, windowId: number | null, foregroundChanged: boolean | null }
|
||||
- getAXState(options?: { emit?: boolean, disableDiffing?: boolean, includeMenus?: boolean, details?: boolean }) -> string
|
||||
- getScreenshot(options?: { emit?: boolean }) -> CuaScreenshot
|
||||
- getAXStateAndScreenshot(options?) -> { state: string, screenshot: CuaScreenshot }
|
||||
- click(target: number | [x, y], options?: { mouseButton?: "left" | "right" | "middle", clickCount?: 1 | 2, holdMs?: number, modifiers?: Array<"command" | "control" | "option" | "shift" | "fn">, relative?: boolean }) -> action result
|
||||
- hover(target: number | [x, y], options?: { relative?: boolean }) -> { dispatched, verified, hoverRoute, systemPointerMoved }
|
||||
- setValue(elementIndex: number, value: string) -> action result
|
||||
- selectText(elementIndex: number, text: string, options?: { prefix?: string, suffix?: string, selectionType?: "text" | "cursor_before" | "cursor_after" }) -> action result
|
||||
- performSecondaryAction(elementIndex: number, action: string) -> action result
|
||||
- scroll({ x, y, deltaLinesX, deltaLinesY, relative? }) -> action result
|
||||
- drag(from: [x, y], to: [x, y], options?: { mouseButton?: "left" | "right" | "middle", modifiers?: Array<"command" | "control" | "option" | "shift" | "fn">, relative?: boolean }) -> action result
|
||||
## Simulated mouse
|
||||
Use an explicit window for screenshots and pointer input.
|
||||
- getScreenshot(options?: { emit?: boolean }) -> { bytes: Uint8Array, mimeType: 'image/png' | 'image/jpeg' | 'image/webp' | 'image/gif', pixelWidth: number, pixelHeight: number, windowId: number } — capture the selected window as one encoded image; bytes is a JavaScript byte array, and mimeType identifies its encoding, so do not assume PNG.
|
||||
- mouseMove(target: string | [number, number], options?: { relative?: boolean }) -> { dispatched: boolean, foregroundChanged?: boolean | null, hoverRoute: 'accessibility' | 'system-pointer', systemPointerMoved: boolean } — reveal AX alternate UI or move the system pointer without pressing; inspect hoverRoute and systemPointerMoved, and expect no automatic pointer restoration.
|
||||
- click(target: string | [number, number], options?: { mouseButton?: 'left' | 'right' | 'middle', clickCount?: 1 | 2, modifiers?: ('command' | 'control' | 'option' | 'shift' | 'fn')[], holdMs?: number, relative?: boolean }) -> ActionResult — click an element id or point, or perform a left-button double-click; holdMs controls the press duration of each press.
|
||||
- scroll(options: { x: number, y: number, deltaLinesX: number, deltaLinesY: number, relative?: boolean }) -> ActionResult — scroll at a point; deltas are integers from -10 to 10 with at least one nonzero, and positive values reveal content above or left.
|
||||
- drag(from: [number, number], to: [number, number], options?: { mouseButton?: 'left' | 'right' | 'middle', modifiers?: ('command' | 'control' | 'option' | 'shift' | 'fn')[], relative?: boolean }) -> ActionResult — drag between two points of one window.
|
||||
|
||||
Raw operation tools use { method, params }, including params: {} for parameterless methods. getApp selects without activating the app. Keep applications in the background for observation and actions that the provider supports without foreground access. Do not activate or raise an app during initialization, before each action, or merely because another app is foreground. Prefer advertised AX actions. If a window is hidden or minimized, try app.show() or window.show() and read fresh state. show requests unhide and unminimize without activation or raising; it chooses the explicit, focused, main or sole AX window. A null windowId means no window was selected for restoration. foregroundChanged compares the front app and top window before and after the request; null means unavailable, and delayed focus changes remain possible. If restoration is unsupported or the window remains unusable, explain the needed foreground step, use app.activate() or an advertised raise action, and read fresh state. Completing the task takes priority when foreground access is necessary. Avoid repeated activation when the user switches away; continue background-capable steps and report what requires foreground access. Applications may still change focus in response to input.
|
||||
A string target is an element id; coordinate targets are [x, y] and are pixels in the latest returned screenshot by default. With relative: true they are integer thousandths of the window width and height from 0 to 999, based on the current AX observation or screenshot. Left clicks support one or two presses; right and middle clicks support one. FOREGROUND_REQUIRED means explain the need, then explicitly activate or raise and observe again; activation is never automatic. Read fresh state after actions, and treat WINDOW_CHANGED, STALE_SNAPSHOT, and SCREENSHOT_REQUIRED as requiring the stated fresh observation or capture before any retry.
|
||||
|
||||
Await operations sequentially. After input, read fresh state to verify the effect. A successful action means dispatched, not verified completion. Cancellation may leave partial effects; never retry uncertain input without observing. STALE_APP requires getApp again; STALE_SNAPSHOT requires getState again. Absolute coordinate input requires a fresh screenshot of that window. Relative input can use a successful AX observation or screenshot. SCREENSHOT_REQUIRED means capture a fresh screenshot before using pixel coordinates. For coordinate input, targets must be [x, y], for example click([650, 400], {relative:true}). A bare number such as click(650) is always an AX element index. The relative option applies only to coordinate targets, not AX indices. Window resizing since the latest observation rejects coordinate input; movement alone preserves window-local coordinates. WINDOW_CHANGED requires fresh state before any retry; use an advertised AX action or explicitly restore the window before screenshot or coordinate input, then observe again. PERMISSION_DENIED requires the user to address the reported system access restriction. After granting access, observe fresh state before continuing; do not automatically replay prior input.
|
||||
## Simulated keyboard
|
||||
Available at app or window level depending on the platform.
|
||||
- typeText(text: string) -> ActionResult — type literal text at the app's current focus; line endings send Return and tabs send Tab.
|
||||
- paste(text: string, options?: { format?: 'text' | 'html' }) -> ActionResult — deliver text or HTML through the app's Paste command, restoring prior clipboard contents unless another writer changes them; a Paste failure is surfaced without substituting typeText, and PASTE_NOT_CONSUMED means inspect before any retry.
|
||||
- pressKey(key: string, modifiers?: string[]) -> ActionResult — press and release one complete keystroke. For example pressKey('Tab', ['shift']) or pressKey('+'). Key and modifier names are platform-defined strings. Pass key and modifiers separately; do not pack them into a chord string.
|
||||
|
||||
Hover reveals alternate UI through a provider-supported accessibility action or moves the system pointer without pressing a button. Wait for the application's hover delay and observe before clicking; subsequent observations do not restore the pointer. Use successive hover calls for intermediate points. Other input and application changes can interrupt the interaction. click accepts optional holdMs, a nonnegative integer in milliseconds for each press, including double-click presses. Zero releases without an additional hold. The configured maximum is stated below. Explicit holdMs uses pointer input even for AX targets; omission preserves ordinary clicks. The operation deadline applies, and cancellation or timeout releases the button.
|
||||
|
||||
Use selectText only on controls advertising selectionWritable. It selects a unique literal occurrence, or places the cursor before/after it; prefix and suffix must be immediately adjacent. AMBIGUOUS_TEXT requires more context; TEXT_NOT_FOUND or SELECTION_CHANGED requires a fresh observation before deciding on input. Left clicks support single/double clicks; right and middle clicks support single clicks. Modified clicks use pointer input. Keypad names such as KP_0 and KP_Enter retain their physical key identity. Window objects expose observation and pointer/AX actions; keyboard input remains app-scoped and follows current focus. Scroll deltas are integer lines from -10 to 10; positive values reveal content above or left. Paste delivers plain text or HTML through the system clipboard and the application's paste action. Original clipboard formats are restored if the clipboard still belongs to this paste; a later copy is preserved. Unsupported Paste commands fail without typing fallback. CLIPBOARD_CHANGED means another writer intervened; observe before deciding on further input. Browser applications can be controlled with getApp like other native apps; browser tabs are unavailable. Installed-app search covers the locations documented by the platform provider; use an absolute application path for another location. Launch does not restore a window or document, and cancellation may leave an application already started. The execution host must install the object API and own variable persistence; this plugin does not preload a cua global. In persistent PTC, use the host-provided initialization snippet below once. Its installation-owned file URL works independently of the project directory. Retain app objects between calls. Observation methods emit by default and also return their value. Use void await app.getAXState() to avoid a duplicate REPL completion, or pass emit:false and print the returned text once. End screenshot calls with void await app.getScreenshot() or void await app.getAXStateAndScreenshot() so image bytes are not serialized as the program completion value.
|
||||
|
||||
macOS: getApp also accepts bundle IDs and .app paths. findApps searches /Applications, /System/Applications and ~/Applications. Accessibility and Screen Recording permissions belong to the responsible launching application. Use the Computer Use permission page to request access, open System Settings, and check again. Native window and AX bounds use macOS points. show does not switch Spaces. Stage Manager can present reduced window geometry; explain any required foreground step before activate or raise.
|
||||
|
||||
Hover first uses advertised AX alternate UI without moving the system cursor, preserving revealed descendants until the pointer target leaves their region. The system-pointer fallback requires the target point to be unobscured in the foreground app. FOREGROUND_REQUIRED requires explicit activation or raising and a fresh observation before retrying.
|
||||
|
||||
Paste delivers plain text or HTML through the application's enabled standard Paste menu command. Paste follows the target app's focused text input. Foreground requirements vary by application; for example, ChatGPT/Codex was observed to accept background Paste without reading clipboard data. PASTE_NOT_CONSUMED means the command was dispatched but no clipboard read was observed before the configured deadline. Inspect the target contents, explicitly activate the app and focus the intended input, then observe again before deciding whether to retry; a delayed paste may still run.
|
||||
Raw tools use { method, params }: include appHandle for app calls and windowId for window calls, and use snapshotId with elementId for element routes. Object options such as emit and disableDiffing are facade-only: emit is local output control and neither is a raw RPC parameter, so raw callers must follow the raw operation schema for admitted params. getAccessibilityTree and getWindow read through raw getState; getWindows maps to listWindows; getScreenshot maps to screenshot; cua.getState maps to listApps; every other named operation uses the same method name.
|
||||
|
||||
Maximum click holdMs: 5000 ms.
|
||||
|
||||
|
||||
@@ -35,48 +35,54 @@ Start independent delegations with `subagent` or `subagent_fork` together in one
|
||||
|
||||
Bash, formatters, code generators, and scripts can overwrite concurrent edits without conflict detection; coordinate their changes explicitly. The agent integrating the work must review the final diff and run relevant tests.
|
||||
|
||||
Use cua with method getCapabilities and params {} to inspect permissions and sessionLocked. If sessionLocked is true or an operation reports SESSION_LOCKED, stop UI attempts and ask the user to unlock macOS. A null sessionLocked means no usable console session was verified. After unlocking, read fresh state; screenshots from a locked desktop are not current observations. Use listApps or cua.getState to discover running applications, and findApps(query) to search installed applications by name, bundle ID or absolute path. getApp requests user consent before launching a stopped application in the background, or before observations and input on a running one. If access is denied or revoked, stop using that application until the user explicitly permits another authorization request. Never switch tools to bypass a refusal. Keep the returned appHandle; every app operation requires it. listWindows returns window IDs and point bounds. Raw getState returns AX text and snapshotId. Menu-bar entries are shown by default; includeMenus=true expands their contents. details=true adds identifiers, global-point bounds and complete action names. Both options also apply to getAXState. Each read replaces the indices for that app/window scope. Use its element indices only with that snapshotId. Raw screenshot and coordinate operations require a windowId. Coordinates are pixels in the latest returned screenshot, measured from its top-left. Set relative:true for integer thousandths from 0 to 999 instead. The Host converts both modes to native window points; AX detail bounds remain global macOS points. A verified window can expose no AX elements. Its empty snapshot permits screenshots and coordinate input; element-index actions require an element from a fresh AX observation.
|
||||
macOS: accessibility tree operations are available at app or window level. Simulated keyboard input is available at app level. Simulated mouse input is available at window level. getApp also accepts bundle IDs and .app paths, and findApps searches /Applications, /System/Applications, and ~/Applications. Use the Computer Use permission page to grant the helper application Accessibility and Screen Recording permissions in System Settings. Native window and AX bounds use macOS points, while delivered screenshots carry their own pixel dimensions. show does not switch Spaces, and Stage Manager can present reduced window geometry; explain any required foreground step before activate or raise.
|
||||
|
||||
Initialize the object API once in persistent PTC. Reuse its variables in later calls; put temporary declarations inside { ... } blocks to avoid redeclaration errors. All object methods are async. getAXState returns text, not a structured state object. Its text separates values and placeholders, and reports subroles, help, URLs, disabled/focused/selected/expanded state and UTF-16 selection ranges when available. Writability is true, false or unknown on relevant controls; omitted state fields do not assert false. selection=unavailable means the reported range exceeds the observed text. Compact output omits ordinary Press/Pick (click), ShowMenu (right click) and ScrollToVisible actions; details=true lists every available action. Use the displayed action name with performSecondaryAction; ambiguous aliases require the complete name from details=true. Truncation reports the reached provider limit. Narrow the window or increase that provider limit before reading again.
|
||||
Use cua to observe and operate native applications. Always use cua.getCapabilities() to confirm whether a specific API is available on the current platform; its reported capabilities take precedence over availability statements in these instructions. All methods are async; observation options default to { emit: true }, and emit: false suppresses sink output only, leaving returned values unchanged. Actions report dispatch evidence, not verified completion. If sessionLocked is true or an operation reports SESSION_LOCKED, stop UI actions and ask the user to unlock the desktop session; null or unknown means verify a usable unlocked desktop session first. cua.getApp requests user consent; if access is denied or revoked, stop using that application until the user explicitly permits another request, and never bypass a refusal or switch tools to avoid it. Never replay uncertain input without a fresh observation.
|
||||
|
||||
Use app methods directly for single-window tasks; getApp already reads initial AX state. Call app.getWindows() and await app.getWindow(windowId) only to target a particular window or resolve ambiguous window selection. App screenshots and coordinate actions use the latest AX observation: its focused on-screen, nonzero-size window, otherwise the sole such window. They do not refresh focus automatically. Read app.getAXState() to follow changed focus.
|
||||
In persistent PTC, use the host-provided bridge initialization once and retain the returned objects; cua comes from that bridge and is not a preinstalled global. Use void await for observations that emit, and never stringify screenshot image bytes.
|
||||
|
||||
CuaScreenshot contains { bytes: Uint8Array, mimeType: "image/png" | "image/jpeg" | "image/webp" | "image/gif", pixelWidth: number, pixelHeight: number, windowId: number }. Use mimeType to choose a file extension or declare an image; do not assume PNG.
|
||||
Signatures show resolved return values after await. ActionResult = { dispatched: boolean, foregroundChanged?: boolean | null }; foregroundChanged is omitted when not sampled and null when the sample is unavailable. JsonValue = string | number | boolean | null | JsonValue[] | { [key: string]: JsonValue }; numbers must be finite. App is the application object with readonly app.appHandle: string (opaque) and the methods listed below for app level. Window is the window object with readonly window.windowId: number and the methods listed below for window level. Neither object carries pid.
|
||||
|
||||
Object API signatures (options default to { emit: true }):
|
||||
Setup and discovery:
|
||||
- cua.getCapabilities(options?: { emit?: boolean }) -> { platform: string, permissions: { accessibility: boolean, screenRecording: boolean }, appLevelAccessibility: boolean, keyboard: ('app' | 'window')[], sessionLocked: boolean | null, methods: string[], coordinateSpace: 'screenshot-pixels', browser: false } — inspect reported permissions, sessionLocked, methods, and keyboard scopes without requesting permissions.
|
||||
- cua.listApps(options?: { emit?: boolean }) -> { id: string, displayName: string, isRunning: true, pid: number }[] — enumerate running applications.
|
||||
- cua.findApps(query: string, options?: { emit?: boolean }) -> ({ id: string, displayName: string, path: string } & ({ isRunning: true, pid: number } | { isRunning: false }))[] — search installed applications, including stopped matches; only running matches carry pid.
|
||||
- cua.getState(options?: { emit?: boolean }) -> { apps: { id: string, displayName: string, isRunning: true, pid: number }[], browsers: [] } — summarize running applications; this is not an accessibility observation.
|
||||
- cua.getApp(selector: string) -> App — select an application by name, bundle ID, absolute path, or running PID string, obtain consent, launch in the background if needed, and read initial AX state silently; it does not activate the app.
|
||||
- app.getWindows(options?: { emit?: boolean }) -> { windowId: number, appName: string, title: string, bounds: { x: number, y: number, width: number, height: number }, isOnScreen: boolean, onCurrentSpace: boolean | null }[] — enumerate windows with global native bounds; null onCurrentSpace means unknown Space membership.
|
||||
- app.getWindow(windowId: number) -> Window — select one window and read its initial AX state silently; do this before any window screenshot or coordinate input.
|
||||
- app.activate() -> ActionResult — request explicit foreground activation; use only when foreground access is required.
|
||||
- app.show() -> { dispatched: boolean, windowId: number | null, foregroundChanged: boolean | null } — unhide the app and restore its selected, focused, main, or sole window without activation.
|
||||
- window.show() -> { dispatched: boolean, windowId: number | null, foregroundChanged: boolean | null } — unhide and restore that explicit window without activation.
|
||||
|
||||
- cua.getCapabilities(options?) -> { platform, permissions, sessionLocked, methods, coordinateSpace, browser }
|
||||
- cua.listApps(options?) -> Array<{ id, displayName, isRunning: true, pid }>
|
||||
- cua.findApps(query: string, options?) -> Array<{ id, displayName, path, isRunning, pid? }>
|
||||
- cua.getState(options?) -> { apps, browsers: [] }
|
||||
- cua.getApp(selector: string) -> App
|
||||
- app.activate() -> { dispatched: true, verified: false }
|
||||
- app.getWindows(options?) -> Array<{ windowId, pid, appName, title, bounds, isOnScreen, onCurrentSpace }>
|
||||
- app.getWindow(windowId: number) -> Window
|
||||
- app.typeText(text: string), app.pressKey(chord: string) -> action result
|
||||
- app.paste(text: string, options?: { format?: "text" | "html" }) -> action result
|
||||
## Accessibility tree
|
||||
Available at app or window level depending on the platform.
|
||||
|
||||
App and Window methods:
|
||||
Element ids are opaque strings from the latest successful tree; the object facade keeps one current snapshot per app across its repeated app and window objects, a new read replaces it, and a failed read invalidates the previous one. Repeated strings can appear for different elements, so do not reuse an old id blindly, and element operations do not fall back to pointer input or typing.
|
||||
- getAccessibilityTree(options?: { emit?: boolean, includeMenus?: boolean, details?: boolean, disableDiffing?: boolean }) -> string — read the full tree text; includeMenus expands menu contents and details adds identifiers, native bounds, and complete action names.
|
||||
- listActions(elementId: string) -> string[] — list the actions an element advertises.
|
||||
- invoke(elementId: string) -> ActionResult — invoke the element's default action; dispatched is false when it has none.
|
||||
- performAction(elementId: string, action: string, argument?: JsonValue) -> ActionResult — perform an observed action with an optional direct JSON argument.
|
||||
- setValue(elementId: string, value: JsonValue) -> ActionResult — write a control value.
|
||||
- selectText(elementId: string, text: string, options?: { prefix?: string, suffix?: string, selectionType?: 'text' | 'cursor_before' | 'cursor_after' }) -> ActionResult — select one literal occurrence or place the cursor beside it, using adjacent context to disambiguate.
|
||||
|
||||
- show() -> { dispatched: true, verified: false, windowId: number | null, foregroundChanged: boolean | null }
|
||||
- getAXState(options?: { emit?: boolean, disableDiffing?: boolean, includeMenus?: boolean, details?: boolean }) -> string
|
||||
- getScreenshot(options?: { emit?: boolean }) -> CuaScreenshot
|
||||
- getAXStateAndScreenshot(options?) -> { state: string, screenshot: CuaScreenshot }
|
||||
- hover(target: number | [x, y], options?: { relative?: boolean }) -> { dispatched, verified, hoverRoute, systemPointerMoved }
|
||||
- click(target: number | [x, y], options?: { mouseButton?: "left" | "right" | "middle", clickCount?: 1 | 2, holdMs?: number, modifiers?: Array<"command" | "control" | "option" | "shift" | "fn">, relative?: boolean }) -> action result
|
||||
- setValue(elementIndex: number, value: string) -> action result
|
||||
- selectText(elementIndex: number, text: string, options?: { prefix?: string, suffix?: string, selectionType?: "text" | "cursor_before" | "cursor_after" }) -> action result
|
||||
- performSecondaryAction(elementIndex: number, action: string) -> action result
|
||||
- scroll({ x, y, deltaLinesX, deltaLinesY, relative? }) -> action result
|
||||
- drag(from: [x, y], to: [x, y], options?: { mouseButton?: "left" | "right" | "middle", modifiers?: Array<"command" | "control" | "option" | "shift" | "fn">, relative?: boolean }) -> action result
|
||||
## Simulated mouse
|
||||
Use an explicit window for screenshots and pointer input.
|
||||
- getScreenshot(options?: { emit?: boolean }) -> { bytes: Uint8Array, mimeType: 'image/png' | 'image/jpeg' | 'image/webp' | 'image/gif', pixelWidth: number, pixelHeight: number, windowId: number } — capture the selected window as one encoded image; bytes is a JavaScript byte array, and mimeType identifies its encoding, so do not assume PNG.
|
||||
- mouseMove(target: string | [number, number], options?: { relative?: boolean }) -> { dispatched: boolean, foregroundChanged?: boolean | null, hoverRoute: 'accessibility' | 'system-pointer', systemPointerMoved: boolean } — reveal AX alternate UI or move the system pointer without pressing; inspect hoverRoute and systemPointerMoved, and expect no automatic pointer restoration.
|
||||
- click(target: string | [number, number], options?: { mouseButton?: 'left' | 'right' | 'middle', clickCount?: 1 | 2, modifiers?: ('command' | 'control' | 'option' | 'shift' | 'fn')[], holdMs?: number, relative?: boolean }) -> ActionResult — click an element id or point, or perform a left-button double-click; holdMs controls the press duration of each press.
|
||||
- scroll(options: { x: number, y: number, deltaLinesX: number, deltaLinesY: number, relative?: boolean }) -> ActionResult — scroll at a point; deltas are integers from -10 to 10 with at least one nonzero, and positive values reveal content above or left.
|
||||
- drag(from: [number, number], to: [number, number], options?: { mouseButton?: 'left' | 'right' | 'middle', modifiers?: ('command' | 'control' | 'option' | 'shift' | 'fn')[], relative?: boolean }) -> ActionResult — drag between two points of one window.
|
||||
|
||||
Raw operation tools use { method, params }, including params: {} for parameterless methods. getApp selects without activating the app. Keep applications in the background for observation and routine input. Do not activate or raise an app during initialization, before each action, or merely because another app is foreground. Prefer advertised AX actions. If a window is hidden or minimized, try app.show() or window.show() and read fresh state. show requests unhide and unminimize without activation, raising or switching Spaces; it chooses the explicit, focused, main or sole AX window. A null windowId means no window was selected for restoration. foregroundChanged compares the front app and top window before and after the request; null means unavailable, and delayed focus changes remain possible. If restoration is unsupported or the window remains unusable, including another Space or Stage Manager, explain the needed foreground step, use app.activate() or an advertised raise action, and read fresh state. Completing the task takes priority when foreground access is necessary. Avoid repeated activation when the user switches away; continue background-capable steps and report what requires foreground access. Applications may still change focus in response to input.
|
||||
A string target is an element id; coordinate targets are [x, y] and are pixels in the latest returned screenshot by default. With relative: true they are integer thousandths of the window width and height from 0 to 999, based on the current AX observation or screenshot. Left clicks support one or two presses; right and middle clicks support one. FOREGROUND_REQUIRED means explain the need, then explicitly activate or raise and observe again; activation is never automatic. Read fresh state after actions, and treat WINDOW_CHANGED, STALE_SNAPSHOT, and SCREENSHOT_REQUIRED as requiring the stated fresh observation or capture before any retry.
|
||||
|
||||
Await operations sequentially. After input, read fresh state to verify the effect. A successful action means dispatched, not verified completion. Cancellation may leave partial effects; never retry uncertain input without observing. STALE_APP requires getApp again; STALE_SNAPSHOT requires getState again. Absolute coordinate input requires a fresh screenshot of that window. Relative input can use a successful AX observation or screenshot. SCREENSHOT_REQUIRED means capture a fresh screenshot before using pixel coordinates. For click and hover, coordinate targets must be [x, y], for example hover([650, 400], {relative:true}). A bare number such as hover(650) is always an AX element index. The relative option applies only to coordinate targets, not AX indices. Window resizing since the latest observation rejects coordinate input; movement alone preserves window-local coordinates. WINDOW_CHANGED requires fresh state before any retry; use an advertised AX action or explicitly restore the window before screenshot or coordinate input, then observe again. PERMISSION_DENIED requires the user to grant the required macOS system permissions for macOS Accessibility or Screen Recording. After granting access, observe fresh state before continuing; do not automatically replay prior input.
|
||||
## Simulated keyboard
|
||||
Available at app or window level depending on the platform.
|
||||
- typeText(text: string) -> ActionResult — type literal text at the app's current focus; line endings send Return and tabs send Tab.
|
||||
- paste(text: string, options?: { format?: 'text' | 'html' }) -> ActionResult — deliver text or HTML through the app's Paste command, restoring prior clipboard contents unless another writer changes them; a Paste failure is surfaced without substituting typeText, and PASTE_NOT_CONSUMED means inspect before any retry.
|
||||
- pressKey(key: string, modifiers?: string[]) -> ActionResult — press and release one complete keystroke. For example pressKey('Tab', ['shift']) or pressKey('+'). Key and modifier names are platform-defined strings. Pass key and modifiers separately; do not pack them into a chord string.
|
||||
|
||||
Hover first uses advertised AX alternate UI without moving the system cursor, preserving revealed descendants until the pointer target leaves their region. When unavailable, it moves the visible system cursor without pressing a button; that route requires the target point to be unobscured in the foreground app. FOREGROUND_REQUIRED requires explicit activation or raising and a fresh observation before retrying. Returning, AX reads and screenshots do not restore the previous point. Wait for the application's hover delay, observe, then hover into the revealed controls before clicking. Use successive hover calls for intermediate points. The result reports hoverRoute and systemPointerMoved. Other input or application changes can interrupt the interaction; app objects do not reserve applications between calls.
|
||||
|
||||
Use selectText only on controls advertising selectionWritable. It selects a unique literal occurrence, or places the cursor before/after it; prefix and suffix must be immediately adjacent. AMBIGUOUS_TEXT requires more context; TEXT_NOT_FOUND or SELECTION_CHANGED requires a fresh observation before deciding on input. Left clicks support single/double clicks; right and middle clicks support single clicks. Modified clicks use pointer input. Optional holdMs is a nonnegative integer in milliseconds for each press, including each press of a double-click. Zero releases without an additional hold. The configured maximum is stated below. It uses pointer input even for AX targets; omission preserves ordinary clicks. The operation deadline still applies, and cancellation or timeout releases the held button. Keypad names such as KP_0 and KP_Enter retain their physical key identity. Window objects expose observation and pointer/AX actions; keyboard input remains app-scoped and follows current focus. Scroll deltas are integer lines from -10 to 10; positive values reveal content above or left. Paste delivers plain text or HTML through the application's enabled standard Paste menu command. Paste follows the target app's focused text input. Foreground requirements vary by application; for example, ChatGPT/Codex was observed to accept background Paste without reading clipboard data. PASTE_NOT_CONSUMED means the command was dispatched but no clipboard read was observed before the configured deadline. Inspect the target contents, explicitly activate the app and focus the intended input, then observe again before deciding whether to retry; a delayed paste may still run. Original clipboard formats are restored if the clipboard still belongs to this paste; a later copy is preserved. Unsupported Paste commands fail without typing fallback. CLIPBOARD_CHANGED means another writer intervened; observe before deciding on further input. Browser applications can be controlled with getApp like other native apps; browser tabs are unavailable. Installed-app search covers standard macOS application directories; use an absolute application path for another location. Launch does not restore a window or document, and cancellation may leave an application already started. The execution host must install the object API and own variable persistence; this plugin does not preload a cua global. In persistent PTC, use the host-provided initialization snippet below once. Its installation-owned file URL works independently of the project directory. Retain app objects between calls. Observation methods emit by default and also return their value. Use void await app.getAXState() to avoid a duplicate REPL completion, or pass emit:false and print the returned text once. End screenshot calls with void await app.getScreenshot() or void await app.getAXStateAndScreenshot() so image bytes are not serialized as the program completion value.
|
||||
Raw tools use { method, params }: include appHandle for app calls and windowId for window calls, and use snapshotId with elementId for element routes. Object options such as emit and disableDiffing are facade-only: emit is local output control and neither is a raw RPC parameter, so raw callers must follow the raw operation schema for admitted params. getAccessibilityTree and getWindow read through raw getState; getWindows maps to listWindows; getScreenshot maps to screenshot; cua.getState maps to listApps; every other named operation uses the same method name.
|
||||
|
||||
Maximum click holdMs: 5000 ms.
|
||||
|
||||
@@ -139,7 +145,7 @@ interface ToolArgsMap {
|
||||
/** Optional positive safe-integer limit on automatic continuation rounds. */
|
||||
max_goal_rounds?: number;
|
||||
} & Record<string, JsonValue>;
|
||||
/** Inspect applications through the CUA object runtime. */
|
||||
/** Inspect desktop access, select an application and window, read AX state, capture screenshots or dispatch input. Use appHandle from getApp; verify actions by reading fresh state. */
|
||||
cua: unknown;
|
||||
/** Edit an existing UTF-8 text file by replacing literal text. */
|
||||
edit: {
|
||||
|
||||
Reference in new issue
Block a user