feat(cua): select literal text with verified AX ranges

packages/computer-use/cua/README.i18n.yaml

@@ -18,8 +18,8 @@
   en: 494643109e1ede83
   zh: 39e603abb26ed601
 /deepseek-ai-dsh-cua/use-this-package/connect-an-external-execution-environment:
-  en: d4636ec68878d60a
-  zh: f37e82c5194f4c0d
+  en: 0f4faf2efb6fc679
+  zh: 714971f3e93d851c
 /deepseek-ai-dsh-cua/understand-the-implementation:
   en: ee9fd0ddf93d496d
   zh: 38d8d5f8b58e6f50

packages/computer-use/cua/README.md

@@ -75,6 +75,8 @@ Use app methods directly for single-window tasks. `getApp` reads initial AX stat
 
 `getWindow` emits its initial AX tree. An owned window without an AX surface returns an empty snapshot and remains available for screenshots and coordinate input. Window objects retain separate snapshots, so reading one window does not invalidate another. Re-reading the same scope invalidates old indices. An AX node removed during observation rejects the read with `STALE_SNAPSHOT` and publishes no partial tree; read again without repeating the preceding input. App-scoped screenshots and coordinates reuse the latest AX observation: its focused on-screen, nonzero-size window, otherwise the sole such window. They do not query current focus again; read `app.getAXState()` to follow a focus change. Explicit window objects retain their selected target. Keyboard methods stay on `app` because input follows the app’s current focus.
 
+`selectText` selects a unique literal occurrence or places the cursor before/after it through writable `AXSelectedTextRange`. Adjacent prefix/suffix disambiguate repeats. Selection preserves the remaining text and formatting, and verifies the resulting UTF-16 range; unsupported, missing, ambiguous or changed text fails without keyboard fallback. AX text includes `selectionWritable`, selection ranges and selected text when available.
+
 Screenshot data must use canonical Base64. Response byte limits apply before parsing, and attachment admission validates the raster and its configured capacity.
 
 Capture and pointer input reject known disagreement between the selected window’s accessibility frame and displayed bounds with `WINDOW_CHANGED`, before posting pointer events. This includes Stage Manager transformations when AX retains the full window frame. Read fresh state; any activation or window raise remains explicit. Advertised AX actions remain available. Missing AX geometry retains the AXless path and does not establish that a transformed window can be operated.
@@ -126,7 +128,7 @@ The plugin contributes one `cua` operation tool with method-specific parameters.
 ```markdown
 Use cua with method getCapabilities and params {} to inspect permissions and sessionLocked. If sessionLocked is true or an operation reports SESSION_LOCKED, stop UI attempts and ask the user to unlock macOS. A null sessionLocked means no usable console session was verified. After unlocking, read fresh state; screenshots from a locked desktop are not current observations. Use listApps and getApp to select an already running application. getApp requests user consent before application observations or input. If access is denied or revoked, stop using that application until the user explicitly grants access again. Never switch tools to bypass a refusal. Keep the returned appHandle; every app operation requires it. listWindows returns window IDs and point bounds. getState returns a complete AX tree and snapshotId. Use its element indices only with that snapshotId. Raw screenshot and coordinate operations require a windowId. Coordinates are window-local macOS points, which may differ from screenshot pixels. A verified window can expose no AX elements. Its empty snapshot permits screenshots and coordinate input; element-index actions require an element from a fresh AX observation.
 
-Initialize the object API once in persistent PTC. Reuse its variables in later calls; put temporary declarations inside { ... } blocks to avoid redeclaration errors. All object methods are async. getAXState returns text, not a structured state object. Its text marks disabled controls, writable values and available selection state.
+Initialize the object API once in persistent PTC. Reuse its variables in later calls; put temporary declarations inside { ... } blocks to avoid redeclaration errors. All object methods are async. getAXState returns text, not a structured state object. Its text marks disabled controls, valueWritable and selectionWritable capabilities, selected state and UTF-16 selection ranges when available.
 
 Use app methods directly for single-window tasks; getApp already reads initial AX state. Call app.getWindows() and await app.getWindow(windowId) only to target a particular window or resolve ambiguous window selection. App screenshots and coordinate actions use the latest AX observation: its focused on-screen, nonzero-size window, otherwise the sole such window. They do not refresh focus automatically. Read app.getAXState() to follow changed focus.
 
@@ -148,6 +150,7 @@ App and Window methods:
 - getAXStateAndScreenshot(options?) -> { state: string, screenshot: Uint8Array }
 - click(target: number | [x, y], options?: { mouseButton?: "left" | "right" | "middle", clickCount?: 1 | 2, modifiers?: Array<"command" | "control" | "option" | "shift" | "fn"> }) -> action result
 - setValue(elementIndex: number, value: string) -> action result
+- selectText(elementIndex: number, text: string, options?: { prefix?: string, suffix?: string, selectionType?: "text" | "cursor_before" | "cursor_after" }) -> action result
 - performSecondaryAction(elementIndex: number, action: string) -> action result
 - scroll({ x, y, deltaLinesX, deltaLinesY }) -> action result
 - drag(from: [x, y], to: [x, y], options?: { mouseButton?: "left" | "right" | "middle", modifiers?: Array<"command" | "control" | "option" | "shift" | "fn"> }) -> action result
@@ -156,7 +159,7 @@ Raw operation tools use { method, params }, including params: {} for parameterle
 
 Await operations sequentially. After input, read fresh state to verify the effect. A successful action means dispatched, not verified completion. Cancellation may leave partial effects; never retry uncertain input without observing. STALE_APP requires getApp again; STALE_SNAPSHOT requires getState again. WINDOW_CHANGED requires fresh state before any retry; use an advertised AX action or explicitly restore the window before screenshot or coordinate input, then observe again. PERMISSION_DENIED requires the user to grant macOS Accessibility or Screen Recording access in System Settings. After granting access, observe fresh state before continuing; do not automatically replay prior input.
 
-Left clicks support single/double clicks; right and middle clicks support single clicks. Modified clicks use pointer input. Keypad names such as KP_0 and KP_Enter retain their physical key identity. Window objects expose observation and pointer/AX actions; keyboard input remains app-scoped and follows current focus. Scroll deltas are integer lines from -10 to 10; positive values reveal content above or left. Paste inserts plain text using native Unicode input without changing the clipboard; rich-text formats are unavailable. Browser applications can be controlled with getApp like other native apps; browser tabs and app launching are unavailable. The execution host must install the object API and own variable persistence; this plugin does not preload a cua global. In persistent PTC, use the host-provided initialization snippet below once. Its installation-owned file URL works independently of the project directory. Retain app objects between calls. End screenshot calls with void await app.getScreenshot() or void await app.getAXStateAndScreenshot() so image bytes are not serialized as the program completion value.
+Use selectText only on controls advertising selectionWritable. It selects a unique literal occurrence, or places the cursor before/after it; prefix and suffix must be immediately adjacent. AMBIGUOUS_TEXT requires more context; TEXT_NOT_FOUND or SELECTION_CHANGED requires a fresh observation before deciding on input. Left clicks support single/double clicks; right and middle clicks support single clicks. Modified clicks use pointer input. Keypad names such as KP_0 and KP_Enter retain their physical key identity. Window objects expose observation and pointer/AX actions; keyboard input remains app-scoped and follows current focus. Scroll deltas are integer lines from -10 to 10; positive values reveal content above or left. Paste inserts plain text using native Unicode input without changing the clipboard; rich-text formats are unavailable. Browser applications can be controlled with getApp like other native apps; browser tabs and app launching are unavailable. The execution host must install the object API and own variable persistence; this plugin does not preload a cua global. In persistent PTC, use the host-provided initialization snippet below once. Its installation-owned file URL works independently of the project directory. Retain app objects between calls. End screenshot calls with void await app.getScreenshot() or void await app.getAXStateAndScreenshot() so image bytes are not serialized as the program completion value.
 ```
 
 #### Token effect

packages/computer-use/cua/README.zh.md

@@ -75,6 +75,8 @@ void await app.getScreenshot()
 
 `getWindow` 会输出初始 AX 树。已确认归属但没有 AX 界面的窗口返回空快照，仍可截图和发送坐标输入。不同窗口分别保留快照，读取一个窗口不会使另一个窗口失效；重新读取同一范围会使旧索引失效。观察期间节点被移除时，该次读取返回 `STALE_SNAPSHOT`，不发布部分树；可以重新读取，但不要重复此前的输入。应用级截图和坐标操作复用最近一次 AX 观察：优先使用其中在屏幕上且尺寸非零的焦点窗口，否则要求只有一个符合条件的窗口。它们不会重新查询即时焦点；需要跟随焦点变化时，先读取 `app.getAXState()`。显式窗口对象保持选定的目标。键盘方法保留在 `app` 上，因为输入跟随应用的当前焦点。
 
+`selectText` 通过可写的 `AXSelectedTextRange` 选中唯一的字面文本，或把光标放在它的前后。紧邻的前后文用于区分重复文本。选区操作保留其余文字和格式，并核对实际 UTF-16 范围；控件不支持、文本不存在、匹配不唯一或文字变化时明确失败，不自动改用键盘输入。AX 文本包含可用的 `selectionWritable`、选区范围和选中文字。
+
 截图数据必须使用规范 Base64。响应字节上限在解析前执行，附件接入校验图像数据及配置的容量。
 
 截图和鼠标输入会核对所选窗口的辅助功能框与显示边界；已知两者不一致时，在发送任何鼠标事件前返回 `WINDOW_CHANGED`。当 AX 保留完整窗口框时，这也会拦住 Stage Manager 变换后的窗口。需要重新读取状态；激活应用或抬起窗口仍须显式调用。元素声明的 AX 动作保持可用。缺少 AX 几何时保留无 AX 路径，不代表变换后的窗口一定可以操作。
@@ -126,7 +128,7 @@ AppKit sheet 可以通过其窗口 ID 单独选择，也会出现在所属文档
 ```markdown
 Use cua with method getCapabilities and params {} to inspect permissions and sessionLocked. If sessionLocked is true or an operation reports SESSION_LOCKED, stop UI attempts and ask the user to unlock macOS. A null sessionLocked means no usable console session was verified. After unlocking, read fresh state; screenshots from a locked desktop are not current observations. Use listApps and getApp to select an already running application. getApp requests user consent before application observations or input. If access is denied or revoked, stop using that application until the user explicitly grants access again. Never switch tools to bypass a refusal. Keep the returned appHandle; every app operation requires it. listWindows returns window IDs and point bounds. getState returns a complete AX tree and snapshotId. Use its element indices only with that snapshotId. Raw screenshot and coordinate operations require a windowId. Coordinates are window-local macOS points, which may differ from screenshot pixels. A verified window can expose no AX elements. Its empty snapshot permits screenshots and coordinate input; element-index actions require an element from a fresh AX observation.
 
-Initialize the object API once in persistent PTC. Reuse its variables in later calls; put temporary declarations inside { ... } blocks to avoid redeclaration errors. All object methods are async. getAXState returns text, not a structured state object. Its text marks disabled controls, writable values and available selection state.
+Initialize the object API once in persistent PTC. Reuse its variables in later calls; put temporary declarations inside { ... } blocks to avoid redeclaration errors. All object methods are async. getAXState returns text, not a structured state object. Its text marks disabled controls, valueWritable and selectionWritable capabilities, selected state and UTF-16 selection ranges when available.
 
 Use app methods directly for single-window tasks; getApp already reads initial AX state. Call app.getWindows() and await app.getWindow(windowId) only to target a particular window or resolve ambiguous window selection. App screenshots and coordinate actions use the latest AX observation: its focused on-screen, nonzero-size window, otherwise the sole such window. They do not refresh focus automatically. Read app.getAXState() to follow changed focus.
 
@@ -148,6 +150,7 @@ App and Window methods:
 - getAXStateAndScreenshot(options?) -> { state: string, screenshot: Uint8Array }
 - click(target: number | [x, y], options?: { mouseButton?: "left" | "right" | "middle", clickCount?: 1 | 2, modifiers?: Array<"command" | "control" | "option" | "shift" | "fn"> }) -> action result
 - setValue(elementIndex: number, value: string) -> action result
+- selectText(elementIndex: number, text: string, options?: { prefix?: string, suffix?: string, selectionType?: "text" | "cursor_before" | "cursor_after" }) -> action result
 - performSecondaryAction(elementIndex: number, action: string) -> action result
 - scroll({ x, y, deltaLinesX, deltaLinesY }) -> action result
 - drag(from: [x, y], to: [x, y], options?: { mouseButton?: "left" | "right" | "middle", modifiers?: Array<"command" | "control" | "option" | "shift" | "fn"> }) -> action result
@@ -156,7 +159,7 @@ Raw operation tools use { method, params }, including params: {} for parameterle
 
 Await operations sequentially. After input, read fresh state to verify the effect. A successful action means dispatched, not verified completion. Cancellation may leave partial effects; never retry uncertain input without observing. STALE_APP requires getApp again; STALE_SNAPSHOT requires getState again. WINDOW_CHANGED requires fresh state before any retry; use an advertised AX action or explicitly restore the window before screenshot or coordinate input, then observe again. PERMISSION_DENIED requires the user to grant macOS Accessibility or Screen Recording access in System Settings. After granting access, observe fresh state before continuing; do not automatically replay prior input.
 
-Left clicks support single/double clicks; right and middle clicks support single clicks. Modified clicks use pointer input. Keypad names such as KP_0 and KP_Enter retain their physical key identity. Window objects expose observation and pointer/AX actions; keyboard input remains app-scoped and follows current focus. Scroll deltas are integer lines from -10 to 10; positive values reveal content above or left. Paste inserts plain text using native Unicode input without changing the clipboard; rich-text formats are unavailable. Browser applications can be controlled with getApp like other native apps; browser tabs and app launching are unavailable. The execution host must install the object API and own variable persistence; this plugin does not preload a cua global. In persistent PTC, use the host-provided initialization snippet below once. Its installation-owned file URL works independently of the project directory. Retain app objects between calls. End screenshot calls with void await app.getScreenshot() or void await app.getAXStateAndScreenshot() so image bytes are not serialized as the program completion value.
+Use selectText only on controls advertising selectionWritable. It selects a unique literal occurrence, or places the cursor before/after it; prefix and suffix must be immediately adjacent. AMBIGUOUS_TEXT requires more context; TEXT_NOT_FOUND or SELECTION_CHANGED requires a fresh observation before deciding on input. Left clicks support single/double clicks; right and middle clicks support single clicks. Modified clicks use pointer input. Keypad names such as KP_0 and KP_Enter retain their physical key identity. Window objects expose observation and pointer/AX actions; keyboard input remains app-scoped and follows current focus. Scroll deltas are integer lines from -10 to 10; positive values reveal content above or left. Paste inserts plain text using native Unicode input without changing the clipboard; rich-text formats are unavailable. Browser applications can be controlled with getApp like other native apps; browser tabs and app launching are unavailable. The execution host must install the object API and own variable persistence; this plugin does not preload a cua global. In persistent PTC, use the host-provided initialization snippet below once. Its installation-owned file URL works independently of the project directory. Retain app objects between calls. End screenshot calls with void await app.getScreenshot() or void await app.getAXStateAndScreenshot() so image bytes are not serialized as the program completion value.
 ```
 
 #### Token 影响

packages/computer-use/cua/native/Sources/CuaNative/AXAccess.swift

@@ -50,7 +50,7 @@ final class AXAccess {
         // and accessibility children remain readable.
         guard error == .failure else { return false }
         let optional: Set<String> = ["AXValue", "AXTitle", "AXDescription", "AXIdentifier", "AXPlaceholderValue",
-                                     "AXSubrole", "AXEnabled", "AXSelected", "AXPosition", "AXSize", "AXMain", "AXMinimized"]
+                                     "AXSubrole", "AXEnabled", "AXSelected", "AXSelectedText", "AXSelectedTextRange", "AXPosition", "AXSize", "AXMain", "AXMinimized"]
         return optional.contains(attribute)
     }
     static func actionsUnavailable(_ error: AXError) -> Bool {
@@ -145,6 +145,38 @@ final class AXAccess {
         let error = AXUIElementSetAttributeValue(element, "AXValue" as CFString, value as CFString)
         guard error == .success else { throw failure(error, "set AXValue") }
     }
+    func selectText(_ element: AXUIElement, text: String, prefix: String?, suffix: String?, mode: String) throws {
+        try requireEnabled(element)
+        guard try writable(element, "AXSelectedTextRange"), let original = try string(element, "AXValue") else {
+            throw NativeFailure(code: "UNSUPPORTED", message: "This control does not expose writable text selection")
+        }
+        var range = try TextSelection.range(in: original, text: text, prefix: prefix, suffix: suffix, mode: mode)
+        guard let value = AXValueCreate(.cfRange, &range) else {
+            throw NativeFailure(code: "NATIVE_ERROR", message: "Could not create the text selection range")
+        }
+        guard try string(element, "AXValue") == original else {
+            throw NativeFailure(code: "STALE_SNAPSHOT", message: "Text changed before selection; read fresh state")
+        }
+        try prepare(element)
+        let error = AXUIElementSetAttributeValue(element, "AXSelectedTextRange" as CFString, value)
+        guard error == .success else { throw failure(error, "set text selection") }
+        let actual = try textRange(attribute(element, "AXSelectedTextRange"))
+        guard actual?.location == range.location, actual?.length == range.length, try string(element, "AXValue") == original else {
+            throw NativeFailure(code: "SELECTION_CHANGED", message: "The requested selection was not retained; read fresh state before input")
+        }
+    }
+    private func textRange(_ raw: CFTypeRef?) throws -> CFRange? {
+        guard let raw else { return nil }
+        guard CFGetTypeID(raw) == AXValueGetTypeID() else {
+            throw NativeFailure(code: "NATIVE_ERROR", message: "AXSelectedTextRange is not an accessibility value")
+        }
+        let value = unsafeDowncast(raw, to: AXValue.self)
+        var range = CFRange()
+        guard AXValueGetType(value) == .cfRange, AXValueGetValue(value, .cfRange, &range), range.location >= 0, range.length >= 0 else {
+            throw NativeFailure(code: "NATIVE_ERROR", message: "AXSelectedTextRange is not a valid range")
+        }
+        return range
+    }
     func bounds(_ element: AXUIElement) throws -> WindowBounds? {
         guard let position = try attribute(element, "AXPosition") else { return nil }
         return try bounds(position: position, size: attribute(element, "AXSize"))
@@ -189,7 +221,7 @@ final class AXAccess {
     private func metadata(_ element: AXUIElement) throws -> [String: CFTypeRef]? {
         try prepare(element)
         let names = ["AXRole", "AXTitle", "AXValue", "AXPlaceholderValue", "AXDescription", "AXEnabled",
-                     "AXIdentifier", "AXPosition", "AXSize", "AXChildren", "AXSelected"]
+                     "AXIdentifier", "AXPosition", "AXSize", "AXChildren", "AXSelected", "AXSelectedText", "AXSelectedTextRange"]
         var output: CFArray?
         let error = AXUIElementCopyMultipleAttributeValues(element, names as CFArray, [], &output)
         if error == .notImplemented || error == .attributeUnsupported { return nil }
@@ -299,6 +331,7 @@ final class AXAccess {
         var lines: [String] = []
         var elements: [JSONValue] = []
         var indexed: [Int: AXUIElement] = [:]
+        let textRoles: Set<String> = ["AXTextField", "AXTextArea", "AXComboBox", "AXStaticText"]
         let valueRoles: Set<String> = ["AXTextField", "AXTextArea", "AXComboBox", "AXSlider", "AXStepper", "AXCheckBox", "AXRadioButton"]
         func walk(_ element: AXUIElement, level: Int, depth: Int) throws {
             try cancellation.check()
@@ -329,7 +362,10 @@ final class AXAccess {
             else { valueWritable = false }
             let enabled = try booleanValue(observed("AXEnabled"), attribute: "AXEnabled")
             let selected = try booleanValue(observed("AXSelected"), attribute: "AXSelected")
-            let addressable = enabled != false && (!actionNames.isEmpty || valueWritable)
+            let selectionWritable = textRoles.contains(role) ? try writable(element, "AXSelectedTextRange") : false
+            let selectedRange = textRoles.contains(role) ? try textRange(observed("AXSelectedTextRange")) : nil
+            let selectedText = textRoles.contains(role) ? try stringValue(observed("AXSelectedText"), attribute: "AXSelectedText") : nil
+            let addressable = enabled != false && (!actionNames.isEmpty || valueWritable || selectionWritable)
             let meaningful = [title, value, description].contains { !($0?.isEmpty ?? true) }
             if !addressable && !meaningful && role != "AXWindow" && role != "AXSheet" {
                 for child in try childValues(observed("AXChildren"), attribute: "AXChildren") { try walk(child, level: level, depth: depth + 1) }
@@ -341,10 +377,13 @@ final class AXAccess {
             line += role
             if enabled == false { line += " disabled" }
             if valueWritable { line += " valueWritable=true" }
+            if selectionWritable { line += " selectionWritable=true" }
             if let selected { line += " selected=\(selected)" }
             if let title, !title.isEmpty { line += " \(String(reflecting: title))" }
             if let value, !value.isEmpty { line += " = \(String(reflecting: value))" }
             if let description, !description.isEmpty { line += " (\(description))" }
+            if let selectedRange { line += " selection=[\(selectedRange.location), \(selectedRange.length)]" }
+            if let selectedText, !selectedText.isEmpty { line += " selectedText=\(String(reflecting: selectedText))" }
             if !actionNames.isEmpty { line += " actions=[\(actionNames.joined(separator: ", "))]" }
             lines.append(line)
             if let index {
@@ -357,7 +396,10 @@ final class AXAccess {
                     "value": value.map(JSONValue.string) ?? .null, "description": description.map(JSONValue.string) ?? .null,
                     "identifier": try stringValue(observed("AXIdentifier"), attribute: "AXIdentifier").map(JSONValue.string) ?? .null,
                     "bounds": frameJSON, "actions": .array(actionNames.map(JSONValue.string)), "enabled": enabled.map(JSONValue.bool) ?? .null,
-                    "valueWritable": .bool(valueWritable), "selected": selected.map(JSONValue.bool) ?? .null,
+                    "valueWritable": .bool(valueWritable), "selectionWritable": .bool(selectionWritable),
+                    "selected": selected.map(JSONValue.bool) ?? .null,
+                    "selectedText": selectedText.map(JSONValue.string) ?? .null,
+                    "selectedRange": selectedRange.map { .array([.number(Double($0.location)), .number(Double($0.length))]) } ?? .null,
                 ]))
             }
             for child in try childValues(observed("AXChildren"), attribute: "AXChildren") { try walk(child, level: level + 1, depth: depth + 1) }

packages/computer-use/cua/native/Sources/CuaNative/AXWorker.swift

@@ -230,6 +230,9 @@ actor AXWorker {
             if method == "typeText" { try inputs().0.typeText(params.string("text"), to: recipient.pid) }
             else { try inputs().0.pressKey(params.string("key"), modifiers: params.modifiers(), to: recipient.pid) }
         case "setValue": try ax().setValue(element(handle, params), params.string("value"))
+        case "selectText":
+            try ax().selectText(element(handle, params), text: params.string("text"), prefix: params.optionalString("prefix"),
+                                suffix: params.optionalString("suffix"), mode: params.optionalString("selectionType") ?? "text")
         case "performSecondaryAction":
             let selected = try element(handle, params)
             let requested = try params.string("action")

packages/computer-use/cua/native/Sources/CuaNative/RequestAdmission.swift

@@ -8,6 +8,7 @@ struct RequestParameters: Sendable {
         "getState": ["appHandle", "windowId", "maxElements", "maxDepth"],
         "click": ["appHandle", "snapshotId", "elementIndex", "windowId", "x", "y", "mouseButton", "clickCount", "modifiers"],
         "setValue": ["appHandle", "snapshotId", "elementIndex", "value"],
+        "selectText": ["appHandle", "snapshotId", "elementIndex", "text", "prefix", "suffix", "selectionType"],
         "performSecondaryAction": ["appHandle", "snapshotId", "elementIndex", "action"],
         "typeText": ["appHandle", "text"], "pressKey": ["appHandle", "key", "modifiers"],
         "scroll": ["appHandle", "windowId", "snapshotId", "x", "y", "deltaLinesX", "deltaLinesY"],

packages/computer-use/cua/native/Sources/CuaNative/TextSelection.swift

@@ -0,0 +1,38 @@
+// Literal text matching returns UTF-16 ranges accepted by macOS accessibility.
+import Foundation
+
+struct TextSelection {
+    static func range(in value: String, text: String, prefix: String?, suffix: String?, mode: String) throws -> CFRange {
+        guard !text.isEmpty else { throw RequestParameters.invalid("Selection text must not be empty") }
+        guard ["text", "cursor_before", "cursor_after"].contains(mode) else {
+            throw RequestParameters.invalid("Unsupported text selection type")
+        }
+        let source = value as NSString
+        let before = (prefix ?? "") as NSString
+        let after = (suffix ?? "") as NSString
+        var start = 0
+        var match: NSRange?
+        while start <= source.length {
+            let candidate = source.range(of: text, options: .literal, range: NSRange(location: start, length: source.length - start))
+            if candidate.location == NSNotFound { break }
+            let end = candidate.location + candidate.length
+            let matchesPrefix = candidate.location >= before.length
+                && source.substring(with: NSRange(location: candidate.location - before.length, length: before.length)).utf16.elementsEqual((before as String).utf16)
+            let matchesSuffix = source.length - end >= after.length
+                && source.substring(with: NSRange(location: end, length: after.length)).utf16.elementsEqual((after as String).utf16)
+            if matchesPrefix && matchesSuffix {
+                guard match == nil else {
+                    throw NativeFailure(code: "AMBIGUOUS_TEXT", message: "Text matches more than once; supply adjacent prefix or suffix")
+                }
+                match = candidate
+            }
+            start = candidate.location + 1
+        }
+        guard let match else { throw NativeFailure(code: "TEXT_NOT_FOUND", message: "Text and adjacent context were not found in this control") }
+        switch mode {
+        case "cursor_before": return CFRange(location: match.location, length: 0)
+        case "cursor_after": return CFRange(location: match.location + match.length, length: 0)
+        default: return CFRange(location: match.location, length: match.length)
+        }
+    }
+}

packages/computer-use/cua/native/protocol.schema.json

@@ -688,6 +688,63 @@
         "method",
         "params"
       ]
+    },
+    {
+      "type": "object",
+      "additionalProperties": false,
+      "properties": {
+        "id": {
+          "type": "string"
+        },
+        "method": {
+          "const": "selectText"
+        },
+        "params": {
+          "type": "object",
+          "additionalProperties": false,
+          "properties": {
+            "appHandle": {
+              "type": "string"
+            },
+            "snapshotId": {
+              "type": "string"
+            },
+            "elementIndex": {
+              "type": "integer",
+              "minimum": 0
+            },
+            "text": {
+              "type": "string",
+              "minLength": 1
+            },
+            "prefix": {
+              "type": "string"
+            },
+            "suffix": {
+              "type": "string"
+            },
+            "selectionType": {
+              "type": "string",
+              "enum": [
+                "text",
+                "cursor_before",
+                "cursor_after"
+              ]
+            }
+          },
+          "required": [
+            "appHandle",
+            "snapshotId",
+            "elementIndex",
+            "text"
+          ]
+        }
+      },
+      "required": [
+        "id",
+        "method",
+        "params"
+      ]
     }
   ]
 }

packages/computer-use/cua/native/tests/swift/TextSelectionTests.swift

@@ -0,0 +1,47 @@
+// Literal selection disambiguation preserves UTF-16 positions without rewriting text.
+import Foundation
+import Testing
+@testable import CuaNative
+
+@Test func selectsOnlyTheRepeatedOccurrenceWithAdjacentContext() throws {
+    let range = try TextSelection.range(in: "a😀 target / b😀 target", text: "target", prefix: "b😀 ", suffix: nil, mode: "text")
+    #expect(range.location == 17)
+    #expect(range.length == 6)
+    let emoji = try TextSelection.range(in: "😀中😀中", text: "😀", prefix: "中", suffix: nil, mode: "text")
+    #expect(emoji.location == 3)
+    #expect(emoji.length == 2)
+}
+
+@Test func placesTheCursorBeforeOrAfterTheMatchedText() throws {
+    let before = try TextSelection.range(in: "a😀 target / b😀 target", text: "target", prefix: "b😀 ", suffix: nil, mode: "cursor_before")
+    let after = try TextSelection.range(in: "a😀 target / b😀 target", text: "target", prefix: "b😀 ", suffix: nil, mode: "cursor_after")
+    #expect(before.location == 17 && before.length == 0)
+    #expect(after.location == 23 && after.length == 0)
+    let suffixed = try TextSelection.range(in: "target first; target second", text: "target", prefix: nil, suffix: " second", mode: "text")
+    #expect(suffixed.location == 14 && suffixed.length == 6)
+}
+
+@Test(arguments: [("target target", "target", "AMBIGUOUS_TEXT"), ("aaaa", "aa", "AMBIGUOUS_TEXT"),
+                  ("other", "target", "TEXT_NOT_FOUND"), ("", "target", "TEXT_NOT_FOUND")])
+func rejectsMissingOrAmbiguousText(_ value: String, _ text: String, _ code: String) {
+    do {
+        _ = try TextSelection.range(in: value, text: text, prefix: nil, suffix: nil, mode: "text")
+        Issue.record("Invalid selection was accepted")
+    } catch let error as NativeFailure { #expect(error.code == code) }
+    catch { Issue.record("Unexpected selection failure: \(error)") }
+}
+
+@Test func selectionContextIsAdjacentAndMatchingIsLiteral() throws {
+    #expect(throws: NativeFailure.self) {
+        try TextSelection.range(in: "prefix gap target", text: "target", prefix: "prefix", suffix: nil, mode: "text")
+    }
+    #expect(throws: NativeFailure.self) {
+        try TextSelection.range(in: "e\u{301}", text: "é", prefix: nil, suffix: nil, mode: "text")
+    }
+    #expect(throws: NativeFailure.self) {
+        try TextSelection.range(in: "hello", text: "", prefix: nil, suffix: nil, mode: "text")
+    }
+    #expect(throws: NativeFailure.self) {
+        try TextSelection.range(in: "hello", text: "hello", prefix: nil, suffix: nil, mode: "unknown")
+    }
+}

packages/computer-use/cua/native/upstream.json

@@ -28,7 +28,7 @@
     {
       "local": "Sources/CuaNative/AXAccess.swift",
       "upstream": "libs/cua-driver/rust/crates/platform-macos/src/ax/tree.rs",
-      "changes": "Reimplement AX traversal, selected-window filtering and display projection in Swift. Retain CFEqual identity handling, scoped AX references, checked discovery failures, AX-less owned windows and DSH system-panel ownership validation. Batch per-node observation attributes while retaining per-attribute errors and individual reads for unsupported batch APIs. Resolve exact-window AX frames for DSH capture and pointer geometry checks. Include value writability, disabled and selected state in the model-facing tree."
+      "changes": "Reimplement AX traversal, selected-window filtering and display projection in Swift. Retain CFEqual identity handling, scoped AX references, checked discovery failures, AX-less owned windows and DSH system-panel ownership validation. Batch per-node observation attributes while retaining per-attribute errors and individual reads for unsupported batch APIs. Resolve exact-window AX frames for DSH capture and pointer geometry checks. Include value writability, disabled and selected state in the model-facing tree. Select uniquely matched literal text through AXSelectedTextRange and verify the UTF-16 range without rewriting content."
     },
     {
       "local": "Sources/CuaNative/NativePlatform.swift",
@@ -44,7 +44,8 @@
     "Sources/CuaNative/RequestCoordinator.swift",
     "Sources/CuaNative/Runtime.swift",
     "Sources/CuaNative/WindowCapture.swift",
-    "Sources/CuaNative/WindowGeometry.swift"
+    "Sources/CuaNative/WindowGeometry.swift",
+    "Sources/CuaNative/TextSelection.swift"
   ],
   "references": [
     {

packages/computer-use/cua/src/api.ts

@@ -101,6 +101,13 @@ export function createCuaApi(dispatch: CuaDispatch, sink?: CuaObservationSink):
         return action('click', { ...snapshotParams(), windowId: windowId(), x, y, mouseButton, clickCount, ...modifiers })
       },
       async setValue(index, value) { return action('setValue', { ...snapshotParams(), elementIndex: index, value }) },
+      async selectText(index, text, options) {
+        return action('selectText', { ...snapshotParams(), elementIndex: index, text,
+          ...options?.prefix === undefined ? {} : { prefix: options.prefix },
+          ...options?.suffix === undefined ? {} : { suffix: options.suffix },
+          ...options?.selectionType === undefined ? {} : { selectionType: options.selectionType },
+        })
+      },
       async performSecondaryAction(index, name) { return action('performSecondaryAction', { ...snapshotParams(), elementIndex: index, action: name }) },
       async scroll(options) {
         if (options.deltaLinesX === 0 && options.deltaLinesY === 0) throw new Error('At least one scroll delta must be nonzero')

packages/computer-use/cua/src/index.ts

@@ -82,7 +82,7 @@ export const Config = Schema.object({
 /** Instructions for raw operation calls and optional object-API bindings. */
 export const GUIDANCE = `Use cua with method getCapabilities and params {} to inspect permissions and sessionLocked. If sessionLocked is true or an operation reports SESSION_LOCKED, stop UI attempts and ask the user to unlock macOS. A null sessionLocked means no usable console session was verified. After unlocking, read fresh state; screenshots from a locked desktop are not current observations. Use listApps and getApp to select an already running application. getApp requests user consent before application observations or input. If access is denied or revoked, stop using that application until the user explicitly grants access again. Never switch tools to bypass a refusal. Keep the returned appHandle; every app operation requires it. listWindows returns window IDs and point bounds. getState returns a complete AX tree and snapshotId. Use its element indices only with that snapshotId. Raw screenshot and coordinate operations require a windowId. Coordinates are window-local macOS points, which may differ from screenshot pixels. A verified window can expose no AX elements. Its empty snapshot permits screenshots and coordinate input; element-index actions require an element from a fresh AX observation.
 
-Initialize the object API once in persistent PTC. Reuse its variables in later calls; put temporary declarations inside { ... } blocks to avoid redeclaration errors. All object methods are async. getAXState returns text, not a structured state object. Its text marks disabled controls, writable values and available selection state.
+Initialize the object API once in persistent PTC. Reuse its variables in later calls; put temporary declarations inside { ... } blocks to avoid redeclaration errors. All object methods are async. getAXState returns text, not a structured state object. Its text marks disabled controls, valueWritable and selectionWritable capabilities, selected state and UTF-16 selection ranges when available.
 
 Use app methods directly for single-window tasks; getApp already reads initial AX state. Call app.getWindows() and await app.getWindow(windowId) only to target a particular window or resolve ambiguous window selection. App screenshots and coordinate actions use the latest AX observation: its focused on-screen, nonzero-size window, otherwise the sole such window. They do not refresh focus automatically. Read app.getAXState() to follow changed focus.
 
@@ -104,6 +104,7 @@ App and Window methods:
 - getAXStateAndScreenshot(options?) -> { state: string, screenshot: Uint8Array }
 - click(target: number | [x, y], options?: { mouseButton?: "left" | "right" | "middle", clickCount?: 1 | 2, modifiers?: Array<"command" | "control" | "option" | "shift" | "fn"> }) -> action result
 - setValue(elementIndex: number, value: string) -> action result
+- selectText(elementIndex: number, text: string, options?: { prefix?: string, suffix?: string, selectionType?: "text" | "cursor_before" | "cursor_after" }) -> action result
 - performSecondaryAction(elementIndex: number, action: string) -> action result
 - scroll({ x, y, deltaLinesX, deltaLinesY }) -> action result
 - drag(from: [x, y], to: [x, y], options?: { mouseButton?: "left" | "right" | "middle", modifiers?: Array<"command" | "control" | "option" | "shift" | "fn"> }) -> action result
@@ -112,7 +113,7 @@ Raw operation tools use { method, params }, including params: {} for parameterle
 
 Await operations sequentially. After input, read fresh state to verify the effect. A successful action means dispatched, not verified completion. Cancellation may leave partial effects; never retry uncertain input without observing. STALE_APP requires getApp again; STALE_SNAPSHOT requires getState again. WINDOW_CHANGED requires fresh state before any retry; use an advertised AX action or explicitly restore the window before screenshot or coordinate input, then observe again. PERMISSION_DENIED requires the user to grant macOS Accessibility or Screen Recording access in System Settings. After granting access, observe fresh state before continuing; do not automatically replay prior input.
 
-Left clicks support single/double clicks; right and middle clicks support single clicks. Modified clicks use pointer input. Keypad names such as KP_0 and KP_Enter retain their physical key identity. Window objects expose observation and pointer/AX actions; keyboard input remains app-scoped and follows current focus. Scroll deltas are integer lines from -10 to 10; positive values reveal content above or left. Paste inserts plain text using native Unicode input without changing the clipboard; rich-text formats are unavailable. Browser applications can be controlled with getApp like other native apps; browser tabs and app launching are unavailable. The execution host must install the object API and own variable persistence; this plugin does not preload a cua global. In persistent PTC, use the host-provided initialization snippet below once. Its installation-owned file URL works independently of the project directory. Retain app objects between calls. End screenshot calls with void await app.getScreenshot() or void await app.getAXStateAndScreenshot() so image bytes are not serialized as the program completion value.`
+Use selectText only on controls advertising selectionWritable. It selects a unique literal occurrence, or places the cursor before/after it; prefix and suffix must be immediately adjacent. AMBIGUOUS_TEXT requires more context; TEXT_NOT_FOUND or SELECTION_CHANGED requires a fresh observation before deciding on input. Left clicks support single/double clicks; right and middle clicks support single clicks. Modified clicks use pointer input. Keypad names such as KP_0 and KP_Enter retain their physical key identity. Window objects expose observation and pointer/AX actions; keyboard input remains app-scoped and follows current focus. Scroll deltas are integer lines from -10 to 10; positive values reveal content above or left. Paste inserts plain text using native Unicode input without changing the clipboard; rich-text formats are unavailable. Browser applications can be controlled with getApp like other native apps; browser tabs and app launching are unavailable. The execution host must install the object API and own variable persistence; this plugin does not preload a cua global. In persistent PTC, use the host-provided initialization snippet below once. Its installation-owned file URL works independently of the project directory. Retain app objects between calls. End screenshot calls with void await app.getScreenshot() or void await app.getAXStateAndScreenshot() so image bytes are not serialized as the program completion value.`
 
 function executionAgent(execution: Readonly<ToolExecution>): Agent {
   if (!execution.agent) throw new Error('dsh-cua requires an agent-owned execution')

packages/computer-use/cua/src/protocol.ts

@@ -35,6 +35,8 @@ export const Request = z.discriminatedUnion('method', [
     z.object({ ...coordinate, ...pointer, clickCount: z.union([z.literal(1), z.literal(2)]).optional() }).strict(),
   ]) }).strict(),
   operation('setValue', { ...snapshot, value: z.string() }),
+  operation('selectText', { ...snapshot, text: z.string().min(1), prefix: z.string().optional(), suffix: z.string().optional(),
+    selectionType: z.enum(['text', 'cursor_before', 'cursor_after']).optional() }),
   operation('performSecondaryAction', { ...snapshot, action: z.string().min(1) }),
   operation('typeText', { ...app, text: z.string() }),
   operation('pressKey', { ...app, key: z.string().min(1), modifiers: z.array(z.enum(['command', 'control', 'option', 'shift', 'fn'])).optional() }),

packages/computer-use/cua/src/types.ts

@@ -61,6 +61,8 @@ export type CuaMouseButton = 'left' | 'right' | 'middle'
 export interface CuaClickOptions { mouseButton?: CuaMouseButton; clickCount?: 1 | 2; modifiers?: readonly CuaPointerModifier[] }
 /** One same-window drag; omitted button is left and omitted modifiers are empty. */
 export interface CuaDragOptions { mouseButton?: CuaMouseButton; modifiers?: readonly CuaPointerModifier[] }
+/** Adjacent literal context disambiguates repeated text; cursor modes collapse the matched range. */
+export interface CuaSelectTextOptions { prefix?: string; suffix?: string; selectionType?: 'text' | 'cursor_before' | 'cursor_after' }
 /** Positive deltas reveal content above/left; each axis is an integer from -10 to 10. */
 export interface CuaScrollOptions { x: number; y: number; deltaLinesX: number; deltaLinesY: number }
 /** An observed AX tree and independently captured PNG; capture is not atomic. */
@@ -103,6 +105,14 @@ export interface CuaSurface {
    * @returns dispatch evidence.
    */
   setValue(index: number, value: string): Promise<CuaActionResult>
+  /**
+   * Select one literal text occurrence through writable AXSelectedTextRange and verify its range.
+   * @param index - Text control index from this surface's latest tree.
+   * @param text - Nonempty literal text; multiple matches require adjacent context.
+   * @param options - Adjacent prefix/suffix and selection or cursor placement.
+   * @returns Dispatch evidence; rejects unsupported, ambiguous or changed text without keyboard fallback.
+   */
+  selectText(index: number, text: string, options?: CuaSelectTextOptions): Promise<CuaActionResult>
   /**
    * Invoke an action advertised by an observed element; raising a window activates its app.
    * @param index - Index from this surface's latest AX tree.

packages/computer-use/cua/tests/api.spec.ts

@@ -115,6 +115,25 @@ describe('pointer buttons and modifiers', () => {
   })
 })
 
+describe('text selections', () => {
+  it('binds literal text and optional disambiguation to the latest observed control', async () => {
+    const f = fixture()
+    const app = await f.api.getApp('Editor')
+    await app.selectText(1, 'target')
+    expect(f.dispatch).toHaveBeenLastCalledWith({ method: 'selectText', params: {
+      appHandle: 'app-17', snapshotId: 'snapshot-1', elementIndex: 1, text: 'target',
+    } })
+    await app.selectText(1, 'target', { prefix: 'second ', suffix: '.', selectionType: 'cursor_after' })
+    expect(f.dispatch).toHaveBeenLastCalledWith({ method: 'selectText', params: {
+      appHandle: 'app-17', snapshotId: 'snapshot-1', elementIndex: 1, text: 'target',
+      prefix: 'second ', suffix: '.', selectionType: 'cursor_after',
+    } })
+    const calls = f.dispatch.mock.calls.length
+    await expect(app.selectText(1, '')).rejects.toThrow()
+    expect(f.dispatch).toHaveBeenCalledTimes(calls)
+  })
+})
+
 describe('application and window observations', () => {
   it('keeps the latest AX index scope separate for an app and each selected window', async () => {
     const f = fixture()