Files
deepseek-harness/packages/computer-use/cua

description, kind
description kind
Inspect and operate native macOS applications through scoped app and window objects. package-reference

@deepseek-ai/dsh-cua

English | 中文

Summary

Inspect macOS permissions, select running applications and windows, read accessibility trees, capture screenshots and send input. The object API can be retained by an external persistent execution environment. The Swift helper uses macOS frameworks directly and adapts selected MIT-licensed Cua Driver behavior. This package owns neither a PTC runtime nor variable persistence.

Table of Contents


Use this package

Enable this provider through explicit source composition on macOS 14 or later. Shipped profiles do not mount it. Prepare a native helper and load the registry/provider overlay:

swift build --package-path packages/computer-use/cua/native --configuration release
cua_native_root="$PWD/packages/computer-use/cua/native"
export DSH_CUA_HELPER="$cua_native_root/.build/release/dsh-cua-native"
pnpm dsh headless --patch packages/computer-use/cua/examples/macos.cordis.yml --dump-config

The dump command checks profile composition without a model request. For a real task, replace --dump-config with a prompt and configure the profile’s model credentials. The overlay mounts the computer-use registry and this provider; remove an existing competing provider before using it. The responsible host/helper needs Accessibility permission for AX/input and Screen Recording permission for capture. getCapabilities reports permissions and session lock state without opening a system dialog. Locked or unavailable desktop sessions reject UI observations and input before dispatch; the user must unlock macOS before resuming. Start the target application yourself. Select it by name, bundle ID, absolute application path, or PID. getApp reads the selected application without activating it. Model guidance starts with background observation and input, without activation during initialization or before each action. If a hidden window, another Space or a Stage Manager thumbnail prevents observation, explain the foreground-dependent step before app.activate() or an observed window’s raise action, then read fresh state. Applications may change focus after input; background delivery does not guarantee an undisturbed user session.

Field Default Meaning
allowedApps [] Live profile-local application consent, keyed by executable path and verified signing requirement.
helperPath package-owned executable Optional absolute native executable override.
buildTimeoutMs 300000 Maximum native preparation time.
timeoutMs 30000 Per-operation deadline; cancellation can leave partial input.
axTimeoutMs 3000 Deadline per Accessibility system call, configured independently of the complete operation.
maxRequestBytes / maxResponseBytes 1048576 / 33554432 UTF-8 request and response limits.
maxElements / maxDepth 2000 / 25 Accessibility traversal limits.
dragDurationMs / dragSteps 400 / 20 Duration and movement segments for a drag.
textEventIntervalMs 0.25 Delay after each Unicode down/up event, from 0 to 1000 ms. Chord timing is independent.

The configuration catalog lists all fields. Native tool calls use {method, params}; the complete method-specific schema is exposed to the model. For example, getApp returns the appHandle needed by listWindows, getState and input operations.

Keypad names (KP_0 through KP_9, KP_Enter, and keypad operators) retain their physical key codes. Key chords use + between modifiers and the key. Use + for the plus key, cmd++ for Command-plus, or the key name plus; an unfinished chord such as cmd+ is rejected.

Agent disposal and plugin unload cancel and await the complete tool call, including image admission and attachment storage. An image write already in progress must settle before disposal returns; a canceled call cannot return a successful screenshot.

Application consent is enforced before AX reads, window-title reads, screenshots, activation, and input through the CUA tool. User Questions offers task access or denial, plus saved access when the application has a verifiable signature and the composition provides settings persistence. Task access is bound to the selected process handle; saved access is bound to its canonical executable path and designated code-signing requirement. Denial prevents repeated prompts in that task. Removing a saved allowedApps entry cancels affected operations and invalidates their helper handles; reauthorization requires fresh selection. Other handles in the same helper also expire, while saved grants remain reusable. Agent/provider disposal joins helper cleanup. See the application consent decision. These checks govern CUA calls; persistent PTC and separately authorized shell tools retain their own execution authority.

Raw getState exposes the validated state fields and its complete tree. The native helper also returns structured element diagnostics for native checks; the tool projection omits that duplicate array from model text and structured tool values.

Native preparation serializes compiler checks and executable publication with the shared file lock, retains the native build cache, and replaces the executable atomically only when its bytes change. Cancellation and preparation failure finish managed process cleanup before releasing the lock.

Connect an external execution environment

To run through persistent PTC, add --patch packages/computer-use/cua/examples/persistent-ptc.cordis.yml after the macOS overlay. This provider keeps variables for the live Agent and runs without a file sandbox. The caller imports the bridge and supplies the current authorized tools.cua binding. Persistent PTC receives an installation-owned bridge file URL in its CUA instructions, so the project does not need to install this package. The example below is for an external host that resolves the package from its own installation and supplies tools; the package does not install a global object or keep variables alive:

import { connectCua } from '@deepseek-ai/dsh-cua/bridge'
const cua = connectCua(() => tools.cua, text => console.log(text))
const app = await cua.getApp("com.apple.TextEdit")
void await app.getScreenshot()

Use app methods directly for single-window tasks. getApp reads initial AX state, so a screenshot needs no window enumeration or additional AX read. Use app.getWindows() and app.getWindow(windowId) when the task needs a particular window or default selection is ambiguous; choose by the observed title and bounds.

getWindow emits its initial AX tree. An owned window without an AX surface returns an empty snapshot and remains available for screenshots and coordinate input. Window objects retain separate snapshots, so reading one window does not invalidate another. Re-reading the same scope invalidates old indices. An AX node removed during observation rejects the read with STALE_SNAPSHOT and publishes no partial tree; read again without repeating the preceding input. App-scoped screenshots and coordinates reuse the latest AX observation: its focused on-screen, nonzero-size window, otherwise the sole such window. They do not query current focus again; read app.getAXState() to follow a focus change. Explicit window objects retain their selected target. Keyboard methods stay on app because input follows the app’s current focus.

selectText selects a unique literal occurrence or places the cursor before/after it through writable AXSelectedTextRange. Adjacent prefix/suffix disambiguate repeats. Selection preserves the remaining text and formatting, and verifies the resulting UTF-16 range; unsupported, missing, ambiguous or changed text fails without keyboard fallback. AX text includes selectionWritable, selection ranges and selected text when available.

Screenshot data must use canonical Base64. Response byte limits apply before parsing, and attachment admission validates the raster and its configured capacity.

Capture and pointer input reject known disagreement between the selected window’s accessibility frame and displayed bounds with WINDOW_CHANGED, before posting pointer events. This includes Stage Manager transformations when AX retains the full window frame. Read fresh state; any activation or window raise remains explicit. Advertised AX actions remain available. Missing AX geometry retains the AXless path and does not establish that a transformed window can be operated.

Retained objects resolve the current tool binding on every call. The external runtime owns persistence, imports, cancellation and completion-value serialization. Screenshots return PNG bytes; the standard tool pipeline admits the image attachment exactly once. emit: false suppresses facade output only. A host that uses another dispatcher can build objects through createCuaApi from ./api.

AppKit sheets are selectable by their window IDs and remain visible in their document window’s tree. Keyboard input follows a system Open/Save panel only when the application’s focused sheet identifies one Apple panel-service window and that service still focuses the same panel. Unrelated or ambiguous remote recipients fail before dispatch.

Indexed clicks support a single right/middle click or a single/double left click. Click and drag options accept a button and modifier keys; a modified click uses pointer input. Drags hold that button and carry its modifiers through every movement and release. AX text marks disabled controls, valueWritable=true, and available selected state. The helper uses an advertised accessibility action for unmodified clicks when available; pointer delivery resolves the element’s current bounds and verifies its owning window. Actions report dispatch, not application success. A visual pointer or click marker can be rendered independently of input; AX actions do not require moving the system cursor. Window-directed mouse input can still change application focus, selection or dialogs. Read state after input. STALE_APP requires acquiring the application again; STALE_SNAPSHOT requires a fresh AX read. Transport timeout/cancellation terminates that Agent’s helper and invalidates its handles. Never retry uncertain input automatically. Closing an Agent releases its helper and AX references, without undoing changes to applications.


Understand the implementation

Dispatch and resource ownership

The object facade translates app/window methods into JSON operations. The bridge routes them through the normal tool pipeline, preserving authorization, nested logs and image admission. A separate serial helper belongs to each Agent. Native handles bind a process identity to its kernel start time, and snapshots retain native AX references only inside that helper. The same operation protocol can serve a future platform provider without adding platform logic to the object API.

Native request IDs match the logged tool call IDs. Successful activation and window-raise results retain foreground request details and immediate before/after focus observations in the raw tool result. An already-frontmost app skips redundant activation; a window raise is skipped only when the exact window is also focused, main, unminimized and first among visible normal windows. Unknown observations retain the requested operation. These observations do not establish who caused an unrelated focus change or verify completion.

Node roles and metadata share one AX batch per observation. Group and scroll-area nodes contribute only children; unused metadata errors do not reject them. Unsupported batching retains individual attribute reads. AX traversal visits each native identity once, including proxy references that compare equal, so cyclic accessibility children cannot repeat whole subtrees. The source manifest records adapted upstream behavior and local Swift modules. AppKit runs on the main event loop; a separate serial executor owns AX references and synchronous input. A request retains exclusive execution through capture and response delivery. No invariant companion is published: there is no independently projected registry or durable evaluator state to reconcile. Protocol validation, target identity and resource cleanup are checked by their owning implementations.


Further Exploration


Model Experience

Native application operations

What the model sees

The plugin contributes one cua operation tool with method-specific parameters. Standard PTC presentation exposes it as tools.cua through run_code. The following package-owned guidance is emitted verbatim:

Operation guidance (verbatim)
Use cua with method getCapabilities and params {} to inspect permissions and sessionLocked. If sessionLocked is true or an operation reports SESSION_LOCKED, stop UI attempts and ask the user to unlock macOS. A null sessionLocked means no usable console session was verified. After unlocking, read fresh state; screenshots from a locked desktop are not current observations. Use listApps and getApp to select an already running application. getApp requests user consent before application observations or input. If access is denied or revoked, stop using that application until the user explicitly grants access again. Never switch tools to bypass a refusal. Keep the returned appHandle; every app operation requires it. listWindows returns window IDs and point bounds. getState returns a complete AX tree and snapshotId. Use its element indices only with that snapshotId. Raw screenshot and coordinate operations require a windowId. Coordinates are window-local macOS points, which may differ from screenshot pixels. A verified window can expose no AX elements. Its empty snapshot permits screenshots and coordinate input; element-index actions require an element from a fresh AX observation.

Initialize the object API once in persistent PTC. Reuse its variables in later calls; put temporary declarations inside { ... } blocks to avoid redeclaration errors. All object methods are async. getAXState returns text, not a structured state object. Its text marks disabled controls, valueWritable and selectionWritable capabilities, selected state and UTF-16 selection ranges when available.

Use app methods directly for single-window tasks; getApp already reads initial AX state. Call app.getWindows() and await app.getWindow(windowId) only to target a particular window or resolve ambiguous window selection. App screenshots and coordinate actions use the latest AX observation: its focused on-screen, nonzero-size window, otherwise the sole such window. They do not refresh focus automatically. Read app.getAXState() to follow changed focus.

Object API signatures (options default to { emit: true }):

- cua.getCapabilities(options?) -> { platform, permissions, sessionLocked, methods, coordinateSpace, browser }
- cua.listApps(options?) -> Array<{ id, displayName, isRunning, pid }>
- cua.getState(options?) -> { apps, browsers: [] }
- cua.getApp(selector: string) -> App
- app.activate() -> { dispatched: true, verified: false }
- app.getWindows(options?) -> Array<{ windowId, pid, appName, title, bounds, isOnScreen, onCurrentSpace }>
- app.getWindow(windowId: number) -> Window
- app.typeText(text: string), app.paste(text: string), app.pressKey(chord: string) -> action result

App and Window methods:

- getAXState(options?: { emit?: boolean, disableDiffing?: boolean }) -> string
- getScreenshot(options?: { emit?: boolean }) -> Uint8Array
- getAXStateAndScreenshot(options?) -> { state: string, screenshot: Uint8Array }
- click(target: number | [x, y], options?: { mouseButton?: "left" | "right" | "middle", clickCount?: 1 | 2, modifiers?: Array<"command" | "control" | "option" | "shift" | "fn"> }) -> action result
- setValue(elementIndex: number, value: string) -> action result
- selectText(elementIndex: number, text: string, options?: { prefix?: string, suffix?: string, selectionType?: "text" | "cursor_before" | "cursor_after" }) -> action result
- performSecondaryAction(elementIndex: number, action: string) -> action result
- scroll({ x, y, deltaLinesX, deltaLinesY }) -> action result
- drag(from: [x, y], to: [x, y], options?: { mouseButton?: "left" | "right" | "middle", modifiers?: Array<"command" | "control" | "option" | "shift" | "fn"> }) -> action result

Raw operation tools use { method, params }, including params: {} for parameterless methods. getApp selects without activating the app. Keep applications in the background for observation and routine input. Do not activate or raise an app during initialization, before each action, or merely because another app is foreground. Prefer advertised AX actions. If a hidden window, another Space or Stage Manager prevents usable observation, explain the foreground-dependent step before app.activate() or an advertised raise action, then read fresh state. Avoid repeated activation when the user switches away; continue background-capable steps and report what requires foreground access. Applications may still change focus in response to input.

Await operations sequentially. After input, read fresh state to verify the effect. A successful action means dispatched, not verified completion. Cancellation may leave partial effects; never retry uncertain input without observing. STALE_APP requires getApp again; STALE_SNAPSHOT requires getState again. WINDOW_CHANGED requires fresh state before any retry; use an advertised AX action or explicitly restore the window before screenshot or coordinate input, then observe again. PERMISSION_DENIED requires the user to grant macOS Accessibility or Screen Recording access in System Settings. After granting access, observe fresh state before continuing; do not automatically replay prior input.

Use selectText only on controls advertising selectionWritable. It selects a unique literal occurrence, or places the cursor before/after it; prefix and suffix must be immediately adjacent. AMBIGUOUS_TEXT requires more context; TEXT_NOT_FOUND or SELECTION_CHANGED requires a fresh observation before deciding on input. Left clicks support single/double clicks; right and middle clicks support single clicks. Modified clicks use pointer input. Keypad names such as KP_0 and KP_Enter retain their physical key identity. Window objects expose observation and pointer/AX actions; keyboard input remains app-scoped and follows current focus. Scroll deltas are integer lines from -10 to 10; positive values reveal content above or left. Paste inserts plain text using native Unicode input without changing the clipboard; rich-text formats are unavailable. Browser applications can be controlled with getApp like other native apps; browser tabs and app launching are unavailable. The execution host must install the object API and own variable persistence; this plugin does not preload a cua global. In persistent PTC, use the host-provided initialization snippet below once. Its installation-owned file URL works independently of the project directory. Retain app objects between calls. End screenshot calls with void await app.getScreenshot() or void await app.getAXStateAndScreenshot() so image bytes are not serialized as the program completion value.

Token effect

Enabling the plugin adds a fixed schema and guidance. AX observations, operation results and admitted images add model input. Retained objects are not serialized into prompts by this package.

KV Cache effect

Tool results append to Session history. Changing plugin configuration or tool presentation can change the request prefix; retaining an app object does not change existing history.

Known Limitations and Deferred Work

  • Only the macOS native implementation is provided. Browser tabs, app launching, rich-text clipboard formats, AX diffs, unassociated system panels and accessory-only apps are unsupported.
  • Coordinates use window-local points including the title bar. PNG pixel dimensions can differ. Window movement, application changes and other desktop users can race observations and input.
  • macOS Stage Manager can reduce background window bounds and screenshots to thumbnails. Full-window observation and coordinate input in this state are not guaranteed; model guidance requires explaining foreground needs and avoiding repeated focus reclamation.
  • Background input depends on private SkyLight symbols; macOS updates can make them unavailable. Validate each target OS version.
  • Observed failures include dispatched Unicode text not appearing in Blender and background Command+A not selecting all text in AppKit. Inspect application state after input; a successful tool result confirms dispatch only.
  • Native binaries require a local build. Signing, distribution, and system-permission onboarding remain deferred.
  • Persistent PTC is an external dependency assumption, not an implementation in this package. Real-model task success requires separate evaluation beyond native and protocol fixtures.

Dev Note

Working context for maintainers

None.