feat(cua): expose plain-text paste and persistent PTC composition

This commit is contained in:
creatixchu committed 2026-09-23 20:12:21 +08:00
1 parent 9bd3aca35e
commit 56f3d3135d
9 files changed
+46 -10

No files matched your search

+2 -2
View File
@@ -2,5 +2,5 @@
# side as of the last confirmed-consistent state. Both languages carry equal authority;
# after editing either side, bring the other along and re-record with:
# pnpm run verify-translation-pairing --write packages/experimental/cua/README.md
README.md: a9d43bcb031971cdef1aaf78d05411ae774ed816
README.zh.md: c1fe27d903a29b6755579dffe64554c696354a5b
README.md: 04209c613df7024f696b19325ee1c3144bab4e0b
README.zh.md: e7c4531ed925e7bc287d1af89eedf32d68a1f3e8
+3 -3
View File
@@ -48,7 +48,7 @@ The [configuration catalog](../../../docs/config-catalog.md#deepseek-aidsh-exper
### Connect an external execution environment
The caller imports the bridge and supplies the current authorized `tools.cua` binding. This integration snippet assumes an execution host has supplied `tools`; the package does not install a global object or keep variables alive:
To run through persistent PTC, add `--patch packages/experimental/cua/examples/persistent-ptc.cordis.yml` after the macOS overlay. This provider keeps variables for the live Agent and runs without a file sandbox. The caller imports the bridge and supplies the current authorized `tools.cua` binding. This integration snippet assumes an execution host has supplied `tools`; the package does not install a global object or keep variables alive:
```javascript
import { connectCua } from '@deepseek-ai/dsh-experimental-cua/bridge'
@@ -106,7 +106,7 @@ Use cua with method getCapabilities to inspect permissions, then listApps and ge
Await operations sequentially. After input, read fresh state to verify the effect. A successful action means dispatched, not verified completion. Cancellation may leave partial effects; never retry uncertain input without observing. STALE_APP requires getApp again; STALE_SNAPSHOT requires getState again. PERMISSION_DENIED requires the user to grant macOS Accessibility or Screen Recording permission.
The optional object API exposes getCapabilities, listApps, getState and getApp. App objects expose getWindows, getWindow(windowId), getAXState, getScreenshot, getAXStateAndScreenshot, click, setValue, scroll, drag, typeText and pressKey. Window objects expose observation and pointer/AX actions; keyboard input remains app-scoped and follows current focus. Scroll deltas are integer lines from -10 to 10; positive values reveal content above or left. Browser tabs, app launching and clipboard paste are unavailable. The execution host must install the object API and own variable persistence; this plugin does not preload a cua global.
The optional object API exposes getCapabilities, listApps, getState and getApp. App objects expose getWindows, getWindow(windowId), getAXState, getScreenshot, getAXStateAndScreenshot, click, setValue, scroll, drag, typeText, paste and pressKey. Window objects expose observation and pointer/AX actions; keyboard input remains app-scoped and follows current focus. Scroll deltas are integer lines from -10 to 10; positive values reveal content above or left. Paste inserts plain text using native Unicode input without changing the clipboard; rich-text formats are unavailable. Browser applications can be controlled with getApp like other native apps; browser tabs and app launching are unavailable. The execution host must install the object API and own variable persistence; this plugin does not preload a cua global. In persistent PTC, initialize once with const { connectCua } = await import("@deepseek-ai/dsh-experimental-cua/bridge"); const cua = connectCua(() => tools.cua, text => console.log(text)). Retain app objects between calls. End screenshot calls with void await app.getScreenshot() or void await app.getAXStateAndScreenshot() so image bytes are not serialized as the program completion value.
```
#### Token effect
@@ -121,7 +121,7 @@ Tool results append to Session history. Changing plugin configuration or tool pr
<a id="known-limitations-and-deferred-work"></a>
- Only the macOS native implementation is provided. Browser tabs, app launching, clipboard paste, AX diffs, separate-process Open/Save panels and accessory-only apps are unsupported.
- Only the macOS native implementation is provided. Browser tabs, app launching, rich-text clipboard formats, AX diffs, separate-process Open/Save panels and accessory-only apps are unsupported.
- Coordinates use window-local points including the title bar. PNG pixel dimensions can differ. Window movement, application changes and other desktop users can race observations and input.
- Native binaries require a local build. Signing, distribution and permission onboarding remain future work.
- Persistent PTC is an external dependency assumption, not an implementation in this package. Real-model task success requires separate evaluation beyond native and protocol fixtures.
+3 -3
View File
@@ -48,7 +48,7 @@ pnpm dsh headless --patch packages/experimental/cua/examples/macos.cordis.yml --
### 接入外部执行环境
调用方导入适配模块,并提供当前执行中已授权的 `tools.cua` 绑定。下面的接入代码假设执行宿主已提供 `tools`;本包不安装全局对象,也不负责保留变量:
使用 persistent PTC 时,在 macOS 配置后追加 `--patch packages/experimental/cua/examples/persistent-ptc.cordis.yml`。该 provider 为存活的 Agent 保留变量,不启用文件沙箱。调用方导入适配模块,并提供当前执行中已授权的 `tools.cua` 绑定。下面的接入代码假设执行宿主已提供 `tools`;本包不安装全局对象,也不负责保留变量:
```javascript
import { connectCua } from '@deepseek-ai/dsh-experimental-cua/bridge'
@@ -106,7 +106,7 @@ Use cua with method getCapabilities to inspect permissions, then listApps and ge
Await operations sequentially. After input, read fresh state to verify the effect. A successful action means dispatched, not verified completion. Cancellation may leave partial effects; never retry uncertain input without observing. STALE_APP requires getApp again; STALE_SNAPSHOT requires getState again. PERMISSION_DENIED requires the user to grant macOS Accessibility or Screen Recording permission.
The optional object API exposes getCapabilities, listApps, getState and getApp. App objects expose getWindows, getWindow(windowId), getAXState, getScreenshot, getAXStateAndScreenshot, click, setValue, scroll, drag, typeText and pressKey. Window objects expose observation and pointer/AX actions; keyboard input remains app-scoped and follows current focus. Scroll deltas are integer lines from -10 to 10; positive values reveal content above or left. Browser tabs, app launching and clipboard paste are unavailable. The execution host must install the object API and own variable persistence; this plugin does not preload a cua global.
The optional object API exposes getCapabilities, listApps, getState and getApp. App objects expose getWindows, getWindow(windowId), getAXState, getScreenshot, getAXStateAndScreenshot, click, setValue, scroll, drag, typeText, paste and pressKey. Window objects expose observation and pointer/AX actions; keyboard input remains app-scoped and follows current focus. Scroll deltas are integer lines from -10 to 10; positive values reveal content above or left. Paste inserts plain text using native Unicode input without changing the clipboard; rich-text formats are unavailable. Browser applications can be controlled with getApp like other native apps; browser tabs and app launching are unavailable. The execution host must install the object API and own variable persistence; this plugin does not preload a cua global. In persistent PTC, initialize once with const { connectCua } = await import("@deepseek-ai/dsh-experimental-cua/bridge"); const cua = connectCua(() => tools.cua, text => console.log(text)). Retain app objects between calls. End screenshot calls with void await app.getScreenshot() or void await app.getAXStateAndScreenshot() so image bytes are not serialized as the program completion value.
```
#### Token 影响
@@ -121,7 +121,7 @@ The optional object API exposes getCapabilities, listApps, getState and getApp.
<a id="known-limitations-and-deferred-work"></a>
- 只提供 macOS 原生实现。不支持浏览器标签、启动应用、剪贴板粘贴、AX 差分、独立进程中的打开或保存对话框,以及仅后台辅助类型的应用。
- 只提供 macOS 原生实现。不支持浏览器标签、启动应用、富文本剪贴板格式、AX 差分、独立进程中的打开或保存对话框,以及仅后台辅助类型的应用。
- 坐标使用包含标题栏的窗口内部 point,可能不同于 PNG 像素尺寸。窗口移动、应用变化及其他桌面使用者的操作可能与观察和输入交错。
- 原生二进制需要本地构建。签名、分发和权限引导尚未实现。
- 持久 PTC 是对外部运行环境的假设,本包不实现它。真实模型任务成功率需要在原生测试和协议测试之外单独评估。
@@ -0,0 +1,8 @@
- id: ptc-runtime
disabled: true
- id: tools
config:
mode: ptc
- insert:
- id: ptc-runtime-persistent
name: '@deepseek-ai/dsh-ptc-runtime-node/persistent'
+1
View File
@@ -144,6 +144,7 @@ export function createCuaApi(dispatch: CuaDispatch, sink?: CuaObservationSink):
return window
},
async typeText(text) { return action('typeText', { appHandle, text }) },
async paste(text) { return action('typeText', { appHandle, text }) },
async pressKey(key) { return action('pressKey', { appHandle, ...keyChord(key) }) },
}
await app.getAXState()
+1 -1
View File
@@ -58,7 +58,7 @@ export const GUIDANCE = `Use cua with method getCapabilities to inspect permissi
Await operations sequentially. After input, read fresh state to verify the effect. A successful action means dispatched, not verified completion. Cancellation may leave partial effects; never retry uncertain input without observing. STALE_APP requires getApp again; STALE_SNAPSHOT requires getState again. PERMISSION_DENIED requires the user to grant macOS Accessibility or Screen Recording permission.
The optional object API exposes getCapabilities, listApps, getState and getApp. App objects expose getWindows, getWindow(windowId), getAXState, getScreenshot, getAXStateAndScreenshot, click, setValue, scroll, drag, typeText and pressKey. Window objects expose observation and pointer/AX actions; keyboard input remains app-scoped and follows current focus. Scroll deltas are integer lines from -10 to 10; positive values reveal content above or left. Browser tabs, app launching and clipboard paste are unavailable. The execution host must install the object API and own variable persistence; this plugin does not preload a cua global.`
The optional object API exposes getCapabilities, listApps, getState and getApp. App objects expose getWindows, getWindow(windowId), getAXState, getScreenshot, getAXStateAndScreenshot, click, setValue, scroll, drag, typeText, paste and pressKey. Window objects expose observation and pointer/AX actions; keyboard input remains app-scoped and follows current focus. Scroll deltas are integer lines from -10 to 10; positive values reveal content above or left. Paste inserts plain text using native Unicode input without changing the clipboard; rich-text formats are unavailable. Browser applications can be controlled with getApp like other native apps; browser tabs and app launching are unavailable. The execution host must install the object API and own variable persistence; this plugin does not preload a cua global. In persistent PTC, initialize once with const { connectCua } = await import("@deepseek-ai/dsh-experimental-cua/bridge"); const cua = connectCua(() => tools.cua, text => console.log(text)). Retain app objects between calls. End screenshot calls with void await app.getScreenshot() or void await app.getAXStateAndScreenshot() so image bytes are not serialized as the program completion value.`
/**
* Mount logged native operations with a separate helper per agent.
+6
View File
@@ -129,6 +129,12 @@ export interface CuaApp extends CuaSurface {
* @returns dispatch evidence.
*/
typeText(text: string): Promise<CuaActionResult>
/**
* Insert plain text at current focus through native Unicode input, preserving the system clipboard.
* @param text - Literal text; rich-text clipboard formats are unsupported.
* @returns Dispatch evidence; cancellation may leave a prefix entered.
*/
paste(text: string): Promise<CuaActionResult>
/**
* Send a key/chord at current focus.
* @param key - xdotool-style chord such as super+c or shift+Tab.
+3 -1
View File
@@ -314,10 +314,12 @@ describe('input actions', () => {
const f = fixture()
const app = await f.api.getApp('Editor')
await app.typeText('中文\nline two')
await app.paste('粘贴\nplain text')
await app.pressKey('Control_L+shift+a')
await app.pressKey('Return')
expect(f.dispatch.mock.calls.slice(-3).map(([operation]) => operation)).toEqual([
expect(f.dispatch.mock.calls.slice(-4).map(([operation]) => operation)).toEqual([
{ method: 'typeText', params: { appHandle: 'app-17', text: '中文\nline two' } },
{ method: 'typeText', params: { appHandle: 'app-17', text: '粘贴\nplain text' } },
{ method: 'pressKey', params: { appHandle: 'app-17', key: 'a', modifiers: ['control', 'shift'] } },
{ method: 'pressKey', params: { appHandle: 'app-17', key: 'Return', modifiers: [] } },
])
@@ -219,6 +219,25 @@
"type": "integer",
"minimum": 0,
"maximum": 9007199254740991
},
"mouseButton": {
"type": "string",
"enum": [
"left",
"right"
]
},
"clickCount": {
"anyOf": [
{
"type": "number",
"const": 1
},
{
"type": "number",
"const": 2
}
]
}
},
"required": [