feat(cua): expose plain-text paste and persistent PTC composition
This commit is contained in:
1 parent
9bd3aca35e
commit
56f3d3135d
9 files changed
+46
-10
No files matched your search
@@ -2,5 +2,5 @@
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write packages/experimental/cua/README.md
|
||||
README.md: a9d43bcb031971cdef1aaf78d05411ae774ed816
|
||||
README.zh.md: c1fe27d903a29b6755579dffe64554c696354a5b
|
||||
README.md: 04209c613df7024f696b19325ee1c3144bab4e0b
|
||||
README.zh.md: e7c4531ed925e7bc287d1af89eedf32d68a1f3e8
|
||||
@@ -48,7 +48,7 @@ The [configuration catalog](../../../docs/config-catalog.md#deepseek-aidsh-exper
|
||||
|
||||
### Connect an external execution environment
|
||||
|
||||
The caller imports the bridge and supplies the current authorized `tools.cua` binding. This integration snippet assumes an execution host has supplied `tools`; the package does not install a global object or keep variables alive:
|
||||
To run through persistent PTC, add `--patch packages/experimental/cua/examples/persistent-ptc.cordis.yml` after the macOS overlay. This provider keeps variables for the live Agent and runs without a file sandbox. The caller imports the bridge and supplies the current authorized `tools.cua` binding. This integration snippet assumes an execution host has supplied `tools`; the package does not install a global object or keep variables alive:
|
||||
|
||||
```javascript
|
||||
import { connectCua } from '@deepseek-ai/dsh-experimental-cua/bridge'
|
||||
@@ -106,7 +106,7 @@ Use cua with method getCapabilities to inspect permissions, then listApps and ge
|
||||
|
||||
Await operations sequentially. After input, read fresh state to verify the effect. A successful action means dispatched, not verified completion. Cancellation may leave partial effects; never retry uncertain input without observing. STALE_APP requires getApp again; STALE_SNAPSHOT requires getState again. PERMISSION_DENIED requires the user to grant macOS Accessibility or Screen Recording permission.
|
||||
|
||||
The optional object API exposes getCapabilities, listApps, getState and getApp. App objects expose getWindows, getWindow(windowId), getAXState, getScreenshot, getAXStateAndScreenshot, click, setValue, scroll, drag, typeText and pressKey. Window objects expose observation and pointer/AX actions; keyboard input remains app-scoped and follows current focus. Scroll deltas are integer lines from -10 to 10; positive values reveal content above or left. Browser tabs, app launching and clipboard paste are unavailable. The execution host must install the object API and own variable persistence; this plugin does not preload a cua global.
|
||||
The optional object API exposes getCapabilities, listApps, getState and getApp. App objects expose getWindows, getWindow(windowId), getAXState, getScreenshot, getAXStateAndScreenshot, click, setValue, scroll, drag, typeText, paste and pressKey. Window objects expose observation and pointer/AX actions; keyboard input remains app-scoped and follows current focus. Scroll deltas are integer lines from -10 to 10; positive values reveal content above or left. Paste inserts plain text using native Unicode input without changing the clipboard; rich-text formats are unavailable. Browser applications can be controlled with getApp like other native apps; browser tabs and app launching are unavailable. The execution host must install the object API and own variable persistence; this plugin does not preload a cua global. In persistent PTC, initialize once with const { connectCua } = await import("@deepseek-ai/dsh-experimental-cua/bridge"); const cua = connectCua(() => tools.cua, text => console.log(text)). Retain app objects between calls. End screenshot calls with void await app.getScreenshot() or void await app.getAXStateAndScreenshot() so image bytes are not serialized as the program completion value.
|
||||
```
|
||||
|
||||
#### Token effect
|
||||
@@ -121,7 +121,7 @@ Tool results append to Session history. Changing plugin configuration or tool pr
|
||||
|
||||
<a id="known-limitations-and-deferred-work"></a>
|
||||
|
||||
- Only the macOS native implementation is provided. Browser tabs, app launching, clipboard paste, AX diffs, separate-process Open/Save panels and accessory-only apps are unsupported.
|
||||
- Only the macOS native implementation is provided. Browser tabs, app launching, rich-text clipboard formats, AX diffs, separate-process Open/Save panels and accessory-only apps are unsupported.
|
||||
- Coordinates use window-local points including the title bar. PNG pixel dimensions can differ. Window movement, application changes and other desktop users can race observations and input.
|
||||
- Native binaries require a local build. Signing, distribution and permission onboarding remain future work.
|
||||
- Persistent PTC is an external dependency assumption, not an implementation in this package. Real-model task success requires separate evaluation beyond native and protocol fixtures.
|
||||
|
||||
@@ -48,7 +48,7 @@ pnpm dsh headless --patch packages/experimental/cua/examples/macos.cordis.yml --
|
||||
|
||||
### 接入外部执行环境
|
||||
|
||||
调用方导入适配模块,并提供当前执行中已授权的 `tools.cua` 绑定。下面的接入代码假设执行宿主已提供 `tools`;本包不安装全局对象,也不负责保留变量:
|
||||
使用 persistent PTC 时,在 macOS 配置后追加 `--patch packages/experimental/cua/examples/persistent-ptc.cordis.yml`。该 provider 为存活的 Agent 保留变量,不启用文件沙箱。调用方导入适配模块,并提供当前执行中已授权的 `tools.cua` 绑定。下面的接入代码假设执行宿主已提供 `tools`;本包不安装全局对象,也不负责保留变量:
|
||||
|
||||
```javascript
|
||||
import { connectCua } from '@deepseek-ai/dsh-experimental-cua/bridge'
|
||||
@@ -106,7 +106,7 @@ Use cua with method getCapabilities to inspect permissions, then listApps and ge
|
||||
|
||||
Await operations sequentially. After input, read fresh state to verify the effect. A successful action means dispatched, not verified completion. Cancellation may leave partial effects; never retry uncertain input without observing. STALE_APP requires getApp again; STALE_SNAPSHOT requires getState again. PERMISSION_DENIED requires the user to grant macOS Accessibility or Screen Recording permission.
|
||||
|
||||
The optional object API exposes getCapabilities, listApps, getState and getApp. App objects expose getWindows, getWindow(windowId), getAXState, getScreenshot, getAXStateAndScreenshot, click, setValue, scroll, drag, typeText and pressKey. Window objects expose observation and pointer/AX actions; keyboard input remains app-scoped and follows current focus. Scroll deltas are integer lines from -10 to 10; positive values reveal content above or left. Browser tabs, app launching and clipboard paste are unavailable. The execution host must install the object API and own variable persistence; this plugin does not preload a cua global.
|
||||
The optional object API exposes getCapabilities, listApps, getState and getApp. App objects expose getWindows, getWindow(windowId), getAXState, getScreenshot, getAXStateAndScreenshot, click, setValue, scroll, drag, typeText, paste and pressKey. Window objects expose observation and pointer/AX actions; keyboard input remains app-scoped and follows current focus. Scroll deltas are integer lines from -10 to 10; positive values reveal content above or left. Paste inserts plain text using native Unicode input without changing the clipboard; rich-text formats are unavailable. Browser applications can be controlled with getApp like other native apps; browser tabs and app launching are unavailable. The execution host must install the object API and own variable persistence; this plugin does not preload a cua global. In persistent PTC, initialize once with const { connectCua } = await import("@deepseek-ai/dsh-experimental-cua/bridge"); const cua = connectCua(() => tools.cua, text => console.log(text)). Retain app objects between calls. End screenshot calls with void await app.getScreenshot() or void await app.getAXStateAndScreenshot() so image bytes are not serialized as the program completion value.
|
||||
```
|
||||
|
||||
#### Token 影响
|
||||
@@ -121,7 +121,7 @@ The optional object API exposes getCapabilities, listApps, getState and getApp.
|
||||
|
||||
<a id="known-limitations-and-deferred-work"></a>
|
||||
|
||||
- 只提供 macOS 原生实现。不支持浏览器标签、启动应用、剪贴板粘贴、AX 差分、独立进程中的打开或保存对话框,以及仅后台辅助类型的应用。
|
||||
- 只提供 macOS 原生实现。不支持浏览器标签、启动应用、富文本剪贴板格式、AX 差分、独立进程中的打开或保存对话框,以及仅后台辅助类型的应用。
|
||||
- 坐标使用包含标题栏的窗口内部 point,可能不同于 PNG 像素尺寸。窗口移动、应用变化及其他桌面使用者的操作可能与观察和输入交错。
|
||||
- 原生二进制需要本地构建。签名、分发和权限引导尚未实现。
|
||||
- 持久 PTC 是对外部运行环境的假设,本包不实现它。真实模型任务成功率需要在原生测试和协议测试之外单独评估。
|
||||
|
||||
@@ -0,0 +1,8 @@
|
||||
- id: ptc-runtime
|
||||
disabled: true
|
||||
- id: tools
|
||||
config:
|
||||
mode: ptc
|
||||
- insert:
|
||||
- id: ptc-runtime-persistent
|
||||
name: '@deepseek-ai/dsh-ptc-runtime-node/persistent'
|
||||
@@ -144,6 +144,7 @@ export function createCuaApi(dispatch: CuaDispatch, sink?: CuaObservationSink):
|
||||
return window
|
||||
},
|
||||
async typeText(text) { return action('typeText', { appHandle, text }) },
|
||||
async paste(text) { return action('typeText', { appHandle, text }) },
|
||||
async pressKey(key) { return action('pressKey', { appHandle, ...keyChord(key) }) },
|
||||
}
|
||||
await app.getAXState()
|
||||
|
||||
@@ -58,7 +58,7 @@ export const GUIDANCE = `Use cua with method getCapabilities to inspect permissi
|
||||
|
||||
Await operations sequentially. After input, read fresh state to verify the effect. A successful action means dispatched, not verified completion. Cancellation may leave partial effects; never retry uncertain input without observing. STALE_APP requires getApp again; STALE_SNAPSHOT requires getState again. PERMISSION_DENIED requires the user to grant macOS Accessibility or Screen Recording permission.
|
||||
|
||||
The optional object API exposes getCapabilities, listApps, getState and getApp. App objects expose getWindows, getWindow(windowId), getAXState, getScreenshot, getAXStateAndScreenshot, click, setValue, scroll, drag, typeText and pressKey. Window objects expose observation and pointer/AX actions; keyboard input remains app-scoped and follows current focus. Scroll deltas are integer lines from -10 to 10; positive values reveal content above or left. Browser tabs, app launching and clipboard paste are unavailable. The execution host must install the object API and own variable persistence; this plugin does not preload a cua global.`
|
||||
The optional object API exposes getCapabilities, listApps, getState and getApp. App objects expose getWindows, getWindow(windowId), getAXState, getScreenshot, getAXStateAndScreenshot, click, setValue, scroll, drag, typeText, paste and pressKey. Window objects expose observation and pointer/AX actions; keyboard input remains app-scoped and follows current focus. Scroll deltas are integer lines from -10 to 10; positive values reveal content above or left. Paste inserts plain text using native Unicode input without changing the clipboard; rich-text formats are unavailable. Browser applications can be controlled with getApp like other native apps; browser tabs and app launching are unavailable. The execution host must install the object API and own variable persistence; this plugin does not preload a cua global. In persistent PTC, initialize once with const { connectCua } = await import("@deepseek-ai/dsh-experimental-cua/bridge"); const cua = connectCua(() => tools.cua, text => console.log(text)). Retain app objects between calls. End screenshot calls with void await app.getScreenshot() or void await app.getAXStateAndScreenshot() so image bytes are not serialized as the program completion value.`
|
||||
|
||||
/**
|
||||
* Mount logged native operations with a separate helper per agent.
|
||||
|
||||
@@ -129,6 +129,12 @@ export interface CuaApp extends CuaSurface {
|
||||
* @returns dispatch evidence.
|
||||
*/
|
||||
typeText(text: string): Promise<CuaActionResult>
|
||||
/**
|
||||
* Insert plain text at current focus through native Unicode input, preserving the system clipboard.
|
||||
* @param text - Literal text; rich-text clipboard formats are unsupported.
|
||||
* @returns Dispatch evidence; cancellation may leave a prefix entered.
|
||||
*/
|
||||
paste(text: string): Promise<CuaActionResult>
|
||||
/**
|
||||
* Send a key/chord at current focus.
|
||||
* @param key - xdotool-style chord such as super+c or shift+Tab.
|
||||
|
||||
@@ -314,10 +314,12 @@ describe('input actions', () => {
|
||||
const f = fixture()
|
||||
const app = await f.api.getApp('Editor')
|
||||
await app.typeText('中文\nline two')
|
||||
await app.paste('粘贴\nplain text')
|
||||
await app.pressKey('Control_L+shift+a')
|
||||
await app.pressKey('Return')
|
||||
expect(f.dispatch.mock.calls.slice(-3).map(([operation]) => operation)).toEqual([
|
||||
expect(f.dispatch.mock.calls.slice(-4).map(([operation]) => operation)).toEqual([
|
||||
{ method: 'typeText', params: { appHandle: 'app-17', text: '中文\nline two' } },
|
||||
{ method: 'typeText', params: { appHandle: 'app-17', text: '粘贴\nplain text' } },
|
||||
{ method: 'pressKey', params: { appHandle: 'app-17', key: 'a', modifiers: ['control', 'shift'] } },
|
||||
{ method: 'pressKey', params: { appHandle: 'app-17', key: 'Return', modifiers: [] } },
|
||||
])
|
||||
|
||||
@@ -219,6 +219,25 @@
|
||||
"type": "integer",
|
||||
"minimum": 0,
|
||||
"maximum": 9007199254740991
|
||||
},
|
||||
"mouseButton": {
|
||||
"type": "string",
|
||||
"enum": [
|
||||
"left",
|
||||
"right"
|
||||
]
|
||||
},
|
||||
"clickCount": {
|
||||
"anyOf": [
|
||||
{
|
||||
"type": "number",
|
||||
"const": 1
|
||||
},
|
||||
{
|
||||
"type": "number",
|
||||
"const": 2
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
|
||||
Reference in new issue
Block a user