chore(ptc): align fixtures and docs with simplified image handling
This commit is contained in:
1 parent
724e56b286
commit
4e72e43037
28 files changed
+97
-91
No files matched your search
@@ -12,8 +12,8 @@
|
||||
en: 182147ddf388c600
|
||||
zh: c0a7d77602c78bcd
|
||||
/ptc-runtime/captured-output-and-the-failure-taxonomy:
|
||||
en: 2c142aef2340f62f
|
||||
zh: 707ae1c35ab1e5ab
|
||||
en: 033ce544b9974ba8
|
||||
zh: b7be881985ca3c34
|
||||
/ptc-runtime/the-service:
|
||||
en: a82f828fa006c7e9
|
||||
zh: 2279d36867335b57
|
||||
@@ -186,7 +186,7 @@ type PtcBindingFunction = (args: unknown) => Promise<PtcJsonValue>
|
||||
|
||||
## Captured output and the failure taxonomy
|
||||
|
||||
Logs are plain strings. Each source channel preserves emission order; interleaving across independent channels is backend-dependent. The Node provider also captures explicitly inspected images as `PtcImageOutput` entries, whose `afterLog` positions place them among text entries. Each host-issued `PtcImage` exposes a readable execution-world `path` and the stored pixel dimensions. The Node provider caps serialized logs, image descriptors, and completion or diagnostic payloads together; overflow is an explicit failure.
|
||||
Logs are plain strings. Each source channel preserves emission order; interleaving across independent channels is backend-dependent. The Node provider also captures explicitly inspected images as `PtcImageOutput` entries, whose `afterLog` positions place their attachment references among text entries. Host bindings return `PtcImage` descriptors containing the attachment reference and a readable execution-world `path`; the program receives an `Image` with that path and the stored pixel dimensions. The Node provider caps serialized logs, image attachment references, and completion or diagnostic payloads together; overflow is an explicit failure.
|
||||
|
||||
Failure kinds are **orthogonal outcomes reported independently** (per [defensive-patterns](../defensive-patterns/general.md)): a budget expiry is not an exception, an abort is not a timeout, and a substrate death (e.g. OOM) is neither:
|
||||
|
||||
|
||||
@@ -186,7 +186,7 @@ type PtcBindingFunction = (args: unknown) => Promise<PtcJsonValue>
|
||||
|
||||
## 捕获的输出与失败分类体系
|
||||
|
||||
日志是普通字符串。每个输出通道保留发出顺序;独立通道之间的交错取决于后端。Node 提供方还将显式展示的图片记录为 `PtcImageOutput`,通过 `afterLog` 将图片定位在文本条目之间。宿主创建的每个 `PtcImage` 提供执行环境可读取的 `path` 和已存储图片的像素尺寸。Node 提供方对序列化日志、图片描述以及完成值或诊断信息统一计算预算,超限会明确失败。
|
||||
日志是普通字符串。每个输出通道保留发出顺序;独立通道之间的交错取决于后端。Node 提供方还将显式展示的图片记录为 `PtcImageOutput`,通过 `afterLog` 将附件引用定位在文本条目之间。宿主绑定返回包含附件引用及执行环境可读取的 `path` 的 `PtcImage` 描述;程序接收包含该路径和已存储图片像素尺寸的 `Image`。Node 提供方对序列化日志、图片附件引用以及完成值或诊断信息统一计算预算,超限会明确失败。
|
||||
|
||||
失败类型是**正交的结果,独立报告**(见 [defensive-patterns](../defensive-patterns/general.zh.md)):预算耗尽不是异常,中止不是超时,基底崩溃(如 OOM)也不是二者中的任何一个:
|
||||
|
||||
|
||||
@@ -12,5 +12,5 @@
|
||||
en: 3c69ee58e2670c62
|
||||
zh: 5876ec7a7d7d66a6
|
||||
/explicit-output-in-typescript-ptc-programs/migration:
|
||||
en: 0a72bb2c0f388ebf
|
||||
zh: 166d5fac652ffdc3
|
||||
en: 7d7ff08089e5e34e
|
||||
zh: 4f73b263ee518b76
|
||||
@@ -24,7 +24,7 @@ CUA screenshots and `read_image` return ordinary `Image` values in TypeScript pr
|
||||
inspect(result)
|
||||
```
|
||||
|
||||
2. Update CUA bridge callbacks to `text => inspect(text)`.
|
||||
2. Update CUA bridge callbacks to `text => inspect(text)`. Direct `createCuaApi(dispatch, onText?)` callers supply a dispatcher that returns program Images for screenshots and pass the text callback as the second argument. Call `window.getScreenshot()` and pass its returned Image to `inspect`.
|
||||
3. For integrations that require console methods, set `enableConsole: true` on the `@deepseek-ai/dsh-ptc-runtime-node` provider row. Direct runtime consumers can set `discardCompletion: true` to select the same explicit-output behavior.
|
||||
4. Run an updated program and verify that inspected values appear once and its final expression adds no output.
|
||||
|
||||
|
||||
@@ -24,7 +24,7 @@ CUA 截图与 `read_image` 在 TypeScript 程序中返回普通 `Image`,其 `p
|
||||
inspect(result)
|
||||
```
|
||||
|
||||
2. 将 CUA 桥接回调更新为 `text => inspect(text)`。
|
||||
2. 将 CUA 桥接回调更新为 `text => inspect(text)`。直接调用 `createCuaApi(dispatch, onText?)` 时,提供截图返回程序 `Image` 的分派器,并将文本回调作为第二个参数。调用 `window.getScreenshot()`,再将返回的 `Image` 传给 `inspect`。
|
||||
3. 对于需要 console 方法的集成,在 `@deepseek-ai/dsh-ptc-runtime-node` 提供方配置行中设置 `enableConsole: true`。直接调用运行时的消费方可设置 `discardCompletion: true`,选择相同的显式输出行为。
|
||||
4. 运行更新后的程序,确认 inspect 的值只出现一次,且末尾表达式不额外输出。
|
||||
|
||||
|
||||
@@ -18,8 +18,8 @@
|
||||
en: 60f20c802585ed1a
|
||||
zh: d06a7dd8c41ab5a1
|
||||
/deepseek-ai-dsh-experimental-cua-macos/use-this-package/connect-an-external-execution-environment:
|
||||
en: b734a0907747580e
|
||||
zh: 8110c5951b70e100
|
||||
en: 8f30c605504d54d0
|
||||
zh: f291cff031b232b8
|
||||
/deepseek-ai-dsh-experimental-cua-macos/understand-the-implementation:
|
||||
en: 4e82b85a5aa72eee
|
||||
zh: 4e004b7a205d8688
|
||||
|
||||
@@ -110,7 +110,7 @@ const app = await cua.getApp("com.apple.TextEdit")
|
||||
const windows = await app.getWindows()
|
||||
if (windows.length === 0) throw new Error("Open a TextEdit window first")
|
||||
const window = await app.getWindow(windows[0].windowId)
|
||||
void await window.getScreenshot()
|
||||
inspect(await window.getScreenshot())
|
||||
```
|
||||
|
||||
Use the `cua` API to discover and select the application (`listApps`, `getState`, `findApps`, `getApp`); use the selected app to activate it, enumerate its windows, and select one. Window selection is required for screenshots and pointer input, not only when several windows exist. `getApp` and `app.getWindow(windowId)` read their initial trees silently; choose the window by its observed title and bounds.
|
||||
@@ -129,7 +129,7 @@ Use `win.click(id)` to send mouse events at the center of an element, or `win.cl
|
||||
|
||||
Absolute coordinates for `mouseMove`, `click`, and `drag` use `[x, y]` tuples; `scroll` uses named `x` and `y`. All use pixels in the latest returned screenshot. Pass `relative: true` for integer thousandths of window width and height, from 0 to 999; omission or `false` uses pixels. Relative input can use a successful AX read or screenshot, while absolute input requires a screenshot. The Host maps the delivered image to native points, including attachment and model-request resizing. A value of 999 remains inside the window edge. Coordinate options do not apply to element IDs. A width or height change since the latest observation returns `WINDOW_CHANGED` before input, including raw coordinate calls without a snapshot ID; read fresh AX or take a new screenshot before proceeding. Window movement alone preserves window-local coordinates. Capture and pointer input reject known disagreement between the selected window’s accessibility frame and displayed bounds with `WINDOW_CHANGED`, before posting pointer events. This includes Stage Manager transformations when AX retains the full window frame. Read fresh state; any activation or window raise remains explicit. Advertised AX actions remain available. Missing AX geometry retains the AXless path and does not establish that a transformed window can be operated.
|
||||
|
||||
Retained objects resolve the current tool binding on every call. The external runtime owns persistence, imports, cancellation and completion-value serialization. Screenshots return encoded bytes with their MIME type, pixel dimensions and window ID; the standard tool pipeline admits the image attachment exactly once. `emit: false` suppresses facade output only. A host that uses another dispatcher can build objects through `createCuaApi` from `./api`.
|
||||
Retained objects resolve the current tool binding on every call. The external runtime owns persistence, imports, cancellation and completion-value serialization. Screenshots return stored `Image` values for explicit display through `inspect`. `emit: false` suppresses text observations. A caller with another program dispatcher can build objects through `createCuaApi(dispatch, onText?)` from `./api`.
|
||||
|
||||
An AppKit sheet can appear in its document window’s tree with a separate window ID. For pointer input on the sheet, use `app.getWindow(sheetId)` and an element ID from that window’s tree. Keyboard input follows a system Open/Save panel only when the application’s focused sheet identifies one Apple panel-service window and that service still focuses the same panel. Unrelated or ambiguous remote recipients fail before dispatch.
|
||||
|
||||
@@ -182,7 +182,7 @@ macOS: accessibility tree operations are available at app or window level. Simul
|
||||
|
||||
Use cua to observe and operate native applications. Always use cua.getCapabilities() to confirm whether a specific API is available on the current platform; its reported capabilities take precedence over availability statements in these instructions. All methods are async; observation options default to { emit: true }, and emit: false suppresses sink output only, leaving returned values unchanged. Actions report dispatch evidence, not verified completion. If sessionLocked is true or an operation reports SESSION_LOCKED, stop UI actions and ask the user to unlock the desktop session; null or unknown means verify a usable unlocked desktop session first. cua.getApp requests user consent; if access is denied or revoked, stop using that application until the user explicitly permits another request, and never bypass a refusal or switch tools to avoid it. Never replay uncertain input without a fresh observation.
|
||||
|
||||
In persistent PTC, use the host-provided bridge initialization once and retain the returned objects; cua comes from that bridge and is not a preinstalled global. Observations emit through inspect; never stringify screenshot image bytes.
|
||||
In persistent PTC, use the host-provided bridge initialization once and retain the returned objects; cua comes from that bridge and is not a preinstalled global. Text observations emit through inspect.
|
||||
|
||||
Signatures show resolved return values after await. ActionResult = { dispatched: boolean, foregroundChanged?: boolean | null }; foregroundChanged is omitted when not sampled and null when the sample is unavailable. JsonValue = string | number | boolean | null | JsonValue[] | { [key: string]: JsonValue }; numbers must be finite. App is the application object with readonly app.appHandle: string (opaque) and the methods listed below for app level. Window is the window object with readonly window.windowId: number and the methods listed below for window level. Neither object carries pid.
|
||||
|
||||
@@ -211,7 +211,7 @@ Element ids are opaque strings from the latest successful tree; the object facad
|
||||
|
||||
## Simulated mouse
|
||||
Use an explicit window for screenshots and pointer input.
|
||||
- getScreenshot(options?: { emit?: boolean }) -> { bytes: Uint8Array, mimeType: 'image/png' | 'image/jpeg' | 'image/webp' | 'image/gif', pixelWidth: number, pixelHeight: number, windowId: number } — capture the selected window as one encoded image; bytes is a JavaScript byte array, and mimeType identifies its encoding, so do not assume PNG.
|
||||
- getScreenshot() -> Image — capture the selected window as a stored image with readonly path, width and height; pass it to inspect to display it.
|
||||
- mouseMove(target: string | [number, number], options?: { relative?: boolean }) -> { dispatched: boolean, foregroundChanged?: boolean | null, hoverRoute: 'accessibility' | 'system-pointer', systemPointerMoved: boolean } — request a mouse move to the target without pressing a button; inspect hoverRoute and systemPointerMoved, and expect no automatic pointer restoration. Default hover requires both AXShowAlternateUI and AXShowDefaultUI on an enabled element or ancestor. Without an AX route it returns HOVER_UNAVAILABLE before sending input. Prefer click on a visible control or performAction with an advertised action. For web content without an AX hover action, use browser tools. Only systemPointerHover: true permits moving the visible cursor. If this returns FOREGROUND_REQUIRED, activate the app and check that the point is unobscured.
|
||||
- click(target: string | [number, number], options?: { mouseButton?: 'left' | 'right' | 'middle', clickCount?: 1 | 2, modifiers?: ('command' | 'control' | 'option' | 'shift' | 'fn')[], holdMs?: number, relative?: boolean }) -> ActionResult — click an element id or point, or perform a left-button double-click; holdMs controls the press duration of each press. click is a Window method: select a window with const win = await app.getWindow(windowId), then call win.click(target). For a background control’s default action, use app.invoke(id). For a mouse click on a background app, call app.activate() first and re-read the target. Both target forms send mouse-down and mouse-up events. click(id) uses the center of the full element bounds. If that center is outside the window, take a fresh win.getScreenshot() and use click([x, y]) on a visible part of the control in screenshot pixels. A background app can ignore the events even when dispatched is true. Verify the expected effect, such as a changed selection, before typing or retrying.
|
||||
- scroll(options: { x: number, y: number, deltaLinesX: number, deltaLinesY: number, relative?: boolean }) -> ActionResult — scroll at a point; deltas are integers from -10 to 10 with at least one nonzero, and positive values reveal content above or left.
|
||||
|
||||
@@ -110,7 +110,7 @@ const app = await cua.getApp("com.apple.TextEdit")
|
||||
const windows = await app.getWindows()
|
||||
if (windows.length === 0) throw new Error("Open a TextEdit window first")
|
||||
const window = await app.getWindow(windows[0].windowId)
|
||||
void await window.getScreenshot()
|
||||
inspect(await window.getScreenshot())
|
||||
```
|
||||
|
||||
使用 `cua` API 发现并选择应用(`listApps`、`getState`、`findApps`、`getApp`);使用选定的应用来激活它、枚举其窗口并选择窗口。截图和指针输入都要求选择窗口,不只在存在多个窗口时才需要。`getApp` 和 `app.getWindow(windowId)` 静默读取初始树;根据观察到的标题与位置选择窗口。
|
||||
@@ -129,7 +129,7 @@ void await window.getScreenshot()
|
||||
|
||||
`mouseMove`、`click` 和 `drag` 的绝对坐标使用 `[x, y]` 元组;`scroll` 使用命名的 `x` 和 `y`。它们都使用最近返回截图中的像素。传入 `relative: true` 时使用 0~999 的千分比整数;省略或传入 `false` 时使用像素。相对坐标可基于成功的 AX 读取或截图,绝对坐标必须先截图。Host 根据实际送入模型的图片尺寸换算原生 point,包含附件归一化和模型请求阶段的缩放。999 映射到窗口边缘以内。坐标选项不适用于元素 ID。窗口宽高与最近一次观察不同时,在输入前返回 `WINDOW_CHANGED`,即使原始坐标调用不传快照 ID 也会检查;重新读取 AX 或截图后再操作。单纯移动窗口不使窗口内坐标失效。截图和鼠标输入会核对所选窗口的辅助功能框与显示边界;已知两者不一致时,在发送任何鼠标事件前返回 `WINDOW_CHANGED`。当 AX 保留完整窗口框时,这也会拦住 Stage Manager 变换后的窗口。需要重新读取状态;激活应用或抬起窗口仍须显式调用。元素声明的 AX 动作保持可用。缺少 AX 几何时保留无 AX 路径,不代表变换后的窗口一定可以操作。
|
||||
|
||||
保留的对象会在每次调用时获取当前工具绑定。外部运行时负责持久化、导入、取消和最终表达式值的序列化。截图返回编码字节及其 MIME 类型、像素尺寸和窗口 ID,标准工具链负责将图片作为附件传递一次。`emit: false` 只控制对象层的输出。使用其他分派器的宿主可以通过 `./api` 的 `createCuaApi` 创建对象。
|
||||
保留的对象会在每次调用时获取当前工具绑定。外部运行时负责持久化、导入、取消和最终表达式值的序列化。截图返回保存的 `Image` 值,通过 `inspect` 显式展示。`emit: false` 控制文本观察输出。使用其他程序分派器的调用方可以通过 `./api` 的 `createCuaApi(dispatch, onText?)` 创建对象。
|
||||
|
||||
AppKit 的 sheet 可以出现在文档窗口的树里,并有独立的窗口 ID。对 sheet 发送鼠标输入时,先用 `app.getWindow(sheetId)` 选择它,再使用该窗口树中的元素 ID。只有应用当前获得焦点的 sheet 指向唯一的 Apple 面板服务窗口,且该服务仍聚焦于同一面板时,键盘输入才转交系统打开或保存面板。无法确认或存在歧义的远端接收方会在输入前报错。
|
||||
|
||||
@@ -182,7 +182,7 @@ macOS: accessibility tree operations are available at app or window level. Simul
|
||||
|
||||
Use cua to observe and operate native applications. Always use cua.getCapabilities() to confirm whether a specific API is available on the current platform; its reported capabilities take precedence over availability statements in these instructions. All methods are async; observation options default to { emit: true }, and emit: false suppresses sink output only, leaving returned values unchanged. Actions report dispatch evidence, not verified completion. If sessionLocked is true or an operation reports SESSION_LOCKED, stop UI actions and ask the user to unlock the desktop session; null or unknown means verify a usable unlocked desktop session first. cua.getApp requests user consent; if access is denied or revoked, stop using that application until the user explicitly permits another request, and never bypass a refusal or switch tools to avoid it. Never replay uncertain input without a fresh observation.
|
||||
|
||||
In persistent PTC, use the host-provided bridge initialization once and retain the returned objects; cua comes from that bridge and is not a preinstalled global. Observations emit through inspect; never stringify screenshot image bytes.
|
||||
In persistent PTC, use the host-provided bridge initialization once and retain the returned objects; cua comes from that bridge and is not a preinstalled global. Text observations emit through inspect.
|
||||
|
||||
Signatures show resolved return values after await. ActionResult = { dispatched: boolean, foregroundChanged?: boolean | null }; foregroundChanged is omitted when not sampled and null when the sample is unavailable. JsonValue = string | number | boolean | null | JsonValue[] | { [key: string]: JsonValue }; numbers must be finite. App is the application object with readonly app.appHandle: string (opaque) and the methods listed below for app level. Window is the window object with readonly window.windowId: number and the methods listed below for window level. Neither object carries pid.
|
||||
|
||||
@@ -211,7 +211,7 @@ Element ids are opaque strings from the latest successful tree; the object facad
|
||||
|
||||
## Simulated mouse
|
||||
Use an explicit window for screenshots and pointer input.
|
||||
- getScreenshot(options?: { emit?: boolean }) -> { bytes: Uint8Array, mimeType: 'image/png' | 'image/jpeg' | 'image/webp' | 'image/gif', pixelWidth: number, pixelHeight: number, windowId: number } — capture the selected window as one encoded image; bytes is a JavaScript byte array, and mimeType identifies its encoding, so do not assume PNG.
|
||||
- getScreenshot() -> Image — capture the selected window as a stored image with readonly path, width and height; pass it to inspect to display it.
|
||||
- mouseMove(target: string | [number, number], options?: { relative?: boolean }) -> { dispatched: boolean, foregroundChanged?: boolean | null, hoverRoute: 'accessibility' | 'system-pointer', systemPointerMoved: boolean } — request a mouse move to the target without pressing a button; inspect hoverRoute and systemPointerMoved, and expect no automatic pointer restoration. Default hover requires both AXShowAlternateUI and AXShowDefaultUI on an enabled element or ancestor. Without an AX route it returns HOVER_UNAVAILABLE before sending input. Prefer click on a visible control or performAction with an advertised action. For web content without an AX hover action, use browser tools. Only systemPointerHover: true permits moving the visible cursor. If this returns FOREGROUND_REQUIRED, activate the app and check that the point is unobscured.
|
||||
- click(target: string | [number, number], options?: { mouseButton?: 'left' | 'right' | 'middle', clickCount?: 1 | 2, modifiers?: ('command' | 'control' | 'option' | 'shift' | 'fn')[], holdMs?: number, relative?: boolean }) -> ActionResult — click an element id or point, or perform a left-button double-click; holdMs controls the press duration of each press. click is a Window method: select a window with const win = await app.getWindow(windowId), then call win.click(target). For a background control’s default action, use app.invoke(id). For a mouse click on a background app, call app.activate() first and re-read the target. Both target forms send mouse-down and mouse-up events. click(id) uses the center of the full element bounds. If that center is outside the window, take a fresh win.getScreenshot() and use click([x, y]) on a visible part of the control in screenshot pixels. A background app can ignore the events even when dispatched is true. Verify the expected effect, such as a changed selection, before typing or retrying.
|
||||
- scroll(options: { x: number, y: number, deltaLinesX: number, deltaLinesY: number, relative?: boolean }) -> ActionResult — scroll at a point; deltas are integers from -10 to 10 with at least one nonzero, and positive values reveal content above or left.
|
||||
|
||||
@@ -64,24 +64,25 @@ it('enumerates every available method once in its catalog entry', () => {
|
||||
|
||||
it('documents option fields and return fields', () => {
|
||||
for (const token of ['emit', 'includeMenus', 'details', 'relative', 'mouseButton', 'clickCount', 'holdMs', 'modifiers', 'format', 'prefix', 'suffix', 'selectionType']) expect(COMMON_GUIDANCE).toContain(token)
|
||||
for (const token of ['dispatched', 'foregroundChanged', 'hoverRoute', 'systemPointerMoved', 'bytes', 'mimeType', 'pixelWidth', 'pixelHeight', 'windowId']) expect(COMMON_GUIDANCE).toContain(token)
|
||||
for (const token of ['dispatched', 'foregroundChanged', 'hoverRoute', 'systemPointerMoved', 'path', 'width', 'height', 'windowId']) expect(COMMON_GUIDANCE).toContain(token)
|
||||
})
|
||||
|
||||
it('inlines discovery, restoration, screenshot, and mouse-move return records', () => {
|
||||
it('inlines discovery, restoration, and mouse-move return records', () => {
|
||||
const lines = COMMON_GUIDANCE.split('\n')
|
||||
for (const name of ['cua.getCapabilities', 'cua.listApps', 'cua.findApps', 'cua.getState', 'app.getWindows', 'app.show', 'window.show', 'getScreenshot', 'mouseMove']) {
|
||||
for (const name of ['cua.getCapabilities', 'cua.listApps', 'cua.findApps', 'cua.getState', 'app.getWindows', 'app.show', 'window.show', 'mouseMove']) {
|
||||
expect(lines.find(line => line.startsWith(`- ${name}(`))).toMatch(/ -> \(?\{ /)
|
||||
}
|
||||
expect(COMMON_GUIDANCE).toContain("{ platform: string, permissions: { accessibility: boolean, screenRecording: boolean }, appLevelAccessibility: boolean, keyboard: ('app' | 'window')[], sessionLocked: boolean | null, methods: string[], coordinateSpace: 'screenshot-pixels', browser: false }")
|
||||
expect(COMMON_GUIDANCE).toContain('{ id: string, displayName: string, isRunning: true, pid: number }[]')
|
||||
expect(COMMON_GUIDANCE).toContain('({ id: string, displayName: string, path: string } & ({ isRunning: true, pid: number } | { isRunning: false }))[]')
|
||||
expect(COMMON_GUIDANCE).toContain('getScreenshot() -> Image')
|
||||
expect(COMMON_GUIDANCE).toContain('bounds: { x: number, y: number, width: number, height: number }, isOnScreen: boolean, onCurrentSpace: boolean | null')
|
||||
})
|
||||
|
||||
it('defines every non-builtin type name used in method signatures', () => {
|
||||
const signatures = COMMON_GUIDANCE.split('\n').filter(line => line.startsWith('- ')).map(line => line.split(' — ')[0]).join('\n')
|
||||
const names = signatures.replace(/'[^']*'/g, '').match(/\b[A-Z][A-Za-z0-9_]*\b/g) ?? []
|
||||
expect([...new Set(names)].sort()).toEqual(['ActionResult', 'App', 'JsonValue', 'Uint8Array', 'Window'])
|
||||
expect([...new Set(names)].sort()).toEqual(['ActionResult', 'App', 'Image', 'JsonValue', 'Window'])
|
||||
expect(COMMON_GUIDANCE).toContain('ActionResult = { dispatched: boolean, foregroundChanged?: boolean | null }')
|
||||
expect(COMMON_GUIDANCE).toContain('JsonValue = string | number | boolean | null | JsonValue[] | { [key: string]: JsonValue }')
|
||||
expect(COMMON_GUIDANCE).toContain('App is the application object with readonly app.appHandle: string (opaque) and the methods listed below for app level')
|
||||
@@ -89,8 +90,7 @@ it('defines every non-builtin type name used in method signatures', () => {
|
||||
expect(COMMON_GUIDANCE).toContain('Signatures show resolved return values after await')
|
||||
})
|
||||
|
||||
it('preserves image formats and pointer modifier enums without restricting keyboard names', () => {
|
||||
expect(COMMON_GUIDANCE).toContain("mimeType: 'image/png' | 'image/jpeg' | 'image/webp' | 'image/gif'")
|
||||
it('documents pointer modifiers and platform-defined keyboard names', () => {
|
||||
const lines = COMMON_GUIDANCE.split('\n')
|
||||
for (const name of ['click', 'drag']) {
|
||||
expect(lines.find(line => line.startsWith(`- ${name}(`))).toContain("modifiers?: ('command' | 'control' | 'option' | 'shift' | 'fn')[]")
|
||||
@@ -135,7 +135,7 @@ it('separates facade options from raw RPC parameters', () => {
|
||||
it('documents facade initialization, emitted observations, and separate key arguments', () => {
|
||||
expect(COMMON_GUIDANCE).toContain('host-provided bridge initialization once')
|
||||
expect(COMMON_GUIDANCE).toContain('not a preinstalled global')
|
||||
expect(COMMON_GUIDANCE).toContain('Observations emit through inspect')
|
||||
expect(COMMON_GUIDANCE).toContain('Text observations emit through inspect')
|
||||
expect(COMMON_GUIDANCE).toContain('do not pack them into a chord string')
|
||||
expect(COMMON_GUIDANCE).toContain('perform a left-button double-click')
|
||||
})
|
||||
|
||||
@@ -169,6 +169,12 @@ it.skipIf(process.platform === 'win32')('shares one application between retained
|
||||
const result = await executeToolFixture(context, { agent: owner, signal: new AbortController().signal,
|
||||
callId: ToolCallId(`retained-object-${++objectCall}`), name: 'cua', arguments: operation })
|
||||
if (result.isError) throw new Error(JSON.stringify(result.content))
|
||||
const image = result.content.find(block => block.type === 'image')
|
||||
if (image !== undefined) {
|
||||
const path = context.attachments.imageHostPath(image.attachment)
|
||||
if (path === undefined) throw new Error('Fixture image path is unavailable')
|
||||
return { path, width: image.attachment.width, height: image.attachment.height }
|
||||
}
|
||||
return z.json().parse(result.value)
|
||||
})
|
||||
const retainedApp = await objectApi(agent).getApp('test.fixture')
|
||||
@@ -178,9 +184,9 @@ it.skipIf(process.platform === 'win32')('shares one application between retained
|
||||
const peerApp = await objectApi(peer.agent).getApp('test.fixture')
|
||||
const peerWindow = await peerApp.getWindow(7)
|
||||
expect(peerApp.appHandle).not.toBe(retainedApp.appHandle)
|
||||
const screenshot = await peerWindow.getScreenshot({ emit: false })
|
||||
expect(screenshot).toMatchObject({ mimeType: 'image/png', pixelWidth: 1, pixelHeight: 1, windowId: 7 })
|
||||
expect(Buffer.from(screenshot.bytes).toString('base64')).toBe(png)
|
||||
const screenshot = await peerWindow.getScreenshot()
|
||||
expect(screenshot).toMatchObject({ width: 1, height: 1 })
|
||||
expect((await readFile(screenshot.path)).toString('base64')).toBe(png)
|
||||
await peerApp.getWindow(8)
|
||||
const recovery = 'Snapshot belongs to window 8; read getAccessibilityTree on window 7'
|
||||
await expect(peerWindow.click(element('0'))).rejects.toThrow(recovery)
|
||||
|
||||
@@ -15,8 +15,8 @@
|
||||
en: 5db4d4fea25493b9
|
||||
zh: 65ab64fc24eb0d09
|
||||
/deepseek-ai-dsh-cua/use-this-package:
|
||||
en: 1783c564f234d2b1
|
||||
zh: 9dd72bb36f21b3a4
|
||||
en: f848fe35252a8367
|
||||
zh: 6cb8a7f15fec4c07
|
||||
/deepseek-ai-dsh-cua/use-this-package/permission-controls:
|
||||
en: d92d5948a7941e8d
|
||||
zh: d43491ae6521b87e
|
||||
|
||||
@@ -35,7 +35,7 @@ Use `cua.getApp(selector)` to select an application; it obtains consent, optiona
|
||||
|
||||
Keyboard methods are available in the scopes a provider declares; the provider owns how input is delivered within each scope. A lost connection invalidates its handles and coordinates. Failed cleanup is reported before that connection is retired; select the application again and inspect fresh state, because earlier input may have partially executed. Screenshot image errors retain the failing stage and attachment error code. Invalid request arguments reject before dispatch with a readable method, parameter and requirement.
|
||||
|
||||
Observation methods emit and return their result. Connect PTC text output with `connectCua(() => tools.cua, text => inspect(text))`. `window.getScreenshot()` returns an ordinary `Image`: `path` is an attachment file readable by the program, and `width` and `height` describe its stored pixels. Use `inspect("before", before, "after", after)` to display selected screenshots; read the window ID from `window.windowId`. Host integrations calling `createCuaApi` receive an encoded `CuaScreenshot` and control output through their image sink.
|
||||
Observation methods emit and return their result. Connect PTC text output with `connectCua(() => tools.cua, text => inspect(text))`. `window.getScreenshot()` returns an ordinary `Image`: `path` is an attachment file readable by the program, and `width` and `height` describe its stored pixels. Use `inspect("before", before, "after", after)` to display selected screenshots; read the window ID from `window.windowId`. `createCuaApi(dispatch, onText?)` accepts the same program dispatcher and text callback.
|
||||
|
||||
`tools.cua` screenshot operations return `Image` in TypeScript programs. Host-side canonical screenshot results carry encoded image data and a durable `attachment` reference; the native renderer projects coordinate metadata and the image into model content. Capture requires an attachment store and a resolved model route declaring image input. Admission and storage failures reject the call.
|
||||
|
||||
|
||||
@@ -35,7 +35,7 @@ kind: "package-reference"
|
||||
|
||||
键盘方法可在 provider 声明的作用域中使用;每个作用域内的输入投递由 provider 负责。连接丢失会使其句柄和坐标失效。清理失败会先报告,再释放该连接;应重新选择应用并读取新状态,因为先前的输入可能已部分执行。截图图像错误会保留失败阶段和附件错误码。请求参数校验失败时,在分派前返回包含方法、参数和要求的可读消息。
|
||||
|
||||
观察方法输出并返回结果。通过 `connectCua(() => tools.cua, text => inspect(text))` 连接 PTC 文本输出。`window.getScreenshot()` 返回普通 `Image`,其 `path` 指向执行环境可读取的附件文件,`width` 和 `height` 对应保存图片的像素尺寸。使用 `inspect("操作前", before, "操作后", after)` 选择展示截图;窗口 ID 从 `window.windowId` 读取。直接调用 `createCuaApi` 的宿主集成接收包含编码字节和 MIME 类型的 `CuaScreenshot`,并通过图片回调控制输出。
|
||||
观察方法输出并返回结果。通过 `connectCua(() => tools.cua, text => inspect(text))` 连接 PTC 文本输出。`window.getScreenshot()` 返回普通 `Image`,其 `path` 指向执行环境可读取的附件文件,`width` 和 `height` 对应保存图片的像素尺寸。使用 `inspect("操作前", before, "操作后", after)` 选择展示截图;窗口 ID 从 `window.windowId` 读取。`createCuaApi(dispatch, onText?)` 接收相同的程序分派器和文本回调。
|
||||
|
||||
`tools.cua` 直接返回操作结果。截图结果保留图片 Base64,并增加供工具渲染使用的持久 `attachment` 引用;模型内容只包含坐标元数据和一张图片,不包含编码字节。截图前要求附件服务可用,并确认当前模型支持图片输入。图片接入或保存失败会拒绝调用。对象桥接保留截图字节和已校验的元数据,不重复上传或输出同一张图片。
|
||||
|
||||
|
||||
@@ -1,15 +1,14 @@
|
||||
/** App and window behavior over a fixture operation dispatcher, without desktop access. */
|
||||
import { Buffer } from 'node:buffer'
|
||||
import { describe, expect, expectTypeOf, it, vi } from 'vitest'
|
||||
import { brandString } from '@deepseek-ai/dsh-brand'
|
||||
import type { JsonValue } from '@deepseek-ai/dsh-util-values'
|
||||
import { createCuaApi } from '../src/api.ts'
|
||||
import type { CuaApp, CuaCapabilities, CuaElementId, CuaOperation, CuaScreenshot, CuaWindow } from '../src/types.ts'
|
||||
import type { CuaApp, CuaCapabilities, CuaElementId, CuaOperation, CuaWindow } from '../src/types.ts'
|
||||
|
||||
/** Element ids are opaque strings; these fixtures use the macOS decimal form. */
|
||||
const element = (id: string): CuaElementId => brandString<CuaElementId>(id)
|
||||
|
||||
const PNG = 'iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAIAAACQd1PeAAAACXBIWXMAAAPoAAAD6AG1e1JrAAAADElEQVQImWNgZGIGAAAOAAeCcsnOAAAAAElFTkSuQmCC'
|
||||
const IMAGE = Object.freeze({ path: '/images/screenshot.png', width: 1, height: 1 })
|
||||
const bounds = { x: -640, y: 100, width: 640, height: 480 }
|
||||
const selected = { appHandle: 'app-17', pid: 17, name: 'Editor', bundleId: 'test.editor', running: true }
|
||||
const delivered = { dispatched: true }
|
||||
@@ -23,7 +22,7 @@ function fixture(withSink = true) {
|
||||
let windows = [windowInfo(23)]
|
||||
let currentCapabilities: CuaCapabilities = capabilities
|
||||
let generation = 0
|
||||
const sink = { text: vi.fn<(value: string) => void>(), image: vi.fn<(screenshot: CuaScreenshot) => void>() }
|
||||
const sink = { text: vi.fn<(value: string) => void>() }
|
||||
const dispatch = vi.fn(async ({ method, params }: CuaOperation): Promise<JsonValue> => {
|
||||
switch (method) {
|
||||
case 'getCapabilities': return currentCapabilities
|
||||
@@ -37,13 +36,13 @@ function fixture(withSink = true) {
|
||||
snapshotId: `snapshot-${++generation}`, tree: `Tree ${generation}`, truncated: false,
|
||||
windows: params.windowId === undefined ? windows : windows.filter(window => window.windowId === params.windowId),
|
||||
}
|
||||
case 'screenshot': return { mimeType: 'image/png', data: PNG, pixelWidth: 1, pixelHeight: 1, windowId: params.windowId, bounds, coordinateSpace: 'screenshot-pixels' }
|
||||
case 'screenshot': return IMAGE
|
||||
case 'listActions': return ['AXPress', 'AXShowMenu', 'AXTextArea.AXValue.Set']
|
||||
default: return delivered
|
||||
}
|
||||
})
|
||||
return {
|
||||
dispatch, sink, api: createCuaApi(dispatch, withSink ? sink : undefined),
|
||||
dispatch, sink, api: createCuaApi(dispatch, withSink ? sink.text : undefined),
|
||||
setWindows: (ids: number[]) => { windows = ids.map(windowInfo) },
|
||||
setWindowRecords: (records: ReturnType<typeof windowInfo>[]) => { windows = records },
|
||||
setCapabilities: (value: CuaCapabilities) => { currentCapabilities = value },
|
||||
@@ -54,10 +53,6 @@ function state(overrides: Record<string, JsonValue> = {}): JsonValue {
|
||||
return { appHandle: selected.appHandle, windowId: null, snapshotId: 'changed', tree: 'Changed tree', windows: [windowInfo(23)], truncated: false, ...overrides }
|
||||
}
|
||||
|
||||
function screenshot(overrides: Record<string, JsonValue> = {}): JsonValue {
|
||||
return { mimeType: 'image/png', data: PNG, pixelWidth: 1, pixelHeight: 1, windowId: 23, bounds, coordinateSpace: 'screenshot-pixels', ...overrides }
|
||||
}
|
||||
|
||||
describe('application discovery', () => {
|
||||
it('reports permissions and discovers both bundle and PID selectors without activating an app', async () => {
|
||||
const f = fixture()
|
||||
@@ -81,13 +76,12 @@ describe('application discovery', () => {
|
||||
expectTypeOf<CuaApp>().not.toHaveProperty('pid')
|
||||
expect(app.appHandle).toBe('app-17')
|
||||
const window = await app.getWindow(23)
|
||||
expect(await window.getScreenshot()).toEqual({ bytes: Uint8Array.from(Buffer.from(PNG, 'base64')), mimeType: 'image/png', pixelWidth: 1, pixelHeight: 1, windowId: 23 })
|
||||
expect(await window.getScreenshot()).toEqual(IMAGE)
|
||||
expect(f.dispatch.mock.calls.map(([operation]) => operation.method)).toEqual([
|
||||
'getCapabilities', 'listApps', 'getApp', 'getState', 'getState', 'screenshot',
|
||||
])
|
||||
expect(await app.getWindows()).toEqual([windowInfo(23)])
|
||||
expect(f.sink.text).not.toHaveBeenCalled()
|
||||
expect(f.sink.image).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('searches installed applications without turning ordinary discovery into an installed-app inventory', async () => {
|
||||
@@ -162,7 +156,7 @@ describe('mouseMove across observations', () => {
|
||||
f.dispatch.mockClear()
|
||||
expect(await window.mouseMove(element('0'))).toEqual({ ...delivered, hoverRoute: 'accessibility', systemPointerMoved: false })
|
||||
await window.getAccessibilityTree({ emit: false })
|
||||
await window.getScreenshot({ emit: false })
|
||||
await window.getScreenshot()
|
||||
expect(await window.mouseMove([30, 40])).toEqual({ ...delivered, hoverRoute: 'accessibility', systemPointerMoved: false })
|
||||
expect(f.dispatch.mock.calls.map(([operation]) => operation)).toEqual([
|
||||
{ method: 'mouseMove', params: { appHandle: 'app-17', snapshotId: 'snapshot-2', elementId: '0', windowId: 23 } },
|
||||
@@ -401,7 +395,7 @@ describe('application and window observations', () => {
|
||||
f.dispatch.mockResolvedValueOnce(state({ windowId: 23, snapshotId: 'axless', tree: '', elements: [] }))
|
||||
const window = await app.getWindow(23)
|
||||
expect(f.sink.text).not.toHaveBeenCalled()
|
||||
await expect(window.getScreenshot()).resolves.toEqual({ bytes: Uint8Array.from(Buffer.from(PNG, 'base64')), mimeType: 'image/png', pixelWidth: 1, pixelHeight: 1, windowId: 23 })
|
||||
await expect(window.getScreenshot()).resolves.toEqual(IMAGE)
|
||||
await window.click([10, 20])
|
||||
await window.scroll({ x: 10, y: 20, deltaLinesX: 0, deltaLinesY: -1 })
|
||||
await window.drag([10, 20], [30, 40])
|
||||
@@ -570,12 +564,11 @@ describe('application and window observations', () => {
|
||||
await window.getScreenshot()
|
||||
expect(f.dispatch.mock.calls.slice(calls).map(([operation]) => operation.method)).toEqual(['screenshot'])
|
||||
expect(f.sink.text).not.toHaveBeenCalled()
|
||||
expect(f.sink.image).toHaveBeenCalledTimes(1)
|
||||
})
|
||||
})
|
||||
|
||||
describe('screenshots', () => {
|
||||
it('returns screenshot metadata independently of input point bounds and supports selective output', async () => {
|
||||
it('returns the stored screenshot Image and emits selected accessibility text', async () => {
|
||||
const f = fixture()
|
||||
f.setWindows([23, 24])
|
||||
const app = await f.api.getApp('Editor')
|
||||
@@ -584,34 +577,20 @@ describe('screenshots', () => {
|
||||
const tree = await window.getAccessibilityTree()
|
||||
const image = await window.getScreenshot()
|
||||
expect(tree).toBe('Tree 3')
|
||||
expect(image).toEqual({ bytes: Uint8Array.from(Buffer.from(PNG, 'base64')), mimeType: 'image/png', pixelWidth: 1, pixelHeight: 1, windowId: 24 })
|
||||
expect(image).toBe(IMAGE)
|
||||
expect(f.dispatch).toHaveBeenLastCalledWith({ method: 'screenshot', params: { appHandle: 'app-17', windowId: 24 } })
|
||||
expect(f.sink.text).toHaveBeenCalledWith('Tree 3')
|
||||
expect(f.sink.image).toHaveBeenCalledWith(image)
|
||||
await window.getAccessibilityTree({ emit: false })
|
||||
await window.getScreenshot({ emit: false })
|
||||
await window.getScreenshot()
|
||||
expect(f.sink.text).toHaveBeenCalledTimes(1)
|
||||
expect(f.sink.image).toHaveBeenCalledTimes(1)
|
||||
})
|
||||
|
||||
it('rejects a screenshot from another window before emitting its image', async () => {
|
||||
it.each([null, 'text only', {}, { path: '/image.png', width: 1 }])('rejects an invalid screenshot result %j', async (value) => {
|
||||
const f = fixture()
|
||||
const app = await f.api.getApp('Editor')
|
||||
const window = await app.getWindow(23)
|
||||
f.dispatch.mockResolvedValueOnce(screenshot({ windowId: 24 }))
|
||||
await expect(window.getScreenshot()).rejects.toThrow('belongs to another window')
|
||||
expect(f.sink.image).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it.each(['image/jpeg', 'image/webp', 'image/gif'])('preserves Host-admitted %s bytes and metadata in the return value and image sink', async (mimeType) => {
|
||||
const f = fixture()
|
||||
const app = await f.api.getApp('Editor')
|
||||
const window = await app.getWindow(23)
|
||||
const data = Buffer.from('admitted request image').toString('base64')
|
||||
f.dispatch.mockResolvedValueOnce(screenshot({ mimeType, data, pixelWidth: 320, pixelHeight: 240 }))
|
||||
const image = await window.getScreenshot()
|
||||
expect(image).toEqual({ bytes: Uint8Array.from(Buffer.from(data, 'base64')), mimeType, pixelWidth: 320, pixelHeight: 240, windowId: 23 })
|
||||
expect(f.sink.image).toHaveBeenCalledExactlyOnceWith(image)
|
||||
f.dispatch.mockResolvedValueOnce(value)
|
||||
await expect(window.getScreenshot()).rejects.toThrow('must return an Image')
|
||||
})
|
||||
|
||||
})
|
||||
|
||||
@@ -305,8 +305,8 @@ describe('read_image happy path', () => {
|
||||
expect(value).toBeInstanceOf(PtcImage)
|
||||
if (!(value instanceof PtcImage)) throw new Error('read_image must return an Image')
|
||||
expect(await readFile(value.path)).toEqual(PNG_1X1)
|
||||
expect([value.width, value.height]).toEqual([1, 1])
|
||||
return { logs: ['before', 'after'], images: [{ afterLog: 1, image: value }] }
|
||||
expect([value.attachment.width, value.attachment.height]).toEqual([1, 1])
|
||||
return { logs: ['before', 'after'], images: [{ afterLog: 1, attachment: value.attachment }] }
|
||||
}
|
||||
|
||||
const result = await call(ctx, RUN_CODE_NAME, {
|
||||
|
||||
@@ -70,6 +70,7 @@
|
||||
"@deepseek-ai/dsh-fs-local": "workspace:*",
|
||||
"@deepseek-ai/dsh-subprocess-local": "workspace:*",
|
||||
"@deepseek-ai/dsh-session-projection": "workspace:*",
|
||||
"@deepseek-ai/dsh-sandbox-local": "workspace:*"
|
||||
"@deepseek-ai/dsh-sandbox-local": "workspace:*",
|
||||
"@deepseek-ai/dsh-attachment": "workspace:*"
|
||||
}
|
||||
}
|
||||
@@ -1,4 +1,4 @@
|
||||
import { PtcImage } from '@deepseek-ai/dsh-ptc-runtime'
|
||||
import { AttachmentId } from '@deepseek-ai/dsh-attachment'
|
||||
import { expect, it } from 'vitest'
|
||||
import { OutputLedger } from '../src/output-ledger.ts'
|
||||
|
||||
@@ -28,8 +28,8 @@ it('retains a bounded prefix when logs or a failure diagnostic exceed the limit'
|
||||
|
||||
|
||||
it('charges ordered image descriptors and text against one budget', () => {
|
||||
const image = new PtcImage('/image.png', 8, 4)
|
||||
const entry = { afterLog: 0, image }
|
||||
const attachment = { attachmentId: AttachmentId('sha256:image'), mediaType: 'image/png' as const, bytes: 64, width: 8, height: 4 }
|
||||
const entry = { afterLog: 0, attachment }
|
||||
const bytes = Buffer.byteLength(JSON.stringify([entry]))
|
||||
const ledger = new OutputLedger(bytes + 2)
|
||||
expect(ledger.admitImage(entry)).toBe(true)
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { AttachmentId } from '@deepseek-ai/dsh-attachment'
|
||||
import { PtcImage } from '@deepseek-ai/dsh-ptc-runtime'
|
||||
import { Context } from '@deepseek-ai/cordis'
|
||||
import { describe, expect, it } from 'vitest'
|
||||
@@ -31,8 +32,8 @@ describe.each(modes)('program output in $executionMode with retention $scriptPer
|
||||
|
||||
it('emits selected images among text and objects and preserves immutable image properties', async () => {
|
||||
const { runtime } = await setup()
|
||||
const before = new PtcImage('/images/before.png', 8, 4)
|
||||
const after = new PtcImage('/images/after.png', 16, 8)
|
||||
const before = new PtcImage('/images/before.png', { attachmentId: AttachmentId('sha256:before'), mediaType: 'image/png', bytes: 64, width: 8, height: 4 })
|
||||
const after = new PtcImage('/images/after.png', { attachmentId: AttachmentId('sha256:after'), mediaType: 'image/png', bytes: 64, width: 16, height: 8 })
|
||||
const result = await runtime.run(runtime.resolve({
|
||||
program: `
|
||||
const before = await tools.before({});
|
||||
@@ -53,19 +54,23 @@ describe.each(modes)('program output in $executionMode with retention $scriptPer
|
||||
expect(result.logs[1]).toBe('before { windowId: 23 }')
|
||||
expect(result.logs[2]).toContain('after [')
|
||||
expect(result.images).toEqual([
|
||||
{ afterLog: 2, image: before }, { afterLog: 3, image: after }, { afterLog: 3, image: before },
|
||||
{ afterLog: 2, attachment: before.attachment },
|
||||
{ afterLog: 3, attachment: after.attachment },
|
||||
{ afterLog: 3, attachment: before.attachment },
|
||||
])
|
||||
expect(result.images?.[0]?.attachment).toBe(before.attachment)
|
||||
expect(result.images?.[1]?.attachment).toBe(after.attachment)
|
||||
})
|
||||
|
||||
it('keeps uninspected images in the program and exposes image output before an exception', async () => {
|
||||
const { runtime } = await setup()
|
||||
const image = new PtcImage('/images/selected.png', 8, 4)
|
||||
const image = new PtcImage('/images/selected.png', { attachmentId: AttachmentId('sha256:selected'), mediaType: 'image/png', bytes: 64, width: 8, height: 4 })
|
||||
const bindings = [{ global: 'tools', functions: { image: () => Promise.resolve(image) } }]
|
||||
const quiet = await runtime.run(runtime.resolve({ program: 'await tools.image({})', bindings, discardCompletion: true }))
|
||||
expect(quiet.images).toBeUndefined()
|
||||
expect(quiet.logs).toEqual([])
|
||||
const failed = await runtime.run(runtime.resolve({ program: 'inspect(await tools.image({})); throw new Error("after image")', bindings, discardCompletion: true }))
|
||||
expect(failed.images).toEqual([{ afterLog: 0, image }])
|
||||
expect(failed.images).toEqual([{ afterLog: 0, attachment: image.attachment }])
|
||||
expect(failed.error?.kind).toBe('exception')
|
||||
})
|
||||
|
||||
@@ -73,7 +78,7 @@ describe.each(modes)('program output in $executionMode with retention $scriptPer
|
||||
const { runtime } = await setup()
|
||||
if (!runtime.retainsState) return
|
||||
const stateOwner = {}
|
||||
const image = new PtcImage('/images/retained.png', 8, 4)
|
||||
const image = new PtcImage('/images/retained.png', { attachmentId: AttachmentId('sha256:retained'), mediaType: 'image/png', bytes: 64, width: 8, height: 4 })
|
||||
const first = await runtime.run(runtime.resolve({
|
||||
program: 'var retainedImage = await tools.image({});', stateOwner, discardCompletion: true,
|
||||
bindings: [{ global: 'tools', functions: { image: () => Promise.resolve(image) } }],
|
||||
@@ -82,13 +87,13 @@ describe.each(modes)('program output in $executionMode with retention $scriptPer
|
||||
expect(first.images).toBeUndefined()
|
||||
const second = await runtime.run(runtime.resolve({ program: 'inspect(retainedImage)', stateOwner, discardCompletion: true, bindings: [] }))
|
||||
expect(second.error).toBeUndefined()
|
||||
expect(second.images).toEqual([{ afterLog: 0, image }])
|
||||
expect(second.images).toEqual([{ afterLog: 0, attachment: image.attachment }])
|
||||
await runtime.release(stateOwner)
|
||||
})
|
||||
|
||||
it('bounds explicitly inspected image descriptors while allowing uninspected images', async () => {
|
||||
const { runtime } = await mountRuntime(new Context(), { ...mode, maxOutputBytes: 128 })
|
||||
const image = new PtcImage('/images/' + 'x'.repeat(200), 8, 4)
|
||||
const image = new PtcImage('/images/large.png', { attachmentId: AttachmentId('sha256:large'), mediaType: 'image/png', bytes: 64, width: 8, height: 4, name: 'x'.repeat(200) })
|
||||
const bindings = [{ global: 'tools', functions: { image: () => Promise.resolve(image) } }]
|
||||
const quiet = await runtime.run(runtime.resolve({ program: 'await tools.image({})', bindings, discardCompletion: true }))
|
||||
expect(quiet.error).toBeUndefined()
|
||||
|
||||
@@ -18,8 +18,8 @@
|
||||
en: 346f62bbfd0e6d9d
|
||||
zh: bf86b2d7a26e8193
|
||||
/deepseek-ai-dsh-ptc-runtime/use-this-package/run-a-program:
|
||||
en: 7703e18ca431e362
|
||||
zh: 6240c5fcb5125baa
|
||||
en: 280cd8a6f6a0d946
|
||||
zh: 995402924714d36d
|
||||
/deepseek-ai-dsh-ptc-runtime/use-this-package/choose-a-backend:
|
||||
en: fc0d515f0c3664f3
|
||||
zh: 0a1b646b5cf92232
|
||||
|
||||
@@ -33,6 +33,8 @@ Call `resolve(request)` followed by `run(spec)` with a program and binding names
|
||||
|
||||
Set `discardCompletion: true` to omit the successful completion without JSON validation or serialization; captured logs and failures are still returned. The TypeScript tool executor selects this option for explicit program output, while workflow consumers retain completion values.
|
||||
|
||||
Host bindings can return `new PtcImage(path, attachment)` to expose a stored image in Node programs. Explicit inspection returns the same attachment reference in `result.images`, positioned among logs by `afterLog`.
|
||||
|
||||
```text
|
||||
const spec = ctx.ptcRuntime.resolve({
|
||||
program: 'return await tools.add({ a: 1, b: 2 })',
|
||||
|
||||
@@ -33,6 +33,8 @@ kind: "package-reference"
|
||||
|
||||
设置 `discardCompletion: true` 可跳过成功完成值的 JSON 验证与序列化,并从结果中省略该值;捕获日志与失败信息仍会返回。TypeScript 工具执行器通过此选项提供显式程序输出,workflow 消费方则继续保留完成值。
|
||||
|
||||
宿主绑定可以返回 `new PtcImage(path, attachment)`,向 Node 程序提供已存储的图片。显式展示后,`result.images` 返回同一附件引用,并通过 `afterLog` 定位在日志条目之间。
|
||||
|
||||
```text
|
||||
const spec = ctx.ptcRuntime.resolve({
|
||||
program: 'return await tools.add({ a: 1, b: 2 })',
|
||||
|
||||
@@ -31,10 +31,12 @@
|
||||
"license": "MIT",
|
||||
"peerDependencies": {
|
||||
"@deepseek-ai/cordis": "workspace:~",
|
||||
"@deepseek-ai/dsh-sandbox": "workspace:*"
|
||||
"@deepseek-ai/dsh-sandbox": "workspace:*",
|
||||
"@deepseek-ai/dsh-attachment": "workspace:*"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@deepseek-ai/cordis": "workspace:~",
|
||||
"@deepseek-ai/dsh-sandbox": "workspace:*"
|
||||
"@deepseek-ai/dsh-sandbox": "workspace:*",
|
||||
"@deepseek-ai/dsh-attachment": "workspace:*"
|
||||
}
|
||||
}
|
||||
@@ -16,6 +16,9 @@
|
||||
},
|
||||
{
|
||||
"path": "../../sandbox/sandbox"
|
||||
},
|
||||
{
|
||||
"path": "../../attachment/attachment"
|
||||
}
|
||||
]
|
||||
}
|
||||
Generated
+6
@@ -14127,6 +14127,9 @@ importers:
|
||||
'@deepseek-ai/cordis':
|
||||
specifier: workspace:~
|
||||
version: link:../../../vendor/cordis
|
||||
'@deepseek-ai/dsh-attachment':
|
||||
specifier: workspace:*
|
||||
version: link:../../attachment/attachment
|
||||
'@deepseek-ai/dsh-sandbox':
|
||||
specifier: workspace:*
|
||||
version: link:../../sandbox/sandbox
|
||||
@@ -14152,6 +14155,9 @@ importers:
|
||||
'@deepseek-ai/cordis':
|
||||
specifier: workspace:~
|
||||
version: link:../../../vendor/cordis
|
||||
'@deepseek-ai/dsh-attachment':
|
||||
specifier: workspace:*
|
||||
version: link:../../attachment/attachment
|
||||
'@deepseek-ai/dsh-fs':
|
||||
specifier: workspace:*
|
||||
version: link:../../fs/fs
|
||||
|
||||
@@ -37,7 +37,7 @@ macOS: accessibility tree operations are available at app or window level. Simul
|
||||
|
||||
Use cua to observe and operate native applications. Always use cua.getCapabilities() to confirm whether a specific API is available on the current platform; its reported capabilities take precedence over availability statements in these instructions. All methods are async; observation options default to { emit: true }, and emit: false suppresses sink output only, leaving returned values unchanged. Actions report dispatch evidence, not verified completion. If sessionLocked is true or an operation reports SESSION_LOCKED, stop UI actions and ask the user to unlock the desktop session; null or unknown means verify a usable unlocked desktop session first. cua.getApp requests user consent; if access is denied or revoked, stop using that application until the user explicitly permits another request, and never bypass a refusal or switch tools to avoid it. Never replay uncertain input without a fresh observation.
|
||||
|
||||
In persistent PTC, use the host-provided bridge initialization once and retain the returned objects; cua comes from that bridge and is not a preinstalled global. Observations emit through inspect; never stringify screenshot image bytes.
|
||||
In persistent PTC, use the host-provided bridge initialization once and retain the returned objects; cua comes from that bridge and is not a preinstalled global. Text observations emit through inspect.
|
||||
|
||||
Signatures show resolved return values after await. ActionResult = { dispatched: boolean, foregroundChanged?: boolean | null }; foregroundChanged is omitted when not sampled and null when the sample is unavailable. JsonValue = string | number | boolean | null | JsonValue[] | { [key: string]: JsonValue }; numbers must be finite. App is the application object with readonly app.appHandle: string (opaque) and the methods listed below for app level. Window is the window object with readonly window.windowId: number and the methods listed below for window level. Neither object carries pid.
|
||||
|
||||
@@ -66,7 +66,7 @@ Element ids are opaque strings from the latest successful tree; the object facad
|
||||
|
||||
## Simulated mouse
|
||||
Use an explicit window for screenshots and pointer input.
|
||||
- getScreenshot(options?: { emit?: boolean }) -> Image — capture the selected window as a stored image with readonly path, width, and height. Pass selected screenshots to inspect(...values) to display them; read the window ID from window.windowId.
|
||||
- getScreenshot() -> Image — capture the selected window as a stored image with readonly path, width, and height. Pass selected screenshots to inspect(...values) to display them; read the window ID from window.windowId.
|
||||
- mouseMove(target: string | [number, number], options?: { relative?: boolean }) -> { dispatched: boolean, foregroundChanged?: boolean | null, hoverRoute: 'accessibility' | 'system-pointer', systemPointerMoved: boolean } — request a mouse move to the target without pressing a button; inspect hoverRoute and systemPointerMoved, and expect no automatic pointer restoration. Default hover requires both AXShowAlternateUI and AXShowDefaultUI on an enabled element or ancestor. Without an AX route it returns HOVER_UNAVAILABLE before sending input. Prefer click on a visible control or performAction with an advertised action. For web content without an AX hover action, use browser tools. Only systemPointerHover: true permits moving the visible cursor. If this returns FOREGROUND_REQUIRED, activate the app and check that the point is unobscured.
|
||||
- click(target: string | [number, number], options?: { mouseButton?: 'left' | 'right' | 'middle', clickCount?: 1 | 2, modifiers?: ('command' | 'control' | 'option' | 'shift' | 'fn')[], holdMs?: number, relative?: boolean }) -> ActionResult — click an element id or point, or perform a left-button double-click; holdMs controls the press duration of each press. click is a Window method: select a window with const win = await app.getWindow(windowId), then call win.click(target). For a background control’s default action, use app.invoke(id). For a mouse click on a background app, call app.activate() first and re-read the target. Both target forms send mouse-down and mouse-up events. click(id) uses the center of the full element bounds. If that center is outside the window, take a fresh win.getScreenshot() and use click([x, y]) on a visible part of the control in screenshot pixels. A background app can ignore the events even when dispatched is true. Verify the expected effect, such as a changed selection, before typing or retrying.
|
||||
- scroll(options: { x: number, y: number, deltaLinesX: number, deltaLinesY: number, relative?: boolean }) -> ActionResult — scroll at a point; deltas are integers from -10 to 10 with at least one nonzero, and positive values reveal content above or left.
|
||||
|
||||
@@ -39,7 +39,7 @@ macOS: accessibility tree operations are available at app or window level. Simul
|
||||
|
||||
Use cua to observe and operate native applications. Always use cua.getCapabilities() to confirm whether a specific API is available on the current platform; its reported capabilities take precedence over availability statements in these instructions. All methods are async; observation options default to { emit: true }, and emit: false suppresses sink output only, leaving returned values unchanged. Actions report dispatch evidence, not verified completion. If sessionLocked is true or an operation reports SESSION_LOCKED, stop UI actions and ask the user to unlock the desktop session; null or unknown means verify a usable unlocked desktop session first. cua.getApp requests user consent; if access is denied or revoked, stop using that application until the user explicitly permits another request, and never bypass a refusal or switch tools to avoid it. Never replay uncertain input without a fresh observation.
|
||||
|
||||
In persistent PTC, use the host-provided bridge initialization once and retain the returned objects; cua comes from that bridge and is not a preinstalled global. Observations emit through inspect; never stringify screenshot image bytes.
|
||||
In persistent PTC, use the host-provided bridge initialization once and retain the returned objects; cua comes from that bridge and is not a preinstalled global. Text observations emit through inspect.
|
||||
|
||||
Signatures show resolved return values after await. ActionResult = { dispatched: boolean, foregroundChanged?: boolean | null }; foregroundChanged is omitted when not sampled and null when the sample is unavailable. JsonValue = string | number | boolean | null | JsonValue[] | { [key: string]: JsonValue }; numbers must be finite. App is the application object with readonly app.appHandle: string (opaque) and the methods listed below for app level. Window is the window object with readonly window.windowId: number and the methods listed below for window level. Neither object carries pid.
|
||||
|
||||
@@ -68,7 +68,7 @@ Element ids are opaque strings from the latest successful tree; the object facad
|
||||
|
||||
## Simulated mouse
|
||||
Use an explicit window for screenshots and pointer input.
|
||||
- getScreenshot(options?: { emit?: boolean }) -> Image — capture the selected window as a stored image with readonly path, width, and height. Pass selected screenshots to inspect(...values) to display them; read the window ID from window.windowId.
|
||||
- getScreenshot() -> Image — capture the selected window as a stored image with readonly path, width, and height. Pass selected screenshots to inspect(...values) to display them; read the window ID from window.windowId.
|
||||
- mouseMove(target: string | [number, number], options?: { relative?: boolean }) -> { dispatched: boolean, foregroundChanged?: boolean | null, hoverRoute: 'accessibility' | 'system-pointer', systemPointerMoved: boolean } — request a mouse move to the target without pressing a button; inspect hoverRoute and systemPointerMoved, and expect no automatic pointer restoration. Default hover requires both AXShowAlternateUI and AXShowDefaultUI on an enabled element or ancestor. Without an AX route it returns HOVER_UNAVAILABLE before sending input. Prefer click on a visible control or performAction with an advertised action. For web content without an AX hover action, use browser tools. Only systemPointerHover: true permits moving the visible cursor. If this returns FOREGROUND_REQUIRED, activate the app and check that the point is unobscured.
|
||||
- click(target: string | [number, number], options?: { mouseButton?: 'left' | 'right' | 'middle', clickCount?: 1 | 2, modifiers?: ('command' | 'control' | 'option' | 'shift' | 'fn')[], holdMs?: number, relative?: boolean }) -> ActionResult — click an element id or point, or perform a left-button double-click; holdMs controls the press duration of each press. click is a Window method: select a window with const win = await app.getWindow(windowId), then call win.click(target). For a background control’s default action, use app.invoke(id). For a mouse click on a background app, call app.activate() first and re-read the target. Both target forms send mouse-down and mouse-up events. click(id) uses the center of the full element bounds. If that center is outside the window, take a fresh win.getScreenshot() and use click([x, y]) on a visible part of the control in screenshot pixels. A background app can ignore the events even when dispatched is true. Verify the expected effect, such as a changed selection, before typing or retrying.
|
||||
- scroll(options: { x: number, y: number, deltaLinesX: number, deltaLinesY: number, relative?: boolean }) -> ActionResult — scroll at a point; deltas are integers from -10 to 10 with at least one nonzero, and positive values reveal content above or left.
|
||||
|
||||
@@ -39,7 +39,7 @@ macOS: accessibility tree operations are available at app or window level. Simul
|
||||
|
||||
Use cua to observe and operate native applications. Always use cua.getCapabilities() to confirm whether a specific API is available on the current platform; its reported capabilities take precedence over availability statements in these instructions. All methods are async; observation options default to { emit: true }, and emit: false suppresses sink output only, leaving returned values unchanged. Actions report dispatch evidence, not verified completion. If sessionLocked is true or an operation reports SESSION_LOCKED, stop UI actions and ask the user to unlock the desktop session; null or unknown means verify a usable unlocked desktop session first. cua.getApp requests user consent; if access is denied or revoked, stop using that application until the user explicitly permits another request, and never bypass a refusal or switch tools to avoid it. Never replay uncertain input without a fresh observation.
|
||||
|
||||
In persistent PTC, use the host-provided bridge initialization once and retain the returned objects; cua comes from that bridge and is not a preinstalled global. Observations emit through inspect; never stringify screenshot image bytes.
|
||||
In persistent PTC, use the host-provided bridge initialization once and retain the returned objects; cua comes from that bridge and is not a preinstalled global. Text observations emit through inspect.
|
||||
|
||||
Signatures show resolved return values after await. ActionResult = { dispatched: boolean, foregroundChanged?: boolean | null }; foregroundChanged is omitted when not sampled and null when the sample is unavailable. JsonValue = string | number | boolean | null | JsonValue[] | { [key: string]: JsonValue }; numbers must be finite. App is the application object with readonly app.appHandle: string (opaque) and the methods listed below for app level. Window is the window object with readonly window.windowId: number and the methods listed below for window level. Neither object carries pid.
|
||||
|
||||
@@ -68,7 +68,7 @@ Element ids are opaque strings from the latest successful tree; the object facad
|
||||
|
||||
## Simulated mouse
|
||||
Use an explicit window for screenshots and pointer input.
|
||||
- getScreenshot(options?: { emit?: boolean }) -> Image — capture the selected window as a stored image with readonly path, width, and height. Pass selected screenshots to inspect(...values) to display them; read the window ID from window.windowId.
|
||||
- getScreenshot() -> Image — capture the selected window as a stored image with readonly path, width, and height. Pass selected screenshots to inspect(...values) to display them; read the window ID from window.windowId.
|
||||
- mouseMove(target: string | [number, number], options?: { relative?: boolean }) -> { dispatched: boolean, foregroundChanged?: boolean | null, hoverRoute: 'accessibility' | 'system-pointer', systemPointerMoved: boolean } — request a mouse move to the target without pressing a button; inspect hoverRoute and systemPointerMoved, and expect no automatic pointer restoration. Default hover requires both AXShowAlternateUI and AXShowDefaultUI on an enabled element or ancestor. Without an AX route it returns HOVER_UNAVAILABLE before sending input. Prefer click on a visible control or performAction with an advertised action. For web content without an AX hover action, use browser tools. Only systemPointerHover: true permits moving the visible cursor. If this returns FOREGROUND_REQUIRED, activate the app and check that the point is unobscured.
|
||||
- click(target: string | [number, number], options?: { mouseButton?: 'left' | 'right' | 'middle', clickCount?: 1 | 2, modifiers?: ('command' | 'control' | 'option' | 'shift' | 'fn')[], holdMs?: number, relative?: boolean }) -> ActionResult — click an element id or point, or perform a left-button double-click; holdMs controls the press duration of each press. click is a Window method: select a window with const win = await app.getWindow(windowId), then call win.click(target). For a background control’s default action, use app.invoke(id). For a mouse click on a background app, call app.activate() first and re-read the target. Both target forms send mouse-down and mouse-up events. click(id) uses the center of the full element bounds. If that center is outside the window, take a fresh win.getScreenshot() and use click([x, y]) on a visible part of the control in screenshot pixels. A background app can ignore the events even when dispatched is true. Verify the expected effect, such as a changed selection, before typing or retrying.
|
||||
- scroll(options: { x: number, y: number, deltaLinesX: number, deltaLinesY: number, relative?: boolean }) -> ActionResult — scroll at a point; deltas are integers from -10 to 10 with at least one nonzero, and positive values reveal content above or left.
|
||||
|
||||
Reference in new issue
Block a user