fix(cua): normalize relative coordinates to 0–999

This commit is contained in:
creatixchu committed 2026-10-21 12:46:20 +08:00
1 parent 224516ec2f
commit 7e8bf2808d
16 files changed
+102 -29

No files matched your search

@@ -0,0 +1,16 @@
# Bilingual-pair consistency record for guide.md (docs/i18n/README.md): per heading
# section, a hash of its English and Chinese blocks outside code blocks and generated regions.
# After editing either side, bring the other along and re-record with:
# pnpm run verify-translation-pairing --write docs/upgrade-guide/v0.2.1-rc.2/cua-relative-coordinates/guide.md
/:
en: f96c9622117dfe19
zh: 7913a9d0d66068d1
/cua-relative-coordinates-use-0-999:
en: 1952d155be13e0d1
zh: 01b2f9f27499f033
/cua-relative-coordinates-use-0-999/change:
en: 757250b2702a35f1
zh: ce715cd07bfcc951
/cua-relative-coordinates-use-0-999/migration:
en: fe92d979c7edb475
zh: 9722f1adf7e3c920
@@ -0,0 +1,17 @@
---
kind: upgrade-guide
description: "CUA relative coordinates accept integers from 0 to 999 and normalize by 999."
---
# CUA relative coordinates use 0–999
English | [中文](guide.zh.md)
## Change
The `cua` tool and application/window object methods now accept integers from 0 to 999 when `relative: true`. The previous range was 0 to 1000. Coordinates are divided by 999 instead of 1000 before scaling to window dimensions; the upper endpoint stays inside the window. This affects `click`, `hover`, `scroll`, and both `drag` endpoints. Absolute screenshot pixels and AX element indices are unchanged.
## Migration
1. Update scripts and prompts that generate relative coordinates. For a normalized fraction `r` between 0 and 1, use `Math.round(r * 999)`. Convert stored coordinates on the old scale with `Math.round(oldValue * 999 / 1000)`; an old endpoint of 1000 becomes 999.
2. Read fresh window state, then invoke the affected action with integer coordinates in 0–999. Verify the result with a fresh observation. Relative values of 1000, negative values, and fractions are rejected before native input.
@@ -0,0 +1,17 @@
---
kind: upgrade-guide
description: "CUA 相对坐标接受 0~999 的整数,并按 999 归一化。"
---
# CUA 相对坐标使用 0~999
[English](guide.md) | 中文
## 变更
`cua` 工具和应用或窗口对象方法在 `relative: true` 时接受 0~999 的整数,原范围为 0~1000。坐标按窗口尺寸缩放前的除数由 1000 改为 999,上限坐标仍位于窗口内部。这影响 `click`、`hover`、`scroll` 和 `drag` 的起终点。绝对截图像素和 AX 元素编号保持原有行为。
## 迁移
1. 更新生成相对坐标的脚本和提示。对于 0~1 的归一化比例 `r`,使用 `Math.round(r * 999)`。已保存的旧坐标使用 `Math.round(oldValue * 999 / 1000)` 转换;旧端点 1000 转换为 999。
2. 读取新的窗口状态后,使用 0~999 的整数坐标调用相应动作,再读取新状态验证结果。相对坐标为 1000、负数或小数时,会在原生输入前被拒绝。
+1 -1
View File
@@ -155,7 +155,7 @@ The plugin contributes one `cua` operation tool with method-specific parameters.
##### Operation guidance (verbatim)
```markdown
Use cua with method getCapabilities and params {} to inspect permissions and sessionLocked. If sessionLocked is true or an operation reports SESSION_LOCKED, stop UI attempts and ask the user to unlock macOS. A null sessionLocked means no usable console session was verified. After unlocking, read fresh state; screenshots from a locked desktop are not current observations. Use listApps or cua.getState to discover running applications, and findApps(query) to search installed applications by name, bundle ID or absolute path. getApp requests user consent before launching a stopped application in the background, or before observations and input on a running one. If access is denied or revoked, stop using that application until the user explicitly permits another authorization request. Never switch tools to bypass a refusal. Keep the returned appHandle; every app operation requires it. listWindows returns window IDs and point bounds. Raw getState returns AX text and snapshotId. Menu-bar entries are shown by default; includeMenus=true expands their contents. details=true adds identifiers, global-point bounds and complete action names. Both options also apply to getAXState. Each read replaces the indices for that app/window scope. Use its element indices only with that snapshotId. Raw screenshot and coordinate operations require a windowId. Coordinates are pixels in the latest returned screenshot, measured from its top-left. Set relative:true for integer thousandths from 0 to 1000 instead. The Host converts both modes to native window points; AX detail bounds remain global macOS points. A verified window can expose no AX elements. Its empty snapshot permits screenshots and coordinate input; element-index actions require an element from a fresh AX observation.
Use cua with method getCapabilities and params {} to inspect permissions and sessionLocked. If sessionLocked is true or an operation reports SESSION_LOCKED, stop UI attempts and ask the user to unlock macOS. A null sessionLocked means no usable console session was verified. After unlocking, read fresh state; screenshots from a locked desktop are not current observations. Use listApps or cua.getState to discover running applications, and findApps(query) to search installed applications by name, bundle ID or absolute path. getApp requests user consent before launching a stopped application in the background, or before observations and input on a running one. If access is denied or revoked, stop using that application until the user explicitly permits another authorization request. Never switch tools to bypass a refusal. Keep the returned appHandle; every app operation requires it. listWindows returns window IDs and point bounds. Raw getState returns AX text and snapshotId. Menu-bar entries are shown by default; includeMenus=true expands their contents. details=true adds identifiers, global-point bounds and complete action names. Both options also apply to getAXState. Each read replaces the indices for that app/window scope. Use its element indices only with that snapshotId. Raw screenshot and coordinate operations require a windowId. Coordinates are pixels in the latest returned screenshot, measured from its top-left. Set relative:true for integer coordinates from 0 to 999, normalized by 999 to the window width and height. The Host converts both modes to native window points; AX detail bounds remain global macOS points. A verified window can expose no AX elements. Its empty snapshot permits screenshots and coordinate input; element-index actions require an element from a fresh AX observation.
Initialize the object API once in persistent PTC. Reuse its variables in later calls; put temporary declarations inside { ... } blocks to avoid redeclaration errors. All object methods are async. getAXState returns text, not a structured state object. Its text separates values and placeholders, and reports subroles, help, URLs, disabled/focused/selected/expanded state and UTF-16 selection ranges when available. Writability is true, false or unknown on relevant controls; omitted state fields do not assert false. selection=unavailable means the reported range exceeds the observed text. Compact output omits ordinary Press/Pick (click), ShowMenu (right click) and ScrollToVisible actions; details=true lists every available action. Use the displayed action name with performSecondaryAction; ambiguous aliases require the complete name from details=true. Truncation reports the reached provider limit. Narrow the window or increase that provider limit before reading again.
+1 -1
View File
@@ -155,7 +155,7 @@ AppKit sheet 可以通过其窗口 ID 单独选择,也会出现在所属文档
##### 操作指导文本(原文)
```markdown
Use cua with method getCapabilities and params {} to inspect permissions and sessionLocked. If sessionLocked is true or an operation reports SESSION_LOCKED, stop UI attempts and ask the user to unlock macOS. A null sessionLocked means no usable console session was verified. After unlocking, read fresh state; screenshots from a locked desktop are not current observations. Use listApps or cua.getState to discover running applications, and findApps(query) to search installed applications by name, bundle ID or absolute path. getApp requests user consent before launching a stopped application in the background, or before observations and input on a running one. If access is denied or revoked, stop using that application until the user explicitly permits another authorization request. Never switch tools to bypass a refusal. Keep the returned appHandle; every app operation requires it. listWindows returns window IDs and point bounds. Raw getState returns AX text and snapshotId. Menu-bar entries are shown by default; includeMenus=true expands their contents. details=true adds identifiers, global-point bounds and complete action names. Both options also apply to getAXState. Each read replaces the indices for that app/window scope. Use its element indices only with that snapshotId. Raw screenshot and coordinate operations require a windowId. Coordinates are pixels in the latest returned screenshot, measured from its top-left. Set relative:true for integer thousandths from 0 to 1000 instead. The Host converts both modes to native window points; AX detail bounds remain global macOS points. A verified window can expose no AX elements. Its empty snapshot permits screenshots and coordinate input; element-index actions require an element from a fresh AX observation.
Use cua with method getCapabilities and params {} to inspect permissions and sessionLocked. If sessionLocked is true or an operation reports SESSION_LOCKED, stop UI attempts and ask the user to unlock macOS. A null sessionLocked means no usable console session was verified. After unlocking, read fresh state; screenshots from a locked desktop are not current observations. Use listApps or cua.getState to discover running applications, and findApps(query) to search installed applications by name, bundle ID or absolute path. getApp requests user consent before launching a stopped application in the background, or before observations and input on a running one. If access is denied or revoked, stop using that application until the user explicitly permits another authorization request. Never switch tools to bypass a refusal. Keep the returned appHandle; every app operation requires it. listWindows returns window IDs and point bounds. Raw getState returns AX text and snapshotId. Menu-bar entries are shown by default; includeMenus=true expands their contents. details=true adds identifiers, global-point bounds and complete action names. Both options also apply to getAXState. Each read replaces the indices for that app/window scope. Use its element indices only with that snapshotId. Raw screenshot and coordinate operations require a windowId. Coordinates are pixels in the latest returned screenshot, measured from its top-left. Set relative:true for integer coordinates from 0 to 999, normalized by 999 to the window width and height. The Host converts both modes to native window points; AX detail bounds remain global macOS points. A verified window can expose no AX elements. Its empty snapshot permits screenshots and coordinate input; element-index actions require an element from a fresh AX observation.
Initialize the object API once in persistent PTC. Reuse its variables in later calls; put temporary declarations inside { ... } blocks to avoid redeclaration errors. All object methods are async. getAXState returns text, not a structured state object. Its text separates values and placeholders, and reports subroles, help, URLs, disabled/focused/selected/expanded state and UTF-16 selection ranges when available. Writability is true, false or unknown on relevant controls; omitted state fields do not assert false. selection=unavailable means the reported range exceeds the observed text. Compact output omits ordinary Press/Pick (click), ShowMenu (right click) and ScrollToVisible actions; details=true lists every available action. Use the displayed action name with performSecondaryAction; ambiguous aliases require the complete name from details=true. Truncation reports the reached provider limit. Narrow the window or increase that provider limit before reading again.
@@ -1,6 +1,6 @@
/** macOS operation guidance supplied with the shared CUA object API. */
/** Instructions for raw operation calls and optional object-API bindings. */
export const GUIDANCE = `Use cua with method getCapabilities and params {} to inspect permissions and sessionLocked. If sessionLocked is true or an operation reports SESSION_LOCKED, stop UI attempts and ask the user to unlock macOS. A null sessionLocked means no usable console session was verified. After unlocking, read fresh state; screenshots from a locked desktop are not current observations. Use listApps or cua.getState to discover running applications, and findApps(query) to search installed applications by name, bundle ID or absolute path. getApp requests user consent before launching a stopped application in the background, or before observations and input on a running one. If access is denied or revoked, stop using that application until the user explicitly permits another authorization request. Never switch tools to bypass a refusal. Keep the returned appHandle; every app operation requires it. listWindows returns window IDs and point bounds. Raw getState returns AX text and snapshotId. Menu-bar entries are shown by default; includeMenus=true expands their contents. details=true adds identifiers, global-point bounds and complete action names. Both options also apply to getAXState. Each read replaces the indices for that app/window scope. Use its element indices only with that snapshotId. Raw screenshot and coordinate operations require a windowId. Coordinates are pixels in the latest returned screenshot, measured from its top-left. Set relative:true for integer thousandths from 0 to 1000 instead. The Host converts both modes to native window points; AX detail bounds remain global macOS points. A verified window can expose no AX elements. Its empty snapshot permits screenshots and coordinate input; element-index actions require an element from a fresh AX observation.
export const GUIDANCE = `Use cua with method getCapabilities and params {} to inspect permissions and sessionLocked. If sessionLocked is true or an operation reports SESSION_LOCKED, stop UI attempts and ask the user to unlock macOS. A null sessionLocked means no usable console session was verified. After unlocking, read fresh state; screenshots from a locked desktop are not current observations. Use listApps or cua.getState to discover running applications, and findApps(query) to search installed applications by name, bundle ID or absolute path. getApp requests user consent before launching a stopped application in the background, or before observations and input on a running one. If access is denied or revoked, stop using that application until the user explicitly permits another authorization request. Never switch tools to bypass a refusal. Keep the returned appHandle; every app operation requires it. listWindows returns window IDs and point bounds. Raw getState returns AX text and snapshotId. Menu-bar entries are shown by default; includeMenus=true expands their contents. details=true adds identifiers, global-point bounds and complete action names. Both options also apply to getAXState. Each read replaces the indices for that app/window scope. Use its element indices only with that snapshotId. Raw screenshot and coordinate operations require a windowId. Coordinates are pixels in the latest returned screenshot, measured from its top-left. Set relative:true for integer coordinates from 0 to 999, normalized by 999 to the window width and height. The Host converts both modes to native window points; AX detail bounds remain global macOS points. A verified window can expose no AX elements. Its empty snapshot permits screenshots and coordinate input; element-index actions require an element from a fresh AX observation.
Initialize the object API once in persistent PTC. Reuse its variables in later calls; put temporary declarations inside { ... } blocks to avoid redeclaration errors. All object methods are async. getAXState returns text, not a structured state object. Its text separates values and placeholders, and reports subroles, help, URLs, disabled/focused/selected/expanded state and UTF-16 selection ranges when available. Writability is true, false or unknown on relevant controls; omitted state fields do not assert false. selection=unavailable means the reported range exceeds the observed text. Compact output omits ordinary Press/Pick (click), ShowMenu (right click) and ScrollToVisible actions; details=true lists every available action. Use the displayed action name with performSecondaryAction; ambiguous aliases require the complete name from details=true. Truncation reports the reached provider limit. Narrow the window or increase that provider limit before reading again.
@@ -453,8 +453,8 @@ it('maps the actual model-request image pixels back to native points without sto
native.request.mockResolvedValue({ dispatched: true, verified: false })
expect((await f.execute('click', { appHandle: 'app-one', windowId: 7, x: 5, y: 2.5 })).isError).toBe(false)
expect(native.request).toHaveBeenLastCalledWith('click', { appHandle: 'app-one', windowId: 7, x: 200, y: 100 }, expect.any(AbortSignal), expect.any(String))
expect((await f.execute('click', { appHandle: 'app-one', windowId: 7, x: 500, y: 500, relative: true })).isError).toBe(false)
expect(native.request).toHaveBeenLastCalledWith('click', { appHandle: 'app-one', windowId: 7, x: 200, y: 100 }, expect.any(AbortSignal), expect.any(String))
expect((await f.execute('click', { appHandle: 'app-one', windowId: 7, x: 999, y: 999, relative: true })).isError).toBe(false)
expect(native.request).toHaveBeenLastCalledWith('click', { appHandle: 'app-one', windowId: 7, x: 399, y: 199 }, expect.any(AbortSignal), expect.any(String))
})
it.each([{ pixelWidth: 2 }, { pixelHeight: 2 }])('rejects native screenshot dimensions that disagree with decoded bytes: %j', async (invalid) => {
+2 -2
View File
@@ -15,8 +15,8 @@
en: 4081cf444a77bf20
zh: def91d58969ec281
/deepseek-ai-dsh-cua/use-this-package:
en: ceb54d964dc355b6
zh: 0892066fbceef7b9
en: 20d3ea010381090d
zh: 29d36bcafb1f4cfa
/deepseek-ai-dsh-cua/use-this-package/permission-controls:
en: be53792edc381ea1
zh: 037407055d9f34a0
+1 -1
View File
@@ -37,7 +37,7 @@ Observation methods emit and return their result. Use `void await app.getAXState
Application consent precedes launch, observation, capture, and input. Session access is kept in memory by session and application identity, survives agent and helper replacement, and expires when DSH restarts or Computer Use is disabled. Saved grants require platform-verified executable identity. Refusal and revocation close affected Agent connections, so their old objects require fresh selection. No input is replayed after cancellation, timeout, or reconnect.
Actions return `dispatched:true, verified:false`; read fresh state to verify the result. Coordinate actions default to pixels in the latest returned screenshot; `relative: true` accepts integers from 0 to 1000 after a successful window observation. The Host resolves attachment normalization and the selected model route’s image projection before returning the screenshot, then converts input to native window points. Resizing invalidates the screenshot mapping; `SCREENSHOT_REQUIRED` requires a new screenshot for absolute coordinates. Resizing rejects old coordinates with `WINDOW_CHANGED`; movement alone preserves window-local coordinates.
Actions return `dispatched:true, verified:false`; read fresh state to verify the result. Coordinate actions default to pixels in the latest returned screenshot; `relative: true` accepts integers from 0 to 999 after a successful window observation. Values are divided by 999 and scaled to the window dimensions; 999 is clamped to the last point inside the window. The Host resolves attachment normalization and the selected model route’s image projection before returning the screenshot, then converts input to native window points. Resizing invalidates the screenshot mapping; `SCREENSHOT_REQUIRED` requires a new screenshot for absolute coordinates. Resizing rejects old coordinates with `WINDOW_CHANGED`; movement alone preserves window-local coordinates.
`click(target, { holdMs: 800 })` holds the mouse button for 800 milliseconds before release. `holdMs` accepts nonnegative integers and applies to each press of a double-click; zero releases without an additional hold. The provider config `maxClickHoldMs` limits explicit holds and defaults to 5000 ms. Explicit holds use pointer input for both AX indices and coordinates; omission preserves ordinary click behavior. The operation deadline includes the hold, and cancellation or timeout releases the button. Received input cannot be undone; forced helper termination cannot guarantee release.
+1 -1
View File
@@ -37,7 +37,7 @@ kind: "package-reference"
应用授权发生在启动、观察、截图和输入之前。会话许可按会话和应用身份保存在内存中,Agent 或 helper 重建后仍可复用,重启 DSH 或关闭 Computer Use 后失效。长期许可要求平台验证可执行文件身份。拒绝或撤销会关闭受影响的 Agent 连接,因此旧对象必须重新选择目标。取消、超时或重新连接后均不重放输入。
动作返回 `dispatched:true, verified:false`;请读取新状态以验证结果。坐标操作默认使用最近返回截图中的像素;成功观察窗口后,传入 `relative: true` 可使用 0~1000 的整数比例。Host 在返回截图前解析附件归一化和当前模型的图片缩放规则,再将输入换算为原生窗口 point。窗口尺寸变化会使截图映射失效,绝对坐标遇到 `SCREENSHOT_REQUIRED` 时需要重新截图。窗口缩放后以 `WINDOW_CHANGED` 拒绝旧坐标;单纯移动窗口不改变窗口内坐标。
动作返回 `dispatched:true, verified:false`;请读取新状态以验证结果。坐标操作默认使用最近返回截图中的像素;成功观察窗口后,传入 `relative: true` 可使用 0~999 的整数坐标。数值除以 999 后按窗口尺寸缩放;999 会限制在窗口内最后一个 point。Host 在返回截图前解析附件归一化和当前模型的图片缩放规则,再将输入换算为原生窗口 point。窗口尺寸变化会使截图映射失效,绝对坐标遇到 `SCREENSHOT_REQUIRED` 时需要重新截图。窗口缩放后以 `WINDOW_CHANGED` 拒绝旧坐标;单纯移动窗口不改变窗口内坐标。
`click(target, { holdMs: 800 })` 按住鼠标按钮 800 毫秒后松开。`holdMs` 接受非负整数,双击时应用于每次按下;0 表示按下后立即松开,不额外等待。提供方配置 `maxClickHoldMs` 限制显式按住时长,默认 5000 毫秒。指定时长后,AX 索引和坐标目标都使用鼠标事件;省略时保留普通点击行为。按住时间计入操作超时,取消或超时会松开按钮。已接收的输入无法撤销;辅助进程被强制终止时无法保证松开。
+1 -1
View File
@@ -62,7 +62,7 @@ export class CuaCoordinates {
if (!relative && !frame.image) throw new NativeError('SCREENSHOT_REQUIRED', 'Take a fresh screenshot before using absolute pixel coordinates')
const convert = (value: number, axis: 'width' | 'height'): number => {
const pixels = frame.image?.[axis] ?? frame[axis]
if (relative) return Math.min(value / 1000 * frame[axis], Math.max(0, frame[axis] - 1))
if (relative) return Math.min(value / 999 * frame[axis], Math.max(0, frame[axis] - 1))
if (value >= pixels) throw new NativeError('INVALID_ARGUMENT', 'Coordinate is outside the latest screenshot')
return value * frame[axis] / pixels
}
+2 -2
View File
@@ -6,7 +6,7 @@ import type { CuaAppHandle, CuaSnapshotId } from './types.ts'
const id = z.number().int().positive().max(2 ** 32 - 1)
const pid = z.number().int().positive().max(2 ** 31 - 1)
const point = z.number().nonnegative()
const relativePoint = z.number().int().min(0).max(1000)
const relativePoint = z.number().int().min(0).max(999)
const app = { appHandle: z.string().min(1) }
const pointer = { mouseButton: z.enum(['left', 'right', 'middle']).optional(),
modifiers: z.array(z.enum(['command', 'control', 'option', 'shift', 'fn'])).optional() }
@@ -17,7 +17,7 @@ const window = { ...app, windowId: id }
const snapshot = { ...app, snapshotId: z.string().min(1), elementIndex: z.number().int().nonnegative() }
const coordinate = { ...window, snapshotId: z.string().min(1).optional(), x: point, y: point }
const absoluteCoordinate = { ...coordinate, relative: z.literal(false).optional().describe('Omitted or false uses pixels in the latest screenshot returned by this tool.') }
const relativeCoordinate = { ...coordinate, relative: z.literal(true).describe('Use integer thousandths of the window width and height, from 0 to 1000.'), x: relativePoint, y: relativePoint }
const relativeCoordinate = { ...coordinate, relative: z.literal(true).describe('Use integer coordinates from 0 to 999, normalized by 999 to the window width and height.'), x: relativePoint, y: relativePoint }
const bounds = z.object({
x: z.number(), y: z.number(), width: z.number().nonnegative(), height: z.number().nonnegative(),
})
+5 -2
View File
@@ -55,9 +55,12 @@ export type CuaWindowInfo = z.output<typeof WindowsResult>[number]
export type CuaActionResult = z.output<typeof ActionResult>
/** Hover dispatch and the effect on the user's system pointer. */
export type CuaHoverResult = z.output<typeof HoverResult>
/** Select screenshot pixels by default or integer thousandths when explicitly requested. */
/** Select screenshot pixels by default or normalized integer coordinates when explicitly requested. */
export interface CuaCoordinateOptions {
/** True uses integers from 0 to 1000; omitted or false uses the latest screenshot's pixels. Applies only to coordinate targets. */
/**
* True uses integers from 0 to 999, normalized by 999; omitted or false uses the latest screenshot's pixels.
* Applies only to coordinate targets.
*/
relative?: boolean
}
/** Restoration dispatch; foregroundChanged compares the front app and top window before and after the request. */
+3 -3
View File
@@ -425,12 +425,12 @@ describe('input actions', () => {
const app = await f.api.getApp('Editor')
await app.click([500, 250], { relative: true })
await app.hover([500, 250], { relative: true })
await app.drag([0, 0], [1000, 1000], { relative: true })
await app.drag([0, 0], [999, 999], { relative: true })
await app.scroll({ x: 500, y: 250, relative: true, deltaLinesX: 0, deltaLinesY: -1 })
expect(f.dispatch.mock.calls.slice(-4).map(([request]) => request.params)).toEqual([
expect.objectContaining({ x: 500, y: 250, relative: true }),
expect.objectContaining({ x: 500, y: 250, relative: true }),
expect.objectContaining({ fromX: 0, toX: 1000, relative: true }),
expect.objectContaining({ fromX: 0, toX: 999, relative: true }),
expect.objectContaining({ x: 500, y: 250, relative: true }),
])
await expect(app.click(1, { relative: true })).rejects.toThrow()
@@ -492,7 +492,7 @@ describe('input actions', () => {
['getState windowId: Too small: expected number to be >0', app => app.getWindow(0)],
['selectText text: Too small: expected string to have >=1 characters', app => app.selectText(0, '')],
['hover x: Invalid input: expected number, received NaN', app => app.hover([NaN, 0])],
['hover x: Too big: expected number to be <=1000', app => app.hover([1001, 0], { relative: true })],
['hover x: Too big: expected number to be <=999', app => app.hover([1000, 0], { relative: true })],
['click relative: not accepted for these arguments', app => app.click(650, { relative: true })],
])('reports a readable validation failure: %s', async (message, run) => {
const f = fixture()
@@ -22,14 +22,31 @@ it('maps Retina and downscaled screenshot pixels to the same native point', () =
}
})
it('maps integer ratios after AX observation and keeps endpoint 1000 inside the window', () => {
it.each(['click', 'hover', 'scroll'])('maps 0–999 coordinates for %s independently of screenshot resolution', (method) => {
const coordinates = new CuaCoordinates()
coordinates.observe(handle, [window()])
expect(coordinates.parameters(call('hover', { x: 500, y: 250, relative: true }))).toEqual({ appHandle: handle, windowId: 7, x: 320, y: 120 })
expect(coordinates.parameters(call('click', { x: 1000, y: 1000, relative: true }))).toMatchObject({ x: 639, y: 479 })
coordinates.screenshot(handle, 7, window().bounds, { width: 1280, height: 960 })
expect(coordinates.parameters(call('click', { x: 1000, y: 1000, relative: true }))).toMatchObject({ x: 639, y: 479 })
expect(coordinates.parameters(call('click', { x: 0, y: 0, relative: true }))).toMatchObject({ x: 0, y: 0 })
const bounds = window(1200, 900).bounds
coordinates.observe(handle, [window(1200, 900)])
const params = { relative: true, ...method === 'scroll' ? { deltaLinesX: 0, deltaLinesY: -1 } : {} }
const assertPoints = () => {
expect(coordinates.parameters(call(method, { ...params, x: 333, y: 666 }))).toMatchObject({ x: 400, y: 600 })
expect(coordinates.parameters(call(method, { ...params, x: 999, y: 999 }))).toMatchObject({ x: 1199, y: 899 })
expect(coordinates.parameters(call(method, { ...params, x: 0, y: 0 }))).toMatchObject({ x: 0, y: 0 })
}
assertPoints()
for (const image of [{ width: 2400, height: 1800 }, { width: 300, height: 225 }]) {
coordinates.screenshot(handle, 7, bounds, image)
assertPoints()
}
})
it('maps relative drag endpoints and keeps 999 inside a one-point window', () => {
const coordinates = new CuaCoordinates()
coordinates.observe(handle, [window(1200, 900)])
const params = { fromX: 0, fromY: 333, toX: 999, toY: 666, relative: true }
expect(coordinates.parameters(call('drag', params))).toEqual({ appHandle: handle, windowId: 7,
fromX: 0, fromY: 300, toX: 1199, toY: 600 })
coordinates.observe(handle, [window(1, 1)])
expect(coordinates.parameters(call('click', { x: 999, y: 999, relative: true }))).toMatchObject({ x: 0, y: 0 })
})
it('maps both drag endpoints and preserves button, modifiers and snapshot identity', () => {
@@ -48,7 +65,7 @@ it('preserves pixels after movement but requires a new image after resize or clo
expect(coordinates.parameters(request)).toMatchObject({ x: 320, y: 240 })
coordinates.observe(handle, [window(800, 600)])
expect(() => coordinates.parameters(request)).toThrow('SCREENSHOT_REQUIRED')
expect(coordinates.parameters(call('click', { x: 500, y: 500, relative: true }))).toMatchObject({ x: 400, y: 300 })
expect(coordinates.parameters(call('click', { x: 999, y: 999, relative: true }))).toMatchObject({ x: 799, y: 599 })
coordinates.observe(handle, [])
expect(() => coordinates.parameters(request)).toThrow('STALE_SNAPSHOT')
})
@@ -76,13 +76,16 @@ it.each([
expect(Request.safeParse({ method: 'hover', params }).success).toBe(false)
})
it.each(['click', 'hover', 'scroll', 'drag'])('validates integer thousandths for %s', (method) => {
it.each(['click', 'hover', 'scroll', 'drag'])('validates 0–999 integer coordinates for %s', (method) => {
const params = { appHandle: 'app', windowId: 7, relative: true,
...method === 'drag' ? { fromX: 0, fromY: 0, toX: 1000, toY: 1000 } : { x: 0, y: 1000 },
...method === 'drag' ? { fromX: 0, fromY: 0, toX: 999, toY: 999 } : { x: 0, y: 999 },
...method === 'scroll' ? { deltaLinesX: 0, deltaLinesY: -1 } : {},
}
expect(Request.safeParse({ method, params }).success).toBe(true)
for (const value of [-1, 1001, 0.5]) {
expect(Request.safeParse({ method, params: { ...params, [method === 'drag' ? 'toX' : 'x']: value } }).success).toBe(false)
for (const field of method === 'drag' ? ['fromX', 'fromY', 'toX', 'toY'] : ['x', 'y']) {
for (const value of [-1, 1000, 0.5]) {
expect(Request.safeParse({ method, params: { ...params, [field]: value } }).success).toBe(false)
}
expect(Request.safeParse({ method, params: { ...params, relative: false, [field]: 1000 } }).success).toBe(true)
}
})