fix(benchmarks): sample Trajectory readiness on animation frames

This commit is contained in:
Tianyi Cui committed 2026-09-13 14:03:32 +08:00
1 parent 99a73ca1d4
commit c042eb2560
7 files changed
+56 -6

No files matched your search

@@ -2,5 +2,5 @@
# side as of the last confirmed-consistent state. Both languages carry equal authority;
# after editing either side, bring the other along and re-record with:
# pnpm run verify-translation-pairing --write .agents/notes/implemented/testing/2026-09-06-frontend-performance-budgets.md
2026-09-06-frontend-performance-budgets.md: 2ccc992add9a1f7fc7824a3e0ed604a9582def56
2026-09-06-frontend-performance-budgets.zh.md: bfe31fd4c8c61b3b89336a8d9d8234d1e76ba040
2026-09-06-frontend-performance-budgets.md: 36a7e6fc97a2ca69d5b58e9cebae4278344c659d
2026-09-06-frontend-performance-budgets.zh.md: bdb27d782b97bde4a02f116de0f0522a0a1531c5
@@ -68,6 +68,14 @@ Run 34036109842, job 101494445658 records open samples of 875.306861/1083.683529
A controlled mouse-refocus delay waits for the real DONE marker without pausing replay: the mouse path rejects a trusted input after DONE, while Enter submission and keyboard-only draft input pass all three samples under the same control. The delay is diagnostic-only. A clean three-sample run on arm64 Node 24.19.0 / Chromium 149.0.7827.55 reports first-reply/input/complete-wall medians of 288.823/418.868/2567.328 ms, with actual overlap and post-DONE rejection in every sample. This proves removal of the mouse-action scheduling dependency, not the cause of a particular hosted stall; all workload constants and budgets remain fixed.
### Trajectory readiness polling
Trajectory activation samples its visible searchbox and a visible record inside `[data-trajectory-scroll]` on animation frames. Playwright's selector polling uses increasing delays up to 500 ms, so DOM readiness can precede the observer by hundreds of milliseconds. The trusted tab click, its actionability checks and the final two-frame wait remain timed; workload and the 625 ms budget are unchanged.
A controlled macOS arm64 diagnostic on Node 24.19.0, Chromium 149.0.7827.55 and Playwright 1.61.1 makes the searchbox and record ready after 300 or 350 ms. Selector waits take 793.082/791.053 ms and observe readiness 493.5/439.2 ms late; frame sampling takes 307.324/357.423 ms and observes it 8.1/7.9 ms late. This isolates conditional observer delay, not product latency or the cause of the first #4119 CI miss.
Complete local 240-turn runs produce Trajectory samples of 157.795/146.737/148.571 ms before the observer change and 130.184/191.247/157.543 ms afterward, with medians 148.571 and 157.543 ms. Both runs pass every fixed endpoint budget; no local speedup or hosted calibration is claimed. The readiness regression rejects a hidden searchbox or record despite a visible unrelated table row; restoring the original selector waits makes that test fail.
## Alternatives considered
**Use the Node fold as paint evidence.** Rejected because it never performs DOM mutation, layout, or browser scheduling. The focused reconnect case likewise makes no GUI speed claim.
@@ -78,6 +86,8 @@ A controlled mouse-refocus delay waits for the real DONE marker without pausing
**Search the entire loaded history for every stream marker.** Rejected because Playwright injects text and accessibility scans into the same renderer whose CPU the benchmark measures. Scoping queries to the composer and latest Assistant preserves visible completion checks without making observer cost proportional to loaded history.
**Raise the Trajectory budget or retry failed samples.** This retains the measured selector backoff and can accept slower product behavior. Frame-based readiness removes that observer delay without changing the budget.
**Measure stream CPU alone.** Rejected because transport stalls and final-settlement delays can leave main-thread CPU low. The independent input, first-reply, and complete-wall budgets cover those waits.
## Consequences
@@ -68,6 +68,14 @@ Node 对话折叠很快,并不能证明浏览器能绘制长对话或在流式
受控的鼠标重新聚焦延迟等待真实 DONE 标记,不暂停重放:鼠标路径拒绝 DONE 之后的真实输入,而 Enter 提交与纯键盘草稿输入在相同对照下通过全部三个样本。该延迟仅用于诊断。在 arm64 Node 24.19.0 / Chromium 149.0.7827.55 上,不含延迟的三个样本报告首段回复/输入/完整壁钟中位数 288.823/418.868/2567.328 ms,每个样本均满足实际重叠并拒绝 DONE 之后的输入。这证明移除了鼠标操作调度依赖,并不证明某次托管停顿的原因;全部工作负载常量和预算保持固定。
### Trajectory 就绪轮询
Trajectory 激活在动画帧上采样可见搜索框与 `[data-trajectory-scroll]` 内的可见记录。Playwright 选择器轮询使用逐步增加、最高 500 ms 的延迟,因此 DOM 可能比观察器早数百毫秒就绪。真实 tab 点击、其可交互性检查与最后两帧等待仍计时;工作负载与 625 ms 预算不变。
在 macOS arm64、Node 24.19.0、Chromium 149.0.7827.55 和 Playwright 1.61.1 上,受控诊断使搜索框与记录在 300 或 350 ms 后就绪。选择器等待耗时 793.082/791.053 ms,晚 493.5/439.2 ms 观察到就绪;动画帧采样耗时 307.324/357.423 ms,晚 8.1/7.9 ms 观察到就绪。这隔离了有条件发生的观察器延迟,不代表产品延迟,也不能证明 #4119 首次 CI 超限的原因。
完整本地 240 轮运行在修改观察器前得到 Trajectory 样本 157.795/146.737/148.571 ms,修改后为 130.184/191.247/157.543 ms,中位数分别为 148.571 和 157.543 ms。两次运行均通过所有固定终点预算;不声称本地提速或托管环境校准。即使存在可见的无关表格行,就绪回归测试仍拒绝隐藏的搜索框或记录;恢复原选择器等待会使该测试失败。
## 考虑过的替代方案
**用 Node 折叠作为绘制证据。** 拒绝,因为它不执行 DOM 修改、布局或浏览器调度。聚焦重连用例同样不声称 GUI 提速。
@@ -78,6 +86,8 @@ Node 对话折叠很快,并不能证明浏览器能绘制长对话或在流式
**每次流式标记查找都搜索全部已加载历史。** 拒绝,因为 Playwright 把文本与无障碍扫描注入同一个被测 CPU 的渲染进程。将查询限定在输入框和最新 Assistant 中,可保留可见完成检查,同时避免观察器成本随已加载历史增长。
**提高 Trajectory 预算或重试失败样本。** 这会保留已测得的选择器退避,并可能接受更慢的产品行为。基于动画帧的就绪观察移除该观察器延迟,不改变预算。
**只测量流式 CPU。** 拒绝,因为传输停顿和最终结算延迟可能不增加主线程 CPU。独立的输入、首段回复和完整壁钟预算覆盖这些等待。
## 影响
@@ -2,5 +2,5 @@
# side as of the last confirmed-consistent state. Both languages carry equal authority;
# after editing either side, bring the other along and re-record with:
# pnpm run verify-translation-pairing --write benchmarks/long-session-browser/README.md
README.md: d6bd253dedb2c6cdea7fec1019a14bfcc3c4c56f
README.zh.md: fe593b833002ece8efd18d07707c10708cb76e2e
README.md: 05ae2321aa661da3aec6074c4b9dd90bec955390
README.zh.md: 5d5b005bc1be9a9acb48874748ff7eec0ac43772
@@ -12,6 +12,8 @@ The required Chromium workflow in [long-session.bench.ts](long-session.bench.ts)
Three fresh browser processes and scaffold worlds produce raw samples and median verdicts. Open and paging end after the expected transcript state and two animation frames; this includes a rendering opportunity, not a hardware presentation timestamp. Paging reports every page and gates the median of each sample’s slowest page. Stream reports first visible reply, trusted draft typing, complete reply wall time, and Chromium main-thread task duration. Enter submits from the focused composer; draft typing retains that focus without a mouse click. Reply-marker lookups and the input-event text witness read only the latest Assistant step, avoiding repeated whole-history text and accessibility scans. The input witness is installed before submission, and draft typing starts as soon as the first marker is visible, without an extra pre-input animation-frame wait. Reply markers are sampled on animation frames, with visible text required inside the latest step. The first observation captures marker state and focus in the browser; its diagnostics are retrieved after typing so they add no pre-input round trip. Diagnostics also include browser-clock timestamps and focus at the first input event. They do not pause replay; a delayed first observation or input can still fail overlap. The actual first input event must observe an unfinished reply; completion waits for the new rendered turn-tail after Host settlement. After measurement, a trusted keystroke after DONE must fail the same overlap assertion. Open, the slowest older page, and first Trajectory use standard-hosted expectations of 900/700/500 ms. Shared 1.25× headroom gives limits of 1125/875/625 ms respectively; stream endpoint overhead budgets are unchanged. Heap after forced GC and DOM counts are diagnostics, not leak budgets.
Trajectory readiness is sampled on animation frames from the visible searchbox and a visible record inside its own table. The trusted tab click and final two-frame wait remain inside the measured interval; hidden records and unrelated table rows cannot satisfy readiness.
The fixture reserves an empty system head before the first user message, with each user message inside its step. It contains mixed-language prompts, prose, reasoning, 20 code fences, and 40 synthetic tool results. Every historical Assistant includes a compact stream built by the production accumulator from matching reasoning, text, tool arguments, usage, and finish chunks. No model, tool, external network, recorded Session, or private Harness home supplies its content. Streaming uses 120 text deltas at 16 ms replay pacing through the real composer, agent loop, transport, and persistence.
The [decision record](../../.agents/notes/implemented/testing/2026-09-06-frontend-performance-budgets.md) owns calibration, exclusions, and alternatives. The larger [manual diagnostic](../../apps/web/tests/complex-history.perf.ts) remains separate.
@@ -12,6 +12,8 @@
三个全新浏览器进程与 scaffold 环境产生原始样本及中位数判定。打开和分页在预期 transcript(文本记录)状态出现且经过两次动画帧后结束;这包含一次渲染机会,而非硬件显示时间戳。分页报告每一页,并对各样本最慢分页时间的中位数执行预算检查。流式报告首次可见回复、受信任的草稿键入、完整回复壁钟时间和 Chromium 主线程任务时间。Enter 从已聚焦的输入框提交;草稿键入保留该焦点,不执行鼠标点击。回复标记查找与输入事件文本观察器仅读取最新 Assistant 步骤,避免重复进行全量历史文本扫描和无障碍扫描。输入事件文本观察器在提交前安装,首个标记可见后立即开始草稿键入,不额外等待输入前动画帧。回复标记在动画帧上采样,要求文本在最新步骤内可见。首次观察在浏览器内记录标记状态与焦点;诊断在键入后取回,不增加输入前的通信往返。诊断还包含首个输入事件的浏览器时钟时间戳与焦点。诊断不会暂停回放;首次观察或输入延迟仍可能导致重叠失败。实际首个输入事件必须观察到未完成的回复;完整回复的测量会在 Host 结算后等待新 turn-tail 渲染完成。测量后,在 DONE 之后发送的受信任按键必须无法通过同一个重叠断言。打开、最慢的较早页面和首次 Trajectory 使用标准托管预期 900/700/500 ms。共享的 1.25× 余量分别产生 1125/875/625 ms 上限;流式终点的额外开销预算不变。强制 GC 后的 heap 与 DOM 数量仅供诊断,不作为泄漏预算。
Trajectory 就绪状态在动画帧上采样,要求搜索框可见且其自身表格中存在可见记录。真实 tab 点击和最后的两帧等待仍计入测量区间;隐藏记录与无关表格行不能满足就绪条件。
fixture(测试前置数据)在首条用户消息前保留空 system 头节点,每条用户消息都位于其步骤内。它包含混合语言提示词、正文、推理(reasoning)、20 个围栏代码块和 40 个合成工具结果。每条历史 Assistant 都含紧凑流,由生产 accumulator 从匹配的推理、文本、工具参数、usage 和 finish 分片构建。其内容不来自模型、工具、外部网络、录制会话或私有 Harness 主目录。流式回复以 16 ms 回放间隔发送 120 个文本 delta,经过真实输入框、agent loop(智能体循环)、传输与持久化。
[决策记录](../../.agents/notes/implemented/testing/2026-09-06-frontend-performance-budgets.zh.md)拥有校准、排除项与替代方案。更大规模的[手动诊断](../../apps/web/tests/complex-history.perf.ts)保持独立。
@@ -68,6 +68,17 @@ async function waitForReplyMarker(page: Page, marker: string, timeout = 30000) {
}, { marker, first: FIRST, done: DONE }, { polling: 'raf', timeout })
}
async function waitForTrajectory(page: Page, timeout = 30000): Promise<void> {
// Selector retry backoff can delay observation after the view is already ready.
const ready = await page.waitForFunction(() => {
const search = document.querySelector('input[type="search"][aria-label="Search trajectory"]')
if (!search?.checkVisibility({ checkVisibilityCSS: true })) return false
return Array.from(document.querySelectorAll('[data-trajectory-scroll] [data-trajectory-row-key]'))
.some(row => row.checkVisibility({ checkVisibilityCSS: true }))
}, undefined, { polling: 'raf', timeout })
await ready.dispose()
}
async function watchInputOverlap(composer: Locator): Promise<void> {
await composer.evaluate((element, markers) => {
element.removeAttribute('data-benchmark-input-witness')
@@ -127,6 +138,22 @@ it('waits for visible marker text in the latest Assistant step', async () => {
}
})
it('waits for visible Trajectory controls and records rather than unrelated table rows', async () => {
const browser = await chromium.launch({ headless: true })
try {
const page = await browser.newPage()
await page.setContent('<input type="search" aria-label="Search trajectory" style="visibility:hidden"><table><tbody><tr><td>unrelated</td></tr></tbody></table><div data-trajectory-scroll><table><tbody><tr data-trajectory-row-key="record"><td>record</td></tr></tbody></table></div>')
await expect(waitForTrajectory(page, 100)).rejects.toThrow('Timeout')
await page.locator('[data-trajectory-row-key]').evaluate(element => { element.style.display = 'none' })
await page.locator('input').evaluate(element => { element.style.visibility = 'visible' })
await expect(waitForTrajectory(page, 100)).rejects.toThrow('Timeout')
await page.locator('[data-trajectory-row-key]').evaluate(element => { element.style.display = 'table-row' })
await waitForTrajectory(page)
} finally {
await browser.close()
}
})
it('opens, pages, navigates and streams into a 240-turn browser history', async () => {
if (webSnapshotMode() !== 'replay') throw new Error('browser benchmarks require keyless replay mode')
const samples: { open: number; page: number; trajectory: number; first: number; streamTask: number; streamWall: number; input: number; inputOverlapped: boolean; heapMb: number; nodes: number }[] = []
@@ -172,8 +199,7 @@ it('opens, pages, navigates and streams into a 240-turn browser history', async
}
const trajectory = await measure(page, async () => {
await page.getByRole('tab', { name: 'Trajectory', exact: true }).click()
await page.getByRole('searchbox', { name: 'Search trajectory', exact: true }).waitFor()
await page.getByRole('row').last().waitFor()
await waitForTrajectory(page)
})
await page.getByRole('tab', { name: 'Chat', exact: true }).click()
await page.waitForFunction(({ selector, expected }) => document.querySelectorAll(selector).length === expected, { selector: TAIL, expected: HISTORY_TURNS })