test: respect startup and teardown budgets in Windows fixtures

This commit is contained in:
creatixchu committed 2026-09-09 16:08:19 +08:00
1 parent a8faac2850
commit a43fa81f82
5 files changed
+82 -20

No files matched your search

@@ -2,5 +2,5 @@
# side as of the last confirmed-consistent state. Both languages carry equal authority;
# after editing either side, bring the other along and re-record with:
# pnpm run verify-translation-pairing --write .agents/notes/implemented/testing/2026-09-08-ci-readiness-and-completion.md
2026-09-08-ci-readiness-and-completion.md: 2249df5467189975aca2d73ec56c6cf82ec7b62f
2026-09-08-ci-readiness-and-completion.zh.md: ccf8e7c16903b66e39bc48e99460e5c5179ab51b
2026-09-08-ci-readiness-and-completion.md: c72048ef90b987e7d27992b1754736b2ce895093
2026-09-08-ci-readiness-and-completion.zh.md: 1188d66249b3092416435da2e8496a657ad00677
@@ -14,6 +14,8 @@ The [ACP coverage run](https://github.com/deepseek-harness/deepseek-harness/acti
A [worker-runtime coverage failure](https://github.com/deepseek-harness/deepseek-harness/actions/runs/34248221544/job/102135631932) exhausts the slow-binding fixture's one-second compute allowance. Concurrent native Windows reproductions exceed that allowance before calling the binding. Worker initialization contributes measured active time; the delayed binding contributes idle time.
The [Windows coverage run](https://github.com/deepseek-harness/deepseek-harness/actions/runs/34324325375/job/102377982193) reports an SDK subprocess exit beyond a fixture's 200 ms confirmation window and an Inspector Worker startup beyond its ten-second default. Neither the protocol-error routing case nor the Cordis tree projection case measures those latency guarantees.
## Decision
The [webhook browser test](../../../../apps/web/tests/github-ready-review.e2e.ts) observes the model request caused by delivery before checking Session registration. The [feedback test](../../../../apps/web/tests/feedback-command.e2e.ts) waits for the empty composer and enabled attachment control before comparing ARIA output. Matching consecutive snapshots cannot prove that the command RPC has settled: its event stream can publish the acknowledgement first.
@@ -28,6 +30,8 @@ The [subagent teardown decision](2026-09-07-subagent-teardown-test-budgets.md) o
The [worker-runtime binding test](../../../../packages/code-runtime/code-runtime-worker-thread/tests/runtime.spec.ts) allows five seconds of compute for source-worker initialization and delays the binding for 6.5 seconds. Charging that idle delay would still exceed the entire compute allowance. The case retains its 15-second test limit and 30-second wall ceiling, registers Context and reply-timer cleanup, and leaves the hot-loop, decoy-dispatch, wall-ceiling, and abort controls at their existing limits. Production budgets remain unchanged.
The [SDK subagent protocol-error test](../../../../packages/subagent/subagent-dsh-sdk/tests/subagent-dsh-sdk.spec.ts) uses the provider's normal shutdown and exit grace periods and registers disposal before its assertions. The [Inspector tree tests](../../../../packages/experimental/inspector/tests/cordis-tree.host.spec.ts) pass the active test budget to Worker startup and register cleanup while startup is still pending. A cancelled test cannot receive a late-ready handle; cleanup awaits initialization and closes a successfully started Worker. Failed initialization already terminates the Worker before rejecting. A controlled late-start test verifies cancellation and closure through the real Worker's HTTP endpoint. Production defaults remain unchanged.
## Alternatives considered
**Larger independent waits.** Rejected where a completion promise already exists. A separate polling deadline continues to compete with the execution lane's budget.
@@ -14,6 +14,8 @@ Status: implemented
一次 [worker runtime coverage 失败](https://github.com/deepseek-harness/deepseek-harness/actions/runs/34248221544/job/102135631932)耗尽了慢 binding 夹具的一秒计算额度。原生 Windows 并发复现在调用 binding 前已超过该额度。Worker 初始化会累计所测的活跃时间;延迟的 binding 累计空闲时间。
[Windows 覆盖率运行](https://github.com/deepseek-harness/deepseek-harness/actions/runs/34324325375/job/102377982193)报告了 SDK 子进程退出超过测试设置的 200 毫秒确认期限,以及 Inspector Worker 启动超过十秒默认期限。协议错误转发用例和 Cordis 树投影用例都不衡量这些延迟保证。
## 决策
[Webhook 浏览器测试](../../../../apps/web/tests/github-ready-review.e2e.ts)观察投递触发的模型请求后再检查 Session 注册。[反馈测试](../../../../apps/web/tests/feedback-command.e2e.ts)在比较 ARIA 输出前等待输入框清空且附件按钮启用。连续两次快照相同不能证明命令 RPC 已完成:事件流可能先发布确认消息。
@@ -28,6 +30,8 @@ Status: implemented
[Worker runtime binding 测试](../../../../packages/code-runtime/code-runtime-worker-thread/tests/runtime.spec.ts)为源码 worker 初始化保留五秒计算额度,并将 binding 延迟设为 6.5 秒。若将该空闲延迟计费,仍会超过整个计算额度。用例保留 15 秒测试期限与 30 秒墙钟上限,登记 Context 和回复定时器的清理,并保持热循环、诱饵 dispatch、墙钟上限及取消控制用例的原有限制。生产预算不变。
[SDK 子 Agent 协议错误测试](../../../../packages/subagent/subagent-dsh-sdk/tests/subagent-dsh-sdk.spec.ts)使用提供方正常的关闭和退出等待时间,并在断言前登记清理。[Inspector 树测试](../../../../packages/experimental/inspector/tests/cordis-tree.host.spec.ts)将当前测试预算传给 Worker 启动,并在启动尚未完成时登记清理。取消后的测试不会收到随后才就绪的实例;清理等待初始化完成,并关闭成功启动的 Worker。初始化失败时,启动操作会在拒绝前终止 Worker。受控的延迟启动测试通过真实 Worker 的 HTTP 端点验证取消和关闭。生产默认值不变。
## 考虑过的替代方案
**增大独立等待时限。** 已有完成 Promise 时不采用。独立轮询期限仍会与执行通道的预算竞争。
@@ -2,10 +2,11 @@
import { Context } from '@deepseek-ai/cordis'
import WebSocket, { type RawData } from 'ws'
import { afterEach, describe, expect, it, vi } from 'vitest'
import { afterEach, describe, expect, it, vi, type TestContext } from 'vitest'
import { CordisTreeCollector } from '../src/shared/cordis/collector.ts'
import { observeCordisTree } from '../src/shared/cordis/observer.ts'
import { startInspector, type InspectorHandle } from '../src/host/bridge/controller.ts'
import * as inspectorBridge from '../src/host/bridge/controller.ts'
import type { InspectorHandle, InspectorOptions } from '../src/host/bridge/controller.ts'
import { publishCordisTree as publishHostCordisTree } from '../src/host/inspection/cordis.ts'
import { parseCordisTreeSnapshot, type CordisTreeNode } from '../src/shared/cordis/snapshot.ts'
import { inspectorId } from '../src/shared/bridge/ids.ts'
@@ -76,6 +77,19 @@ class CdpClient {
}
}
/** Own pending startup through cancellation; tree assertions do not measure Worker cold-start latency. */
async function startTestInspector(options: InspectorOptions, test: TestContext): Promise<InspectorHandle> {
const pending = inspectorBridge.startInspector({ ...options, startupTimeoutMs: test.task.timeout })
test.onTestFinished(async () => {
// Failed starts terminate their Worker before rejecting.
const [started] = await Promise.allSettled([pending])
if (started.status === 'fulfilled') await started.value.close()
})
const handle = await pending
test.signal.throwIfAborted()
return handle
}
describe('Cordis tree inspection', () => {
let inspector: InspectorHandle | undefined
let cdp: CdpClient | undefined
@@ -84,6 +98,48 @@ describe('Cordis tree inspection', () => {
const observers: Array<() => void> = []
const fibers: Array<{ dispose(): Promise<void> }> = []
it('closes a late-starting Worker after test cancellation without returning its handle', async (test) => {
const release = Promise.withResolvers<undefined>()
const controller = new AbortController()
const cleanups: Array<Parameters<TestContext['onTestFinished']>[0]> = []
let starting: Promise<InspectorHandle> | undefined
const start = inspectorBridge.startInspector
const spy = vi.spyOn(inspectorBridge, 'startInspector').mockImplementation(async (options) => {
starting = start(options)
const handle = await starting
await release.promise
return handle
})
test.onTestFinished(async () => {
release.resolve(undefined)
spy.mockRestore()
if (starting === undefined) return
const [started] = await Promise.allSettled([starting])
if (started.status === 'fulfilled') await started.value.close()
})
const pending = startTestInspector({ port: 0, captureFetch: false }, {
...test,
signal: controller.signal,
onTestFinished: (cleanup) => { cleanups.push(cleanup) },
})
const result = pending.then(
handle => ({ status: 'fulfilled' as const, handle }),
(reason: unknown) => ({ status: 'rejected' as const, reason }),
)
if (starting === undefined) throw new Error('Inspector startup was not called')
const handle = await starting
expect(spy).toHaveBeenCalledWith(expect.objectContaining({ startupTimeoutMs: test.task.timeout }))
expect(cleanups).toHaveLength(1)
const cancelled = new Error('test cancelled during startup')
controller.abort(cancelled)
release.resolve(undefined)
const outcome = await result
expect(outcome.status).toBe('rejected')
expect('reason' in outcome ? outcome.reason : undefined).toBe(cancelled)
await cleanups[0]!(test)
await expect(fetch(handle.endpoint.httpUrl)).rejects.toThrow()
})
afterEach(async () => {
for (const dispose of observers.splice(0).reverse()) dispose()
for (const fiber of fibers.splice(0).reverse()) await fiber.dispose()
@@ -362,8 +418,8 @@ describe('Cordis tree inspection', () => {
backend.close()
})
it('projects Host and Client trees and resolves both node kinds to RemoteObjects', async () => {
inspector = await startInspector({ port: 0, captureFetch: false, maxCordisNodes: 100 })
it('projects Host and Client trees and resolves both node kinds to RemoteObjects', async (test) => {
inspector = await startTestInspector({ port: 0, captureFetch: false, maxCordisNodes: 100 }, test)
const host = new Context()
const hostFiber = host.plugin({ name: 'host-child', apply() {} })
fibers.push(hostFiber)
@@ -540,8 +596,8 @@ describe('Cordis tree inspection', () => {
expect(disconnectedTree.clients[0]?.connection.state).toBe('disconnected')
})
it('emits only node-level DOM changes for Client snapshots', async () => {
inspector = await startInspector({ port: 0, captureFetch: false, maxCordisNodes: 100 })
it('emits only node-level DOM changes for Client snapshots', async (test) => {
inspector = await startTestInspector({ port: 0, captureFetch: false, maxCordisNodes: 100 }, test)
cdp = await CdpClient.connect(inspector.endpoint.webSocketDebuggerUrl)
const initialDocument = (await cdp.call('DOM.getDocument')).result?.root as CdpNode
const clientsNode = initialDocument.children?.find(node => node.localName === 'clients')
@@ -598,8 +654,8 @@ describe('Cordis tree inspection', () => {
})
})
it('serves three document levels by default and withheld levels on demand', async () => {
inspector = await startInspector({ port: 0, captureFetch: false, maxCordisNodes: 100 })
it('serves three document levels by default and withheld levels on demand', async (test) => {
inspector = await startTestInspector({ port: 0, captureFetch: false, maxCordisNodes: 100 }, test)
const host = new Context()
let innerFiber: { uid: number | null } | undefined
const outer = host.plugin({
@@ -680,14 +736,14 @@ describe('Cordis tree inspection', () => {
expect(secondCdp.events.slice(offset).some(event => event.method === 'DOM.setChildNodes')).toBe(false)
})
it('restores a disconnected Client tree from a new transport generation', async () => {
inspector = await startInspector({
it('restores a disconnected Client tree from a new transport generation', async (test) => {
inspector = await startTestInspector({
port: 0,
captureFetch: false,
maxCordisNodes: 100,
clientReconnectBaseMs: 10,
clientReconnectMaxMs: 20,
})
}, test)
clientSource = await InspectorClientFixture.start(inspector.endpoint.client, { label: 'Reconnect Client' })
cdp = await CdpClient.connect(inspector.endpoint.webSocketDebuggerUrl)
await cdp.call('Runtime.enable')
@@ -6,7 +6,7 @@
* quiescent disposal are all exercised end to end. No model, no key.
*/
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
import { afterEach, beforeEach, describe, expect, it, onTestFinished, vi } from 'vitest'
import { Context } from '@deepseek-ai/cordis'
import { existsSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs'
import { tmpdir } from 'node:os'
@@ -751,19 +751,17 @@ describe('dsh-subagent-dsh-sdk provider', () => {
cwd: process.cwd(),
provider: 'p',
model: 'm',
// The fake dies as soon as the prompt arrives: FAKE_HANG_PROMPT plus a
// short-lived process is simulated by killing via dispose below instead;
// here use FAKE_MALFORMED to make the prompt reply violate the protocol.
env: { FAKE_MALFORMED_PROMPT: '1' },
shutdownTimeoutMs: 100,
disposeEofGraceMs: 200,
disposeGraceMs: 200,
shutdownTimeoutMs: DEFAULT_SHUTDOWN_TIMEOUT_MS,
disposeEofGraceMs: DEFAULT_DISPOSE_EOF_GRACE_MS,
disposeGraceMs: DEFAULT_DISPOSE_GRACE_MS,
onError: (error) => {
seen.push(error.message)
throw new Error('sink failure must be contained')
},
}
const run = await startSdkRun(request(), spec)
onTestFinished(() => run.dispose())
const result = await run.result
expect(result.stopReason).toBe('error')
expect(result.diagnostic).toBe(