diff --git a/.agents/notes/implemented/testing/2026-09-08-ci-readiness-and-completion.i18n.yaml b/.agents/notes/implemented/testing/2026-09-08-ci-readiness-and-completion.i18n.yaml index dbbc4ea266..f96c72c081 100644 --- a/.agents/notes/implemented/testing/2026-09-08-ci-readiness-and-completion.i18n.yaml +++ b/.agents/notes/implemented/testing/2026-09-08-ci-readiness-and-completion.i18n.yaml @@ -2,5 +2,5 @@ # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: # pnpm run verify-translation-pairing --write .agents/notes/implemented/testing/2026-09-08-ci-readiness-and-completion.md -2026-09-08-ci-readiness-and-completion.md: 2249df5467189975aca2d73ec56c6cf82ec7b62f -2026-09-08-ci-readiness-and-completion.zh.md: ccf8e7c16903b66e39bc48e99460e5c5179ab51b +2026-09-08-ci-readiness-and-completion.md: c72048ef90b987e7d27992b1754736b2ce895093 +2026-09-08-ci-readiness-and-completion.zh.md: 1188d66249b3092416435da2e8496a657ad00677 diff --git a/.agents/notes/implemented/testing/2026-09-08-ci-readiness-and-completion.md b/.agents/notes/implemented/testing/2026-09-08-ci-readiness-and-completion.md index 2249df5467..c72048ef90 100644 --- a/.agents/notes/implemented/testing/2026-09-08-ci-readiness-and-completion.md +++ b/.agents/notes/implemented/testing/2026-09-08-ci-readiness-and-completion.md @@ -14,6 +14,8 @@ The [ACP coverage run](https://github.com/deepseek-harness/deepseek-harness/acti A [worker-runtime coverage failure](https://github.com/deepseek-harness/deepseek-harness/actions/runs/34248221544/job/102135631932) exhausts the slow-binding fixture's one-second compute allowance. Concurrent native Windows reproductions exceed that allowance before calling the binding. Worker initialization contributes measured active time; the delayed binding contributes idle time. +The [Windows coverage run](https://github.com/deepseek-harness/deepseek-harness/actions/runs/34324325375/job/102377982193) reports an SDK subprocess exit beyond a fixture's 200 ms confirmation window and an Inspector Worker startup beyond its ten-second default. Neither the protocol-error routing case nor the Cordis tree projection case measures those latency guarantees. + ## Decision The [webhook browser test](../../../../apps/web/tests/github-ready-review.e2e.ts) observes the model request caused by delivery before checking Session registration. The [feedback test](../../../../apps/web/tests/feedback-command.e2e.ts) waits for the empty composer and enabled attachment control before comparing ARIA output. Matching consecutive snapshots cannot prove that the command RPC has settled: its event stream can publish the acknowledgement first. @@ -28,6 +30,8 @@ The [subagent teardown decision](2026-09-07-subagent-teardown-test-budgets.md) o The [worker-runtime binding test](../../../../packages/code-runtime/code-runtime-worker-thread/tests/runtime.spec.ts) allows five seconds of compute for source-worker initialization and delays the binding for 6.5 seconds. Charging that idle delay would still exceed the entire compute allowance. The case retains its 15-second test limit and 30-second wall ceiling, registers Context and reply-timer cleanup, and leaves the hot-loop, decoy-dispatch, wall-ceiling, and abort controls at their existing limits. Production budgets remain unchanged. +The [SDK subagent protocol-error test](../../../../packages/subagent/subagent-dsh-sdk/tests/subagent-dsh-sdk.spec.ts) uses the provider's normal shutdown and exit grace periods and registers disposal before its assertions. The [Inspector tree tests](../../../../packages/experimental/inspector/tests/cordis-tree.host.spec.ts) pass the active test budget to Worker startup and register cleanup while startup is still pending. A cancelled test cannot receive a late-ready handle; cleanup awaits initialization and closes a successfully started Worker. Failed initialization already terminates the Worker before rejecting. A controlled late-start test verifies cancellation and closure through the real Worker's HTTP endpoint. Production defaults remain unchanged. + ## Alternatives considered **Larger independent waits.** Rejected where a completion promise already exists. A separate polling deadline continues to compete with the execution lane's budget. diff --git a/.agents/notes/implemented/testing/2026-09-08-ci-readiness-and-completion.zh.md b/.agents/notes/implemented/testing/2026-09-08-ci-readiness-and-completion.zh.md index ccf8e7c169..1188d66249 100644 --- a/.agents/notes/implemented/testing/2026-09-08-ci-readiness-and-completion.zh.md +++ b/.agents/notes/implemented/testing/2026-09-08-ci-readiness-and-completion.zh.md @@ -14,6 +14,8 @@ Status: implemented 一次 [worker runtime coverage 失败](https://github.com/deepseek-harness/deepseek-harness/actions/runs/34248221544/job/102135631932)耗尽了慢 binding 夹具的一秒计算额度。原生 Windows 并发复现在调用 binding 前已超过该额度。Worker 初始化会累计所测的活跃时间；延迟的 binding 累计空闲时间。 +[Windows 覆盖率运行](https://github.com/deepseek-harness/deepseek-harness/actions/runs/34324325375/job/102377982193)报告了 SDK 子进程退出超过测试设置的 200 毫秒确认期限，以及 Inspector Worker 启动超过十秒默认期限。协议错误转发用例和 Cordis 树投影用例都不衡量这些延迟保证。 + ## 决策 [Webhook 浏览器测试](../../../../apps/web/tests/github-ready-review.e2e.ts)观察投递触发的模型请求后再检查 Session 注册。[反馈测试](../../../../apps/web/tests/feedback-command.e2e.ts)在比较 ARIA 输出前等待输入框清空且附件按钮启用。连续两次快照相同不能证明命令 RPC 已完成：事件流可能先发布确认消息。 @@ -28,6 +30,8 @@ Status: implemented [Worker runtime binding 测试](../../../../packages/code-runtime/code-runtime-worker-thread/tests/runtime.spec.ts)为源码 worker 初始化保留五秒计算额度，并将 binding 延迟设为 6.5 秒。若将该空闲延迟计费，仍会超过整个计算额度。用例保留 15 秒测试期限与 30 秒墙钟上限，登记 Context 和回复定时器的清理，并保持热循环、诱饵 dispatch、墙钟上限及取消控制用例的原有限制。生产预算不变。 +[SDK 子 Agent 协议错误测试](../../../../packages/subagent/subagent-dsh-sdk/tests/subagent-dsh-sdk.spec.ts)使用提供方正常的关闭和退出等待时间，并在断言前登记清理。[Inspector 树测试](../../../../packages/experimental/inspector/tests/cordis-tree.host.spec.ts)将当前测试预算传给 Worker 启动，并在启动尚未完成时登记清理。取消后的测试不会收到随后才就绪的实例；清理等待初始化完成，并关闭成功启动的 Worker。初始化失败时，启动操作会在拒绝前终止 Worker。受控的延迟启动测试通过真实 Worker 的 HTTP 端点验证取消和关闭。生产默认值不变。 + ## 考虑过的替代方案 **增大独立等待时限。** 已有完成 Promise 时不采用。独立轮询期限仍会与执行通道的预算竞争。 diff --git a/packages/experimental/inspector/tests/cordis-tree.host.spec.ts b/packages/experimental/inspector/tests/cordis-tree.host.spec.ts index 46bbad99be..9cabf89f48 100644 --- a/packages/experimental/inspector/tests/cordis-tree.host.spec.ts +++ b/packages/experimental/inspector/tests/cordis-tree.host.spec.ts @@ -2,10 +2,11 @@ import { Context } from '@deepseek-ai/cordis' import WebSocket, { type RawData } from 'ws' -import { afterEach, describe, expect, it, vi } from 'vitest' +import { afterEach, describe, expect, it, vi, type TestContext } from 'vitest' import { CordisTreeCollector } from '../src/shared/cordis/collector.ts' import { observeCordisTree } from '../src/shared/cordis/observer.ts' -import { startInspector, type InspectorHandle } from '../src/host/bridge/controller.ts' +import * as inspectorBridge from '../src/host/bridge/controller.ts' +import type { InspectorHandle, InspectorOptions } from '../src/host/bridge/controller.ts' import { publishCordisTree as publishHostCordisTree } from '../src/host/inspection/cordis.ts' import { parseCordisTreeSnapshot, type CordisTreeNode } from '../src/shared/cordis/snapshot.ts' import { inspectorId } from '../src/shared/bridge/ids.ts' @@ -76,6 +77,19 @@ class CdpClient { } } +/** Own pending startup through cancellation; tree assertions do not measure Worker cold-start latency. */ +async function startTestInspector(options: InspectorOptions, test: TestContext): Promise { + const pending = inspectorBridge.startInspector({ ...options, startupTimeoutMs: test.task.timeout }) + test.onTestFinished(async () => { + // Failed starts terminate their Worker before rejecting. + const [started] = await Promise.allSettled([pending]) + if (started.status === 'fulfilled') await started.value.close() + }) + const handle = await pending + test.signal.throwIfAborted() + return handle +} + describe('Cordis tree inspection', () => { let inspector: InspectorHandle | undefined let cdp: CdpClient | undefined @@ -84,6 +98,48 @@ describe('Cordis tree inspection', () => { const observers: Array<() => void> = [] const fibers: Array<{ dispose(): Promise }> = [] + it('closes a late-starting Worker after test cancellation without returning its handle', async (test) => { + const release = Promise.withResolvers() + const controller = new AbortController() + const cleanups: Array[0]> = [] + let starting: Promise | undefined + const start = inspectorBridge.startInspector + const spy = vi.spyOn(inspectorBridge, 'startInspector').mockImplementation(async (options) => { + starting = start(options) + const handle = await starting + await release.promise + return handle + }) + test.onTestFinished(async () => { + release.resolve(undefined) + spy.mockRestore() + if (starting === undefined) return + const [started] = await Promise.allSettled([starting]) + if (started.status === 'fulfilled') await started.value.close() + }) + const pending = startTestInspector({ port: 0, captureFetch: false }, { + ...test, + signal: controller.signal, + onTestFinished: (cleanup) => { cleanups.push(cleanup) }, + }) + const result = pending.then( + handle => ({ status: 'fulfilled' as const, handle }), + (reason: unknown) => ({ status: 'rejected' as const, reason }), + ) + if (starting === undefined) throw new Error('Inspector startup was not called') + const handle = await starting + expect(spy).toHaveBeenCalledWith(expect.objectContaining({ startupTimeoutMs: test.task.timeout })) + expect(cleanups).toHaveLength(1) + const cancelled = new Error('test cancelled during startup') + controller.abort(cancelled) + release.resolve(undefined) + const outcome = await result + expect(outcome.status).toBe('rejected') + expect('reason' in outcome ? outcome.reason : undefined).toBe(cancelled) + await cleanups[0]!(test) + await expect(fetch(handle.endpoint.httpUrl)).rejects.toThrow() + }) + afterEach(async () => { for (const dispose of observers.splice(0).reverse()) dispose() for (const fiber of fibers.splice(0).reverse()) await fiber.dispose() @@ -362,8 +418,8 @@ describe('Cordis tree inspection', () => { backend.close() }) - it('projects Host and Client trees and resolves both node kinds to RemoteObjects', async () => { - inspector = await startInspector({ port: 0, captureFetch: false, maxCordisNodes: 100 }) + it('projects Host and Client trees and resolves both node kinds to RemoteObjects', async (test) => { + inspector = await startTestInspector({ port: 0, captureFetch: false, maxCordisNodes: 100 }, test) const host = new Context() const hostFiber = host.plugin({ name: 'host-child', apply() {} }) fibers.push(hostFiber) @@ -540,8 +596,8 @@ describe('Cordis tree inspection', () => { expect(disconnectedTree.clients[0]?.connection.state).toBe('disconnected') }) - it('emits only node-level DOM changes for Client snapshots', async () => { - inspector = await startInspector({ port: 0, captureFetch: false, maxCordisNodes: 100 }) + it('emits only node-level DOM changes for Client snapshots', async (test) => { + inspector = await startTestInspector({ port: 0, captureFetch: false, maxCordisNodes: 100 }, test) cdp = await CdpClient.connect(inspector.endpoint.webSocketDebuggerUrl) const initialDocument = (await cdp.call('DOM.getDocument')).result?.root as CdpNode const clientsNode = initialDocument.children?.find(node => node.localName === 'clients') @@ -598,8 +654,8 @@ describe('Cordis tree inspection', () => { }) }) - it('serves three document levels by default and withheld levels on demand', async () => { - inspector = await startInspector({ port: 0, captureFetch: false, maxCordisNodes: 100 }) + it('serves three document levels by default and withheld levels on demand', async (test) => { + inspector = await startTestInspector({ port: 0, captureFetch: false, maxCordisNodes: 100 }, test) const host = new Context() let innerFiber: { uid: number | null } | undefined const outer = host.plugin({ @@ -680,14 +736,14 @@ describe('Cordis tree inspection', () => { expect(secondCdp.events.slice(offset).some(event => event.method === 'DOM.setChildNodes')).toBe(false) }) - it('restores a disconnected Client tree from a new transport generation', async () => { - inspector = await startInspector({ + it('restores a disconnected Client tree from a new transport generation', async (test) => { + inspector = await startTestInspector({ port: 0, captureFetch: false, maxCordisNodes: 100, clientReconnectBaseMs: 10, clientReconnectMaxMs: 20, - }) + }, test) clientSource = await InspectorClientFixture.start(inspector.endpoint.client, { label: 'Reconnect Client' }) cdp = await CdpClient.connect(inspector.endpoint.webSocketDebuggerUrl) await cdp.call('Runtime.enable') diff --git a/packages/subagent/subagent-dsh-sdk/tests/subagent-dsh-sdk.spec.ts b/packages/subagent/subagent-dsh-sdk/tests/subagent-dsh-sdk.spec.ts index b9fa8c55c0..ba26c0c69c 100644 --- a/packages/subagent/subagent-dsh-sdk/tests/subagent-dsh-sdk.spec.ts +++ b/packages/subagent/subagent-dsh-sdk/tests/subagent-dsh-sdk.spec.ts @@ -6,7 +6,7 @@ * quiescent disposal are all exercised end to end. No model, no key. */ -import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' +import { afterEach, beforeEach, describe, expect, it, onTestFinished, vi } from 'vitest' import { Context } from '@deepseek-ai/cordis' import { existsSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs' import { tmpdir } from 'node:os' @@ -751,19 +751,17 @@ describe('dsh-subagent-dsh-sdk provider', () => { cwd: process.cwd(), provider: 'p', model: 'm', - // The fake dies as soon as the prompt arrives: FAKE_HANG_PROMPT plus a - // short-lived process is simulated by killing via dispose below instead; - // here use FAKE_MALFORMED to make the prompt reply violate the protocol. env: { FAKE_MALFORMED_PROMPT: '1' }, - shutdownTimeoutMs: 100, - disposeEofGraceMs: 200, - disposeGraceMs: 200, + shutdownTimeoutMs: DEFAULT_SHUTDOWN_TIMEOUT_MS, + disposeEofGraceMs: DEFAULT_DISPOSE_EOF_GRACE_MS, + disposeGraceMs: DEFAULT_DISPOSE_GRACE_MS, onError: (error) => { seen.push(error.message) throw new Error('sink failure must be contained') }, } const run = await startSdkRun(request(), spec) + onTestFinished(() => run.dispose()) const result = await run.result expect(result.stopReason).toBe('error') expect(result.diagnostic).toBe(