diff --git a/packages/session/session-persistence-jsonl/README.i18n.yaml b/packages/session/session-persistence-jsonl/README.i18n.yaml index efd3f41872..7c5cf29e6b 100644 --- a/packages/session/session-persistence-jsonl/README.i18n.yaml +++ b/packages/session/session-persistence-jsonl/README.i18n.yaml @@ -30,14 +30,14 @@ en: 1eaa87736cac029b zh: 26224e8889786d69 /deepseek-ai-dsh-session-persistence-jsonl/use-this-package/reading-the-logs: - en: c06983c4a73bebe1 - zh: 7319f8f44223685f + en: 7b15500e627a0fee + zh: 9a1ff27749bbcfc2 /deepseek-ai-dsh-session-persistence-jsonl/understand-the-implementation: en: a35407bb25671e7e zh: 6a00f9510c85adcb /deepseek-ai-dsh-session-persistence-jsonl/understand-the-implementation/design-concept: - en: b69d4565a11b6f02 - zh: c8309690fc931f8d + en: a503a68cc65d378a + zh: a84500e276fe6073 /deepseek-ai-dsh-session-persistence-jsonl/understand-the-implementation/physical-encoding: en: 011d56563b81acd9 zh: 69122dbade603fe9 diff --git a/packages/session/session-persistence-jsonl/README.md b/packages/session/session-persistence-jsonl/README.md index 5846404961..f9dfaf091b 100644 --- a/packages/session/session-persistence-jsonl/README.md +++ b/packages/session/session-persistence-jsonl/README.md @@ -79,7 +79,7 @@ The current-generation scanner applies the current codec owner’s structural ad ### Reading the logs -`open(id, 'read'|'write')` selects the highest canonical generation. Current input follows the ordinary fast path. For historical input, a read open decodes and migrates the source once, validates the current logical result, and returns it without publishing a successor. A write open reuses that revision-keyed preparation when available, or performs the same preparation, then encodes a same-directory temporary file in bounded chunks, verifies it in a Worker Thread, rechecks the source revision, and publishes the current successor without overwrite before returning. The source remains byte-identical. Source drift after preparation rejects that write open without replacing the logical history already returned to readers; a later write open prepares the new revision. The backend marks decoded event graphs `shared-frozen` when it freezes them before memoization; every nested object and array is frozen, and handle reads and slices preserve that state, including empty slices. Only an unmaterialized pending log reports `detached`. `stat(id)` and `list()` select and translate only the highest generation header without reading event rows or starting migration; snapshots carry the selected file’s `sizeBytes` and a best-effort revision. Current revisions identify that file; historical revisions also fingerprint the selected files across the persistence root, so child changes invalidate cached logical events. Fingerprinting reads filesystem metadata only; unrelated changes conservatively invalidate historical revisions. One `list()` call shares a corpus fingerprint across its historical entries. With `compression: 'none'`, the log is newline-delimited text an external reader can consume directly; the compressed default must be read through the backend. +`open(id, 'read'|'write')` selects the highest canonical generation. Current input follows the ordinary fast path. For historical input, a read open decodes and migrates the source once, validates the current logical result, and returns it without publishing a successor. A write open reuses that revision-keyed preparation when available, or performs the same preparation, then encodes a same-directory temporary file in bounded chunks, runs isolated verification, rechecks the source revision, and publishes the current successor without overwrite before returning. The source remains byte-identical. Source drift after preparation rejects that write open without replacing the logical history already returned to readers; a later write open prepares the new revision. The backend marks decoded event graphs `shared-frozen` when it freezes them before memoization; every nested object and array is frozen, and handle reads and slices preserve that state, including empty slices. Only an unmaterialized pending log reports `detached`. `stat(id)` and `list()` select and translate only the highest generation header without reading event rows or starting migration; snapshots carry the selected file’s `sizeBytes` and a best-effort revision. Current revisions identify that file; historical revisions also fingerprint the selected files across the persistence root, so child changes invalidate cached logical events. Fingerprinting reads filesystem metadata only; unrelated changes conservatively invalidate historical revisions. One `list()` call shares a corpus fingerprint across its historical entries. With `compression: 'none'`, the log is newline-delimited text an external reader can consume directly; the compressed default must be read through the backend. Historical body preparation completes the parent catalog through [V3→V4](../session-format-v3-to-v4/README.md). It finds candidate direct children from headers, reads each child's own descriptor through historical codecs, and retains compact evidence plus source revisions. It neither prepares child catalogs nor publishes child successors. Unreadable or unsupported headers, including corrupt Zstandard header frames, are omitted from discovery and `list()`. Direct access to a corrupt compressed header still rejects; header I/O and cancellation errors propagate. Child decoding and descriptor-field failures produce warnings naming the child path and retain header identity through `subagent/catalog` when the parent has no complete entry; healthy children and existing parent catalog entries remain available. Opening a damaged child still reports that child's error. Missing, unsupported, or multiple descriptors likewise produce unknown-mode membership without inventing a label. Published unknown entries remain browsable; child history reads retry the actual log and resolve its mode from a valid descriptor. Preparation rechecks membership and inspected source revisions before returning, reuse, and publication, including failed child reads so a repaired child invalidates stale preparation. Source drift retries a read open once and refuses a write open. Cancellation still aborts the operation. Current V4 opens bypass discovery and validate catalog fields, uniqueness, and current delivery ownership before exposing events. @@ -97,6 +97,8 @@ This section explains the physical encoding and write path; the observable contr ### Design concept +Published verifiers run the bundled JavaScript in fresh Worker Threads. Source execution uses isolated Node processes with the ESM TypeScript loader, avoiding loader registration inside short-lived threads. Both paths retain the same verification, cancellation, and two-isolate concurrency limit; teardown waits for the owned isolate to stop. + The backend owns its complete storage runtime (`src/storage.ts`): `JsonlSessionHandle` carries the per-handle mutation chain, the routed live-event buffer with its fixed batching window and single-flight drain, monotonic reads, and idempotent close; a tracker holds the in-process single-writer claims, the open-handle set teardown sweeps, and the created-but-unmaterialized pending sessions the backend's own session listeners route into. Historical body reads share one per-session Decode/Migrate preparation, and a bounded revision-keyed memo lets an immediate observe-to-resume handoff reuse that parse; the backend deep-freezes each event graph once before memoization, so later handle reads reuse it without copying or freezing. Only a write open publishes the prepared successor. The package deliberately exposes only its default plugin export plus configuration types — the concrete class is not a named export, so consumers couple to `ctx.sessionPersistence`, and the shared seam suites (`runPersistenceContract`/`runLiveWritePathContract`) pin its observable behavior. Physical revisions combine device, inode, size, and nanosecond timestamps for the preparation memo, stable-read retries, and publication checks. Historical public revisions add a SHA-256 fingerprint of sorted selected paths and their physical revisions; lock files and retained unselected generations do not contribute. ### Physical encoding diff --git a/packages/session/session-persistence-jsonl/README.zh.md b/packages/session/session-persistence-jsonl/README.zh.md index b7dd646e57..4a79ca35b0 100644 --- a/packages/session/session-persistence-jsonl/README.zh.md +++ b/packages/session/session-persistence-jsonl/README.zh.md @@ -79,7 +79,7 @@ kind: "package-reference" ### 读取日志 -`open(id, 'read'|'write')` 选择最高规范 generation。当前格式输入走普通快速路径。对于历史输入,只读 open 会单遍解码并迁移源、校验当前逻辑结果,然后在不发布后继的情况下返回。写 open 会在可用时复用按 revision 为键的 preparation,否则执行同一套 preparation,再按有界分片编码同目录临时文件、在 Worker Thread 中校验、复查源修订,并在返回前以不覆盖方式发布当前后继。源保持逐字节不变。如果源在 preparation 后发生变化,该次写 open 会失败,已经返回给读方的逻辑历史不会被替换;后续写 open 会针对新的 revision 重新执行 preparation。后端在 memo 化前冻结已解码的 event graph,并在此时将其标记为 `shared-frozen`;每个嵌套对象和数组都会冻结,句柄读取和 slice 即使为空也保留该状态。只有尚未实体化的 pending 空日志报告 `detached`。`stat(id)` 与 `list()` 只选择并转换最高 generation 的 header,不读取事件行,也不启动迁移;快照携带所选文件的 `sizeBytes` 与尽力而为的修订号。当前格式修订号标识该文件;历史格式修订号还包含持久化根目录中所选文件的指纹,因此子日志变化会使缓存的逻辑事件失效。指纹只读取文件系统元数据;无关变化也会保守地使历史修订号失效。一次 `list()` 为其历史条目共用一个全库指纹。选择 `compression: 'none'` 后,日志是外部读取方可直接消费的换行分隔文本;压缩默认值必须经后端读取。 +`open(id, 'read'|'write')` 选择最高规范 generation。当前格式输入走普通快速路径。对于历史输入,只读 open 会单遍解码并迁移源、校验当前逻辑结果,然后在不发布后继的情况下返回。写 open 会在可用时复用按 revision 为键的 preparation,否则执行同一套 preparation,再按有界分片编码同目录临时文件、执行隔离校验、复查源修订,并在返回前以不覆盖方式发布当前后继。源保持逐字节不变。如果源在 preparation 后发生变化,该次写 open 会失败,已经返回给读方的逻辑历史不会被替换;后续写 open 会针对新的 revision 重新执行 preparation。后端在 memo 化前冻结已解码的 event graph,并在此时将其标记为 `shared-frozen`;每个嵌套对象和数组都会冻结,句柄读取和 slice 即使为空也保留该状态。只有尚未实体化的 pending 空日志报告 `detached`。`stat(id)` 与 `list()` 只选择并转换最高 generation 的 header,不读取事件行,也不启动迁移;快照携带所选文件的 `sizeBytes` 与尽力而为的修订号。当前格式修订号标识该文件;历史格式修订号还包含持久化根目录中所选文件的指纹,因此子日志变化会使缓存的逻辑事件失效。指纹只读取文件系统元数据;无关变化也会保守地使历史修订号失效。一次 `list()` 为其历史条目共用一个全库指纹。选择 `compression: 'none'` 后,日志是外部读取方可直接消费的换行分隔文本;压缩默认值必须经后端读取。 历史正文准备通过 [V3→V4](../session-format-v3-to-v4/README.zh.md) 补齐父目录:从 header 找到候选直属子 Session,通过历史编解码器逐个读取其自身 descriptor,保留紧凑证据与来源修订。此过程不准备子目录,也不发布子后继。不可读或不支持的 header(包括损坏的 Zstandard header 帧)不参与发现,也不出现在 `list()` 中。直接访问损坏的压缩 header 仍会失败;header 的 I/O 错误与取消错误继续传播。子日志解码或 descriptor 字段失败会产生带子路径的警告;父目录没有完整条目时,通过 `subagent/catalog` 保留 header 身份信息。健康子项和已有父目录项仍可使用。打开损坏子 Session 时仍报告该子会话的错误。缺失、不支持或多个 descriptor 同样生成模式未知的目录项,不编造标签。已发布的未知条目仍可浏览;读取子历史时会重试实际日志,并从有效 descriptor 确定模式。准备返回、复用与发布前会重新检查成员集合及已检查来源的修订,也包括读取失败的子日志,使修复后的子日志能够使旧准备缓存失效。来源变化时只读打开重试一次,写打开拒绝发布。取消仍会中止操作。当前 V4 打开跳过发现,并在暴露事件前校验目录字段、唯一性及当前投递归属。 @@ -97,6 +97,8 @@ kind: "package-reference" ### 设计理念 +发布产物在新建的 Worker Thread 中运行打包后的 JavaScript。源码执行使用加载 ESM TypeScript 加载器的独立 Node 进程,避免在短生命周期线程内注册加载器。两条路径保持相同的校验、取消及最多两个隔离执行单元并发的限制;清理会等待所属线程或进程停止。 + 该后端拥有自己完整的存储运行时(`src/storage.ts`):`JsonlSessionHandle` 承载逐句柄修改链、带固定批处理窗口与 single-flight 排空的已路由实时事件缓冲、单调读取与幂等 close;一个 tracker 持有进程内单写者认领、teardown 清扫所遍历的打开句柄集合,以及后端自己的会话监听器所路由进的已创建但未实体化待定会话。历史正文读取共享每个 Session 唯一的一次 Decode/Migrate preparation,按 revision 为键的有界 memo 让紧接的观察到恢复交接复用该解析;backend 在 memo 化前只对每个 event graph 深度冻结一次,因此后续 handle read 无需复制或再次冻结。只有写 open 才发布准备好的后继。本包有意只暴露默认插件导出与配置类型——具体类不是具名导出,因此消费方只耦合 `ctx.sessionPersistence`,其可观察行为由共享 seam 测试套件(`runPersistenceContract`/`runLiveWritePathContract`)钉住。物理修订号组合 device、inode、size 与纳秒时间戳,供准备缓存、稳定读取重试及发布检查使用。历史格式的公开修订号还加入所选路径排序后及其物理修订号的 SHA-256 指纹;锁文件与保留但未被选中的代际不参与计算。 ### 物理编码 diff --git a/packages/session/session-persistence-jsonl/src/migration-verifier.ts b/packages/session/session-persistence-jsonl/src/migration-verifier.ts index 75ef5db3b1..bb878f3ac0 100644 --- a/packages/session/session-persistence-jsonl/src/migration-verifier.ts +++ b/packages/session/session-persistence-jsonl/src/migration-verifier.ts @@ -1,9 +1,9 @@ /** Isolated verification for a staged or competing current JSONL generation. */ import { Worker } from 'node:worker_threads' -import type { WorkerOptions } from 'node:worker_threads' import type { JsonlCompression } from './format.ts' import type { JsonlExpectedPrefix, JsonlVerifiedGeneration } from './generation.ts' +import { spawnSourceVerifier } from './verifier-source-process.ts' interface VerificationRequest { readonly path: string @@ -70,33 +70,16 @@ class VerificationScheduler { const verificationScheduler = new VerificationScheduler() -function workerSpawn(request: VerificationRequest): { readonly entry: string | URL; readonly options: WorkerOptions } { - /* v8 ignore next 3 -- built-worker coverage owns the bundled path. */ +function spawnVerifier(request: VerificationRequest) { + /* v8 ignore next 2 -- built-worker coverage owns the bundled path. */ if (!import.meta.url.endsWith('.ts')) { - return { - entry: new URL('./worker.cjs', import.meta.url), - options: { workerData: request, execArgv: [] }, - } - } - const workerEntry = new URL('./worker.ts', import.meta.url) - const bootstrap = [ - `import { register as registerEsm } from ${JSON.stringify(import.meta.resolve('tsx/esm/api'))}`, - `import { register as registerCjs } from ${JSON.stringify(import.meta.resolve('tsx/cjs/api'))}`, - 'registerCjs()', - 'registerEsm()', - `await import(${JSON.stringify(workerEntry.href)})`, - ].join('\n') - return { - entry: new URL(`data:text/javascript,${encodeURIComponent(bootstrap)}`), - options: { - workerData: request, - execArgv: [], - }, + return new Worker(new URL('./worker.cjs', import.meta.url), { workerData: request, execArgv: [] }) } + return spawnSourceVerifier(new URL('./worker.ts', import.meta.url), request) } /** - * Verify one current generation in a fresh Worker Thread. + * Verify one current generation in a fresh Worker Thread, or a process for source execution. * @param path - staged or competing current-generation path. * @param compression - configured physical encoding. * @param expectedId - Session id expected in the decoded header. @@ -136,8 +119,7 @@ function runVerificationWorker( path, compression, expectedId, expectedEventCount, ...(expectedPrefix === undefined ? {} : { expectedPrefix }), } - const { entry, options } = workerSpawn(request) - const worker = new Worker(entry, options) + const worker = spawnVerifier(request) return new Promise((resolve, reject) => { let settled = false const cleanup = (): void => { diff --git a/packages/session/session-persistence-jsonl/src/verifier-source-process.ts b/packages/session/session-persistence-jsonl/src/verifier-source-process.ts new file mode 100644 index 0000000000..e8afa2f94a --- /dev/null +++ b/packages/session/session-persistence-jsonl/src/verifier-source-process.ts @@ -0,0 +1,35 @@ +/** Source-only verifier isolation without TypeScript hooks inside Worker Threads. */ + +import { fork, type ChildProcess, type Serializable } from 'node:child_process' +import { fileURLToPath } from 'node:url' + +/** + * Start the source verifier in a Node process with advanced IPC serialization. + * @param entry - source verifier entry module. + * @param request - request validated by the verifier entry. + * @returns message and exit events, plus termination that waits for process closure. + */ +export function spawnSourceVerifier( + entry: URL, request: Serializable, +): Pick & { terminate(): Promise } { + const child = fork(fileURLToPath(entry), [], { + execArgv: ['--import', import.meta.resolve('tsx/esm')], + serialization: 'advanced', + stdio: ['ignore', 'ignore', 'inherit', 'ipc'], + }) + let closed = false + const completion = new Promise((resolve) => { + child.once('close', (code) => { + closed = true + resolve(code) + }) + }) + child.send(request) + return { + once: child.once.bind(child), + terminate(): Promise { + if (!closed) child.kill('SIGKILL') + return completion + }, + } +} diff --git a/packages/session/session-persistence-jsonl/src/worker.ts b/packages/session/session-persistence-jsonl/src/worker.ts index cc77016c30..511751b08e 100644 --- a/packages/session/session-persistence-jsonl/src/worker.ts +++ b/packages/session/session-persistence-jsonl/src/worker.ts @@ -1,6 +1,7 @@ /** Worker entry for current-generation physical and logical verification. */ import { parentPort, workerData } from 'node:worker_threads' +import type { MessagePort } from 'node:worker_threads' import { verifyJsonlCurrentGeneration } from './generation.ts' import type { JsonlExpectedPrefix } from './generation.ts' import type { JsonlCompression } from './format.ts' @@ -30,12 +31,7 @@ function parseRequest(value: unknown): VerificationRequest { return request as VerificationRequest } -if (parentPort === null) throw new Error('migration verifier requires a parent port') -const port = parentPort - -const request = parseRequest(workerData) - -async function verify(): Promise { +async function verify(request: VerificationRequest, port: Pick): Promise { try { const result = await verifyJsonlCurrentGeneration( request.path, @@ -53,4 +49,15 @@ async function verify(): Promise { } } -void verify() +if (parentPort !== null) { + void verify(parseRequest(workerData), parentPort) +} else { + const send = process.send?.bind(process) + if (send === undefined) throw new Error('migration verifier requires a parent port or IPC channel') + process.once('message', (value: unknown) => { + void verify(parseRequest(value), { + postMessage: (response) => { send(response) }, + close: () => { process.disconnect() }, + }) + }) +} diff --git a/packages/session/session-persistence-jsonl/tests/migration-verifier.spec.ts b/packages/session/session-persistence-jsonl/tests/migration-verifier.spec.ts index 54004522a2..b03249c349 100644 --- a/packages/session/session-persistence-jsonl/tests/migration-verifier.spec.ts +++ b/packages/session/session-persistence-jsonl/tests/migration-verifier.spec.ts @@ -4,7 +4,13 @@ import { verifyCurrentGenerationInWorker } from '../src/migration-verifier.ts' const state = vi.hoisted(() => ({ workers: [] as unknown[] })) vi.mock('node:worker_threads', () => ({ - Worker: class { + Worker: vi.fn(function forbiddenSourceWorker() { + throw new Error('source verification must not register TypeScript hooks in a Worker Thread') + }), +})) + +vi.mock('../src/verifier-source-process.ts', () => { + class Verifier { readonly listeners = new Map void>() readonly terminate = vi.fn<() => Promise>(() => Promise.resolve(0)) @@ -20,8 +26,11 @@ vi.mock('node:worker_threads', () => ({ emit(event: string, value: unknown): void { this.listeners.get(event)?.(value as never) } - }, -})) + } + return { + spawnSourceVerifier: (entry: URL, request: unknown) => new Verifier(entry, { workerData: request }), + } +}) interface FakeWorker { readonly entry: string | URL diff --git a/packages/session/session-persistence-jsonl/tests/verifier-source-process.spec.ts b/packages/session/session-persistence-jsonl/tests/verifier-source-process.spec.ts new file mode 100644 index 0000000000..89924abcfc --- /dev/null +++ b/packages/session/session-persistence-jsonl/tests/verifier-source-process.spec.ts @@ -0,0 +1,75 @@ +/** Source verifier transport, exit observation, and cancellation ownership. */ + +import { EventEmitter } from 'node:events' +import { pathToFileURL } from 'node:url' +import { resolve } from 'node:path' +import { afterEach, expect, it, vi } from 'vitest' +import { spawnSourceVerifier } from '../src/verifier-source-process.ts' + +const fork = vi.hoisted(() => vi.fn()) +vi.mock('node:child_process', () => ({ fork })) + +afterEach(() => { fork.mockReset() }) + +function childProcess() { + const child = Object.assign(new EventEmitter(), { send: vi.fn(), kill: vi.fn() }) + fork.mockReturnValue(child) + return child +} + +it('sends the request to an ESM source process with BigInt-capable IPC', () => { + const child = childProcess() + const request = { path: '/stage', expectedId: 'session' } + const entry = resolve('worker.ts') + spawnSourceVerifier(pathToFileURL(entry), request) + + expect(fork).toHaveBeenCalledWith(entry, [], { + execArgv: ['--import', import.meta.resolve('tsx/esm')], + serialization: 'advanced', stdio: ['ignore', 'ignore', 'inherit', 'ipc'], + }) + expect(child.send).toHaveBeenCalledWith(request) +}) + +it('forwards response, error, and exit observations', () => { + const child = childProcess() + const verifier = spawnSourceVerifier(pathToFileURL(resolve('worker.ts')), {}) + const response = vi.fn() + const error = vi.fn() + const exit = vi.fn() + verifier.once('message', response) + verifier.once('error', error) + verifier.once('exit', exit) + const result = { identity: { ino: 1n } } + const failure = new Error('spawn failed') + child.emit('message', result) + child.emit('error', failure) + child.emit('exit', 7, null) + + expect(response).toHaveBeenCalledWith(result) + expect(error).toHaveBeenCalledWith(failure) + expect(exit).toHaveBeenCalledWith(7, null) +}) + +it('waits for process closure after requesting termination', async () => { + const child = childProcess() + const verifier = spawnSourceVerifier(pathToFileURL(resolve('worker.ts')), {}) + const terminated = verifier.terminate() + let closed = false + void terminated.then(() => { closed = true }) + + expect(child.kill).toHaveBeenCalledWith('SIGKILL') + child.emit('exit', null, 'SIGKILL') + await Promise.resolve() + expect(closed).toBe(false) + child.emit('close', null, 'SIGKILL') + await expect(terminated).resolves.toBeNull() +}) + +it('keeps a closed process untouched when termination is requested later', async () => { + const child = childProcess() + const verifier = spawnSourceVerifier(pathToFileURL(resolve('worker.ts')), {}) + child.emit('close', 0, null) + + await expect(verifier.terminate()).resolves.toBe(0) + expect(child.kill).not.toHaveBeenCalled() +})