fix(session): isolate corrupt compressed headers during discovery

This commit is contained in:
Dudu-0223 committed 2026-09-20 15:44:52 +08:00
1 parent 21ec132c0e
commit e44efedeb0
7 files changed
+60 -27

No files matched your search

@@ -51,8 +51,14 @@ describe.skipIf(webSnapshotMode() === 'record')('historical preset restoration t
data: { turn: 1, reason: { kind: 'completed' } } }) + '\n')),
]))
const brokenSource = await readFile(brokenPath)
const corruptHeaderId = SessionId('preset-migration-corrupt-header')
const corruptHeaderPath = generationLogPath(scaffold.persistenceRoot, scaffold.workspaceCwd, corruptHeaderId, 2, 'zstd')
const corruptHeaderSource = Buffer.from('invalid Zstandard header frame')
await mkdir(dirname(corruptHeaderPath), { recursive: true })
await writeFile(corruptHeaderPath, corruptHeaderSource)
const listed = await scaffold.ctx.sessionController.list({}, new AbortController().signal)
expect(listed.items.map(item => item.sessionId)).toEqual(expect.arrayContaining([id, childId, brokenId]))
expect(listed.items.map(item => item.sessionId)).not.toContain(corruptHeaderId)
const reader = await scaffold.ctx.sessionPersistence.open(id, 'read')
try {
@@ -123,6 +129,8 @@ describe.skipIf(webSnapshotMode() === 'record')('historical preset restoration t
const context = { sessionIds: [id], cwd: scaffold.workspaceCwd }
expect(normalizeSessionSnapshots([published], context)).toEqual(normalizeSessionSnapshots([expected], context))
expect(await readFile(predecessor)).toEqual(source)
await expect(scaffold.ctx.sessionPersistence.open(corruptHeaderId, 'read')).rejects.toThrow(/invalid frame magic/)
expect(await readFile(corruptHeaderPath)).toEqual(corruptHeaderSource)
expect((await readdir(dirname(predecessor))).filter(name => name.endsWith('.jsonl.zstd')).sort())
.toEqual(['session.v2.jsonl.zstd', `session.v${SESSION_FORMAT_VERSION}.jsonl.zstd`])
} finally {
@@ -2,5 +2,5 @@
# side as of the last confirmed-consistent state. Both languages carry equal authority;
# after editing either side, bring the other along and re-record with:
# pnpm run verify-translation-pairing --write packages/session/session-persistence-jsonl/README.md
README.md: 609368aa42f1ba75d818a74877fcd574bf9d5ddf
README.zh.md: 07c0dc94270ddb69812ec8443688a505269eba59
README.md: 5846404961caaef7a91fb4dda562655d15fd2be9
README.zh.md: b7dd646e57b75fe47f3683d2b230dace628f8eee
@@ -81,7 +81,7 @@ The current-generation scanner applies the current codec owner’s structural ad
`open(id, 'read'|'write')` selects the highest canonical generation. Current input follows the ordinary fast path. For historical input, a read open decodes and migrates the source once, validates the current logical result, and returns it without publishing a successor. A write open reuses that revision-keyed preparation when available, or performs the same preparation, then encodes a same-directory temporary file in bounded chunks, verifies it in a Worker Thread, rechecks the source revision, and publishes the current successor without overwrite before returning. The source remains byte-identical. Source drift after preparation rejects that write open without replacing the logical history already returned to readers; a later write open prepares the new revision. The backend marks decoded event graphs `shared-frozen` when it freezes them before memoization; every nested object and array is frozen, and handle reads and slices preserve that state, including empty slices. Only an unmaterialized pending log reports `detached`. `stat(id)` and `list()` select and translate only the highest generation header without reading event rows or starting migration; snapshots carry the selected file’s `sizeBytes` and a best-effort revision. Current revisions identify that file; historical revisions also fingerprint the selected files across the persistence root, so child changes invalidate cached logical events. Fingerprinting reads filesystem metadata only; unrelated changes conservatively invalidate historical revisions. One `list()` call shares a corpus fingerprint across its historical entries. With `compression: 'none'`, the log is newline-delimited text an external reader can consume directly; the compressed default must be read through the backend.
Historical body preparation completes the parent catalog through [V3→V4](../session-format-v3-to-v4/README.md). It finds candidate direct children from headers, reads each child's own descriptor through historical codecs, and retains compact evidence plus source revisions. It neither prepares child catalogs nor publishes child successors. Unreadable or unsupported headers are omitted from discovery. Child decoding and descriptor-field failures produce warnings naming the child path and retain header identity through `subagent/catalog` when the parent has no complete entry; healthy children and existing parent catalog entries remain available. Opening a damaged child still reports that child's error. Missing, unsupported, or multiple descriptors likewise produce unknown-mode membership without inventing a label. Published unknown entries remain browsable; child history reads retry the actual log and resolve its mode from a valid descriptor. Preparation rechecks membership and inspected source revisions before returning, reuse, and publication, including failed child reads so a repaired child invalidates stale preparation. Source drift retries a read open once and refuses a write open. Cancellation still aborts the operation. Current V4 opens bypass discovery and validate catalog fields, uniqueness, and current delivery ownership before exposing events.
Historical body preparation completes the parent catalog through [V3→V4](../session-format-v3-to-v4/README.md). It finds candidate direct children from headers, reads each child's own descriptor through historical codecs, and retains compact evidence plus source revisions. It neither prepares child catalogs nor publishes child successors. Unreadable or unsupported headers, including corrupt Zstandard header frames, are omitted from discovery and `list()`. Direct access to a corrupt compressed header still rejects; header I/O and cancellation errors propagate. Child decoding and descriptor-field failures produce warnings naming the child path and retain header identity through `subagent/catalog` when the parent has no complete entry; healthy children and existing parent catalog entries remain available. Opening a damaged child still reports that child's error. Missing, unsupported, or multiple descriptors likewise produce unknown-mode membership without inventing a label. Published unknown entries remain browsable; child history reads retry the actual log and resolve its mode from a valid descriptor. Preparation rechecks membership and inspected source revisions before returning, reuse, and publication, including failed child reads so a repaired child invalidates stale preparation. Source drift retries a read open once and refuses a write open. Cancellation still aborts the operation. Current V4 opens bypass discovery and validate catalog fields, uniqueness, and current delivery ownership before exposing events.
Historical `stat` and `list` revisions require metadata work proportional to the root’s Session count. Fresh body preparation scans all selected headers and decodes direct-child bodies; memo reuse still scans membership and checks revisions. Read-only access never publishes an upgrade, so cold processes and evicted preparations repeat that work. Current V4 body reads and revisions avoid the historical corpus scan. See the [measured costs and diagnostic command](../../../.agents/notes/implemented/architecture/2026-08-31-released-session-format-migrations.md#catalog-scan-measurements).
@@ -81,7 +81,7 @@ kind: "package-reference"
`open(id, 'read'|'write')` 选择最高规范 generation。当前格式输入走普通快速路径。对于历史输入,只读 open 会单遍解码并迁移源、校验当前逻辑结果,然后在不发布后继的情况下返回。写 open 会在可用时复用按 revision 为键的 preparation,否则执行同一套 preparation,再按有界分片编码同目录临时文件、在 Worker Thread 中校验、复查源修订,并在返回前以不覆盖方式发布当前后继。源保持逐字节不变。如果源在 preparation 后发生变化,该次写 open 会失败,已经返回给读方的逻辑历史不会被替换;后续写 open 会针对新的 revision 重新执行 preparation。后端在 memo 化前冻结已解码的 event graph,并在此时将其标记为 `shared-frozen`;每个嵌套对象和数组都会冻结,句柄读取和 slice 即使为空也保留该状态。只有尚未实体化的 pending 空日志报告 `detached`。`stat(id)` 与 `list()` 只选择并转换最高 generation 的 header,不读取事件行,也不启动迁移;快照携带所选文件的 `sizeBytes` 与尽力而为的修订号。当前格式修订号标识该文件;历史格式修订号还包含持久化根目录中所选文件的指纹,因此子日志变化会使缓存的逻辑事件失效。指纹只读取文件系统元数据;无关变化也会保守地使历史修订号失效。一次 `list()` 为其历史条目共用一个全库指纹。选择 `compression: 'none'` 后,日志是外部读取方可直接消费的换行分隔文本;压缩默认值必须经后端读取。
历史正文准备通过 [V3→V4](../session-format-v3-to-v4/README.zh.md) 补齐父目录:从 header 找到候选直属子 Session,通过历史编解码器逐个读取其自身 descriptor,保留紧凑证据与来源修订。此过程不准备子目录,也不发布子后继。不可读或不支持的 header 不参与发现。子日志解码或 descriptor 字段失败会产生带子路径的警告;父目录没有完整条目时,通过 `subagent/catalog` 保留 header 身份信息。健康子项和已有父目录项仍可使用。打开损坏子 Session 时仍报告该子会话的错误。缺失、不支持或多个 descriptor 同样生成模式未知的目录项,不编造标签。已发布的未知条目仍可浏览;读取子历史时会重试实际日志,并从有效 descriptor 确定模式。准备返回、复用与发布前会重新检查成员集合及已检查来源的修订,也包括读取失败的子日志,使修复后的子日志能够使旧准备缓存失效。来源变化时只读打开重试一次,写打开拒绝发布。取消仍会中止操作。当前 V4 打开跳过发现,并在暴露事件前校验目录字段、唯一性及当前投递归属。
历史正文准备通过 [V3→V4](../session-format-v3-to-v4/README.zh.md) 补齐父目录:从 header 找到候选直属子 Session,通过历史编解码器逐个读取其自身 descriptor,保留紧凑证据与来源修订。此过程不准备子目录,也不发布子后继。不可读或不支持的 header(包括损坏的 Zstandard header 帧)不参与发现,也不出现在 `list()` 中。直接访问损坏的压缩 header 仍会失败;header 的 I/O 错误与取消错误继续传播。子日志解码或 descriptor 字段失败会产生带子路径的警告;父目录没有完整条目时,通过 `subagent/catalog` 保留 header 身份信息。健康子项和已有父目录项仍可使用。打开损坏子 Session 时仍报告该子会话的错误。缺失、不支持或多个 descriptor 同样生成模式未知的目录项,不编造标签。已发布的未知条目仍可浏览;读取子历史时会重试实际日志,并从有效 descriptor 确定模式。准备返回、复用与发布前会重新检查成员集合及已检查来源的修订,也包括读取失败的子日志,使修复后的子日志能够使旧准备缓存失效。来源变化时只读打开重试一次,写打开拒绝发布。取消仍会中止操作。当前 V4 打开跳过发现,并在暴露事件前校验目录字段、唯一性及当前投递归属。
历史格式的 `stat` 与 `list` 修订号需要与根目录 Session 数量成正比的元数据工作。首次正文准备扫描所有所选 header 并解码直属子正文;复用准备缓存仍扫描成员集合并检查修订。只读访问从不发布升级,因此冷进程与被淘汰的准备缓存会重复这些工作。当前 V4 正文读取和修订号避开历史全库扫描。见[实测成本与诊断命令](../../../.agents/notes/implemented/architecture/2026-08-31-released-session-format-migrations.zh.md#catalog-scan-measurements)。
@@ -1067,7 +1067,7 @@ class JsonlSessionPersistence extends SessionPersistence {
try {
header = await this.readGenerationHeader(selected, undefined, signal)
} catch (error: unknown) {
if (error instanceof SessionFormatUnsupportedError) continue
if (error instanceof SessionFormatUnsupportedError || error instanceof SessionPersistenceCorruptionError) continue
throw error
}
if (header === undefined) {
@@ -1406,7 +1406,7 @@ class JsonlSessionPersistence extends SessionPersistence {
}
}
/** Read and validate only the independently compressed header frame. */
/** Read only the header frame; compression failures reject as corruption, while I/O and cancellation propagate. */
private async readFirstZstdLine(path: string, signal?: AbortSignal): Promise<string | undefined> {
signal?.throwIfAborted()
const handle = await open(path, 'r')
@@ -1422,21 +1422,21 @@ class JsonlSessionPersistence extends SessionPersistence {
signal?.throwIfAborted()
content = Buffer.concat([content, chunk.subarray(0, bytesRead)])
signal?.throwIfAborted()
const first = scanZstdFrames(content, 1).frames[0]
signal?.throwIfAborted()
if (first === undefined) continue
let plaintext: Buffer
try {
const first = scanZstdFrames(content, 1).frames[0]
if (first === undefined) continue
const plaintext = await decompressZstdFrame(content.subarray(first.start, first.end))
signal?.throwIfAborted()
plaintext = await decompressZstdFrame(content.subarray(first.start, first.end))
assertZstdHeaderFrame(plaintext)
return plaintext.subarray(0, -1).toString('utf8')
} catch (error) {
/* v8 ignore next -- decoder failure plus concurrent abort is timing-dependent */
if (signal?.aborted) signal.throwIfAborted()
throw new Error('corrupt Zstandard session log: header frame failed validation', { cause: error })
throw new SessionPersistenceCorruptionError(
`corrupt Zstandard session log: header frame failed validation: ${String(error)} (raw log: ${path})`,
{ cause: error },
)
}
signal?.throwIfAborted()
assertZstdHeaderFrame(plaintext)
return plaintext.subarray(0, -1).toString('utf8')
}
} finally {
await handle.close()
@@ -58,15 +58,36 @@ describe.each(['none', 'zstd'] as const)('historical catalog publication (%s)',
await expect(f.ctx.sessionPersistence.open(f.parent, access)).rejects.toThrow('wrong Session')
})
it('opens the parent when another Session header cannot be read', async () => {
const f = await fixture()
const path = await f.write('child', [], true)
await writeFile(path, compression === 'none' ? '{broken header\n' : await compressZstdFrame('{broken header\n'))
expect(await f.read()).toEqual([])
const writer = await f.ctx.sessionPersistence.open(f.parent, 'write')
await writer.close()
expect(await readFile(generationLogPath(f.root, undefined, f.parent, 4, compression))).not.toHaveLength(0)
})
it.each(compression === 'none' ? ['json'] as const : ['json', 'magic', 'checksum', 'lines'] as const)(
'opens the parent when another Session header has corrupt %s', async (damage) => {
const f = await fixture()
await f.write('child', [f.descriptor], true)
const path = await f.write('unrelated', [])
let damaged: Buffer
if (damage === 'json') {
damaged = compression === 'none' ? Buffer.from('{broken header\n') : await compressZstdFrame('{broken header\n')
} else if (damage === 'lines') {
damaged = await compressZstdFrame('{}\n{}\n')
} else {
damaged = await readFile(path)
const offset = damage === 'magic' ? 0 : damaged.length - 1
damaged[offset] = damaged[offset]! ^ 0xFF
}
await writeFile(path, damaged)
expect((await f.ctx.sessionPersistence.list()).map(row => row.header.id).sort()).toEqual(['child', f.parent])
const expected = [{ type: 'subagent/catalog', data: { childId: 'child', mode: 'continuable', label: 'old child' } }]
expect(await f.read()).toMatchObject(expected)
const writer = await f.ctx.sessionPersistence.open(f.parent, 'write')
try { expect((await writer.read()).events).toMatchObject(expected) } finally { await writer.close() }
expect(await readFile(generationLogPath(f.root, undefined, f.parent, 4, compression))).not.toHaveLength(0)
for (const access of ['read', 'write'] as const) {
await expect(f.ctx.sessionPersistence.open(SessionId('unrelated'), access)).rejects.toThrow()
}
expect(await readFile(path)).toEqual(damaged)
await expect(readFile(generationLogPath(f.root, undefined, SessionId('unrelated'), 4, compression)))
.rejects.toMatchObject({ code: 'ENOENT' })
},
)
it('invalidates lightweight revisions on child membership and contents while retaining stable tokens', async () => {
const f = await fixture()
@@ -782,7 +782,7 @@ describe('JsonlSessionPersistence: default Zstandard encoding', () => {
expect((await readAll(ctx.sessionPersistence, header.id)).events).toEqual([...oneTurnLog(), ...secondTurn])
})
it('skips empty, incomplete, and non-header compressed artifacts while rejecting malformed header frames', async () => {
it('skips unreadable compressed headers during discovery while rejecting targeted reads', async () => {
const root = await freshRoot()
for (const [id, content] of [
['empty', Buffer.alloc(0)],
@@ -803,7 +803,7 @@ describe('JsonlSessionPersistence: default Zstandard encoding', () => {
JSON.stringify({ type: 'turn/start' }),
'',
].join('\n')))
await expect(ctx.sessionPersistence.list()).rejects.toThrow(/first frame is not exactly one header line/)
expect(await ctx.sessionPersistence.list()).toEqual([])
await expect(ctx.sessionPersistence.open(twoLinesId, 'read'))
.rejects.toThrow(/first frame is not exactly one header line/)
})
@@ -824,7 +824,11 @@ describe('JsonlSessionPersistence: default Zstandard encoding', () => {
.rejects.toThrow(/empty or header-less Zstandard session log/)
await expect(ctx.sessionPersistence.open(SessionId('empty-header'), 'read'))
.rejects.toThrow(/first frame is not exactly one header line/)
await expect(ctx.sessionPersistence.list()).rejects.toThrow(/header frame failed validation/)
expect(await ctx.sessionPersistence.list()).toEqual([])
await expect(ctx.sessionPersistence.stat(SessionId('bad-checksum')))
.rejects.toThrow(/header frame failed validation/)
await expect(ctx.sessionPersistence.open(SessionId('bad-checksum'), 'read'))
.rejects.toThrow(/frame at byte 0 failed validation/)
})
})