diff --git a/.agents/notes/implemented/architecture/2026-09-17-native-v4-read-validation.i18n.yaml b/.agents/notes/implemented/architecture/2026-09-17-native-v4-read-validation.i18n.yaml index 91d23342da..c3afeacd89 100644 --- a/.agents/notes/implemented/architecture/2026-09-17-native-v4-read-validation.i18n.yaml +++ b/.agents/notes/implemented/architecture/2026-09-17-native-v4-read-validation.i18n.yaml @@ -2,5 +2,5 @@ # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: # pnpm run verify-translation-pairing --write .agents/notes/implemented/architecture/2026-09-17-native-v4-read-validation.md -2026-09-17-native-v4-read-validation.md: e98cfff679254557ebd5be35c23810cbd2b920f9 -2026-09-17-native-v4-read-validation.zh.md: 2b586bc3eb45a20875b69ce789c995d0ab208f7d +2026-09-17-native-v4-read-validation.md: b111e9f1dd4aae6557b7893a9f334445ef0c85bc +2026-09-17-native-v4-read-validation.zh.md: 20f1dc53b6eda42834e169c18f7b269e2933d27c diff --git a/.agents/notes/implemented/architecture/2026-09-17-native-v4-read-validation.md b/.agents/notes/implemented/architecture/2026-09-17-native-v4-read-validation.md index e98cfff679..b111e9f1dd 100644 --- a/.agents/notes/implemented/architecture/2026-09-17-native-v4-read-validation.md +++ b/.agents/notes/implemented/architecture/2026-09-17-native-v4-read-validation.md @@ -18,6 +18,8 @@ An unfinished tail may retain an open turn, step, tool call, or compaction. A cl Installed Session adoption continues to own event envelopes, message roles and metadata, canonical request headers, and surface replacement reference coverage. Optional runtime invariants provide diagnostics; their installation is not required for durable admission. Native system-message admission also validates coordinates, identity, content blocks, and known image/tool-call fields before physical recovery can discard rows. Additional JSON fields and unknown nonempty content tags remain intact; retired tool-result wrappers are refused. Retired `request/header.header.system` and required `tool/code-dispatch*` tags remain hard refusals before recovery; obsolete ignorable PTC records remain opaque. Physical framing alone does not establish semantic validity. +A retired tool-result wrapper outside its owning `tool/result` row would require changing the event, role, or source to represent it in V4. The adjacent migration lifts only the canonical owned wrapper; target restoration refuses wrappers remaining in interpreted content. Native row admission applies the same refusal before recoverable tail suppression. The check follows declared message, content, and Assistant-stream slots instead of recursively inspecting JSON, so tool arguments, replay state, schema parameters, nested extensions, and unknown ignorable payloads retain their recorded values. + `EpochHeader.system` explicitly reserves the retired key as `system?: never` with `@persistenceReserved`. The catalog retains its empty value set so permitting a value is a change to a forbidden field, rather than an ordinary optional-field addition. The native reader and Session adoption continue to reject that key; system prompts remain `system/message` events. ## Alternatives considered @@ -28,4 +30,4 @@ Installed Session adoption continues to own event envelopes, message roles and m ## Consequences -The native restorer maintains one bounded state machine per read and rejects contradictions before successor publication. Frozen V0–V3 validators and committed Session generations remain unchanged. Public catalog regressions cover invalid relationships alongside partial tails, inherited cuts, immutable unknown attribution, and complete tool exchanges; these checks define finalized V4 admission. Breaking changes require a successor; compatible additions use new acknowledgements, and implementation fixes preserve the declared meaning. +The native restorer maintains one bounded state machine per read and rejects contradictions before successor publication. Frozen V0–V3 validators and committed Session generations remain unchanged. Public catalog regressions cover invalid relationships alongside partial tails, inherited cuts, immutable unknown attribution, and complete tool exchanges; these checks enforce the declared V4 representation without changing its schemas. Breaking changes require a successor; compatible additions use new acknowledgements, and implementation fixes preserve the declared meaning. diff --git a/.agents/notes/implemented/architecture/2026-09-17-native-v4-read-validation.zh.md b/.agents/notes/implemented/architecture/2026-09-17-native-v4-read-validation.zh.md index 2b586bc3eb..20f1dc53b6 100644 --- a/.agents/notes/implemented/architecture/2026-09-17-native-v4-read-validation.zh.md +++ b/.agents/notes/implemented/architecture/2026-09-17-native-v4-read-validation.zh.md @@ -18,6 +18,8 @@ V4 格式恢复器与当前 JSONL 扫描器共用同一强制跨事件关系校 已安装的 Session 接纳器继续负责事件封装、消息角色与元数据、规范请求头,以及 surface 替换的引用覆盖。可选运行时不变式提供诊断;持久化准入不要求安装它们。原生 system 消息准入还会在物理恢复可能丢弃行之前,校验坐标、身份、内容块及已知 image/tool-call 字段。额外 JSON 字段和非空的未知内容标签保持不变,退役的 tool-result 包装会被拒绝。退役的 `request/header.header.system` 和必需的 `tool/code-dispatch*` 标签仍在恢复前强制拒绝;标记为可忽略的过时 PTC 记录保持不透明。仅通过物理分帧不能证明语义有效。 +退役的 tool-result 包装若不属于对应的 `tool/result` 行,就需要改变事件、角色或来源才能在 V4 中表示。相邻迁移只提升规范的自有包装;目标恢复器拒绝解释内容中剩余的包装。原生行准入在可恢复尾部被忽略之前执行同样的拒绝。检查沿声明的消息、内容和 Assistant stream 字段进行,不递归扫描 JSON,因此工具参数、replay state、schema 参数、嵌套扩展和未知可忽略载荷保留其记录值。 + `EpochHeader.system` 通过带有 `@persistenceReserved` 的 `system?: never` 显式保留退役键。目录保留其空值集合,使允许某个值被识别为对禁止字段的修改,而非普通可选字段添加。原生读取方和 Session 接纳继续拒绝该键;系统提示词仍由 `system/message` 事件承载。 ## Alternatives considered @@ -28,4 +30,4 @@ V4 格式恢复器与当前 JSONL 扫描器共用同一强制跨事件关系校 ## Consequences -原生恢复器为每次读取维护独立状态机,在发布后继文件之前拒绝矛盾。冻结的 V0–V3 校验器及已提交的 Session 代际保持不变。公开目录回归覆盖无效关系、部分尾部、继承切分点、未知归属的不可变保存,以及完整工具交互;这些检查定义已定稿 V4 的接纳规则。破坏性变更要求后继版本;兼容新增使用新的确认记录,实现修复须保留声明的含义。 +原生恢复器为每次读取维护独立状态机,在发布后继文件之前拒绝矛盾。冻结的 V0–V3 校验器及已提交的 Session 代际保持不变。公开目录回归覆盖无效关系、部分尾部、继承切分点、未知归属的不可变保存,以及完整工具交互;这些检查落实已声明的 V4 表示,不改变其 schema。破坏性变更要求后继版本;兼容新增使用新的确认记录,实现修复须保留声明的含义。 diff --git a/packages/session/session-format-v3-to-v4/src/codec.ts b/packages/session/session-format-v3-to-v4/src/codec.ts index 7cfae7fe24..b7cf4e7a2b 100644 --- a/packages/session/session-format-v3-to-v4/src/codec.ts +++ b/packages/session/session-format-v3-to-v4/src/codec.ts @@ -44,7 +44,10 @@ export const releasedV4SessionFormatCodec = Object.freeze({ return { ...releasedV2SessionFormatCodec.encodeHeader({ ...header, version: 2 }, inheritedEventCount), version: 4 } }, encodeEvent(event: SessionFormatEvent) { - if (event.type === 'developer/message' && event['ignorable'] === true) assertV4DeveloperData(event) + if (event.type === 'developer/message' && event['ignorable'] === true) { + assertV4DeveloperData(event) + assertV4RetiredSyntax(event) + } assertV4RowAdmission(event) return releasedV2SessionFormatCodec.encodeEvent(event) }, diff --git a/packages/session/session-format-v3-to-v4/src/retired-syntax.ts b/packages/session/session-format-v3-to-v4/src/retired-syntax.ts index 6644f60c3b..6571f3171b 100644 --- a/packages/session/session-format-v3-to-v4/src/retired-syntax.ts +++ b/packages/session/session-format-v3-to-v4/src/retired-syntax.ts @@ -2,8 +2,23 @@ import { SessionFormatError, SessionFormatUnsupportedMigrationError, isSessionFormatJsonObject } from '@deepseek-ai/dsh-session-format' +function assertBlock(block: unknown, subject: string): void { + if (isSessionFormatJsonObject(block) && block['type'] === 'tool-result') { + throw new SessionFormatError(`${subject} must not contain a released tool-result wrapper`) + } +} + +function assertContent(content: unknown, subject: string): void { + if (Array.isArray(content)) content.forEach((block) => { assertBlock(block, subject) }) +} + +function assertMessageContent(message: unknown, subject: string): void { + if (isSessionFormatJsonObject(message)) assertContent(message['content'], subject) +} + /** - * Refuse retired request-header system text and predecessor PTC tags without interpreting extension payloads. + * Refuse retired headers, PTC tags, and tool-result blocks in interpreted content slots. + * Tool arguments, replay state, schema parameters, and nested extension values remain opaque. * @param row - parsed physical row or complete native logical event. */ export function assertV4RetiredSyntax(row: unknown): void { @@ -11,12 +26,50 @@ export function assertV4RetiredSyntax(row: unknown): void { if ((row['type'] === 'tool/code-dispatch-start' || row['type'] === 'tool/code-dispatch') && row['ignorable'] !== true) { throw new SessionFormatUnsupportedMigrationError(`format v4 rejects retired event type ${row['type']}`) } - if (row['type'] !== 'request/header') return const data = row['data'] - if (!isSessionFormatJsonObject(data) || !isSessionFormatJsonObject(data['header'])) { - throw new SessionFormatError('format v4 request/header requires data and header objects') + if (row['type'] === 'request/header') { + if (!isSessionFormatJsonObject(data) || !isSessionFormatJsonObject(data['header'])) { + throw new SessionFormatError('format v4 request/header requires data and header objects') + } + if (Object.hasOwn(data['header'], 'system')) { + throw new SessionFormatError('format v4 request/header rejects retired header.system') + } + return } - if (Object.hasOwn(data['header'], 'system')) { - throw new SessionFormatError('format v4 request/header rejects retired header.system') + if (!isSessionFormatJsonObject(data)) return + const subject = `format v4 ${row['type']} at seq ${row['seq']} content` + switch (row['type']) { + case 'user/message': + assertContent(data['content'], subject) + break + case 'developer/message': + case 'assistant/message': + case 'team/message/queued': + assertMessageContent(data['message'], subject) + break + case 'agent/inbox/spliced': + case 'session/title-llm-request': { + const messages = data[row['type'] === 'agent/inbox/spliced' ? 'inserted' : 'messages'] + if (Array.isArray(messages)) messages.forEach((message) => { assertMessageContent(message, subject) }) + break + } + case 'compaction/summary': + assertContent(data['summary'], subject) + assertContent(data['rawOutput'], subject) + break + case 'tool/ptc-dispatch': + assertContent(data['content'], subject) + break + } + if ((row['type'] === 'assistant/message' || row['type'] === 'assistant/attempt') && Array.isArray(data['stream'])) { + for (const entry of data['stream']) { + if (!isSessionFormatJsonObject(entry) || entry['type'] !== 'chunk') continue + const chunk = entry['chunk'] + if (!isSessionFormatJsonObject(chunk)) continue + if (chunk['type'] === 'block-end') assertBlock(chunk['block'], subject) + if (chunk['type'] === 'block-start' && chunk['blockType'] === 'tool-result') { + throw new SessionFormatError(`${subject} must not contain a released tool-result wrapper`) + } + } } } diff --git a/packages/session/session-format-v3-to-v4/tests/retired-content.spec.ts b/packages/session/session-format-v3-to-v4/tests/retired-content.spec.ts new file mode 100644 index 0000000000..68f70c9d38 --- /dev/null +++ b/packages/session/session-format-v3-to-v4/tests/retired-content.spec.ts @@ -0,0 +1,112 @@ +import { describe, expect, it } from 'vitest' +import { SessionFormatEventCollector } from '@deepseek-ai/dsh-session-format' +import type { SessionFormatEvent, SessionFormatJsonObject } from '@deepseek-ai/dsh-session-format' +import { createSessionFormatCatalogWithChildren, historicalSessionFormatCatalog } from '@deepseek-ai/dsh-session-format-catalog' +import { assertV4RowAdmission, releasedV4SessionFormatCodec, restoreReleasedV4Artifact } from '../src/index.ts' + +const header = { version: 4, id: 'retired-content', createdAt: 1, isSeeded: false, delegationDepth: 0 } +const wrapper: SessionFormatJsonObject = { + type: 'tool-result', toolCallId: 'retired-call', content: [{ type: 'text', text: 'saved result' }], +} +const content = [wrapper] +const user = { id: 'user', role: 'user', source: { kind: 'user' }, content } +const message = (role: string): SessionFormatJsonObject => ({ + id: role, role, content, + source: role === 'assistant' ? { kind: 'model', provider: 'test', model: 'test' } + : role === 'tool' ? { kind: 'tool', callId: 'call' } : { kind: `${role}-prompt` }, + ...(role === 'tool' ? { toolCallId: 'call' } : {}), +}) +const cases: readonly { name: string; type: string; data: SessionFormatJsonObject; error?: RegExp }[] = [ + { name: 'user message', type: 'user/message', data: user }, + ...['system', 'developer', 'assistant'].map(role => ({ + name: `${role} message`, type: `${role}/message`, data: { turn: 1, step: 1, message: message(role) }, + ...(role === 'system' ? { error: /rejects retired tool-result/ } : {}), + })), + { name: 'native tool message', type: 'tool/result', data: { turn: 1, step: 1, message: message('tool') } }, + { name: 'queued input', type: 'agent/inbox/spliced', data: { target: 'next-turn', start: 0, inserted: [user] } }, + { name: 'title request', type: 'session/title-llm-request', data: { messages: [user] } }, + { name: 'compaction summary', type: 'compaction/summary', data: { summary: content } }, + { name: 'compaction raw output', type: 'compaction/summary', data: { summary: [], rawOutput: content } }, + { name: 'PTC result', type: 'tool/ptc-dispatch', data: { content } }, + { name: 'team message', type: 'team/message/queued', data: { message: user } }, + ...['assistant/message', 'assistant/attempt'].flatMap(type => ['block-start', 'block-end'].map(kind => ({ + name: `${type} stream ${kind}`, type, data: { + turn: 1, step: 1, message: { ...message('assistant'), content: [] }, + stream: [{ type: 'chunk', elapsedMs: 0, chunk: kind === 'block-start' + ? { type: kind, index: 0, blockType: 'tool-result' } + : { type: kind, index: 0, block: wrapper } }], + }, + }))), +] + +function row(type: string, data: SessionFormatJsonObject): SessionFormatEvent { + return { type, data, seq: 0, time: 1 } +} + +describe('retired tool-result content admission', () => { + it.each(cases)('refuses $name in native encoding, restoration and recoverable suffixes', ({ type, data, error = /released tool-result wrapper/ }) => { + const event = row(type, data) + const before = structuredClone(event) + expect(() => releasedV4SessionFormatCodec.encodeEvent(event)).toThrow(error) + expect(() => restoreReleasedV4Artifact({ header, inheritedEventCount: 0, events: [event] }, new Set([type]))) + .toThrow(error) + for (const recovery of ['strict', 'recoverable'] as const) { + const decoder = releasedV4SessionFormatCodec.createDecoder({ type: 'session', ...header }, recovery) + const collector = new SessionFormatEventCollector() + if (recovery === 'recoverable') decoder.decodeRow(null, collector) + expect(() => { decoder.decodeRow(event, collector) }).toThrow(error) + } + expect(event).toEqual(before) + }) + + it('refuses a V3 user wrapper at the V4 target while leaving the released reader and source intact', () => { + const event = { ...row('user/message', user), surfaceOp: 'append' } as SessionFormatEvent + const before = structuredClone(event) + const physical = { type: 'session', ...header, version: 3 } + const historical = historicalSessionFormatCatalog.createRestore(physical, { recovery: 'strict', validation: 'current' }) + historical.decodeRow(event) + expect(historical.finish().events).toEqual([event]) + const current = createSessionFormatCatalogWithChildren([]).createRestore(physical, { recovery: 'strict', validation: 'current' }) + expect(() => { current.decodeRow(event); current.finish() }).toThrow(/released tool-result wrapper/) + expect(event).toEqual(before) + }) + + it('interprets ignorable developer content only for a reader that knows the event, or for encoding', () => { + const event = { ...row('developer/message', { turn: 1, step: 1, message: message('developer') }), ignorable: true } + const collector = new SessionFormatEventCollector() + const decoder = releasedV4SessionFormatCodec.createDecoder({ type: 'session', ...header }, 'strict') + decoder.decodeRow(event, collector) + expect(collector.values).toEqual([event]) + expect(() => { assertV4RowAdmission(event, new Set()) }).not.toThrow() + expect(() => { assertV4RowAdmission(event, new Set(['developer/message'])) }).toThrow(/released tool-result wrapper/) + expect(() => releasedV4SessionFormatCodec.encodeEvent(event)).toThrow(/released tool-result wrapper/) + const artifact = { header, inheritedEventCount: 0, events: [event] } + expect(restoreReleasedV4Artifact(artifact, new Set())).toBe(artifact) + }) + + it('keeps tool arguments, replay state, schemas, nested metadata, and unknown ignorable payloads opaque', () => { + const rows: SessionFormatEvent[] = [ + row('user/message', { ...user, content: [{ type: 'text', text: 'text', metadata: wrapper }], replayState: { content } }), + row('tool/call', { arguments: JSON.stringify(wrapper), metadata: { content } }), + row('tool/ptc-dispatch', { content: [{ type: 'text', text: 'result' }], arguments: wrapper }), + row('request/header', { header: { config: { provider: 'test', model: 'test' }, tools: [{ + name: 'example', description: '', parameters: { type: 'object', examples: [wrapper] }, + }] } }), + { ...row('future/opaque', { content, message: user, stream: [{ type: 'chunk', chunk: { type: 'block-end', block: wrapper } }] }), ignorable: true }, + ] + for (const event of rows) { + const before = structuredClone(event) + const encoded = releasedV4SessionFormatCodec.encodeEvent(event) + const collector = new SessionFormatEventCollector() + releasedV4SessionFormatCodec.createDecoder({ type: 'session', ...header }, 'strict').decodeRow(encoded, collector) + expect(collector.values).toEqual([before]) + expect(event).toEqual(before) + } + const opaque = rows.at(-1)! + const migrated = createSessionFormatCatalogWithChildren([]).createRestore({ type: 'session', ...header, version: 3 }, { + recovery: 'strict', validation: 'current', + }) + migrated.decodeRow(opaque) + expect(migrated.finish().events).toEqual([opaque]) + }) +}) diff --git a/packages/session/session-persistence-jsonl/tests/retired-content-admission.spec.ts b/packages/session/session-persistence-jsonl/tests/retired-content-admission.spec.ts new file mode 100644 index 0000000000..eb7368af5e --- /dev/null +++ b/packages/session/session-persistence-jsonl/tests/retired-content-admission.spec.ts @@ -0,0 +1,56 @@ +/** Interpreted retired content must fail before migration publication or recoverable native-tail suppression. */ +import { Context } from '@deepseek-ai/cordis' +import { SessionId } from '@deepseek-ai/dsh-session' +import { SessionFormatUnsupportedError, SessionPersistenceCorruptionError } from '@deepseek-ai/dsh-session-persistence' +import JsonlSessionPersistence from '@deepseek-ai/dsh-session-persistence-jsonl' +import { mkdir, mkdtemp, readFile, readdir, rm, writeFile } from 'node:fs/promises' +import { tmpdir } from 'node:os' +import { basename, dirname, join } from 'node:path' +import { describe, expect, it, onTestFinished } from 'vitest' +import { generationLogPath } from '../src/format.ts' +import { compressZstdFrame } from '../src/zstd.ts' + +const id = SessionId('retired-content-admission') +const wrapper = { type: 'tool-result', toolCallId: 'retired-call', content: [{ type: 'text', text: 'result' }] } +const modes = (['none', 'zstd'] as const).flatMap(compression => + (['read', 'write'] as const).map(access => ({ compression, access }))) + +describe.each(modes)('retired content in JSONL ($compression, $access)', ({ compression, access }) => { + async function stored(version: 3 | 4, opaque: boolean, corruptPrefix: boolean) { + const root = await mkdtemp(join(tmpdir(), 'dsh-retired-content-')) + const ctx = new Context() + onTestFinished(async () => { + try { await ctx.fiber.dispose() } finally { await rm(root, { recursive: true, force: true }) } + }) + const event = { type: opaque ? 'future/opaque' : 'user/message', seq: 0, time: 1, + ...(opaque ? { ignorable: true } : { surfaceOp: 'append' }), + data: { id: 'user', role: 'user', source: { kind: 'user' }, content: [wrapper] }, + } + const header = { type: 'session', version, id, createdAt: 1, delegationDepth: 0, isSeeded: false } + const lines = [JSON.stringify(header) + '\n', ...(corruptPrefix ? ['{invalid json}\n'] : []), JSON.stringify(event) + '\n'] + const bytes = compression === 'none' ? Buffer.from(lines.join('')) + : Buffer.concat(await Promise.all(lines.map(compressZstdFrame))) + const path = generationLogPath(root, undefined, id, version, compression) + await mkdir(dirname(path), { recursive: true }) + await writeFile(path, bytes) + await ctx.plugin(JsonlSessionPersistence, { root, compression }) + return { ctx, path, bytes, event } + } + + it.each([{ version: 3, corruptPrefix: false }, { version: 4, corruptPrefix: false }, { version: 4, corruptPrefix: true }] as const) + ('refuses V$version content with corruptPrefix=$corruptPrefix and preserves the only generation', async ({ version, corruptPrefix }) => { + const { ctx, path, bytes } = await stored(version, false, corruptPrefix) + const opened = ctx.sessionPersistence.open(id, access).then(async (handle) => { await handle.close() }) + await expect(opened).rejects.toBeInstanceOf(version === 3 ? SessionFormatUnsupportedError : SessionPersistenceCorruptionError) + await expect(opened).rejects.toThrow(/released tool-result wrapper/) + expect(await readFile(path)).toEqual(bytes) + expect((await readdir(dirname(path))).filter(name => name !== 'session.lock')).toEqual([basename(path)]) + }) + + it.each([3, 4] as const)('preserves an unknown ignorable V%s payload without interpreting its content', async (version) => { + const { ctx, path, bytes, event } = await stored(version, true, false) + const handle = await ctx.sessionPersistence.open(id, access) + try { expect((await handle.read()).events).toEqual([event]) } finally { await handle.close() } + expect(await readFile(path)).toEqual(bytes) + }) +})