deepseek-harness/.agents/notes/archived/architecture/2026-06-14-session-persistence.md

Agent Note: Session persistence as an abstract service over the existing SessionEvent

Status: implemented Archived: 2026-09-30

English | 中文

Problem

Sessions lived only in memory. The example session-jsonl.ts plugin (duplicated byte-for-byte in both examples) was write-only telemetry: it buffered session/event and appended JSON lines, with no read/replay path, no crash-safety (no fsync, no atomic write, a fire-and-forget dispose drain), no listing, and no format versioning. Nothing could rehydrate a past session from disk into a live agent, so durable resume, durable forking, and host-side session browsing were all impossible.

The event-sourced model makes the append-only log the single source of truth and derives LLM history from it. Persistence had to stay faithful to that: persist the existing SessionEvent directly, with no parallel "persisted message" type that the log is converted to and from. The backend also had to be swappable — a file store now, a database store later — behind one interface.

Decision

Persistence is a capability seam with an abstract Service Definition (capability seams, the dsh-shell template), not loop or core logic:

1. Interface (dsh-session-persistence, ctx.sessionPersistence) — an abstract SessionPersistence service: create/open/stat/list/flush, with create/open returning per-session SessionHandles that carry read/append/flush/close (handle-based seam). Its persisted unit IS the existing SessionEvent ({ type, seq, time, data }), reused verbatim — no conversion type.
2. Implementation (dsh-session-persistence-jsonl) — an append-only logical JSONL log per session: a SessionHeader line followed by storage records that losslessly represent the contiguous SessionEvent stream. The current format writes one event per row; frozen v0 and v1 readers retain their historical packed-delta representation. Checksummed Zstandard frames are the default physical encoding, with raw lines configurable.

Key durable, contested choices:

• The canonical durable log persists every current SessionEvent losslessly. One assistant/message or assistant/attempt embeds the exact timed provider stream for an attempt; deriveMessages() projects only the surface message. Dropping embedded stream members is tempting, but it loses replay, timing, usage, partial-failure, and diagnostic facts. Removing a complete event likewise requires dense renumbering because seq = log.length and events[i].seq === i; the v1-to-v2 migration performs that rewrite explicitly rather than filtering the canonical log.
• Append-only; a crashed turn is closed, never truncated. Flushed events are never rewritten. The semantic checkpoint policy drains the request before model dispatch, a recorded top-level call before tool dispatch, and the complete response/result batch after a step; the loop drains the final turn boundary. Because one interrupted turn may contain substantial valid work, persistence returns its contiguous, parseable events unmodified; the reader owns balancing — resume computes risk-classified error results for unanswered assistant calls, a missing step/end, and turn/end with { kind: 'interrupted' } (interruptedTurnClosers) and appends them through its write handle, while read-only observers add the same closers in memory. The synthetic results keep resumed provider transcripts valid. Only the incomplete fragment of a torn final append is discarded — complete records recovered from it are durably rewritten by the write path before its first new append; a parse error or sequence gap in the committed prefix is corruption and makes the session unloadable.
• The file backend is canonical while the service remains extensible. dsh-session-persistence-jsonl is the sole first-party provider and passes runPersistenceContract; the abstract service remains available to out-of-tree providers. The JSONL-only persistence decision owns removal of the first-party database provider and its deliberate compatibility cut.
• Metadata is out-of-log. Format version, cwd, and lineage are storage concerns, not replayable conversation state, so they live in a SessionHeader owned by dsh-session and attached to a Session via a new readonly session.header — never in SessionEventMap, never reaching deriveMessages(). createdAt is non-negative safe-integer Unix epoch milliseconds: live creation and persistence registration reject fractional values, and JSONL validates the decoded header. The alternative (a merge-extensible session/meta event as log line 0) was rejected: an in-log event would ride along with a seeded/forked session for free, but metadata is not replayable state, so the explicit out-of-log header boundary is the cleaner cost. (The header was originally split into an immutable SessionHeader plus a mutable SessionSummary whose union was SessionMeta; the mutable summary was later removed as dead state — see Drop the mutable session summary.)
• ctx.agents.create() and ctx.agents.resume() are async factories; resume additionally crosses the persistence boundary. ctx.agents.resume({ resumeSessionId }) opens the session's write handle, reads the stored log, and publishes the prepared Session under the persisted id, continuing its projections. The Session preparation decision owns the unpublished-Session ownership window. The agent-loop does NOT hard-inject sessionPersistence (that would pend non-persistent demos forever); resume rejects with a clear error when it is absent.

Alternatives considered

Each key choice above records its rejected alternative where the choice is stated: a stream-filtered canonical log — loses attempt evidence, while removing events without an explicit migration breaks contiguous sequence numbers; truncating a crashed turn — silently destroys a long autonomous run's real work; an in-log session/meta event as log line 0 — metadata is not replayable state; finite fractional createdAt values — have no producer and diverge from integer Unix-millisecond storage; hard-injecting sessionPersistence into the loop — would pend non-persistent demos forever.

Format versioning: the header carries a version; handles expose only the logical format selected by SESSION_FORMAT_VERSION (version authority). JSONL event-body reads compose the complete static adjacent migration chain before returning a handle; each edge owns its historical transformations. V0 remains at suffixless session.jsonl[.zstd], while positive versions use immutable lowercase session.vN.jsonl[.zstd] names (released Session migration). Current-generation append and flush are robust to partial trailing writes; a future provider or write-ahead log needs its own power-loss and recovery contract.

Consequences

The Service Definition, JSONL provider, and metadata contract in dsh-session (session.header, the create(header, options?) signature) buy durable resume/fork, a read/replay path, crash tolerance, and host-side session access over the existing event-sourced log. The reusable runPersistenceContract suite holds the provider and future implementations to the same append-only, contiguous-seq, lazy-materialization, logical-recovery, integer-metadata, and serializability semantics. Persisting the full logical log also settles event fidelity: every Assistant attempt retains its exact compact timed stream in one durable settlement.