Commit Graph
26 Commits
Author SHA1 Message Date
Magolor 5e5b8bee6e fix(storage): unify versioned batch writes and deletion recovery 2026-10-26 13:04:41 +08:00
Magolor 88b0425dfc fix(storage): contain failed writes 2026-10-26 13:04:41 +08:00
Magolor 84210fa1e9 feat(storage): support compatible whole-unit versions 2026-10-26 13:04:41 +08:00
Chinesezjc 6db5041812 fix(storage): keep batch deletion tracked and consistent 2026-10-22 17:54:21 +08:00
Chinesezjc 3c5ab55d16 perf(storage): delete task rows as one durable batch 2026-10-21 12:21:11 +08:00
Chinesezjc 7a362b263b Merge pull request #4335 from deepseek-harness/feat/schedule-storage-phase1
feat(schedule): persist tasks and resume due sessions
2026-09-23 23:33:33 +08:00
imccyu c917fe6d46 Merge remote-tracking branch 'origin/master' into merge/projcache-v6-compat-into-master
# Conflicts:
#	packages/session/session-projection-cache/src/spec.ts
#	packages/storage/storage-json/src/per-record-unit.ts
#	packages/storage/storage-json/tests/json-backend.spec.ts
2026-09-02 16:55:00 +08:00
imccyu fcd109d29a feat(storage): version read compatibility and backup-and-skip salvage for per-record units
A DomainSpec may declare compatibleVersions: older domain versions whose
stored records the current record schemas still accept. The json backend's
per-record reads admit documents stamped with a declared version (writes
always stamp the current one), and the legacy whole-unit bootstrap migrates
only a file whose stored version is in the accepted set — previously it
migrated any version and stamped the records current, turning a discardable
stale cache into invalid-record failures that refused the whole domain at
open and permanently poisoned the new tree on first boot.

A DomainSpec may also declare invalidRecords: 'backup-and-skip' for domains
whose records are disposable derived data: a stored record failing its zod
schema is moved aside through the new optional KvUnit.backupRecord
(<key>.json.bak.<YYYYMMDDHHmm> under the json backend), logged with its
cause, and skipped, instead of rejecting the open. The default stays
fail-loud, and so do backends without backupRecord.
2026-09-02 15:25:07 +08:00
Magolor d921d4b357 fix(storage-json): reject cross-version legacy bootstrap (#3431)
* fix(storage-json): reject cross-version legacy bootstrap

* test(webworker): sync projection cache fixture version

* docs(storage): record legacy bootstrap version ownership
2026-09-02 04:37:15 +00:00
Turtle 15f2997bcb cleanup: omit unneeded invariant companions 2026-08-28 13:12:52 +08:00
_Kerman 380334436e fix(storage-json): preserve legacy cache after bootstrap 2026-08-25 21:02:39 +08:00
_Kerman 83459fa476 perf(storage-json): load record files concurrently 2026-08-25 20:50:54 +08:00
_Kerman 87c01d9966 test(storage-json): reach the unreadable-record branch on every platform
The per-record contract test forced readFile to fail via chmod 0o000,
which is a no-op on win32, so the readRecord catch stayed uncovered on
the windows native coverage gate. Route record documents without an
isFile pre-filter: a directory where the document should be throws
EISDIR on every platform, and readRecord's existing contract already
reads an unreadable document as absent. Drop the win32 skip.
2026-08-21 11:34:22 +08:00
_Kerman cdb918c4b0 chore(storage-json): exempt the shared unit lifecycle from the duplication gate
The two standalone unit classes deliberately mirror the KvUnit drain/guard
lifecycle (close + assertOpen); mark the block with a reason-carrying
jscpd ignore so the duplication gate stays green.
2026-08-20 17:34:08 +08:00
_Kerman 08e546eff1 feat(storage-json): one-time migration of a legacy whole-unit file to per-record
Opening a per-record unit splits a legacy `<root>/<name>.json` (the
pre-per-record single-file layout) into per-record documents; an
already-present new record wins, and the legacy file is deleted once
every record migrated. The migration also runs when the new tree is
absent — the fresh-upgrade shape — and foreign, shapeless, or malformed
legacy files are left alone. This preserves previously cached session
titles, list metadata, stats, and subagent identity across the medium
change.
2026-08-20 16:25:39 +08:00
_Kerman 501f387b46 feat(storage): add the per-record layout to the json backend
The json backend now serves two layouts. single (the default) keeps the
whole unit as one document at <root>/<name>.json; per-record keeps one
version-stamped document per record at <root>/<name>/<table>/<key>.json
(plus global.json), so one write rewrites one record instead of the whole
unit. The per-record unit is stateless — the directory is the state,
loadAll re-reads the tree, and every write is a single durable file
operation — while single keeps its authoritative in-memory state and
whole-file publish. Records keys must be path-safe ([a-zA-Z0-9_-]+);
an unsafe key rejects. A record document that is malformed or stamped
with another version reads as an absent record: one bad or stale file
never bricks the unit, and a version bump discards stale records instead
of migrating them. DomainSpec and KvUnitDescriptor gain the optional
layout field (defineDomain validates it, descriptorOf projects it).
2026-08-20 13:47:39 +08:00
_Kerman 3f4c5f0563 fix(session-projection-cache): address review — atomic-write reuse, sqlite no-path, ordering and drain
- Write through @deepseek-ai/dsh-atomic-write with { mode: 0o600,
  dirMode: 0o700 } instead of exporting a second atomic-write primitive
  from dsh-storage-json; the session tree stays owner-only like the jsonl
  backend's own directories.
- Serialize atomic replacements per cache path so an older cut can never
  overwrite a newer one; track in-flight writes and drain them on
  disposal so a late flush cannot land after teardown.
- Detect the absent per-session directory before the checkpoint cut and
  durability flush: sqlite-style backends no-op the write entirely.
- Cold-read write-back path and identity both come from the stored log
  header (tail.meta), so a stale caller header cannot mint an orphan
  cache file.
- Add no-path coverage (write no-op, cachedSnapshot undefined, cold
  fallback to the full-log rung) and a concurrent-write ordering test;
  the package now holds 100% statement/branch/function/line coverage.
- Sync README.md/zh (inject list, coldSnapshot signature, per-session
  file read wording), package description, the Agent Note alternatives,
  and the superseded proposed/implemented notes (EN/ZH); add the
  concurrent-checkpoint Known Limitation.
2026-08-19 21:40:12 +08:00
_Kerman cdb4cc3c68 feat(session-projection-cache): store one projection_cache.json per session
Replace the single global session_projcache domain with a per-session
cache file inside the session's own persistence directory, resolved
through sessionPersistence.locate(meta) — the persistence backend owns
the session-directory layout, the cache service keeps every checkpoint
and cold-read responsibility.

- cachedSnapshot(meta) becomes async (one file read per session);
  coldSnapshot takes the session header so it can locate the file, with
  the stored log header remaining the identity witness.
- Backends without a per-session directory (sqlite) disable the durable
  cache: writes no-op and cold reads fall to the full-log rung. An
  obsolete global cache is never read — derived data refolds on first
  cold read (no migration).
- writeAtomic is exported from dsh-storage-json as the shared atomic
  whole-file replace primitive; api-proxy listing and subagent cold
  reads await the now-async cachedSnapshot.
- READMEs and a new Agent Note document the per-session medium.
2026-08-19 21:03:20 +08:00
imccyu ec601ca13d build(vendor): rescope the vendored Cordis packages into @deepseek-ai
Machine-produced by `pnpm run rescope-vendor --apply` plus the regeneration it
prints: `pnpm install` for the lockfile, `pnpm run gen-third-party-notices`,
`verify-translation-pairing --write` for the touched bilingual pairs,
`gen-doc-graphs`, and one typert snapshot whose ids embed character offsets.
`pnpm run rescope-vendor --check` verifies the result.

Renames nine vendored packages (cordis, cosmokit, schemastery and the six
@cordisjs plugins) and every reference that resolves them: manifest names and
dependency keys, module specifiers including declare-module merges, cordis.yml
plugin names, tsconfig paths, every Markdown fence, and `docs/` prose.
Directory names, upstream versions, and dependency ranges are unchanged, so
vendor/README.md still reads as an upstream snapshot; its manifest table gains
an upstream-name column so THIRD_PARTY_NOTICES keeps MIT attribution pointed
at each fork's origin.

The tutorial tier follows the rename end to end: its yaml fences named plugins
the Loader can no longer resolve, its `ts ignore-check` fences disagreed with
the compiled fences beside them, and its prose quoted both. The contracts that
told readers to keep upstream names — the root convention and the vendoring
cookbook's tree comment and manifest invariant — now say to rescope instead.

Two rules read `@deepseek-ai/` as "another workspace plugin": the client bundle
purity gate now names the vendored libraries a browser bundle inlines, and the
files where a bare `cordis` is an agent-preset id keep that product data.
2026-08-10 22:04:13 +08:00
Turtle 95a995968b refactor: migrate linting to Oxlint 2026-07-29 14:32:11 +08:00
imccyu 9eb9c70a8a feat(web): add workspace-aware session flow 2026-07-25 17:45:26 +08:00
imccyu 1bddf5269a fix(storage): keep the json loadAll closed guard a rejection
Dropping async for the lint gate turned the closed-unit guard into a
synchronous throw, breaking the conformance clause that every post-close
operation rejects. Restore async with a justified require-await disable;
the guard's rejection semantics are what the shared suite pins.
2026-07-25 11:08:04 +08:00
imccyu 2e986ee1e3 style(storage,workspace): satisfy the repository lint gate
eslint --fix formatting sweep plus the manual residue: sync method
bodies drop async behind Promise-returning signatures (the sqlite unit
routes primitives through a settle() guard preserving the never-throws-
synchronously contract), catch callbacks type their reason as unknown,
loadAll's global slot is plain unknown (null semantics stay in JSDoc),
a non-null assertion becomes a narrowing, and unsafe any assignments in
tests gain explicit types. One justified eslint-disable for
prefer-promise-reject-errors follows the core/session precedent —
wrapping would discard the original StorageError code.
2026-07-25 11:08:04 +08:00
imccyu 0559e0207e chore(storage,workspace): gates — coverage, catalogs, bilingual note
- Per-file 100% coverage across the five new packages (invariant
  companion suites, failure-injection negatives, lifecycle and
  malformed-medium branches).
- Canonical README Model Experience / Known Limitations sections; new
  storage/ and workspace/ group READMEs; packages/README.md rows (budget
  ceiling raised 760 → 790 for the two new groups).
- Cordis catalog/type-link registrations, service-role classification,
  and regenerated catalogs/graphs for the new services and events.
- Agent Note: English body + i18n pairing record; design-sketch fences
  opted out of doc-typecheck as ignore-check.
- Two exactOptionalPropertyTypes/discriminant fixes in new tests.

doc-sync (24 gates), typecheck, hygiene, and the five-package suite
(92 tests) all pass.
2026-07-25 11:08:04 +08:00
imccyu 80b3b6d917 fix(storage,workspace): post-review hardening
Review findings applied across the group:
- storage hub: stale disposers no longer remove a successor registration;
  the package now default-exports the Storage service class per the
  service-package export shape.
- json backend: failed publishes roll back the authoritative memory state
  (a rejected write can no longer resurface via get() or ride the next
  publish); close() drains in-flight writes and blocks in-flight opens;
  double-open rejects as a plain caller error instead of malformed-medium.
- sqlite backend: loadAll builds records on a null prototype (__proto__
  keys round-trip instead of polluting), user_version is stamped only
  after the schema is fully created, and corrupt record JSON rejects as
  malformed-medium instead of a bare SyntaxError.
- domain form: writes persist before mutating authoritative memory or
  emitting; DomainChanged is a put/deleted discriminated union.
- workspace: attach/detach idempotence decided on the write chain (stale
  snapshots no longer short-circuit), create() requires a directory, and
  startup fails loud on duplicate stored paths.

Eleven regression tests pin the fixed behaviors.
2026-07-25 11:08:04 +08:00
imccyu 1529be6fd4 feat(storage): json backend — one human-readable file per unit
Atomic whole-file replacement (same-dir temp + fsync + rename + parent
fsync); the in-memory unit state is authoritative and the file is always
the current net state, pretty-printed. Missing files open as empty units
and materialize on first write; foreign or unparsable files reject with
malformed-medium, stored-version drift with version-mismatch.
2026-07-25 11:08:04 +08:00