Commit Graph
45 Commits
Author SHA1 Message Date
creatixchu 413ac14b16 refactor: move image offload projection into its plugin 2026-09-11 15:24:45 +08:00
creatixchu ec3560c42e Merge master 16bf879af2 into image offload branch 2026-09-11 13:40:19 +08:00
creatixchu b5e7fca4a5 feat(compaction): record image offload decisions without message replacement 2026-09-10 16:47:50 +08:00
_Kerman 37abcb74c5 chore(session): permit existing deprecated history reads 2026-09-09 12:33:53 +08:00
Tianyi Cui 1adcd8203a fix(token-meter): anchor usage after admitted prompt inputs
The loop appends step/start before system/message and the entered user
messages. Capturing nodes at step/start therefore omits inputs already
included in the provider's successful usage, then adds those inputs back
as a positive surface delta. Prompt replacement can also incorrectly add
or subtract the difference from the prior prompt on a completed call.

Snapshot the current priced surface immediately before assistant/message
commits. Keep provider output separate from the durable assistant node so
listener rewrites retain their signed delta. The invariant is zero delta
immediately after an unchanged successful output: provider usage already
includes every admitted prompt input. Later appends/replacements still
produce signed deltas, and low or absent usage keeps heuristic fallback.

Delete stepStart.nodes rather than adding prompt-specific corrections or
another request snapshot: the existing transactional surface fold already
contains the successful request inputs, including replacements made during
same-step retry recovery. Keep turn/step state and all overlap, mismatch,
and late-assistant lifecycle validation. Retry attempts are log-only and
request middleware changes configuration; injected messages remain queued
until admission. No loop, event format, projection, or retry policy changes.

Exercise the real loop with reported usage and initial, growing, shrinking,
and empty prompts; same-step failed attempt plus retry prompt replacement;
request middleware; eager observation and fresh seeded replay. The two
regressions fail before the fix with spurious deltas of +48 and +18 tokens.
Retain existing durable-output rewrite, route repricing, missing/low usage,
transactional failure, and lifecycle tests. Update README EN/ZH and the
existing system-prompt surface-node Agent Note, including pairing records.

Validation (dedicated worktree, no full unit suite):
- pnpm exec vitest run packages/llm/token-meter/tests packages/compaction/compaction-basic/tests/compaction-loop-repro.spec.ts --coverage --coverage.include='packages/llm/token-meter/src/index.ts'
  118 passed; exact changed runtime file 100% statements/branches/functions/lines.
- pnpm exec vitest run packages/core/agent-loop/tests/request-reconstruction.spec.ts packages/compaction/compaction-basic/tests/compaction-basic.spec.ts packages/compaction/compaction-basic/tests/loader-composition.spec.ts
  118 passed, including retry reconstruction and real Loader composition.
- pnpm run doc-sync: 33 gates passed.
- pnpm run test:docs: 15 gates passed.
- pnpm run lint: passed, 0 warnings/errors.
- git diff --cached --check: passed.

Baseline normalize.ts comment conflict marker is intentionally untouched.
2026-09-06 20:42:43 +08:00
Tianyi Cui ee956c720d refactor(session): represent the system prompt as surface node zero
Consolidate the representation-change PR and its rebase reconciliations into one baseline. Preserve the exact tree and keep the in-history feature in the dependent PR. Follow-up fixes remain separate.
2026-09-06 20:42:43 +08:00
Tianyi Cui 165cc31eb8 perf(llm,host): read embedded Assistant streams per compact record
Session format v2 embeds each attempt's compact stream in
assistant/message and assistant/attempt, but Host and client consumers
still expanded it into per-member TimedStreamChunk arrays and did
per-member work; expandAssistantStream materializes the full array before
find/toReversed/break can answer. Session Stats (the projection phase of
every Session open), the token meter's usage and provider-assembly folds,
the subagent output fold, and the Session Controller image lookup still
paid O(members) allocation and time per settlement.

The Chat and Trajectory definitions were already settled from
message.content on master; the remaining per-member folds stay.

dsh-llm now exports record-level readers (first token, visible content,
visible text, last raw chunk of a type, raw chunks of a type, joined
text, run-aware assembly, per-run first-token/first-visible times) that
scan the compact records once with early exit. Session Stats reads
assistantStreamFirstTokenTime, the token meter reads
lastAssistantStreamChunk(stream, 'usage') and assembles through
assembleAssistantStream, the subagent output fold appends
joinAssistantStreamText, and the Session Controller scans
assistantStreamChunks(stream, 'block-end').

expandAssistantStream is deliberately not memoized: retaining expansions
costs roughly ten times the compact stream for the Session's lifetime.
It remains the validating path at durable boundaries.

Synthetic 200-turn v0 migration benchmark, median of five: first-open
projection 28.0 ms -> 5.4 ms, first-open total 76.9 -> 50.0 ms, peak RSS
137.2 -> 94.9 MB; reopen projection 17.8 -> 5.6 ms; all phase budgets and
the 128 MB heap constraint keep passing.
2026-09-06 12:41:25 +08:00
creatixchu fb743759a7 Merge remote-tracking branch 'origin/master' into worktree/2984-generic-file-upload
# Conflicts:
#	apps/web/tests/scaffold.ts
#	docs/config-catalog.i18n.yaml
#	docs/config-catalog.md
#	docs/config-catalog.zh.md
#	docs/event-producer-consumer.i18n.yaml
#	docs/event-producer-consumer.md
#	docs/event-producer-consumer.zh.md
#	packages/api/session-controller/README.i18n.yaml
#	packages/api/session-controller/README.md
#	packages/api/session-controller/README.zh.md
#	packages/extensions/tool-cordis/src/api-catalog.ts
#	packages/llm/token-meter/src/index.ts
#	packages/session-query/session-log-export/README.i18n.yaml
#	packages/session-query/session-log-export/README.md
#	packages/session-query/session-log-export/README.zh.md
#	packages/session-query/session-log-export/src/archive.ts
#	scripts/package-dependency-policy.ts
2026-09-04 10:12:19 +08:00
creatixchu 3b9ac476fb Merge remote-tracking branch 'origin/master' into worktree/2984-generic-file-upload
# Conflicts:
#	docs/config-catalog.i18n.yaml
#	docs/config-catalog.md
#	docs/event-producer-consumer.i18n.yaml
#	docs/event-producer-consumer.md
#	docs/event-producer-consumer.zh.md
#	docs/module-graph.i18n.yaml
#	docs/module-graph.md
#	docs/module-graph.zh.md
#	docs/persistence-catalog.i18n.yaml
#	docs/persistence-catalog.md
#	docs/persistence-catalog.zh.md
#	packages/api/session-controller/src/commands.ts
#	packages/api/session-controller/src/index.ts
#	packages/api/session-controller/tests/client-contract.client.spec.ts
#	packages/api/session-controller/tests/session-pending-submissions.client.spec.ts
#	packages/client/ui-attachment/src/AttachmentRail.module.css
#	packages/client/ui-chat/src/client/chat/MessageItem.tsx
#	packages/client/ui-conversation/src/client/skeleton/InputBar.tsx
#	packages/client/ui-trajectory/README.i18n.yaml
#	packages/client/ui-trajectory/README.md
#	packages/client/ui-trajectory/README.zh.md
#	packages/extensions/cordis-client-runner/src/client/api-catalog.ts
#	packages/llm/token-meter/src/surface-fold.ts
#	packages/session-query/session-log-export/src/archive.ts
#	packages/session-query/session-log-export/tests/archive.host.spec.ts
2026-09-02 10:29:53 +08:00
Tianyi Cui f99b06eaed feat(session)!: embed assistant streams in format v2 2026-09-02 04:00:01 +08:00
Tianyi Cui 27bf1039db refactor(session)!: distinguish event seqs from log offsets 2026-09-01 20:36:00 +08:00
creatixchu 65d2015b09 feat(attachment): store and project generic files 2026-09-01 16:11:43 +08:00
_Kerman 6c63e708b8 Merge remote-tracking branch 'origin/master' into dshw/pr-deepseek-harness-deepseek-harness-2907
# Conflicts:
#	docs/config-catalog.i18n.yaml
#	docs/persistence-catalog.i18n.yaml
#	docs/subsystems/session.i18n.yaml
#	packages/api/session-controller/tests/transport.host.spec.ts
#	packages/bundle/headless/src/index.ts
#	packages/core/session/README.i18n.yaml
#	packages/schedule/schedule/README.i18n.yaml
#	packages/schedule/schedule/README.md
#	packages/schedule/schedule/README.zh.md
#	packages/session-query/tool-session-query/src/workspace-access.ts
#	packages/session/session-log-deepseek/src/index.ts
2026-08-31 10:56:53 +08:00
imccyu 9135a13a8b refactor(consumers): remove cross-package runtime relays 2026-08-30 02:29:52 +08:00
_Kerman 5660f44d29 perf(session): separate indexed and snapshot log reads 2026-08-28 13:25:58 +08:00
_Kerman a6c7c70d4f fix(session-projection): close review findings from the fold migration
- permission-presets: register the permissions unit synchronously before the
  existing-session sweep, so a remount reads folded knob state instead of
  treating every session as fresh; add a regression test for that path
- sandbox-policy/terminal-bash: mount the projection registry in the 5 pwsh
  terminal tests, the sdk-minimal bundle, and the e2b fixture composition;
  declare the new package dependency in both manifests
- plan-mode: restore .strict() on the plan unit state schema and drop the
  deleted foldPlanMode from the bilingual READMEs
- session-title/session-projection: sync bilingual READMEs to the mandatory
  projection seam and re-record translation pairing
- docs: turnBoundary reader contract, subagent schema comment, token-meter
  import comment, sandbox-policy module docstring
2026-08-26 14:13:42 +08:00
_Kerman 4e6a1f8d21 Merge origin/xtr/projection-per-session-cache into xtr/session-projection-migrations 2026-08-25 21:42:06 +08:00
creatixchu 1c065f3cd4 Merge remote-tracking branch 'origin/master' into worktree/2848-image-token-pressure
# Conflicts:
#	apps/cli/tests/profiles/acp/image-compaction.cordis.snapshot.yml
#	apps/cli/tests/profiles/acp/image-compaction.cordis.yml
#	apps/cli/tests/profiles/acp/tests/snapshots/image-compaction/input.json
#	apps/cli/tests/profiles/acp/tests/snapshots/image-compaction/session.jsonl
#	apps/cli/tests/profiles/acp/tests/snapshots/image-compaction/stdout.expected.jsonl
#	docs/config-catalog.i18n.yaml
#	docs/config-catalog.md
#	docs/config-catalog.zh.md
#	examples/acp-agent/tests/acp.snapshot.ts
#	packages/llm/token-meter/README.i18n.yaml
#	packages/llm/token-meter/README.md
#	packages/llm/token-meter/README.zh.md
#	packages/llm/token-meter/src/index.ts
#	packages/llm/token-meter/src/surface-fold.ts
2026-08-25 11:15:35 +08:00
creatixchu 42164508c8 feat(llm): 在 compaction 中按路由为图片请求压力计价
Closes #2848.

- dsh-llm 新增 LlmAdapter.imageRequestPricing 同步钩子与 LlmImageRequestPricing/LlmImageRequestPrice 词汇,ctx.llm 按路由解析
- llm-deepseek 用官方公布的 v4 视觉计算器逐句移植(14px patch、3:1 降采样、384 上限、最坏对齐 pad)实现该钩子,复现请求投影的最旧优先 offload 与像素预算缩放;纯几何 requestImageDimensions 上移到 dsh-attachment
- token-meter 表层 fold 存储与路由无关的节点事实,measure() 按生效 envelope 的路由为图片出现处定价;锚点存快照并按同一路由重定价;TokenSurfaceNode 同时携带路由价 tokens 与固定启发式 heuristicTokens
- compaction 触发、保留与选段读取同一套路由价,记录的 shadowedTokenCount 保持启发式以维持 O(1) 投影 fold 一致
- llm-replay 支持按模型的 imageRequestTokens 声明;新增 keyless 的 image-compaction ACP 快照场景端到端验证装配应用
2026-08-24 19:15:30 +08:00
Hypatia May 58a0e450b3 perf(token-meter): commit the surface fold in place through a plan/commit pair
foldSurfaceTokens rebuilt the priced surface on every surface event: an
append allocated [...nodes, node] and a replacement copied the whole
array before splicing, charging every well-formed event O(surface) for
an atomicity property only malformed events need. Benchmarks put the
copy at ~99.9% of an append's cost (100µs at a 50k-node surface vs
0.1µs for pricing) with O(S²) accumulation over a session, inside the
synchronous session/event publication path.

Split the fold into the session core's planSurfaceEvent/applySurfacePlan
shape: planSurfaceTokens performs every fallible step against the
read-only surface, commitSurfaceTokens applies the plan in place and is
infallible by construction. _foldEvent plans first, runs the remaining
fallible anchor validation, and only then commits, so retry identity is
preserved by ordering instead of by allocation. Appends drop to
amortized O(1) (100.3µs -> 1.9µs at 50k nodes); replacements keep their
O(surface) findIndex but stop paying the extra full copy (21µs -> 4.2µs).

A new regression test pins the one hazard this introduces: an event
whose surface plan is valid but whose later anchor validation throws
must leave the priced surface and running total uncommitted across
repeated failures.
2026-08-24 16:24:12 +08:00
_Kerman 1a72ae202a refactor(session): migrate host state reads to projections 2026-08-19 13:57:58 +08:00
Tianyi Cui a2d0f7f411 refactor: apply repository naming contract
Apply the accepted pre-release package, service, type, directory, and role renames as one repository-wide change.
2026-08-13 00:54:38 +08:00
imccyu ec601ca13d build(vendor): rescope the vendored Cordis packages into @deepseek-ai
Machine-produced by `pnpm run rescope-vendor --apply` plus the regeneration it
prints: `pnpm install` for the lockfile, `pnpm run gen-third-party-notices`,
`verify-translation-pairing --write` for the touched bilingual pairs,
`gen-doc-graphs`, and one typert snapshot whose ids embed character offsets.
`pnpm run rescope-vendor --check` verifies the result.

Renames nine vendored packages (cordis, cosmokit, schemastery and the six
@cordisjs plugins) and every reference that resolves them: manifest names and
dependency keys, module specifiers including declare-module merges, cordis.yml
plugin names, tsconfig paths, every Markdown fence, and `docs/` prose.
Directory names, upstream versions, and dependency ranges are unchanged, so
vendor/README.md still reads as an upstream snapshot; its manifest table gains
an upstream-name column so THIRD_PARTY_NOTICES keeps MIT attribution pointed
at each fork's origin.

The tutorial tier follows the rename end to end: its yaml fences named plugins
the Loader can no longer resolve, its `ts ignore-check` fences disagreed with
the compiled fences beside them, and its prose quoted both. The contracts that
told readers to keep upstream names — the root convention and the vendoring
cookbook's tree comment and manifest invariant — now say to rescope instead.

Two rules read `@deepseek-ai/` as "another workspace plugin": the client bundle
purity gate now names the vendored libraries a browser bundle inlines, and the
files where a bare `cordis` is an agent-preset id keep that product data.
2026-08-10 22:04:13 +08:00
Turtle a27efdef36 docs: make technical prose concrete 2026-08-10 16:34:20 +08:00
Turtle dda02250f5 docs: reserve seam for complete capabilities 2026-08-09 17:26:57 +08:00
Turtle 9704749b01 docs: replace vague provenance prose with recorded facts 2026-08-09 15:35:02 +08:00
imccyu a17eebcaca docs(metrics): clarify projection and latency contracts 2026-08-06 00:22:09 +08:00
Yichen Jiang e43b08d7b8 Merge remote-tracking branch 'origin/master' into feat/add-session-data-preview
# Conflicts:
#	docs/core-data-structures/session.i18n.yaml
#	packages/client/ui-conversation/README.i18n.yaml
#	packages/client/ui-conversation/README.md
#	packages/llm/token-meter/README.i18n.yaml
#	packages/llm/token-meter/README.md
#	packages/llm/token-meter/README.zh.md
#	packages/llm/token-meter/src/projection.ts
2026-08-05 17:11:16 +08:00
Yichen Jiang e62cbe12e4 refactor(token-meter): fold the surface once for the meter and the projection
`contextBreakdown.messageTokens` and `measure().surfaceTokens` answer the same
question in the same heuristic vocabulary, and the panel's composition rows are
only honest while they agree. Each owner carried its own copy of the positional
fold — same pricing, same `{seq, tokens}` node list, same replace-range lookup
and guard, differing only in mutable versus immutable application — so an edit
to either one would have moved the panel away from `measure()` with both sides
still green. The duplication gate caught the shared 62 tokens.

`src/surface-fold.ts` now owns `foldSurfaceTokens`: total, allocation-fresh,
returning the event's price, the next surface, and the signed total delta. The
service assigns that result where it used to prepare a commit closure, which
keeps its validate-before-mutate replay transaction intact — the fold throws
before any state is touched, so a malformed event still fails identically on
every retry. `_prepareSurfaceMutation` and `_estimateSurfaceEvent` go away with
it, and the projection's apply drops to one call.

Covers the identity with a session that appends and then compacts, asserting
the projection figure equals the service surface at each boundary; the test
fails when either side of the fold is perturbed.
2026-08-05 16:29:51 +08:00
Turtle 8af3babaea docs: rescan rebased documentation hierarchy 2026-08-05 16:16:57 +08:00
Yif 0073d6aaa1 feat(web): turn speed metrics and composer context meter
Assistant footers and the stats line gain TTFT/tok-per-second readings
folded from step timings; context occupancy moves off the stats line onto
a composer ring whose panel shows a heuristic system/tools/messages
breakdown from the new token-meter contextBreakdown session projection.
2026-08-05 13:54:46 +08:00
Hypatia May 7f30a7e30c Merge remote-tracking branch 'origin/master' into codex/status-bar-token-metrics
Conflict resolution notes:

- StatsLine: master redesigned the row into pipe-separated groups with LLM and
  tool wall times. Kept that design and swapped only the token accounting
  source, so counts and durations stay window-scoped while billing and context
  occupancy read the durable projections.
- Generated artifacts (cordis catalog, module graph, event producer/consumer,
  i18n pairing hashes) and web snapshots took master's side; they are
  regenerated and re-recorded after this merge.
- Web e2e goldens and details-panel/timeline assertions took master's side:
  that evolution is unrelated to this branch.
- ui-conversation package.json: kept master's devDependency ordering, re-adding
  only the token-meter entry this branch needs.
2026-07-30 13:56:39 +08:00
Hypatia May 4819210142 refactor(token-meter): make context occupancy durable projection state
Replace the transient `session/model-request` mux frame with ordinary durable
session state. Occupancy now rides two last-wins projection fields instead of a
non-replayable frame that needed removal tombstones and cross-stream fencing.

The frame was the only non-replayable class on the mux stream. Because host and
mux are independent SSE streams with no cross-stream order, a request emitted
before a removal could arrive after `host/session-removed`, and a legitimate
request for a new lifecycle reusing the same id could be fenced by a late
removal. Fixing that needed a lifecycle generation on every frame; the frame
itself was the problem.

Removed: the `session/model-request` frame and schema, the `agent/model-request`
core event, the ApiProxy measurement point, the client-side telemetry map and
removal tombstone, and the synthetic `cancelled` open error used to signal
reconnect through the error channel.

Added: `request/context`, a log-only session event recording the
registration-bound capacity of the route a request resolved to, appended beside
`request/header` from the lookup that already prepared the call and skipped when
the route is unchanged. Capacity stays out of `EpochHeader` because it is
adapter metadata about a route, not an input the request was built from, so it
must not join request reconstruction or header equality.

The `contextPressure` projection pairs the newest provider-reported prompt size
with the newest recorded capacity. The two are deliberately not one atomic
request observation: switching models can pair a fresh capacity with the prior
route's pressure until the next request reports usage. The figure is a
user-facing reference, and this matches how the TUI status line has always
computed occupancy.
2026-07-30 13:53:08 +08:00
Hypatia May e37cb23336 feat(token-meter): project durable token usage 2026-07-29 14:35:24 +08:00
Turtle 95a995968b refactor: migrate linting to Oxlint 2026-07-29 14:32:11 +08:00
_Kerman fbf87e660c refactor: identify and freeze messages at creation 2026-07-28 13:55:59 +08:00
_Kerman 009d113e0e refactor(agent): trim obsolete loop surfaces 2026-07-24 21:58:07 +08:00
Yichen Jiang 49a9bca9f6 fix: validate compaction config pairs 2026-07-21 14:49:57 +08:00
Yichen Jiang cfa180c127 Resolve compaction policy per routed model 2026-07-20 15:34:00 +08:00
Tianyi Cui 06ea8643ff Merge newest origin/master into token-meter-service 2026-07-17 22:49:54 +08:00
Hypatia May 06e0450b8d fix(token-meter): keep pressure anchors conservative 2026-07-16 15:26:30 +08:00
Hypatia May 6d96b3e4a8 refactor(token-meter): merge measurement snapshots (round 1) 2026-07-16 14:38:24 +08:00
Hypatia May 19a56ec542 fix(token-meter): reject stale config (round 2) 2026-07-16 13:23:20 +08:00
Hypatia May 5c243e8a8d refactor(token-meter): simplify singleton service (round 1) 2026-07-16 12:58:07 +08:00
Hypatia May f038780ff6 feat(llm): add replay token metering (PR2 round 1) 2026-07-15 14:47:29 +08:00