diff --git a/.agents/notes/implemented/architecture/2026-09-02-image-offload-watermark.i18n.yaml b/.agents/notes/implemented/architecture/2026-09-02-durable-image-offload.i18n.yaml similarity index 57% rename from .agents/notes/implemented/architecture/2026-09-02-image-offload-watermark.i18n.yaml rename to .agents/notes/implemented/architecture/2026-09-02-durable-image-offload.i18n.yaml index 2f0427cb1e..a2c691105e 100644 --- a/.agents/notes/implemented/architecture/2026-09-02-image-offload-watermark.i18n.yaml +++ b/.agents/notes/implemented/architecture/2026-09-02-durable-image-offload.i18n.yaml @@ -1,6 +1,6 @@ # Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: -# pnpm run verify-translation-pairing --write .agents/notes/implemented/architecture/2026-09-02-image-offload-watermark.md -2026-09-02-image-offload-watermark.md: 1197fd06950aa2caa54e124fdbdfddeaff07d250 -2026-09-02-image-offload-watermark.zh.md: 548e4a0f13a5f3a60a5636b0cfe00b6210b695f5 +# pnpm run verify-translation-pairing --write .agents/notes/implemented/architecture/2026-09-02-durable-image-offload.md +2026-09-02-durable-image-offload.md: db77e4743f1f81c7425b3dd63d32b36f161eafec +2026-09-02-durable-image-offload.zh.md: 9b0a424aff24acea130980a61095929b834a3145 diff --git a/.agents/notes/implemented/architecture/2026-09-02-durable-image-offload.md b/.agents/notes/implemented/architecture/2026-09-02-durable-image-offload.md new file mode 100644 index 0000000000..db77e4743f --- /dev/null +++ b/.agents/notes/implemented/architecture/2026-09-02-durable-image-offload.md @@ -0,0 +1,55 @@ +# Agent Note: Durable image offload by surface replacement + +Status: implemented + +English | [中文](2026-09-02-durable-image-offload.zh.md) + +## Problem + +Request-size image offload was recomputed from scratch on every request. Each route collected every image occurrence on the derived surface, oldest first, and once the accumulated bytes exceeded its budget it rounded the excess up to a whole removal quantum and replaced that many oldest occurrences with placeholder text, per the [unified image request pipeline](../feature/2026-08-20-unified-image-request-pipeline.md). Nothing remembered where the previous request stopped; the prefix was stable only because the arithmetic over an append-only history repeated itself. + +That stability failed wherever the arithmetic inputs moved. The [Files inline fallback](../../archived/bug-fix/2026-08-21-deepseek-files-inline-fallback.md) rebuilt a request under a 20 MiB inline budget with a 10 MiB quantum, offloading far more images for that one request, and the next request in file mode brought them back. The pi-ai route used a quantum of one byte, so its prefix moved on almost every request. Compaction lowered the total and returned previously offloaded images. A route switch moved every step boundary. Each move changed the model-visible prefix and invalidated the provider cache prefix. + +The same recomputation broke the repository invariant that model-visible input is reconstructable from the session log. Which representation was dispatched, the exact derived request-version byte lengths, and the route budgets and quanta were runtime or configuration facts that never entered the log; `request/header` records call config, system prompt, and tools only. Provider usage anchors token totals and cannot recover the image set, and the [route-priced estimate](../../archived/feature/2026-08-24-route-priced-image-request-pressure.md) documented that it did not reproduce the fallback budget. No consumer could pair a logged assistant response with the image set its request carried. + +## Decision + +The offload of an image occurrence is a durable surface fact recorded the way compaction records its reductions: a `surfaceOp: replace` node. + +**Marked copies on the surface.** `ImageBlock` gains `offloaded?: true`. An offloaded occurrence lives in a replacement event of the same type as the node that carried it (`user/message` or `tool/result`) whose content is a copy of the original with those blocks marked; `sourceEventSeqs` points at the replaced node. The session core, its event map, its derivation, and its validation are unchanged. `deriveMessages()` sends the marked block; serialization renders every marked block as `offloadedImageText` with the currently resolved access path and prepares only retained occurrences. + +**Only advances.** A replacement never reverts. When a budget grows, a route changes, or compaction lowers the total, the marked copies stay, so the model-visible prefix and the provider cache prefix move only forward. + +**Adapters project, never decide.** An image-capable route enforces an `LlmImageRequestBudget` (`representation`, `maxBytes`, `maxImages`, both quanta) over the retained occurrences' exact request-version bytes. When they still exceed the budget, in file mode, under the inline fallback's tighter budget, or under the pi-ai bound, the adapter fails the attempt with `IMAGE_OFFLOAD_REQUIRED` and `LlmFailure.offloadImages` naming how many more oldest occurrences must be offloaded, computed with the shared `requiredImageOffload()`. Nothing plans an offload before dispatch. + +**Recovery owned by `dsh-llm-retry`.** The retry executor already sits on the `agent/request-error` waterfall. Before any provider policy it handles `IMAGE_OFFLOAD_REQUIRED`: it walks the surface in model request order, marks the first `offloadImages` retained occurrences, appends one replacement per node that carried any of them, and returns the `retry` action without spending the provider retry budget or logging `llm/retry`. Assistant nodes carry model output, not input images, and are skipped. When nothing remains to offload it delegates and the failure reaches ordinary recovery. The loop re-runs the step over the replaced surface and logs the fresh `request/header` every surface replacement gets; the agent loop is unchanged. + +**Token accounting.** `priceImages` receives the surface's `ImageBlock`s and prices a marked one as its placeholder text; the DeepSeek and replay pricing no longer reproduce any offload arithmetic. The meter needs no new state: a replacement reprices the node like any other replacement. + +**Other consumers.** Compaction reconstructs each selected event through `deriveEventMessage()` and sees the marks. Resume, fork, and replay reproduce the surface from the log. Text-only routes keep their separate whole-history substitution. + +## Alternatives considered + +**Keep recomputing the offload point per request.** Stable only while the arithmetic inputs held still; the inline fallback, the pi-ai quantum, compaction, and route switches all moved the prefix, and no consumer could reconstruct a historical request's image set. + +**Record each request's projection outcome as a log-only event.** Restores reconstructability but not stability: the recorded outcome is not a decision input, so every oscillation still happens and the log merely documents it, with two sources of truth for the offloaded set that can disagree. + +**Log the full projected request body.** Everything except the offload decision is already derivable; repeating the history per request grows the log quadratically to record one position. + +**A log-only `image/offload` watermark event applied by session derivation.** An earlier revision recorded the offload point as a position (event seq plus block path) and had `Session.deriveMessages()` mark every occurrence at or before it. That gave the session a second mechanism for changing model-visible history next to `surfaceOp: replace`, with its own validation, fold, cache invalidation, and required-on-read event, and placed the logic in the core instead of on the plugin that decides. Issue #3041 asks that a permanent eviction use a surface-changing event rather than a request-projection event; because the offload never reverts, it is one. + +**Let each adapter append the replacement.** The adapter owns the budgets but not the session surface; a surface change appended below the loop lets two adapters define the surface differently. Adapters report the count they need instead. + +**Plan the offload before dispatch, in the loop or in a plugin.** The loop knows the exact prepared route before deriving each request, so planning there never spends a failed attempt; but it puts a route-specific policy into the one component every profile shares, changes the documented step order, and bypasses the `agent/request-error` waterfall that context-overflow compaction and retry already use for the same repair-and-retry pattern. A pre-step plugin keeps the loop unchanged but cannot see the step's own messages or the first request's route, so the failure path stays necessary, and the plan needs every route to declare its budget on its model info. Handling only the failure costs one attempt per quantum crossing (64 MiB in DeepSeek file mode, 20 MiB on pi-ai) and matches the planned direction: routes will stop checking sizes locally, send every image, and the provider will report what it cannot cache, which is exactly a failure naming an offload point. + +**Keep a transient extra offload for the inline fallback and exact-byte overflow.** Would have sent an unlogged projection in exactly the cases the invariant exists for; the failure-and-advance path costs one serialization attempt and keeps every dispatched request derivable from the log. + +## Consequences + +Offloaded images never return automatically when budgets grow, a larger route is selected, or compaction lowers the total; recovery is the read-only path in the placeholder, which the model uses deliberately. A Files outage or a temporary switch to a small-budget route offloads permanently; both are accepted for the same reason. Each offload copies the carrying node's message into the log; images are references, so the copy is the node's text plus block metadata. + +Every dispatched request's image set is determined by the log alone, across file mode, inline fallback, resume, fork, retry, and compaction, and the provider cache prefix no longer oscillates. The execution-world access path embedded in placeholder and handle text is still resolved at serialization time; that gap exists for retained images too and belongs to a separate decision about recording the execution-world mapping. The route-local byte checks are provisional: once the provider reports uncacheable images itself, `requiredImageOffload()` and the route budgets go away while the marked copies and the recovery branch stay as they are. + +## Testing + +`packages/llm/llm/tests/content.spec.ts` pins arbitrary-depth image traversal, the offload count including the 129-to-64 MiB quantum example, and placeholder projection. `packages/llm/llm-retry/tests/image-offload.spec.ts` pins the `IMAGE_OFFLOAD_REQUIRED` replace-and-retry path without a retry event, the untouched original node, request-order counts after an earlier surface replacement, and the exhausted case that delegates downstream. Adapter specs pin placeholder projection, prepared-only-retained reads, and the exact-byte failure with its count; `route-pricing.spec.ts` pins placeholder pricing for a marked replacement. The `inline-image-prompt` TypeScript SDK snapshot replays an authored `IMAGE_OFFLOAD_REQUIRED` attempt through the shipped profile and pins the replacement and the retried request. diff --git a/.agents/notes/implemented/architecture/2026-09-02-durable-image-offload.zh.md b/.agents/notes/implemented/architecture/2026-09-02-durable-image-offload.zh.md new file mode 100644 index 0000000000..9b0a424aff --- /dev/null +++ b/.agents/notes/implemented/architecture/2026-09-02-durable-image-offload.zh.md @@ -0,0 +1,55 @@ +# Agent Note: 以表层替换实现持久的图片 offload + +Status: implemented + +[English](2026-09-02-durable-image-offload.md) | 中文 + +## 问题 + +请求级图片 offload 过去在每次请求时都从头重算。每条路由按最老优先的顺序收集派生表层上的全部图片出现位置,累计字节一旦超过预算,就把超出部分向上取整到整个删除量子,把这么多最老的出现位置替换为占位文本,见[统一图片请求管线](../feature/2026-08-20-unified-image-request-pipeline.zh.md)。没有任何东西记住上一次请求停在哪里,前缀稳定只是因为对 append-only 历史做同样的算术会得到同样的结果。 + +算术的输入一动,这种稳定就失效。[Files 内联回退](../../archived/bug-fix/2026-08-21-deepseek-files-inline-fallback.md)会用 20 MiB 内联预算和 10 MiB 量子重建请求,那一次省略多得多的图片,下一次 file 模式的请求又把它们带回来。pi-ai 路由的量子是一个字节,前缀几乎每次请求都会移动。compaction 降低总量,让先前省略的图片回归。切换路由会移动每个台阶边界。每一次移动都改变模型可见前缀,并让 provider 的缓存前缀失效。 + +同样的重算也破坏了仓库不变量:模型可见输入必须能从 session log 重建。实际发出的表示方式、派生请求版本的精确字节长度、路由预算和量子都是运行时或配置事实,从不进入日志,`request/header` 只记录调用配置、系统提示词和工具。provider usage 只锚定 token 总量,恢复不了图片集合,[按路由定价的估计](../../archived/feature/2026-08-24-route-priced-image-request-pressure.md)也写明它不复现回退预算。没有任何消费方能把一条已记录的助手响应和它的请求携带的图片集合配对。 + +## 决定 + +一个图片出现位置的省略是一条持久的表层事实,记录方式和 compaction 记录它的缩减一样:一个 `surfaceOp: replace` 节点。 + +**表层上带标记的副本。** `ImageBlock` 新增 `offloaded?: true`。被省略的出现位置存在于一条与原承载节点同类型的替换事件里(`user/message` 或 `tool/result`),内容是原消息的副本,只把那些块打上标记;`sourceEventSeqs` 指向被替换的节点。session 核心、它的事件表、派生和校验都不变。`deriveMessages()` 原样发送带标记的块;序列化把每个带标记的块渲染为带当前已解析访问路径的 `offloadedImageText`,只准备保留的出现位置。 + +**只前进。** 替换永不回退。预算变大、路由切换或 compaction 降低总量时,带标记的副本留在原地,所以模型可见前缀和 provider 缓存前缀只向前移动。 + +**adapter 只投影,不决定。** 支持图片的路由按保留的出现位置的精确请求版本字节执行一个 `LlmImageRequestBudget`(`representation`、`maxBytes`、`maxImages` 与两个量子)。当它们仍超过预算,无论是 file 模式、内联回退更紧的预算还是 pi-ai 上限,adapter 都以 `IMAGE_OFFLOAD_REQUIRED` 让本次尝试失败,并在 `LlmFailure.offloadImages` 中用共享的 `requiredImageOffload()` 算出还需省略多少最老的出现位置。发送前没有任何规划。 + +**恢复归 `dsh-llm-retry`。** 重试执行器本来就在 `agent/request-error` waterfall 上。它在任何提供方策略之前处理 `IMAGE_OFFLOAD_REQUIRED`:按模型请求顺序遍历表层,给前 `offloadImages` 个保留的出现位置打标记,为每个承载了其中任一位置的节点追加一条替换,然后返回 `retry` 动作,不占提供方重试预算,也不记录 `llm/retry`。assistant 节点承载的是模型输出而不是输入图片,直接跳过。没有可省略的出现位置时向下游委托,失败进入普通恢复路径。循环在替换后的表层上重跑该 step,并像每次表层替换后一样记录新的 `request/header`;agent loop 不变。 + +**token 记账。** `priceImages` 接收表层的 `ImageBlock`,把带标记的按占位文本定价;DeepSeek 和 replay 的定价不再复现任何 offload 算术。meter 不需要新状态:一次替换和其他替换一样重新为该节点定价。 + +**其他消费方。** compaction 通过 `deriveEventMessage()` 重建每个选中事件,看得到标记。resume、fork 和重放从日志复现表层。纯文本路由保留各自的全历史替换。 + +## 考虑过的替代方案 + +**继续每次请求重算 offload 位置。** 只在算术输入不动时稳定,内联回退、pi-ai 量子、compaction 和路由切换都会移动前缀,且没有消费方能重建历史请求的图片集合。 + +**用 log-only 事件记录每次请求的投影结果。** 恢复了可重建性但没有稳定性:记录的结果不是决策输入,每一种抖动照旧发生,日志只是把它记下来,且省略集合有两个可能不一致的事实来源。 + +**记录完整的投影后请求体。** 除 offload 决定外一切都已可派生,为记录一个位置而每次请求重复整段历史会让日志平方级增长。 + +**由 session 派生应用的 log-only `image/offload` 水位事件。** 更早的一版把 offload 点记成一个位置(事件序号加块路径),由 `Session.deriveMessages()` 给位于它及之前的每个出现位置打标记。这让 session 在 `surfaceOp: replace` 之外多出第二种改变模型可见历史的机制,带着自己的校验、折叠、缓存失效和读取时必须识别的事件,并且把逻辑放进了核心而不是做决定的插件。issue #3041 要求永久淘汰使用改变表层的事件而不是请求投影事件;省略永不回退,它就是永久淘汰。 + +**让各个 adapter 自己追加替换。** adapter 拥有预算,但不拥有会话表层;在循环之下追加表层变更会让两个 adapter 对表层做出不同定义。adapter 改为上报它需要的数量。 + +**在发送前规划省略,无论放在循环里还是插件里。** 循环在派生每个请求之前就知道精确的已准备路由,在那里规划永远不会多花一次失败的尝试;但这会把一条路由专属的策略放进所有 profile 共用的那个组件,改变已记录的 step 顺序,还绕过了上下文溢出 compaction 和重试已经在用的同一套 `agent/request-error` waterfall。pre-step 插件不改循环,但看不到 step 自己的消息和第一个请求的路由,失败路径仍然必需,而且规划要求每条路由在模型信息上声明预算。只处理失败的代价是每越过一个量子多一次尝试(DeepSeek file 模式 64 MiB,pi-ai 20 MiB),并且和既定方向一致:路由将不再本地检查大小,全部发送,由 provider 报告无法缓存的部分,那正是一个指明省略点的失败。 + +**为内联回退和精确字节溢出保留临时的额外省略。** 恰好会在不变量所针对的场景发送未记录的投影;失败再推进的路径只多花一次序列化尝试,且让每个已发出请求都可由日志派生。 + +## 后果 + +预算变大、选中更大的路由或 compaction 降低总量时,被省略的图片不会自动回归;恢复手段是占位文本中的只读路径,模型需要时主动使用。一次 Files 故障或临时切到小预算路由会永久省略,两者出于同一理由被接受。每次省略会把承载节点的消息复制一份进日志;图片只是引用,复制的是该节点的文本和块元数据。 + +每个已发出请求的图片集合都仅由日志决定,覆盖 file 模式、内联回退、resume、fork、重试和 compaction,provider 缓存前缀不再来回变化。占位文本和句柄文本里嵌入的执行世界访问路径仍在序列化时解析;这个缺口对保留的图片同样存在,属于另一个关于记录执行世界映射的决定。路由本地的字节检查是过渡实现:provider 自己报告无法缓存的图片之后,`requiredImageOffload()` 和路由预算会被删掉,带标记的副本和恢复分支保持不变。 + +## 测试 + +`packages/llm/llm/tests/content.spec.ts` 钉住任意深度的图片遍历、包括 129 到 64 MiB 量子示例在内的省略计数,以及占位投影。`packages/llm/llm-retry/tests/image-offload.spec.ts` 钉住 `IMAGE_OFFLOAD_REQUIRED` 替换并重试且不追加重试事件的路径、原节点保持不变、更早一次表层替换之后的请求顺序计数,以及向下游委托的耗尽情况。adapter 测试钉住占位投影、只读取保留图片以及带数量的精确字节失败;`route-pricing.spec.ts` 钉住带标记替换的占位定价。`inline-image-prompt` TypeScript SDK 快照通过发布的 profile 重放一次手工编写的 `IMAGE_OFFLOAD_REQUIRED` 尝试,钉住替换和重试后的请求。 diff --git a/.agents/notes/implemented/architecture/2026-09-02-image-offload-watermark.md b/.agents/notes/implemented/architecture/2026-09-02-image-offload-watermark.md deleted file mode 100644 index 1197fd0695..0000000000 --- a/.agents/notes/implemented/architecture/2026-09-02-image-offload-watermark.md +++ /dev/null @@ -1,55 +0,0 @@ -# Agent Note: Durable image offload watermark - -Status: implemented - -English | [中文](2026-09-02-image-offload-watermark.zh.md) - -## Problem - -Request-size image offload was recomputed from scratch on every request. Each route collected every image occurrence on the derived surface, oldest first, and once the accumulated bytes exceeded its budget it rounded the excess up to a whole removal quantum and replaced that many oldest occurrences with placeholder text, per the [unified image request pipeline](../feature/2026-08-20-unified-image-request-pipeline.md). Nothing remembered where the previous request stopped; the prefix was stable only because the arithmetic over an append-only history repeated itself. - -That stability failed wherever the arithmetic inputs moved. The [Files inline fallback](../../archived/bug-fix/2026-08-21-deepseek-files-inline-fallback.md) rebuilt a request under a 20 MiB inline budget with a 10 MiB quantum, offloading far more images for that one request, and the next request in file mode brought them back. The pi-ai route used a quantum of one byte, so its prefix moved on almost every request. Compaction lowered the total and returned previously offloaded images. A route switch moved every step boundary. Each move changed the model-visible prefix and invalidated the provider cache prefix. - -The same recomputation broke the repository invariant that model-visible input is reconstructable from the session log. Which representation was dispatched, the exact derived request-version byte lengths, and the route budgets and quanta were runtime or configuration facts that never entered the log; `request/header` records call config, system prompt, and tools only. Provider usage anchors token totals and cannot recover the image set, and the [route-priced estimate](../../archived/feature/2026-08-24-route-priced-image-request-pressure.md) documented that it did not reproduce the fallback budget. No consumer could pair a logged assistant response with the image set its request carried. - -## Decision - -The offload point is a durable session fact: the core `image/offload` event records an image offload watermark that only advances, and the derived surface, every route, and the token meter read the offloaded set from it. - -**Event.** `image/offload` carries `{ turn, step, watermark }`, where `watermark` is an `ImageOccurrencePosition`: the seq of the event carrying the last offloaded occurrence and its complete nested block path inside that event's content. Positions order by seq, then by path, so a newer event always lies after an older one regardless of surface replacements. `Session.deriveMessages()` and `Session.deriveEventMessage()` mark every occurrence positioned at or before the watermark `offloaded: true` on the `ImageBlock`; the marked copies are frozen and the durable event content is untouched. `Session.append` and seeding reject a malformed or non-advancing watermark and require its path to identify an image on the current surface. They also reject a logged message containing the request-only `offloaded` marker. The event is required-on-read because it changes the derived surface; adding it does not change the log envelope structure. - -**Only advances.** The watermark never retreats when a budget grows, a route changes, or compaction lowers the total, so the model-visible prefix and the provider cache prefix move only forward. An occurrence below the watermark that compaction later shadows leaves the watermark valid, because the comparison is positional. - -**Decision owned by `dsh-llm-retry`, budgets enforced by routes.** An image-capable route enforces an `LlmImageRequestBudget` (`representation`, `maxBytes`, `maxImages`, both quanta) over its retained occurrences' exact request-version bytes and reports the overflow as a failure; nothing plans an offload before dispatch. `dsh-llm-retry`, the recovery executor already on the `agent/request-error` waterfall, handles `IMAGE_OFFLOAD_REQUIRED` before any provider policy: it walks the surface in model request order, converts the reported count to the greatest durable position in that request-order prefix (a surface replacement can put a newer event before older ones, so the advance may offload additional occurrences but always covers the prefix), appends `image/offload`, and returns the `retry` action without spending the provider retry budget or logging `llm/retry`. The loop re-runs the step over the derived surface; the agent loop is unchanged. - -**Adapters project, never decide.** Serialization renders every `offloaded` block as `offloadedImageText` with the currently resolved access path and prepares only retained occurrences. When the retained occurrences' exact request-version bytes still exceed the route budget, in file mode, under the inline fallback's tighter budget, or under the pi-ai bound, the adapter fails the attempt with `IMAGE_OFFLOAD_REQUIRED` and `LlmFailure.offloadImages` naming how many more oldest occurrences must be offloaded, computed with the shared `requiredImageOffload()`. When nothing remains to offload, `dsh-llm-retry` delegates and the failure reaches ordinary recovery. - -**Token accounting.** `priceImages` receives surface `ImageBlock`s and prices an `offloaded` one as its placeholder text; the DeepSeek and replay pricing no longer reproduce any offload arithmetic. The meter folds `image/offload` into its replay state, prices the current surface under the current watermark and each usage anchor under the watermark its request was derived with. Provider usage remains the anchor for completed requests. - -**Other consumers.** Compaction reconstructs each selected event through `Session.deriveEventMessage()`, which applies the same current watermark as `deriveMessages()`, before its direct `ctx.llm.stream` call. Resume, fork, and replay reproduce the surface from the log. Text-only routes keep their separate whole-history substitution. - -## Alternatives considered - -**Keep recomputing the offload point per request.** Stable only while the arithmetic inputs held still; the inline fallback, the pi-ai quantum, compaction, and route switches all moved the prefix, and no consumer could reconstruct a historical request's image set. - -**Record each request's projection outcome as a log-only event.** Restores reconstructability but not stability: the recorded outcome is not a decision input, so every oscillation still happens and the log merely documents it, with two sources of truth for the offloaded set that can disagree. - -**Log the full projected request body.** Everything except the offload decision is already derivable; repeating the history per request grows the log quadratically to record one position. - -**Identify the watermark by attachment id or by occurrence count.** Attachment ids repeat for duplicate attachments, so the position is ambiguous; counts shift when compaction prunes earlier occurrences. A seq plus block path is unambiguous and survives pruning. - -**Let each adapter append the event.** The adapter owns the budgets but not the session surface; a surface fact appended below the loop bypasses the session's derived history and lets two adapters define the surface differently. Adapters report the count they need instead. - -**Plan the offload before dispatch, in the loop or in a plugin.** The loop knows the exact prepared route before deriving each request, so planning there never spends a failed attempt; but it puts a route-specific policy into the one component every profile shares, changes the documented step order, and bypasses the `agent/request-error` waterfall that context-overflow compaction and retry already use for the same repair-and-retry pattern. A pre-step plugin keeps the loop unchanged but cannot see the step's own messages or the first request's route, so the failure path stays necessary, and the plan needs every route to declare its budget on its model info. Handling only the failure costs one attempt per quantum crossing (64 MiB in DeepSeek file mode, 20 MiB on pi-ai) and matches the planned direction: routes will stop checking sizes locally, send every image, and the provider will report what it cannot cache, which is exactly a failure naming an offload point. - -**Keep a transient extra offload for the inline fallback and exact-byte overflow.** Would have sent an unlogged projection in exactly the cases the invariant exists for; the failure-and-advance path costs one serialization attempt and keeps every dispatched request derivable from the log. - -## Consequences - -Offloaded images never return automatically when budgets grow, a larger route is selected, or compaction lowers the total; recovery is the read-only path in the placeholder, which the model uses deliberately. A Files outage or a temporary switch to a small-budget route advances the watermark permanently; both are accepted for the same reason. The route-local byte checks are provisional: once the provider reports uncacheable images itself, `requiredImageOffload()` and the route budgets go away while the event, the derivation, and the recovery branch stay as they are. - -Every dispatched request's image set is determined by the log alone, across file mode, inline fallback, resume, fork, retry, and compaction, and the provider cache prefix no longer oscillates. The execution-world access path embedded in placeholder and handle text is still resolved at serialization time; that gap exists for retained images too and belongs to a separate decision about recording the execution-world mapping. - -## Testing - -`packages/llm/llm/tests/content.spec.ts` pins arbitrary-depth image traversal, represented bytes, projection, and the prefix count, including the 129-to-64 MiB quantum example. `packages/core/session/tests/image-offload.spec.ts` pins append and seed validation, exact current-surface image paths, rejection of persisted derived markers, strict advance, arbitrary-depth marking, frozen copies, cache rebuild, and scratch-replay equality. `packages/llm/llm-retry/tests/image-offload.spec.ts` pins the `IMAGE_OFFLOAD_REQUIRED` advance-and-retry path without a retry event, request-order counts after a surface replacement, and the exhausted case that delegates downstream. Compaction tests pin watermark application to direct summarization input. Adapter specs pin placeholder projection, prepared-only-retained reads, and the exact-byte failure with its count; `route-pricing.spec.ts` pins watermark pricing. The `inline-image-prompt` TypeScript SDK snapshot replays an authored `IMAGE_OFFLOAD_REQUIRED` attempt through the shipped profile and pins the advance and the retried request. The Python SDK notification test pins lossless forwarding of the new event and its nested watermark. diff --git a/.agents/notes/implemented/architecture/2026-09-02-image-offload-watermark.zh.md b/.agents/notes/implemented/architecture/2026-09-02-image-offload-watermark.zh.md deleted file mode 100644 index 548e4a0f13..0000000000 --- a/.agents/notes/implemented/architecture/2026-09-02-image-offload-watermark.zh.md +++ /dev/null @@ -1,55 +0,0 @@ -# Agent Note: 持久的图片 offload 水位 - -Status: implemented - -[English](2026-09-02-image-offload-watermark.md) | 中文 - -## 问题 - -请求级图片 offload 过去在每次请求时都从头重算。每条路由按最老优先的顺序收集派生表层上的全部图片出现位置,累计字节一旦超过预算,就把超出部分向上取整到整个删除量子,把这么多最老的出现位置替换为占位文本,见[统一图片请求管线](../feature/2026-08-20-unified-image-request-pipeline.zh.md)。没有任何东西记住上一次请求停在哪里,前缀稳定只是因为对 append-only 历史做同样的算术会得到同样的结果。 - -算术的输入一动,这种稳定就失效。[Files 内联回退](../../archived/bug-fix/2026-08-21-deepseek-files-inline-fallback.md)会用 20 MiB 内联预算和 10 MiB 量子重建请求,那一次省略多得多的图片,下一次 file 模式的请求又把它们带回来。pi-ai 路由的量子是一个字节,前缀几乎每次请求都会移动。compaction 降低总量,让先前省略的图片回归。切换路由会移动每个台阶边界。每一次移动都改变模型可见前缀,并让 provider 的缓存前缀失效。 - -同样的重算也破坏了仓库不变量:模型可见输入必须能从 session log 重建。实际发出的表示方式、派生请求版本的精确字节长度、路由预算和量子都是运行时或配置事实,从不进入日志,`request/header` 只记录调用配置、系统提示词和工具。provider usage 只锚定 token 总量,恢复不了图片集合,[按路由定价的估计](../../archived/feature/2026-08-24-route-priced-image-request-pressure.md)也写明它不复现回退预算。没有任何消费方能把一条已记录的助手响应和它的请求携带的图片集合配对。 - -## 决定 - -offload 位置是持久的会话事实:核心事件 `image/offload` 记录一条只会前进的图片 offload 水位,派生表层、每条路由和 token meter 都从它读取省略集合。 - -**事件。** `image/offload` 携带 `{ turn, step, watermark }`,其中 `watermark` 是一个 `ImageOccurrencePosition`:承载最后一个被省略出现位置的事件序号,以及它在该事件内容中的完整嵌套块路径。位置先按序号再按路径排序,所以无论表层如何替换,较新的事件总在较老的之后。`Session.deriveMessages()` 与 `Session.deriveEventMessage()` 把位于水位及之前的每个出现位置在 `ImageBlock` 上标为 `offloaded: true`;标记后的副本被冻结,持久事件内容不受影响。`Session.append` 与 seed 拒绝畸形或没有严格前进的水位,并要求路径指向当前表层中的图片。它们还会拒绝包含请求专用 `offloaded` 标记的持久消息。该事件改变派生表层,因此读取时必须识别;增加该事件不改变日志信封结构。 - -**只前进。** 预算变大、路由切换或 compaction 降低总量时水位永不回退,所以模型可见前缀和 provider 缓存前缀只向前移动。水位之下的出现位置后来被 compaction 遮蔽也不影响水位有效性,因为比较是按位置进行的。 - -**决定权在 `dsh-llm-retry`,预算由路由执行。** 支持图片的路由按保留的出现位置的精确请求版本字节执行一个 `LlmImageRequestBudget`(`representation`、`maxBytes`、`maxImages` 与两个量子),超出时以失败上报;发送前没有任何规划。`dsh-llm-retry` 已经是 `agent/request-error` waterfall 上的恢复执行器,它在任何提供方策略之前处理 `IMAGE_OFFLOAD_REQUIRED`:按模型请求顺序遍历表层,把上报的数量换算成该请求序前缀中最大的持久位置(表层替换可能把较新的事件放到较老事件之前,推进因此可能额外省略一些出现位置,但一定覆盖该前缀),追加 `image/offload`,返回 `retry` 动作,不占提供方重试预算,也不记录 `llm/retry`。循环在派生表层上重跑该 step;agent loop 不变。 - -**adapter 只投影,不决定。** 序列化把每个 `offloaded` 块渲染为带当前已解析访问路径的 `offloadedImageText`,只准备保留的出现位置。当保留的出现位置按精确请求版本字节仍超过路由预算,无论是 file 模式、内联回退更紧的预算还是 pi-ai 上限,adapter 都以 `IMAGE_OFFLOAD_REQUIRED` 让本次尝试失败,并在 `LlmFailure.offloadImages` 中用共享的 `requiredImageOffload()` 算出还需省略多少最老的出现位置。没有可省略的出现位置时,`dsh-llm-retry` 向下游委托,失败进入普通恢复路径。 - -**token 记账。** `priceImages` 接收表层的 `ImageBlock`,把 `offloaded` 的按占位文本定价;DeepSeek 和 replay 的定价不再复现任何 offload 算术。meter 把 `image/offload` 折进其重放状态,按当前水位为当前表层定价,按每个 usage 锚点的请求派生时的水位为该锚点定价。已完成请求仍以 provider usage 为锚点。 - -**其他消费方。** compaction 在直接调用 `ctx.llm.stream` 之前,通过 `Session.deriveEventMessage()` 重建每个选中事件,因此会应用与 `deriveMessages()` 相同的当前水位。resume、fork 和重放从日志复现表层。纯文本路由保留各自的全历史替换。 - -## 考虑过的替代方案 - -**继续每次请求重算 offload 位置。** 只在算术输入不动时稳定,内联回退、pi-ai 量子、compaction 和路由切换都会移动前缀,且没有消费方能重建历史请求的图片集合。 - -**用 log-only 事件记录每次请求的投影结果。** 恢复了可重建性但没有稳定性:记录的结果不是决策输入,每一种抖动照旧发生,日志只是把它记下来,且省略集合有两个可能不一致的事实来源。 - -**记录完整的投影后请求体。** 除 offload 决定外一切都已可派生,为记录一个位置而每次请求重复整段历史会让日志平方级增长。 - -**用附件 id 或出现次数标识水位。** 附件 id 在重复附加时重复,位置因此含糊;compaction 剪掉更早的出现位置后计数会漂移。序号加块路径没有歧义,也不受剪枝影响。 - -**让各个 adapter 自己追加事件。** adapter 拥有预算,但不拥有会话表层;在循环之下追加表层事实绕过了会话的派生历史,也会让两个 adapter 对表层做出不同定义。adapter 改为上报它需要的数量。 - -**在发送前规划省略,无论放在循环里还是插件里。** 循环在派生每个请求之前就知道精确的已准备路由,在那里规划永远不会多花一次失败的尝试;但这会把一条路由专属的策略放进所有 profile 共用的那个组件,改变已记录的 step 顺序,还绕过了上下文溢出 compaction 和重试已经在用的同一套 `agent/request-error` waterfall。pre-step 插件不改循环,但看不到 step 自己的消息和第一个请求的路由,失败路径仍然必需,而且规划要求每条路由在模型信息上声明预算。只处理失败的代价是每越过一个量子多一次尝试(DeepSeek file 模式 64 MiB,pi-ai 20 MiB),并且和既定方向一致:路由将不再本地检查大小,全部发送,由 provider 报告无法缓存的部分,那正是一个指明省略点的失败。 - -**为内联回退和精确字节溢出保留临时的额外省略。** 恰好会在不变量所针对的场景发送未记录的投影;失败再推进的路径只多花一次序列化尝试,且让每个已发出请求都可由日志派生。 - -## 后果 - -预算变大、选中更大的路由或 compaction 降低总量时,被省略的图片不会自动回归;恢复手段是占位文本中的只读路径,模型需要时主动使用。一次 Files 故障或临时切到小预算路由会永久推进水位,两者出于同一理由被接受。路由本地的字节检查是过渡实现:provider 自己报告无法缓存的图片之后,`requiredImageOffload()` 和路由预算会被删掉,事件、派生和恢复分支保持不变。 - -每个已发出请求的图片集合仅由日志决定,覆盖 file 模式、内联回退、resume、fork、retry 和 compaction,provider 缓存前缀不再抖动。占位和句柄文本中嵌入的执行世界访问路径仍在序列化时解析,这个缺口对保留的图片同样存在,属于另一个关于记录执行世界映射的决定。 - -## 测试 - -`packages/llm/llm/tests/content.spec.ts` 钉住任意深度的图片遍历、表示字节、投影和前缀计数,包括 129 到 64 MiB 的量子示例。`packages/core/session/tests/image-offload.spec.ts` 钉住追加与 seed 校验、当前表层图片路径、拒绝持久化派生标记、严格推进、任意深度标记、冻结副本、缓存重建和从头重放的一致性。`packages/llm/llm-retry/tests/image-offload.spec.ts` 钉住 `IMAGE_OFFLOAD_REQUIRED` 推进并重试且不追加重试事件的路径、表层替换后的请求顺序计数,以及向下游委托的耗尽情况。compaction 测试钉住直接摘要输入中的水位应用。adapter 测试钉住占位投影、只读取保留图片以及带数量的精确字节失败;`route-pricing.spec.ts` 钉住水位定价。`inline-image-prompt` TypeScript SDK 快照通过发布的 profile 重放一次手工编写的 `IMAGE_OFFLOAD_REQUIRED` 尝试,钉住推进和重试后的请求。Python SDK 通知测试钉住新事件及其嵌套水位的无损转发。 diff --git a/docs/persistence-catalog.i18n.yaml b/docs/persistence-catalog.i18n.yaml index fab47e0319..6775871bcd 100644 --- a/docs/persistence-catalog.i18n.yaml +++ b/docs/persistence-catalog.i18n.yaml @@ -2,5 +2,5 @@ # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: # pnpm run verify-translation-pairing --write docs/persistence-catalog.md -persistence-catalog.md: f03164165bb23acf87948ae36960b00e0cf84559 -persistence-catalog.zh.md: ba2e6aa7ce2953b04019a34ac7365c9c77ad52e6 +persistence-catalog.md: 8d6f66dc949b6d41530eb387534689dc66f1e99f +persistence-catalog.zh.md: 06d07ceab984ad20ee6637227d54fafbbb58bdd4 diff --git a/docs/persistence-catalog.md b/docs/persistence-catalog.md index f03164165b..8d6f66dc94 100644 --- a/docs/persistence-catalog.md +++ b/docs/persistence-catalog.md @@ -88,7 +88,7 @@ export type SessionEvent = { }[T] ``` -Sources: [`packages/core/session/src/types.ts:410`](../packages/core/session/src/types.ts) · [`packages/core/session/src/types.ts:418`](../packages/core/session/src/types.ts) · [`packages/core/session/src/types.ts:447`](../packages/core/session/src/types.ts) · [`packages/core/session/src/types.ts:478`](../packages/core/session/src/types.ts) +Sources: [`packages/core/session/src/types.ts:385`](../packages/core/session/src/types.ts) · [`packages/core/session/src/types.ts:393`](../packages/core/session/src/types.ts) · [`packages/core/session/src/types.ts:422`](../packages/core/session/src/types.ts) · [`packages/core/session/src/types.ts:453`](../packages/core/session/src/types.ts) ## Events @@ -215,7 +215,7 @@ Source: [`packages/interaction/user-approval/src/index.ts:33`](../packages/inter 'assistant/attempt': { turn: number; step: number; stream: AssistantStreamRecord[] } ``` -Source: [`packages/core/session/src/types.ts:333`](../packages/core/session/src/types.ts) +Source: [`packages/core/session/src/types.ts:319`](../packages/core/session/src/types.ts) @@ -245,7 +245,7 @@ Source: [`packages/core/session/src/types.ts:333`](../packages/core/session/src/ Types: [TokenUsage](subsystems/llm-streaming.md) -Source: [`packages/core/session/src/types.ts:319`](../packages/core/session/src/types.ts) +Source: [`packages/core/session/src/types.ts:305`](../packages/core/session/src/types.ts) ### `command/*` @@ -504,28 +504,6 @@ Source: [`packages/hooks/hook-protocol/src/types.ts:19`](../packages/hooks/hook- Source: [`packages/hooks/hook-protocol/src/types.ts:31`](../packages/hooks/hook-protocol/src/types.ts) -### `image/*` - - - -#### `image/offload` — log-only - -```ts persistence-catalog -/** - * Advances the durable image offload watermark before a request in step - * `step` of turn `turn` is dispatched. Every image occurrence positioned at - * or before `watermark` derives with `offloaded: true`, so each route sends - * its placeholder text instead of the image; occurrences after it stay - * retained. The watermark only advances: each event names a position - * strictly after the previous one, and no later budget, route change, or - * compaction moves it back. It is a log-only event that changes the derived - * surface, so a build that does not know the type refuses the log. - */ -'image/offload': { turn: number; step: number; watermark: ImageOccurrencePosition } -``` - -Source: [`packages/core/session/src/types.ts:383`](../packages/core/session/src/types.ts) - ### `llm/*` @@ -615,7 +593,7 @@ Source: [`packages/plan/plan-mode/src/index.ts:46`](../packages/plan/plan-mode/s 'request/context': RequestContext ``` -Source: [`packages/core/session/src/types.ts:372`](../packages/core/session/src/types.ts) +Source: [`packages/core/session/src/types.ts:358`](../packages/core/session/src/types.ts) @@ -634,7 +612,7 @@ Source: [`packages/core/session/src/types.ts:372`](../packages/core/session/src/ } ``` -Source: [`packages/core/session/src/types.ts:362`](../packages/core/session/src/types.ts) +Source: [`packages/core/session/src/types.ts:348`](../packages/core/session/src/types.ts) ### `sandbox/*` @@ -709,7 +687,7 @@ Source: [`packages/schedule/schedule/src/types.ts:219`](../packages/schedule/sch 'session/end-seed': { inherited?: true } ``` -Source: [`packages/core/session/src/types.ts:406`](../packages/core/session/src/types.ts) +Source: [`packages/core/session/src/types.ts:381`](../packages/core/session/src/types.ts) @@ -771,7 +749,7 @@ Source: [`packages/session/session-log-deepseek/src/types.ts:59`](../packages/se 'step/end': { turn: number; step: number } ``` -Source: [`packages/core/session/src/types.ts:300`](../packages/core/session/src/types.ts) +Source: [`packages/core/session/src/types.ts:286`](../packages/core/session/src/types.ts) @@ -782,7 +760,7 @@ Source: [`packages/core/session/src/types.ts:300`](../packages/core/session/src/ 'step/start': { turn: number; step: number } ``` -Source: [`packages/core/session/src/types.ts:298`](../packages/core/session/src/types.ts) +Source: [`packages/core/session/src/types.ts:284`](../packages/core/session/src/types.ts) ### `subagent/*` @@ -913,7 +891,7 @@ Source: [`packages/todo/tool-todo/src/types.ts:31`](../packages/todo/tool-todo/s Types: [ToolCallId](subsystems/core.md) -Source: [`packages/core/session/src/types.ts:339`](../packages/core/session/src/types.ts) +Source: [`packages/core/session/src/types.ts:325`](../packages/core/session/src/types.ts) @@ -988,7 +966,7 @@ Source: [`packages/core/tools/src/types.ts:40`](../packages/core/tools/src/types } ``` -Source: [`packages/core/session/src/types.ts:351`](../packages/core/session/src/types.ts) +Source: [`packages/core/session/src/types.ts:337`](../packages/core/session/src/types.ts) ### `tool-workflow/*` @@ -1068,7 +1046,7 @@ Source: [`packages/workflow/tool-workflow/src/types.ts:47`](../packages/workflow Types: [TurnEndReason](subsystems/session.md) -Source: [`packages/core/session/src/types.ts:296`](../packages/core/session/src/types.ts) +Source: [`packages/core/session/src/types.ts:282`](../packages/core/session/src/types.ts) @@ -1084,7 +1062,7 @@ Source: [`packages/core/session/src/types.ts:296`](../packages/core/session/src/ 'turn/start': { turn: number } ``` -Source: [`packages/core/session/src/types.ts:287`](../packages/core/session/src/types.ts) +Source: [`packages/core/session/src/types.ts:273`](../packages/core/session/src/types.ts) ### `user/*` @@ -1103,7 +1081,7 @@ Source: [`packages/core/session/src/types.ts:287`](../packages/core/session/src/ 'user/message': UserMessage ``` -Source: [`packages/core/session/src/types.ts:308`](../packages/core/session/src/types.ts) +Source: [`packages/core/session/src/types.ts:294`](../packages/core/session/src/types.ts) ### `web/*` diff --git a/docs/persistence-catalog.zh.md b/docs/persistence-catalog.zh.md index ba2e6aa7ce..06d07ceab9 100644 --- a/docs/persistence-catalog.zh.md +++ b/docs/persistence-catalog.zh.md @@ -506,28 +506,6 @@ export type SessionEvent = { 来源:[`packages/hooks/hook-protocol/src/types.ts:31`](../packages/hooks/hook-protocol/src/types.ts) -### `image/*` - - - -#### `image/offload` — log-only - -```ts persistence-catalog -/** - * Advances the durable image offload watermark before a request in step - * `step` of turn `turn` is dispatched. Every image occurrence positioned at - * or before `watermark` derives with `offloaded: true`, so each route sends - * its placeholder text instead of the image; occurrences after it stay - * retained. The watermark only advances: each event names a position - * strictly after the previous one, and no later budget, route change, or - * compaction moves it back. It is a log-only event that changes the derived - * surface, so a build that does not know the type refuses the log. - */ -'image/offload': { turn: number; step: number; watermark: ImageOccurrencePosition } -``` - -来源:[`packages/core/session/src/types.ts:366`](../packages/core/session/src/types.ts) - ### `llm/*` diff --git a/docs/subsystems/core.i18n.yaml b/docs/subsystems/core.i18n.yaml index d92af9b9b1..25c3042d2f 100644 --- a/docs/subsystems/core.i18n.yaml +++ b/docs/subsystems/core.i18n.yaml @@ -2,5 +2,5 @@ # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: # pnpm run verify-translation-pairing --write docs/subsystems/core.md -core.md: 0f64cd32eb74ccd17e74312d11ed4bb236098f7a -core.zh.md: 245b6c0f08f9902adcf56d9221406692dd87c346 +core.md: 4af3a22478f324655dea1b132f0c8a8528f4b883 +core.zh.md: 7359e5aaa75e9f750d44f1386acab388665e1e1e diff --git a/docs/subsystems/core.md b/docs/subsystems/core.md index 0f64cd32eb..4af3a22478 100644 --- a/docs/subsystems/core.md +++ b/docs/subsystems/core.md @@ -347,7 +347,7 @@ type SessionStartSource = 'startup' | 'resume' | 'clear' | 'compact' A `Session` is an **append-only log** of typed `SessionEvent`s — the single source of truth. The LLM message history is *derived* from the log (`deriveMessages()`), not stored separately. Every entry carries a monotonic `seq`, a `time`, and a `type`-discriminated `data` payload; surface variants may also list cited earlier events in `sourceEventSeqs` and carry a `surfaceOp`. -The `SessionEvent` envelope's exact conditional fields, the thirteen core event variants (`turn/start`, `turn/end`, `step/start`, `step/end`, `user/message`, `assistant/message`, `assistant/attempt`, `tool/call`, `tool/result`, `request/header`, `request/context`, `image/offload`, `session/end-seed`), the `deriveMessages()` projection rules, the `TurnEndReason` reasons, and the execution-enclosure and standalone-event rules are on **[session.md](session.md)**. How the log is made durable — the `SessionPersistence` interface, JSONL provider, `session/flush` checkpoint, crash recovery, and `SessionHeader` — is on **[persistence.md](persistence.md)**. +The `SessionEvent` envelope's exact conditional fields, the twelve core event variants (`turn/start`, `turn/end`, `step/start`, `step/end`, `user/message`, `assistant/message`, `assistant/attempt`, `tool/call`, `tool/result`, `request/header`, `request/context`, `session/end-seed`), the `deriveMessages()` projection rules, the `TurnEndReason` reasons, and the execution-enclosure and standalone-event rules are on **[session.md](session.md)**. How the log is made durable — the `SessionPersistence` interface, JSONL provider, `session/flush` checkpoint, crash recovery, and `SessionHeader` — is on **[persistence.md](persistence.md)**. ## `ToolDefinition` diff --git a/docs/subsystems/core.zh.md b/docs/subsystems/core.zh.md index 245b6c0f08..7359e5aaa7 100644 --- a/docs/subsystems/core.zh.md +++ b/docs/subsystems/core.zh.md @@ -355,7 +355,7 @@ type SessionStartSource = 'startup' | 'resume' | 'clear' | 'compact' `Session` 是一份类型化 `SessionEvent` 的**仅追加日志**——唯一的真源。LLM 消息历史从日志*派生*(`deriveMessages()`),而非单独存储。每个条目携带单调的 `seq`、`time` 与按 `type` 判别的 `data` payload;surface 变体还可以在 `sourceEventSeqs` 中列出被引用的较早事件,并携带 `surfaceOp`。 -`SessionEvent` 信封的确切条件字段、十三种核心事件变体(`turn/start`、`turn/end`、`step/start`、`step/end`、`user/message`、`assistant/message`、`assistant/attempt`、`tool/call`、`tool/result`、`request/header`、`request/context`、`image/offload`、`session/end-seed`)、`deriveMessages()` 投影规则、`TurnEndReason` 原因以及执行封闭和独立事件规则都在 **[session.md](session.zh.md)** 中。日志如何持久化——`SessionPersistence` 接口、JSONL provider、`session/flush` 检查点、崩溃恢复与 `SessionHeader`——则在 **[persistence.md](persistence.zh.md)** 中。 +`SessionEvent` 信封的确切条件字段、十二种核心事件变体(`turn/start`、`turn/end`、`step/start`、`step/end`、`user/message`、`assistant/message`、`assistant/attempt`、`tool/call`、`tool/result`、`request/header`、`request/context`、`session/end-seed`)、`deriveMessages()` 投影规则、`TurnEndReason` 原因以及执行封闭和独立事件规则都在 **[session.md](session.zh.md)** 中。日志如何持久化——`SessionPersistence` 接口、JSONL provider、`session/flush` 检查点、崩溃恢复与 `SessionHeader`——则在 **[persistence.md](persistence.zh.md)** 中。 ## `ToolDefinition` diff --git a/docs/subsystems/llm-streaming.i18n.yaml b/docs/subsystems/llm-streaming.i18n.yaml index d06acdc2c0..4f7cf608c3 100644 --- a/docs/subsystems/llm-streaming.i18n.yaml +++ b/docs/subsystems/llm-streaming.i18n.yaml @@ -2,5 +2,5 @@ # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: # pnpm run verify-translation-pairing --write docs/subsystems/llm-streaming.md -llm-streaming.md: f2c8c09779e3e8e808dc62f9861230b7ac69cb87 -llm-streaming.zh.md: ddf7ae1dd5a514b58178521c526b0de1d6c165e6 +llm-streaming.md: 7eb1610febca36973f2892f2e15ad8af52a62b44 +llm-streaming.zh.md: b1cab08a1193ab8f70fcf570dcc57a573b5c96cd diff --git a/docs/subsystems/llm-streaming.md b/docs/subsystems/llm-streaming.md index f2c8c09779..7eb1610feb 100644 --- a/docs/subsystems/llm-streaming.md +++ b/docs/subsystems/llm-streaming.md @@ -247,8 +247,9 @@ interface LlmFailure { /** * With code `IMAGE_OFFLOAD_REQUIRED`: how many more of the oldest retained * image occurrences the route needs offloaded before the same request fits - * its exact byte accounting. `dsh-llm-retry` advances the session's durable - * `image/offload` watermark by this count and retries the step. + * its exact byte accounting. `dsh-llm-retry` replaces the surface nodes + * carrying that many oldest retained occurrences with copies marked + * `offloaded` and retries the step. */ readonly offloadImages?: number } diff --git a/docs/subsystems/llm-streaming.zh.md b/docs/subsystems/llm-streaming.zh.md index ddf7ae1dd5..b1cab08a11 100644 --- a/docs/subsystems/llm-streaming.zh.md +++ b/docs/subsystems/llm-streaming.zh.md @@ -249,8 +249,9 @@ interface LlmFailure { /** * With code `IMAGE_OFFLOAD_REQUIRED`: how many more of the oldest retained * image occurrences the route needs offloaded before the same request fits - * its exact byte accounting. `dsh-llm-retry` advances the session's durable - * `image/offload` watermark by this count and retries the step. + * its exact byte accounting. `dsh-llm-retry` replaces the surface nodes + * carrying that many oldest retained occurrences with copies marked + * `offloaded` and retries the step. */ readonly offloadImages?: number } diff --git a/docs/subsystems/session.i18n.yaml b/docs/subsystems/session.i18n.yaml index 6477c348d7..592b0599b3 100644 --- a/docs/subsystems/session.i18n.yaml +++ b/docs/subsystems/session.i18n.yaml @@ -2,5 +2,5 @@ # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: # pnpm run verify-translation-pairing --write docs/subsystems/session.md -session.md: df91b16dcd2125c33ce92d62079f7afc84cc47e5 -session.zh.md: f107b1f4d0eac7c49206b7018f44d0653091a6af +session.md: 90051ca8668173daab4a003cf574f5ec64073223 +session.zh.md: 5de78f22697f71ec8c15824d36b4b93f31fc6a94 diff --git a/docs/subsystems/session.md b/docs/subsystems/session.md index df91b16dcd..90051ca866 100644 --- a/docs/subsystems/session.md +++ b/docs/subsystems/session.md @@ -117,17 +117,6 @@ interface SessionEventMap { * changes. It does not participate in request reconstruction or header equality. */ 'request/context': RequestContext - /** - * Advances the durable image offload watermark before a request in step - * `step` of turn `turn` is dispatched. Every image occurrence positioned at - * or before `watermark` derives with `offloaded: true`, so each route sends - * its placeholder text instead of the image; occurrences after it stay - * retained. The watermark only advances: each event names a position - * strictly after the previous one, and no later budget, route change, or - * compaction moves it back. It is a log-only event that changes the derived - * surface, so a build that does not know the type refuses the log. - */ - 'image/offload': { turn: number; step: number; watermark: ImageOccurrencePosition } /** * Marks the end of a constructor seed. Events before it have smaller seq * values and came from the seed (resume, fork, or replay); this lifecycle @@ -198,26 +187,6 @@ interface RequestContext { } ``` -### The image offload event: `image/offload` - -`dsh-llm-retry` appends `image/offload` on the `agent/request-error` waterfall when an adapter fails an attempt with `IMAGE_OFFLOAD_REQUIRED`, before the step retries. Its `watermark` names the last offloaded occurrence; `deriveMessages()` and the Session instance's `deriveEventMessage()` mark every occurrence positioned at or before it `offloaded: true`, and each route renders those marks as placeholder text. `Session.append` and seeding require the watermark to identify an image on the current surface, reject malformed and non-advancing positions, and reject request-only `offloaded` markers in durable messages. Like `request/header`, it is not a `SurfaceEventType`; unlike it, it changes the derived surface, so it stays required-on-read ([decision](../../.agents/notes/implemented/architecture/2026-09-02-image-offload-watermark.md)). - -```ts type-equiv -/** - * Durable position of one image occurrence on the model-visible surface: the - * seq of the event carrying it and the block path inside that event's - * content. Every nested tool-result contributes another path index. Positions - * order by seq, then by path, so a newer event always lies after an older one - * regardless of surface replacements. - */ -interface ImageOccurrencePosition { - /** Seq of the surface message event carrying the occurrence. */ - seq: SessionSeq - /** Block index path inside that event's message content. */ - path: number[] -} -``` - ## `SessionEvent` — one log entry A proper discriminated union over `type` (not independent `type`/`data` unions), so `switch (event.type)` narrows `event.data` without casts. `seq` is the monotonic position in the log (`seq = log.length`); `time` is epoch ms. @@ -588,9 +557,7 @@ declare class Session { * * CACHED: each surface node is projected exactly once, when first seen — a * call costs O(new nodes), and a surface rewrite (a `replace`; - * {@link SessionSurface.replaceGeneration}) or an `image/offload` advance - * rebuilds, and image occurrences at or before the watermark derive with - * `offloaded: true` ({@link markImageOffload}). The returned array is + * {@link SessionSurface.replaceGeneration}) rebuilds. The returned array is * a fresh snapshot per call (later appends never grow an array a caller * already holds); the `Message` objects in it are SHARED and **deep-frozen**. * Their content reuses the already frozen durable event data, so the cache @@ -599,9 +566,8 @@ declare class Session { */ deriveMessages(): Message[]; /** - * Derive one event under the session's current image offload watermark. - * Direct model consumers use this instance method so their per-event - * reconstruction matches {@link deriveMessages}. + * Instance face of the pure per-node `deriveEventMessage` export from + * `surface.ts`. * @param event - the event to project. * @returns the derived message, or null when the event produces none. */ @@ -611,7 +577,7 @@ declare class Session { ## Derived history: `deriveMessages()` and `deriveEventMessage()` -`Session.deriveMessages()` projects the event log into the `Message[]` the model sees — cached (each surface node projected once, when first seen; a surface rewrite or image watermark advance rebuilds) and frozen (a fresh array per call over shared, deep-frozen messages, so mutating logged history through a projection is unrepresentable). The pure `deriveEventMessage(event)` export applies the message-type rules without session state. The Session instance method of the same name additionally applies the current image watermark and is the per-event API for direct model consumers such as compaction. The projection rules: +`Session.deriveMessages()` projects the event log into the `Message[]` the model sees — cached (each surface node projected once, when first seen; a surface rewrite rebuilds) and frozen (a fresh array per call over shared, deep-frozen messages, so mutating logged history through a projection is unrepresentable). `deriveEventMessage(event)` is the per-node pure function the fold applies — public so external reconstructors and the dev invariant project a log prefix with exactly the same rules and cannot disagree with the cache. The projection rules: - `user/message` → a user message carrying exact `content`; an optional envelope remains log-only display metadata. - `assistant/message` → an assistant message with the provider and model that produced it plus optional adapter-private replay state. Its embedded compact stream is replay, usage, and UI evidence rather than a second message. An **empty-content** `assistant/message` is also skipped — a max-tokens step cut off with no content still records an `assistant/message` to hold its stream, usage, provider, and model, but a content-less assistant turn must not enter the provider transcript. diff --git a/docs/subsystems/session.zh.md b/docs/subsystems/session.zh.md index f107b1f4d0..5de78f2269 100644 --- a/docs/subsystems/session.zh.md +++ b/docs/subsystems/session.zh.md @@ -117,17 +117,6 @@ interface SessionEventMap { * changes. It does not participate in request reconstruction or header equality. */ 'request/context': RequestContext - /** - * Advances the durable image offload watermark before a request in step - * `step` of turn `turn` is dispatched. Every image occurrence positioned at - * or before `watermark` derives with `offloaded: true`, so each route sends - * its placeholder text instead of the image; occurrences after it stay - * retained. The watermark only advances: each event names a position - * strictly after the previous one, and no later budget, route change, or - * compaction moves it back. It is a log-only event that changes the derived - * surface, so a build that does not know the type refuses the log. - */ - 'image/offload': { turn: number; step: number; watermark: ImageOccurrencePosition } /** * Marks the end of a constructor seed. Events before it have smaller seq * values and came from the seed (resume, fork, or replay); this lifecycle @@ -198,26 +187,6 @@ interface RequestContext { } ``` -### 图片 offload 事件:`image/offload` - -adapter 以 `IMAGE_OFFLOAD_REQUIRED` 让一次尝试失败时,`dsh-llm-retry` 在 `agent/request-error` waterfall 上、step 重试前追加 `image/offload`。其 `watermark` 指向最后一个被省略的出现位置;`deriveMessages()` 与 Session 实例的 `deriveEventMessage()` 把位于它及之前的每个出现位置标为 `offloaded: true`,每条路由把这些标记渲染为占位文本。`Session.append` 与 seed 要求水位指向当前表层的图片,拒绝畸形和没有严格前进的位置,并拒绝持久消息中的请求专用 `offloaded` 标记。它和 `request/header` 一样不是 `SurfaceEventType`;与之不同的是它改变派生表层,因此读取时必须识别([决定](../../.agents/notes/implemented/architecture/2026-09-02-image-offload-watermark.zh.md))。 - -```ts type-equiv -/** - * Durable position of one image occurrence on the model-visible surface: the - * seq of the event carrying it and the block path inside that event's - * content. Every nested tool-result contributes another path index. Positions - * order by seq, then by path, so a newer event always lies after an older one - * regardless of surface replacements. - */ -interface ImageOccurrencePosition { - /** Seq of the surface message event carrying the occurrence. */ - seq: SessionSeq - /** Block index path inside that event's message content. */ - path: number[] -} -``` - ## `SessionEvent`:一条日志条目 基于 `type` 的真正可辨识联合(而非独立的 `type`/`data` 联合),因此 `switch (event.type)` 能直接收窄 `event.data`,无需类型断言。`seq` 是日志中的单调递增位置(`seq = log.length`);`time` 为 epoch 毫秒。 @@ -590,9 +559,7 @@ declare class Session { * * CACHED: each surface node is projected exactly once, when first seen — a * call costs O(new nodes), and a surface rewrite (a `replace`; - * {@link SessionSurface.replaceGeneration}) or an `image/offload` advance - * rebuilds, and image occurrences at or before the watermark derive with - * `offloaded: true` ({@link markImageOffload}). The returned array is + * {@link SessionSurface.replaceGeneration}) rebuilds. The returned array is * a fresh snapshot per call (later appends never grow an array a caller * already holds); the `Message` objects in it are SHARED and **deep-frozen**. * Their content reuses the already frozen durable event data, so the cache @@ -601,9 +568,8 @@ declare class Session { */ deriveMessages(): Message[]; /** - * Derive one event under the session's current image offload watermark. - * Direct model consumers use this instance method so their per-event - * reconstruction matches {@link deriveMessages}. + * Instance face of the pure per-node `deriveEventMessage` export from + * `surface.ts`. * @param event - the event to project. * @returns the derived message, or null when the event produces none. */ @@ -613,7 +579,7 @@ declare class Session { ## 派生历史:`deriveMessages()` 与 `deriveEventMessage()` -`Session.deriveMessages()` 将事件日志投影为模型看到的 `Message[]`。它是缓存的(每个 surface 节点在首次出现时投影一次;surface 重写或图片水位推进触发重建)且冻结的(每次调用返回一个新数组,引用共享的深冻结消息,因此通过投影修改已记录的历史在类型上不可表达)。纯函数导出 `deriveEventMessage(event)` 只应用消息类型规则。同名 Session 实例方法还会应用当前图片水位,是 compaction 等直接模型消费方使用的逐事件接口。投影规则: +`Session.deriveMessages()` 将事件日志投影为模型看到的 `Message[]`。它是缓存的(每个 surface 节点在首次出现时投影一次;surface 重写触发重建)且冻结的(每次调用返回一个新数组,引用共享的深冻结消息,因此通过投影修改已记录的历史在类型上不可表达)。`deriveEventMessage(event)` 是折叠所应用的逐节点纯函数,公开暴露以便外部重建器和开发不变式检查能以完全相同的规则投影日志前缀,不会与缓存产生分歧。投影规则: - `user/message` → 一条携带确切 `content` 的 user 消息;可选 envelope 仅作为日志中的展示元数据保留。 - `assistant/message` → 一条 assistant 消息,包含生成它的提供方和模型,以及可选的适配器私有回放状态。其嵌入式紧凑 stream 是回放、usage 与 UI 证据,而不是第二条 message。**内容为空的** `assistant/message` 也会跳过:因 max-tokens 而截断且无内容的步骤仍会记录一条 `assistant/message` 来保存 stream、usage、提供方和模型,但无内容的 assistant 轮次不得进入提供方 transcript(文本记录)。 diff --git a/packages/compaction/compaction-basic/tests/compaction-basic.spec.ts b/packages/compaction/compaction-basic/tests/compaction-basic.spec.ts index 28aebda515..76ddb5f021 100644 --- a/packages/compaction/compaction-basic/tests/compaction-basic.spec.ts +++ b/packages/compaction/compaction-basic/tests/compaction-basic.spec.ts @@ -908,46 +908,6 @@ describe('compaction region transaction', () => { expect(summarizedText(input)).toContain('fixture user 1') }) - it('applies the durable image watermark to the replayed summarization prefix', async () => { - const compact = service() - const session = conversation(3) - const replaced = session.surface.nodes[0]! - const replacement = session.append('user/message', createUserMessage({ - content: [ - { type: 'text', text: 'large image context '.repeat(100) }, - { - type: 'image', - attachment: { - attachmentId: AttachmentId(`sha256:${'d'.repeat(64)}`), - mediaType: 'image/png', - bytes: 1, - width: 1, - height: 1, - }, - }, - ], - source: { kind: 'user' }, - }), { - surfaceOp: { op: 'replace', start: replaced, end: replaced }, - sourceEventSeqs: [replaced], - }) - session.append('image/offload', { - turn: 4, - step: 1, - watermark: { seq: replacement.seq, path: [1] }, - }) - - await compact.compactRegion(replacement.seq, replacement.seq, agent(session, MODEL), SIGNAL) - - expect(compact.calls[0]!.input.messages[0]!.content[1]).toMatchObject({ - type: 'image', - offloaded: true, - }) - const durable = session.eventAt(replacement.seq)! - expect(durable.type === 'user/message' ? durable.data.content[1] : undefined) - .not.toHaveProperty('offloaded') - }) - it.each([ ['start missing', 9_001, undefined, /start seq 9001 not found/], ['end missing', undefined, 9_002, /end seq 9002 not found/], diff --git a/packages/core/session/README.i18n.yaml b/packages/core/session/README.i18n.yaml index 376d694b90..3d5d742b46 100644 --- a/packages/core/session/README.i18n.yaml +++ b/packages/core/session/README.i18n.yaml @@ -2,5 +2,5 @@ # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: # pnpm run verify-translation-pairing --write packages/core/session/README.md -README.md: 584c4921811e08f7d5d5450f5d2f3211e137df6f -README.zh.md: ea82ccebab44f571a481d85cdcf18648c5caa9a7 +README.md: e810299678c93fd0b5c740dcd2485df746eee0e0 +README.zh.md: 9d8250053a6b11fd9ac71839bd6517cb58c3d90a diff --git a/packages/core/session/README.md b/packages/core/session/README.md index 584c492181..e810299678 100644 --- a/packages/core/session/README.md +++ b/packages/core/session/README.md @@ -83,15 +83,13 @@ The package is built on event sourcing: a `Session` is an append-only log of typ `request/header` stores a full canonical snapshot of the non-history request envelope with reason `initial`, `resume`, `change`, or `series`. An explicit message-series start or a surface replacement writes a `series` snapshot when the envelope is unchanged; a simultaneous change uses `startsSeries: true`. Same-series steps, retries, and ordinary later turns inherit the latest snapshot. `adapterDefaults` distinguishes values resolved by the adapter from explicit settings, and `foldRequestHeader()` selects the latest snapshot. This self-contained record supports partial-window rendering and exact reconstruction at the cost of growth per message series; the [reconstructable-requests Agent Note](../../../.agents/notes/implemented/architecture/2026-07-05-reconstructable-requests.md) owns the detail. -`image/offload` records the durable image offload watermark `dsh-llm-retry` advances when an adapter reports that a route's request-image budget is exceeded: an `ImageOccurrencePosition` naming the last offloaded occurrence by event seq and its complete nested block path. The derivation marks every occurrence at or before it `offloaded: true`, so each route sends its placeholder text instead of the image. The position only advances; `Session.append` and seeding require it to identify a current surface image and reject request-only `offloaded` markers in durable messages. Because the event changes the derived surface it is required-on-read: a build without the type refuses the log rather than replaying images the model never saw ([decision](../../../.agents/notes/implemented/architecture/2026-09-02-image-offload-watermark.md)). - ### Source map | File | Role | |---|---| | [`src/index.ts`](src/index.ts) | Plugin entry: `SessionStore` service, store lifecycle, `fork`, `flush` | | [`src/types.ts`](src/types.ts) | `SessionEventMap`, `SessionEvent`, `UserMessage`, `SessionHeader`, `TurnEndReasonMap` | -| [`src/surface.ts`](src/surface.ts) | Ordered surface projection, replacement validation, `deriveEventMessage`, `image/offload` marking | +| [`src/surface.ts`](src/surface.ts) | Ordered surface projection, replacement validation, `deriveEventMessage` | | [`src/request-header.ts`](src/request-header.ts) | `request/header` folding and reconstruction | | [`dsh-util-values`](../../util/values/README.md) | Shared lossless JSON validation and detached snapshots | | [`src/repair.ts`](src/repair.ts) | Cold repair of crash-orphaned logs | @@ -107,7 +105,7 @@ Every append uses the shared iterative `snapshotJsonValue()` pass, which reads, ### The request header -The loop logs a full canonical `request/header` snapshot (call config, adapter defaults, rendered system prompt, assembled tool schemas) at each loop-instance boundary and on change; `foldRequestHeader(events)` reconstructs it by selecting the latest snapshot, making every conversation request a pure function of the log. Route metadata (`request/context`) is separate logged state appended only when the provider, model, or capacity differs. The image offload watermark (`image/offload`) is durable surface state: every image occurrence positioned at or before it derives with `offloaded: true`, each event must advance strictly past the previous one. +The loop logs a full canonical `request/header` snapshot (call config, adapter defaults, rendered system prompt, assembled tool schemas) at each loop-instance boundary and on change; `foldRequestHeader(events)` reconstructs it by selecting the latest snapshot, making every conversation request a pure function of the log. Route metadata (`request/context`) is separate logged state appended only when the provider, model, or capacity differs. diff --git a/packages/core/session/README.zh.md b/packages/core/session/README.zh.md index ea82ccebab..9d8250053a 100644 --- a/packages/core/session/README.zh.md +++ b/packages/core/session/README.zh.md @@ -83,15 +83,13 @@ session.deriveMessages() // the derived model history `request/header` 存储非历史请求 envelope 的完整规范快照,原因为 `initial`、`resume`、`change` 或 `series`。显式消息序列起点或表层替换会在 envelope 不变时写入 `series` 快照;同时发生变化时使用 `startsSeries: true`。同一序列内的步骤、重试与普通后续轮次继承最新快照。`adapterDefaults` 区分由适配器解析的值与显式设置,`foldRequestHeader()` 选择最新快照。这种自包含记录以每个消息序列增加存储为代价,支持局部窗口渲染与精确重建;细节由[可重建请求 Agent Note](../../../.agents/notes/implemented/architecture/2026-07-05-reconstructable-requests.zh.md)负责。 -`image/offload` 记录 `dsh-llm-retry` 在 adapter 报告路由请求图片预算被超过时推进的持久图片 offload 水位:一个以事件序号加完整嵌套块路径指向最后一个被省略出现位置的 `ImageOccurrencePosition`。派生把位于水位及之前的每个出现位置标为 `offloaded: true`,于是每条路由发送其占位文本而不是图片。位置只会推进;`Session.append` 与 seed 要求它指向当前表层图片,并拒绝持久消息中的请求专用 `offloaded` 标记。该事件改变派生表层,因此读取时必须识别:不认识该类型的构建拒绝这份日志,而不是带着模型从未见过的图片重放([决定](../../../.agents/notes/implemented/architecture/2026-09-02-image-offload-watermark.zh.md))。 - ### 源码地图 | 文件 | 职责 | |---|---| | [`src/index.ts`](src/index.ts) | 插件入口:`SessionStore` 服务、存储生命周期、`fork`、`flush` | | [`src/types.ts`](src/types.ts) | `SessionEventMap`、`SessionEvent`、`UserMessage`、`SessionHeader`、`TurnEndReasonMap` | -| [`src/surface.ts`](src/surface.ts) | 有序 surface 投影、替换校验、`deriveEventMessage`、`image/offload` 标记 | +| [`src/surface.ts`](src/surface.ts) | 有序 surface 投影、替换校验、`deriveEventMessage` | | [`src/request-header.ts`](src/request-header.ts) | `request/header` 折叠与重建 | | [`dsh-util-values`](../../util/values/README.zh.md) | 共享无损 JSON 校验与分离式快照 | | [`src/repair.ts`](src/repair.ts) | 崩溃遗留日志的冷修复 | @@ -107,7 +105,7 @@ session.deriveMessages() // the derived model history ### 请求头 -循环在每个循环实例边界及变更时记录完整规范 `request/header` 快照(调用配置、适配器默认值、渲染后的系统提示词、组装后的工具 schema);`foldRequestHeader(events)` 通过选择最新快照来重建它,使每个对话请求都成为日志的纯函数。路由元数据(`request/context`)是独立的已记录状态,仅在提供方、模型或容量变化时追加。图片 offload 水位(`image/offload`)是持久的表层状态:位于水位及之前的每个图片出现位置派生时带 `offloaded: true`,每条事件必须严格越过前一条。 +循环在每个循环实例边界及变更时记录完整规范 `request/header` 快照(调用配置、适配器默认值、渲染后的系统提示词、组装后的工具 schema);`foldRequestHeader(events)` 通过选择最新快照来重建它,使每个对话请求都成为日志的纯函数。路由元数据(`request/context`)是独立的已记录状态,仅在提供方、模型或容量变化时追加。 diff --git a/packages/core/session/src/index.ts b/packages/core/session/src/index.ts index 2c16d89569..14e889ad87 100644 --- a/packages/core/session/src/index.ts +++ b/packages/core/session/src/index.ts @@ -12,11 +12,11 @@ import { brandString } from '@deepseek-ai/dsh-brand' import { assertNever, deepFreeze, snapshotJsonValue } from '@deepseek-ai/dsh-util-values' import { scopeOf, scopeTarget } from '@deepseek-ai/dsh-scope' import type { Scoped } from '@deepseek-ai/dsh-scope' -import type { ContentBlock, Message } from '@deepseek-ai/dsh-llm' +import type { Message } from '@deepseek-ai/dsh-llm' import { SESSION_FORMAT_VERSION, SessionLogOffset, SessionSeq } from './types.ts' import type { TypertLookup } from '@deepseek-ai/dsh-typert-protocol' -import type { CreateSessionOptions, EpochHeader, ImageOccurrencePosition, PrepareSessionOptions, RequestContext, SessionEvent, SessionEventMap, SessionEventType, SessionHeader, SessionId, SessionSeedEventState, SurfaceIntent, SurfaceEventType } from './types.ts' -import { compareImagePositions, deriveEventMessage, markImageOffload, SurfaceManager } from './surface.ts' +import type { CreateSessionOptions, EpochHeader, PrepareSessionOptions, RequestContext, SessionEvent, SessionEventMap, SessionEventType, SessionHeader, SessionId, SessionSeedEventState, SurfaceIntent, SurfaceEventType } from './types.ts' +import { deriveEventMessage, SurfaceManager } from './surface.ts' import type { SessionSurface } from './surface.ts' import { foldRequestHeader } from './request-header.ts' @@ -26,7 +26,7 @@ export type { SessionPreparationOptions } from './preparation.ts' export type { AssistantMessage, ToolResultMessage, UserMessage } from '@deepseek-ai/dsh-llm' export { interruptedTurnClosers, TOOL_NOT_STARTED, TOOL_OUTCOME_UNKNOWN } from './repair.ts' export type { SessionSurface, SurfaceFoldReplacement, SurfaceFoldResult } from './surface.ts' -export { compareImagePositions, deriveEventMessage, foldSurface, isAppendSurfaceEvent, isReplacementSurfaceEvent, isSurfaceEvent, isSurfaceEligibleType } from './surface.ts' +export { deriveEventMessage, foldSurface, isAppendSurfaceEvent, isReplacementSurfaceEvent, isSurfaceEvent, isSurfaceEligibleType } from './surface.ts' export { canonicalHeader, foldRequestHeader, headerEquals } from './request-header.ts' export { KNOWN_SESSION_EVENT_TYPES } from './known-event-types.ts' @@ -335,7 +335,6 @@ function assertMessageEventShape(event: Record, subject: string if (!Array.isArray(messageRecord['content'])) { throw new Error(`${subject} message has invalid content`) } - assertNoLoggedImageOffloadMarker(type, data, subject) const sourceRecord = source as Record if (type === 'assistant/message') { if (sourceRecord['kind'] !== 'model' || !hasProviderModel(sourceRecord)) { @@ -361,58 +360,6 @@ function assertMessageEventShape(event: Record, subject: string } } -/** Reject the request-only image marker from durable message content at every nesting depth. */ -function assertNoLoggedImageOffloadMarker(type: string, data: unknown, subject: string): void { - if (type !== 'user/message' && type !== 'assistant/message' && type !== 'tool/result') return - const message = (type === 'user/message' ? data : (data as { message: unknown }).message) as { content?: unknown } - const visit = (blocks: unknown): void => { - if (!Array.isArray(blocks)) return - for (const block of blocks as ({ type?: unknown; content?: unknown } | null)[]) { - if (block?.type === 'image' && 'offloaded' in block) { - throw new Error(`${subject} stores the request-only image offload marker`) - } - if (block?.type === 'tool-result') visit(block.content) - } - } - visit(message.content) -} - -/** Return whether a block path identifies an image occurrence. */ -function pathIdentifiesImage(content: readonly ContentBlock[], [index, ...rest]: readonly number[]): boolean { - const block = index === undefined ? undefined : content[index] - if (block === undefined) return false - if (rest.length === 0) return block.type === 'image' - return block.type === 'tool-result' && pathIdentifiesImage(block.content, rest) -} - -/** - * Validate one `image/offload` payload at the append or seed boundary: the - * position must name an image occurrence on the current surface and lie - * strictly after the watermark in force. - */ -function assertImageOffloadAdvance( - data: unknown, - log: readonly SessionEvent[], - surfaceNodes: readonly SessionSeq[], - prior: ImageOccurrencePosition | undefined, - location: string, -): void { - const watermark = (data as { watermark?: { seq?: unknown; path?: unknown } } | null)?.watermark - if (typeof watermark?.seq !== 'number' || !Array.isArray(watermark.path) - || !watermark.path.every(index => Number.isInteger(index) && index >= 0)) { - throw new Error(`${location} names an invalid image offload watermark`) - } - const position = { seq: watermark.seq as SessionSeq, path: watermark.path as number[] } - if (prior !== undefined && compareImagePositions(position, prior) <= 0) { - throw new Error(`${location} does not advance the image offload watermark`) - } - // oxlint-disable-next-line typescript/no-non-null-assertion -- surface nodes index the accepted log - const message = surfaceNodes.includes(position.seq) ? deriveEventMessage(log[position.seq]!) : null - if (message === null || !pathIdentifiesImage(message.content, position.path)) { - throw new Error(`${location} watermark does not identify an image on the current surface`) - } -} - /** Whether an unknown value carries the current provider/model pair. */ function hasProviderModel(value: unknown): boolean { if (typeof value !== 'object' || value === null) return false @@ -474,8 +421,6 @@ const attachments = new WeakMap() */ export class Session { private log: SessionEvent[] = [] - /** Watermark in force after the last accepted `image/offload` event. */ - private imageOffloadFold: ImageOccurrencePosition | undefined /** Single incremental owner of surface acceptance and projection state. */ private readonly surfaceManager = new SurfaceManager(this.log) @@ -598,9 +543,6 @@ export class Session { throw new Error(`seed event at index ${index} is not losslessly JSON-serializable`) } assertSessionEventEnvelope(snapshot, index) - if (snapshot.type === 'image/offload') { - assertImageOffloadAdvance(snapshot.data, this.log, this.surface.nodes, this.imageOffloadFold, `seed event at index ${index}`) - } if (snapshot.seq !== index) { throw new Error(`seed event at index ${index} has seq ${snapshot.seq} (expected ${index}); seed must be contiguous from 0`) } @@ -613,8 +555,6 @@ export class Session { throw new Error(`invalid seed event at index ${index}: ${error instanceof Error ? error.message : 'invalid surface metadata'}`) } this.log.push(mode === 'snapshot' ? deepFreeze(snapshot) : snapshot) - // oxlint-disable-next-line typescript/no-non-null-assertion -- the event was just appended - this.foldImageOffload(this.log.at(-1)!) } } this.firstLiveSeq = SessionLogOffset(this.log.length) @@ -749,10 +689,6 @@ export class Session { if (dataSnapshot === undefined) { throw new Error(`session event "${type}" carries non-JSON-serializable data`) } - assertNoLoggedImageOffloadMarker(type, dataSnapshot, `session event "${type}"`) - if (type === 'image/offload') { - assertImageOffloadAdvance(dataSnapshot, this.log, this.surface.nodes, this.imageOffloadFold, `session event "${type}"`) - } const surfaceMetadataSnapshot = snapshotJsonValue(surfaceMetadata) if (surfaceMetadataSnapshot === undefined) { throw new Error(`session event "${type}" carries non-JSON-serializable surface metadata`) @@ -778,7 +714,6 @@ export class Session { callbacks = collectSessionCallbacks(entry.emitCtx, [entry.carrier, 'session/event', ...callbackArgs]) } this.log.push(event as SessionEvent) - this.foldImageOffload(event as SessionEvent) this.eventsSnapshot = undefined if (callbacks !== undefined && entry !== undefined) { invokeContainedSessionObservers(entry.emitCtx, 'session/event', entry.id, callbackArgs, callbacks) @@ -836,21 +771,12 @@ export class Session { return this.contextFold } - /** Record an accepted `image/offload` event as the watermark in force. */ - private foldImageOffload(event: SessionEvent): void { - if (event.type !== 'image/offload') return - const { watermark } = event.data - this.imageOffloadFold = deepFreeze({ seq: watermark.seq, path: [...watermark.path] }) - } - /** The derived-message cache: frozen projections, extended per unseen node. */ private derived: Message[] = [] /** Surface position (nodes projected) the cache has reached. */ private derivedNodes = 0 /** {@link SurfaceManager.replaceGeneration} the cache was built under. */ private derivedGeneration = 0 - /** Watermark the cache was built under; an advance rebuilds every node. */ - private derivedWatermark: ImageOccurrencePosition | undefined /** * Derive the LLM message history by walking the ordered sequences of @@ -863,9 +789,7 @@ export class Session { * * CACHED: each surface node is projected exactly once, when first seen — a * call costs O(new nodes), and a surface rewrite (a `replace`; - * {@link SessionSurface.replaceGeneration}) or an `image/offload` advance - * rebuilds, and image occurrences at or before the watermark derive with - * `offloaded: true` ({@link markImageOffload}). The returned array is + * {@link SessionSurface.replaceGeneration}) rebuilds. The returned array is * a fresh snapshot per call (later appends never grow an array a caller * already holds); the `Message` objects in it are SHARED and **deep-frozen**. * Their content reuses the already frozen durable event data, so the cache @@ -876,12 +800,10 @@ export class Session { const surface = this.surface const nodes = surface.nodes const generation = surface.replaceGeneration - const watermark = this.imageOffloadFold - if (generation !== this.derivedGeneration || watermark !== this.derivedWatermark) { + if (generation !== this.derivedGeneration) { this.derived = [] this.derivedNodes = 0 this.derivedGeneration = generation - this.derivedWatermark = watermark } for (const seq of nodes.slice(this.derivedNodes)) { // Surface sequences are built from this.log — seq is always a valid @@ -898,18 +820,13 @@ export class Session { } /** - * Derive one event under the session's current image offload watermark. - * Direct model consumers use this instance method so their per-event - * reconstruction matches {@link deriveMessages}. + * Instance face of the pure per-node `deriveEventMessage` export from + * `surface.ts`. * @param event - the event to project. * @returns the derived message, or null when the event produces none. */ deriveEventMessage(event: SessionEvent): Message | null { - const message = deriveEventMessage(event) - if (message === null) return null - // A marked copy is frozen like the durable original it derives from. - const marked = markImageOffload(message, event.seq, this.imageOffloadFold) - return marked === message ? message : deepFreeze(marked) + return deriveEventMessage(event) } } diff --git a/packages/core/session/src/invariant.ts b/packages/core/session/src/invariant.ts index aad11350db..3242691324 100644 --- a/packages/core/session/src/invariant.ts +++ b/packages/core/session/src/invariant.ts @@ -148,8 +148,7 @@ function validateEvent( // Unconstrained: an unbalanced seed legally puts it inside an open turn. break case 'request/header': - case 'request/context': - case 'image/offload': { + case 'request/context': { if (trace.openTurn === null) { fail(`${event.type} appended outside any open turn (core execution events must be turn-enclosed)`) } diff --git a/packages/core/session/src/known-event-types.ts b/packages/core/session/src/known-event-types.ts index 749ab47fc3..3bc31ec32d 100644 --- a/packages/core/session/src/known-event-types.ts +++ b/packages/core/session/src/known-event-types.ts @@ -39,7 +39,6 @@ export const KNOWN_SESSION_EVENT_TYPES: ReadonlySet = new Set([ 'goal/change', 'hook/invoked', 'hook/result', - 'image/offload', 'llm/retry', 'llm/retry-started', 'model/selection', diff --git a/packages/core/session/src/surface.ts b/packages/core/session/src/surface.ts index 1593a94f27..1cec97e3cd 100644 --- a/packages/core/session/src/surface.ts +++ b/packages/core/session/src/surface.ts @@ -8,10 +8,9 @@ * @module @deepseek-ai/dsh-session/surface */ -import type { ContentBlock, ImageBlockPath, Message } from '@deepseek-ai/dsh-llm' +import type { Message } from '@deepseek-ai/dsh-llm' import { SessionLogOffset, SessionSeq } from './types.ts' import type { - ImageOccurrencePosition, SessionEvent, SessionSeqCursor, SurfaceEvent, @@ -121,76 +120,6 @@ export function deriveEventMessage(event: SessionEvent): Message | null { } } -/** Compare two nested block paths in message order. */ -function compareImageBlockPaths(a: ImageBlockPath, b: ImageBlockPath): number { - const length = Math.min(a.length, b.length) - for (let index = 0; index < length; index += 1) { - // oxlint-disable-next-line typescript/no-non-null-assertion -- index is below both lengths - const delta = a[index]! - b[index]! - if (delta !== 0) return delta - } - return a.length - b.length -} - -/** - * Compare two image occurrence positions in log order: by event seq, then by - * block path inside the event's content. - * @param a - first position. - * @param b - second position. - * @returns negative, zero, or positive as `a` lies before, at, or after `b`. - */ -export function compareImagePositions(a: ImageOccurrencePosition, b: ImageOccurrencePosition): number { - return a.seq === b.seq ? compareImageBlockPaths(a.path, b.path) : a.seq - b.seq -} - -/** Mark matching image occurrences while preserving unchanged durable content by identity. */ -function markOffloadedImages( - content: readonly ContentBlock[], - offloaded: (path: ImageBlockPath) => boolean, - prefix: ImageBlockPath = [], -): readonly ContentBlock[] { - let next: ContentBlock[] | undefined - for (const [index, block] of content.entries()) { - const path = [...prefix, index] - if (block.type === 'image' && block.offloaded !== true && offloaded(path)) { - next ??= content.slice(0, index) - next.push({ ...block, offloaded: true }) - continue - } - if (block.type === 'tool-result') { - const inner = markOffloadedImages(block.content, offloaded, path) - if (inner !== block.content) { - next ??= content.slice(0, index) - next.push({ ...block, content: inner as ContentBlock[] }) - continue - } - } - next?.push(block) - } - return next ?? content -} - -/** - * Project one derived surface message under the `image/offload` watermark: - * every image occurrence positioned at or before it carries `offloaded: true`. - * @param message - the node's derived message. - * @param seq - the node's event seq. - * @param watermark - watermark in force, or undefined when nothing is offloaded. - * @returns the same message when nothing changes, otherwise a shallow copy with marked content. - */ -export function markImageOffload( - message: Message, - seq: SessionSeq, - watermark: ImageOccurrencePosition | undefined, -): Message { - if (watermark === undefined || seq > watermark.seq) return message - const content = markOffloadedImages( - message.content, - path => seq < watermark.seq || compareImageBlockPaths(path, watermark.path) <= 0, - ) - return content === message.content ? message : { ...message, content: content as Message['content'] } -} - /** One replacement operation observed while folding a session surface. */ export interface SurfaceFoldReplacement { /** Seq of the event that replaced the prior surface range. */ diff --git a/packages/core/session/src/types.ts b/packages/core/session/src/types.ts index 9808f23f64..3007650a85 100644 --- a/packages/core/session/src/types.ts +++ b/packages/core/session/src/types.ts @@ -247,20 +247,6 @@ export interface RequestContext { contextWindow?: number } -/** - * Durable position of one image occurrence on the model-visible surface: the - * seq of the event carrying it and the block path inside that event's - * content. Every nested tool-result contributes another path index. Positions - * order by seq, then by path, so a newer event always lies after an older one - * regardless of surface replacements. - */ -export interface ImageOccurrencePosition { - /** Seq of the surface message event carrying the occurrence. */ - seq: SessionSeq - /** Block index path inside that event's message content. */ - path: number[] -} - /** * Why a `request/header` snapshot was appended: `'initial'` — the log's first * header (a new conversation); `'resume'` — a loop instance's first request @@ -370,17 +356,6 @@ export interface SessionEventMap { * changes. It does not participate in request reconstruction or header equality. */ 'request/context': RequestContext - /** - * Advances the durable image offload watermark before a request in step - * `step` of turn `turn` is dispatched. Every image occurrence positioned at - * or before `watermark` derives with `offloaded: true`, so each route sends - * its placeholder text instead of the image; occurrences after it stay - * retained. The watermark only advances: each event names a position - * strictly after the previous one, and no later budget, route change, or - * compaction moves it back. It is a log-only event that changes the derived - * surface, so a build that does not know the type refuses the log. - */ - 'image/offload': { turn: number; step: number; watermark: ImageOccurrencePosition } /** * Marks the end of a constructor seed. Events before it have smaller seq * values and came from the seed (resume, fork, or replay); this lifecycle diff --git a/packages/core/session/tests/image-offload.spec.ts b/packages/core/session/tests/image-offload.spec.ts deleted file mode 100644 index 946d64cf76..0000000000 --- a/packages/core/session/tests/image-offload.spec.ts +++ /dev/null @@ -1,234 +0,0 @@ -/** - * Durable image offload watermark: append/seed validation, monotonic - * advance, derived-surface marking, cache rebuild, and replay equality. - */ - -import { describe, expect, it } from 'vitest' -import { createUserMessage, ToolCallId } from '@deepseek-ai/dsh-llm' -import type { ContentBlock } from '@deepseek-ai/dsh-llm' -import { compareImagePositions, Session, SessionId, SessionSeq } from '@deepseek-ai/dsh-session' - -function image(name: string): Extract { - return { - type: 'image', - attachment: { - attachmentId: `sha256:${'a'.repeat(64)}` as never, - name, - mediaType: 'image/png', - bytes: 3, - width: 1, - height: 1, - }, - } -} - -function toolResult(name: string): ContentBlock { - return { type: 'tool-result', toolCallId: ToolCallId(name), content: [{ type: 'text', text: 'shot' }, image(name)] } -} - -function seeded(): Session { - const session = Session.create(SessionId('offload')) - session.append('turn/start', { turn: 1 }) - session.append('step/start', { turn: 1, step: 1 }) - session.append('user/message', createUserMessage({ - content: [{ type: 'text', text: 'no images here' }], source: { kind: 'user' }, - }), { surfaceOp: 'append' }) - session.append('user/message', createUserMessage({ - content: [image('first'), image('second')], source: { kind: 'user' }, - }), { surfaceOp: 'append' }) - session.append('user/message', createUserMessage({ - content: [toolResult('third'), image('fourth')], source: { kind: 'user' }, - }), { surfaceOp: 'append' }) - return session -} - -function offloadedNames(session: Session): string[] { - const names: string[] = [] - const visit = (content: readonly ContentBlock[]): void => { - for (const block of content) { - if (block.type === 'image' && block.offloaded === true) names.push(block.attachment.name ?? '') - if (block.type === 'tool-result') visit(block.content) - } - } - for (const message of session.deriveMessages()) { - visit(message.content) - } - return names -} - -describe('image/offload append validation', () => { - it('rejects a malformed or out-of-range watermark', () => { - const session = seeded() - expect(() => session.append('image/offload', { turn: 1, step: 1, watermark: { seq: SessionSeq(99), path: [0] } })) - .toThrow('watermark does not identify an image on the current surface') - expect(() => session.append('image/offload', { turn: 1, step: 1, watermark: { seq: SessionSeq(3), path: [] } })) - .toThrow('watermark does not identify an image on the current surface') - expect(() => session.append('image/offload', { turn: 1, step: 1, watermark: { seq: SessionSeq(3), path: [-1] } })) - .toThrow('names an invalid image offload watermark') - expect(() => session.append('image/offload', { turn: 1, step: 1, watermark: null as never })) - .toThrow('names an invalid image offload watermark') - expect(() => session.append('image/offload', 7 as never)) - .toThrow('names an invalid image offload watermark') - expect(offloadedNames(session)).toEqual([]) - }) - - it('requires the watermark path to identify a current surface image', () => { - const session = seeded() - expect(() => session.append('image/offload', { turn: 1, step: 1, watermark: { seq: SessionSeq(2), path: [0] } })) - .toThrow('watermark does not identify an image on the current surface') - expect(() => session.append('image/offload', { turn: 1, step: 1, watermark: { seq: SessionSeq(3), path: [9] } })) - .toThrow('watermark does not identify an image on the current surface') - expect(() => session.append('image/offload', { turn: 1, step: 1, watermark: { seq: SessionSeq(3), path: [0, 0] } })) - .toThrow('watermark does not identify an image on the current surface') - session.append('user/message', createUserMessage({ - content: [{ type: 'text', text: 'replacement' }], source: { kind: 'user' }, - }), { - surfaceOp: { op: 'replace', start: SessionSeq(3), end: SessionSeq(3) }, - sourceEventSeqs: [SessionSeq(3)], - }) - expect(() => session.append('image/offload', { turn: 1, step: 1, watermark: { seq: SessionSeq(3), path: [0] } })) - .toThrow('watermark does not identify an image on the current surface') - }) - - it('rejects request-only offload markers from appended and restored messages', () => { - const session = seeded() - expect(() => session.append('assistant/message', { - turn: 1, - step: 1, - message: {}, - } as never, { surfaceOp: 'append' })).not.toThrow() - expect(() => session.append('user/message', createUserMessage({ - content: [{ ...image('marked'), offloaded: true }], source: { kind: 'user' }, - }), { surfaceOp: 'append' })) - .toThrow('stores the request-only image offload marker') - - const events = [...session.snapshotEvents()] - const event = events[3]! - const marked = { - ...event, - data: { ...event.data, content: [null, { ...image('restored-marked'), offloaded: true }] }, - } - events[3] = marked as never - expect(() => Session.create(SessionId('offload-marked-seed'), events)) - .toThrow('seed user/message at index 3 stores the request-only image offload marker') - }) - - it('accepts only strictly advancing positions', () => { - const session = seeded() - session.append('image/offload', { turn: 1, step: 1, watermark: { seq: SessionSeq(3), path: [1] } }) - expect(offloadedNames(session)).toEqual(['first', 'second']) - expect(() => session.append('image/offload', { turn: 1, step: 1, watermark: { seq: SessionSeq(3), path: [1] } })) - .toThrow('does not advance the image offload watermark') - expect(() => session.append('image/offload', { turn: 1, step: 1, watermark: { seq: SessionSeq(3), path: [0] } })) - .toThrow('does not advance the image offload watermark') - session.append('image/offload', { turn: 1, step: 1, watermark: { seq: SessionSeq(4), path: [0, 1] } }) - expect(offloadedNames(session)).toEqual(['first', 'second', 'third']) - }) - - it('validates a seed with the same rules and folds its watermark', () => { - const session = seeded() - session.append('image/offload', { turn: 1, step: 1, watermark: { seq: SessionSeq(3), path: [0] } }) - const replayed = Session.create(SessionId('offload-replay'), session.snapshotEvents()) - expect(offloadedNames(replayed)).toEqual(['first']) - expect(replayed.deriveMessages()).toEqual(session.deriveMessages()) - - const events = session.snapshotEvents().map(event => ( - event.type === 'image/offload' - ? { ...event, data: { ...event.data, watermark: { seq: 50, path: [0] } } } - : event - )) - expect(() => Session.create(SessionId('offload-bad-seed'), events as never)) - .toThrow('seed event at index 5 watermark does not identify an image on the current surface') - }) -}) - -describe('image/offload surface derivation', () => { - it('marks occurrences at or before the watermark, including nested tool results', () => { - const session = seeded() - expect(offloadedNames(session)).toEqual([]) - session.append('image/offload', { turn: 1, step: 1, watermark: { seq: SessionSeq(3), path: [0] } }) - expect(offloadedNames(session)).toEqual(['first']) - session.append('image/offload', { turn: 1, step: 1, watermark: { seq: SessionSeq(4), path: [0, 1] } }) - expect(offloadedNames(session)).toEqual(['first', 'second', 'third']) - session.append('image/offload', { turn: 1, step: 1, watermark: { seq: SessionSeq(4), path: [1] } }) - expect(offloadedNames(session)).toEqual(['first', 'second', 'third', 'fourth']) - }) - - it('marks images below two nested tool-result levels', () => { - const session = seeded() - const deep = session.append('user/message', createUserMessage({ - content: [{ - type: 'tool-result', - toolCallId: ToolCallId('outer'), - content: [{ - type: 'tool-result', - toolCallId: ToolCallId('middle'), - content: [{ type: 'tool-result', toolCallId: ToolCallId('inner'), content: [image('deep')] }], - }], - }], - source: { kind: 'user' }, - }), { surfaceOp: 'append' }) - session.append('image/offload', { - turn: 1, - step: 1, - watermark: { seq: deep.seq, path: [0, 0, 0, 0] }, - }) - expect(offloadedNames(session)).toContain('deep') - }) - - it('freezes marked copies, keeps the durable event content untouched, and matches a scratch replay', () => { - const session = seeded() - const before = session.deriveMessages() - session.append('image/offload', { turn: 1, step: 1, watermark: { seq: SessionSeq(3), path: [0] } }) - const after = session.deriveMessages() - expect(after[0]).toBe(before[0]) - expect(after[1]).not.toBe(before[1]) - expect(Object.isFrozen(after[1])).toBe(true) - expect(Object.isFrozen(after[1]!.content[0])).toBe(true) - expect(after[2]).toBe(before[2]) - expect(before[1]!.content[0]).toEqual(image('first')) - expect(session.eventAt(SessionSeq(3))!.data).toMatchObject({ content: [image('first'), image('second')] }) - const scratch = Session.create(SessionId('offload-scratch'), session.snapshotEvents()) - expect(scratch.deriveMessages()).toEqual(after) - }) - - it('leaves later appended nodes retained without rebuilding the cache', () => { - const session = seeded() - session.append('image/offload', { turn: 1, step: 1, watermark: { seq: SessionSeq(4), path: [1] } }) - const marked = session.deriveMessages() - session.append('user/message', createUserMessage({ - content: [image('fifth')], source: { kind: 'user' }, - }), { surfaceOp: 'append' }) - const grown = session.deriveMessages() - expect(grown[0]).toBe(marked[0]) - expect(grown.at(-1)!.content[0]).toEqual(image('fifth')) - expect(offloadedNames(session)).toEqual(['first', 'second', 'third', 'fourth']) - }) -}) - -describe('image offload helpers', () => { - it('preserves a nested block by identity when only an earlier sibling is offloaded', () => { - const session = Session.create(SessionId('offload-sibling')) - session.append('turn/start', { turn: 1 }) - const message = createUserMessage({ - content: [image('first'), toolResult('later')], - source: { kind: 'user' }, - }) - const seq = session.append('user/message', message, { surfaceOp: 'append' }).seq - const before = session.deriveMessages()[0]! - session.append('image/offload', { turn: 1, step: 1, watermark: { seq, path: [0] } }) - const projected = session.deriveMessages()[0]! - expect(projected.content[0]).toEqual({ ...image('first'), offloaded: true }) - expect(projected.content[1]).toBe(before.content[1]) - }) - - it('orders positions by seq then path and folds the latest watermark', () => { - expect(compareImagePositions({ seq: SessionSeq(1), path: [5] }, { seq: SessionSeq(3), path: [0] })).toBeLessThan(0) - expect(compareImagePositions({ seq: SessionSeq(4), path: [1] }, { seq: SessionSeq(3), path: [0, 3] })).toBeGreaterThan(0) - const session = seeded() - session.append('image/offload', { turn: 1, step: 1, watermark: { seq: SessionSeq(3), path: [0] } }) - expect(offloadedNames(session)).toEqual(['first']) - session.append('image/offload', { turn: 1, step: 1, watermark: { seq: SessionSeq(4), path: [1] } }) - expect(offloadedNames(session)).toEqual(['first', 'second', 'third', 'fourth']) - }) -}) diff --git a/packages/extensions/tool-cordis/src/api-catalog.ts b/packages/extensions/tool-cordis/src/api-catalog.ts index 70e3415c99..211686b403 100644 --- a/packages/extensions/tool-cordis/src/api-catalog.ts +++ b/packages/extensions/tool-cordis/src/api-catalog.ts @@ -4311,10 +4311,6 @@ export const TYPE_API: readonly TypeApiEntry[] = [ name: 'ImageMediaType', declaration: 'export type ImageMediaType = \'image/png\' | \'image/jpeg\' | \'image/webp\' | \'image/gif\';', }, - { - name: 'ImageOccurrencePosition', - declaration: 'export interface ImageOccurrencePosition {\n seq: SessionSeq;\n path: number[];\n}', - }, { name: 'ImageRequestPolicy', declaration: 'export interface ImageRequestPolicy {\n maxPixels: number;\n maxBytes: number;\n}', @@ -5061,7 +5057,7 @@ export const TYPE_API: readonly TypeApiEntry[] = [ }, { name: 'SessionEventMap', - declaration: 'export interface SessionEventMap {\n \'turn/start\': {\n turn: number;\n };\n \'turn/end\': {\n turn: number;\n reason: TurnEndReason;\n };\n \'step/start\': {\n turn: number;\n step: number;\n };\n \'step/end\': {\n turn: number;\n step: number;\n };\n \'user/message\': UserMessage;\n \'assistant/message\': {\n turn: number;\n step: number;\n message: AssistantMessage;\n stream: AssistantStreamRecord[];\n usage?: TokenUsage;\n interrupted?: true;\n };\n \'assistant/attempt\': {\n turn: number;\n step: number;\n stream: AssistantStreamRecord[];\n };\n \'tool/call\': {\n turn: number;\n step: number;\n callId: ToolCallId;\n name: string;\n arguments: string;\n };\n \'tool/result\': {\n turn: number;\n step: number;\n message: ToolResultMessage;\n error?: {\n name: string;\n code: string;\n };\n meta?: JsonValue;\n };\n \'request/header\': {\n header: EpochHeader;\n reason: RequestHeaderReason;\n startsSeries?: true;\n };\n \'request/context\': RequestContext;\n \'image/offload\': {\n turn: number;\n step: number;\n watermark: ImageOccurrencePosition;\n };\n \'session/end-seed\': {\n inherited?: true;\n };\n}', + declaration: 'export interface SessionEventMap {\n \'turn/start\': {\n turn: number;\n };\n \'turn/end\': {\n turn: number;\n reason: TurnEndReason;\n };\n \'step/start\': {\n turn: number;\n step: number;\n };\n \'step/end\': {\n turn: number;\n step: number;\n };\n \'user/message\': UserMessage;\n \'assistant/message\': {\n turn: number;\n step: number;\n message: AssistantMessage;\n stream: AssistantStreamRecord[];\n usage?: TokenUsage;\n interrupted?: true;\n };\n \'assistant/attempt\': {\n turn: number;\n step: number;\n stream: AssistantStreamRecord[];\n };\n \'tool/call\': {\n turn: number;\n step: number;\n callId: ToolCallId;\n name: string;\n arguments: string;\n };\n \'tool/result\': {\n turn: number;\n step: number;\n message: ToolResultMessage;\n error?: {\n name: string;\n code: string;\n };\n meta?: JsonValue;\n };\n \'request/header\': {\n header: EpochHeader;\n reason: RequestHeaderReason;\n startsSeries?: true;\n };\n \'request/context\': RequestContext;\n \'session/end-seed\': {\n inherited?: true;\n };\n}', }, { name: 'SessionEventMetadataFilter', diff --git a/packages/llm/llm-deepseek/README.i18n.yaml b/packages/llm/llm-deepseek/README.i18n.yaml index df870c0d50..7a002de1b1 100644 --- a/packages/llm/llm-deepseek/README.i18n.yaml +++ b/packages/llm/llm-deepseek/README.i18n.yaml @@ -2,5 +2,5 @@ # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: # pnpm run verify-translation-pairing --write packages/llm/llm-deepseek/README.md -README.md: ad9b89a35fb6bd53579d573aa8d88ee12dc80a77 -README.zh.md: dce5b3f7332173f4ef9a82d1dd6845b0597d009d +README.md: 476dfa93129caa867aacd9d7120de58a7ac6700c +README.zh.md: c6fc5c0ced2da5afd78bfc101f68e7b3c70c107a diff --git a/packages/llm/llm-deepseek/README.md b/packages/llm/llm-deepseek/README.md index ad9b89a35f..476dfa9312 100644 --- a/packages/llm/llm-deepseek/README.md +++ b/packages/llm/llm-deepseek/README.md @@ -156,7 +156,7 @@ The selected DeepSeek model receives the harness system prompt, message history, #### Token effect -Provider tokenization governs exact text and image-token input. The adapter declares per-route `imageRequestPricing`: it prices each occurrence the session's `image/offload` watermark marks offloaded as its placeholder text and each retained image at its projected dimensions with the published v4 vision accounting (14px patch grid, 3:1 downsampling, 384-token cap, worst-case alignment pad). This lets the token meter price image pressure before a request; reported usage remains authoritative. Reasoning passback carries every reasoned turn's chain of thought into later requests, while offloaded images stop costing visual tokens. A request whose retained occurrences exceed the file-mode or inline-fallback budget (`maxRequestFilesBytes`, `maxImagesPerRequest`, both quanta) at their exact request-version bytes fails with `IMAGE_OFFLOAD_REQUIRED` naming the additional oldest occurrences to offload, and `dsh-llm-retry` advances the watermark and retries. Cache-read usage is reported when available. `totalTokens` is the exact `prompt_tokens + completion_tokens` aggregate and is omitted if a supplied `total_tokens` disagrees. +Provider tokenization governs exact text and image-token input. The adapter declares per-route `imageRequestPricing`: it prices each occurrence a surface replacement marks offloaded as its placeholder text and each retained image at its projected dimensions with the published v4 vision accounting (14px patch grid, 3:1 downsampling, 384-token cap, worst-case alignment pad). This lets the token meter price image pressure before a request; reported usage remains authoritative. Reasoning passback carries every reasoned turn's chain of thought into later requests, while offloaded images stop costing visual tokens. A request whose retained occurrences exceed the file-mode or inline-fallback budget (`maxRequestFilesBytes`, `maxImagesPerRequest`, both quanta) at their exact request-version bytes fails with `IMAGE_OFFLOAD_REQUIRED` naming the additional oldest occurrences to offload, and `dsh-llm-retry` replaces the carrying surface nodes with marked copies and retries. Cache-read usage is reported when available. `totalTokens` is the exact `prompt_tokens + completion_tokens` aggregate and is omitted if a supplied `total_tokens` disagrees. #### KV Cache effect diff --git a/packages/llm/llm-deepseek/README.zh.md b/packages/llm/llm-deepseek/README.zh.md index dce5b3f733..c6fc5c0ced 100644 --- a/packages/llm/llm-deepseek/README.zh.md +++ b/packages/llm/llm-deepseek/README.zh.md @@ -156,7 +156,7 @@ Files 模式通过 `maxRequestFilesBytes` 与 `maxImagesPerRequest` 限制保留 #### Token 影响 -提供方分词决定精确的文本与图片 token 输入。适配器声明按路由的 `imageRequestPricing`:把会话 `image/offload` 水位标记为已省略的每个出现位置按其占位文本计价,并按投影后的尺寸使用官方公布的 v4 视觉计量(14px patch 网格、3:1 降采样、单图 384 token 上限、最坏对齐 pad)为每张保留图片计价。这使 token 计量服务可以在请求发出前为图片压力定价;上报的 usage 仍是权威值。推理回传会把每个推理轮次的思维链带进后续请求,而已省略的图片不再消耗视觉 token。保留的出现位置按精确请求版本字节超过 file 模式或内联回退预算(`maxRequestFilesBytes`、`maxImagesPerRequest` 与两个量子)的请求,以 `IMAGE_OFFLOAD_REQUIRED` 失败并说明还需省略多少最老的出现位置,由 `dsh-llm-retry` 推进水位并重试。可用时报告缓存读取用量。`totalTokens` 是精确的 `prompt_tokens + completion_tokens` 汇总值;提供方给出的 `total_tokens` 不一致时省略该值。 +提供方分词决定精确的文本与图片 token 输入。适配器声明按路由的 `imageRequestPricing`:把表层替换标记为已省略的每个出现位置按其占位文本计价,并按投影后的尺寸使用官方公布的 v4 视觉计量(14px patch 网格、3:1 降采样、单图 384 token 上限、最坏对齐 pad)为每张保留图片计价。这使 token 计量服务可以在请求发出前为图片压力定价;上报的 usage 仍是权威值。推理回传会把每个推理轮次的思维链带进后续请求,而已省略的图片不再消耗视觉 token。保留的出现位置按精确请求版本字节超过 file 模式或内联回退预算(`maxRequestFilesBytes`、`maxImagesPerRequest` 与两个量子)的请求,以 `IMAGE_OFFLOAD_REQUIRED` 失败并说明还需省略多少最老的出现位置,由 `dsh-llm-retry` 用带标记的副本替换承载节点并重试。可用时报告缓存读取用量。`totalTokens` 是精确的 `prompt_tokens + completion_tokens` 汇总值;提供方给出的 `total_tokens` 不一致时省略该值。 #### KV Cache 影响 diff --git a/packages/llm/llm-pi-ai/README.i18n.yaml b/packages/llm/llm-pi-ai/README.i18n.yaml index e926a274ee..75b1476aeb 100644 --- a/packages/llm/llm-pi-ai/README.i18n.yaml +++ b/packages/llm/llm-pi-ai/README.i18n.yaml @@ -2,5 +2,5 @@ # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: # pnpm run verify-translation-pairing --write packages/llm/llm-pi-ai/README.md -README.md: 56d68aa62fbb8497d4f6e34ded79d1ad27f41985 -README.zh.md: ff6734e57a4e2e725b4d45fee5b24aa01221cb21 +README.md: 326028b8c2f64dec2a49241752e2285308031de8 +README.zh.md: 502390b50de611f434e561f0567e5fc319fe7550 diff --git a/packages/llm/llm-pi-ai/README.md b/packages/llm/llm-pi-ai/README.md index 56d68aa62f..326028b8c2 100644 --- a/packages/llm/llm-pi-ai/README.md +++ b/packages/llm/llm-pi-ai/README.md @@ -176,7 +176,7 @@ Read these pages when the package-level contract is not enough. They move from t #### What the model sees -The selected catalog model receives `GenerateOptions.system`, history, tools, and sampling fields supported by pi-ai's common streaming API. Each retained image is preceded by text naming its complete attachment id and actual request dimensions. When the current execution filesystem maps the attachment provider's host object, the text also carries a read-only normalized-object path and warns that normalization or request projection may have resized or re-encoded the upload. Each occurrence the session's `image/offload` watermark marks offloaded keeps its own identity and currently resolved access in replacement text, and its normalized attachment is not read or transformed. When the retained occurrences' exact base64 payload still exceeds the route's `maxRequestImageBytes`, the call fails with `IMAGE_OFFLOAD_REQUIRED` so `dsh-llm-retry` advances the watermark and retries the step. Provider-native replay metadata is restored only when the adapter validates it for the historical content. +The selected catalog model receives `GenerateOptions.system`, history, tools, and sampling fields supported by pi-ai's common streaming API. Each retained image is preceded by text naming its complete attachment id and actual request dimensions. When the current execution filesystem maps the attachment provider's host object, the text also carries a read-only normalized-object path and warns that normalization or request projection may have resized or re-encoded the upload. Each occurrence a surface replacement marks offloaded keeps its own identity and currently resolved access in replacement text, and its normalized attachment is not read or transformed. When the retained occurrences' exact base64 payload still exceeds the route's `maxRequestImageBytes`, the call fails with `IMAGE_OFFLOAD_REQUIRED` so `dsh-llm-retry` replaces the carrying surface nodes with marked copies and retries the step. Provider-native replay metadata is restored only when the adapter validates it for the historical content. #### Token effect @@ -184,7 +184,7 @@ Provider tokenization governs exact input. Retained images add the stable attach #### KV Cache effect -Conversion preserves logical request order, while image handles and offload placeholders add model-visible text. A changed execution-world path rewrites a historical handle and can prevent reuse from that image even when attachment identity and request bytes stay stable. Changing adapter instance, provider, model, or another upstream token has the same suffix effect. An `image/offload` advance replaces an earlier image with placeholder text, so reuse ends at that message; the watermark never retreats, so the prefix stays stable afterwards. +Conversion preserves logical request order, while image handles and offload placeholders add model-visible text. A changed execution-world path rewrites a historical handle and can prevent reuse from that image even when attachment identity and request bytes stay stable. Changing adapter instance, provider, model, or another upstream token has the same suffix effect. An offload replacement turns an earlier image into placeholder text, so reuse ends at that message; the replacement never reverts, so the prefix stays stable afterwards. ### Provider response diff --git a/packages/llm/llm-pi-ai/README.zh.md b/packages/llm/llm-pi-ai/README.zh.md index ff6734e57a..502390b50d 100644 --- a/packages/llm/llm-pi-ai/README.zh.md +++ b/packages/llm/llm-pi-ai/README.zh.md @@ -176,7 +176,7 @@ pi-ai 不提供的路由需要 `api`、`baseURL` 与非空 `models` 列表;无 #### 模型看到什么 -所选目录模型会收到 `GenerateOptions.system`、历史、工具与 pi-ai 通用流式 API 支持的采样字段。每张保留图片前都会有文本,注明其完整附件 id 与实际请求尺寸。当前执行文件系统可以映射附件提供方的宿主对象时,该文本还会携带只读规范化对象路径,并警告规范化或请求投影可能缩放或重新编码上传内容。会话 `image/offload` 水位标记为已卸载的每个出现位置都会在替换文本中保留自己的身份与当前已解析访问方式,其规范化附件不会读取或变换。当保留的出现位置按精确 base64 载荷仍超过路由的 `maxRequestImageBytes` 时,调用以 `IMAGE_OFFLOAD_REQUIRED` 失败,由 `dsh-llm-retry` 推进水位并重试 step。提供方原生回放元数据只在适配器针对历史内容校验通过后恢复。 +所选目录模型会收到 `GenerateOptions.system`、历史、工具与 pi-ai 通用流式 API 支持的采样字段。每张保留图片前都会有文本,注明其完整附件 id 与实际请求尺寸。当前执行文件系统可以映射附件提供方的宿主对象时,该文本还会携带只读规范化对象路径,并警告规范化或请求投影可能缩放或重新编码上传内容。表层替换标记为已卸载的每个出现位置都会在替换文本中保留自己的身份与当前已解析访问方式,其规范化附件不会读取或变换。当保留的出现位置按精确 base64 载荷仍超过路由的 `maxRequestImageBytes` 时,调用以 `IMAGE_OFFLOAD_REQUIRED` 失败,由 `dsh-llm-retry` 用带标记的副本替换承载节点并重试 step。提供方原生回放元数据只在适配器针对历史内容校验通过后恢复。 #### Token 影响 @@ -184,7 +184,7 @@ pi-ai 不提供的路由需要 `api`、`baseURL` 与非空 `models` 列表;无 #### KV Cache 影响 -转换保持逻辑请求顺序,图片句柄与卸载占位符则会添加模型可见文本。即使附件身份与请求字节保持稳定,执行世界路径变化也会改写历史句柄,并可能从该图片起阻止复用。更换适配器实例、提供方、模型或其他上游 token 具有相同的后缀影响。一次 `image/offload` 推进会把较早图片替换为占位文本,因此复用在该消息处结束;水位永不回退,此后前缀保持稳定。 +转换保持逻辑请求顺序,图片句柄与卸载占位符则会添加模型可见文本。即使附件身份与请求字节保持稳定,执行世界路径变化也会改写历史句柄,并可能从该图片起阻止复用。更换适配器实例、提供方、模型或其他上游 token 具有相同的后缀影响。一次省略替换会把较早图片换成占位文本,因此复用在该消息处结束;替换永不回退,此后前缀保持稳定。 ### 提供方响应 diff --git a/packages/llm/llm-retry/README.i18n.yaml b/packages/llm/llm-retry/README.i18n.yaml index aa54d6047a..9084498243 100644 --- a/packages/llm/llm-retry/README.i18n.yaml +++ b/packages/llm/llm-retry/README.i18n.yaml @@ -2,5 +2,5 @@ # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: # pnpm run verify-translation-pairing --write packages/llm/llm-retry/README.md -README.md: e5c97f602b4db1450ae1e2566e34b40fc6c385fd -README.zh.md: 2c3c310c9cc86e439f57de8c2f3068cc26176312 +README.md: 52740300adb9e791fb6bcbcd1f4845e684f10667 +README.zh.md: 0855be01fa06e9f2b8194480dcfbb8faf2e54451 diff --git a/packages/llm/llm-retry/README.md b/packages/llm/llm-retry/README.md index e5c97f602b..52740300ad 100644 --- a/packages/llm/llm-retry/README.md +++ b/packages/llm/llm-retry/README.md @@ -9,7 +9,7 @@ English | [中文](README.zh.md) ## Summary -`@deepseek-ai/dsh-llm-retry` is the retry executor for failed model requests: it applies each provider's resolved retry policy at the agent loop's open-step `agent/request-error` extension point, so every retry re-runs the same step inside the same open turn over the same durable history. It does not wrap the streaming call itself — every adapter call remains one provider attempt, and direct `ctx.llm.stream()` consumers stay single-attempt. Retry scheduling is durable: the plugin appends `llm/retry` events to the session log before waiting, and cancellation during backoff leaves the log consistent. Normal mode retries a bounded set of failure codes up to `maxRetries` with exponential backoff; always mode asks downstream recovery first, then retries every failure without an attempt limit. It also owns one durable repair: an adapter's `IMAGE_OFFLOAD_REQUIRED` failure advances the session's `image/offload` watermark by the count the failure names and retries the step, outside any provider retry budget. +`@deepseek-ai/dsh-llm-retry` is the retry executor for failed model requests: it applies each provider's resolved retry policy at the agent loop's open-step `agent/request-error` extension point, so every retry re-runs the same step inside the same open turn over the same durable history. It does not wrap the streaming call itself — every adapter call remains one provider attempt, and direct `ctx.llm.stream()` consumers stay single-attempt. Retry scheduling is durable: the plugin appends `llm/retry` events to the session log before waiting, and cancellation during backoff leaves the log consistent. Normal mode retries a bounded set of failure codes up to `maxRetries` with exponential backoff; always mode asks downstream recovery first, then retries every failure without an attempt limit. It also owns one durable repair: an adapter's `IMAGE_OFFLOAD_REQUIRED` failure replaces the surface nodes carrying the count of oldest images the failure names with copies marked `offloaded`, then retries the step outside any provider retry budget. ## Table of Contents @@ -55,7 +55,7 @@ Each scheduled retry is durable before its wait: the plugin appends a non-surfac ### Failures and recovery -A failure before any final adapter is selected has no provider policy and delegates downstream unchanged. In normal mode, a failure code outside the eligible set, or an exhausted budget, delegates; in always mode, an over-cap provider delay uses the configured local backoff so the policy cannot terminate on that instruction. Nothing here is model-visible: no retry event, delay, provider error, or failed partial output reaches the model or derived messages. An `IMAGE_OFFLOAD_REQUIRED` failure carrying `offloadImages` is handled before any policy: the plugin appends one `image/offload` event naming the greatest durable position among that many oldest retained image occurrences in request order and returns `retry`; when nothing remains to offload it delegates. That retry spends no retry budget and logs no `llm/retry` event, and the retried request sends placeholder text for the offloaded images ([decision](../../../.agents/notes/implemented/architecture/2026-09-02-image-offload-watermark.md)). +A failure before any final adapter is selected has no provider policy and delegates downstream unchanged. In normal mode, a failure code outside the eligible set, or an exhausted budget, delegates; in always mode, an over-cap provider delay uses the configured local backoff so the policy cannot terminate on that instruction. Nothing here is model-visible: no retry event, delay, provider error, or failed partial output reaches the model or derived messages. An `IMAGE_OFFLOAD_REQUIRED` failure carrying `offloadImages` is handled before any policy: the plugin walks the surface in request order, replaces each node carrying one of that many oldest retained image occurrences with a copy whose occurrences are marked `offloaded` (a `surfaceOp: replace` like compaction's), and returns `retry`; when nothing remains to offload it delegates. That retry spends no retry budget and logs no `llm/retry` event, and the retried request sends placeholder text for the marked images ([decision](../../../.agents/notes/implemented/architecture/2026-09-02-durable-image-offload.md)). ----- @@ -77,13 +77,13 @@ The executor is built on one rule: **durable before wait, open-step boundaries.* |---|---| | [`src/index.ts`](src/index.ts) | The function plugin: waterfall listener, policy lookup, backoff, durable event appends | | [`src/history.ts`](src/history.ts) | Durable retry-history lookup from the session log | -| [`src/image-offload.ts`](src/image-offload.ts) | The `image/offload` advance for `IMAGE_OFFLOAD_REQUIRED` failures | +| [`src/image-offload.ts`](src/image-offload.ts) | The surface replacements that offload images for `IMAGE_OFFLOAD_REQUIRED` failures | | [`src/types.ts`](src/types.ts) | Browser-safe `llm/retry` and `llm/retry-started` event payload types | | [`src/brand.ts`](src/brand.ts) | The `RetryId` brand shared by the event payloads | ### Recovery flow -A failed step arrives on the waterfall with its provider and resolved policy. An `IMAGE_OFFLOAD_REQUIRED` failure naming `offloadImages` is repaired at once: the watermark advances and the step retries with no delay. Always mode settles downstream recovery first and honors a downstream `retry` decision; normal mode first checks that the failure code is eligible and the budget is not exhausted. The plugin computes the delay — provider `Retry-After` when valid and within bounds, otherwise local bounded exponential backoff with symmetric jitter — appends the `llm/retry` event, waits on a cancellable timer, appends `llm/retry-started`, and returns `{ kind: 'retry' }`. The loop then re-runs the failed step inside the same open turn over the same durable history. +A failed step arrives on the waterfall with its provider and resolved policy. An `IMAGE_OFFLOAD_REQUIRED` failure naming `offloadImages` is repaired at once: the carrying surface nodes are replaced and the step retries with no delay. Always mode settles downstream recovery first and honors a downstream `retry` decision; normal mode first checks that the failure code is eligible and the budget is not exhausted. The plugin computes the delay — provider `Retry-After` when valid and within bounds, otherwise local bounded exponential backoff with symmetric jitter — appends the `llm/retry` event, waits on a cancellable timer, appends `llm/retry-started`, and returns `{ kind: 'retry' }`. The loop then re-runs the failed step inside the same open turn over the same durable history. ### Waterfall composition diff --git a/packages/llm/llm-retry/README.zh.md b/packages/llm/llm-retry/README.zh.md index 2c3c310c9c..0855be01fa 100644 --- a/packages/llm/llm-retry/README.zh.md +++ b/packages/llm/llm-retry/README.zh.md @@ -9,7 +9,7 @@ kind: "package-reference" ## 概述 -`@deepseek-ai/dsh-llm-retry` 是失败模型请求的重试执行器:它在 agent loop 的打开步骤 `agent/request-error` 扩展点上应用各提供方解析后的重试策略,因此每次重试都会在同一个打开的轮次内重跑同一个步骤(基于同一份持久历史)。它不包装流式调用本身——每次适配器调用仍是一次提供方尝试,直接 `ctx.llm.stream()` 消费方仍是单次尝试。重试调度是持久的:插件在等待之前就把 `llm/retry` 事件追加进会话日志,退避期间取消会让日志保持一致。normal mode 以指数退避重试一组有界的失败 code,最多 `maxRetries` 次;always mode 先询问下游恢复,然后无尝试上限地重试每个失败。它还负责一种持久修复:adapter 以 `IMAGE_OFFLOAD_REQUIRED` 失败时,按失败给出的数量推进会话的 `image/offload` 水位并重试该步骤,不占任何提供方重试预算。 +`@deepseek-ai/dsh-llm-retry` 是失败模型请求的重试执行器:它在 agent loop 的打开步骤 `agent/request-error` 扩展点上应用各提供方解析后的重试策略,因此每次重试都会在同一个打开的轮次内重跑同一个步骤(基于同一份持久历史)。它不包装流式调用本身——每次适配器调用仍是一次提供方尝试,直接 `ctx.llm.stream()` 消费方仍是单次尝试。重试调度是持久的:插件在等待之前就把 `llm/retry` 事件追加进会话日志,退避期间取消会让日志保持一致。normal mode 以指数退避重试一组有界的失败 code,最多 `maxRetries` 次;always mode 先询问下游恢复,然后无尝试上限地重试每个失败。它还负责一种持久修复:adapter 以 `IMAGE_OFFLOAD_REQUIRED` 失败时,把承载失败所报数量的最老图片的表层节点替换为标了 `offloaded` 的副本,再重试该步骤,不占任何提供方重试预算。 ## 目录 @@ -55,7 +55,7 @@ kind: "package-reference" ### 失败与恢复 -在任何最终适配器被选中之前发生的失败没有提供方策略,原样委派下游。normal mode 中,不在合格集合内的失败 code 或已耗尽的预算会委派;always mode 中,超上限的提供方延迟使用配置的本地退避,因此策略不会因该指令终止。这里没有任何模型可见内容:重试事件、延迟、提供方错误或失败的部分输出都不会到达模型或派生消息。带 `offloadImages` 的 `IMAGE_OFFLOAD_REQUIRED` 失败在任何策略之前处理:插件追加一个 `image/offload` 事件,指向按请求顺序最老的那些保留图片中最大的持久位置,然后返回 `retry`;没有可省略的图片时委派下游。这次重试不占重试预算,也不追加 `llm/retry` 事件,重试的请求为被省略的图片发送占位文本([决定](../../../.agents/notes/implemented/architecture/2026-09-02-image-offload-watermark.zh.md))。 +在任何最终适配器被选中之前发生的失败没有提供方策略,原样委派下游。normal mode 中,不在合格集合内的失败 code 或已耗尽的预算会委派;always mode 中,超上限的提供方延迟使用配置的本地退避,因此策略不会因该指令终止。这里没有任何模型可见内容:重试事件、延迟、提供方错误或失败的部分输出都不会到达模型或派生消息。带 `offloadImages` 的 `IMAGE_OFFLOAD_REQUIRED` 失败在任何策略之前处理:插件按请求顺序遍历表层,把承载那些最老保留图片的每个节点替换为标了 `offloaded` 的副本(和 compaction 一样的 `surfaceOp: replace`),然后返回 `retry`;没有可省略的图片时委派下游。这次重试不占重试预算,也不追加 `llm/retry` 事件,重试的请求为被标记的图片发送占位文本([决定](../../../.agents/notes/implemented/architecture/2026-09-02-durable-image-offload.zh.md))。 ----- @@ -77,13 +77,13 @@ kind: "package-reference" |---|---| | [`src/index.ts`](src/index.ts) | 函数插件:waterfall 监听器、策略查找、退避、持久事件追加 | | [`src/history.ts`](src/history.ts) | 从会话日志查找持久重试历史 | -| [`src/image-offload.ts`](src/image-offload.ts) | 针对 `IMAGE_OFFLOAD_REQUIRED` 失败的 `image/offload` 推进 | +| [`src/image-offload.ts`](src/image-offload.ts) | 为 `IMAGE_OFFLOAD_REQUIRED` 失败省略图片的表层替换 | | [`src/types.ts`](src/types.ts) | 浏览器安全的 `llm/retry` 与 `llm/retry-started` 事件载荷类型 | | [`src/brand.ts`](src/brand.ts) | 事件载荷共享的 `RetryId` 品牌 | ### 恢复流程 -失败步骤连同其提供方与解析后的策略一起到达 waterfall。带 `offloadImages` 的 `IMAGE_OFFLOAD_REQUIRED` 失败会立即修复:推进水位,步骤无延迟重试。always mode 先结算下游恢复,并遵循下游的 `retry` 决定;normal mode 先检查失败 code 是否合格、预算是否未耗尽。插件计算延迟——有效且在边界内的提供方 `Retry-After`,否则带对称抖动的本地有界指数退避——追加 `llm/retry` 事件,在可取消定时器上等待,追加 `llm/retry-started`,然后返回 `{ kind: 'retry' }`。loop 随后在同一个打开的轮次内重跑失败步骤(仍基于同一份持久历史)。 +失败步骤连同其提供方与解析后的策略一起到达 waterfall。带 `offloadImages` 的 `IMAGE_OFFLOAD_REQUIRED` 失败会立即修复:替换承载节点,步骤无延迟重试。always mode 先结算下游恢复,并遵循下游的 `retry` 决定;normal mode 先检查失败 code 是否合格、预算是否未耗尽。插件计算延迟——有效且在边界内的提供方 `Retry-After`,否则带对称抖动的本地有界指数退避——追加 `llm/retry` 事件,在可取消定时器上等待,追加 `llm/retry-started`,然后返回 `{ kind: 'retry' }`。loop 随后在同一个打开的轮次内重跑失败步骤(仍基于同一份持久历史)。 ### Waterfall 组合 diff --git a/packages/llm/llm-retry/src/image-offload.ts b/packages/llm/llm-retry/src/image-offload.ts index ffa59dc6b9..2888bfc03c 100644 --- a/packages/llm/llm-retry/src/image-offload.ts +++ b/packages/llm/llm-retry/src/image-offload.ts @@ -1,44 +1,81 @@ /** - * Durable image offload recovery: advance the session's `image/offload` - * watermark past the oldest retained image occurrences an adapter's - * `IMAGE_OFFLOAD_REQUIRED` failure names, so the retried step derives the - * offloaded set the log records. + * Image offload recovery: replace the surface nodes carrying the oldest + * retained image occurrences with copies that mark them `offloaded`, so the + * retried request sends placeholder text for exactly the occurrences an + * adapter's `IMAGE_OFFLOAD_REQUIRED` failure named. The replacement is a + * durable surface fact: it never reverts, and every later request and + * measurement reads it from the log. * * @module @deepseek-ai/dsh-llm-retry/image-offload */ -import { visitImageBlocks } from '@deepseek-ai/dsh-llm' -import { compareImagePositions } from '@deepseek-ai/dsh-session' -import type { ImageOccurrencePosition, Session } from '@deepseek-ai/dsh-session' +import type { ContentBlock, ToolResultMessage } from '@deepseek-ai/dsh-llm' +import type { Session, SessionEvent } from '@deepseek-ai/dsh-session' + +/** Mark the first `remaining.count` retained occurrences, keeping unchanged blocks by identity. */ +function markOldestImages(blocks: readonly ContentBlock[], remaining: { count: number }): readonly ContentBlock[] { + let next: ContentBlock[] | undefined + for (const [index, block] of blocks.entries()) { + if (remaining.count > 0 && block.type === 'image' && block.offloaded !== true) { + remaining.count -= 1 + next ??= blocks.slice(0, index) + next.push({ ...block, offloaded: true }) + continue + } + if (remaining.count > 0 && block.type === 'tool-result') { + const content = markOldestImages(block.content, remaining) + if (content !== block.content) { + next ??= blocks.slice(0, index) + next.push({ ...block, content: content as ContentBlock[] }) + continue + } + } + next?.push(block) + } + return next ?? blocks +} + +/** The surface node types that carry model input images. */ +type ImageNode = SessionEvent<'user/message'> | SessionEvent<'tool/result'> + +/** Append the marked copy of one surface node in place of the original. */ +function replaceNode(session: Session, event: ImageNode, content: readonly ContentBlock[]): void { + const options = { + surfaceOp: { op: 'replace' as const, start: event.seq, end: event.seq }, + sourceEventSeqs: [event.seq], + } + if (event.type === 'user/message') { + session.append('user/message', { ...event.data, content: content as ContentBlock[] }, options) + } else { + session.append('tool/result', { + ...event.data, + message: { ...event.data.message, content: content as ToolResultMessage['content'] }, + }, options) + } +} /** - * Append one `image/offload` advance covering the `count` oldest image - * occurrences the surface still sends, in model request order. A surface - * replacement can place a newer event before older ones, so the advance - * names the greatest durable position in that prefix; it may offload - * additional occurrences but always covers the prefix. + * Offload the `count` oldest retained image occurrences, in model request + * order, by replacing each surface node that carries one with a copy whose + * occurrences are marked `offloaded`. Assistant nodes carry model output, not + * input images, and are skipped. * @param session - the session whose derived surface the retried request reads. - * @param turn - the open turn. - * @param step - the step being retried. * @param count - how many more oldest retained occurrences the adapter needs offloaded. * @returns whether any occurrence remained to offload. */ -export function advanceImageOffload(session: Session, turn: number, step: number, count: number): boolean { - let remaining = count - let watermark: ImageOccurrencePosition | undefined - for (const seq of session.surface.nodes) { - if (remaining === 0) break +export function offloadOldestImages(session: Session, count: number): boolean { + const remaining = { count } + let replaced = false + for (const seq of [...session.surface.nodes]) { + if (remaining.count === 0) break // oxlint-disable-next-line typescript/no-non-null-assertion -- surface nodes index the durable log - const message = session.deriveEventMessage(session.eventAt(seq)!) - if (message === null) continue - visitImageBlocks(message.content, (block, path) => { - if (remaining === 0 || block.offloaded === true) return - remaining -= 1 - const position = { seq, path: [...path] } - if (watermark === undefined || compareImagePositions(position, watermark) > 0) watermark = position - }) + const event = session.eventAt(seq)! + if (event.type !== 'user/message' && event.type !== 'tool/result') continue + const original = event.type === 'user/message' ? event.data.content : event.data.message.content + const content = markOldestImages(original, remaining) + if (content === original) continue + replaceNode(session, event, content) + replaced = true } - if (watermark === undefined) return false - session.append('image/offload', { turn, step, watermark }) - return true + return replaced } diff --git a/packages/llm/llm-retry/src/index.ts b/packages/llm/llm-retry/src/index.ts index 9bc7b2acb0..0971ff6263 100644 --- a/packages/llm/llm-retry/src/index.ts +++ b/packages/llm/llm-retry/src/index.ts @@ -14,7 +14,7 @@ import { IMAGE_OFFLOAD_REQUIRED_CODE } from '@deepseek-ai/dsh-llm' import type { LlmFailure, ResolvedRetryPolicy } from '@deepseek-ai/dsh-llm' import type {} from '@deepseek-ai/dsh-session-projection' import { RetryId } from './brand.ts' -import { advanceImageOffload } from './image-offload.ts' +import { offloadOldestImages } from './image-offload.ts' import type { LlmRetryEventData } from './types.ts' export type { LlmRetryEventData, LlmRetryStartedEventData } from './types.ts' @@ -199,7 +199,7 @@ export function apply(ctx: Context, config: Config = {}, internals: RetryInterna ): Promise { // A durable surface repair, not a provider retry: it spends no retry budget and logs no retry event. if (failure.code === IMAGE_OFFLOAD_REQUIRED_CODE && failure.offloadImages !== undefined - && advanceImageOffload(agent.session, turn, step, failure.offloadImages)) { + && offloadOldestImages(agent.session, failure.offloadImages)) { return { kind: 'retry' } } if (policy === undefined) return next() diff --git a/packages/llm/llm-retry/tests/image-offload.spec.ts b/packages/llm/llm-retry/tests/image-offload.spec.ts index 41f08d2b96..8a5be7f21e 100644 --- a/packages/llm/llm-retry/tests/image-offload.spec.ts +++ b/packages/llm/llm-retry/tests/image-offload.spec.ts @@ -1,16 +1,16 @@ /** * Image offload recovery: an adapter's `IMAGE_OFFLOAD_REQUIRED` failure - * advances the durable watermark by the named count and retries the step - * without spending the provider retry budget. + * replaces the surface nodes carrying the named count of oldest images with + * marked copies and retries the step without spending the provider retry budget. */ import { describe, expect, it } from 'vitest' import { Context } from '@deepseek-ai/cordis' import AgentLoop from '@deepseek-ai/dsh-agent-loop' import { mountAgentLoopTestDependencies } from '@deepseek-ai/dsh-agent-loop-testkit' -import { createAssistantMessage, createUserMessage, IMAGE_OFFLOAD_REQUIRED_CODE, LlmAdapter, LlmError, ToolCallId } from '@deepseek-ai/dsh-llm' +import { createAssistantMessage, createToolResultMessage, createUserMessage, IMAGE_OFFLOAD_REQUIRED_CODE, LlmAdapter, LlmError, ToolCallId } from '@deepseek-ai/dsh-llm' import type { ContentBlock, GenerateOptions, StreamChunk } from '@deepseek-ai/dsh-llm' -import { SessionId } from '@deepseek-ai/dsh-session' +import { isReplacementSurfaceEvent, SessionId, SessionSeq } from '@deepseek-ai/dsh-session' import type { Session } from '@deepseek-ai/dsh-session' import * as retry from '../src/index.ts' @@ -75,15 +75,11 @@ function offloadedNames(options: GenerateOptions): string[] { return names } -function offloadEvents(session: Session): { turn: number; step: number; seq: number; path: number[] }[] { +/** Surface replacements appended by the recovery, as `[original seq, replacement seq]` pairs. */ +function replacements(session: Session): [number, number][] { return session.snapshotEvents() - .filter(event => event.type === 'image/offload') - .map(event => ({ - turn: event.data.turn, - step: event.data.step, - seq: Number(event.data.watermark.seq), - path: event.data.watermark.path, - })) + .filter(isReplacementSurfaceEvent) + .map(event => [Number(event.sourceEventSeqs?.[0]), Number(event.seq)]) } describe('image offload recovery', () => { @@ -106,13 +102,20 @@ describe('image offload recovery', () => { expect(adapter.requests).toHaveLength(2) expect(offloadedNames(adapter.requests[0]!)).toEqual([]) expect(offloadedNames(adapter.requests[1]!)).toEqual(['a', 'b']) - expect(offloadEvents(agent.session)).toMatchObject([{ turn: 1, step: 1, path: [1, 0] }]) expect(delegated).toEqual([]) - const types = agent.session.snapshotEvents().map(event => event.type) + const events = agent.session.snapshotEvents() + const types = events.map(event => event.type) expect(types.filter(type => type === 'llm/retry')).toHaveLength(0) expect(types.filter(type => type === 'assistant/attempt')).toHaveLength(1) - expect(types.indexOf('assistant/attempt')).toBeLessThan(types.indexOf('image/offload')) - expect(types.indexOf('image/offload')).toBeLessThan(types.indexOf('assistant/message')) + // One node carried both occurrences, so one replacement lands between the failed attempt and the retry. + expect(replacements(agent.session)).toHaveLength(1) + const [original, replacement] = replacements(agent.session)[0]! + expect(events[original]).toMatchObject({ type: 'user/message', surfaceOp: 'append' }) + expect(types.indexOf('assistant/attempt')).toBeLessThan(replacement) + expect(replacement).toBeLessThan(types.indexOf('assistant/message')) + // The original event keeps its content; only the replacement carries the marks. + const durable = events[original]! + expect(durable.type === 'user/message' ? durable.data.content[0] : undefined).not.toHaveProperty('offloaded') }) it('counts the adapter prefix in request order after a surface replacement', async () => { @@ -144,8 +147,49 @@ describe('image offload recovery', () => { expect(adapter.requests).toHaveLength(2) expect(offloadedNames(adapter.requests[1]!)).toEqual(['replacement', 'second']) - // The replacement node carries the newest seq, so it is the greatest position in the prefix. - expect(offloadEvents(agent.session).at(-1)).toMatchObject({ seq: 3, path: [0] }) + // Both nodes carried one occurrence, so the recovery replaced the replacement node and 'second'. + expect(replacements(agent.session).slice(1).map(([original]) => original)).toEqual([3, 2]) + }) + + it('replaces a tool result node and leaves blocks after the count untouched', async () => { + const adapter = new ScriptedAdapter([offloadRequired(1), textResponse('sent')]) + const ctx = await harness(adapter) + const agent = await ctx.agentLoop.create(SessionId('offload-tool-result'), { provider: 'mock', model: 'mock' }) + const callId = ToolCallId('shot') + agent.session.append('turn/start', { turn: 0 }) + agent.session.append('assistant/message', { + turn: 0, + step: 1, + message: createAssistantMessage({ + content: [{ type: 'tool-call', id: callId, name: 'read_image', arguments: '{}' }], + source: { provider: 'mock', model: 'mock' }, + }), + stream: [], + }, { surfaceOp: 'append' }) + agent.session.append('tool/call', { turn: 0, step: 1, callId, name: 'read_image', arguments: '{}' }) + const result = agent.session.append('tool/result', { + turn: 0, + step: 1, + message: createToolResultMessage({ + callId, + content: [ + { type: 'tool-result', toolCallId: ToolCallId('empty'), content: [{ type: 'text', text: 'no image' }] }, + image('first'), + { type: 'tool-result', toolCallId: ToolCallId('inner'), content: [image('second')] }, + ], + isError: false, + }), + }, { surfaceOp: 'append' }) + agent.session.append('turn/end', { turn: 0, reason: { kind: 'completed' } }) + + agent.followup(createUserMessage({ content: [{ type: 'text', text: 'send' }], source: { kind: 'user' } })) + await agent.whenIdle() + + expect(offloadedNames(adapter.requests[1]!)).toEqual(['first']) + const [replacement] = replacements(agent.session) + expect(replacement).toEqual([Number(result.seq), expect.any(Number) as never]) + const replaced = agent.session.eventAt(SessionSeq(replacement![1]))! + expect(replaced.type === 'tool/result' ? replaced.data.message.source.callId : undefined).toBe(callId) }) it('delegates IMAGE_OFFLOAD_REQUIRED once nothing remains to offload', async () => { @@ -160,7 +204,7 @@ describe('image offload recovery', () => { agent.followup(createUserMessage({ content: [{ type: 'text', text: 'no images' }], source: { kind: 'user' } })) await agent.whenIdle() expect(delegated).toEqual([IMAGE_OFFLOAD_REQUIRED_CODE]) - expect(offloadEvents(agent.session)).toHaveLength(0) + expect(replacements(agent.session)).toHaveLength(0) expect(agent.session.snapshotEvents().at(-1)).toMatchObject({ type: 'turn/end', data: { reason: { kind: 'error' } } }) }) }) diff --git a/packages/llm/llm/README.i18n.yaml b/packages/llm/llm/README.i18n.yaml index e4c929a014..5c59f00429 100644 --- a/packages/llm/llm/README.i18n.yaml +++ b/packages/llm/llm/README.i18n.yaml @@ -2,5 +2,5 @@ # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: # pnpm run verify-translation-pairing --write packages/llm/llm/README.md -README.md: 0fec0b06c9e9d7725e9efe6b98e27ccae1cb1b65 -README.zh.md: 4b1d9738769f3568c66d2bad5a124f0087eaced3 +README.md: 398c6e17d5b5da631a48d510844b2156a8f0579d +README.zh.md: 83293a94d5e5560ef1f5efeee4d4c3db038e9be4 diff --git a/packages/llm/llm/README.md b/packages/llm/llm/README.md index 0fec0b06c9..398c6e17d5 100644 --- a/packages/llm/llm/README.md +++ b/packages/llm/llm/README.md @@ -95,13 +95,13 @@ The service is built on one separation: **the logical contract is provider-neutr | [`src/call-config.ts`](src/call-config.ts) | Call-config validation, adapter-default materialization, and request freezing | | [`src/retry-policy.ts`](src/retry-policy.ts) | Provider-owned retry policy resolution (normal and always modes) | | [`src/error.ts`](src/error.ts) | `HarnessError`/`LlmError` taxonomy and provider-neutral failure codes | -| [`src/content.ts`](src/content.ts) | Shared image-content helpers: the image walk, offload watermark planning, and offloaded-image projection | +| [`src/content.ts`](src/content.ts) | Shared image-content helpers: the image walk, offload accounting, and offloaded-image projection | | [`src/api-key.ts`](src/api-key.ts) | Credential format check shared by every adapter | | [`src/adapter-failure.ts`](src/adapter-failure.ts) | Failure normalization into terminal finish chunks | ### Main flow -A request is validated against its exact model's capability — context window, output default, reasoning efforts, and input modalities — and any adapter-configured defaults are materialized, then the whole request is deep-frozen. `prepareCall()` binds those facts, detached context, and retry policy to the exact adapter generation that performs terminal dispatch, so HMR or dynamic settings cannot combine one generation's image capability with another generation's endpoint. An image-capable adapter projects durable references into route-specific request versions; `resolveImageAttachmentAccess()` separately maps an attachment provider's optional host object into the current tool execution world without changing the request image or its `variantId`. A text-only route receives deterministic per-image placeholders, including nested tool-result images, without rewriting append-only session history. Durable `FileBlock` references never reach any adapter: request assembly replaces each one, nested tool-result occurrences included, with deterministic handle text naming the file and its saved read-only path, resolved through the mounted attachment and filesystem providers. `ctx.llm.fileRequestText(ref)` exposes that exact synchronous projection to request measurement. Surface derivation marks occurrences at or before the session's durable `image/offload` watermark `offloaded: true`, and `projectOffloadedImages()` renders those marks as route-owned placeholder text. An image-capable route whose retained occurrences exceed its `LlmImageRequestBudget` at their exact bytes fails with `IMAGE_OFFLOAD_REQUIRED` naming the additional oldest occurrences (`requiredImageOffload()`), never with an unlogged projection; `dsh-llm-retry` advances the watermark and retries. Adapters that charge visual tokens declare per-route `imageRequestPricing`, which `ctx.llm.imageRequestPricing(provider, model)` resolves synchronously for the token meter. Dispatch goes through the `llm/stream` waterfall, then chunks return as token-level deltas and every adapter outcome reaches the consumer as one terminal `finish` chunk. +A request is validated against its exact model's capability — context window, output default, reasoning efforts, and input modalities — and any adapter-configured defaults are materialized, then the whole request is deep-frozen. `prepareCall()` binds those facts, detached context, and retry policy to the exact adapter generation that performs terminal dispatch, so HMR or dynamic settings cannot combine one generation's image capability with another generation's endpoint. An image-capable adapter projects durable references into route-specific request versions; `resolveImageAttachmentAccess()` separately maps an attachment provider's optional host object into the current tool execution world without changing the request image or its `variantId`. A text-only route receives deterministic per-image placeholders, including nested tool-result images, without rewriting append-only session history. Durable `FileBlock` references never reach any adapter: request assembly replaces each one, nested tool-result occurrences included, with deterministic handle text naming the file and its saved read-only path, resolved through the mounted attachment and filesystem providers. `ctx.llm.fileRequestText(ref)` exposes that exact synchronous projection to request measurement. An image occurrence a surface replacement marked `offloaded: true` reaches every route as placeholder text through `projectOffloadedImages()`. An image-capable route whose retained occurrences exceed its `LlmImageRequestBudget` at their exact bytes fails with `IMAGE_OFFLOAD_REQUIRED` naming the additional oldest occurrences (`requiredImageOffload()`), never with an unlogged projection; `dsh-llm-retry` replaces the carrying surface nodes with marked copies and retries. Adapters that charge visual tokens declare per-route `imageRequestPricing`, which `ctx.llm.imageRequestPricing(provider, model)` resolves synchronously for the token meter. Dispatch goes through the `llm/stream` waterfall, then chunks return as token-level deltas and every adapter outcome reaches the consumer as one terminal `finish` chunk. ### Invariants @@ -139,7 +139,7 @@ None, as the LLM service adds no content; adapters choose when to add the shared #### KV Cache effect -Reasoning-effort materialization preserves the assembled request prefix. Image identity and request-preview text are deterministic, while an optional execution-world path is resolved for each request; a changed path or an `image/offload` advance can prevent reuse from that image. +Reasoning-effort materialization preserves the assembled request prefix. Image identity and request-preview text are deterministic, while an optional execution-world path is resolved for each request; a changed path or an offload replacement can prevent reuse from that image. ## Known Limitations and Deferred Work diff --git a/packages/llm/llm/README.zh.md b/packages/llm/llm/README.zh.md index 4b1d973876..83293a94d5 100644 --- a/packages/llm/llm/README.zh.md +++ b/packages/llm/llm/README.zh.md @@ -95,13 +95,13 @@ for await (const chunk of ctx.llm.stream({ | [`src/call-config.ts`](src/call-config.ts) | 调用配置校验、适配器默认值填入与请求冻结 | | [`src/retry-policy.ts`](src/retry-policy.ts) | 提供方自有重试策略解析(normal 与 always 模式) | | [`src/error.ts`](src/error.ts) | `HarnessError`/`LlmError` 分类体系与提供方无关失败 code | -| [`src/content.ts`](src/content.ts) | 共享图片内容辅助函数:图片遍历、卸载水位规划与已卸载图片投影 | +| [`src/content.ts`](src/content.ts) | 共享图片内容辅助函数:图片遍历、省略计数与已省略图片投影 | | [`src/api-key.ts`](src/api-key.ts) | 每个适配器共享的凭据格式校验 | | [`src/adapter-failure.ts`](src/adapter-failure.ts) | 把失败归一化为终止 finish 分片 | ### 主流程 -请求会对照其精确模型的能力——上下文窗口、输出默认值、推理强度与输入模态——校验,填入任何适配器配置的默认值,然后整个请求被深度冻结。`prepareCall()` 把这些事实、分离的上下文与重试策略绑定到执行最终分发的精确适配器代次,因此 HMR 或动态设置无法把一个代次的图片能力与另一代次的端点混用。支持图片的适配器把持久引用投影为路由专用请求版本;`resolveImageAttachmentAccess()` 会单独把附件提供方的可选宿主对象映射进当前工具执行世界,而不改变请求图片或其 `variantId`。纯文本路由接收确定性的逐图片占位符,包括嵌套工具结果图片,而不会改写仅追加会话历史。持久 `FileBlock` 引用永远不会到达任何适配器:请求组装把每个引用(包括嵌套工具结果中的出现)替换为确定性 handle 文本,指出文件与其只读保存路径,路径经由挂载的附件与文件系统提供方解析。`ctx.llm.fileRequestText(ref)` 向请求计量公开相同的同步投影。表层派生把位于会话持久 `image/offload` 水位及之前的出现位置标为 `offloaded: true`,`projectOffloadedImages()` 把这些标记渲染为路由所属的占位文本。支持图片的路由在保留的出现位置按精确字节超过其 `LlmImageRequestBudget` 时,以 `IMAGE_OFFLOAD_REQUIRED` 失败并说明还需省略多少最老的出现位置(`requiredImageOffload()`),绝不发送未记录的投影;`dsh-llm-retry` 推进水位并重试。对视觉 token 收费的适配器声明按路由的 `imageRequestPricing`,`ctx.llm.imageRequestPricing(provider, model)` 为 token meter 同步解析它。分发经过 `llm/stream` waterfall,随后分片以 token 级增量返回,每个适配器结果都以唯一一个终止 `finish` 分片到达消费方。 +请求会对照其精确模型的能力——上下文窗口、输出默认值、推理强度与输入模态——校验,填入任何适配器配置的默认值,然后整个请求被深度冻结。`prepareCall()` 把这些事实、分离的上下文与重试策略绑定到执行最终分发的精确适配器代次,因此 HMR 或动态设置无法把一个代次的图片能力与另一代次的端点混用。支持图片的适配器把持久引用投影为路由专用请求版本;`resolveImageAttachmentAccess()` 会单独把附件提供方的可选宿主对象映射进当前工具执行世界,而不改变请求图片或其 `variantId`。纯文本路由接收确定性的逐图片占位符,包括嵌套工具结果图片,而不会改写仅追加会话历史。持久 `FileBlock` 引用永远不会到达任何适配器:请求组装把每个引用(包括嵌套工具结果中的出现)替换为确定性 handle 文本,指出文件与其只读保存路径,路径经由挂载的附件与文件系统提供方解析。`ctx.llm.fileRequestText(ref)` 向请求计量公开相同的同步投影。表层替换标为 `offloaded: true` 的图片出现位置,经 `projectOffloadedImages()` 以占位文本到达每条路由。支持图片的路由在保留的出现位置按精确字节超过其 `LlmImageRequestBudget` 时,以 `IMAGE_OFFLOAD_REQUIRED` 失败并说明还需省略多少最老的出现位置(`requiredImageOffload()`),绝不发送未记录的投影;`dsh-llm-retry` 用带标记的副本替换承载这些图片的表层节点并重试。对视觉 token 收费的适配器声明按路由的 `imageRequestPricing`,`ctx.llm.imageRequestPricing(provider, model)` 为 token meter 同步解析它。分发经过 `llm/stream` waterfall,随后分片以 token 级增量返回,每个适配器结果都以唯一一个终止 `finish` 分片到达消费方。 ### 不变式 @@ -139,7 +139,7 @@ for await (const chunk of ctx.llm.stream({ #### KV Cache 影响 -推理强度的具体化会保留已组装请求前缀。图片身份与请求预览文本是确定性的,可选执行世界路径则按请求解析;路径变化或一次 `image/offload` 推进可能从该图片起阻止复用。 +推理强度的具体化会保留已组装请求前缀。图片身份与请求预览文本是确定性的,可选执行世界路径则按请求解析;路径变化或一次省略替换可能从该图片起阻止复用。 ## 已知限制与延期工作 diff --git a/packages/llm/llm/src/content.ts b/packages/llm/llm/src/content.ts index f0a513b45f..33ea56d7b5 100644 --- a/packages/llm/llm/src/content.ts +++ b/packages/llm/llm/src/content.ts @@ -205,37 +205,18 @@ function base64Length(bytes: number): number { return Math.ceil(bytes / 3) * 4 } -/** - * Block index path of one image occurrence inside message content. Each - * nested tool-result contributes another index. Paths compare - * lexicographically in message order. - */ -export type ImageBlockPath = readonly number[] - -function visitImageBlocksAt( - content: readonly ContentBlock[], - prefix: ImageBlockPath, - visit: (block: ImageBlock, path: ImageBlockPath) => void, -): void { - for (const [index, block] of content.entries()) { - const path = [...prefix, index] - if (block.type === 'image') visit(block, path) - else if (block.type === 'tool-result') visitImageBlocksAt(block.content, path, visit) - } -} - /** * Visit every image occurrence of typed content in message order, including - * nested tool-result content, with the block path that identifies it. Budget - * planning, adapter validation, and pricing share this request-content walk. + * nested tool-result content. Budget accounting, adapter validation, and + * pricing share this request-content walk. * @param content - typed model content blocks. - * @param visit - called once per occurrence with the block and its path. + * @param visit - called once per occurrence. */ -export function visitImageBlocks( - content: readonly ContentBlock[], - visit: (block: ImageBlock, path: ImageBlockPath) => void, -): void { - visitImageBlocksAt(content, [], visit) +export function visitImageBlocks(content: readonly ContentBlock[], visit: (block: ImageBlock) => void): void { + for (const block of content) { + if (block.type === 'image') visit(block) + else if (block.type === 'tool-result') visitImageBlocks(block.content, visit) + } } /** Replace every offloaded occurrence, including nested tool results, with its placeholder. */ diff --git a/packages/llm/llm/src/error.ts b/packages/llm/llm/src/error.ts index e031fcf970..6c5f7ad5b4 100644 --- a/packages/llm/llm/src/error.ts +++ b/packages/llm/llm/src/error.ts @@ -163,10 +163,10 @@ export function isHarnessError(value: unknown): value is HarnessError { } /** - * Canonical code for a request an image-capable route cannot send until the - * session's durable `image/offload` watermark advances. The failure's - * `offloadImages` names how many more of the oldest retained occurrences must - * be offloaded; `dsh-llm-retry` appends the advance and retries the step, so - * the model never receives an unlogged projection. + * Canonical code for a request an image-capable route cannot send until more + * of its images are offloaded. The failure's `offloadImages` names how many + * more of the oldest retained occurrences must be offloaded; `dsh-llm-retry` + * replaces the surface nodes carrying them with copies marked `offloaded` and + * retries the step, so the model never receives an unlogged projection. */ export const IMAGE_OFFLOAD_REQUIRED_CODE = 'IMAGE_OFFLOAD_REQUIRED' diff --git a/packages/llm/llm/src/types.ts b/packages/llm/llm/src/types.ts index fd8c86f8bc..b0df82c35b 100644 --- a/packages/llm/llm/src/types.ts +++ b/packages/llm/llm/src/types.ts @@ -51,8 +51,9 @@ export interface LlmFailure { /** * With code `IMAGE_OFFLOAD_REQUIRED`: how many more of the oldest retained * image occurrences the route needs offloaded before the same request fits - * its exact byte accounting. `dsh-llm-retry` advances the session's durable - * `image/offload` watermark by this count and retries the step. + * its exact byte accounting. `dsh-llm-retry` replaces the surface nodes + * carrying that many oldest retained occurrences with copies marked + * `offloaded` and retries the step. */ readonly offloadImages?: number } @@ -80,9 +81,9 @@ export interface ImageBlock { /** Immutable bytes and intrinsic display metadata owned by the attachment service. */ attachment: ImageAttachmentRef /** - * Set by surface derivation when the occurrence lies at or before the - * session's durable `image/offload` watermark: every route sends its - * placeholder text instead of the image. Never stored in a logged message. + * Set on a surface replacement once the occurrence is offloaded from + * requests: every route sends its placeholder text, which names the image + * and its read-only path, instead of the image. */ offloaded?: true } diff --git a/packages/llm/llm/tests/content.spec.ts b/packages/llm/llm/tests/content.spec.ts index 9d2c3a1ff7..cf22656a99 100644 --- a/packages/llm/llm/tests/content.spec.ts +++ b/packages/llm/llm/tests/content.spec.ts @@ -15,7 +15,7 @@ import { requestImageHandleText, visitImageBlocks, } from '../src/index.ts' -import type { ContentBlock, ImageBlockPath } from '../src/index.ts' +import type { ContentBlock } from '../src/index.ts' const source = { kind: 'plugin' as const, plugin: 'test' } @@ -36,8 +36,8 @@ function image(bytes: number, offloaded?: true): Extract { - it('visits image occurrences at every tool-result depth with their block paths', () => { - const seen: ImageBlockPath[] = [] + it('visits image occurrences at every tool-result depth in message order', () => { + const seen: number[] = [] visitImageBlocks([ { type: 'text', text: 'before' }, image(1), @@ -55,8 +55,8 @@ describe('visitImageBlocks', () => { ], }, image(3), - ], (block, path) => seen.push([...path, block.attachment.bytes])) - expect(seen).toEqual([[1, 1], [2, 1, 2], [2, 2, 0, 0, 4], [3, 3]]) + ], block => seen.push(block.attachment.bytes)) + expect(seen).toEqual([1, 2, 4, 3]) }) }) diff --git a/packages/llm/token-meter/README.i18n.yaml b/packages/llm/token-meter/README.i18n.yaml index a1cfd47b79..9b88f82260 100644 --- a/packages/llm/token-meter/README.i18n.yaml +++ b/packages/llm/token-meter/README.i18n.yaml @@ -2,5 +2,5 @@ # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: # pnpm run verify-translation-pairing --write packages/llm/token-meter/README.md -README.md: 9bd737a1ffc9fe34cc1fa9819bd1676da996cb49 -README.zh.md: 0588ce576677e2fd6b7be1fe2a76154053c1a30d +README.md: 85108bf138f3148898121935f78021fa597a23f5 +README.zh.md: 73707d6198022dc2d7d960f5e67978da426527f2 diff --git a/packages/llm/token-meter/README.md b/packages/llm/token-meter/README.md index 9bd737a1ff..85108bf138 100644 --- a/packages/llm/token-meter/README.md +++ b/packages/llm/token-meter/README.md @@ -40,7 +40,7 @@ const { totalTokens, surfaceTokens, nodes } = ctx.tokenMeter.measure(session) const price = ctx.tokenMeter.estimateMessage(message) ``` -Each measurement resolves the effective envelope's provider/model through the optional `llm` service. Image occurrences use the routed request's visual-token price plus model-visible text when the adapter declares pricing, and occurrences at or before the session's durable `image/offload` watermark are priced as the route's placeholder text exactly as the derived surface sends them; other routes keep the fixed heuristic. File occurrences use the exact route-independent handle text that the same `llm` service resolves for adapter dispatch, including its current execution-world path or explicit no-path message. Each node also carries route-independent `heuristicTokens` for replacement shadow prices. Provider usage is reused only when the latest successful call's canonical request envelope matches the measured envelope and its total is no lower than that call's full route-priced anchor; otherwise the complete current envelope and surface are estimated. Surface changes stay signed relative to a matching anchor repriced under the same route, including negative deltas after shrinking replacements. +Each measurement resolves the effective envelope's provider/model through the optional `llm` service. Image occurrences use the routed request's visual-token price plus model-visible text when the adapter declares pricing, and occurrences a surface replacement marked `offloaded` are priced as the route's placeholder text exactly as the surface sends them; other routes keep the fixed heuristic. File occurrences use the exact route-independent handle text that the same `llm` service resolves for adapter dispatch, including its current execution-world path or explicit no-path message. Each node also carries route-independent `heuristicTokens` for replacement shadow prices. Provider usage is reused only when the latest successful call's canonical request envelope matches the measured envelope and its total is no lower than that call's full route-priced anchor; otherwise the complete current envelope and surface are estimated. Surface changes stay signed relative to a matching anchor repriced under the same route, including negative deltas after shrinking replacements. ### Session projections diff --git a/packages/llm/token-meter/README.zh.md b/packages/llm/token-meter/README.zh.md index 0588ce5766..73707d6198 100644 --- a/packages/llm/token-meter/README.zh.md +++ b/packages/llm/token-meter/README.zh.md @@ -40,7 +40,7 @@ const { totalTokens, surfaceTokens, nodes } = ctx.tokenMeter.measure(session) const price = ctx.tokenMeter.estimateMessage(message) ``` -每次测量都会通过可选的 `llm` 服务解析生效 envelope 的提供方/模型。适配器声明图片定价时,图片出现处使用路由请求的视觉 token 价格加模型可见文本,位于会话持久 `image/offload` 水位及之前的出现位置则按路由的占位文本计价,与派生表层实际发送的一致;其他路由保持固定启发式规则。文件出现处使用同一个 `llm` 服务为适配器分发解析的确切、与路由无关的 handle 文本,其中包含当前执行世界路径或明确的无路径说明。每个节点还携带与路由无关的 `heuristicTokens`,供替换影子价使用。只有当最新成功调用的规范请求 envelope 与已测量 envelope 匹配、且其总量不低于该调用完整路由定价锚点时,才复用提供方用量;否则会对完整当前 envelope 与表面做估算。表面变更保持相对于按同一路由重新定价的匹配锚点的带符号值,包括缩减替换后的负 delta。 +每次测量都会通过可选的 `llm` 服务解析生效 envelope 的提供方/模型。适配器声明图片定价时,图片出现处使用路由请求的视觉 token 价格加模型可见文本,表层替换标为已省略的出现位置则按路由的占位文本计价,与表层实际发送的一致;其他路由保持固定启发式规则。文件出现处使用同一个 `llm` 服务为适配器分发解析的确切、与路由无关的 handle 文本,其中包含当前执行世界路径或明确的无路径说明。每个节点还携带与路由无关的 `heuristicTokens`,供替换影子价使用。只有当最新成功调用的规范请求 envelope 与已测量 envelope 匹配、且其总量不低于该调用完整路由定价锚点时,才复用提供方用量;否则会对完整当前 envelope 与表面做估算。表面变更保持相对于按同一路由重新定价的匹配锚点的带符号值,包括缩减替换后的负 delta。 ### 会话投影 diff --git a/packages/llm/token-meter/src/index.ts b/packages/llm/token-meter/src/index.ts index b7ea85076b..e28476d61a 100644 --- a/packages/llm/token-meter/src/index.ts +++ b/packages/llm/token-meter/src/index.ts @@ -10,7 +10,6 @@ import { assembleAssistantStream } from '@deepseek-ai/dsh-llm' import type { LlmImageRequestPricing, LlmRuntime, Message, TokenUsage } from '@deepseek-ai/dsh-llm' import { deepFreeze } from '@deepseek-ai/dsh-util-values' import type { - ImageOccurrencePosition, EpochHeader, Session, SessionEvent, @@ -53,8 +52,6 @@ interface MeasurementAnchor { readonly header: EpochHeader | undefined /** Surface snapshot the anchored request was derived from. */ readonly nodes: readonly MeterSurfaceNode[] - /** Offload watermark the anchored request was derived under. */ - readonly watermark: ImageOccurrencePosition | undefined /** Fixed-heuristic price of the call's provider output. */ readonly assistantTokens: number /** Provider usage of the call, when it reported one under a known header. */ @@ -65,8 +62,6 @@ interface ReplayState { consumedEvents: SessionLogOffsetType header: EpochHeader | undefined surface: MeterSurfaceNode[] - /** Durable `image/offload` watermark in force after the consumed events. */ - watermark: ImageOccurrencePosition | undefined stepStart: { turn: number; step: number; nodes: readonly MeterSurfaceNode[] } | undefined anchor: MeasurementAnchor | undefined } @@ -153,7 +148,7 @@ export class TokenMeter extends Service { : canonicalHeader(requestHeader) const pricing = this._routeImagePricing(header) const fileText = this._fileRequestText() - const surface = priceSurface(state.surface, pricing, fileText, state.watermark) + const surface = priceSurface(state.surface, pricing, fileText) const anchor = state.anchor let baseline: TokenMeasurementBaseline @@ -162,7 +157,7 @@ export class TokenMeter extends Service { // Matching headers share one route, so the anchored snapshot reprices // under the same pricing as the current surface and the signed delta // compares like with like. - const anchorSurfaceTokens = priceSurface(anchor.nodes, pricing, fileText, anchor.watermark).surfaceTokens + const anchorSurfaceTokens = priceSurface(anchor.nodes, pricing, fileText).surfaceTokens + anchor.assistantTokens const estimatedAnchorTokens = estimateHeader(header) + anchorSurfaceTokens const usage = anchor.usage @@ -226,7 +221,6 @@ export class TokenMeter extends Service { consumedEvents: SessionLogOffset(0), header: undefined, surface: [], - watermark: undefined, stepStart: undefined, anchor: undefined, } @@ -249,7 +243,6 @@ export class TokenMeter extends Service { */ private _foldEvent(state: ReplayState, event: SessionEvent): void { let nextHeader = state.header - let nextWatermark = state.watermark let nextStepStart = state.stepStart let nextAnchor = state.anchor @@ -257,9 +250,6 @@ export class TokenMeter extends Service { case 'request/header': nextHeader = canonicalHeader(event.data.header) break - case 'image/offload': - nextWatermark = event.data.watermark - break case 'step/start': if (state.stepStart !== undefined) { throw new Error( @@ -299,7 +289,6 @@ export class TokenMeter extends Service { nextAnchor = { header: nextHeader, nodes: stepStart.nodes, - watermark: nextWatermark, assistantTokens: this._estimateProviderAssistant(event), usage: event.data.usage, } @@ -307,7 +296,6 @@ export class TokenMeter extends Service { nextAnchor = { header: nextHeader, nodes: stepStart.nodes, - watermark: nextWatermark, assistantTokens: eventTokens, usage: undefined, } @@ -315,7 +303,6 @@ export class TokenMeter extends Service { } state.header = nextHeader - state.watermark = nextWatermark state.stepStart = nextStepStart if (plan !== undefined) { commitSurfaceTokens(state.surface, plan) diff --git a/packages/llm/token-meter/src/route-pricing.ts b/packages/llm/token-meter/src/route-pricing.ts index 7ca6156067..03a1c519c3 100644 --- a/packages/llm/token-meter/src/route-pricing.ts +++ b/packages/llm/token-meter/src/route-pricing.ts @@ -5,9 +5,7 @@ * @module @deepseek-ai/dsh-token-meter/route-pricing */ -import type { ContentBlock, ImageBlock, LlmImageRequestPricing } from '@deepseek-ai/dsh-llm' -import { compareImagePositions } from '@deepseek-ai/dsh-session' -import type { ImageOccurrencePosition } from '@deepseek-ai/dsh-session' +import type { ContentBlock, LlmImageRequestPricing } from '@deepseek-ai/dsh-llm' import { estimateContent } from './estimate.ts' import type { MeterSurfaceNode } from './surface-fold.ts' import type { TokenSurfaceNode } from './types.ts' @@ -23,13 +21,10 @@ export interface PricedSurface { } /** - * Price one ordered surface under its model-request attachment projection. Every - * occurrence positioned at or before `watermark` is priced as the route's - * offloaded placeholder, exactly as the derived surface sends it. + * Price one ordered surface under its model-request attachment projection. * @param nodes - the fold's current or snapshotted surface, in model-visible order. * @param pricing - the routed model's image pricing, or undefined to keep the fixed heuristic. * @param fileText - exact file handle projection used by the mounted LLM service. - * @param watermark - the durable `image/offload` watermark in force for this surface, if any. * @returns detached public nodes and their route-priced total. * @throws when the pricing answers a different occurrence count than it was * asked — misalignment would silently misprice nodes, so it must fail loud. @@ -37,16 +32,9 @@ export interface PricedSurface { export function priceSurface( nodes: readonly MeterSurfaceNode[], pricing: LlmImageRequestPricing | undefined, - fileText: ((ref: FileAttachmentRef) => string) | undefined, - watermark?: ImageOccurrencePosition, + fileText?: (ref: FileAttachmentRef) => string, ): PricedSurface { - const images: ImageBlock[] = pricing === undefined - ? [] - : nodes.flatMap(node => node.images.map(({ attachment, path }) => ( - watermark !== undefined && compareImagePositions({ seq: node.seq, path: [...path] }, watermark) <= 0 - ? { type: 'image' as const, attachment, offloaded: true as const } - : { type: 'image' as const, attachment } - ))) + const images = pricing === undefined ? [] : nodes.flatMap(node => node.images) const hasFiles = fileText !== undefined && nodes.some(node => node.files.length > 0) if ((pricing === undefined || images.length === 0) && !hasFiles) { let surfaceTokens = 0 diff --git a/packages/llm/token-meter/src/surface-fold.ts b/packages/llm/token-meter/src/surface-fold.ts index 6ce0b8fcf2..af51e66183 100644 --- a/packages/llm/token-meter/src/surface-fold.ts +++ b/packages/llm/token-meter/src/surface-fold.ts @@ -18,19 +18,9 @@ import { deriveEventMessage } from '@deepseek-ai/dsh-session' import type { SessionSeq, SurfaceEvent } from '@deepseek-ai/dsh-session' -import { visitImageBlocks } from '@deepseek-ai/dsh-llm' -import type { ContentBlock, ImageBlockPath, Message } from '@deepseek-ai/dsh-llm' -import type { ImageAttachmentRef } from '@deepseek-ai/dsh-attachment' +import type { ContentBlock, ImageBlock, Message } from '@deepseek-ai/dsh-llm' import { estimateMessage, estimateStructuralBlock } from './estimate.ts' -/** One durable image occurrence of a surface node, positioned for the offload watermark. */ -export interface MeterImageOccurrence { - /** Durable normalized attachment reference of the occurrence. */ - readonly attachment: ImageAttachmentRef - /** Block path of the occurrence inside the node's message content. */ - readonly path: ImageBlockPath -} - type FileAttachmentRef = Extract['attachment'] /** One priced surface node with the image occurrences route pricing replaces. */ @@ -43,8 +33,8 @@ export interface MeterSurfaceNode { readonly imageStructuralTokens: number /** Structural JSON price replaced when request assembly projects files to text. */ readonly fileStructuralTokens: number - /** Durable image occurrences in message order; empty for image-free nodes. */ - readonly images: readonly MeterImageOccurrence[] + /** Durable image blocks in message order, `offloaded` marks included; empty for image-free nodes. */ + readonly images: readonly ImageBlock[] /** Durable file occurrences in message order; empty for file-free nodes. */ readonly files: readonly FileAttachmentRef[] } @@ -61,28 +51,28 @@ export interface SurfaceTokenPlan { readonly target: 'append' | { readonly startIdx: number; readonly endIdx: number } } -/** Collect image occurrences with their block paths and total their structural prices. */ -function collectImages(blocks: readonly ContentBlock[], images: MeterImageOccurrence[]): number { - let structuralTokens = 0 - visitImageBlocks(blocks, (block, path) => { - images.push({ attachment: block.attachment, path }) - structuralTokens += estimateStructuralBlock(block) - }) - return structuralTokens -} - -/** Collect file occurrences recursively and total their structural prices. */ -function collectFiles(blocks: readonly ContentBlock[], files: FileAttachmentRef[]): number { +/** Collect projected attachment occurrences and their structural prices. */ +function collectProjectedAttachments( + blocks: readonly ContentBlock[], + images: ImageBlock[], + files: FileAttachmentRef[], +): { readonly imageTokens: number; readonly fileTokens: number } { + let imageTokens = 0 let fileTokens = 0 for (const block of blocks) { - if (block.type === 'file') { + if (block.type === 'image') { + images.push(block) + imageTokens += estimateStructuralBlock(block) + } else if (block.type === 'file') { files.push(block.attachment) fileTokens += estimateStructuralBlock(block) } else if (block.type === 'tool-result') { - fileTokens += collectFiles(block.content, files) + const nested = collectProjectedAttachments(block.content, images, files) + imageTokens += nested.imageTokens + fileTokens += nested.fileTokens } } - return fileTokens + return { imageTokens, fileTokens } } /** Build one priced node from a surface event's derived message. */ @@ -98,15 +88,14 @@ function analyzeNode(seq: SessionSeq, message: Message | null): MeterSurfaceNode } } const heuristicTokens = estimateMessage(message) - const images: MeterImageOccurrence[] = [] - const imageStructuralTokens = collectImages(message.content, images) + const images: ImageBlock[] = [] const files: FileAttachmentRef[] = [] - const fileStructuralTokens = collectFiles(message.content, files) + const structural = collectProjectedAttachments(message.content, images, files) return { seq, heuristicTokens, - imageStructuralTokens, - fileStructuralTokens, + imageStructuralTokens: structural.imageTokens, + fileStructuralTokens: structural.fileTokens, images, files, } diff --git a/packages/llm/token-meter/tests/route-pricing.spec.ts b/packages/llm/token-meter/tests/route-pricing.spec.ts index b80df6b16a..4499970b2a 100644 --- a/packages/llm/token-meter/tests/route-pricing.spec.ts +++ b/packages/llm/token-meter/tests/route-pricing.spec.ts @@ -198,14 +198,14 @@ describe('request projection pricing', () => { expect(unknownRoute.nodes[0]!.tokens).toBe(estimateMessage(message)) }) - it('prices occurrences at or before the image/offload watermark as the route placeholder', async () => { + it('prices an occurrence a surface replacement marked offloaded as the route placeholder', async () => { const placeholder = '[offloaded]' - const watermarkPricing: LlmImageRequestPricing = { + const markedPricing: LlmImageRequestPricing = { priceImages: images => images.map(block => (block.offloaded === true ? { visualTokens: 0, text: placeholder } : { visualTokens: VISUAL_TOKENS, text: HANDLE_TEXT })), } - const { meter, session } = await harness(() => watermarkPricing) + const { meter, session } = await harness(() => markedPricing) session.append('turn/start', { turn: 1 }) const older = imageMessage('older') const newer = imageMessage('newer') @@ -215,8 +215,10 @@ describe('request projection pricing', () => { const before = meter.measure(session) expect(before.nodes.map(node => node.tokens)).toEqual([routedMessageTokens(older), routedMessageTokens(newer)]) - session.append('step/start', { turn: 1, step: 1 }) - session.append('image/offload', { turn: 1, step: 1, watermark: { seq: olderSeq, path: [1] } }) + session.append('user/message', { + ...older, + content: older.content.map(block => (block.type === 'image' ? { ...block, offloaded: true as const } : block)), + }, { surfaceOp: { op: 'replace', start: olderSeq, end: olderSeq }, sourceEventSeqs: [olderSeq] }) const after = meter.measure(session) const imageFree = estimateMessage({ ...older, content: older.content.filter(block => block.type !== 'image') }) expect(after.nodes[0]!.tokens).toBe(imageFree + estimateContent([{ type: 'text', text: placeholder }])) diff --git a/python/sdk/tests/test_client.py b/python/sdk/tests/test_client.py index 122c92cbcc..0af0db29ad 100644 --- a/python/sdk/tests/test_client.py +++ b/python/sdk/tests/test_client.py @@ -43,17 +43,6 @@ for line in sys.stdin: print(json.dumps({"jsonrpc": "2.0", "method": "session.event", "params": {"sessionId": params["sessionId"], "event": {"type": "agent/inbox/spliced", "data": {"target": "next-turn", "start": 0, "inserted": [{"id": "message-1"}]}}}}), flush=True) print(json.dumps({"jsonrpc": "2.0", "method": "session.status", "params": {"sessionId": params["sessionId"], "status": "running"}}), flush=True) print(json.dumps({"jsonrpc": "2.0", "id": msg["id"], "result": {"messageId": "message-1"}}), flush=True) - print(json.dumps({ - "jsonrpc": "2.0", - "method": "session.event", - "params": { - "sessionId": params["sessionId"], - "event": { - "type": "image/offload", - "data": {"turn": 1, "step": 1, "watermark": {"seq": 7, "path": [3]}}, - }, - }, - }), flush=True) print(json.dumps({ "jsonrpc": "2.0", "method": "session.event", @@ -120,10 +109,6 @@ for line in sys.stdin: assert result.final_response == "hello from runtime" assert result.finish_reason == "max-tokens" - assert result.events[1] == { - "type": "image/offload", - "data": {"turn": 1, "step": 1, "watermark": {"seq": 7, "path": [3]}}, - } assert result.events[-1]["type"] == "turn/end" dumped_env = json.loads(env_dump.read_text()) assert dumped_env["DEEPSEEK_API_KEY"] == "env-key" diff --git a/scripts/type-equiv.manifest.json b/scripts/type-equiv.manifest.json index 164c8395a3..c843a901ed 100644 --- a/scripts/type-equiv.manifest.json +++ b/scripts/type-equiv.manifest.json @@ -474,11 +474,6 @@ "symbol": "RequestContext", "source": "packages/core/session/src/types.ts" }, - { - "doc": "docs/subsystems/session.md", - "symbol": "ImageOccurrencePosition", - "source": "packages/core/session/src/types.ts" - }, { "doc": "docs/subsystems/session.md", "symbol": "SessionSeq", diff --git a/snapshots/sdk/inline-image-prompt/session.v2.jsonl b/snapshots/sdk/inline-image-prompt/session.v2.jsonl index 612b5a2983..a4af1cdade 100644 --- a/snapshots/sdk/inline-image-prompt/session.v2.jsonl +++ b/snapshots/sdk/inline-image-prompt/session.v2.jsonl @@ -11,8 +11,9 @@ {"type":"session/title","data":{"title":"Inspect these images, then reply","messageSeqs":[7],"source":{"kind":"fallback"}}} {"type":"request/header","data":{"header":{"config":{"provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"system":"{{system}}","tools":"{{tools}}"},"reason":"initial"}} {"type":"request/context","data":{"provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"}} -{"type":"assistant/attempt","data":{"turn":1,"step":1,"stream":[{"type":"chunk","time":1788752482896,"chunk":{"type":"finish","reason":{"kind":"error","failure":{"message":"DeepSeek base64 request images exceed the route budget; 3 more oldest occurrence(s) must be offloaded.","code":"IMAGE_OFFLOAD_REQUIRED","offloadImages":3}}}}]}} -{"type":"image/offload","data":{"turn":1,"step":1,"watermark":{"seq":7,"path":[3]}}} -{"type":"assistant/message","data":{"turn":1,"step":1,"message":{"role":"assistant","content":[{"type":"text","text":"DONE"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:3}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1788752482896,"chunk":{"type":"block-start","index":0,"blockType":"text"}},{"type":"chunk","time":1788752482896,"chunk":{"type":"block-end","index":0,"block":{"type":"text","text":"DONE"}}},{"type":"chunk","time":1788752482896,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1788752482896,"chunk":{"type":"finish","reason":{"kind":"stop"}}}]},"surfaceOp":"append"} +{"type":"assistant/attempt","data":{"turn":1,"step":1,"stream":[{"type":"chunk","time":1788839885455,"chunk":{"type":"finish","reason":{"kind":"error","failure":{"message":"DeepSeek base64 request images exceed the route budget; 3 more oldest occurrence(s) must be offloaded.","code":"IMAGE_OFFLOAD_REQUIRED","offloadImages":3}}}}]}} +{"type":"user/message","data":{"content":[{"type":"text","text":"Inspect these images, then reply with exactly "},{"type":"image","attachment":{"attachmentId":"sha256:b1ff9c8ea3a780bad09b346c423d2d0e46815926879b18e841d928376a946640","mediaType":"image/png","width":1,"height":1,"bytes":69},"offloaded":true},{"type":"image","attachment":{"attachmentId":"sha256:b1ff9c8ea3a780bad09b346c423d2d0e46815926879b18e841d928376a946640","mediaType":"image/png","width":1,"height":1,"bytes":69},"offloaded":true},{"type":"image","attachment":{"attachmentId":"sha256:b1ff9c8ea3a780bad09b346c423d2d0e46815926879b18e841d928376a946640","mediaType":"image/png","width":1,"height":1,"bytes":69},"offloaded":true},{"type":"image","attachment":{"attachmentId":"sha256:b1ff9c8ea3a780bad09b346c423d2d0e46815926879b18e841d928376a946640","mediaType":"image/png","width":1,"height":1,"bytes":69}},{"type":"image","attachment":{"attachmentId":"sha256:b1ff9c8ea3a780bad09b346c423d2d0e46815926879b18e841d928376a946640","mediaType":"image/png","width":1,"height":1,"bytes":69}},{"type":"image","attachment":{"attachmentId":"sha256:b1ff9c8ea3a780bad09b346c423d2d0e46815926879b18e841d928376a946640","mediaType":"image/png","width":1,"height":1,"bytes":69}},{"type":"text","text":"the single word DONE."}],"source":{"kind":"user"},"role":"user","id":"{{message:1}}"},"sourceEventSeqs":[7],"surfaceOp":{"op":"replace","start":7,"end":7}} +{"type":"request/header","data":{"header":{"config":{"provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"system":"{{system}}","tools":"{{tools}}"},"reason":"series"}} +{"type":"assistant/message","data":{"turn":1,"step":1,"message":{"role":"assistant","content":[{"type":"text","text":"DONE"}],"source":{"kind":"model","provider":"deepseek-official","model":"deepseek-v4-flash-vision-exp"},"id":"{{message:3}}"},"usage":{"inputTokens":3,"outputTokens":3},"stream":[{"type":"chunk","time":1788839885461,"chunk":{"type":"block-start","index":0,"blockType":"text"}},{"type":"chunk","time":1788839885461,"chunk":{"type":"block-end","index":0,"block":{"type":"text","text":"DONE"}}},{"type":"chunk","time":1788839885461,"chunk":{"type":"usage","usage":{"inputTokens":3,"outputTokens":3}}},{"type":"chunk","time":1788839885461,"chunk":{"type":"finish","reason":{"kind":"stop"}}}]},"surfaceOp":"append"} {"type":"step/end","data":{"turn":1,"step":1}} {"type":"turn/end","data":{"turn":1,"reason":{"kind":"completed"}}}