diff --git a/.agents/notes/implemented/architecture/2026-07-30-web-config-plane.i18n.yaml b/.agents/notes/implemented/architecture/2026-07-30-web-config-plane.i18n.yaml index ea59089b5d..6fb4396a74 100644 --- a/.agents/notes/implemented/architecture/2026-07-30-web-config-plane.i18n.yaml +++ b/.agents/notes/implemented/architecture/2026-07-30-web-config-plane.i18n.yaml @@ -2,5 +2,5 @@ # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: # pnpm run verify-translation-pairing --write .agents/notes/implemented/architecture/2026-07-30-web-config-plane.md -2026-07-30-web-config-plane.md: 1326f50a792b6c6f791c9515ea03cc362d16fc7e -2026-07-30-web-config-plane.zh.md: bea4bbc237f44864a5fa9bf677267ca0055c13ed +2026-07-30-web-config-plane.md: ac989cb100190e9a41ebf04b5b2d80125d49e0cb +2026-07-30-web-config-plane.zh.md: 26f11352b4362d3cbded3d1dc9205e289ee9b18c diff --git a/.agents/notes/implemented/architecture/2026-07-30-web-config-plane.md b/.agents/notes/implemented/architecture/2026-07-30-web-config-plane.md index 1326f50a79..ac989cb100 100644 --- a/.agents/notes/implemented/architecture/2026-07-30-web-config-plane.md +++ b/.agents/notes/implemented/architecture/2026-07-30-web-config-plane.md @@ -22,7 +22,7 @@ The request-level configuration seam made LLM adapter configuration restart-free **A hand-written editor over a schema model layer.** `ctx.settingsSchema`, provided by `dsh-client-ui-settings`, rehydrates the wire's `toJSON()` envelope into live schemastery nodes for validation, path resolution, and immutable draft editing — but no generic rendering: the first cut shipped a full schema-driven form renderer, and the resulting page was an unstyled schema dump (every advanced field flattened onto the card, raw field names as labels, the `retryPolicy` unsupported-fallback in the main flow). The hand-written direction won over adding a hint/grouping system, and a further simplification removed the reference input entirely: the card's primary field is one **API key** input, a whole-section provider without a configured key opens as its setup card, and the collapsed 自定义设置 fold carries the curated per-family extras (`baseURL` for both families, `reasoningEffort` for deepseek / `reasoning` for pi-ai, plus direct DeepSeek model rows with `id`, `name`, and `contextWindow`). Existing model fields outside that visible set survive array edits; retry policy, timeouts, and other fields remain owned by `settings.yaml`. Validation still runs the rehydrated schema before writing, while adapter-specific checks reject catalog invariants that the serialized schema cannot express. The card's colors resolve through the `--dsw-alias-*` design tokens; it had named `--border`/`--surface`/`--text-*`, which nothing in this app defines, so it rendered their light-mode fallbacks and stayed light under the dark theme. The model catalog takes the row shape the pi-ai provider form introduces: one bordered entry per model, id and display name on the row, and the capacities behind the row's own disclosure, so the two editors read as one design rather than diverging. Every field keeps the indexed `aria-label` that names it. Both capacities are text fields reading a decimal `K`/`M` suffix (`1M` is 1000K, matching how capacities are quoted) and storing the plain count: a field holds the typed text while it has focus, because re-deriving it from the parsed count on every keystroke would rewrite `1000` to `1K` mid-word, and text that does not parse stays on screen so the save-time rejection names a row the user can still see. The shared class names carry only declared token spellings: `--dsw-alias-border-subtle`, `--dsw-alias-text-tertiary`, and `--dsw-alias-text-primary` are undeclared, so naming them resolves to the light-mode literals in their fallback slots. A styles test now rejects any `--dsw-*` name the token sheet does not declare, so the next editor to name one fails rather than shipping a light-only surface. -**The Models page is a three-domain join with service-shaped apply semantics.** Rows are configured providers; the add card's select is the dormant directory remainder. Route liveness still gates readiness and invalidates the join, but the page does not render it as provider status because configuration presence and runtime availability are distinct. The key path stays reference-shaped without ever showing a reference: a typed key stores **write-only** through `credentials.set` under the profile's `apiKeyEnv`, deriving `_API_KEY` when none exists (the pi-ai profile records the derivation only when a key is entered), so `settings.yaml` never carries a key value; a blank pi-ai key materializes a reference-free profile and preserves provider-native authentication. Profile edits and removals land as minimal path-addressed `settings.mutate` operations against the redacted user section, which never names a secret the page did not receive. Removing a user-layer provider first opens a localized confirmation dialog whose row actions, title, description, and final action identify the same provider; confirmation removes an exact configured+writable derived credential before the profile, while custom, environment, and unidentified targets remain untouched. Both stages are idempotent and a partial failure stays in the dialog for retry. DeepSeek's model list is array-replace configuration: inherited effective rows remain visible until the first edit materializes the complete list in the user layer, and reset unsets the list override. The partial-commit and credential-ownership rationale lives in the [provider credential lifecycle note](../bug-fix/2026-08-06-provider-credential-lifecycle.md). +**The Models page is a three-domain join with service-shaped apply semantics.** Rows are configured providers; the add card's select is the dormant directory remainder. Route liveness still gates readiness and invalidates the join, but the page does not render it as provider status because configuration presence and runtime availability are distinct. The key path stays reference-shaped without ever showing a reference: a typed key stores **write-only** through `credentials.set` under the profile's `apiKeyEnv`, deriving `_API_KEY` when none exists (the pi-ai profile records the derivation only when a key is entered), so `settings.yaml` never carries a key value; a blank pi-ai key materializes a reference-free profile and preserves provider-native authentication. Profile edits and removals land as minimal path-addressed `settings.mutate` operations against the redacted user section, which never names a secret the page did not receive. Removing a user-layer provider first opens a localized confirmation dialog whose row actions, title, description, and final action identify the same provider; confirmation removes an exact configured+writable derived credential before the profile, while custom, environment, and unidentified targets remain untouched. Both stages are idempotent and a partial failure stays in the dialog for retry. DeepSeek's model list is array-replace configuration: inherited effective rows remain visible until the first edit materializes the complete list in the user layer, and reset unsets the list override. `llm.discoverModels` results stay in picker-local state until **Add selected**; configured ids start unchecked, while **Select all** / **Deselect all** changes only that local set, so bulk selection preserves the same capacity-protection rule. The partial-commit and credential-ownership rationale lives in the [provider credential lifecycle note](../bug-fix/2026-08-06-provider-credential-lifecycle.md). ## Alternatives considered @@ -36,4 +36,4 @@ The request-level configuration seam made LLM adapter configuration restart-free ## Consequences -The whole loop is pinned keyless in the browser lane (`apps/web/tests/models-settings.e2e.ts`): the add card offers the dormant pi-ai catalog, adding `minimax-cn` with a typed key writes the reference-only profile into `settings.yaml`, stores the value into the harness home's `.env` under the derived `MINIMAX_CN_API_KEY`, registers the route live on the topology frame, and the customized fold merges `reasoning` beside the reference — zero model calls, ARIA goldens for the add-card, configured, and identified delete-confirmation states, plus a scaffold `harnessHome` so tests never touch a real `~/.dsh` (the provider under test is one whose derived reference cannot collide with a developer's exported keys). The settings-shell scenario intercepts the pathless native intent; Service Definition, provider, wire, React, and native-opener tests separately pin provider absence, custom-path resolution, absent-file materialization, owner-only permissions, hidden remote/unavailable states, duplicate-click collapse, localized failure, macOS text-editor dispatch, and Linux/Windows desktop dispatch. The removal scenario proves cancellation leaves both profile and key intact, then confirmation removes both the profile and its identified managed credential. The DeepSeek onboarding fixture edits the default catalog into a user-owned list, persists an arbitrary model id/name/context window, removes the active row, and observes the model selector's empty-selection fallback. The rename touched 239 files (fixtures, goldens, docs, python) with no compatibility alias. The renderer replacement needed no wire change: apply semantics, redaction, and the directory join were renderer-agnostic all along. Deferred: a per-row models preview (the picker already lists models) and a page address for live routes that never declared configurability. +The whole loop is pinned keyless in the browser lane (`apps/web/tests/models-settings.e2e.ts`): the add card offers the dormant pi-ai catalog, adding `minimax-cn` with a typed key writes the reference-only profile into `settings.yaml`, stores the value into the harness home's `.env` under the derived `MINIMAX_CN_API_KEY`, registers the route live on the topology frame, and the customized fold merges `reasoning` beside the reference — zero model calls, ARIA goldens for the add-card, configured, model-picker, and identified delete-confirmation states, plus a scaffold `harnessHome` so tests never touch a real `~/.dsh` (the provider under test is one whose derived reference cannot collide with a developer's exported keys). The component suite pins configured-id exclusions and both directions of the bulk toggle. The settings-shell scenario intercepts the pathless native intent; Service Definition, provider, wire, React, and native-opener tests separately pin provider absence, custom-path resolution, absent-file materialization, owner-only permissions, hidden remote/unavailable states, duplicate-click collapse, localized failure, macOS text-editor dispatch, and Linux/Windows desktop dispatch. The removal scenario proves cancellation leaves both profile and key intact, then confirmation removes both the profile and its identified managed credential. The DeepSeek onboarding fixture edits the default catalog into a user-owned list, persists an arbitrary model id/name/context window, removes the active row, and observes the model selector's empty-selection fallback. The rename touched 239 files (fixtures, goldens, docs, python) with no compatibility alias. The renderer replacement needed no wire change: apply semantics, redaction, and the directory join were renderer-agnostic all along. Deferred: a per-row models preview (the picker already lists models) and a page address for live routes that never declared configurability. diff --git a/.agents/notes/implemented/architecture/2026-07-30-web-config-plane.zh.md b/.agents/notes/implemented/architecture/2026-07-30-web-config-plane.zh.md index bea4bbc237..26f11352b4 100644 --- a/.agents/notes/implemented/architecture/2026-07-30-web-config-plane.zh.md +++ b/.agents/notes/implemented/architecture/2026-07-30-web-config-plane.zh.md @@ -22,7 +22,7 @@ Status: implemented **架在 schema 模型层之上的手写编辑器。**`dsh-client-ui-settings` 提供的 `ctx.settingsSchema` 把 wire 的 `toJSON()` 信封还原（rehydrate）为活的 schemastery 节点，用于校验、路径解析与不可变草稿编辑——但不做通用渲染：第一版交付了完整的 schema 驱动表单渲染器，得到的却是一个未加样式、把 schema 原样倾倒出来的页面（每个进阶字段都平铺到卡片上、原始字段名直接充当标签、`retryPolicy` 的「不支持」回退落在主流程里）。手写方向胜过了再加一套提示／分组系统，进一步的简化又把引用输入框整个移除：卡片的主字段是一个 **API 密钥** 输入框，未配置密钥的整分节提供方会以其设置卡片的形式打开，收起的「自定义设置」折叠区承载按家族精选的额外字段（两个家族都有 `baseURL`，deepseek 有 `reasoningEffort`／pi-ai 有 `reasoning`，另有直接 DeepSeek 模型行的 `id`、`name` 和 `contextWindow`）。现有模型字段中不在可见集合内的部分会在数组编辑后保留；重试策略、超时及其他字段仍归 `settings.yaml` 所有。校验仍会在写入前运行还原出的 schema，适配器特有的检查则会拒绝序列化 schema 无法表达的目录不变量。卡片的颜色经 `--dsw-alias-*` 设计 token 解析；它此前引用的 `--border`／`--surface`／`--text-*` 在本应用中无人定义，于是渲染出的是它们的亮色模式回退值，在暗色主题下依旧保持亮色。模型目录采用 pi-ai 提供方表单引入的行形态：每个模型一个带边框的条目，ID 与显示名称落在行上，容量则收在该行自己的折叠区里，使两个编辑器呈现为同一套设计，而不是各自分岔。每个字段都保留那个为其命名的带序号 `aria-label`。两项容量都是文本输入框，读取十进制的 `K`／`M` 后缀（`1M` 即 1000K，与容量的通行标注方式一致）并存储纯数值：字段持有焦点期间保留键入的文本，因为若每次按键都从解析出的数值重新推导该文本，`1000` 会在尚未输完时就被改写成 `1K`；无法解析的文本也会留在屏幕上，因此保存时的拒绝点名的是用户仍能看见的那一行。共用的类名只承载已声明的 token 写法：`--dsw-alias-border-subtle`、`--dsw-alias-text-tertiary` 和 `--dsw-alias-text-primary` 均未声明，写出它们就会解析为各自回退槽位中的亮色模式字面值。现在有一个样式测试会拒绝 token 表未声明的任何 `--dsw-*` 名称，因此下一个写出这类名称的编辑者会当场失败，而不是交付一个只有亮色的界面。 -**Models 页是一次三领域联接，应用语义与服务同形。**每一行是一个已配置的提供方；「新增」卡片的选择框是可配置提供方目录中剩余的休眠条目。路由存活状态仍用于就绪判定，并会使该联接失效，但页面不将其渲染为提供方状态，因为配置存在与运行时可用性是两个不同概念。密钥通道保持引用形态，却从不展示任何引用：键入的密钥经 `credentials.set` **只写**存入 profile 的 `apiKeyEnv` 之下，引用不存在时便派生 `_API_KEY`（仅在输入密钥时，pi-ai profile 才会记录该派生），因此 `settings.yaml` 从不携带密钥值；留空 pi-ai 密钥会具化一个不带引用的 profile，并保留提供方原生认证。profile 的编辑和删除会针对脱敏后的用户分节，以按路径寻址的最小 `settings.mutate` 操作落地，绝不会点名页面未收到的机密。删除用户层提供方时，会先打开本地化确认对话框，其行操作、标题、说明和最终操作都会点名同一个提供方；确认后会先清除与派生目标精确匹配且已配置、可写的凭据，再删除 profile，自定义目标、环境目标和无法识别的目标则保持不变。两个阶段都具备幂等性，部分失败会留在对话框中供重试。DeepSeek 的模型列表是数组替换配置：继承而来的生效模型行会一直显示，直到第一次编辑将完整列表具化到用户层；重置则会取消设置该列表覆盖。部分提交与凭据所有权的理由记录在[提供方凭据生命周期 note](../bug-fix/2026-08-06-provider-credential-lifecycle.md)中。 +**Models 页是一次三领域联接，应用语义与服务同形。**每一行是一个已配置的提供方；「新增」卡片的选择框是可配置提供方目录中剩余的休眠条目。路由存活状态仍用于就绪判定，并会使该联接失效，但页面不将其渲染为提供方状态，因为配置存在与运行时可用性是两个不同概念。密钥通道保持引用形态，却从不展示任何引用：键入的密钥经 `credentials.set` **只写**存入 profile 的 `apiKeyEnv` 之下，引用不存在时便派生 `_API_KEY`（仅在输入密钥时，pi-ai profile 才会记录该派生），因此 `settings.yaml` 从不携带密钥值；留空 pi-ai 密钥会具化一个不带引用的 profile，并保留提供方原生认证。profile 的编辑和删除会针对脱敏后的用户分节，以按路径寻址的最小 `settings.mutate` 操作落地，绝不会点名页面未收到的机密。删除用户层提供方时，会先打开本地化确认对话框，其行操作、标题、说明和最终操作都会点名同一个提供方；确认后会先清除与派生目标精确匹配且已配置、可写的凭据，再删除 profile，自定义目标、环境目标和无法识别的目标则保持不变。两个阶段都具备幂等性，部分失败会留在对话框中供重试。DeepSeek 的模型列表是数组替换配置：继承而来的生效模型行会一直显示，直到第一次编辑将完整列表具化到用户层；重置则会取消设置该列表覆盖。`llm.discoverModels` 的结果会留在选择框的本地状态中，直至点击**添加所选**；已配置的 ID 默认不勾选，**全选**／**取消全选**也只会改变这一本地集合，因此批量选择仍遵守同一条容量保护规则。部分提交与凭据所有权的理由记录在[提供方凭据生命周期 note](../bug-fix/2026-08-06-provider-credential-lifecycle.md)中。 ## 曾考虑的替代方案 @@ -36,4 +36,4 @@ Status: implemented ## 后果 -整条闭环以无密钥方式固定在浏览器测试通道（`apps/web/tests/models-settings.e2e.ts`）：「新增」卡片提供休眠的 pi-ai catalog，携键入的密钥添加 `minimax-cn` 会把只含引用的 profile 写入 `settings.yaml`、把密钥值存入 harness 家目录 `.env` 中派生的 `MINIMAX_CN_API_KEY` 之下、路由随拓扑帧注册为存活，「自定义设置」折叠区则把 `reasoning` 合并到引用旁边——全程零模型调用，「新增」卡片态、已配置态与已点名目标的删除确认态各有 ARIA golden，另有脚手架式的 `harnessHome`，测试绝不触碰真实的 `~/.dsh`（受测提供方是派生引用不可能与开发者已导出密钥相撞的那一个）。设置外壳场景会截获无路径参数的原生意图；Service Definition、提供方、wire、React 与原生打开器测试分别固定了提供方缺失、自定义路径解析、缺失文件创建、仅属主权限、远程／不可用时隐藏、重复点击合并、本地化失败、macOS 文本编辑器分发，以及 Linux/Windows 桌面分发。删除场景证明，取消会保留 profile 和密钥，随后的确认会同时删除 profile 及其已识别的受管凭据。DeepSeek 首次使用 fixture 会把默认目录编辑为用户自有列表、持久化任意模型的 ID／名称／上下文窗口、移除活动模型行，并观察模型选择器的空选择回退。这次重命名触及 239 个文件（fixture（测试前置数据）、golden、文档、python），未保留兼容别名。替换渲染器不需要任何 wire 变更：应用语义、脱敏与目录联接从一开始就与渲染器无关。延后事项：每行的模型预览（选择器已能列出模型）和为从未声明可配置性的存活路由提供页面地址。 +整条闭环以无密钥方式固定在浏览器测试通道（`apps/web/tests/models-settings.e2e.ts`）：「新增」卡片提供休眠的 pi-ai catalog，携键入的密钥添加 `minimax-cn` 会把只含引用的 profile 写入 `settings.yaml`、把密钥值存入 harness 家目录 `.env` 中派生的 `MINIMAX_CN_API_KEY` 之下、路由随拓扑帧注册为存活，「自定义设置」折叠区则把 `reasoning` 合并到引用旁边——全程零模型调用，「新增」卡片态、已配置态、模型选择框态与已点名目标的删除确认态各有 ARIA golden，另有脚手架式的 `harnessHome`，测试绝不触碰真实的 `~/.dsh`（受测提供方是派生引用不可能与开发者已导出密钥相撞的那一个）。组件测试固定了已配置 ID 的排除与批量切换的两个方向。设置外壳场景会截获无路径参数的原生意图；Service Definition、提供方、wire、React 与原生打开器测试分别固定了提供方缺失、自定义路径解析、缺失文件创建、仅属主权限、远程／不可用时隐藏、重复点击合并、本地化失败、macOS 文本编辑器分发，以及 Linux/Windows 桌面分发。删除场景证明，取消会保留 profile 和密钥，随后的确认会同时删除 profile 及其已识别的受管凭据。DeepSeek 首次使用 fixture 会把默认目录编辑为用户自有列表、持久化任意模型的 ID／名称／上下文窗口、移除活动模型行，并观察模型选择器的空选择回退。这次重命名触及 239 个文件（fixture（测试前置数据）、golden、文档、python），未保留兼容别名。替换渲染器不需要任何 wire 变更：应用语义、脱敏与目录联接从一开始就与渲染器无关。延后事项：每行的模型预览（选择器已能列出模型）和为从未声明可配置性的存活路由提供页面地址。 diff --git a/.agents/notes/implemented/bug-fix/2026-08-18-request-image-payload-bound.i18n.yaml b/.agents/notes/implemented/bug-fix/2026-08-18-request-image-payload-bound.i18n.yaml new file mode 100644 index 0000000000..f797c08086 --- /dev/null +++ b/.agents/notes/implemented/bug-fix/2026-08-18-request-image-payload-bound.i18n.yaml @@ -0,0 +1,6 @@ +# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each +# side as of the last confirmed-consistent state. Both languages carry equal authority; +# after editing either side, bring the other along and re-record with: +# pnpm run verify-translation-pairing --write .agents/notes/implemented/bug-fix/2026-08-18-request-image-payload-bound.md +2026-08-18-request-image-payload-bound.md: df10df39d18c7da4660b566e8f0b6a5a60ff8dc1 +2026-08-18-request-image-payload-bound.zh.md: 070f2d194f1459f3f2728fdf9d5d2db2c3a24385 diff --git a/.agents/notes/implemented/bug-fix/2026-08-18-request-image-payload-bound.md b/.agents/notes/implemented/bug-fix/2026-08-18-request-image-payload-bound.md new file mode 100644 index 0000000000..df10df39d1 --- /dev/null +++ b/.agents/notes/implemented/bug-fix/2026-08-18-request-image-payload-bound.md @@ -0,0 +1,35 @@ +# Agent Note: Request-level image payload bound + +Status: implemented + +English | [中文](2026-08-18-request-image-payload-bound.zh.md) + +## Problem + +Every image in session history is base64-inlined into every model request by the pi-ai adapter, so a long session's request body grows monotonically with each admitted image. Gateways cap request-body size; once the accumulated payload crossed such a cap the request was rejected with 413 (`Failed to buffer the request body: length limit exceeded`), and because nothing bounds or trims the assembled request, every retry resent the same oversized body. The session was permanently unusable, and the failure text matched no `classifyPiAiError` rule, so it surfaced as the generic `PI_AI_ERROR`. Admission bounds (per image, per message) cannot prevent this: each image is individually admissible, and the sum still grows without bound. Two screenshots were enough to trigger it in production. + +## Decision + +The pi-ai provider profile carries `maxRequestImageBytes` (default `DEFAULT_MAX_REQUEST_IMAGE_BYTES = 20MiB`, a positive integer, per route, changeable from cordis.yml and the `llm-pi-ai` settings section). At request conversion, `toPiContext` sums the base64 length of every image in history (derived from `ImageAttachmentRef.bytes` without reading data) and, while the sum exceeds the bound, replaces the oldest images with a fixed model-facing placeholder. The placeholder tells the model to read the file again when a path is available or ask the user to attach the image again. The most recent images are omitted last; an image larger than the bound is itself omitted. Offload locations use message and nested block indexes rather than object identity, so replaying the same JSON log produces the same request. Offloaded images are never read from the attachment store. `classifyPiAiError` classifies 413 and specific request-body-cap wording as `INVALID_REQUEST` (resending the same body cannot succeed). Four images admitted at the attachment store's 3.5MiB raw-image default occupy at most 18.67MiB after base64 expansion. The 20MiB request-image default therefore retains four such images and reserves the rest of a 32MiB request for system prompts, history, tools, and JSON. Deployments behind stricter gateways lower the value per route. + +## Offload is conversion, not history + +The placeholder is model-visible but not logged as a session event. It stays within the model-visible ⟺ logged invariant the same way the adapter's other serialization does (`(no output)` fallbacks, text-only folding): the offload locations are a pure function of the logged history and the route configuration, so the exact request remains reconstructable from the session log plus the composition. A logged elision event becomes necessary only when offload decisions gain non-deterministic inputs (for example live gateway feedback), which belongs to the deferred capability-metadata design. + +## Alternatives considered + +- **Fail the request with a clear error instead of offloading.** Keeps the model informed but leaves the session wedged: the user cannot remove images from durable history, so a hard failure at the bound is permanent. Offload keeps the session serviceable, which is the point of the fix. +- **Upload images once and reference them by URL / file id.** Removes the linear body growth entirely and is the right medium-term shape (providers and the internal gateway both document a Files path), but it introduces upload lifecycle management across providers and is far beyond a P0 hotfix. +- **Count the full request body, not only images.** Text and tools contribute little and their sizes are only known after full serialization per protocol; bounding the dominant term with explicit headroom is accurate enough for the failure being fixed and much simpler. Revisit inside the route-capability design. +- **Trim at admission instead.** Admission cannot see future accumulation; only the assembled request knows its total. Admission-side bounds (per-side dimension, bytes) remain as the first layer and are owned by [the dimension-limit note](2026-08-17-image-dimension-admission-limit.md). + +## Related + +- [Per-side image dimension admission limit](2026-08-17-image-dimension-admission-limit.md) — the admission-layer companion fix; together they close the two observed session-poisoning failures (400 dimension, 413 body size). + +## Consequences + +- An image-heavy long session keeps completing requests. The oldest images are omitted first; the most recent image is omitted only when it cannot fit within the bound. +- Crossing the bound rewrites an early message, so the provider prompt-cache prefix ends at the newly offloaded image until the offloaded prefix stabilizes. +- The bound counts base64 image payload only; deployments must keep it below their gateway's request-body cap with headroom, and the shipped default cannot know a private gateway's cap. +- Route capability metadata driving admission and assembly together (image count, per-image size, request size, provider token formulas) remains deferred design work tracked outside this fix. diff --git a/.agents/notes/implemented/bug-fix/2026-08-18-request-image-payload-bound.zh.md b/.agents/notes/implemented/bug-fix/2026-08-18-request-image-payload-bound.zh.md new file mode 100644 index 0000000000..070f2d194f --- /dev/null +++ b/.agents/notes/implemented/bug-fix/2026-08-18-request-image-payload-bound.zh.md @@ -0,0 +1,35 @@ +# Agent Note: 请求级图片载荷上限 + +Status: implemented + +[English](2026-08-18-request-image-payload-bound.md) | 中文 + +## Problem + +pi-ai 适配器把会话历史中的每张图片 base64 内联进每一个模型请求，长会话的请求体随每张入库图片单调增长。网关对请求体大小设有上限；累积载荷一旦越线，请求被以 413 拒绝（`Failed to buffer the request body: length limit exceeded`），而组装层没有任何约束或裁剪，每次重试都会原样重发同一个超限请求体，会话永久不可用。该报错文本不匹配 `classifyPiAiError` 的任何规则，只能落进笼统的 `PI_AI_ERROR`。准入上限（单图、单消息）无法阻止这一点：每张图片单独看都合规，总和仍然无界增长。线上两张截图即可触发。 + +## Decision + +pi-ai provider profile 增加 `maxRequestImageBytes`（默认 `DEFAULT_MAX_REQUEST_IMAGE_BYTES = 20MiB`，正整数，按路由生效，可从 cordis.yml 与 `llm-pi-ai` settings 段修改）。请求转换时，`toPiContext` 由 `ImageAttachmentRef.bytes` 推算每张历史图片的 base64 长度（无需读取数据）求和，总和超过上限时从最老的图片开始替换为一段固定的模型可见占位文本。占位文本要求模型在有路径时重新读取文件，否则请用户重新附上图片。越新的图片越晚被省略；单张图片本身超过上限时也会被省略。offload 位置用消息与嵌套块的索引表示，不依赖对象身份，因此重放同一份 JSON 日志会产生相同请求。被 offload 的图片不会从附件存储读取。`classifyPiAiError` 把 413 与明确的请求体上限措辞归类为 `INVALID_REQUEST`（原样重发不可能成功）。四张按附件存储默认上限准入的 3.5MiB 原始图片，经 base64 膨胀后最多占 18.67MiB。20MiB 请求图片默认上限因此可保留四张这样的图片，并在 32MiB 请求内为系统提示词、历史、工具与 JSON 保留其余容量。网关更严格的部署按路由调低该值。 + +## offload 是转换而非历史 + +占位文本模型可见，但不记录为会话事件。它与适配器的其他序列化（`(no output)` 回退、纯文本折叠）以同样的方式满足「模型可见 ⟺ 已记录」不变量：offload 位置是已记录历史与路由配置的纯函数，确切请求仍可由会话日志加组合配置重建。只有当 offload 决策引入非确定性输入（例如网关的实时反馈）时才需要记录省略事件，那属于暂缓的能力元数据设计。 + +## Alternatives considered + +- **在上限处直接报错而不 offload。** 模型知情，但会话仍然卡死：用户无法从持久历史中删除图片，越线即永久失败。offload 让会话保持可用，这正是本修复的目标。 +- **图片上传一次、按 URL / file id 引用。** 从结构上消除请求体线性增长，是正确的中期形态（各提供方与内部网关都有 Files 路径），但要跨提供方管理上传生命周期，远超 P0 热修复范围。 +- **统计完整请求体而非只统计图片。** 文本与工具占比很小，且其大小要到按协议完整序列化后才可知；对主导项设上限并留出显式余量，对所修故障足够精确且简单得多。留到路由能力设计中再议。 +- **改在准入侧裁剪。** 准入看不到未来的累积，只有组装后的请求知道自己的总量。准入侧上限（单边尺寸、字节）作为第一层保留，归[尺寸上限笔记](2026-08-17-image-dimension-admission-limit.md)所有。 + +## Related + +- [图片单边尺寸准入上限](2026-08-17-image-dimension-admission-limit.md)，准入层的配套修复；两者合起来封住已观测到的两类会话毒化故障（400 尺寸、413 请求体）。 + +## Consequences + +- 图片较多的长会话持续可用。最老的图片优先省略；仅当最新图片本身无法装进上限时才会省略它。 +- 越过上限会改写较早的一条消息，提供方 prompt cache 前缀在新被 offload 的图片处截止，直到被 offload 的前缀稳定。 +- 上限只统计 base64 图片载荷；部署必须让它低于自家网关的请求体上限并留出余量，发行默认值无法预知私有网关的上限。 +- 由路由能力元数据同时驱动准入与组装（图片数量、单图大小、请求大小、提供方 token 公式）的设计仍为暂缓工作，在本修复之外跟踪。 diff --git a/.agents/notes/implemented/feature/2026-07-22-web-multimodal-image-input-and-durable-attachments.i18n.yaml b/.agents/notes/implemented/feature/2026-07-22-web-multimodal-image-input-and-durable-attachments.i18n.yaml index 8695a95bec..842f2c9cd3 100644 --- a/.agents/notes/implemented/feature/2026-07-22-web-multimodal-image-input-and-durable-attachments.i18n.yaml +++ b/.agents/notes/implemented/feature/2026-07-22-web-multimodal-image-input-and-durable-attachments.i18n.yaml @@ -2,5 +2,5 @@ # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: # pnpm run verify-translation-pairing --write .agents/notes/implemented/feature/2026-07-22-web-multimodal-image-input-and-durable-attachments.md -2026-07-22-web-multimodal-image-input-and-durable-attachments.md: 8c07b8b786aeeb87a4c2db7c0e6e49928b0ddf2c -2026-07-22-web-multimodal-image-input-and-durable-attachments.zh.md: 5d2d2963d6bee729362852701ea58f191f9721b2 +2026-07-22-web-multimodal-image-input-and-durable-attachments.md: 6a12380992c3205d6bb3f5701f093b3335fae2f9 +2026-07-22-web-multimodal-image-input-and-durable-attachments.zh.md: a8e94154e435a5359e7928d3b11eba7d1aee92f7 diff --git a/.agents/notes/implemented/feature/2026-07-22-web-multimodal-image-input-and-durable-attachments.md b/.agents/notes/implemented/feature/2026-07-22-web-multimodal-image-input-and-durable-attachments.md index 8c07b8b786..6a12380992 100644 --- a/.agents/notes/implemented/feature/2026-07-22-web-multimodal-image-input-and-durable-attachments.md +++ b/.agents/notes/implemented/feature/2026-07-22-web-multimodal-image-input-and-durable-attachments.md @@ -140,7 +140,7 @@ Composer thumbnails and each `MessageImage` own ephemeral original-preview state ### Limits and trust boundaries -Version one accepts PNG, JPEG, WebP, and GIF only. SVG and remote URLs are excluded. Default limits are 5 MiB per image, 20 images and 100 MiB aggregate image bytes per message, 40 million intrinsic pixels per image, and 2000 pixels on either side. These deployment-varying limits are validated backend configuration and enforced by the host before persistence. The client connection carrier has an independent configurable `maxRequestBodyBytes` cap (160 MiB by default) for every API request and fails load if it cannot hold the attachment service's aggregate image limit after base64 and envelope expansion; lowering image policy therefore never silently lowers the carrier limit for valid text or other RPCs. A body without a declared length is rejected the moment it crosses the cap rather than drained to its end. +Version one accepts PNG, JPEG, WebP, and GIF only. SVG and remote URLs are excluded. Default limits are 3.5 MiB per image, 20 images and 100 MiB aggregate image bytes per message, 40 million intrinsic pixels per image, and 2000 pixels on either side. These deployment-varying limits are validated backend configuration and enforced by the host before persistence. The client connection carrier has an independent configurable `maxRequestBodyBytes` cap (160 MiB by default) for every API request and fails load if it cannot hold the attachment service's aggregate image limit after base64 and envelope expansion; lowering image policy therefore never silently lowers the carrier limit for valid text or other RPCs. A body without a declared length is rejected the moment it crosses the cap rather than drained to its end. Malformed base64, unsupported or mismatched media, truncated image payloads, excess bytes, excess image count, excess pixels, excess per-side dimensions, missing objects, and integrity mismatches return stable structured failures. Original filenames are reduced to a display basename, control characters are removed, and no local path is logged or returned to the browser. diff --git a/.agents/notes/implemented/feature/2026-07-22-web-multimodal-image-input-and-durable-attachments.zh.md b/.agents/notes/implemented/feature/2026-07-22-web-multimodal-image-input-and-durable-attachments.zh.md index 5d2d2963d6..a8e94154e4 100644 --- a/.agents/notes/implemented/feature/2026-07-22-web-multimodal-image-input-and-durable-attachments.zh.md +++ b/.agents/notes/implemented/feature/2026-07-22-web-multimodal-image-input-and-durable-attachments.zh.md @@ -140,7 +140,7 @@ Pi-AI 适配器是首条视觉输入路径：它在请求时解析 `ctx.attachme ### 限制与信任边界 -第一版仅接受 PNG、JPEG、WebP 和 GIF。不接受 SVG 和远程 URL。默认限制为每张图片 5 MiB、每条消息 20 张图片和 100 MiB 图片总字节数、每张图片 4,000 万个固有像素，以及任一边 2,000 像素。这些随部署变化的限制属于经过校验的后端配置，并由宿主在持久化前强制执行。客户端连接载体为每个 API 请求设置独立且可配置的 `maxRequestBodyBytes` 上限（默认 160 MiB）；如果该上限无法容纳附件服务的图片总量限制经 base64 和请求封装膨胀后的大小，加载就会失败。因此，降低图片策略绝不会静默降低有效文本或其他 RPC 的载体上限。未声明长度的请求体在越过上限的瞬间即被拒绝，而不是先读完再拒。 +第一版仅接受 PNG、JPEG、WebP 和 GIF。不接受 SVG 和远程 URL。默认限制为每张图片 3.5 MiB、每条消息 20 张图片和 100 MiB 图片总字节数、每张图片 4,000 万个固有像素，以及任一边 2,000 像素。这些随部署变化的限制属于经过校验的后端配置，并由宿主在持久化前强制执行。客户端连接载体为每个 API 请求设置独立且可配置的 `maxRequestBodyBytes` 上限（默认 160 MiB）；如果该上限无法容纳附件服务的图片总量限制经 base64 和请求封装膨胀后的大小，加载就会失败。因此，降低图片策略绝不会静默降低有效文本或其他 RPC 的载体上限。未声明长度的请求体在越过上限的瞬间即被拒绝，而不是先读完再拒。 格式错误的 base64、不支持或不匹配的媒体、截断的图片数据、超出字节限制、超出图片数量、超出像素限制、超出单边尺寸限制、对象缺失和完整性不匹配都会返回稳定的结构化错误。原始文件名只保留用于显示的末段，控制字符会被移除，并且任何本地路径都不会写入日志或返回浏览器。 diff --git a/.agents/notes/implemented/feature/2026-08-12-web-image-intake-and-limits-alignment.i18n.yaml b/.agents/notes/implemented/feature/2026-08-12-web-image-intake-and-limits-alignment.i18n.yaml index 1ea48bbef6..01144af297 100644 --- a/.agents/notes/implemented/feature/2026-08-12-web-image-intake-and-limits-alignment.i18n.yaml +++ b/.agents/notes/implemented/feature/2026-08-12-web-image-intake-and-limits-alignment.i18n.yaml @@ -2,5 +2,5 @@ # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: # pnpm run verify-translation-pairing --write .agents/notes/implemented/feature/2026-08-12-web-image-intake-and-limits-alignment.md -2026-08-12-web-image-intake-and-limits-alignment.md: 2f8b99bb4850d9875dcba0a03ae8ad9f340d1506 -2026-08-12-web-image-intake-and-limits-alignment.zh.md: 62d5ebd54275ae8de0e0b9ba701ba34042dfcde7 +2026-08-12-web-image-intake-and-limits-alignment.md: 00cf7ea99d63e848c4b5839da1d97d94c9fb8464 +2026-08-12-web-image-intake-and-limits-alignment.zh.md: d88ba25a3f3e3be4a3655080a45123ada6cdebcd diff --git a/.agents/notes/implemented/feature/2026-08-12-web-image-intake-and-limits-alignment.md b/.agents/notes/implemented/feature/2026-08-12-web-image-intake-and-limits-alignment.md index 2f8b99bb48..00cf7ea99d 100644 --- a/.agents/notes/implemented/feature/2026-08-12-web-image-intake-and-limits-alignment.md +++ b/.agents/notes/implemented/feature/2026-08-12-web-image-intake-and-limits-alignment.md @@ -16,7 +16,7 @@ The second alignment step for issue #2248, after the [attachment display note](2 **History thumbnails (DeepSeek Chat rules).** A message's lone image renders at 240px on its long edge with the displayed ratio clamped to [0.25, 4], cropped by `cover` with the anchor at the top of very tall images and the left of very wide ones, never upscaled; several images render as fixed 64px square tiles in one wrapping row (10px gap, user messages right-aligned). Consecutive assistant `image` blocks merge into one gallery so they tile instead of each opening a one-image row. -**Limits aligned and projected.** Defaults are 20 images / 5 MiB per image / 100 MiB aggregate (`attachment-local`), with the HTTP carrier cap raised to one shared `DEFAULT_MAX_REQUEST_BODY_BYTES = 160 MiB` (http-bridge, previously two independent 32 MiB literals) to satisfy the load-time capacity assertion (aggregate × 4/3 + headroom ≈ 134.3 MiB). Consumer products cluster at 10–20 attachments (ChatGPT 10, Gemini 10, Claude 20; DeepSeek Chat's 50 is the outlier), and a vision-model image costs roughly 1300–4800 tokens, so 50 images can fill a 200k context in one message. The 5 MiB per-image default admits images across Anthropic routes that impose either a 5 MiB or 10 MiB maximum; deployments using only routes with the larger limit can override it. A 512 MiB aggregate cannot pass this transport because base64-in-JSON would need a single JSON string past V8's ~512 MiB string ceiling. The limits reach clients as the `imageLimits` session projection — a constant-per-boot unit (`apply` returns the same state reference, so baselines alone carry it and no change frames exist) registered by **apiproxy**, not the attachment Service Definition: `dsh-llm` depends on `dsh-attachment` (`ImageBlock` → `ImageAttachmentRef`), so the seam package referencing `dsh-session-projection` (whose graph reaches `dsh-llm` through `dsh-session`) closes a project-reference cycle, and the per-message count/aggregate rules the value describes are the proxy's own admission checks anyway. The `SessionProjectionMap` merge rides the proxy's sessions wire-contract file, which every client program already includes through the carrier's type re-exports. +**Limits aligned and projected.** Defaults are 20 images / 3.5 MiB per image / 100 MiB aggregate (`attachment-local`), with the HTTP carrier cap raised to one shared `DEFAULT_MAX_REQUEST_BODY_BYTES = 160 MiB` (http-bridge, previously two independent 32 MiB literals) to satisfy the load-time capacity assertion (aggregate × 4/3 + headroom ≈ 134.3 MiB). Consumer products cluster at 10–20 attachments (ChatGPT 10, Gemini 10, Claude 20; DeepSeek Chat's 50 is the outlier), and a vision-model image costs roughly 1300–4800 tokens, so 50 images can fill a 200k context in one message. Including base64 padding, a 3.5 MiB encoded file occupies at most 4.67 MiB and leaves 0.33 MiB below a 5 MiB route check. Deployments using only routes with larger limits can override it. A 512 MiB aggregate cannot pass this transport because base64-in-JSON would need a single JSON string past V8's ~512 MiB string ceiling. The limits reach clients as the `imageLimits` session projection — a constant-per-boot unit (`apply` returns the same state reference, so baselines alone carry it and no change frames exist) registered by **apiproxy**, not the attachment Service Definition: `dsh-llm` depends on `dsh-attachment` (`ImageBlock` → `ImageAttachmentRef`), so the seam package referencing `dsh-session-projection` (whose graph reaches `dsh-llm` through `dsh-session`) closes a project-reference cycle, and the per-message count/aggregate rules the value describes are the proxy's own admission checks anyway. The `SessionProjectionMap` merge rides the proxy's sessions wire-contract file, which every client program already includes through the carrier's type re-exports. **Intake pre-check and error copy.** Both intake gestures converge on one `intakeImages` wrapper in InputBar that checks count, per-image bytes, and aggregate bytes against the projection before `addImages`: a violating batch is refused whole (DeepSeek Chat semantics) with an immediate banner naming the limit — no submit-time rollback theater. The host checks stay as the backstop for callers that bypass the composer. Banner copy follows one principle the user set: reasons a user can act on (model without vision, count, size, resolution, format — now a positive list of supported formats instead of echoing the rejected MIME type) get product sentences naming the way out; reasons they cannot act on (corrupt base64, lost references, read failures) fold into one send-failed sentence that keeps the reason code, because the product currently faces developers and a reportable code beats a dead end. Non-attachment error codes keep the raw message + code presentation. diff --git a/.agents/notes/implemented/feature/2026-08-12-web-image-intake-and-limits-alignment.zh.md b/.agents/notes/implemented/feature/2026-08-12-web-image-intake-and-limits-alignment.zh.md index 62d5ebd542..d88ba25a3f 100644 --- a/.agents/notes/implemented/feature/2026-08-12-web-image-intake-and-limits-alignment.zh.md +++ b/.agents/notes/implemented/feature/2026-08-12-web-image-intake-and-limits-alignment.zh.md @@ -16,7 +16,7 @@ issue #2248 的第二步对齐，接在[附件展示 note](2026-08-11-web-attach **历史缩略图（DeepSeek Chat 规则）。** 一条消息仅有的一张图长边 240px、展示比例钳制在 [0.25, 4]，`cover` 裁切，特别高的图锚定顶部、特别宽的锚定左侧，从不放大；多张图渲染为固定 64px 方块，单个可换行的横排（10px 间距，用户消息右对齐）。assistant 连续的 `image` 块合并进同一个画廊，平铺而不是各占一行。 -**上限对齐并投影。** 默认值为每条消息 20 张、单图 5 MiB、总量 100 MiB（`attachment-local`），HTTP 载体上限提为唯一共享的 `DEFAULT_MAX_REQUEST_BODY_BYTES = 160 MiB`（http-bridge，原先是两个独立的 32 MiB 字面量），以满足加载时的容量断言（总量 × 4/3 加余量 ≈ 134.3 MiB）。消费级产品集中在 10 到 20 个附件（ChatGPT 10、Gemini 10、Claude 20；DeepSeek Chat 的 50 是例外），且视觉模型一张图约 1300 到 4800 token，因此 50 张图可在一条消息中填满 200k 上下文。默认单图上限采用 5 MiB，可适用于分别采用 5 MiB 或 10 MiB 上限的 Anthropic 路由；仅使用较大上限路由的部署可以覆盖该值。512 MiB 总量无法通过当前传输，因为 base64 进 JSON 需要一个超过 V8 约 512 MiB 字符串上限的单个 JSON 字符串。限额以 `imageLimits` 会话投影到达客户端。它是每次启动恒定的单元（`apply` 返回同一状态引用，因此只靠基线携带、不存在变更帧），由 **apiproxy** 而非 attachment Service Definition 注册：`dsh-llm` 依赖 `dsh-attachment`（`ImageBlock` → `ImageAttachmentRef`），seam 包引用 `dsh-session-projection`（其图谱经 `dsh-session` 到达 `dsh-llm`）会闭合 project-reference 环，而该值描述的每消息数量与总量规则本来就是 proxy 自己的准入检查。`SessionProjectionMap` 合并放在 proxy 的 sessions 协议文件里，每个客户端程序都经载体的类型再导出包含它。 +**上限对齐并投影。** 默认值为每条消息 20 张、单图 3.5 MiB、总量 100 MiB（`attachment-local`），HTTP 载体上限提为唯一共享的 `DEFAULT_MAX_REQUEST_BODY_BYTES = 160 MiB`（http-bridge，原先是两个独立的 32 MiB 字面量），以满足加载时的容量断言（总量 × 4/3 加余量 ≈ 134.3 MiB）。消费级产品集中在 10 到 20 个附件（ChatGPT 10、Gemini 10、Claude 20；DeepSeek Chat 的 50 是例外），且视觉模型一张图约 1300 到 4800 token，因此 50 张图可在一条消息中填满 200k 上下文。3.5 MiB 编码文件包括 base64 填充在内最多占 4.67 MiB，在 5 MiB 路由检查下保留 0.33 MiB 余量。仅使用较大上限路由的部署可以覆盖该值。512 MiB 总量无法通过当前传输，因为 base64 进 JSON 需要一个超过 V8 约 512 MiB 字符串上限的单个 JSON 字符串。限额以 `imageLimits` 会话投影到达客户端。它是每次启动恒定的单元（`apply` 返回同一状态引用，因此只靠基线携带、不存在变更帧），由 **apiproxy** 而非 attachment Service Definition 注册：`dsh-llm` 依赖 `dsh-attachment`（`ImageBlock` → `ImageAttachmentRef`），seam 包引用 `dsh-session-projection`（其图谱经 `dsh-session` 到达 `dsh-llm`）会闭合 project-reference 环，而该值描述的每消息数量与总量规则本来就是 proxy 自己的准入检查。`SessionProjectionMap` 合并放在 proxy 的 sessions 协议文件里，每个客户端程序都经载体的类型再导出包含它。 **加入预检与错误文案。** 两种加入手势汇合到 InputBar 的一个 `intakeImages` 包装：在 `addImages` 之前按投影检查数量、单图字节与总字节，违规的一批整体拒收（DeepSeek Chat 语义）并立刻弹出点名上限的横幅——不再有提交时的回滚戏码。宿主检查保留，兜底绕过 composer 的调用方。横幅文案遵循用户定下的一条原则：用户能解决的原因（模型不支持视觉、数量、大小、分辨率、格式——格式改为正面列出支持列表而不是回显被拒的 MIME 类型）用点明出路的产品句子；用户无法解决的原因（base64 损坏、引用丢失、读取失败）折叠为一条保留原因码的发送失败句子，因为产品当前面向开发者，可上报的码好过死胡同。非附件错误码保留原文加错误码的展示。 diff --git a/apps/web/tests/models-settings.e2e.ts b/apps/web/tests/models-settings.e2e.ts index e07426cce1..bb84b2ec03 100644 --- a/apps/web/tests/models-settings.e2e.ts +++ b/apps/web/tests/models-settings.e2e.ts @@ -30,6 +30,7 @@ const EMPTY_EXPECTED = join(SNAPSHOT_DIR, 'empty.expected.md') const CONFIGURED_EXPECTED = join(SNAPSHOT_DIR, 'configured.expected.md') const DECLARED_EXPECTED = join(SNAPSHOT_DIR, 'declared.expected.md') const DECLARED_EDIT_EXPECTED = join(SNAPSHOT_DIR, 'declared-edit.expected.md') +const MODEL_PICKER_EXPECTED = join(SNAPSHOT_DIR, 'model-picker.expected.md') const NATIVE_DELETE_EXPECTED = join(SNAPSHOT_DIR, 'native-delete.expected.md') const DELETE_EXPECTED = join(SNAPSHOT_DIR, 'delete.expected.md') const MODE = webSnapshotMode() @@ -178,6 +179,42 @@ describe('web e2e: Models settings page configures a dormant provider', () => { expect(tripwire.pageErrors).toEqual([]) }, 60_000) + it('selects and clears the discovered model catalog in one action', async () => { + onTestFailed(() => saveFailureShot(page, 'web-e2e-models-picker')) + const settingsDialog = page.getByRole('dialog', { name: '设置' }) + await settingsDialog.getByRole('button', { name: '编辑 minimax-cn' }).click() + await settingsDialog.getByText('自定义设置').click() + await settingsDialog.getByRole('button', { name: '获取可用模型' }).click() + + const picker = page.getByRole('dialog', { name: '选择要添加的模型' }) + await picker.waitFor({ timeout: 10_000 }) + const boxes = picker.getByRole('checkbox') + const count = await boxes.count() + expect(count).toBeGreaterThan(0) + expect(await boxes.evaluateAll(nodes => nodes.map(node => (node as HTMLInputElement).checked))).toEqual( + Array.from({ length: count }, () => true), + ) + + await picker.getByRole('button', { name: '取消全选' }).click() + expect(await boxes.evaluateAll(nodes => nodes.map(node => (node as HTMLInputElement).checked))).toEqual( + Array.from({ length: count }, () => false), + ) + await picker.getByRole('button', { name: '全选' }).waitFor() + const snapshot = await captureStableAria( + page, + '[role="dialog"][aria-label="选择要添加的模型"]', + scaffold.workspaceCwd, + ) + await compareOrRefreshGolden(MODEL_PICKER_EXPECTED, snapshot, MODE) + + await picker.getByRole('button', { name: '全选' }).click() + expect(await boxes.evaluateAll(nodes => nodes.map(node => (node as HTMLInputElement).checked))).toEqual( + Array.from({ length: count }, () => true), + ) + await picker.getByRole('button', { name: '取消', exact: true }).click() + await settingsDialog.getByRole('button', { name: '取消', exact: true }).click() + }, 60_000) + it('declares a route the adapter does not ship', async () => { onTestFailed(() => saveFailureShot(page, 'web-e2e-models-declare')) const dialog = page.getByRole('dialog', { name: '设置' }) @@ -280,7 +317,8 @@ describe('web e2e: Models settings page configures a dormant provider', () => { it.skipIf(MODE === 'record')('keeps the fixture inventory closed', async () => { await assertFixtureInventory(SNAPSHOT_DIR, [ 'configured.expected.md', 'declared-edit.expected.md', 'declared.expected.md', - 'delete.expected.md', 'empty.expected.md', 'native-delete.expected.md', + 'delete.expected.md', 'empty.expected.md', 'model-picker.expected.md', + 'native-delete.expected.md', ]) }) }) diff --git a/apps/web/tests/snapshots/models-settings/model-picker.expected.md b/apps/web/tests/snapshots/models-settings/model-picker.expected.md new file mode 100644 index 0000000000..3a482d8414 --- /dev/null +++ b/apps/web/tests/snapshots/models-settings/model-picker.expected.md @@ -0,0 +1,18 @@ +- dialog "选择要添加的模型": + - heading "选择要添加的模型" [level=2] + - button "关闭": + - img + - paragraph: 以下是模型提供方的可用模型，勾选要添加的模型。 + - button "全选" + - list: + - listitem: + - checkbox "MiniMax-M2.7" + - text: MiniMax-M2.7 + - listitem: + - checkbox "MiniMax-M2.7-highspeed" + - text: MiniMax-M2.7-highspeed + - listitem: + - checkbox "MiniMax-M3" + - text: MiniMax-M3 + - button "取消" + - button "添加所选" diff --git a/docs/config-catalog.i18n.yaml b/docs/config-catalog.i18n.yaml index 5aed4b89fb..74f56627fe 100644 --- a/docs/config-catalog.i18n.yaml +++ b/docs/config-catalog.i18n.yaml @@ -2,5 +2,5 @@ # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: # pnpm run verify-translation-pairing --write docs/config-catalog.md -config-catalog.md: 57761846ca2c170b4335b6331a40910fd032eb25 -config-catalog.zh.md: 52ea5c9692c143cf9b5b1246db8356d7a83f15c4 +config-catalog.md: 54875326197c9d4501a72719005c222b702c3eb5 +config-catalog.zh.md: b832a6f89821bc54ec2c07b2fa7c458f1e22b43d diff --git a/docs/config-catalog.md b/docs/config-catalog.md index 57761846ca..5487532619 100644 --- a/docs/config-catalog.md +++ b/docs/config-catalog.md @@ -985,6 +985,13 @@ export interface PiAiProviderProfile { websocketConnectTimeoutMs?: number /** Maximum provider idle time while one stream read is outstanding. */ streamIdleTimeoutMs?: number + /** + * Maximum base64-encoded image payload per request. When a request's + * accumulated images exceed it, the oldest images are replaced by text + * placeholders until the request fits, so a long session keeps completing + * requests instead of being rejected by a request-size cap. + */ + maxRequestImageBytes?: number /** Provider-owned model-request retry policy; omission uses normal mode with five retries. */ retryPolicy?: RetryPolicyConfig } @@ -1081,7 +1088,7 @@ type WithheldThinkingFormat = 'chat-template' | 'qwen-chat-template' Depends on: `Api` (`@earendil-works/pi-ai`) · `CacheRetention` (`@earendil-works/pi-ai`) · `Model` (`@earendil-works/pi-ai`) · `ModelThinkingLevel` (`@earendil-works/pi-ai`) · `OpenAICompletionsCompat` (`@earendil-works/pi-ai`) · [`RetryPolicyConfig`](../packages/llm/llm/src/index.ts) · `ThinkingBudgets` (`@earendil-works/pi-ai`) · `Transport` (`@earendil-works/pi-ai`) -Source: [`packages/llm/llm-pi-ai/src/config.ts:172`](../packages/llm/llm-pi-ai/src/config.ts) +Source: [`packages/llm/llm-pi-ai/src/config.ts:192`](../packages/llm/llm-pi-ai/src/config.ts) diff --git a/docs/config-catalog.zh.md b/docs/config-catalog.zh.md index 52ea5c9692..b832a6f898 100644 --- a/docs/config-catalog.zh.md +++ b/docs/config-catalog.zh.md @@ -987,6 +987,13 @@ export interface PiAiProviderProfile { websocketConnectTimeoutMs?: number /** Maximum provider idle time while one stream read is outstanding. */ streamIdleTimeoutMs?: number + /** + * Maximum base64-encoded image payload per request. When a request's + * accumulated images exceed it, the oldest images are replaced by text + * placeholders until the request fits, so a long session keeps completing + * requests instead of being rejected by a request-size cap. + */ + maxRequestImageBytes?: number /** Provider-owned model-request retry policy; omission uses normal mode with five retries. */ retryPolicy?: RetryPolicyConfig } @@ -1083,7 +1090,7 @@ type WithheldThinkingFormat = 'chat-template' | 'qwen-chat-template' 依赖：`Api`（`@earendil-works/pi-ai`）· `CacheRetention`（`@earendil-works/pi-ai`）· `Model`（`@earendil-works/pi-ai`）· `ModelThinkingLevel`（`@earendil-works/pi-ai`）· `OpenAICompletionsCompat`（`@earendil-works/pi-ai`）· [`RetryPolicyConfig`](../packages/llm/llm/src/index.ts) · `ThinkingBudgets`（`@earendil-works/pi-ai`）· `Transport`（`@earendil-works/pi-ai`） -来源：[`packages/llm/llm-pi-ai/src/config.ts:172`](../packages/llm/llm-pi-ai/src/config.ts) +来源：[`packages/llm/llm-pi-ai/src/config.ts:192`](../packages/llm/llm-pi-ai/src/config.ts) diff --git a/examples/acp-agent/tests/acp.snapshot.ts b/examples/acp-agent/tests/acp.snapshot.ts index c5bf81fc1f..e617206dcb 100644 --- a/examples/acp-agent/tests/acp.snapshot.ts +++ b/examples/acp-agent/tests/acp.snapshot.ts @@ -1,11 +1,19 @@ import { fileURLToPath } from 'node:url' import { readFileSync } from 'node:fs' import { spawnSync } from 'node:child_process' +import { createServer } from 'node:http' +import type { IncomingMessage, ServerResponse } from 'node:http' import { mkdir, utimes, writeFile } from 'node:fs/promises' import { dirname, join } from 'node:path' import { homedir } from 'node:os' import { expect, it } from 'vitest' -import { defineAcpSnapshotSuite, type Scenario, type SnapshotSuiteOptions } from '@deepseek-ai/dsh-acp-snapshot' +import { + defineAcpSnapshotSuite, + runScenario, + type InputScript, + type Scenario, + type SnapshotSuiteOptions, +} from '@deepseek-ai/dsh-acp-snapshot' import { resolvePwshPath } from '@deepseek-ai/dsh-pwsh-local' import { decodeStorageRecord } from '@deepseek-ai/dsh-session' @@ -40,6 +48,7 @@ const ADVANCED_CONFIG = fileURLToPath(new URL('../advanced.cordis.yml', import.m const FS_CONFIG = fileURLToPath(new URL('../fs.cordis.yml', import.meta.url)) const SESSION_QUERY_CONFIG = fileURLToPath(new URL('../session-query.cordis.yml', import.meta.url)) const IMAGE_CONFIG = fileURLToPath(new URL('../image.cordis.yml', import.meta.url)) +const IMAGE_OFFLOAD_CONFIG = fileURLToPath(new URL('./fixtures/image-offload.cordis.yml', import.meta.url)) const IMAGE_TEXT_ROUTE_CONFIG = fileURLToPath(new URL('../image-text-route.cordis.yml', import.meta.url)) const PTY_CONFIG = fileURLToPath(new URL('../pty.cordis.yml', import.meta.url)) const DEPTH_TWO_CONFIG = fileURLToPath(new URL('../depth-two.cordis.yml', import.meta.url)) @@ -655,6 +664,92 @@ defineAcpSnapshotSuite({ hasPwsh, }) +it('pins pi-ai image offload in the request sent by the assembled app', async () => { + const requests: Record[] = [] + const server = createServer((request: IncomingMessage, response: ServerResponse) => { + let body = '' + request.setEncoding('utf8') + request.on('data', (chunk: string) => { body += chunk }) + request.on('end', () => { + requests.push(JSON.parse(body) as Record) + response.writeHead(200, { 'content-type': 'text/event-stream' }) + response.end([ + 'data: {"choices":[{"delta":{"role":"assistant","content":""},"index":0,"finish_reason":null}]}', + 'data: {"choices":[{"delta":{"content":"DONE"},"index":0,"finish_reason":null}]}', + 'data: {"choices":[{"delta":{},"index":0,"finish_reason":"stop"}],"usage":{"prompt_tokens":3,"completion_tokens":1}}', + 'data: [DONE]', + '', + ].join('\n\n')) + }) + }) + await new Promise(resolve => server.listen(0, '127.0.0.1', resolve)) + const address = server.address() + if (address === null || typeof address === 'string') throw new Error('image-offload snapshot server has no port') + + const image = 'iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAIAAACQd1PeAAAADElEQVR4nGP4z8AAAAMBAQDJ/pLvAAAAAElFTkSuQmCC' + const input: InputScript = { + steps: [ + { op: 'initialize' }, + { op: 'newSession' }, + { + op: 'promptContent', + content: [ + { type: 'text', text: 'Compare the older image ' }, + { type: 'image', data: image, mimeType: 'image/png' }, + { type: 'text', text: ' with the newer image ' }, + { type: 'image', data: image, mimeType: 'image/png' }, + { type: 'text', text: ', then reply with DONE.' }, + ], + }, + ], + } + + try { + const result = await runScenario(input, { + agent: AGENT, + mode: 'record', + configPath: IMAGE_OFFLOAD_CONFIG, + fixtureFile: join(SNAPSHOTS_DIR, 'image-offload-request', 'session.jsonl'), + env: { + DSH_SNAPSHOT_API_KEY: 'snapshot-key', + DSH_SNAPSHOT_BASE_URL: `http://127.0.0.1:${address.port}/v1`, + }, + }) + expect(result.stderr).toBe('') + expect(requests).toHaveLength(1) + const messages = requests[0]?.messages as { content?: unknown }[] | undefined + const offloaded = messages?.find(message => JSON.stringify(message.content).includes('[image omitted')) + expect(offloaded?.content).toMatchInlineSnapshot(` + [ + { + "text": "Compare the older image ", + "type": "text", + }, + { + "text": "[image omitted to keep the request within its image limit; older images are omitted first. If this image is still needed, read its file again when a path is available; otherwise ask the user to attach it again.]", + "type": "text", + }, + { + "text": " with the newer image ", + "type": "text", + }, + { + "image_url": { + "url": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAIAAACQd1PeAAAADElEQVR4nGP4z8AAAAMBAQDJ/pLvAAAAAElFTkSuQmCC", + }, + "type": "image_url", + }, + { + "text": ", then reply with DONE.", + "type": "text", + }, + ] + `) + } finally { + await new Promise(resolve => server.close(() => { resolve() })) + } +}, 45_000) + it('packed ACP fixture retains every chunk row kind without changing the logical session', () => { const source = fixtureRecords(PACKED_CHUNKS_SOURCE) const packed = fixtureRecords('packed-chunks') diff --git a/examples/acp-agent/tests/fixtures/image-offload.cordis.yml b/examples/acp-agent/tests/fixtures/image-offload.cordis.yml new file mode 100644 index 0000000000..6007c5d44d --- /dev/null +++ b/examples/acp-agent/tests/fixtures/image-offload.cordis.yml @@ -0,0 +1,41 @@ +# Keyless assembled-request snapshot for pi-ai image offload. The local +# provider endpoint is supplied by the snapshot test; the real attachment +# store and ACP bridge carry two uploaded images into one model request. +- id: base + name: '@deepseek-ai/cordis-plugin-include' + config: + path: ../../cordis.yml + patches: + - id: llm-deepseek + name: '@deepseek-ai/dsh-llm-deepseek' + disabled: true + - id: acp-agent + name: '@deepseek-ai/dsh-acp-demo' + config: + provider: snapshot + model: vision + persistenceRoot: !!js process.env.DSH_SNAPSHOT_SESSIONS_ROOT ?? './.sessions' + persistenceCompression: none + workspaceContext: + maxBytes: 65536 + persona: | + You are a coding assistant powered by the {{model}} model. Your working directory is {{cwd}}. + + Keep answers brief and factual. + - insert: + - id: attachment-local + name: '@deepseek-ai/dsh-attachment-local' + - id: llm-pi-ai + name: '@deepseek-ai/dsh-llm-pi-ai' + config: + providers: + snapshot: + apiKeyEnv: DSH_SNAPSHOT_API_KEY + api: openai-completions + baseURL: !!js process.env.DSH_SNAPSHOT_BASE_URL + maxRequestImageBytes: 92 + models: + - id: vision + contextWindow: 32768 + maxTokens: 1024 + input: [text, image] diff --git a/packages/attachment/attachment-local/src/index.ts b/packages/attachment/attachment-local/src/index.ts index 344830daee..a4047da1f1 100644 --- a/packages/attachment/attachment-local/src/index.ts +++ b/packages/attachment/attachment-local/src/index.ts @@ -11,7 +11,7 @@ import { readImageFile, saveImageFile, validateImageFile } from './store.ts' export { readImageFile, saveImageFile, validateImageFile } from './store.ts' /** Default maximum encoded bytes for one image. */ -export const DEFAULT_MAX_IMAGE_BYTES = 5 * 1024 * 1024 +export const DEFAULT_MAX_IMAGE_BYTES = 3.5 * 1024 * 1024 /** Default maximum images in one prompt. */ export const DEFAULT_MAX_IMAGES_PER_MESSAGE = 20 /** Default maximum aggregate image bytes in one prompt. */ diff --git a/packages/attachment/attachment-local/tests/index.spec.ts b/packages/attachment/attachment-local/tests/index.spec.ts index 859008c7d7..92bbe3c0aa 100644 --- a/packages/attachment/attachment-local/tests/index.spec.ts +++ b/packages/attachment/attachment-local/tests/index.spec.ts @@ -15,7 +15,7 @@ import LocalAttachmentStore, { describe('local attachment service', () => { it('resolves every omitted admission limit explicitly', () => { const service = new LocalAttachmentStore(new Context(), {}) - expect(DEFAULT_MAX_IMAGE_BYTES).toBe(5 * 1024 * 1024) + expect(DEFAULT_MAX_IMAGE_BYTES).toBe(3.5 * 1024 * 1024) expect(service.imageLimits).toEqual({ maxImageBytes: DEFAULT_MAX_IMAGE_BYTES, maxImagesPerMessage: DEFAULT_MAX_IMAGES_PER_MESSAGE, diff --git a/packages/client/ui-settings-models/README.i18n.yaml b/packages/client/ui-settings-models/README.i18n.yaml index f26ca67bc3..950124960a 100644 --- a/packages/client/ui-settings-models/README.i18n.yaml +++ b/packages/client/ui-settings-models/README.i18n.yaml @@ -2,5 +2,5 @@ # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: # pnpm run verify-translation-pairing --write packages/client/ui-settings-models/README.md -README.md: f0094c4f2687beaf6a5e5f7a1a8dc3d4ebdbcbc0 -README.zh.md: 68bb64ddeebfd65644edbb5e11fd2e147214d44a +README.md: 0d76d5f31585721ad8f5cd76dd223d72353fdd0c +README.zh.md: 8ac336245c9d82420c10132b3b935cf00f5ea889 diff --git a/packages/client/ui-settings-models/README.md b/packages/client/ui-settings-models/README.md index f0094c4f26..0d76d5f315 100644 --- a/packages/client/ui-settings-models/README.md +++ b/packages/client/ui-settings-models/README.md @@ -16,7 +16,7 @@ Every edit lands as `settings.mutate` path ops against the stored section — a A pi-ai profile's `models` list is edited on the card: one row per model showing its id and display name, with the context window and output cap behind a per-row disclosure and two label-free actions — expand and delete — on the right. An empty list means "serve this route's built-in catalog", so a row is only ever added deliberately; clearing a capacity drops it rather than storing a value the schema would reject, and the adapter's route-level fallbacks size whatever configuration leaves out — an empty capacity shows those fallbacks' magnitude as its placeholder, a hint rather than a mirror, since the field counts `K` as 1000 and a deployment may override them. A capacity that is not a positive integer is simply not stored. -**Fetch available models** asks `llm.discoverModels` about the endpoint the form **currently shows**, including a base URL edited but not yet saved and a key typed but not yet stored, so adding a provider is one pass instead of save-then-return. The reply opens a picker rather than being written: candidates already configured start unchecked, so adopting a selection never overwrites a capacity the user corrected. A provider that cannot be interrogated is a detour, not a dead end — the adapter's own message appears beside the rows, which stay editable by hand. +**Fetch available models** asks `llm.discoverModels` about the endpoint the form **currently shows**, including a base URL edited but not yet saved and a key typed but not yet stored, so adding a provider is one pass instead of save-then-return. The reply opens a picker rather than being written: candidates already configured start unchecked, so adopting a selection never overwrites a capacity the user corrected. **Select all** / **Deselect all** changes only the picker's current checkboxes; nothing is written until **Add selected**. A provider that cannot be interrogated is a detour, not a dead end — the adapter's own message appears beside the rows, which stay editable by hand. **Add a custom provider** declares a route pi-ai does not ship. It is its own card rather than the editor with extra fields, because the route id is being chosen here and the settings address does not exist until it is: one `settings.mutate` sets the whole profile at `providers.`, and the key travels separately through `credentials.set` under the same `_API_KEY` derivation an existing provider uses. What a hand-declared route cannot default gates the create button — a unique **Provider ID**, an endpoint, a protocol, and at least one uniquely-identified model — so the failure names the field while the user is still looking at it. The id must start with a lowercase letter, because it is also the stem of the derived credential reference and a reference is a POSIX shell identifier: a digit-leading id otherwise passes every check this card makes and then fails at the credential seam with a raw regular expression. Capacities do not gate it: the adapter's fallbacks size a model the endpoint described by id alone, which is what most listings return. The protocol choices are read out of the namespace's own schema rather than a wire field or a constant, so they cannot drift from the ones the adapter accepts. The card records the conventional `apiKeyEnv` reference only when a key is typed, the same rule the editor applies, so a route declared for provider-native authentication is not born pointing at a reference nothing will ever set. When the profile write lands but the key write fails, the provider already exists: the card settles the fields describing it, retries the credential alone — re-running the profile write would carry the revision that write just superseded, so the Host would answer `settings-conflict` and the key could never be stored from here — and reports the created provider even if the user then cancels. diff --git a/packages/client/ui-settings-models/README.zh.md b/packages/client/ui-settings-models/README.zh.md index 68bb64ddee..8ac336245c 100644 --- a/packages/client/ui-settings-models/README.zh.md +++ b/packages/client/ui-settings-models/README.zh.md @@ -16,7 +16,7 @@ pi-ai profile 的 `models` 列表就在卡片上编辑：一行一个模型，行上显示 id 与显示名称，上下文窗口与输出上限收在该行的展开区内，右侧是两个无文字的操作——展开与删除。空列表意味着「使用该路由的内置 catalog」，因此每一行都只会被刻意添加；清空容量会丢弃它，而不是存入一个 schema 会拒绝的值，配置留空的部分由适配器的路由级回退值定尺寸——留空的容量以这些回退值的量级作为占位符，那只是提示而非镜像：该字段按 1000 计 `K`，且部署可以覆盖这些回退值。不是正整数的容量根本不会被存下。 -**获取可用模型**会针对表单**当前显示**的端点调用 `llm.discoverModels`，包括已修改但尚未保存的 API 地址和已键入但尚未存储的密钥，因此新增一个提供方是一趟走完，而不是「先保存再回来」。回复会打开一个选择框而不是直接写入：已配置过的候选默认不勾选，因此采纳一次选择绝不会覆盖用户已更正的容量。无法被询问的提供方只是绕路而非死路——适配器自己的消息会显示在各行旁边，而这些行仍可手工编辑。 +**获取可用模型**会针对表单**当前显示**的端点调用 `llm.discoverModels`，包括已修改但尚未保存的 API 地址和已键入但尚未存储的密钥，因此新增一个提供方是一趟走完，而不是「先保存再回来」。回复会打开一个选择框而不是直接写入：已配置过的候选默认不勾选，因此采纳一次选择绝不会覆盖用户已更正的容量。**全选**／**取消全选**只会改变选择框当前的勾选状态；点击**添加所选**之前不会写入任何内容。无法被询问的提供方只是绕路而非死路——适配器自己的消息会显示在各行旁边，而这些行仍可手工编辑。 **添加自定义提供方**用来声明 pi-ai 未提供的路由。它是独立的一张卡片而非在编辑器上加字段，因为路由 id 正是在这里被*选定*的，而在选定之前 settings 地址并不存在：一次 `settings.mutate` 在 `providers.` 上设置整个 profile，密钥则经 `credentials.set` 单独传递，使用与既有提供方相同的 `_API_KEY` 派生。手工声明的路由无法默认的东西会门控创建按钮——唯一的 **Provider ID**、端点、协议，以及至少一个标识唯一的模型——因此失败会在用户仍看着该字段时点名它。该 id 必须以小写字母开头，因为它同时是派生凭据引用的词干，而引用是 POSIX shell 标识符：数字开头的 id 否则会通过这张卡片的每一项检查，然后在凭据 seam 上抛出原始正则表达式错误。容量不参与门控：端点只按 id 描述的模型（这正是多数列表返回的形态）由适配器的回退值定尺寸。协议选项读自该 namespace 自己的 schema，而非某个协议字段或常量，因此它们不会与适配器实际接受的集合发生漂移。只有键入了密钥，这张卡片才记录约定的 `apiKeyEnv` 引用，与编辑器同一条规则，因此一条为提供方原生认证声明的路由不会一出生就指向一个永远不会被设置的引用。当 profile 写入成功而密钥写入失败时，提供方已经存在：卡片会把描述它的字段定住，只重试凭据——再跑一次 profile 写入会带着刚被自己这次写入取代的 revision，宿主将以 `settings-conflict` 应答，密钥就再也无法从这里存下——并且即使用户随后取消，也照实报告提供方已创建。 diff --git a/packages/client/ui-settings-models/src/client/ModelListEditor.tsx b/packages/client/ui-settings-models/src/client/ModelListEditor.tsx index 6d88484ac5..a4b80f52a0 100644 --- a/packages/client/ui-settings-models/src/client/ModelListEditor.tsx +++ b/packages/client/ui-settings-models/src/client/ModelListEditor.tsx @@ -290,6 +290,18 @@ export function ModelListEditor(props: ModelListEditorProps): ReactNode { }) } + const activeCandidates = candidates ?? [] + const allCandidatesPicked = activeCandidates.length > 0 + && activeCandidates.every(candidate => picked.has(candidate.id)) + + const toggleAllCandidates = (): void => { + setPicked((current) => { + return activeCandidates.every(candidate => current.has(candidate.id)) + ? new Set() + : new Set(activeCandidates.map(candidate => candidate.id)) + }) + } + // A route the adapter already describes answers without an endpoint; only a // draft with neither has nothing to ask about. const askable = probe.provider !== undefined || (probe.baseURL !== undefined && probe.baseURL.length > 0) @@ -445,6 +457,11 @@ export function ModelListEditor(props: ModelListEditorProps): ReactNode { )} > +
+ +
    {(candidates ?? []).map(candidate => (
  • diff --git a/packages/client/ui-settings-models/src/client/ModelsSection.module.css b/packages/client/ui-settings-models/src/client/ModelsSection.module.css index 5203524517..6b8c2047a4 100644 --- a/packages/client/ui-settings-models/src/client/ModelsSection.module.css +++ b/packages/client/ui-settings-models/src/client/ModelsSection.module.css @@ -644,6 +644,12 @@ select.input { --dsh-scrollbar-thumb-hover: var(--dsw-alias-scrollbar-hover-l2); } +.candidateActions { + display: flex; + justify-content: flex-end; + margin-bottom: 6px; +} + .candidateList { display: flex; flex-direction: column; diff --git a/packages/client/ui-settings-models/src/client/locales.ts b/packages/client/ui-settings-models/src/client/locales.ts index a49a4f0ef2..856ef64c7b 100644 --- a/packages/client/ui-settings-models/src/client/locales.ts +++ b/packages/client/ui-settings-models/src/client/locales.ts @@ -72,6 +72,8 @@ export const en = { fetchEmpty: 'The provider listed no models. Add them by hand.', fetchTitle: 'Choose models to add', fetchDescription: 'These are the models this provider has available. Choose the ones to add.', + fetchSelectAll: 'Select all', + fetchDeselectAll: 'Deselect all', fetchAdopt: 'Add selected', customAdd: 'Add a custom provider', customTitle: 'Custom provider', @@ -172,6 +174,8 @@ export const zh: { [Key in keyof typeof en]: string } = { fetchEmpty: '该提供方没有列出任何模型，请手动添加。', fetchTitle: '选择要添加的模型', fetchDescription: '以下是模型提供方的可用模型，勾选要添加的模型。', + fetchSelectAll: '全选', + fetchDeselectAll: '取消全选', fetchAdopt: '添加所选', customAdd: '添加自定义提供方', customTitle: '自定义提供方', diff --git a/packages/client/ui-settings-models/tests/provider-form.client.spec.tsx b/packages/client/ui-settings-models/tests/provider-form.client.spec.tsx index e5e32498d1..4bc30f558a 100644 --- a/packages/client/ui-settings-models/tests/provider-form.client.spec.tsx +++ b/packages/client/ui-settings-models/tests/provider-form.client.spec.tsx @@ -605,6 +605,27 @@ describe('endpoint interrogation', () => { // A disclosed output cap rides along with the candidate that has one. expect(firstMutate(mutate).ops[0]?.value).toEqual([{ id: 'a' }, { id: 'b', maxTokens: 2048 }]) }) + + it('selects and clears every discovered candidate in one action', async () => { + const discover = vi.fn(() => Promise.resolve(ok({ + models: [{ id: 'a' }, { id: 'b' }, { id: 'c' }], + }))) + await mountSection({ discover }) + openEditor('openai') + + fireEvent.click(screen.getByText(en.fetchModels)) + const dialog = await screen.findByRole('dialog') + const boxes = [...dialog.querySelectorAll('input[type="checkbox"]')] + expect(boxes.map(box => box.checked)).toEqual([true, true, true]) + + fireEvent.click(within_(dialog, en.fetchDeselectAll)) + expect(boxes.map(box => box.checked)).toEqual([false, false, false]) + expect(within_(dialog, en.fetchSelectAll)).toBeTruthy() + + fireEvent.click(within_(dialog, en.fetchSelectAll)) + expect(boxes.map(box => box.checked)).toEqual([true, true, true]) + expect(within_(dialog, en.fetchDeselectAll)).toBeTruthy() + }) }) describe('provider rows', () => { diff --git a/packages/llm/llm-pi-ai/README.i18n.yaml b/packages/llm/llm-pi-ai/README.i18n.yaml index d79bad503b..694da1fb3c 100644 --- a/packages/llm/llm-pi-ai/README.i18n.yaml +++ b/packages/llm/llm-pi-ai/README.i18n.yaml @@ -2,5 +2,5 @@ # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: # pnpm run verify-translation-pairing --write packages/llm/llm-pi-ai/README.md -README.md: 208615408b7ed450961ce5c47bafe8127b273eb8 -README.zh.md: 101fd563ea391c56fac9d6c33ef01c48671f791f +README.md: f696b6bee50b844bfbc6bab7f9def0e785d450c9 +README.zh.md: cace9d1fdfd85a559674b121b79b3dbb86b4337b diff --git a/packages/llm/llm-pi-ai/README.md b/packages/llm/llm-pi-ai/README.md index 208615408b..f696b6bee5 100644 --- a/packages/llm/llm-pi-ai/README.md +++ b/packages/llm/llm-pi-ai/README.md @@ -113,7 +113,7 @@ A model that carries reasoning metadata — from the installed catalog or from i A model **without** that metadata — a hand-declared one whose entry declares no `reasoningEfforts`, and a catalog model pi-ai marks as non-reasoning — exposes no `reasoning` at all. pi-ai reports such a model as supporting the single level `off`, but `off` is translated to *omitting* the reasoning option, which is byte-for-byte the request that naming no effort already produces: selecting it could not disable anything, so a provider whose own default is to think would keep thinking with `off` shown as selected. Reporting the capability as unavailable leaves a surface offering the provider's default and nothing that misrepresents it. The profile `reasoning` value, including `off`, is the deployment default when configured; omitting it preserves the provider default. Per-request `GenerateOptions.reasoningEffort` takes precedence, and a level absent from the exact model capability fails the REQUEST with `UNSUPPORTED_REASONING_EFFORT` before network I/O instead of being clamped. Describing a model never fails that way: the models under one provider disagree about which levels they accept, so `resolveModel` reports a profile level the exact model cannot take as no default at all rather than throwing. A throw there would take the whole provider out of every model catalog built over it — one mis-set profile field hiding even the models that do support the level — so a bad configuration surfaces where it is acted on, not where it is described. pi-ai's common stream options represent `off` by omitting `reasoning`. -Supported profile fields are `apiKeyEnv`, `displayName`, `api`, `baseURL`, `models`, `modelOverrides`, `compat`, `defaultContextWindow`, `defaultMaxTokens`, `defaultInput`, `headers`, `reasoning`, `thinkingBudgets`, `cacheRetention`, `transport`, `timeoutMs`, `websocketConnectTimeoutMs`, `streamIdleTimeoutMs`, and `retryPolicy`. Each resolved profile retry policy is captured with that provider route; omission uses the shared bounded normal default of five retries. The stream-idle interval is a positive finite Node timer delay, defaults to five minutes, and covers only an outstanding provider read, not consumer think time. Harness app attribution wins a conflicting configured header name. +Supported profile fields are `apiKeyEnv`, `displayName`, `api`, `baseURL`, `models`, `modelOverrides`, `compat`, `defaultContextWindow`, `defaultMaxTokens`, `defaultInput`, `headers`, `reasoning`, `thinkingBudgets`, `cacheRetention`, `transport`, `timeoutMs`, `websocketConnectTimeoutMs`, `streamIdleTimeoutMs`, `maxRequestImageBytes`, and `retryPolicy`. Each resolved profile retry policy is captured with that provider route; omission uses the shared bounded normal default of five retries. The stream-idle interval is a positive finite Node timer delay, defaults to five minutes, and covers only an outstanding provider read, not consumer think time. `maxRequestImageBytes` bounds one request's base64-encoded image payload (default 20MiB, a positive integer): every image in history is re-encoded into every request, so when the accumulated payload exceeds the bound, the oldest images are replaced by a fixed text placeholder until the request fits, keeping an image-heavy session serviceable instead of permanently rejected by a gateway request-size cap. The default leaves capacity for system prompts, history, tools, and JSON; deployments behind stricter gateways lower it per route. Harness app attribution wins a conflicting configured header name. The adapter forces pi-ai's SDK `maxRetries` to zero so one `stream()` call makes one provider request. The removed profile fields `maxRetries` and `maxRetryDelayMs` fail load instead of silently multiplying or hiding the separately composed agent-level retry budget. Idle expiry aborts the SDK's stable request signal and surfaces `TIMEOUT`; an earlier caller abort remains `ABORTED`. @@ -163,15 +163,15 @@ pi-ai installs several provider SDKs and lazy-loads the one selected by the cata #### What the model sees -The selected catalog model receives `GenerateOptions.system`, history, tools, and sampling fields supported by pi-ai's common streaming API. This package adds no prompt prose. Provider-native replay metadata is restored only when the adapter validates it for the historical content. +The selected catalog model receives `GenerateOptions.system`, history, tools, and sampling fields supported by pi-ai's common streaming API. This package adds no prompt prose, with one exception: when a request's accumulated base64 image payload exceeds the route's `maxRequestImageBytes`, each offloaded image (oldest first) is replaced by fixed text. The text tells the model to read the file again when a path is available or ask the user to attach the image again. Provider-native replay metadata is restored only when the adapter validates it for the historical content. #### Token effect -Provider tokenization governs exact input. Conversion adds no model-visible text; replay metadata may let a native API reuse provider-side state. +Provider tokenization governs exact input. Conversion adds no model-visible text beyond the image-offload placeholder, which replaces the offloaded image's visual tokens with a short fixed sentence; replay metadata may let a native API reuse provider-side state. #### KV Cache effect -Conversion preserves logical request order without adding text, while the selected provider's serialization and replay state determine reuse. Changing adapter instance, provider, model, or any upstream request token may prevent reuse from the first difference. +Conversion preserves logical request order without adding text, while the selected provider's serialization and replay state determine reuse. Changing adapter instance, provider, model, or any upstream request token may prevent reuse from the first difference. Crossing the image bound rewrites an early message (the newly offloaded image becomes placeholder text), so reuse ends at that message until the offloaded prefix stabilizes. ### Provider response @@ -189,6 +189,7 @@ Recorded response content appends to the next request and does not invalidate it ## Known Limitations and Deferred Work +- **`maxRequestImageBytes` counts base64 image payload only** — text, tools, and JSON structure ride outside the bound, so it must sit below the gateway's request-body cap with headroom. Offload is decided at request conversion as a pure function of history and configuration and is not recorded as a session event; per-route capability metadata (image count, per-image size, total request size) driving admission and assembly together is deferred design work. - **A provider that authenticates through OAuth alone is not offered** — pi-ai resolves OAuth from a *stored* OAuth credential, and this adapter builds its `Models` collection with no credential store and runs no login flow, so every request on such a route fails `Provider is not configured` before it goes out. The configurable-provider directory withholds them; `openai-codex` is the only one the installed catalog ships. A route a settings document already names keeps its entry so a configuration surface can edit or delete it, and `apiKeyEnv` still authenticates it with that key — which for Codex is a token that expires with nothing here to refresh it. - **Provider-native discovery reads the process environment only** — a route naming no credential defers to the catalog provider's own resolution, which interrogates environment variables (`AZURE_OPENAI_API_KEY`, `AWS_PROFILE`, `AWS_ACCESS_KEY_ID`, and each provider's own set). It reads no local credential directory, so `~/.aws/credentials` without an exported `AWS_PROFILE` resolves as unconfigured, and a value held by the harness credential seam is invisible to it unless the process environment carries it too. - **Settings can add or override routes, not remove composition routes** — the user layer merges over the composition `base`, so deleting a `cordis.yml`-provided provider is a composition change; `replace` on the namespace only resets the user layer. diff --git a/packages/llm/llm-pi-ai/README.zh.md b/packages/llm/llm-pi-ai/README.zh.md index 101fd563ea..cace9d1fdf 100644 --- a/packages/llm/llm-pi-ai/README.zh.md +++ b/packages/llm/llm-pi-ai/README.zh.md @@ -114,7 +114,7 @@ profile 的 `models` 列表是*替换*该路由已安装 catalog，而不是扩 **没有**这份元数据的模型——条目未声明 `reasoningEfforts` 的手工声明模型，以及 pi-ai 标记为不具备推理能力的 catalog 模型——完全不公开 `reasoning`。pi-ai 会把这类模型报告为只支持 `off` 一档，但 `off` 会被翻译成*省略* reasoning 选项，而那与「不点名任何档位」产出的请求逐字节相同：选它关不掉任何东西，于是自身默认就在思考的提供方，会在界面显示 `off` 被选中的同时继续思考。把该能力报告为不可用，界面就只剩提供方默认这一项，不会再出现自相矛盾的控件。配置 profile 的 `reasoning` 值（包括 `off`）在存在时是部署默认值；省略它会保留提供方默认值。每次请求的 `GenerateOptions.reasoningEffort` 优先；未出现在确切模型能力中的档位会让**请求**在网络 I/O 前以 `UNSUPPORTED_REASONING_EFFORT` 失败，而不会被自动调整。**描述**一个模型则从不这样失败：同一提供方下各模型接受的档位并不一致，因此 `resolveModel` 对该模型拿不下的 profile 档位报告为「没有默认值」，而不是抛错。在那里抛错会让整个提供方从任何基于它构建的模型目录中消失——一个配错的 profile 字段连支持该档位的模型也一并藏起来——所以坏配置暴露在被执行处，而不是被描述处。pi-ai 的通用流选项通过省略 `reasoning` 表示 `off`。 -受支持的 profile 字段是 `apiKeyEnv`、`displayName`、`api`、`baseURL`、`models`、`modelOverrides`、`compat`、`defaultContextWindow`、`defaultMaxTokens`、`defaultInput`、`headers`、`reasoning`、`thinkingBudgets`、`cacheRetention`、`transport`、`timeoutMs`、`websocketConnectTimeoutMs`、`streamIdleTimeoutMs` 和 `retryPolicy`。每条 profile 解析后的重试策略会随该提供方路由一同捕获；省略时使用共享的有界 normal 默认值并重试五次。流空闲间隔必须是正的有限 Node 定时器延迟，默认为五分钟，且只覆盖未完成提供方读取，不包括消费方思考时间。若已配置标头中有同名项，则以 Harness 应用归因为准。 +受支持的 profile 字段是 `apiKeyEnv`、`displayName`、`api`、`baseURL`、`models`、`modelOverrides`、`compat`、`defaultContextWindow`、`defaultMaxTokens`、`defaultInput`、`headers`、`reasoning`、`thinkingBudgets`、`cacheRetention`、`transport`、`timeoutMs`、`websocketConnectTimeoutMs`、`streamIdleTimeoutMs`、`maxRequestImageBytes` 和 `retryPolicy`。每条 profile 解析后的重试策略会随该提供方路由一同捕获；省略时使用共享的有界 normal 默认值并重试五次。流空闲间隔必须是正的有限 Node 定时器延迟，默认为五分钟，且只覆盖未完成提供方读取，不包括消费方思考时间。`maxRequestImageBytes` 约束单个请求的 base64 编码图片载荷（默认 20MiB，正整数）：历史中的每张图片都会重新编码进每个请求，累积载荷超过上限时，从最老的图片开始替换为固定文本占位，直到请求装得下，使图片较多的会话保持可用，而不是被网关请求体上限永久拒绝。默认值为系统提示词、历史、工具与 JSON 保留请求容量；网关更严格的部署按路由调低该值。若已配置标头中有同名项，则以 Harness 应用归因为准。 适配器强制 pi-ai SDK `maxRetries` 为零，因此一次 `stream()` 调用只会发起一次提供方请求。已移除 profile 字段 `maxRetries` 和 `maxRetryDelayMs` 会使加载失败，而不是静默倍增或隐藏单独组合的 agent（智能体）级重试预算。空闲超时会 abort SDK 的稳定请求信号，并以 `TIMEOUT` 呈现；较早的调用方 abort 仍为 `ABORTED`。 @@ -164,15 +164,15 @@ pi-ai 会安装多个提供方 SDK，并延迟加载 catalog 模型所选的 SDK #### 模型看到的内容 -所选 catalog 模型会收到 `GenerateOptions.system`、历史、工具，以及 pi-ai 通用流式 API 支持的采样字段。本包不添加提示词文本。只有当适配器验证提供方原生回放元数据与历史内容匹配时，才会恢复这些元数据。 +所选 catalog 模型会收到 `GenerateOptions.system`、历史、工具，以及 pi-ai 通用流式 API 支持的采样字段。本包不添加提示词文本，仅有一个例外：请求累积的 base64 图片载荷超过路由的 `maxRequestImageBytes` 时，被 offload 的图片（从最老开始）会被替换为一段固定文本。该文本要求模型在有路径时重新读取文件，否则请用户重新附上图片。只有当适配器验证提供方原生回放元数据与历史内容匹配时，才会恢复这些元数据。 #### Token 影响 -精确输入取决于提供方 tokenization。转换不添加模型可见文本；回放元数据可能让原生 API 复用提供方侧状态。 +精确输入取决于提供方 tokenization。除图片 offload 占位文本外，转换不添加模型可见文本；占位文本用一句固定短句替代被省略图片的视觉 token。回放元数据可能让原生 API 复用提供方侧状态。 #### KV Cache 影响 -转换保留逻辑请求顺序，不添加文本；复用取决于所选提供方的序列化与回放状态。更改适配器实例、提供方、模型或任何上游请求 token，都可能使复用从首个出现差异的 token 起失效。 +转换保留逻辑请求顺序，不添加文本；复用取决于所选提供方的序列化与回放状态。更改适配器实例、提供方、模型或任何上游请求 token，都可能使复用从首个出现差异的 token 起失效。跨过图片上限会改写较早的一条消息（新被 offload 的图片变为占位文本），复用在该消息处截止，直到被 offload 的前缀稳定。 ### 提供方响应 @@ -190,6 +190,7 @@ pi-ai 事件会变为 harness 推理、文本、工具调用、usage 与 finish ## 已知限制与暂缓事项 +- **`maxRequestImageBytes` 只统计 base64 图片载荷**：文本、工具与 JSON 结构不计入上限，因此该值必须低于网关请求体上限并留出余量。offload 在请求转换时决定，是历史与配置的纯函数，不记录为会话事件；由按路由能力元数据（图片数量、单图大小、请求总大小）同时驱动准入与组装的完整设计属于暂缓工作。 - **仅以 OAuth 认证的提供方不予提供**：pi-ai 的 OAuth 只从*已存储*的 OAuth 凭据解析，而本适配器构造 `Models` 集合时不注入凭据存储、也不运行登录流程，因此这类路由的每个请求都会在发出之前以 `Provider is not configured` 失败。可配置提供方目录因此不列出它们；已安装 catalog 中只有 `openai-codex` 属于此类。settings 文档已经写过的路由仍保留目录条目，配置界面据此可以编辑或删除；`apiKeyEnv` 也仍能用该密钥完成认证——对 Codex 而言那是一个会过期、且这里没有任何环节会去刷新的 token。 - **提供方自带的凭据发现只读进程环境**：不指定凭据的路由交由 catalog 提供方自行解析，而它探测的是环境变量（`AZURE_OPENAI_API_KEY`、`AWS_PROFILE`、`AWS_ACCESS_KEY_ID` 以及各提供方自己的那一组）。它不读任何本地凭据目录，因此只有 `~/.aws/credentials` 而未导出 `AWS_PROFILE` 会被解析为未配置；由 harness 凭据 seam 保管的值，除非进程环境里也有，否则对它不可见。 - **settings 能新增或覆盖路由，但不能移除组合路由**：用户层合并在组合 `base` 之上，因此删除 `cordis.yml` 提供的提供方属于组合变更；对该 namespace 执行 `replace` 只会重置用户层。 diff --git a/packages/llm/llm-pi-ai/src/adapter.ts b/packages/llm/llm-pi-ai/src/adapter.ts index ab1c784351..5ecec593da 100644 --- a/packages/llm/llm-pi-ai/src/adapter.ts +++ b/packages/llm/llm-pi-ai/src/adapter.ts @@ -317,7 +317,7 @@ export class PiAiAdapter extends LlmAdapter { } const context = attachments === undefined ? toPiContext(options, undefined, onReplayDegrade) - : await toPiContext(options, attachments, onReplayDegrade) + : await toPiContext(options, attachments, onReplayDegrade, profile.maxRequestImageBytes) const events = snapshot.models.streamSimple(model, context, { ...profileOptions(profile, reasoning, apiKey), ...options.temperature === undefined ? {} : { temperature: options.temperature }, diff --git a/packages/llm/llm-pi-ai/src/config.ts b/packages/llm/llm-pi-ai/src/config.ts index cd143ea135..d1e1f697a9 100644 --- a/packages/llm/llm-pi-ai/src/config.ts +++ b/packages/llm/llm-pi-ai/src/config.ts @@ -34,6 +34,17 @@ import { buildProvider, supportedProtocols } from './provider.ts' /** Default maximum idle interval while an adapter stream read is outstanding. */ export const DEFAULT_STREAM_IDLE_TIMEOUT_MS = 300_000 +/** + * Default request-level bound on base64-encoded image payload. Every image in + * history is re-encoded into every request body, so an unbounded conversation + * eventually exceeds a provider or gateway request-size cap and the session + * can never complete another request. The 20MiB default admits four images at + * the attachment store's 3.5MiB raw-image default after base64 expansion and + * reserves request capacity for system prompts, history, tools, and JSON. + * Deployments behind stricter gateways lower it per route. + */ +export const DEFAULT_MAX_REQUEST_IMAGE_BYTES = 20 * 1024 * 1024 + /** Context capacity assumed for a model neither configuration nor the catalog sizes. */ export const DEFAULT_CONTEXT_WINDOW = 262_144 @@ -136,6 +147,13 @@ export interface PiAiProviderProfile { websocketConnectTimeoutMs?: number /** Maximum provider idle time while one stream read is outstanding. */ streamIdleTimeoutMs?: number + /** + * Maximum base64-encoded image payload per request. When a request's + * accumulated images exceed it, the oldest images are replaced by text + * placeholders until the request fits, so a long session keeps completing + * requests instead of being rejected by a request-size cap. + */ + maxRequestImageBytes?: number /** Provider-owned model-request retry policy; omission uses normal mode with five retries. */ retryPolicy?: RetryPolicyConfig } @@ -151,6 +169,8 @@ export interface ResolvedPiAiProviderProfile apiKeyEnv?: CredentialRef /** Positive finite provider-idle interval after defaulting. */ streamIdleTimeoutMs: number + /** Positive request-level base64 image payload bound after defaulting. */ + maxRequestImageBytes: number /** Immutable retry policy captured with this provider route. */ retryPolicy: ResolvedRetryPolicy /** @@ -248,6 +268,7 @@ const profile = z.object({ timeoutMs: z.natural(), websocketConnectTimeoutMs: z.natural(), streamIdleTimeoutMs: z.number().min(Number.MIN_VALUE).max(MAX_TIMER_DELAY_MS).default(DEFAULT_STREAM_IDLE_TIMEOUT_MS), + maxRequestImageBytes: z.number().step(1).min(1).default(DEFAULT_MAX_REQUEST_IMAGE_BYTES), retryPolicy: RetryPolicySchema, }) @@ -323,6 +344,10 @@ export function resolveProfiles( `llm-pi-ai: provider "${provider}" streamIdleTimeoutMs must be a positive finite number no greater than ${MAX_TIMER_DELAY_MS}`, ) } + const maxRequestImageBytes = source.maxRequestImageBytes ?? DEFAULT_MAX_REQUEST_IMAGE_BYTES + if (!Number.isInteger(maxRequestImageBytes) || maxRequestImageBytes <= 0) { + throw new Error(`llm-pi-ai: provider "${provider}" maxRequestImageBytes must be a positive integer`) + } // Detached from the configuration object because pi-ai types `Model.input` // mutable. The schema's explicit default covers an absent key, so an empty // list here is always one someone typed — and unlike an entry's, nothing @@ -354,6 +379,7 @@ export function resolveProfiles( displayName, ...apiKeyEnv === undefined ? {} : { apiKeyEnv: credentialRef(apiKeyEnv) }, streamIdleTimeoutMs, + maxRequestImageBytes, retryPolicy: resolveRetryPolicy(retryPolicy, `llm-pi-ai: provider "${provider}" retryPolicy`), ...rest.headers === undefined ? {} : { headers: { ...rest.headers } }, ...rest.thinkingBudgets === undefined ? {} : { thinkingBudgets: { ...rest.thinkingBudgets } }, diff --git a/packages/llm/llm-pi-ai/src/context.ts b/packages/llm/llm-pi-ai/src/context.ts index dcbaabc815..5a2d330b7d 100644 --- a/packages/llm/llm-pi-ai/src/context.ts +++ b/packages/llm/llm-pi-ai/src/context.ts @@ -26,17 +26,82 @@ function toolResultText(blocks: readonly ContentBlock[]): string { : block.type === 'tool-result' ? toolResultText(block.content) : '').join('') } +/** Model-facing stand-in for an image dropped to fit the request bound. */ +export const OFFLOADED_IMAGE_TEXT + = '[image omitted to keep the request within its image limit; older images are omitted first. If this image is still needed, read its file again when a path is available; otherwise ask the user to attach it again.]' + +/** Base64 length of `bytes` raw bytes (4 output characters per 3 input bytes, padded). */ +function base64Length(bytes: number): number { + return Math.ceil(bytes / 3) * 4 +} + +/** + * Select the images a request must drop to fit the per-request payload bound. + * History order is oldest-first, so the most recent images are omitted last. + * A single image larger than the bound is itself omitted. Locations use + * message and nested block indexes so JSON replay cannot change the result by + * splitting or preserving shared object identities. + * @param messages - complete request history, oldest first. + * @param maxRequestImageBytes - bound on total base64-encoded image payload; undefined leaves every image in place. + * @returns the image locations the conversion replaces with {@link OFFLOADED_IMAGE_TEXT}. + */ +function offloadedImages( + messages: readonly Message[], + maxRequestImageBytes: number | undefined, +): ReadonlySet { + const offloaded = new Set() + if (maxRequestImageBytes === undefined) return offloaded + const images: { location: string; base64Bytes: number }[] = [] + const collect = (messageIndex: number, blocks: readonly ContentBlock[], prefix: readonly number[] = []): void => { + for (const [blockIndex, block] of blocks.entries()) { + const path = [...prefix, blockIndex] + if (block.type === 'image') { + images.push({ + location: `${messageIndex}:${path.join('.')}`, + base64Bytes: base64Length(block.attachment.bytes), + }) + } else if (block.type === 'tool-result') { + collect(messageIndex, block.content, path) + } + } + } + for (const [messageIndex, message] of messages.entries()) collect(messageIndex, message.content) + let total = images.reduce((sum, image) => sum + image.base64Bytes, 0) + for (const image of images) { + if (total <= maxRequestImageBytes) break + offloaded.add(image.location) + total -= image.base64Bytes + } + return offloaded +} + +interface LocatedContentBlock { + readonly block: ContentBlock + readonly path: readonly number[] +} + +/** Attach stable nested indexes to blocks from one message. */ +function locatedBlocks(blocks: readonly ContentBlock[], prefix: readonly number[] = []): LocatedContentBlock[] { + return blocks.map((block, index) => ({ block, path: [...prefix, index] })) +} + async function userContent( - blocks: readonly ContentBlock[], + blocks: readonly LocatedContentBlock[], attachments: AttachmentStore, + offloaded: ReadonlySet, + messageIndex: number, ): Promise { const content: (TextContent | ImageContent)[] = [] - for (const block of blocks) { + for (const { block, path } of blocks) { switch (block.type) { case 'text': if (block.text.length > 0) content.push({ type: 'text', text: block.text }) break case 'image': { + if (offloaded.has(`${messageIndex}:${path.join('.')}`)) { + content.push({ type: 'text', text: OFFLOADED_IMAGE_TEXT }) + break + } const stored = await attachments.readImage(block.attachment) content.push({ type: 'image', @@ -47,7 +112,7 @@ async function userContent( } case 'tool-result': { - const nested = await userContent(block.content, attachments) + const nested = await userContent(locatedBlocks(block.content, path), attachments, offloaded, messageIndex) if (typeof nested === 'string') { if (nested.length > 0) content.push({ type: 'text', text: nested }) } else { @@ -136,36 +201,44 @@ export function toPiContext( ): PiContext /** * Convert harness history to a pi-ai Context while resolving durable images. - * Tool result names are recovered from preceding assistant tool calls. + * Tool result names are recovered from preceding assistant tool calls. When + * the accumulated base64 image payload exceeds `maxRequestImageBytes`, the + * oldest images are replaced by text placeholders until the request fits, so + * an image-heavy session keeps clearing gateway request-size caps. * @param options - the harness request; `options.system` maps to pi-ai's single `systemPrompt` slot. * @param attachments - durable byte resolver for image references. * @param onReplayDegrade - forwarded to {@link toPiAssistant} for each assistant message. + * @param maxRequestImageBytes - request-level bound on base64-encoded image payload; omission leaves every image in place. * @returns the asynchronously resolved pi-ai context. */ export function toPiContext( options: GenerateOptions, attachments: AttachmentStore, onReplayDegrade?: (reason: string) => void, + maxRequestImageBytes?: number, ): Promise export function toPiContext( options: GenerateOptions, attachments?: AttachmentStore, onReplayDegrade?: (reason: string) => void, + maxRequestImageBytes?: number, ): PiContext | Promise { return attachments === undefined ? textOnlyContext(options, onReplayDegrade) - : toPiContextWithImages(options, attachments, onReplayDegrade) + : toPiContextWithImages(options, attachments, onReplayDegrade, maxRequestImageBytes) } async function toPiContextWithImages( options: GenerateOptions, attachments: AttachmentStore, onReplayDegrade?: (reason: string) => void, + maxRequestImageBytes?: number, ): Promise { + const offloaded = offloadedImages(options.messages, maxRequestImageBytes) const toolNames = new Map() const messages: PiMessage[] = [] - for (const message of options.messages) { + for (const [messageIndex, message] of options.messages.entries()) { if (message.role === 'system') { if (contentHasImage(message.content)) { throw new LlmError('pi-ai cannot represent an image in an in-history system message', 'UNSUPPORTED_CONTENT') @@ -185,14 +258,17 @@ async function toPiContextWithImages( continue } // user role: text + tool results (each result becomes its own message). - const regular = message.content.filter(block => block.type !== 'tool-result') - const content = await userContent(regular, attachments) - const results = message.content.filter(block => block.type === 'tool-result') + const located = locatedBlocks(message.content) + const regular = located.filter(({ block }) => block.type !== 'tool-result') + const content = await userContent(regular, attachments, offloaded, messageIndex) + const results = located.filter((entry): entry is LocatedContentBlock & { block: Extract } => ( + entry.block.type === 'tool-result' + )) if (content.length > 0 || results.length === 0) { messages.push({ role: 'user', content, timestamp: 0 }) } - for (const result of results) { - const resultContent = await userContent(result.content, attachments) + for (const { block: result, path } of results) { + const resultContent = await userContent(locatedBlocks(result.content, path), attachments, offloaded, messageIndex) messages.push({ role: 'toolResult', toolCallId: result.toolCallId, diff --git a/packages/llm/llm-pi-ai/src/stream.ts b/packages/llm/llm-pi-ai/src/stream.ts index 22aa7cb579..d45df45a39 100644 --- a/packages/llm/llm-pi-ai/src/stream.ts +++ b/packages/llm/llm-pi-ai/src/stream.ts @@ -40,6 +40,9 @@ function classifyPiAiError(message: string): string { if (/\b(?:401|403)\b/.test(message)) return 'AUTH' if (isQuotaExceededError(message)) return QUOTA_EXCEEDED_CODE if (/\b429\b|rate.?limit/i.test(message)) return 'RATE_LIMIT' + // A rejected request body (gateway or provider size cap): resending the + // same request cannot succeed, so it is invalid, not transient. + if (/\b413\b|failed to buffer the request body:\s*length limit exceeded|payload too large|request body too large/i.test(message)) return 'INVALID_REQUEST' if (/\b400\b|invalid.?request/i.test(message)) return 'INVALID_REQUEST' if (/\b5\d\d\b/.test(message)) return 'SERVER' if (/\btime(?:d)?\s*out\b|timeout/i.test(message)) return 'TIMEOUT' diff --git a/packages/llm/llm-pi-ai/tests/adapter.spec.ts b/packages/llm/llm-pi-ai/tests/adapter.spec.ts index 82537fd2b3..d45a2e3089 100644 --- a/packages/llm/llm-pi-ai/tests/adapter.spec.ts +++ b/packages/llm/llm-pi-ai/tests/adapter.spec.ts @@ -12,7 +12,7 @@ import * as LlmPiAi from '@deepseek-ai/dsh-llm-pi-ai' import { PiAiAdapter } from '@deepseek-ai/dsh-llm-pi-ai' import { MAX_TIMER_DELAY_MS } from '@deepseek-ai/dsh-timeout' import { getBuiltinModels } from '@earendil-works/pi-ai/providers/all' -import { resolveProfiles } from '../src/config.ts' +import { DEFAULT_MAX_REQUEST_IMAGE_BYTES, resolveProfiles } from '../src/config.ts' import { assemble } from './assemble.ts' import { closeMockServers, mockServer, textEvents } from './mock-server.ts' @@ -697,6 +697,7 @@ describe('provider profile lifecycle', () => { }) it('validates empty, underspecified, legacy-shaped, and explicitly blank profiles', () => { + expect(DEFAULT_MAX_REQUEST_IMAGE_BYTES).toBe(20 * 1024 * 1024) // Empty and omitted dicts are the dormant zero-route posture, not errors. expect(resolveProfiles({}).size).toBe(0) expect(resolveProfiles(undefined).size).toBe(0) @@ -710,6 +711,11 @@ describe('provider profile lifecycle', () => { expect(() => resolveProfiles({ openai: { provider: 'openai' } as never })).toThrow(/moved to the providers dict key/) expect(() => resolveProfiles({ openai: { baseURL: '' } })).toThrow(/empty baseURL/) expect(() => resolveProfiles({ openai: { apiKeyEnv: 'not-a-var!' } })).toThrow(/must match/) + expect(() => resolveProfiles({ openai: { maxRequestImageBytes: 0 } })).toThrow(/maxRequestImageBytes/) + expect(resolveProfiles({ openai: {} }).get('openai')?.maxRequestImageBytes) + .toBe(DEFAULT_MAX_REQUEST_IMAGE_BYTES) + expect(resolveProfiles({ openai: { maxRequestImageBytes: 1024 } }).get('openai')?.maxRequestImageBytes) + .toBe(1024) }) it.each(['maxRetries', 'maxRetryDelayMs'] as const)( @@ -731,6 +737,9 @@ describe('provider profile lifecycle', () => { { streamIdleTimeoutMs: 0 }, { streamIdleTimeoutMs: Number.NaN }, { streamIdleTimeoutMs: MAX_TIMER_DELAY_MS + 1 }, + { maxRequestImageBytes: 0 }, + { maxRequestImageBytes: 1.5 }, + { maxRequestImageBytes: Number.NaN }, ] for (const entry of invalid) { const ctx = new Context() diff --git a/packages/llm/llm-pi-ai/tests/context.spec.ts b/packages/llm/llm-pi-ai/tests/context.spec.ts index 01ed3d080c..be41a12b07 100644 --- a/packages/llm/llm-pi-ai/tests/context.spec.ts +++ b/packages/llm/llm-pi-ai/tests/context.spec.ts @@ -3,7 +3,7 @@ import { AttachmentId } from '@deepseek-ai/dsh-attachment' import type { AttachmentStore, ImageAttachmentRef } from '@deepseek-ai/dsh-attachment' import { CallId, createMessage, createUserMessage } from '@deepseek-ai/dsh-llm' import type { ContentBlock, GenerateOptions, Message } from '@deepseek-ai/dsh-llm' -import { toPiContext } from '../src/context.ts' +import { OFFLOADED_IMAGE_TEXT, toPiContext } from '../src/context.ts' import { toPiAssistant } from '../src/replay.ts' const ref: ImageAttachmentRef = { @@ -140,6 +140,92 @@ describe('pi-ai request context conversion', () => { ]) }) + it('replaces the oldest images with placeholders once the request payload bound is exceeded', async () => { + const readImage = vi.fn(() => Promise.resolve({ ref: { ...ref, bytes: 3 }, data: Uint8Array.of(1, 2, 3) })) + const store = { readImage } as unknown as AttachmentStore + const sized: ImageAttachmentRef = { ...ref, bytes: 3 } + const callId = CallId('shot-call') + // Three 3-byte images cost 4 base64 characters each (12 total); a bound of + // 8 forces exactly the oldest one out, including one nested in a tool result. + const context = await toPiContext(request([ + user([{ + type: 'tool-result', + toolCallId: callId, + content: [{ type: 'image', attachment: sized }], + }]), + user([{ type: 'image', attachment: sized }, { type: 'text', text: 'newer' }]), + user([{ type: 'image', attachment: sized }]), + ]), store, undefined, 8) + + expect(context.messages).toEqual([ + { + role: 'toolResult', + toolCallId: 'shot-call', + toolName: 'unknown', + content: [{ type: 'text', text: OFFLOADED_IMAGE_TEXT }], + isError: false, + timestamp: 0, + }, + { + role: 'user', + content: [ + { type: 'image', data: 'AQID', mimeType: 'image/png' }, + { type: 'text', text: 'newer' }, + ], + timestamp: 0, + }, + { role: 'user', content: [{ type: 'image', data: 'AQID', mimeType: 'image/png' }], timestamp: 0 }, + ]) + expect(readImage).toHaveBeenCalledTimes(2) + }) + + it('keeps every image at exactly the payload bound and drops all of them when even the newest cannot fit', async () => { + const sized: ImageAttachmentRef = { ...ref, bytes: 3 } + const exact = await toPiContext(request([ + user([{ type: 'image', attachment: sized }]), + user([{ type: 'image', attachment: sized }]), + ]), attachments, undefined, 8) + expect(exact.messages).toEqual([ + { role: 'user', content: [expect.objectContaining({ type: 'image' })], timestamp: 0 }, + { role: 'user', content: [expect.objectContaining({ type: 'image' })], timestamp: 0 }, + ]) + + const readImage = vi.fn() + const store = { readImage } as unknown as AttachmentStore + const oversized = await toPiContext(request([ + user([{ type: 'image', attachment: { ...ref, bytes: 300 } }]), + ]), store, undefined, 8) + // All-text content collapses to the string form; the placeholder still reaches the model. + expect(oversized.messages).toEqual([ + { role: 'user', content: OFFLOADED_IMAGE_TEXT, timestamp: 0 }, + ]) + expect(readImage).not.toHaveBeenCalled() + }) + + it('offloads repeated image-block occurrences by position rather than shared object identity', async () => { + const sized: ImageAttachmentRef = { ...ref, bytes: 3 } + const shared: ContentBlock = { type: 'image', attachment: sized } + const readImage = vi.fn(() => Promise.resolve({ ref: sized, data: Uint8Array.of(1, 2, 3) })) + const store = { readImage } as unknown as AttachmentStore + const aliased = await toPiContext(request([user([shared, shared])]), store, undefined, 4) + const replayed = await toPiContext(request([user([ + { type: 'image', attachment: { ...sized } }, + { type: 'image', attachment: { ...sized } }, + ])]), store, undefined, 4) + + const expected = [{ + role: 'user', + content: [ + { type: 'text', text: OFFLOADED_IMAGE_TEXT }, + { type: 'image', data: 'AQID', mimeType: 'image/png' }, + ], + timestamp: 0, + }] + expect(aliased.messages).toEqual(expected) + expect(replayed.messages).toEqual(expected) + expect(readImage).toHaveBeenCalledTimes(2) + }) + it('keeps empty text-only users while separating result-only messages', () => { const callId = CallId('unknown-call') expect(toPiContext(request([ diff --git a/packages/llm/llm-pi-ai/tests/convert.spec.ts b/packages/llm/llm-pi-ai/tests/convert.spec.ts index 1a42e4b085..2a3b41b0c4 100644 --- a/packages/llm/llm-pi-ai/tests/convert.spec.ts +++ b/packages/llm/llm-pi-ai/tests/convert.spec.ts @@ -779,6 +779,16 @@ describe('mapStopReason / mapUsage', () => { stopReason: 'error', errorMessage: 'HTTP 400: invalid input: temperature exceeds maximum allowed value', }))).toMatchObject({ kind: 'error', failure: { code: 'INVALID_REQUEST' } }) + expect(mapStopReason(assistant({ stopReason: 'error', errorMessage: 'HTTP 413: Payload Too Large' }))) + .toMatchObject({ kind: 'error', failure: { code: 'INVALID_REQUEST' } }) + expect(mapStopReason(assistant({ + stopReason: 'error', + errorMessage: 'Failed to buffer the request body: length limit exceeded', + }))).toMatchObject({ kind: 'error', failure: { code: 'INVALID_REQUEST' } }) + expect(mapStopReason(assistant({ + stopReason: 'error', + errorMessage: 'vector length limit exceeded', + }))).toMatchObject({ kind: 'error', failure: { code: 'PI_AI_ERROR' } }) }) it.each([