diff --git a/.agents/notes/implemented/bug-fix/2026-10-26-malformed-charset-declaration.i18n.yaml b/.agents/notes/implemented/bug-fix/2026-10-26-malformed-charset-declaration.i18n.yaml new file mode 100644 index 0000000000..22e56fbdd0 --- /dev/null +++ b/.agents/notes/implemented/bug-fix/2026-10-26-malformed-charset-declaration.i18n.yaml @@ -0,0 +1,19 @@ +# Bilingual-pair consistency record for 2026-10-26-malformed-charset-declaration.md (docs/i18n/README.md): per heading +# section, a hash of its English and Chinese blocks outside code blocks and generated regions. +# After editing either side, bring the other along and re-record with: +# pnpm run verify-translation-pairing --write .agents/notes/implemented/bug-fix/2026-10-26-malformed-charset-declaration.md +/agent-note-malformed-charset-recovery-prefers-utf-8: + en: d8a759b84c36b308 + zh: 3b6ff2eb5519afb1 +/agent-note-malformed-charset-recovery-prefers-utf-8/problem: + en: dce96445b3801a7f + zh: e026d26fa680556a +/agent-note-malformed-charset-recovery-prefers-utf-8/decision: + en: ba74ad57757231ca + zh: 46999cd0100814ae +/agent-note-malformed-charset-recovery-prefers-utf-8/alternatives-considered: + en: 9fafffb6a7a78a29 + zh: ee1dcc27d0315e68 +/agent-note-malformed-charset-recovery-prefers-utf-8/consequences: + en: c8aad3cc3f02e81e + zh: 29d9313afd1c7b5b diff --git a/.agents/notes/implemented/bug-fix/2026-10-26-malformed-charset-declaration.md b/.agents/notes/implemented/bug-fix/2026-10-26-malformed-charset-declaration.md new file mode 100644 index 0000000000..9e36964e09 --- /dev/null +++ b/.agents/notes/implemented/bug-fix/2026-10-26-malformed-charset-declaration.md @@ -0,0 +1,33 @@ +# Agent Note: Malformed charset recovery prefers UTF-8 + +Status: implemented + +English | [中文](2026-10-26-malformed-charset-declaration.zh.md) + +## Problem + +A response can declare `charset=utf-8,gbk` over readable UTF-8 HTML. Passing that entire parameter to `TextDecoder` rejects the page before reading it. Compatibility needs an explicit encoding preference because successful decoding alone cannot identify the server's intended encoding. + +## Decision + +`decoderForCharset` preserves valid single-label declarations and the absent-header UTF-8 default. If a declaration is unsupported as a whole, it checks its comma- and whitespace-separated labels for UTF-8, including standard aliases. A match selects a fatal UTF-8 decoder; otherwise the declaration fails before the response body is read and the provider cancels the stream. + +The provider decodes once. Recovery rejects invalid UTF-8 with `WEB_UNSUPPORTED_CONTENT_TYPE` and preserves the decoding error as its cause. Only an actual byte-cap truncation enables streaming decode, which omits an incomplete suffix and keeps `truncated: true`. Complete responses, including those exactly at the cap, validate the final sequence at EOF. + +UTF-8 recovery is an explicit heuristic, not proof of the original encoding. The bytes `C2 A9` represent `©` in UTF-8 and `漏` in GBK; repeating them preserves that ambiguity at any length. The provider accepts that ambiguity for UTF-8 recovery but does not extend it to arbitrary legacy-encoding guesses. + +## Alternatives considered + +**Choose the first declared encoding that decodes.** A GBK body can also decode as `windows-1252` with different text. Declaration order does not resolve the conflict, and single-byte encodings can turn a decoding failure into silent corruption. Rejected. + +**Use strict probes followed by ordinary decoding.** This decodes successful bodies twice. An unflushed streaming probe also accepts an incomplete EOF sequence that the ordinary decoder replaces. One fatal decode with explicit truncation state is sufficient. + +**Read BOM and HTML meta declarations.** This would add HTML encoding identification and a dependency or parser, with a separate policy for missing metadata and non-HTML content. It remains deferred; the bounded UTF-8 preference resolves the reported response without claiming browser-equivalent detection. + +## Consequences + +- Malformed declarations containing UTF-8 can return readable text; a failed UTF-8 recovery never falls back to another encoding or inserts replacement characters. +- Malformed declarations without a UTF-8 label remain unsupported, including repeated legacy aliases. Valid single-label legacy decoding is unchanged. +- [`fetch-http.spec.ts`](../../../../packages/web/web-fetch-http/tests/fetch-http.spec.ts) covers recovery, conflicting labels, invalid bytes, EOF, truncation, and response-stream cancellation. +- The recorded [`web-fetch`](../../../../snapshots/session/web-fetch/snapshot.yml) scenario serves literal UTF-8 characters with `charset=gbk,utf-8` through the real HTTP provider and verifies the same model-visible text. +- HTML `` remains ignored. The package README documents the remaining ambiguity. diff --git a/.agents/notes/implemented/bug-fix/2026-10-26-malformed-charset-declaration.zh.md b/.agents/notes/implemented/bug-fix/2026-10-26-malformed-charset-declaration.zh.md new file mode 100644 index 0000000000..8e9d1f31ed --- /dev/null +++ b/.agents/notes/implemented/bug-fix/2026-10-26-malformed-charset-declaration.zh.md @@ -0,0 +1,33 @@ +# Agent Note: 畸形 charset 声明的恢复优先使用 UTF-8 + +Status: implemented + +[English](2026-10-26-malformed-charset-declaration.md) | 中文 + +## Problem + +响应可能在可读的 UTF-8 HTML 上声明 `charset=utf-8,gbk`。把整个参数交给 `TextDecoder` 会在读取前拒绝页面。兼容处理需要明确的编码优先规则,因为解码成功本身无法识别服务器原本使用的编码。 + +## Decision + +`decoderForCharset` 保留合法的单标签声明,以及未声明时默认使用 UTF-8 的行为。如果整个声明不受支持,它检查按逗号和空白分隔的标签是否包含 UTF-8(含标准别名)。匹配时选择 fatal UTF-8 解码器;否则在读取正文前拒绝声明,由提供方取消响应流。 + +提供方只解码一次。恢复路径以 `WEB_UNSUPPORTED_CONTENT_TYPE` 拒绝非法 UTF-8,并保留解码错误作为 cause。只有真正触发字节上限截断时才启用流式解码,省略不完整的末尾序列并保持 `truncated: true`。完整响应(包括恰好达到上限的响应)在 EOF 校验末尾序列。 + +UTF-8 恢复是明确的启发式规则,不是原始编码的证明。字节 `C2 A9` 在 UTF-8 下表示 `©`,在 GBK 下表示 `漏`;重复这些字节后,任意长度都保留该歧义。提供方为 UTF-8 恢复接受这种歧义,但不将其扩展为任意传统编码的猜测。 + +## Alternatives considered + +**选择声明中第一个能解码的编码。** GBK 正文也可能按 `windows-1252` 解出不同文本。声明顺序不能解决冲突,单字节编码还可能把解码失败变成静默的内容损坏。拒绝。 + +**先严格探测,再常规解码。** 这会对成功的正文解码两次。未结束的流式探测还会接受 EOF 处不完整的序列,随后被常规解码器替换。一次 fatal 解码配合显式截断状态已经足够。 + +**读取 BOM 和 HTML meta 声明。** 这会增加 HTML 编码识别能力及依赖或解析器,还需单独规定缺少元数据和非 HTML 内容的策略。继续延期;有范围的 UTF-8 优先规则可以处理所报告的响应,无需宣称与浏览器等价的编码检测。 + +## Consequences + +- 含有 UTF-8 的畸形声明可以返回可读文本;UTF-8 恢复失败时不会回退到其他编码,也不会插入替换字符。 +- 不含 UTF-8 标签的畸形声明仍不受支持,包括重复的传统编码别名。合法的单标签传统编码解码保持不变。 +- [`fetch-http.spec.ts`](../../../../packages/web/web-fetch-http/tests/fetch-http.spec.ts) 覆盖恢复、冲突标签、非法字节、EOF、截断和响应流取消。 +- 已录制的 [`web-fetch`](../../../../snapshots/session/web-fetch/snapshot.yml) 场景通过真实 HTTP 提供方,以 `charset=gbk,utf-8` 提供包含原样 UTF-8 字符的页面,并验证相同的模型可见文本。 +- HTML `` 仍然被忽略。包 README 记录了剩余歧义。 diff --git a/packages/web/web-fetch-http/README.i18n.yaml b/packages/web/web-fetch-http/README.i18n.yaml index fcea215f61..e38ab6fa9e 100644 --- a/packages/web/web-fetch-http/README.i18n.yaml +++ b/packages/web/web-fetch-http/README.i18n.yaml @@ -27,11 +27,11 @@ en: 2d18cdcc9b53dd0f zh: 3bc6ca7c5933ee5d /deepseek-ai-dsh-web-fetch-http/use-this-package/transport-behavior: - en: 584054eb465ae615 - zh: c421860ba79bfffc + en: 4c5618124337446e + zh: 45c7378c8b0bba45 /deepseek-ai-dsh-web-fetch-http/use-this-package/failures-and-recovery: - en: 35407f106dc65deb - zh: 02118072fba68b8f + en: 34095d9d4231d73a + zh: be0ed903adbb124f /deepseek-ai-dsh-web-fetch-http/understand-the-implementation: en: 9842c69ad3636bb3 zh: 3cdc3f816cbdd0cb @@ -42,8 +42,8 @@ en: 1f355ec2726ff61b zh: 076a0c259c94f22a /deepseek-ai-dsh-web-fetch-http/understand-the-implementation/read-path: - en: f13cd16555fe39a5 - zh: 8f48cb659f2c272c + en: 4836d06bc065477d + zh: 5a7ddc805a7c67d8 /deepseek-ai-dsh-web-fetch-http/further-exploration: en: eeaaf0b455673a1c zh: 0874f9f44ff9a220 @@ -54,8 +54,8 @@ en: 40294c1c31b74824 zh: d75d48f485447a74 /deepseek-ai-dsh-web-fetch-http/known-limitations-and-deferred-work: - en: aa351dc3edaa6355 - zh: 5bb556857b389df7 + en: 74d63a0c0e7649e1 + zh: 73a59fcca72daba5 /deepseek-ai-dsh-web-fetch-http/known-limitations-and-deferred-work/dev-note: en: e67f633dc6fe7773 zh: 3bb0c20c93f16f98 diff --git a/packages/web/web-fetch-http/README.md b/packages/web/web-fetch-http/README.md index ac56125e1a..4aecd7b649 100644 --- a/packages/web/web-fetch-http/README.md +++ b/packages/web/web-fetch-http/README.md @@ -63,9 +63,11 @@ const page = await ctx.web.fetch({ url: 'https://example.com' }) The provider keeps requests anonymous and bounded: it accepts only `http:` and `https:` URLs without embedded credentials and rejects URLs over 2,048 characters. Requests go through the process-wide dispatcher, so the installed proxy policy applies and the system resolver chooses the address; any destination address is allowed. Same-origin redirects are followed; cross-origin redirects fail and require a fresh call. The provider also enforces byte, character, hop, and time caps, rejects unsupported content types, and sends an explicit product `User-Agent`. +A malformed comma- or whitespace-joined charset declaration containing a UTF-8 label (such as `charset=utf-8,gbk`) is decoded strictly as UTF-8. Recovery rejects invalid UTF-8 without trying another encoding. When the byte cap cuts a character, recovery omits the incomplete suffix and returns `truncated: true`; a complete response using recovery must pass the EOF check. Valid single-label declarations retain ordinary replacement decoding. + ### Failures and recovery -Failures throw `WebError` with a machine-routable code: `WEB_INVALID_URL`, `WEB_BLOCKED_URL`, `WEB_FETCH_TOO_LARGE`, `WEB_FETCH_TIMEOUT`, `WEB_REDIRECT_BLOCKED`, `WEB_UNSUPPORTED_CONTENT_TYPE`, `WEB_ABORTED`, or `WEB_PROVIDER_ERROR`. Direct callers can route on the code; the model-facing `web_fetch` tool surfaces the failure text to the model under its own error wrapper. +Failures throw `WebError` with a machine-routable code: `WEB_INVALID_URL`, `WEB_BLOCKED_URL`, `WEB_FETCH_TOO_LARGE`, `WEB_FETCH_TIMEOUT`, `WEB_REDIRECT_BLOCKED`, `WEB_UNSUPPORTED_CONTENT_TYPE`, `WEB_ABORTED`, or `WEB_PROVIDER_ERROR`. `WEB_UNSUPPORTED_CONTENT_TYPE` covers unsupported media types, unsupported charset declarations, and invalid UTF-8 during malformed-declaration recovery. Direct callers can route on the code; the model-facing `web_fetch` tool surfaces the failure text to the model under its own error wrapper. ----- @@ -94,7 +96,7 @@ The package is built on one separation and one layered timeout: ### Read path -A fetch validates the URL and sends the request. It repeats URL validation for each same-origin redirect; a cross-origin redirect fails before response bytes are accepted. The final response is classified by `Content-Type`, decoded from its declared charset, and read under the byte cap; the decoded text is then truncated to the character cap. +A fetch validates the URL and sends the request. It repeats URL validation for each same-origin redirect; a cross-origin redirect fails before response bytes are accepted. The final response is classified by `Content-Type`, decoded from its declared charset (malformed declarations containing UTF-8 use strict UTF-8 recovery), and read under the byte cap; the decoded text is then truncated to the character cap. @@ -132,7 +134,8 @@ These limits define when the provider is unsafe or a poor fit. They are current - **Private destinations are reachable** — loopback, private-network, and cloud metadata addresses are fetched like any other host. A deployment that must keep the model away from them needs a `tools/pre-execute` policy, network confinement, or no fetch tool. - **Only textual content decodes** — html/xhtml and `text/*` plus JSON/XML families; a missing `Content-Type` or any binary type throws `WEB_UNSUPPORTED_CONTENT_TYPE`, and text-extractable PDF decoding is named deferred work. -- **Charset comes only from the `Content-Type` header** (UTF-8 default) — an HTML `` declaration is ignored, and a declared-but-unrecognized charset label throws rather than falling back. +- **Charset comes only from the `Content-Type` header** (UTF-8 default) — an HTML `` declaration is ignored, and unsupported declarations without a UTF-8 label throw. +- **UTF-8 recovery is a preference, not encoding detection** — bytes can be valid as both UTF-8 and another encoding, even in a long response. Recovery chooses UTF-8 in that case; malformed declarations without a UTF-8 label remain unsupported. ### Dev Note diff --git a/packages/web/web-fetch-http/README.zh.md b/packages/web/web-fetch-http/README.zh.md index 662d3e132c..ba4092e23a 100644 --- a/packages/web/web-fetch-http/README.zh.md +++ b/packages/web/web-fetch-http/README.zh.md @@ -63,9 +63,11 @@ const page = await ctx.web.fetch({ url: 'https://example.com' }) 提供方保持请求匿名且有界:只接受不含内嵌凭据且不超过 2,048 个字符的 `http:` 与 `https:` URL。请求经过进程级 dispatcher,因此已安装的代理策略生效,目标地址由系统解析器选择;任何目标地址都被允许。同源重定向会被跟随;跨源重定向会失败并要求重新调用。提供方还强制执行字节、字符、跳数和时间上限,拒绝不支持的内容类型,并发送显式产品 `User-Agent`。 +对于以逗号或空白拼接、且含有 UTF-8 标签的畸形 charset 声明(例如 `charset=utf-8,gbk`),提供方严格按 UTF-8 解码。恢复路径拒绝非法 UTF-8,不会尝试其他编码。字节上限在字符内部截断时,恢复路径省略不完整的末尾序列并返回 `truncated: true`;使用恢复路径的完整响应必须通过 EOF 检查。合法的单标签声明仍在遇到非法字节时使用替换字符。 + ### 失败与恢复 -失败会抛出 `WebError`,其中包含可供程序路由的错误码:`WEB_INVALID_URL`、`WEB_BLOCKED_URL`、`WEB_FETCH_TOO_LARGE`、`WEB_FETCH_TIMEOUT`、`WEB_REDIRECT_BLOCKED`、`WEB_UNSUPPORTED_CONTENT_TYPE`、`WEB_ABORTED` 或 `WEB_PROVIDER_ERROR`。直接调用方可以按错误码路由;面向模型的 `web_fetch` 工具会在自己的错误包装层内把失败文本呈现给模型。 +失败会抛出 `WebError`,其中包含可供程序路由的错误码:`WEB_INVALID_URL`、`WEB_BLOCKED_URL`、`WEB_FETCH_TOO_LARGE`、`WEB_FETCH_TIMEOUT`、`WEB_REDIRECT_BLOCKED`、`WEB_UNSUPPORTED_CONTENT_TYPE`、`WEB_ABORTED` 或 `WEB_PROVIDER_ERROR`。`WEB_UNSUPPORTED_CONTENT_TYPE` 涵盖不支持的媒体类型、不支持的 charset 声明,以及畸形声明恢复过程中遇到的非法 UTF-8。直接调用方可以按错误码路由;面向模型的 `web_fetch` 工具会在自己的错误包装层内把失败文本呈现给模型。 ----- @@ -94,7 +96,7 @@ const page = await ctx.web.fetch({ url: 'https://example.com' }) ### 读取路径 -抓取先校验 URL,再发送请求。每次同源重定向都重复 URL 校验;跨源重定向在接收响应字节前失败。最终响应按 `Content-Type` 分类、依声明的 charset 解码,并在字节上限内读取;解码后的文本再截断到字符上限。 +抓取先校验 URL,再发送请求。每次同源重定向都重复 URL 校验;跨源重定向在接收响应字节前失败。最终响应按 `Content-Type` 分类、依声明的 charset 解码(含有 UTF-8 的畸形声明使用严格 UTF-8 恢复),并在字节上限内读取;解码后的文本再截断到字符上限。 @@ -132,7 +134,8 @@ const page = await ctx.web.fetch({ url: 'https://example.com' }) - **可以访问私有目标**——loopback、内网与云元数据地址与其他主机一样会被抓取。需要阻止模型访问这些地址的部署必须添加 `tools/pre-execute` 策略、网络隔离,或不挂载抓取工具。 - **只解码文本内容**——包括 html/xhtml 与 `text/*` 加 JSON/XML 家族;缺少 `Content-Type` 或任何二进制类型都会抛出 `WEB_UNSUPPORTED_CONTENT_TYPE`,可提取文本的 PDF 解码属于明确的延期工作。 -- **charset 只来自 `Content-Type` 标头**(默认 UTF-8)——HTML `` 声明会被忽略;声明但无法识别的 charset 标签会抛出异常,而非回退。 +- **charset 只来自 `Content-Type` 标头**(默认 UTF-8)——HTML `` 声明会被忽略;不支持且不含 UTF-8 标签的声明会抛出异常。 +- **UTF-8 恢复是一种优先规则,不是编码检测**——同一组字节可能同时是合法的 UTF-8 和其他编码,长响应也不例外。恢复路径在这种情况下选择 UTF-8;不含 UTF-8 标签的畸形声明仍不受支持。 ### 开发备注 diff --git a/packages/web/web-fetch-http/src/policy.ts b/packages/web/web-fetch-http/src/policy.ts index 47b8fe14ab..a106958f1e 100644 --- a/packages/web/web-fetch-http/src/policy.ts +++ b/packages/web/web-fetch-http/src/policy.ts @@ -1,7 +1,8 @@ /** - * URL validation and content-type classification for the local HTTP(S) fetch - * provider — the pure, network-free half. The provider's `fetch()` composes - * these with transport (redirect following, byte caps, decoding). + * URL validation, content-type classification, and charset resolution for the + * local HTTP(S) fetch provider — the pure, network-free half. The provider's + * `fetch()` composes these with transport (redirect following, byte caps, + * decoding). * * @module @deepseek-ai/dsh-web-fetch-http/policy */ @@ -83,14 +84,10 @@ export function classifyContentType(contentType: string | null): FetchableKind | } /** - * Extract the `charset` parameter from a response `Content-Type`, lower-cased, - * or `undefined` when absent. The provider feeds this label to `TextDecoder` - * so a non-UTF-8 response is decoded with its declared encoding rather than - * silently mangled into replacement characters. + * Extract the lower-cased `charset` parameter from a response `Content-Type`. * - * @param contentType - the raw `Content-Type` header, or `null` when the - * response carries none. - * @returns the lower-cased charset label, or `undefined` when none is declared. + * @param contentType - the raw header, or `null` when absent. + * @returns the charset parameter, or `undefined` when none is declared. */ export function parseCharset(contentType: string | null): string | undefined { const match = /;\s*charset\s*=\s*"?([^";]+)"?/i.exec(contentType ?? '') @@ -98,20 +95,31 @@ export function parseCharset(contentType: string | null): string | undefined { } /** - * Build a `TextDecoder` for the declared charset, falling back to UTF-8 when - * none is declared. Throws {@link WebError} `WEB_UNSUPPORTED_CONTENT_TYPE` when - * the label is present but not a charset `TextDecoder` recognizes — better to - * fail loudly than return mojibake. + * Resolve the declared charset, defaulting to UTF-8 when absent. A malformed + * comma- or whitespace-joined declaration containing a UTF-8 label uses a fatal + * UTF-8 decoder; other unsupported declarations throw `WEB_UNSUPPORTED_CONTENT_TYPE`. + * UTF-8 recovery is a preference, not proof of the original encoding. * - * @param charset - the declared charset label (from {@link parseCharset}), or - * `undefined` to default to UTF-8. - * @returns a decoder for the declared (or defaulted) encoding. + * @param charset - the declared charset parameter, or `undefined` to default. + * @returns a decoder; callers must report fatal decoding failures and flush at + * EOF unless they deliberately truncated the bytes. */ export function decoderForCharset(charset: string | undefined): TextDecoder { - if (charset === undefined) return new TextDecoder('utf-8') + const declared = charset ?? 'utf-8' try { - return new TextDecoder(charset) + return new TextDecoder(declared) } catch (error: unknown) { - throw new WebError(`unsupported charset "${charset}"`, 'WEB_UNSUPPORTED_CONTENT_TYPE', { cause: error }) + if (declared.split(/[\s,]+/).some(isUtf8Label)) return new TextDecoder('utf-8', { fatal: true }) + throw new WebError(`unsupported charset "${declared}"`, 'WEB_UNSUPPORTED_CONTENT_TYPE', { cause: error }) + } +} + +/** Whether a label names UTF-8, including its standard aliases. */ +function isUtf8Label(label: string): boolean { + try { + return new TextDecoder(label).encoding === 'utf-8' + } catch (_error: unknown) { + // An unsupported label does not authorize UTF-8 recovery. + return false } } diff --git a/packages/web/web-fetch-http/src/provider.ts b/packages/web/web-fetch-http/src/provider.ts index 3357124e38..1842e30c8e 100644 --- a/packages/web/web-fetch-http/src/provider.ts +++ b/packages/web/web-fetch-http/src/provider.ts @@ -121,9 +121,7 @@ export class HttpFetchProvider implements WebFetchProvider { throw new WebError(`unsupported content type "${contentType ?? 'unknown'}"`, 'WEB_UNSUPPORTED_CONTENT_TYPE') } - // Resolve the decoder BEFORE reading the body so an unsupported charset - // fails without consuming the stream — but cancel the body on that failure - // so the socket does not leak (matching the unsupported-content-type path). + // Reject unsupported declarations before reading, releasing the response stream. let decoder: TextDecoder try { decoder = decoderForCharset(parseCharset(contentType)) @@ -132,7 +130,17 @@ export class HttpFetchProvider implements WebFetchProvider { throw error } const { bytes, truncatedByBytes } = await this.readCapped(response, signal) - const decoded = decoder.decode(bytes) + let decoded: string + try { + // Only a real byte-cap truncation permits an incomplete UTF-8 suffix. + decoded = decoder.decode(bytes, { stream: decoder.fatal && truncatedByBytes }) + } catch (error: unknown) { + throw new WebError( + 'response body is not valid UTF-8 for the malformed charset declaration', + 'WEB_UNSUPPORTED_CONTENT_TYPE', + { cause: error }, + ) + } const truncatedByChars = decoded.length > this.limits.maxBodyChars const content = truncatedByChars ? decoded.slice(0, this.limits.maxBodyChars) : decoded const body: WebFetchBody = kind === 'html' ? { kind: 'html', content } : { kind: 'text', content } diff --git a/packages/web/web-fetch-http/tests/fetch-http.spec.ts b/packages/web/web-fetch-http/tests/fetch-http.spec.ts index ab2b713936..c66876cc5b 100644 --- a/packages/web/web-fetch-http/tests/fetch-http.spec.ts +++ b/packages/web/web-fetch-http/tests/fetch-http.spec.ts @@ -86,7 +86,21 @@ describe('policy helpers', () => { it('builds a decoder for a charset and defaults to UTF-8', () => { expect(decoderForCharset(undefined).encoding).toBe('utf-8') expect(decoderForCharset('iso-8859-1').encoding).toBe('windows-1252') - expect(() => decoderForCharset('not-a-charset')).toThrow(expect.objectContaining({ code: 'WEB_UNSUPPORTED_CONTENT_TYPE' })) + expect(decoderForCharset('gb2312').encoding).toBe('gbk') + expect(decoderForCharset('utf-8').fatal).toBe(false) + expect(() => decoderForCharset('not-a-charset')).toThrow(expect.objectContaining({ + code: 'WEB_UNSUPPORTED_CONTENT_TYPE', message: 'unsupported charset "not-a-charset"', + })) + }) + + it.each(['utf-8,gbk', 'gbk,utf8', 'not-a-charset, utf-8', 'utf-8 utf8'])('requires strict UTF-8 for %s', (charset) => { + const decoder = decoderForCharset(charset) + expect(decoder.encoding).toBe('utf-8') + expect(decoder.fatal).toBe(true) + }) + + it.each(['gb2312,gbk', 'windows-1252,gbk', ','])('rejects a malformed declaration without UTF-8: %s', (charset) => { + expect(() => decoderForCharset(charset)).toThrow(expect.objectContaining({ code: 'WEB_UNSUPPORTED_CONTENT_TYPE' })) }) }) @@ -181,6 +195,107 @@ describe('HttpFetchProvider caps', () => { }) }) +describe('HttpFetchProvider charset recovery', () => { + it.each(['utf-8,gbk', 'gbk,utf8', 'not-a-charset, utf-8'])('decodes UTF-8 HTML declared as %s', async (charset) => { + handler = (_req, res) => { + res.writeHead(200, { 'content-type': `text/html; charset=${charset}` }) + res.end('

你好

') + } + const result = await provider().fetch({ url: base }) + expect(result.body).toEqual({ kind: 'html', content: '

你好

' }) + expect(result.truncated).toBe(false) + }) + + it.each([ + ['gbk,utf-8', [0xE4, 0xB8]], // Valid GBK, but an incomplete UTF-8 sequence at EOF. + ['gbk,utf-8', [0xE4]], + ['gbk,utf-8', [0x81, 0x20]], + ['gbk,utf-8', [0xD6, 0xD0, 0xCE, 0xC4]], + ['not-a-charset,utf-8', [0xD6, 0xD0, 0xCE, 0xC4]], + ['utf8,utf-8', [0x81, 0x20]], + ] as const)('rejects non-UTF-8 bytes for %s: %j', async (charset, bytes) => { + handler = (_req, res) => { + res.writeHead(200, { 'content-type': `text/plain; charset=${charset}` }) + res.end(Buffer.from(bytes)) + } + await expect(provider().fetch({ url: base })).rejects.toThrow(expect.objectContaining({ + code: 'WEB_UNSUPPORTED_CONTENT_TYPE', + message: 'response body is not valid UTF-8 for the malformed charset declaration', + })) + }) + + it.each(['gb2312,gbk', 'windows-1252,gbk'])('rejects a legacy-only compound declaration: %s', async (charset) => { + handler = (_req, res) => { + res.writeHead(200, { 'content-type': `text/plain; charset=${charset}` }) + res.end(Buffer.from([0xD6, 0xD0, 0xCE, 0xC4])) + } + await expect(provider().fetch({ url: base })).rejects.toThrow(expect.objectContaining({ + code: 'WEB_UNSUPPORTED_CONTENT_TYPE', message: `unsupported charset "${charset}"`, + })) + }) + + it('keeps single-label GBK decoding', async () => { + handler = (_req, res) => { + res.writeHead(200, { 'content-type': 'text/plain; charset=gbk' }) + res.end(Buffer.from([0xD6, 0xD0, 0xCE, 0xC4])) + } + expect((await provider().fetch({ url: base })).body.content).toBe('中文') + }) + + it('keeps replacement decoding for a single declared charset', async () => { + handler = (_req, res) => { + res.writeHead(200, { 'content-type': 'text/plain; charset=utf-8' }) + res.end(Buffer.from([0xE4, 0xB8])) + } + expect((await provider().fetch({ url: base })).body.content).toBe('\uFFFD') + }) + + it.each([ + ['中文', 5, true], + ['中文', 3, true], + ['中', 3, false], + ] as const)('returns complete UTF-8 characters for %j with byte cap %i (truncated: %s)', async (body, maxResponseBytes, truncated) => { + handler = (_req, res) => { + res.writeHead(200, { 'content-type': 'text/plain; charset=gbk,utf-8' }) + res.end(body) + } + const result = await provider({ maxResponseBytes }).fetch({ url: base }) + expect(result.body.content).toBe('中') + expect(result.truncated).toBe(truncated) + }) + + it('validates EOF when the response exactly fills the byte cap', async () => { + handler = (_req, res) => { + res.writeHead(200, { 'content-type': 'text/plain; charset=gbk,utf-8' }) + res.end(Buffer.from([0xE4, 0xB8])) + } + await expect(provider({ maxResponseBytes: 2 }).fetch({ url: base })).rejects.toThrow(expect.objectContaining({ + code: 'WEB_UNSUPPORTED_CONTENT_TYPE', + message: 'response body is not valid UTF-8 for the malformed charset declaration', + })) + }) + + it('rejects invalid UTF-8 before the truncated suffix', async () => { + handler = (_req, res) => { + res.writeHead(200, { 'content-type': 'text/plain; charset=gbk,utf-8' }) + res.end(Buffer.from([0x81, 0x20, 0x61])) + } + await expect(provider({ maxResponseBytes: 2 }).fetch({ url: base })).rejects.toThrow(expect.objectContaining({ + code: 'WEB_UNSUPPORTED_CONTENT_TYPE', + message: 'response body is not valid UTF-8 for the malformed charset declaration', + })) + }) + + it('uses UTF-8 even when a long body also forms valid GBK', async () => { + // C2 A9 represents © in UTF-8 and 漏 in GBK; length cannot disambiguate it. + handler = (_req, res) => { + res.writeHead(200, { 'content-type': 'text/plain; charset=gbk,utf-8' }) + res.end('©'.repeat(1000)) + } + expect((await provider().fetch({ url: base })).body.content).toBe('©'.repeat(1000)) + }) +}) + describe('HttpFetchProvider redirects', () => { it('follows a same-origin redirect and reports the final URL', async () => { handler = (req, res) => { @@ -342,7 +457,6 @@ describe('HttpFetchProvider invalid URLs and abort', () => { await expect(provider().fetch({ url: 'http://127.0.0.1:1/' })) .rejects.toThrow(expect.objectContaining({ code: 'WEB_PROVIDER_ERROR' })) }) - }) describe('HttpFetchProvider body cancellation on error paths', () => { @@ -364,8 +478,8 @@ describe('HttpFetchProvider body cancellation on error paths', () => { await closed }) - it('cancels the body when an unsupported charset is rejected', async () => { - const closed = respondWithOpenBody(200, { 'content-type': 'text/plain; charset=not-a-charset' }) + it.each(['not-a-charset', 'gb2312,gbk', 'windows-1252,gbk'])('cancels the body when charset %s is rejected', async (charset) => { + const closed = respondWithOpenBody(200, { 'content-type': `text/plain; charset=${charset}` }) await expect(provider().fetch({ url: base })) .rejects.toThrow(expect.objectContaining({ code: 'WEB_UNSUPPORTED_CONTENT_TYPE' })) await closed diff --git a/snapshots/session/web-fetch/web-fetch-fixture-server.mjs b/snapshots/session/web-fetch/web-fetch-fixture-server.mjs index f580910d03..84fab269dd 100644 --- a/snapshots/session/web-fetch/web-fetch-fixture-server.mjs +++ b/snapshots/session/web-fetch/web-fetch-fixture-server.mjs @@ -1,6 +1,6 @@ /** * Deterministic HTTP provider for the web-fetch snapshot scenario: a small - * HTML page (headings, named entities, a GFM table, nested formatting) on a + * UTF-8 HTML page with a malformed charset declaration on an * OS-assigned loopback port behind the real HTTP transport. Recording * and replay therefore exercise fetch and markdown rendering without external * network while retaining the recorded request URL. @@ -14,10 +14,10 @@ const RECORDED_URL = 'http://public.test:43117/menu.html' const PAGE = ` Menu -

Café menu

+

Café menu

Prices include service & tax — updated daily.

-
DrinkPrice
Espresso€2
Flat white€3
+
DrinkPrice
Espresso€2
Flat white€3

See today’s specials.

` @@ -62,7 +62,7 @@ export async function apply(ctx) { label: 'web-fetch-fixture-server', requestListener: (req, res) => { if (req.url === '/menu.html') { - res.writeHead(200, { 'content-type': 'text/html; charset=utf-8' }) + res.writeHead(200, { 'content-type': 'text/html; charset=gbk,utf-8' }) res.end(PAGE) return }