Merge pull request #7322 from deepseek-harness/fix/web-fetch-charset-tolerance
fix(web): recover malformed charset declarations with strict UTF-8
This commit is contained in:
Dudu-0223 authored and GitHub committed
commit
cf38ed3205
10 files changed
+267
-46
No files matched your search
@@ -0,0 +1,19 @@
|
||||
# Bilingual-pair consistency record for 2026-10-26-malformed-charset-declaration.md (docs/i18n/README.md): per heading
|
||||
# section, a hash of its English and Chinese blocks outside code blocks and generated regions.
|
||||
# After editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write .agents/notes/implemented/bug-fix/2026-10-26-malformed-charset-declaration.md
|
||||
/agent-note-malformed-charset-recovery-prefers-utf-8:
|
||||
en: d8a759b84c36b308
|
||||
zh: 3b6ff2eb5519afb1
|
||||
/agent-note-malformed-charset-recovery-prefers-utf-8/problem:
|
||||
en: dce96445b3801a7f
|
||||
zh: e026d26fa680556a
|
||||
/agent-note-malformed-charset-recovery-prefers-utf-8/decision:
|
||||
en: ba74ad57757231ca
|
||||
zh: 46999cd0100814ae
|
||||
/agent-note-malformed-charset-recovery-prefers-utf-8/alternatives-considered:
|
||||
en: 9fafffb6a7a78a29
|
||||
zh: ee1dcc27d0315e68
|
||||
/agent-note-malformed-charset-recovery-prefers-utf-8/consequences:
|
||||
en: c8aad3cc3f02e81e
|
||||
zh: 29d9313afd1c7b5b
|
||||
@@ -0,0 +1,33 @@
|
||||
# Agent Note: Malformed charset recovery prefers UTF-8
|
||||
|
||||
Status: implemented
|
||||
|
||||
English | [中文](2026-10-26-malformed-charset-declaration.zh.md)
|
||||
|
||||
## Problem
|
||||
|
||||
A response can declare `charset=utf-8,gbk` over readable UTF-8 HTML. Passing that entire parameter to `TextDecoder` rejects the page before reading it. Compatibility needs an explicit encoding preference because successful decoding alone cannot identify the server's intended encoding.
|
||||
|
||||
## Decision
|
||||
|
||||
`decoderForCharset` preserves valid single-label declarations and the absent-header UTF-8 default. If a declaration is unsupported as a whole, it checks its comma- and whitespace-separated labels for UTF-8, including standard aliases. A match selects a fatal UTF-8 decoder; otherwise the declaration fails before the response body is read and the provider cancels the stream.
|
||||
|
||||
The provider decodes once. Recovery rejects invalid UTF-8 with `WEB_UNSUPPORTED_CONTENT_TYPE` and preserves the decoding error as its cause. Only an actual byte-cap truncation enables streaming decode, which omits an incomplete suffix and keeps `truncated: true`. Complete responses, including those exactly at the cap, validate the final sequence at EOF.
|
||||
|
||||
UTF-8 recovery is an explicit heuristic, not proof of the original encoding. The bytes `C2 A9` represent `©` in UTF-8 and `漏` in GBK; repeating them preserves that ambiguity at any length. The provider accepts that ambiguity for UTF-8 recovery but does not extend it to arbitrary legacy-encoding guesses.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
**Choose the first declared encoding that decodes.** A GBK body can also decode as `windows-1252` with different text. Declaration order does not resolve the conflict, and single-byte encodings can turn a decoding failure into silent corruption. Rejected.
|
||||
|
||||
**Use strict probes followed by ordinary decoding.** This decodes successful bodies twice. An unflushed streaming probe also accepts an incomplete EOF sequence that the ordinary decoder replaces. One fatal decode with explicit truncation state is sufficient.
|
||||
|
||||
**Read BOM and HTML meta declarations.** This would add HTML encoding identification and a dependency or parser, with a separate policy for missing metadata and non-HTML content. It remains deferred; the bounded UTF-8 preference resolves the reported response without claiming browser-equivalent detection.
|
||||
|
||||
## Consequences
|
||||
|
||||
- Malformed declarations containing UTF-8 can return readable text; a failed UTF-8 recovery never falls back to another encoding or inserts replacement characters.
|
||||
- Malformed declarations without a UTF-8 label remain unsupported, including repeated legacy aliases. Valid single-label legacy decoding is unchanged.
|
||||
- [`fetch-http.spec.ts`](../../../../packages/web/web-fetch-http/tests/fetch-http.spec.ts) covers recovery, conflicting labels, invalid bytes, EOF, truncation, and response-stream cancellation.
|
||||
- The recorded [`web-fetch`](../../../../snapshots/session/web-fetch/snapshot.yml) scenario serves literal UTF-8 characters with `charset=gbk,utf-8` through the real HTTP provider and verifies the same model-visible text.
|
||||
- HTML `<meta charset>` remains ignored. The package README documents the remaining ambiguity.
|
||||
@@ -0,0 +1,33 @@
|
||||
# Agent Note: 畸形 charset 声明的恢复优先使用 UTF-8
|
||||
|
||||
Status: implemented
|
||||
|
||||
[English](2026-10-26-malformed-charset-declaration.md) | 中文
|
||||
|
||||
## Problem
|
||||
|
||||
响应可能在可读的 UTF-8 HTML 上声明 `charset=utf-8,gbk`。把整个参数交给 `TextDecoder` 会在读取前拒绝页面。兼容处理需要明确的编码优先规则，因为解码成功本身无法识别服务器原本使用的编码。
|
||||
|
||||
## Decision
|
||||
|
||||
`decoderForCharset` 保留合法的单标签声明，以及未声明时默认使用 UTF-8 的行为。如果整个声明不受支持，它检查按逗号和空白分隔的标签是否包含 UTF-8（含标准别名）。匹配时选择 fatal UTF-8 解码器；否则在读取正文前拒绝声明，由提供方取消响应流。
|
||||
|
||||
提供方只解码一次。恢复路径以 `WEB_UNSUPPORTED_CONTENT_TYPE` 拒绝非法 UTF-8，并保留解码错误作为 cause。只有真正触发字节上限截断时才启用流式解码，省略不完整的末尾序列并保持 `truncated: true`。完整响应（包括恰好达到上限的响应）在 EOF 校验末尾序列。
|
||||
|
||||
UTF-8 恢复是明确的启发式规则，不是原始编码的证明。字节 `C2 A9` 在 UTF-8 下表示 `©`，在 GBK 下表示 `漏`；重复这些字节后，任意长度都保留该歧义。提供方为 UTF-8 恢复接受这种歧义，但不将其扩展为任意传统编码的猜测。
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
**选择声明中第一个能解码的编码。** GBK 正文也可能按 `windows-1252` 解出不同文本。声明顺序不能解决冲突，单字节编码还可能把解码失败变成静默的内容损坏。拒绝。
|
||||
|
||||
**先严格探测，再常规解码。** 这会对成功的正文解码两次。未结束的流式探测还会接受 EOF 处不完整的序列，随后被常规解码器替换。一次 fatal 解码配合显式截断状态已经足够。
|
||||
|
||||
**读取 BOM 和 HTML meta 声明。** 这会增加 HTML 编码识别能力及依赖或解析器，还需单独规定缺少元数据和非 HTML 内容的策略。继续延期；有范围的 UTF-8 优先规则可以处理所报告的响应，无需宣称与浏览器等价的编码检测。
|
||||
|
||||
## Consequences
|
||||
|
||||
- 含有 UTF-8 的畸形声明可以返回可读文本；UTF-8 恢复失败时不会回退到其他编码，也不会插入替换字符。
|
||||
- 不含 UTF-8 标签的畸形声明仍不受支持，包括重复的传统编码别名。合法的单标签传统编码解码保持不变。
|
||||
- [`fetch-http.spec.ts`](../../../../packages/web/web-fetch-http/tests/fetch-http.spec.ts) 覆盖恢复、冲突标签、非法字节、EOF、截断和响应流取消。
|
||||
- 已录制的 [`web-fetch`](../../../../snapshots/session/web-fetch/snapshot.yml) 场景通过真实 HTTP 提供方，以 `charset=gbk,utf-8` 提供包含原样 UTF-8 字符的页面，并验证相同的模型可见文本。
|
||||
- HTML `<meta charset>` 仍然被忽略。包 README 记录了剩余歧义。
|
||||
@@ -27,11 +27,11 @@
|
||||
en: 2d18cdcc9b53dd0f
|
||||
zh: 3bc6ca7c5933ee5d
|
||||
/deepseek-ai-dsh-web-fetch-http/use-this-package/transport-behavior:
|
||||
en: 584054eb465ae615
|
||||
zh: c421860ba79bfffc
|
||||
en: 4c5618124337446e
|
||||
zh: 45c7378c8b0bba45
|
||||
/deepseek-ai-dsh-web-fetch-http/use-this-package/failures-and-recovery:
|
||||
en: 35407f106dc65deb
|
||||
zh: 02118072fba68b8f
|
||||
en: 34095d9d4231d73a
|
||||
zh: be0ed903adbb124f
|
||||
/deepseek-ai-dsh-web-fetch-http/understand-the-implementation:
|
||||
en: 9842c69ad3636bb3
|
||||
zh: 3cdc3f816cbdd0cb
|
||||
@@ -42,8 +42,8 @@
|
||||
en: 1f355ec2726ff61b
|
||||
zh: 076a0c259c94f22a
|
||||
/deepseek-ai-dsh-web-fetch-http/understand-the-implementation/read-path:
|
||||
en: f13cd16555fe39a5
|
||||
zh: 8f48cb659f2c272c
|
||||
en: 4836d06bc065477d
|
||||
zh: 5a7ddc805a7c67d8
|
||||
/deepseek-ai-dsh-web-fetch-http/further-exploration:
|
||||
en: eeaaf0b455673a1c
|
||||
zh: 0874f9f44ff9a220
|
||||
@@ -54,8 +54,8 @@
|
||||
en: 40294c1c31b74824
|
||||
zh: d75d48f485447a74
|
||||
/deepseek-ai-dsh-web-fetch-http/known-limitations-and-deferred-work:
|
||||
en: aa351dc3edaa6355
|
||||
zh: 5bb556857b389df7
|
||||
en: 74d63a0c0e7649e1
|
||||
zh: 73a59fcca72daba5
|
||||
/deepseek-ai-dsh-web-fetch-http/known-limitations-and-deferred-work/dev-note:
|
||||
en: e67f633dc6fe7773
|
||||
zh: 3bb0c20c93f16f98
|
||||
@@ -63,9 +63,11 @@ const page = await ctx.web.fetch({ url: 'https://example.com' })
|
||||
|
||||
The provider keeps requests anonymous and bounded: it accepts only `http:` and `https:` URLs without embedded credentials and rejects URLs over 2,048 characters. Requests go through the process-wide dispatcher, so the installed proxy policy applies and the system resolver chooses the address; any destination address is allowed. Same-origin redirects are followed; cross-origin redirects fail and require a fresh call. The provider also enforces byte, character, hop, and time caps, rejects unsupported content types, and sends an explicit product `User-Agent`.
|
||||
|
||||
A malformed comma- or whitespace-joined charset declaration containing a UTF-8 label (such as `charset=utf-8,gbk`) is decoded strictly as UTF-8. Recovery rejects invalid UTF-8 without trying another encoding. When the byte cap cuts a character, recovery omits the incomplete suffix and returns `truncated: true`; a complete response using recovery must pass the EOF check. Valid single-label declarations retain ordinary replacement decoding.
|
||||
|
||||
### Failures and recovery
|
||||
|
||||
Failures throw `WebError` with a machine-routable code: `WEB_INVALID_URL`, `WEB_BLOCKED_URL`, `WEB_FETCH_TOO_LARGE`, `WEB_FETCH_TIMEOUT`, `WEB_REDIRECT_BLOCKED`, `WEB_UNSUPPORTED_CONTENT_TYPE`, `WEB_ABORTED`, or `WEB_PROVIDER_ERROR`. Direct callers can route on the code; the model-facing `web_fetch` tool surfaces the failure text to the model under its own error wrapper.
|
||||
Failures throw `WebError` with a machine-routable code: `WEB_INVALID_URL`, `WEB_BLOCKED_URL`, `WEB_FETCH_TOO_LARGE`, `WEB_FETCH_TIMEOUT`, `WEB_REDIRECT_BLOCKED`, `WEB_UNSUPPORTED_CONTENT_TYPE`, `WEB_ABORTED`, or `WEB_PROVIDER_ERROR`. `WEB_UNSUPPORTED_CONTENT_TYPE` covers unsupported media types, unsupported charset declarations, and invalid UTF-8 during malformed-declaration recovery. Direct callers can route on the code; the model-facing `web_fetch` tool surfaces the failure text to the model under its own error wrapper.
|
||||
|
||||
-----
|
||||
|
||||
@@ -94,7 +96,7 @@ The package is built on one separation and one layered timeout:
|
||||
|
||||
### Read path
|
||||
|
||||
A fetch validates the URL and sends the request. It repeats URL validation for each same-origin redirect; a cross-origin redirect fails before response bytes are accepted. The final response is classified by `Content-Type`, decoded from its declared charset, and read under the byte cap; the decoded text is then truncated to the character cap.
|
||||
A fetch validates the URL and sends the request. It repeats URL validation for each same-origin redirect; a cross-origin redirect fails before response bytes are accepted. The final response is classified by `Content-Type`, decoded from its declared charset (malformed declarations containing UTF-8 use strict UTF-8 recovery), and read under the byte cap; the decoded text is then truncated to the character cap.
|
||||
|
||||
</details>
|
||||
|
||||
@@ -132,7 +134,8 @@ These limits define when the provider is unsafe or a poor fit. They are current
|
||||
|
||||
- **Private destinations are reachable** — loopback, private-network, and cloud metadata addresses are fetched like any other host. A deployment that must keep the model away from them needs a `tools/pre-execute` policy, network confinement, or no fetch tool.
|
||||
- **Only textual content decodes** — html/xhtml and `text/*` plus JSON/XML families; a missing `Content-Type` or any binary type throws `WEB_UNSUPPORTED_CONTENT_TYPE`, and text-extractable PDF decoding is named deferred work.
|
||||
- **Charset comes only from the `Content-Type` header** (UTF-8 default) — an HTML `<meta charset>` declaration is ignored, and a declared-but-unrecognized charset label throws rather than falling back.
|
||||
- **Charset comes only from the `Content-Type` header** (UTF-8 default) — an HTML `<meta charset>` declaration is ignored, and unsupported declarations without a UTF-8 label throw.
|
||||
- **UTF-8 recovery is a preference, not encoding detection** — bytes can be valid as both UTF-8 and another encoding, even in a long response. Recovery chooses UTF-8 in that case; malformed declarations without a UTF-8 label remain unsupported.
|
||||
|
||||
<a id="dev-note"></a>
|
||||
### Dev Note
|
||||
|
||||
@@ -63,9 +63,11 @@ const page = await ctx.web.fetch({ url: 'https://example.com' })
|
||||
|
||||
提供方保持请求匿名且有界：只接受不含内嵌凭据且不超过 2,048 个字符的 `http:` 与 `https:` URL。请求经过进程级 dispatcher，因此已安装的代理策略生效，目标地址由系统解析器选择；任何目标地址都被允许。同源重定向会被跟随；跨源重定向会失败并要求重新调用。提供方还强制执行字节、字符、跳数和时间上限，拒绝不支持的内容类型，并发送显式产品 `User-Agent`。
|
||||
|
||||
对于以逗号或空白拼接、且含有 UTF-8 标签的畸形 charset 声明（例如 `charset=utf-8,gbk`），提供方严格按 UTF-8 解码。恢复路径拒绝非法 UTF-8，不会尝试其他编码。字节上限在字符内部截断时，恢复路径省略不完整的末尾序列并返回 `truncated: true`；使用恢复路径的完整响应必须通过 EOF 检查。合法的单标签声明仍在遇到非法字节时使用替换字符。
|
||||
|
||||
### 失败与恢复
|
||||
|
||||
失败会抛出 `WebError`，其中包含可供程序路由的错误码：`WEB_INVALID_URL`、`WEB_BLOCKED_URL`、`WEB_FETCH_TOO_LARGE`、`WEB_FETCH_TIMEOUT`、`WEB_REDIRECT_BLOCKED`、`WEB_UNSUPPORTED_CONTENT_TYPE`、`WEB_ABORTED` 或 `WEB_PROVIDER_ERROR`。直接调用方可以按错误码路由；面向模型的 `web_fetch` 工具会在自己的错误包装层内把失败文本呈现给模型。
|
||||
失败会抛出 `WebError`，其中包含可供程序路由的错误码：`WEB_INVALID_URL`、`WEB_BLOCKED_URL`、`WEB_FETCH_TOO_LARGE`、`WEB_FETCH_TIMEOUT`、`WEB_REDIRECT_BLOCKED`、`WEB_UNSUPPORTED_CONTENT_TYPE`、`WEB_ABORTED` 或 `WEB_PROVIDER_ERROR`。`WEB_UNSUPPORTED_CONTENT_TYPE` 涵盖不支持的媒体类型、不支持的 charset 声明，以及畸形声明恢复过程中遇到的非法 UTF-8。直接调用方可以按错误码路由；面向模型的 `web_fetch` 工具会在自己的错误包装层内把失败文本呈现给模型。
|
||||
|
||||
-----
|
||||
|
||||
@@ -94,7 +96,7 @@ const page = await ctx.web.fetch({ url: 'https://example.com' })
|
||||
|
||||
### 读取路径
|
||||
|
||||
抓取先校验 URL，再发送请求。每次同源重定向都重复 URL 校验；跨源重定向在接收响应字节前失败。最终响应按 `Content-Type` 分类、依声明的 charset 解码，并在字节上限内读取；解码后的文本再截断到字符上限。
|
||||
抓取先校验 URL，再发送请求。每次同源重定向都重复 URL 校验；跨源重定向在接收响应字节前失败。最终响应按 `Content-Type` 分类、依声明的 charset 解码（含有 UTF-8 的畸形声明使用严格 UTF-8 恢复），并在字节上限内读取；解码后的文本再截断到字符上限。
|
||||
|
||||
</details>
|
||||
|
||||
@@ -132,7 +134,8 @@ const page = await ctx.web.fetch({ url: 'https://example.com' })
|
||||
|
||||
- **可以访问私有目标**——loopback、内网与云元数据地址与其他主机一样会被抓取。需要阻止模型访问这些地址的部署必须添加 `tools/pre-execute` 策略、网络隔离，或不挂载抓取工具。
|
||||
- **只解码文本内容**——包括 html/xhtml 与 `text/*` 加 JSON/XML 家族；缺少 `Content-Type` 或任何二进制类型都会抛出 `WEB_UNSUPPORTED_CONTENT_TYPE`，可提取文本的 PDF 解码属于明确的延期工作。
|
||||
- **charset 只来自 `Content-Type` 标头**（默认 UTF-8）——HTML `<meta charset>` 声明会被忽略；声明但无法识别的 charset 标签会抛出异常，而非回退。
|
||||
- **charset 只来自 `Content-Type` 标头**（默认 UTF-8）——HTML `<meta charset>` 声明会被忽略；不支持且不含 UTF-8 标签的声明会抛出异常。
|
||||
- **UTF-8 恢复是一种优先规则，不是编码检测**——同一组字节可能同时是合法的 UTF-8 和其他编码，长响应也不例外。恢复路径在这种情况下选择 UTF-8；不含 UTF-8 标签的畸形声明仍不受支持。
|
||||
|
||||
<a id="dev-note"></a>
|
||||
### 开发备注
|
||||
|
||||
@@ -1,7 +1,8 @@
|
||||
/**
|
||||
* URL validation and content-type classification for the local HTTP(S) fetch
|
||||
* provider — the pure, network-free half. The provider's `fetch()` composes
|
||||
* these with transport (redirect following, byte caps, decoding).
|
||||
* URL validation, content-type classification, and charset resolution for the
|
||||
* local HTTP(S) fetch provider — the pure, network-free half. The provider's
|
||||
* `fetch()` composes these with transport (redirect following, byte caps,
|
||||
* decoding).
|
||||
*
|
||||
* @module @deepseek-ai/dsh-web-fetch-http/policy
|
||||
*/
|
||||
@@ -83,14 +84,10 @@ export function classifyContentType(contentType: string | null): FetchableKind |
|
||||
}
|
||||
|
||||
/**
|
||||
* Extract the `charset` parameter from a response `Content-Type`, lower-cased,
|
||||
* or `undefined` when absent. The provider feeds this label to `TextDecoder`
|
||||
* so a non-UTF-8 response is decoded with its declared encoding rather than
|
||||
* silently mangled into replacement characters.
|
||||
* Extract the lower-cased `charset` parameter from a response `Content-Type`.
|
||||
*
|
||||
* @param contentType - the raw `Content-Type` header, or `null` when the
|
||||
* response carries none.
|
||||
* @returns the lower-cased charset label, or `undefined` when none is declared.
|
||||
* @param contentType - the raw header, or `null` when absent.
|
||||
* @returns the charset parameter, or `undefined` when none is declared.
|
||||
*/
|
||||
export function parseCharset(contentType: string | null): string | undefined {
|
||||
const match = /;\s*charset\s*=\s*"?([^";]+)"?/i.exec(contentType ?? '')
|
||||
@@ -98,20 +95,31 @@ export function parseCharset(contentType: string | null): string | undefined {
|
||||
}
|
||||
|
||||
/**
|
||||
* Build a `TextDecoder` for the declared charset, falling back to UTF-8 when
|
||||
* none is declared. Throws {@link WebError} `WEB_UNSUPPORTED_CONTENT_TYPE` when
|
||||
* the label is present but not a charset `TextDecoder` recognizes — better to
|
||||
* fail loudly than return mojibake.
|
||||
* Resolve the declared charset, defaulting to UTF-8 when absent. A malformed
|
||||
* comma- or whitespace-joined declaration containing a UTF-8 label uses a fatal
|
||||
* UTF-8 decoder; other unsupported declarations throw `WEB_UNSUPPORTED_CONTENT_TYPE`.
|
||||
* UTF-8 recovery is a preference, not proof of the original encoding.
|
||||
*
|
||||
* @param charset - the declared charset label (from {@link parseCharset}), or
|
||||
* `undefined` to default to UTF-8.
|
||||
* @returns a decoder for the declared (or defaulted) encoding.
|
||||
* @param charset - the declared charset parameter, or `undefined` to default.
|
||||
* @returns a decoder; callers must report fatal decoding failures and flush at
|
||||
* EOF unless they deliberately truncated the bytes.
|
||||
*/
|
||||
export function decoderForCharset(charset: string | undefined): TextDecoder {
|
||||
if (charset === undefined) return new TextDecoder('utf-8')
|
||||
const declared = charset ?? 'utf-8'
|
||||
try {
|
||||
return new TextDecoder(charset)
|
||||
return new TextDecoder(declared)
|
||||
} catch (error: unknown) {
|
||||
throw new WebError(`unsupported charset "${charset}"`, 'WEB_UNSUPPORTED_CONTENT_TYPE', { cause: error })
|
||||
if (declared.split(/[\s,]+/).some(isUtf8Label)) return new TextDecoder('utf-8', { fatal: true })
|
||||
throw new WebError(`unsupported charset "${declared}"`, 'WEB_UNSUPPORTED_CONTENT_TYPE', { cause: error })
|
||||
}
|
||||
}
|
||||
|
||||
/** Whether a label names UTF-8, including its standard aliases. */
|
||||
function isUtf8Label(label: string): boolean {
|
||||
try {
|
||||
return new TextDecoder(label).encoding === 'utf-8'
|
||||
} catch (_error: unknown) {
|
||||
// An unsupported label does not authorize UTF-8 recovery.
|
||||
return false
|
||||
}
|
||||
}
|
||||
@@ -121,9 +121,7 @@ export class HttpFetchProvider implements WebFetchProvider {
|
||||
throw new WebError(`unsupported content type "${contentType ?? 'unknown'}"`, 'WEB_UNSUPPORTED_CONTENT_TYPE')
|
||||
}
|
||||
|
||||
// Resolve the decoder BEFORE reading the body so an unsupported charset
|
||||
// fails without consuming the stream — but cancel the body on that failure
|
||||
// so the socket does not leak (matching the unsupported-content-type path).
|
||||
// Reject unsupported declarations before reading, releasing the response stream.
|
||||
let decoder: TextDecoder
|
||||
try {
|
||||
decoder = decoderForCharset(parseCharset(contentType))
|
||||
@@ -132,7 +130,17 @@ export class HttpFetchProvider implements WebFetchProvider {
|
||||
throw error
|
||||
}
|
||||
const { bytes, truncatedByBytes } = await this.readCapped(response, signal)
|
||||
const decoded = decoder.decode(bytes)
|
||||
let decoded: string
|
||||
try {
|
||||
// Only a real byte-cap truncation permits an incomplete UTF-8 suffix.
|
||||
decoded = decoder.decode(bytes, { stream: decoder.fatal && truncatedByBytes })
|
||||
} catch (error: unknown) {
|
||||
throw new WebError(
|
||||
'response body is not valid UTF-8 for the malformed charset declaration',
|
||||
'WEB_UNSUPPORTED_CONTENT_TYPE',
|
||||
{ cause: error },
|
||||
)
|
||||
}
|
||||
const truncatedByChars = decoded.length > this.limits.maxBodyChars
|
||||
const content = truncatedByChars ? decoded.slice(0, this.limits.maxBodyChars) : decoded
|
||||
const body: WebFetchBody = kind === 'html' ? { kind: 'html', content } : { kind: 'text', content }
|
||||
|
||||
@@ -86,7 +86,21 @@ describe('policy helpers', () => {
|
||||
it('builds a decoder for a charset and defaults to UTF-8', () => {
|
||||
expect(decoderForCharset(undefined).encoding).toBe('utf-8')
|
||||
expect(decoderForCharset('iso-8859-1').encoding).toBe('windows-1252')
|
||||
expect(() => decoderForCharset('not-a-charset')).toThrow(expect.objectContaining({ code: 'WEB_UNSUPPORTED_CONTENT_TYPE' }))
|
||||
expect(decoderForCharset('gb2312').encoding).toBe('gbk')
|
||||
expect(decoderForCharset('utf-8').fatal).toBe(false)
|
||||
expect(() => decoderForCharset('not-a-charset')).toThrow(expect.objectContaining({
|
||||
code: 'WEB_UNSUPPORTED_CONTENT_TYPE', message: 'unsupported charset "not-a-charset"',
|
||||
}))
|
||||
})
|
||||
|
||||
it.each(['utf-8,gbk', 'gbk,utf8', 'not-a-charset, utf-8', 'utf-8 utf8'])('requires strict UTF-8 for %s', (charset) => {
|
||||
const decoder = decoderForCharset(charset)
|
||||
expect(decoder.encoding).toBe('utf-8')
|
||||
expect(decoder.fatal).toBe(true)
|
||||
})
|
||||
|
||||
it.each(['gb2312,gbk', 'windows-1252,gbk', ','])('rejects a malformed declaration without UTF-8: %s', (charset) => {
|
||||
expect(() => decoderForCharset(charset)).toThrow(expect.objectContaining({ code: 'WEB_UNSUPPORTED_CONTENT_TYPE' }))
|
||||
})
|
||||
})
|
||||
|
||||
@@ -181,6 +195,107 @@ describe('HttpFetchProvider caps', () => {
|
||||
})
|
||||
})
|
||||
|
||||
describe('HttpFetchProvider charset recovery', () => {
|
||||
it.each(['utf-8,gbk', 'gbk,utf8', 'not-a-charset, utf-8'])('decodes UTF-8 HTML declared as %s', async (charset) => {
|
||||
handler = (_req, res) => {
|
||||
res.writeHead(200, { 'content-type': `text/html; charset=${charset}` })
|
||||
res.end('<meta charset="UTF-8"><p>你好</p>')
|
||||
}
|
||||
const result = await provider().fetch({ url: base })
|
||||
expect(result.body).toEqual({ kind: 'html', content: '<meta charset="UTF-8"><p>你好</p>' })
|
||||
expect(result.truncated).toBe(false)
|
||||
})
|
||||
|
||||
it.each([
|
||||
['gbk,utf-8', [0xE4, 0xB8]], // Valid GBK, but an incomplete UTF-8 sequence at EOF.
|
||||
['gbk,utf-8', [0xE4]],
|
||||
['gbk,utf-8', [0x81, 0x20]],
|
||||
['gbk,utf-8', [0xD6, 0xD0, 0xCE, 0xC4]],
|
||||
['not-a-charset,utf-8', [0xD6, 0xD0, 0xCE, 0xC4]],
|
||||
['utf8,utf-8', [0x81, 0x20]],
|
||||
] as const)('rejects non-UTF-8 bytes for %s: %j', async (charset, bytes) => {
|
||||
handler = (_req, res) => {
|
||||
res.writeHead(200, { 'content-type': `text/plain; charset=${charset}` })
|
||||
res.end(Buffer.from(bytes))
|
||||
}
|
||||
await expect(provider().fetch({ url: base })).rejects.toThrow(expect.objectContaining({
|
||||
code: 'WEB_UNSUPPORTED_CONTENT_TYPE',
|
||||
message: 'response body is not valid UTF-8 for the malformed charset declaration',
|
||||
}))
|
||||
})
|
||||
|
||||
it.each(['gb2312,gbk', 'windows-1252,gbk'])('rejects a legacy-only compound declaration: %s', async (charset) => {
|
||||
handler = (_req, res) => {
|
||||
res.writeHead(200, { 'content-type': `text/plain; charset=${charset}` })
|
||||
res.end(Buffer.from([0xD6, 0xD0, 0xCE, 0xC4]))
|
||||
}
|
||||
await expect(provider().fetch({ url: base })).rejects.toThrow(expect.objectContaining({
|
||||
code: 'WEB_UNSUPPORTED_CONTENT_TYPE', message: `unsupported charset "${charset}"`,
|
||||
}))
|
||||
})
|
||||
|
||||
it('keeps single-label GBK decoding', async () => {
|
||||
handler = (_req, res) => {
|
||||
res.writeHead(200, { 'content-type': 'text/plain; charset=gbk' })
|
||||
res.end(Buffer.from([0xD6, 0xD0, 0xCE, 0xC4]))
|
||||
}
|
||||
expect((await provider().fetch({ url: base })).body.content).toBe('中文')
|
||||
})
|
||||
|
||||
it('keeps replacement decoding for a single declared charset', async () => {
|
||||
handler = (_req, res) => {
|
||||
res.writeHead(200, { 'content-type': 'text/plain; charset=utf-8' })
|
||||
res.end(Buffer.from([0xE4, 0xB8]))
|
||||
}
|
||||
expect((await provider().fetch({ url: base })).body.content).toBe('\uFFFD')
|
||||
})
|
||||
|
||||
it.each([
|
||||
['中文', 5, true],
|
||||
['中文', 3, true],
|
||||
['中', 3, false],
|
||||
] as const)('returns complete UTF-8 characters for %j with byte cap %i (truncated: %s)', async (body, maxResponseBytes, truncated) => {
|
||||
handler = (_req, res) => {
|
||||
res.writeHead(200, { 'content-type': 'text/plain; charset=gbk,utf-8' })
|
||||
res.end(body)
|
||||
}
|
||||
const result = await provider({ maxResponseBytes }).fetch({ url: base })
|
||||
expect(result.body.content).toBe('中')
|
||||
expect(result.truncated).toBe(truncated)
|
||||
})
|
||||
|
||||
it('validates EOF when the response exactly fills the byte cap', async () => {
|
||||
handler = (_req, res) => {
|
||||
res.writeHead(200, { 'content-type': 'text/plain; charset=gbk,utf-8' })
|
||||
res.end(Buffer.from([0xE4, 0xB8]))
|
||||
}
|
||||
await expect(provider({ maxResponseBytes: 2 }).fetch({ url: base })).rejects.toThrow(expect.objectContaining({
|
||||
code: 'WEB_UNSUPPORTED_CONTENT_TYPE',
|
||||
message: 'response body is not valid UTF-8 for the malformed charset declaration',
|
||||
}))
|
||||
})
|
||||
|
||||
it('rejects invalid UTF-8 before the truncated suffix', async () => {
|
||||
handler = (_req, res) => {
|
||||
res.writeHead(200, { 'content-type': 'text/plain; charset=gbk,utf-8' })
|
||||
res.end(Buffer.from([0x81, 0x20, 0x61]))
|
||||
}
|
||||
await expect(provider({ maxResponseBytes: 2 }).fetch({ url: base })).rejects.toThrow(expect.objectContaining({
|
||||
code: 'WEB_UNSUPPORTED_CONTENT_TYPE',
|
||||
message: 'response body is not valid UTF-8 for the malformed charset declaration',
|
||||
}))
|
||||
})
|
||||
|
||||
it('uses UTF-8 even when a long body also forms valid GBK', async () => {
|
||||
// C2 A9 represents © in UTF-8 and 漏 in GBK; length cannot disambiguate it.
|
||||
handler = (_req, res) => {
|
||||
res.writeHead(200, { 'content-type': 'text/plain; charset=gbk,utf-8' })
|
||||
res.end('©'.repeat(1000))
|
||||
}
|
||||
expect((await provider().fetch({ url: base })).body.content).toBe('©'.repeat(1000))
|
||||
})
|
||||
})
|
||||
|
||||
describe('HttpFetchProvider redirects', () => {
|
||||
it('follows a same-origin redirect and reports the final URL', async () => {
|
||||
handler = (req, res) => {
|
||||
@@ -342,7 +457,6 @@ describe('HttpFetchProvider invalid URLs and abort', () => {
|
||||
await expect(provider().fetch({ url: 'http://127.0.0.1:1/' }))
|
||||
.rejects.toThrow(expect.objectContaining({ code: 'WEB_PROVIDER_ERROR' }))
|
||||
})
|
||||
|
||||
})
|
||||
|
||||
describe('HttpFetchProvider body cancellation on error paths', () => {
|
||||
@@ -364,8 +478,8 @@ describe('HttpFetchProvider body cancellation on error paths', () => {
|
||||
await closed
|
||||
})
|
||||
|
||||
it('cancels the body when an unsupported charset is rejected', async () => {
|
||||
const closed = respondWithOpenBody(200, { 'content-type': 'text/plain; charset=not-a-charset' })
|
||||
it.each(['not-a-charset', 'gb2312,gbk', 'windows-1252,gbk'])('cancels the body when charset %s is rejected', async (charset) => {
|
||||
const closed = respondWithOpenBody(200, { 'content-type': `text/plain; charset=${charset}` })
|
||||
await expect(provider().fetch({ url: base }))
|
||||
.rejects.toThrow(expect.objectContaining({ code: 'WEB_UNSUPPORTED_CONTENT_TYPE' }))
|
||||
await closed
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
/**
|
||||
* Deterministic HTTP provider for the web-fetch snapshot scenario: a small
|
||||
* HTML page (headings, named entities, a GFM table, nested formatting) on a
|
||||
* UTF-8 HTML page with a malformed charset declaration on an
|
||||
* OS-assigned loopback port behind the real HTTP transport. Recording
|
||||
* and replay therefore exercise fetch and markdown rendering without external
|
||||
* network while retaining the recorded request URL.
|
||||
@@ -14,10 +14,10 @@ const RECORDED_URL = 'http://public.test:43117/menu.html'
|
||||
const PAGE = `<!doctype html>
|
||||
<html><head><title>Menu</title><style>.x{color:red}</style><script>ignored()</script></head>
|
||||
<body>
|
||||
<h1>Café menu</h1>
|
||||
<h1>Café menu</h1>
|
||||
<p>Prices include <strong>service & <em>tax</em></strong> — updated daily.</p>
|
||||
<ul><li>Espresso</li><li>Flat white</li></ul>
|
||||
<table><thead><tr><th>Drink</th><th>Price</th></tr></thead><tbody><tr><td>Espresso</td><td>€2</td></tr><tr><td>Flat white</td><td>€3</td></tr></tbody></table>
|
||||
<table><thead><tr><th>Drink</th><th>Price</th></tr></thead><tbody><tr><td>Espresso</td><td>€2</td></tr><tr><td>Flat white</td><td>€3</td></tr></tbody></table>
|
||||
<p>See <a href="https://fixture.invalid/specials">today’s specials</a>.</p>
|
||||
</body></html>
|
||||
`
|
||||
@@ -62,7 +62,7 @@ export async function apply(ctx) {
|
||||
label: 'web-fetch-fixture-server',
|
||||
requestListener: (req, res) => {
|
||||
if (req.url === '/menu.html') {
|
||||
res.writeHead(200, { 'content-type': 'text/html; charset=utf-8' })
|
||||
res.writeHead(200, { 'content-type': 'text/html; charset=gbk,utf-8' })
|
||||
res.end(PAGE)
|
||||
return
|
||||
}
|
||||
|
||||
Reference in new issue
Block a user