description, kind
| description | kind |
|---|---|
| The anonymous HTTP(S) fetch backend for ctx.web: how deployments mount bounded URL retrieval with same-origin redirects and text-only decoding. | package-reference |
@deepseek-ai/dsh-web-fetch-http
English | 中文
Summary
With dsh-web-fetch-http, the harness can fetch HTTP(S) pages through the web service (ctx.web) and get their status code plus bounded, decoded content without sending credentials. Choose it when a composition needs retrieval with URL validation, same-origin redirects, byte and character caps, and an explicit product User-Agent. It returns non-2xx responses as results rather than errors, and rejects binary data and unsupported content types. It does not restrict destination addresses, so it reaches loopback and private-network hosts. The model-facing web_fetch tool lives in dsh-tool-web, which renders this provider's bodies.
Table of Contents
- Use this package
- Understand the implementation
- Further Exploration
- Model Experience
- Known Limitations and Deferred Work
- Dev Note
Use this package
Mount the provider in a composition that already loads the web service; it registers as the http fetch provider, so ctx.web.fetch() resolves it automatically when it is the only usable fetch backend — or pin it with fetchProvider: http.
When to choose it
Choose this backend when a deployment must fetch pages with bounded output: no credentials are sent, redirects cannot escape the origin, and every response is capped.
Minimal configuration
Load the web service and the provider; configurable limits have safe defaults and validate at plugin construction, so an invalid value fails loudly instead of building a provider with nonsensical caps. The URL security limit is fixed at 2,048 characters.
- name: '@deepseek-ai/dsh-web'
- name: '@deepseek-ai/dsh-web-fetch-http'
| Field | Default | Meaning |
|---|---|---|
maxResponseBytes |
5,000,000 |
Maximum response body size in bytes |
maxBodyChars |
100,000 |
Maximum decoded body length in characters |
timeoutMs |
30,000 |
Fetch timeout — a resource backstop, not the model-facing tool budget |
maxRedirects |
5 |
Maximum same-origin redirect hops (0 follows none) |
userAgent |
deepseek-harness/… |
User-Agent header sent on every request |
The generated configuration catalog is the exhaustive source for every accepted field and its JSDoc.
What a fetch returns
A successful call yields a WebFetchResult: the final URL after allowed redirects, the HTTP status code, a decoded body classified as html or text, and a truncated flag. A non-2xx response is a result, not an error — the status code is part of the fetched resource state; WebError is reserved for failures to safely retrieve or represent the resource.
const page = await ctx.web.fetch({ url: 'https://example.com' })
// page.body.kind === 'html' | 'text'; page.statusCode === 200 | 404 | ...
Transport behavior
The provider keeps requests anonymous and bounded: it accepts only http: and https: URLs without embedded credentials and rejects URLs over 2,048 characters. Requests go through the process-wide dispatcher, so the installed proxy policy applies and the system resolver chooses the address; any destination address is allowed. Same-origin redirects are followed; cross-origin redirects fail and require a fresh call. The provider also enforces byte, character, hop, and time caps, rejects unsupported content types, and sends an explicit product User-Agent.
A malformed comma- or whitespace-joined charset declaration containing a UTF-8 label (such as charset=utf-8,gbk) is decoded strictly as UTF-8. Recovery rejects invalid UTF-8 without trying another encoding. When the byte cap cuts a character, recovery omits the incomplete suffix and returns truncated: true; a complete response using recovery must pass the EOF check. Valid single-label declarations retain ordinary replacement decoding.
Failures and recovery
Failures throw WebError with a machine-routable code: WEB_INVALID_URL, WEB_BLOCKED_URL, WEB_FETCH_TOO_LARGE, WEB_FETCH_TIMEOUT, WEB_REDIRECT_BLOCKED, WEB_UNSUPPORTED_CONTENT_TYPE, WEB_ABORTED, or WEB_PROVIDER_ERROR. WEB_UNSUPPORTED_CONTENT_TYPE covers unsupported media types, unsupported charset declarations, and invalid UTF-8 during malformed-declaration recovery. Direct callers can route on the code; the model-facing web_fetch tool surfaces the failure text to the model under its own error wrapper.
Understand the implementation
Implementation internals — click to expand
This section explains the design decisions behind the provider; the observable behavior is fully covered in Use this package.
Design philosophy
The package is built on one separation and one layered timeout:
- Retrieval vs. presentation. This provider owns URL validation, HTTP transport, redirect policy, caps, charset decoding, and binary rejection;
dsh-tool-webowns HTML→markdown and truncation formatting. A non-2xx response is data, not failure. - Two timeout layers. The provider's
timeoutMsis a resource backstop for directctx.web.fetch()callers; the model-facing tool-call budget belongs todsh-tool-call-timeout-policy, which armsexec.signal. When the outer deadline fires first the provider reportsWEB_ABORTEDand the policy replaces it withTOOL_TIMEOUT;WEB_FETCH_TIMEOUTtherefore identifies a direct service caller whose provider budget elapsed.
Source map
| File | Role |
|---|---|
src/index.ts |
Plugin entry: config schema, limit validation, provider registration |
src/provider.ts |
The HttpFetchProvider: transport, redirect following, capped reads, charset decoding |
src/policy.ts |
URL validation, same-origin checks, content-type classification, charset parsing |
Read path
A fetch validates the URL and sends the request. It repeats URL validation for each same-origin redirect; a cross-origin redirect fails before response bytes are accepted. The final response is classified by Content-Type, decoded from its declared charset (malformed declarations containing UTF-8 use strict UTF-8 recovery), and read under the byte cap; the decoded text is then truncated to the character cap.
Further Exploration
Read these pages when the package-level contract is not enough. They move from the shared vocabulary to the service, the model-facing tools, and the design rationale.
- Web subsystem — the exhaustive fetch request/result vocabulary and error codes.
- Web package map — the six-package family and each role.
- dsh-web — the web service this provider registers into.
- dsh-tool-web — the model-facing
web_fetchtool that renders this provider's bodies. - Generated configuration catalog — every accepted config field and its source declaration.
- Web capability seam decision — why search and fetch share one provider-selection service.
Model Experience
Indirectly, through dsh-tool-web, which renders this provider's maxBodyChars-bounded decoded text or markdown-shaped HTML under its fetch-result wrapper while redirects, headers, and transport limits remain hidden.
KV Cache effect
No direct invalidation; the named consumer owns any request-prefix changes.
Known Limitations and Deferred Work
These limits define when the provider is unsafe or a poor fit. They are current package constraints.
- Private destinations are reachable — loopback, private-network, and cloud metadata addresses are fetched like any other host. A deployment that must keep the model away from them needs a
tools/pre-executepolicy, network confinement, or no fetch tool. - Only textual content decodes — html/xhtml and
text/*plus JSON/XML families; a missingContent-Typeor any binary type throwsWEB_UNSUPPORTED_CONTENT_TYPE, and text-extractable PDF decoding is named deferred work. - Charset comes only from the
Content-Typeheader (UTF-8 default) — an HTML<meta charset>declaration is ignored, and unsupported declarations without a UTF-8 label throw. - UTF-8 recovery is a preference, not encoding detection — bytes can be valid as both UTF-8 and another encoding, even in a long response. Recovery chooses UTF-8 in that case; malformed declarations without a UTF-8 label remain unsupported.
Dev Note
Working context for maintainers — click to expand
None.