diff --git a/.agents/notes/implemented/architecture/2026-08-22-single-dsh-application-launcher.i18n.yaml b/.agents/notes/implemented/architecture/2026-08-22-single-dsh-application-launcher.i18n.yaml index 5e193c9063..ba50248d0c 100644 --- a/.agents/notes/implemented/architecture/2026-08-22-single-dsh-application-launcher.i18n.yaml +++ b/.agents/notes/implemented/architecture/2026-08-22-single-dsh-application-launcher.i18n.yaml @@ -21,8 +21,8 @@ en: 8ee88035cf64eeb3 zh: 6184e50fc4d47348 /agent-note-one-dsh-launcher-for-application-profiles/decision/python-runtime: - en: 5962abadfdece709 - zh: 5db20e96deed49d1 + en: 4b4cf5071b740c85 + zh: 595e5fe8127f9067 /agent-note-one-dsh-launcher-for-application-profiles/decision/enforcement: en: 9ce8d86b65064f38 zh: 27b8ad426cb319a2 diff --git a/.agents/notes/implemented/architecture/2026-08-22-single-dsh-application-launcher.md b/.agents/notes/implemented/architecture/2026-08-22-single-dsh-application-launcher.md index 7d4bb1a9c2..27a8315fe6 100644 --- a/.agents/notes/implemented/architecture/2026-08-22-single-dsh-application-launcher.md +++ b/.agents/notes/implemented/architecture/2026-08-22-single-dsh-application-launcher.md @@ -46,7 +46,7 @@ Direct SDK use follows normal Harness-home resolution: explicit `dshHome`, inher ### Python runtime -The Python runtime wheel stages [`python/sdk-runtime/runtime-bootstrap.mjs`](../../../../python/sdk-runtime/runtime-bootstrap.mjs) as the `dsh-python-runtime-closure` entry. Its ordinary branch calls the public CLI export; a provider-private selector dispatches to the internal subprocess runner before CLI parsing and is not an application entry point. The [native-containment reference](../../../../packages/subprocess/subprocess-local/README.md) owns that private dispatch. The Python client selects `dsh --profile sdk` by default, ordered patch files, and an explicit Harness home; the runnable example under `python/sdk/examples` selects `sdk-minimal`. The installed `dsh` console command exposes the same profile grammar, including the `web` profile. +The Python runtime wheel stages [`python/sdk-runtime/runtime-bootstrap.mjs`](../../../../python/sdk-runtime/runtime-bootstrap.mjs) as the `dsh-python-runtime-closure` entry. Its ordinary branch calls the public CLI export; a provider-private selector dispatches to the internal subprocess runner before CLI parsing and is not an application entry point. A private `DSH_RUNTIME_DOWNLOAD` selector prepares optional resources before application launch and is removed before spawning preparation subprocesses; ordinary CLI arguments do not select downloads. The [native-containment reference](../../../../packages/subprocess/subprocess-local/README.md) owns that private dispatch. The Python client selects `dsh --profile sdk` by default, ordered patch files, and an explicit Harness home; the runnable example under `python/sdk/examples` selects `sdk-minimal`. The installed `dsh` console command exposes the same profile grammar, including the `web` profile. The executable family is `deepseek-harness-sdk-runtime--`. The SDK wire, wheel and import distribution names, sidecar names, and wire identity `deepseek-harness-sdk-runtime` remain stable. The SDK package family is `@deepseek-ai/dsh-sdk-client`, `@deepseek-ai/dsh-sdk-protocol`, and `@deepseek-ai/dsh-sdk-jsonrpc-server`; `@deepseek-ai/dsh-acp` remains the ACP protocol plugin. There is no Python-specific Node application, checked-in complete config, compatibility package, forwarding executable, fallback parser, or SDK/ACP launcher alias. [docs/architecture.md](../../../../docs/architecture.md) owns this launch, and the [`python/sdk-runtime` README](../../../../python/sdk-runtime/README.md) owns the Windows carrier. diff --git a/.agents/notes/implemented/architecture/2026-08-22-single-dsh-application-launcher.zh.md b/.agents/notes/implemented/architecture/2026-08-22-single-dsh-application-launcher.zh.md index eb7d5ff8eb..f9a50ebf5a 100644 --- a/.agents/notes/implemented/architecture/2026-08-22-single-dsh-application-launcher.zh.md +++ b/.agents/notes/implemented/architecture/2026-08-22-single-dsh-application-launcher.zh.md @@ -46,7 +46,7 @@ SDK 用户通过 profile 自定义插件。`dsh plugin --profile ...` 管 ### Python 运行时 -Python 运行时 wheel 将 [`python/sdk-runtime/runtime-bootstrap.mjs`](../../../../python/sdk-runtime/runtime-bootstrap.mjs) 暂存为 `dsh-python-runtime-closure` 入口。其普通分支调用公开 CLI export;提供方私有选择会在 CLI 解析前分派到内部子进程 runner,而不是应用入口。[原生 containment 参考](../../../../packages/subprocess/subprocess-local/README.zh.md)负责该私有分派。Python 客户端默认选择 `dsh --profile sdk`、有序 patch 文件与显式 Harness home;`python/sdk/examples` 下的可运行示例选择 `sdk-minimal`。安装的 `dsh` 控制台命令暴露相同的 profile 语法,包括 `web` profile。 +Python 运行时 wheel 将 [`python/sdk-runtime/runtime-bootstrap.mjs`](../../../../python/sdk-runtime/runtime-bootstrap.mjs) 暂存为 `dsh-python-runtime-closure` 入口。其普通分支调用公开 CLI export;提供方私有选择会在 CLI 解析前分派到内部子进程 runner,而不是应用入口。私有 `DSH_RUNTIME_DOWNLOAD` 选择器在应用启动前准备可选资源,并在启动资源准备子进程前移除;普通 CLI 参数不选择下载。[原生 containment 参考](../../../../packages/subprocess/subprocess-local/README.zh.md)负责该私有分派。Python 客户端默认选择 `dsh --profile sdk`、有序 patch 文件与显式 Harness home;`python/sdk/examples` 下的可运行示例选择 `sdk-minimal`。安装的 `dsh` 控制台命令暴露相同的 profile 语法,包括 `web` profile。 可执行文件族是 `deepseek-harness-sdk-runtime--`。SDK 协议格式、wheel 与 import 分发名称、伴随文件名称,以及协议 identity `deepseek-harness-sdk-runtime` 保持稳定。SDK 包族是 `@deepseek-ai/dsh-sdk-client`、`@deepseek-ai/dsh-sdk-protocol` 与 `@deepseek-ai/dsh-sdk-jsonrpc-server`;`@deepseek-ai/dsh-acp` 继续作为 ACP 协议插件。仓库不保留 Python 专用 Node 应用、检入的完整配置、兼容包、转发可执行文件、后备解析器或 SDK/ACP 启动别名。[docs/architecture.md](../../../../docs/architecture.zh.md)负责该启动方式,[`python/sdk-runtime` README](../../../../python/sdk-runtime/README.zh.md)负责 Windows 载体。 diff --git a/python/sdk-runtime/README.i18n.yaml b/python/sdk-runtime/README.i18n.yaml index 5948bb273e..5780264493 100644 --- a/python/sdk-runtime/README.i18n.yaml +++ b/python/sdk-runtime/README.i18n.yaml @@ -6,14 +6,14 @@ en: 1ea1d8e6d2c14b02 zh: 8e0b9903158667a1 /deepseek-harness-runtime-bin/installed-commands-and-artifacts: - en: fb7f6c3cec95833b - zh: b04e60cbc0ba96fc + en: 89c97974ad5e97a6 + zh: af99634672c80994 /deepseek-harness-runtime-bin/python-module-api: en: 78a200e328d96637 zh: f3b7a0b69b7a39e0 /deepseek-harness-runtime-bin/packaged-profile-resolution: - en: 30b9d8f69da71b23 - zh: 9eb984af84dc7ead + en: f1123b8835b57bbe + zh: 05af612c33c5d855 /deepseek-harness-runtime-bin/build-and-distribution: - en: 1973f696bc202247 - zh: 4642f2414b1618f2 + en: 01b6d838deae9c7b + zh: 88b09ab5f1ae14ca diff --git a/python/sdk-runtime/README.md b/python/sdk-runtime/README.md index 23062200dc..c3d32f9b3e 100644 --- a/python/sdk-runtime/README.md +++ b/python/sdk-runtime/README.md @@ -21,11 +21,11 @@ office = download_office() primary_runtime = download_primary_runtime() ``` -`download_office()` downloads the locked npm Kit, target engine, and complete dependency tree, plus standalone Node for the Kit CLI. `download_primary_runtime()` independently downloads CPython, locked Python libraries, Node, pnpm, and Office skills. Both return absolute resource directories and reuse completed caches. The existing local `installPrimaryRuntime()` operation still copies an already available payload; it does not download. +`download_office()` downloads the locked npm Kit, target engine, and complete dependency tree, plus standalone Node for the Kit CLI. `download_primary_runtime()` independently downloads CPython, locked Python libraries, Node, pnpm, and Office skills. Both return absolute resource directories and reuse completed caches. The authoring download validates its target, interpreters, library directory and skills before returning. The existing local `installPrimaryRuntime()` operation still copies an already available payload; it does not download. -`DSH_RESOURCE_CACHE` selects an absolute cache root, defaulting to `~/.cache/deepseek-harness/resources`. Resource identities include the release's locked inputs. Downloads verify archive hashes and package identities, preserve executable permissions, and publish completed directories atomically. Failed downloads can be retried. `DSH_RESOURCE_DOWNLOAD_TIMEOUT_MS` bounds each archive download, including its body; the default is `300000`, and accepted values are integers from `1` to `2147483647`. Downloads follow the Harness proxy environment policy. Installed wheel files are never modified. +`DSH_RESOURCE_CACHE` selects an absolute cache root, defaulting to `~/.cache/deepseek-harness/resources`. Resource identities include the release's locked inputs. Downloads verify archive hashes and package identities, preserve executable permissions, and publish completed directories atomically. Failed downloads can be retried. `DSH_RESOURCE_DOWNLOAD_TIMEOUT_MS` bounds each archive download, including its body; the default is `300000`, and accepted values are integers from `1` to `2147483647`. Downloads follow the Harness proxy environment policy. Installed wheel files are never modified. Release versions have separate cache directories; unused versions require manual removal. Downloads use the locked upstream URLs without a registry or mirror override. -On subsequent launches the packaged bootstrap exposes downloaded resources to the SDK profile. `DSH_PRIMARY_RUNTIME` overrides the cached authoring environment; an empty value disables its query. `DSH_OFFICE_SIDECAR` can select an absolute sidecar directory with its standalone Node. Office skills are available when either resource is selected; authoring-only deployments expose the skills with the Kit CLI disabled. External authoring payloads retain the `primary-runtime/` plus sibling `office-skills/` layout. The SDK reads resources in place without copying them into `DSH_HOME`. The `sdk-minimal` profile does not mount these providers. +On subsequent launches the packaged bootstrap exposes downloaded resources to the SDK profile. `DSH_PRIMARY_RUNTIME` overrides the cached authoring environment; an empty value disables its query. `DSH_OFFICE_SIDECAR` can select an absolute sidecar directory with its standalone Node. The Office checker is included in the wheel; users may provide their own Python environment and document libraries without downloading the authoring environment. Office skills are available when either resource is selected; authoring-only deployments expose the skills with the Kit CLI disabled. External authoring payloads retain the `primary-runtime/` plus sibling `office-skills/` layout. The SDK reads resources in place without copying them into `DSH_HOME`. The `sdk-minimal` profile does not mount these providers. Skill selection is independent of delivery: project, custom-directory, and user filesystem skills override same-name bundled skills. An SDK patch can disable only the Office provider while retaining the Python query: @@ -56,7 +56,7 @@ Unsupported platforms and missing executables or sidecars raise `FileNotFoundErr `dsh` initializes shipped profiles under the explicit home, composes their bundle patches, and loads bundled plugins from the executable's virtual filesystem. Runtime resolution uses an in-memory generation instead of disk symlinks or proxy packages. Fallback imports use recorded declaring-package paths, including paths inside the executable's virtual filesystem, so built-in rows and external plugin peers share the bundled Cordis/module instance. Native shared libraries and Windows ConPTY addons are packaged with native addons, while ripgrep and the macOS PTY helper remain executable sidecars. -The Python bootstrap resolves the Office kit from its explicitly downloaded or selected sidecar so native helpers and URL Workers use real filesystem paths. The same resource preparation implementation serves repository builds and explicit Python downloads. The kit owns document operations; the Office smoke retrieves CLI paths from the loaded skill and executes capabilities and DOCX conversion with an empty PATH. +Python and the packaged bootstrap share the resource layout: `//office/`, `node/bin/node[.exe]`, and `node_modules/@deepseek-ai/libreoffice-kit/lib/cli.js`. Automatic Office cache discovery requires the `complete` marker to match the manifest identity; an explicit `DSH_OFFICE_SIDECAR` requires the Node and CLI files. The Python bootstrap resolves the Office kit from its explicitly downloaded or selected sidecar so native helpers and URL Workers use real filesystem paths. The same resource preparation implementation serves repository builds and explicit Python downloads. The kit owns document operations; the Office smoke retrieves CLI paths from the loaded skill and executes capabilities and DOCX conversion with an empty PATH. External profile management uses `dsh plugin --profile ...`. That command requires `pnpm` on `PATH`; ordinary SDK/profile execution does not. @@ -64,6 +64,6 @@ External profile management uses `dsh plugin --profile ...`. That command Production deployment permits unused workspace patches for packages outside the runtime closure; patches for included packages must still apply successfully. This exception is confined to the deploy command; repository installation still rejects unused patches. -From the repository root, `pnpm exec tsx scripts/build-exe-for-python-sdk.ts` verifies the closure, builds packages, deploys a symlink-free tree, packages the selected target, and syncs the executable and sidecars into this module. Only the target platform’s PTY prebuilds are packaged. Package README, CHANGELOG, HISTORY Markdown, source maps, and TypeScript declarations outside runtime assets are omitted. Deployment disables workspace hoisting, and packaging excludes the checkout’s node_modules so dependency scans cannot add development packages to the payload. Staging shares root workspace packages with nested consumers of the same pnpm dependency identity, preserving singleton services while retaining different peer resolutions. `scripts/build-python-release.py` stages release-shaped wheels at the root repository version and pins `deepseek-harness-sdk` to the exact runtime version. +From the repository root, `pnpm exec tsx scripts/build-exe-for-python-sdk.ts` verifies the closure, builds packages, deploys a symlink-free tree, packages the selected target, and syncs the executable and sidecars into this module. Only the target platform’s PTY prebuilds are packaged. Package README, CHANGELOG, HISTORY Markdown, source maps, and TypeScript declarations outside runtime assets are omitted. Deployment disables workspace hoisting, and packaging excludes checkout dependencies and workspace source trees outside the deployment so dependency scans cannot add development packages to the payload. Staging shares root workspace packages with nested consumers of the same pnpm dependency identity, preserving singleton services while retaining different peer resolutions. `scripts/build-python-release.py` stages release-shaped wheels at the root repository version and pins `deepseek-harness-sdk` to the exact runtime version. The installed-wheel smoke creates a clean virtual environment outside the checkout, proves the installed distribution and executable identities, then exercises default and customized SDK profiles, external plugins, MCP, native tools, direct JSON-RPC, committed snapshots, and the real provider on trusted runs. Its Office scenario relocates the lightweight target payload, explicitly downloads the sidecar, and converts DOCX with the required platform engine: the target’s declared native engine, or WASM when no native engine is declared. See the [Python contributor workflow](../development.md) and [installed-wheel testing decision](../../.agents/notes/implemented/testing/2026-08-23-installed-python-wheel-black-box-ci.md). diff --git a/python/sdk-runtime/README.zh.md b/python/sdk-runtime/README.zh.md index 08f05e0bd8..8f2c1bf317 100644 --- a/python/sdk-runtime/README.zh.md +++ b/python/sdk-runtime/README.zh.md @@ -21,11 +21,11 @@ office = download_office() primary_runtime = download_primary_runtime() ``` -`download_office()` 下载锁定版本的 npm Kit、目标引擎及完整依赖树,并准备运行 Kit CLI 的独立 Node。`download_primary_runtime()` 独立下载 CPython、锁定的 Python 库、Node、pnpm 和 Office skills。两者返回绝对资源目录,并复用完整缓存。现有本地 `installPrimaryRuntime()` 操作仍只复制已有载荷,不执行下载。 +`download_office()` 下载锁定版本的 npm Kit、目标引擎及完整依赖树,并准备运行 Kit CLI 的独立 Node。`download_primary_runtime()` 独立下载 CPython、锁定的 Python 库、Node、pnpm 和 Office skills。两者返回绝对资源目录,并复用完整缓存。创作下载会在返回前校验目标平台、解释器、库目录与 skills。现有本地 `installPrimaryRuntime()` 操作仍只复制已有载荷,不执行下载。 -`DSH_RESOURCE_CACHE` 指定绝对缓存根目录,默认为 `~/.cache/deepseek-harness/resources`。资源身份包含发布版本锁定的输入。下载会校验归档哈希和包身份、保留执行权限,并原子发布完整目录。下载失败后可重试。`DSH_RESOURCE_DOWNLOAD_TIMEOUT_MS` 限制每个归档的下载耗时,包含响应内容;默认值为 `300000`,接受 `1` 到 `2147483647` 的整数。下载遵循 Harness 代理环境策略。已安装的 wheel 文件不会被修改。 +`DSH_RESOURCE_CACHE` 指定绝对缓存根目录,默认为 `~/.cache/deepseek-harness/resources`。资源身份包含发布版本锁定的输入。下载会校验归档哈希和包身份、保留执行权限,并原子发布完整目录。下载失败后可重试。`DSH_RESOURCE_DOWNLOAD_TIMEOUT_MS` 限制每个归档的下载耗时,包含响应内容;默认值为 `300000`,接受 `1` 到 `2147483647` 的整数。下载遵循 Harness 代理环境策略。已安装的 wheel 文件不会被修改。不同发布版本使用独立缓存目录;不再使用的版本需手动删除。下载使用锁定的上游 URL,不提供 registry 或镜像覆盖。 -后续启动时,打包 bootstrap 将已下载资源提供给 SDK profile。`DSH_PRIMARY_RUNTIME` 覆盖缓存中的创作环境;空值禁用其查询。`DSH_OFFICE_SIDECAR` 可选择包含独立 Node 的绝对 sidecar 目录。选择任一种资源即可使用 Office skills;仅有创作环境时,skills 中的 Kit CLI 被禁用。外部创作载荷保留 `primary-runtime/` 及同级 `office-skills/` 布局。SDK 原位读取资源,不复制到 `DSH_HOME`。`sdk-minimal` profile 不挂载这些提供方。 +后续启动时,打包 bootstrap 将已下载资源提供给 SDK profile。`DSH_PRIMARY_RUNTIME` 覆盖缓存中的创作环境;空值禁用其查询。`DSH_OFFICE_SIDECAR` 可选择包含独立 Node 的绝对 sidecar 目录。Office checker 随 wheel 提供;用户可自行提供 Python 环境和文档库,无需下载创作环境。选择任一种资源即可使用 Office skills;仅有创作环境时,skills 中的 Kit CLI 被禁用。外部创作载荷保留 `primary-runtime/` 及同级 `office-skills/` 布局。SDK 原位读取资源,不复制到 `DSH_HOME`。`sdk-minimal` profile 不挂载这些提供方。 选择 skills 与交付资源相互独立:项目、自定义目录和用户文件系统 skills 优先于同名随包 skills。SDK patch 可以仅禁用 Office 提供方,同时保留 Python 查询: @@ -56,7 +56,7 @@ primary_runtime = download_primary_runtime() `dsh` 在显式指定的主目录下初始化随附 profile、组合其 bundle patch,并从可执行程序的虚拟文件系统加载内置插件。运行时解析使用内存中的 generation,不创建磁盘符号链接或代理包。fallback 导入使用记录的声明包路径,包括可执行程序虚拟文件系统内的路径,因此内置配置项与外部插件 peer 共享内置的 Cordis/模块实例。原生共享库与 Windows ConPTY addon 会同其他原生 addon 一起打包;ripgrep 与 macOS PTY helper 仍是可执行伴随程序。 -Python bootstrap 从显式下载或选择的 sidecar 解析 Office kit,让原生辅助程序与 URL Worker 使用真实文件系统路径。仓库构建与显式 Python 下载共用资源准备实现。kit 负责文档操作;Office smoke 从已加载技能获取 CLI 路径,并在空 PATH 下执行 capabilities 和 DOCX 转换。 +Python 与打包的 bootstrap 共享资源布局:`//office/`、`node/bin/node[.exe]` 与 `node_modules/@deepseek-ai/libreoffice-kit/lib/cli.js`。自动发现 Office 缓存要求 `complete` 标记匹配清单 identity;显式 `DSH_OFFICE_SIDECAR` 要求 Node 与 CLI 文件存在。Python bootstrap 从显式下载或选择的 sidecar 解析 Office kit,让原生辅助程序与 URL Worker 使用真实文件系统路径。仓库构建与显式 Python 下载共用资源准备实现。kit 负责文档操作;Office smoke 从已加载技能获取 CLI 路径,并在空 PATH 下执行 capabilities 和 DOCX 转换。 外部 profile 管理使用 `dsh plugin --profile ...`。该命令要求 `PATH` 中存在 `pnpm`;普通 SDK/profile 运行不需要它。 @@ -64,6 +64,6 @@ Python bootstrap 从显式下载或选择的 sidecar 解析 Office kit,让原 生产部署允许工作区中不属于运行时闭包的补丁保持未使用;闭包内包的补丁仍必须成功应用。此例外仅用于部署命令,仓库安装仍拒绝未使用的补丁。 -在仓库根目录运行 `pnpm exec tsx scripts/build-exe-for-python-sdk.ts`,会校验闭包、构建包、部署无符号链接的文件树、打包所选目标,并把可执行程序及伴随文件同步到本模块。打包时只保留目标平台的 PTY 预构建文件,并排除运行时资源目录之外的包内 README、CHANGELOG、HISTORY Markdown 文档、source map 和 TypeScript 声明。部署会关闭工作区提升,打包时还会排除检出目录的 node_modules,防止依赖扫描把开发包加入载荷。暂存步骤让使用相同 pnpm 依赖标识的嵌套消费者共享根目录的工作区包,保留服务单例,同时保留不同的对等依赖解析。`scripts/build-python-release.py` 按仓库根版本暂存发布形态的 wheel 包,并将 `deepseek-harness-sdk` 固定到完全相同的运行时版本。 +在仓库根目录运行 `pnpm exec tsx scripts/build-exe-for-python-sdk.ts`,会校验闭包、构建包、部署无符号链接的文件树、打包所选目标,并把可执行程序及伴随文件同步到本模块。打包时只保留目标平台的 PTY 预构建文件,并排除运行时资源目录之外的包内 README、CHANGELOG、HISTORY Markdown 文档、source map 和 TypeScript 声明。部署会关闭工作区提升,打包时还会排除部署目录之外的检出依赖和工作区源码树,防止依赖扫描把开发包加入载荷。暂存步骤让使用相同 pnpm 依赖标识的嵌套消费者共享根目录的工作区包,保留服务单例,同时保留不同的对等依赖解析。`scripts/build-python-release.py` 按仓库根版本暂存发布形态的 wheel 包,并将 `deepseek-harness-sdk` 固定到完全相同的运行时版本。 已安装 wheel 包冒烟测试会在检出目录外创建干净的虚拟环境,验证分发物与可执行程序的来源,然后覆盖默认及自定义 SDK profile、外部插件、MCP、原生工具、直接 JSON-RPC、检入快照,以及可信运行中的真实提供方。Office 场景会迁移轻量目标载荷、显式下载 sidecar,并使用所需的平台引擎转换 DOCX:使用目标已声明的原生引擎,未声明原生引擎时使用 WASM。另见 [Python 贡献者工作流](../development.zh.md) 与 [installed-wheel 测试决策](../../.agents/notes/implemented/testing/2026-08-23-installed-python-wheel-black-box-ci.zh.md)。 diff --git a/python/sdk-runtime/runtime-bootstrap.mjs b/python/sdk-runtime/runtime-bootstrap.mjs index a83d877dbc..d860ccafca 100644 --- a/python/sdk-runtime/runtime-bootstrap.mjs +++ b/python/sdk-runtime/runtime-bootstrap.mjs @@ -80,8 +80,11 @@ const aclRunner = process.platform === 'win32' ? fileURLToPath(import.meta.resolve('@deepseek-ai/dsh-sandbox-windows-acl/runner')) : undefined -if (process.argv[2] === '--download-office' || process.argv[2] === '--download-primary-runtime') { - const path = await downloadResource(process.argv[2] === '--download-office' ? 'office' : 'primary') +const resource = process.env.DSH_RUNTIME_DOWNLOAD +if (resource !== undefined) { + Reflect.deleteProperty(process.env, 'DSH_RUNTIME_DOWNLOAD') + if (resource !== 'office' && resource !== 'primary') throw new Error('DSH_RUNTIME_DOWNLOAD must select office or primary') + const path = await downloadResource(resource) console.log(JSON.stringify(path)) } else if (aclRunner !== undefined && process.argv[2] === aclRunner) { process.argv.splice(1, 1) diff --git a/python/sdk-runtime/src/deepseek_harness_runtime/__init__.py b/python/sdk-runtime/src/deepseek_harness_runtime/__init__.py index ef0e01ce74..bd853f73b1 100644 --- a/python/sdk-runtime/src/deepseek_harness_runtime/__init__.py +++ b/python/sdk-runtime/src/deepseek_harness_runtime/__init__.py @@ -28,7 +28,7 @@ import shutil import subprocess import sys from pathlib import Path -from ._resources import validate_downloads, office_launch_args +from ._resources import validate_downloads, validate_resources, office_launch_args PACKAGE_METADATA_FILENAME = "deepseek-harness-runtime.json" @@ -153,8 +153,9 @@ def _node_launch_args() -> tuple[str, str]: return (node, str(bin_js)) -def _download(flag: str) -> Path: - result = subprocess.run((*resolve_bundled_launch_args(), flag), capture_output=True, text=True) +def _download(resource: str) -> Path: + result = subprocess.run(resolve_bundled_launch_args(), capture_output=True, encoding="utf-8", + env={**os.environ, "DSH_RUNTIME_DOWNLOAD": resource}) if result.returncode: raise RuntimeError(result.stderr.strip() or f"resource download failed with exit code {result.returncode}") path = json.loads(result.stdout) @@ -171,7 +172,7 @@ def download_office() -> Path: This explicit operation requires network access on a cache miss, not a Harness home, agent, or model. Other SDK operations never trigger downloads. """ - return _download("--download-office") + return _download("office") def download_primary_runtime() -> Path: @@ -181,7 +182,9 @@ def download_primary_runtime() -> Path: Office engine download and does not copy resources into a Harness home. Completed downloads are reused; DSH_RESOURCE_CACHE selects the cache root. """ - return _download("--download-primary-runtime") + path = _download("primary") + validate_resources(path.parent, _current_platform_tag()) + return path def resolve_office_launch_args() -> tuple[str, str]: diff --git a/python/sdk-runtime/src/deepseek_harness_runtime/_resources.py b/python/sdk-runtime/src/deepseek_harness_runtime/_resources.py index 9898d0829f..ae547ac26c 100644 --- a/python/sdk-runtime/src/deepseek_harness_runtime/_resources.py +++ b/python/sdk-runtime/src/deepseek_harness_runtime/_resources.py @@ -1,4 +1,4 @@ -"""Validate the authoring resources in staged, installed, or archived runtime wheels.""" +"""Validate wheel download manifests, downloaded resources, and Office process launches.""" from __future__ import annotations @@ -72,6 +72,12 @@ def office_launch_args(root: Path, target: str) -> tuple[str, str]: office = Path(os.environ.get("DSH_OFFICE_SIDECAR", str(cache / metadata["identity"] / "office"))) if not office.is_absolute(): raise ValueError("DSH_OFFICE_SIDECAR must be an absolute directory") + if "DSH_OFFICE_SIDECAR" not in os.environ: + complete = office / "complete" + if not complete.is_file(): + raise FileNotFoundError(f"Office download is incomplete: {office}. Call deepseek_harness_runtime.download_office() first.") + if complete.read_text(encoding="utf-8") != metadata["identity"]: + raise ValueError(f"Office cache identity mismatch: {office}") node = office / "node/bin" / ("node.exe" if target == "win-x64" else "node") cli = office / "node_modules/@deepseek-ai/libreoffice-kit/lib/cli.js" for path in (node, cli): @@ -80,16 +86,16 @@ def office_launch_args(root: Path, target: str) -> tuple[str, str]: return str(node), str(cli) -def run_office_process(arguments: Sequence[str], *, capture_output: bool) -> subprocess.CompletedProcess[bytes]: - """Run Office with captured or inherited streams; a second interrupt kills its process tree.""" +def run_office_process(arguments: Sequence[str]) -> subprocess.CompletedProcess[bytes]: + """Run Office with captured streams; a second interrupt kills its process tree.""" taskkill = os.path.join(os.environ.get("SystemRoot", ""), "System32", "taskkill.exe") if os.name == "nt" and not os.path.isabs(taskkill): raise ValueError("Office operations on Windows require an absolute SystemRoot environment value.") with subprocess.Popen( arguments, - stdin=subprocess.DEVNULL if capture_output else None, - stdout=subprocess.PIPE if capture_output else None, - stderr=subprocess.PIPE if capture_output else None, + stdin=subprocess.DEVNULL, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, start_new_session=os.name != "nt", ) as process: try: diff --git a/python/sdk/src/deepseek_harness/office.py b/python/sdk/src/deepseek_harness/office.py index 3a213f0460..9e2dab604d 100644 --- a/python/sdk/src/deepseek_harness/office.py +++ b/python/sdk/src/deepseek_harness/office.py @@ -136,7 +136,6 @@ def _run(arguments: list[str], options: OfficeOptions | None, result_type: type[ process = run_office_process( [*deepseek_harness_runtime.resolve_office_launch_args(), *arguments, *(options._arguments() if options is not None else [])], - capture_output=True, ) stdout, stderr = process.stdout, process.stderr if process.returncode != 0: diff --git a/python/sdk/tests/test_office_interrupts.py b/python/sdk/tests/test_office_interrupts.py new file mode 100644 index 0000000000..2cd4dbd612 --- /dev/null +++ b/python/sdk/tests/test_office_interrupts.py @@ -0,0 +1,106 @@ +"""Office interruption terminates the owned process tree before returning to callers.""" +from __future__ import annotations + +import ntpath +import os +import signal +import subprocess +import sys +import time +from pathlib import Path +from types import SimpleNamespace + +import pytest +from deepseek_harness_runtime import _resources + + +def wait_for_file(path: Path, process: subprocess.Popen, timeout: float = 30) -> None: + deadline = time.monotonic() + timeout + while not path.is_file(): + if process.poll() is not None: + raise AssertionError(f"fixture exited before {path.name}: {process.returncode}") + if time.monotonic() >= deadline: + raise TimeoutError(f"fixture never produced {path}") + time.sleep(0.01) + + +@pytest.mark.skipif(os.name == "nt", reason="POSIX process groups and SIGINT delivery") +def test_second_interrupt_waits_for_the_entire_office_process_tree(tmp_path: Path) -> None: + fixture = tmp_path / "office.py" + ready, interrupted, pidfile = (tmp_path / name for name in ("ready", "interrupted", "pid")) + fixture.write_text("""import json,os,signal,subprocess,sys +from pathlib import Path +root=Path(sys.argv[1]) +(root/'pid').write_text(str(os.getpid())) +signal.signal(signal.SIGINT,lambda *_:(root/'interrupted').touch()) +child=subprocess.Popen([sys.executable,'-c', + "import signal,sys; from pathlib import Path; signal.signal(signal.SIGINT,signal.SIG_IGN); Path(sys.argv[1]).touch(); signal.pause()",str(root/'child-ready')]) +(root/'ready').touch() +child.wait() +""", encoding="utf-8") + code = ("import sys; from deepseek_harness_runtime._resources import run_office_process; " + f"args=[sys.executable,{str(fixture)!r},{str(tmp_path)!r}]\n" + "try: run_office_process(args)\n" + "except KeyboardInterrupt: print('tree terminated',flush=True); sys.exit(73)\n") + process = subprocess.Popen([sys.executable, "-c", code], stdout=subprocess.PIPE, stderr=subprocess.PIPE, + start_new_session=True, env={**os.environ, + "PYTHONPATH": str(Path(_resources.__file__).resolve().parents[1])}) + try: + wait_for_file(ready, process) + wait_for_file(tmp_path / "child-ready", process) + os.kill(process.pid, signal.SIGINT) + wait_for_file(interrupted, process) + os.kill(process.pid, signal.SIGINT) + # The grandchild inherits the captured streams; communicate can finish only after it closes them. + stdout, stderr = process.communicate(timeout=30) + assert process.returncode == 73, stderr + assert stdout == b"tree terminated\n" + finally: + if pidfile.is_file(): + try: + os.killpg(int(pidfile.read_text()), signal.SIGKILL) + except ProcessLookupError as error: + # The successful interruption path already terminated this group. + pass + if process.poll() is None: + process.kill() + process.communicate(timeout=30) + + +def test_windows_second_interrupt_uses_absolute_taskkill_and_waits(monkeypatch: pytest.MonkeyPatch) -> None: + calls = [] + + class Process: + pid = 73 + returncode = -9 + attempts = 0 + + def __enter__(self): + return self + + def __exit__(self, *_): + pass + + def communicate(self): + self.attempts += 1 + if self.attempts < 3: + raise KeyboardInterrupt + return b"", b"" + + process = Process() + + def popen(arguments, **options): + assert options == {"stdin": subprocess.DEVNULL, "stdout": subprocess.PIPE, + "stderr": subprocess.PIPE, "start_new_session": False} + return process + + monkeypatch.setattr(_resources, "os", SimpleNamespace(name="nt", path=ntpath, environ={"SystemRoot": r"C:\Windows"})) + monkeypatch.setattr(_resources, "subprocess", SimpleNamespace( + Popen=popen, DEVNULL=subprocess.DEVNULL, PIPE=subprocess.PIPE, + run=lambda arguments, **options: calls.append((arguments, options)), + )) + with pytest.raises(KeyboardInterrupt): + _resources.run_office_process(["node.exe", "cli.js"]) + assert process.attempts == 3 + assert calls == [([r"C:\Windows\System32\taskkill.exe", "/PID", "73", "/T", "/F"], + {"stdin": subprocess.DEVNULL, "capture_output": True, "check": False})] diff --git a/python/sdk/tests/test_runtime_resolution.py b/python/sdk/tests/test_runtime_resolution.py index 37d60da483..d41d3282aa 100644 --- a/python/sdk/tests/test_runtime_resolution.py +++ b/python/sdk/tests/test_runtime_resolution.py @@ -332,24 +332,57 @@ def test_runtime_starts_independently_of_office_engine( assert runtime.bundled_runtime_path() == executable -@pytest.mark.parametrize("function,flag", [ - (runtime.download_office, "--download-office"), - (runtime.download_primary_runtime, "--download-primary-runtime"), +@pytest.mark.parametrize("function,resource", [ + (runtime.download_office, "office"), + (runtime.download_primary_runtime, "primary"), ]) def test_explicit_download_uses_runtime_without_a_harness_home( - tmp_path: Path, monkeypatch: pytest.MonkeyPatch, function, flag: str, + tmp_path: Path, monkeypatch: pytest.MonkeyPatch, function, resource: str, ) -> None: + root = tmp_path / "中文缓存" + root.mkdir() + _resource_sidecars(root / "deepseek-harness-sdk-runtime-linux-x64") + monkeypatch.setattr(runtime, "_current_platform_tag", lambda: "linux-x64") script = tmp_path / "download.py" - destination = tmp_path / "downloaded" - script.write_text("import json,sys\n" + f"assert sys.argv[1] == {flag!r}\nprint(json.dumps({str(destination)!r}))\n") + destination = root / "linux-x64/primary-runtime" if resource == "primary" else root + script.write_text("import json,os,sys\n" + f"assert os.environ['DSH_RUNTIME_DOWNLOAD'] == {resource!r}\n" + + f"sys.stdout.buffer.write(json.dumps({str(destination)!r}, ensure_ascii=False).encode('utf-8'))\n", encoding="utf-8") monkeypatch.delenv("DSH_HOME", raising=False) monkeypatch.setattr(runtime, "resolve_bundled_launch_args", lambda: (sys.executable, str(script))) assert function() == destination - script.write_text("import sys\nprint('checksum mismatch', file=sys.stderr)\nsys.exit(2)\n") - with pytest.raises(RuntimeError, match="checksum mismatch"): + script.write_text("import sys\nsys.stderr.buffer.write('下载失败'.encode('utf-8'))\nsys.exit(2)\n", encoding="utf-8") + with pytest.raises(RuntimeError, match="下载失败"): function() +@pytest.mark.skipif(os.name == "nt", reason="LC_ALL=C is a POSIX locale fixture") +def test_download_decodes_utf8_in_an_ascii_locale(tmp_path: Path) -> None: + script = tmp_path / "download.py" + destination = tmp_path / "中文缓存" + script.write_text("import sys\nsys.stdout.buffer.write(" + json.dumps(str(destination), ensure_ascii=False).encode('utf-8').__repr__() + ")\n", encoding="utf-8") + code = ("import sys,locale,deepseek_harness_runtime as r; " + "assert locale.getpreferredencoding(False).lower() not in ('utf-8','utf8'); " + f"r.resolve_bundled_launch_args=lambda:(sys.executable,{str(script)!r}); " + f"assert str(r.download_office())=={str(destination)!r}") + result = subprocess.run([sys.executable, "-c", code], capture_output=True, encoding="utf-8", env={ + **os.environ, "LC_ALL": "C", "PYTHONUTF8": "0", "PYTHONCOERCECLOCALE": "0", + "PYTHONPATH": str(Path(runtime.__file__).resolve().parents[1]), + }, timeout=30) + assert result.returncode == 0, result.stderr + + +@pytest.mark.parametrize("missing", ["primary-runtime/runtime.json", "primary-runtime/dependencies/python/bin/python3", + "primary-runtime/dependencies/node/bin/node", "office-skills/scripts/check_office.py"]) +def test_primary_download_rejects_an_incomplete_payload(tmp_path: Path, monkeypatch: pytest.MonkeyPatch, missing: str) -> None: + _resource_sidecars(tmp_path / "deepseek-harness-sdk-runtime-linux-x64") + root = tmp_path / "linux-x64" + (root / missing).unlink() + monkeypatch.setattr(runtime, "_download", lambda resource: root / "primary-runtime") + monkeypatch.setattr(runtime, "_current_platform_tag", lambda: "linux-x64") + with pytest.raises(FileNotFoundError): + runtime.download_primary_runtime() + + def test_office_resolution_requires_explicit_download(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: from deepseek_harness_runtime._resources import office_launch_args @@ -366,4 +399,32 @@ def test_office_resolution_requires_explicit_download(tmp_path: Path, monkeypatc for path in (node, cli): path.parent.mkdir(parents=True, exist_ok=True) path.touch() + with pytest.raises(FileNotFoundError, match="incomplete"): + office_launch_args(tmp_path / "linux-x64", "linux-x64") + (office / "complete").write_text("wrong", encoding="utf-8") + with pytest.raises(ValueError, match="identity mismatch"): + office_launch_args(tmp_path / "linux-x64", "linux-x64") + (office / "complete").write_text("a" * 64, encoding="utf-8") assert office_launch_args(tmp_path / "linux-x64", "linux-x64") == (str(node), str(cli)) + (office / "complete").unlink() + monkeypatch.setenv("DSH_OFFICE_SIDECAR", str(office)) + assert office_launch_args(tmp_path / "linux-x64", "linux-x64") == (str(node), str(cli)) + + +@pytest.mark.parametrize("invalid", ["target", "identity", "office", "pnpm", "skills"]) +def test_download_manifest_rejects_invalid_wheel_resources(tmp_path: Path, invalid: str) -> None: + from deepseek_harness_runtime._resources import validate_downloads + + _resource_sidecars(tmp_path / "deepseek-harness-sdk-runtime-linux-x64") + root = tmp_path / "linux-x64" + path = root / "downloads.json" + manifest = json.loads(path.read_text(encoding="utf-8")) + if invalid == "skills": + (root / "office-skills/scripts/check_office.py").unlink() + with pytest.raises(FileNotFoundError, match="Office skill"): + validate_downloads(root, "linux-x64") + else: + manifest[invalid] = {"target": "win-x64", "identity": "z" * 64, "office": [], "pnpm": []}[invalid] + path.write_text(json.dumps(manifest), encoding="utf-8") + with pytest.raises(ValueError, match="invalid runtime download manifest"): + validate_downloads(root, "linux-x64") diff --git a/scripts/build-exe-for-python-sdk-office.spec.ts b/scripts/build-exe-for-python-sdk-office.spec.ts index 50865f1a9f..64c61687ee 100644 --- a/scripts/build-exe-for-python-sdk-office.spec.ts +++ b/scripts/build-exe-for-python-sdk-office.spec.ts @@ -1,10 +1,12 @@ -import { chmod, mkdir, mkdtemp, readFile, rm, stat, writeFile } from 'node:fs/promises' +import { mkdir, mkdtemp, readFile, rm, writeFile } from 'node:fs/promises' import { tmpdir } from 'node:os' -import { dirname, join } from 'node:path' +import { join } from 'node:path' import { afterEach, expect, it } from 'vitest' import { createHash } from 'node:crypto' import { c as tar } from 'tar' -import { copyOfficeSidecar, downloadOfficeSidecar } from './build-exe-for-python-sdk-office.ts' +import { downloadOfficeSidecar, runtimeNpmArchive } from './build-exe-for-python-sdk-office.ts' + +import { officePackageDirectories } from './libreoffice-packages.mjs' const temporaryDirectories: string[] = [] @@ -26,33 +28,8 @@ async function fixture() { return { root, staging, destination, packageAt } } -it('copies package-owned data, licenses, helper permissions, and nested dependencies', async () => { - const { staging, destination, packageAt } = await fixture() - const entry = await packageAt('@deepseek-ai/libreoffice-kit', { optionalDependencies: { '@deepseek-ai/libreoffice-kit-wasm': '0.0.1' }, dependencies: { decoder: '1' } }) - const engine = await packageAt('@deepseek-ai/libreoffice-kit-wasm') - const decoder = await packageAt('decoder', { dependencies: { codec: '1' } }, entry) - await packageAt('codec', {}, decoder) - await packageAt('codec', { version: '2.0.0' }) - const assets = ['assets/soffice.data', 'prebuilds.json', 'licenses/LICENSE', 'bin/helper'] - for (const name of assets) { - const path = join(engine, name) - await mkdir(dirname(path), { recursive: true }) - await writeFile(path, name) - } - await chmod(join(engine, 'bin/helper'), 0o755) - - const packages = await copyOfficeSidecar(staging, destination, { platform: 'linux', arch: 'x64' }) - - expect(packages).toHaveLength(4) - expect(await readFile(join(destination, 'node_modules/@deepseek-ai/libreoffice-kit-wasm/assets/soffice.data'), 'utf8')).toBe('assets/soffice.data') - expect(await readFile(join(destination, 'node_modules/@deepseek-ai/libreoffice-kit-wasm/licenses/LICENSE'), 'utf8')).toBe('licenses/LICENSE') - expect(await readFile(join(destination, 'node_modules/@deepseek-ai/libreoffice-kit/node_modules/decoder/node_modules/codec/package.json'), 'utf8')).toContain('codec') - await expect(stat(join(destination, 'node_modules/codec'))).rejects.toMatchObject({ code: 'ENOENT' }) - if (process.platform !== 'win32') expect((await stat(join(destination, 'node_modules/@deepseek-ai/libreoffice-kit-wasm/bin/helper'))).mode & 0o111).toBe(0o111) -}) - -it('copies installed target optionals and leaves other platforms and absent optionals out', async () => { - const { staging, destination, packageAt } = await fixture() +it('locates installed target optionals and leaves other platforms and absent optionals out', async () => { + const { staging, packageAt } = await fixture() await packageAt('@deepseek-ai/libreoffice-kit', { optionalDependencies: { native: '1', foreign: '1', absent: '1' }, }) @@ -60,77 +37,74 @@ it('copies installed target optionals and leaves other platforms and absent opti await packageAt('native', { os: ['linux'], cpu: ['x64'] }) await packageAt('foreign', { os: ['darwin'], cpu: ['arm64'] }) - const packages = await copyOfficeSidecar(staging, destination, { platform: 'linux', arch: 'x64' }) + const packages = await officePackageDirectories(staging, { platform: 'linux', arch: 'x64' }) - expect(packages.map(path => path.replaceAll('\\', '/'))).toEqual([ + expect(packages.map(path => path.slice(staging.length + 1).replaceAll('\\', '/'))).toEqual([ 'node_modules/@deepseek-ai/libreoffice-kit', 'node_modules/@deepseek-ai/libreoffice-kit-wasm', 'node_modules/native', ]) }) -it('rejects a missing required dependency before producing a sidecar', async () => { - const { staging, destination, packageAt } = await fixture() +it('rejects a missing required dependency in the deployed closure', async () => { + const { staging, packageAt } = await fixture() await packageAt('@deepseek-ai/libreoffice-kit', { dependencies: { 'dsh-missing-office-fixture': '1' } }) - await expect(copyOfficeSidecar(staging, destination, { platform: 'linux', arch: 'x64' })) + await expect(officePackageDirectories(staging, { platform: 'linux', arch: 'x64' })) .rejects.toThrow('dsh-missing-office-fixture required by @deepseek-ai/libreoffice-kit is missing') - await expect(stat(destination)).rejects.toMatchObject({ code: 'ENOENT' }) }) it('rejects an incomplete installed optional instead of omitting it', async () => { - const { staging, destination, packageAt } = await fixture() + const { staging, packageAt } = await fixture() await packageAt('@deepseek-ai/libreoffice-kit', { optionalDependencies: { broken: '1' } }) await mkdir(join(staging, 'node_modules/broken'), { recursive: true }) - await expect(copyOfficeSidecar(staging, destination, { platform: 'linux', arch: 'x64' })) + await expect(officePackageDirectories(staging, { platform: 'linux', arch: 'x64' })) .rejects.toMatchObject({ code: 'ENOENT' }) }) it('rejects an ancestor dependency outside the deployed closure', async () => { - const { root, staging, destination, packageAt } = await fixture() + const { root, staging, packageAt } = await fixture() await packageAt('@deepseek-ai/libreoffice-kit', { dependencies: { 'ancestor-office-fixture': '1' } }) await packageAt('ancestor-office-fixture', {}, root) - await expect(copyOfficeSidecar(staging, destination, { platform: 'linux', arch: 'x64' })) + await expect(officePackageDirectories(staging, { platform: 'linux', arch: 'x64' })) .rejects.toThrow('outside the deployed closure') }) it.each([true, false])('requires a staged WASM engine when declared=%s even when an ancestor has one', async (declared) => { - const { root, staging, destination, packageAt } = await fixture() + const { root, staging, packageAt } = await fixture() await packageAt('@deepseek-ai/libreoffice-kit', declared ? { optionalDependencies: { '@deepseek-ai/libreoffice-kit-wasm': '0.0.1' } } : {}) await packageAt('@deepseek-ai/libreoffice-kit-wasm', {}, root) - await expect(copyOfficeSidecar(staging, destination, { platform: 'linux', arch: 'x64' })) + await expect(officePackageDirectories(staging, { platform: 'linux', arch: 'x64' })) .rejects.toThrow('Office engine @deepseek-ai/libreoffice-kit-wasm required for linux/x64 is missing.') - await expect(stat(destination)).rejects.toMatchObject({ code: 'ENOENT' }) }) it.each([ ['darwin', 'arm64', 'darwin-arm64'], ['darwin', 'x64', 'darwin-x64'], ['win32', 'arm64', 'win32-arm64'], ['win32', 'x64', 'win32-x64'], ['linux', 'x64', 'linux-x64'], ['linux', 'arm64', 'wasm'], ['freebsd', 'x64', 'wasm'], -])('copies only the %s/%s engine even when other engines are staged', async (platform, arch, selected) => { - const { staging, destination, packageAt } = await fixture() +])('locates only the %s/%s engine even when other engines are staged', async (platform, arch, selected) => { + const { staging, packageAt } = await fixture() const targets = ['darwin-arm64', 'darwin-x64', 'win32-arm64', 'win32-x64', 'linux-x64', 'wasm'] const names = targets.map(target => `@deepseek-ai/libreoffice-kit-${target}`) await packageAt('@deepseek-ai/libreoffice-kit', { optionalDependencies: Object.fromEntries(names.map(name => [name, '0.0.1'])) }) for (const name of names) await packageAt(name) const expected = `@deepseek-ai/libreoffice-kit-${selected}` - const packages = await copyOfficeSidecar(staging, destination, { platform, arch }) - expect(packages.map(path => path.replaceAll('\\', '/'))).toEqual([ + const packages = await officePackageDirectories(staging, { platform, arch }) + expect(packages.map(path => path.slice(staging.length + 1).replaceAll('\\', '/'))).toEqual([ 'node_modules/@deepseek-ai/libreoffice-kit', `node_modules/${expected}`, ]) }) it.each(['darwin', 'win32', 'linux'])('%s requires its native package even when WASM is staged', async (platform) => { - const { staging, destination, packageAt } = await fixture() + const { staging, packageAt } = await fixture() await packageAt('@deepseek-ai/libreoffice-kit', { optionalDependencies: { '@deepseek-ai/libreoffice-kit-wasm': '0.0.1', [`@deepseek-ai/libreoffice-kit-${platform}-arm64`]: '0.0.1', } }) await packageAt('@deepseek-ai/libreoffice-kit-wasm') - await expect(copyOfficeSidecar(staging, destination, { platform, arch: 'arm64' })) + await expect(officePackageDirectories(staging, { platform, arch: 'arm64' })) .rejects.toThrow(`Office engine @deepseek-ai/libreoffice-kit-${platform}-arm64 required for ${platform}/arm64 is missing.`) - await expect(stat(destination)).rejects.toMatchObject({ code: 'ENOENT' }) }) it('downloads pinned npm packages into the sidecar without resolving or executing package scripts', async () => { @@ -157,3 +131,18 @@ it('downloads pinned npm packages into the sidecar without resolving or executin await writeFile(join(cache, checksum.toString('hex')), 'corrupted archive') await expect(downloadOfficeSidecar([artifact], destination, cache)).rejects.toThrow('checksum mismatch') }) + + +it('takes npm integrity from the workspace lock without querying registry metadata', async () => { + const { root } = await fixture() + const lockfile = join(root, 'pnpm-lock.yaml') + const integrity = `sha512-${Buffer.alloc(64, 7).toString('base64')}` + await writeFile(lockfile, JSON.stringify({ packages: { '@scope/fixture@1.2.3': { resolution: { integrity } } } })) + expect(await runtimeNpmArchive('@scope/fixture', '1.2.3', lockfile)).toEqual({ + name: '@scope/fixture', version: '1.2.3', integrity, + url: 'https://registry.npmjs.org/@scope/fixture/-/fixture-1.2.3.tgz', + }) + await expect(runtimeNpmArchive('@scope/fixture', '1.2.4', lockfile)).rejects.toThrow('missing SHA-512 integrity') + await writeFile(lockfile, JSON.stringify({ packages: { '@scope/fixture@1.2.3': { resolution: { integrity: 'sha256-invalid' } } } })) + await expect(runtimeNpmArchive('@scope/fixture', '1.2.3', lockfile)).rejects.toThrow('missing SHA-512 integrity') +}) diff --git a/scripts/build-exe-for-python-sdk-office.ts b/scripts/build-exe-for-python-sdk-office.ts index ac3bff120d..f1ffe4bd63 100644 --- a/scripts/build-exe-for-python-sdk-office.ts +++ b/scripts/build-exe-for-python-sdk-office.ts @@ -1,7 +1,8 @@ /** Keep LibreOffice workers, prebuilt engines, and their dependencies on the real filesystem. */ -import { cp, mkdir, readFile, rm } from 'node:fs/promises' +import { readFile } from 'node:fs/promises' import { isAbsolute, join, relative, sep } from 'node:path' import { downloadRuntimeNpmPackage, type RuntimeNpmArchive } from './primary-runtime/prepare.ts' +import { load as parseYaml } from 'js-yaml' import { officePackageDirectories } from './libreoffice-packages.mjs' /** pkg applies these exclusions to dependency `files` as well as root asset globs. */ @@ -10,33 +11,6 @@ export const OFFICE_ASSET_IGNORES = [ '**/node_modules/@deepseek-ai/libreoffice-kit-*/**', ] -/** - * Copy the installed Office dependency tree without changing package contents or executable modes. - * Harness sidecars require the kit's declared target native engine, or WASM for other targets. - * Missing target engines fail with their package name; missing required dependencies or paths outside the deployed closure also fail. - * @param staging - Symlink-free deployed Node closure. - * @param destination - Target-specific Office directory beside the executable; replaced when present. - * @param target - Node platform and CPU of the executable. - * @returns Relative package directories included in the sidecar. - */ -export async function copyOfficeSidecar( - staging: string, - destination: string, - target: { platform: string; arch: string }, -): Promise { - const directories = await officePackageDirectories(staging, target) - await rm(destination, { recursive: true, force: true }) - await mkdir(destination, { recursive: true }) - for (const source of directories) { - const nestedModules = join(source, 'node_modules') - await cp(source, join(destination, relative(staging, source)), { - recursive: true, - filter: path => path !== nestedModules && !path.startsWith(nestedModules + sep), - }) - } - return directories.map(directory => relative(staging, directory)) -} - /** One npm package at its existing sidecar dependency location. */ export interface OfficePackageArchive extends RuntimeNpmArchive { /** Relative package directory inside the sidecar. */ @@ -44,19 +18,19 @@ export interface OfficePackageArchive extends RuntimeNpmArchive { } /** - * Read exact npm tarball metadata for a version already selected by the workspace lock. + * Resolve one installed npm version against its reviewed workspace lock integrity. * @param name - Published package name. * @param version - Exact installed version. - * @returns Immutable npm tarball identity. + * @param lockfile - Workspace lock recording the exact published archive. + * @returns Locked npm tarball identity; rejects versions without SHA-512 integrity. */ -export async function runtimeNpmArchive(name: string, version: string): Promise { - const response = await fetch(`https://registry.npmjs.org/${encodeURIComponent(name)}/${encodeURIComponent(version)}`) - if (!response.ok) throw new Error(`runtime npm metadata: HTTP ${String(response.status)} for ${name}@${version}`) - const metadata = await response.json() as { name: string; version: string; dist: { tarball: string; integrity: string } } - if (metadata.name !== name || metadata.version !== version || !metadata.dist.integrity.startsWith('sha512-')) { - throw new Error(`runtime npm metadata: invalid identity for ${name}@${version}`) +export async function runtimeNpmArchive(name: string, version: string, lockfile = join(import.meta.dirname, '../pnpm-lock.yaml')): Promise { + const lock = parseYaml(await readFile(lockfile, 'utf8')) as { packages?: Record } + const integrity = lock.packages?.[`${name}@${version}`]?.resolution?.integrity + if (typeof integrity !== 'string' || !/^sha512-[A-Za-z0-9+/]+={0,2}$/u.test(integrity)) { + throw new Error(`runtime npm lock: missing SHA-512 integrity for ${name}@${version}`) } - return { name, version, url: metadata.dist.tarball, integrity: metadata.dist.integrity } + return { name, version, url: `https://registry.npmjs.org/${name}/-/${name.split('/').at(-1)}-${version}.tgz`, integrity } } /** diff --git a/scripts/build-exe-for-python-sdk.ts b/scripts/build-exe-for-python-sdk.ts index 3534c0f1e3..d61453fa76 100644 --- a/scripts/build-exe-for-python-sdk.ts +++ b/scripts/build-exe-for-python-sdk.ts @@ -342,6 +342,8 @@ class SingleExeBuild { const patch = { bin: ENTRY_BIN, pkg: { assets: ASSET_GLOBS, ignore: [ ...OFFICE_ASSET_IGNORES, `${sourceModules}/**`, `${sourceModules}/.pnpm/**`, + ...['packages', 'apps', 'vendor', 'native', 'scripts'].map(directory => `${join(root, directory).replaceAll('\\', '/')}/**`), + `${root.replaceAll('\\', '/')}/tsconfig*.json`, '**/*.map', '**/*.d.ts', '**/*.d.mts', '**/*.d.cts', ] } } const manifestPath = join(this.staging, 'package.json') @@ -355,11 +357,24 @@ class SingleExeBuild { if (!existsSync(join(this.staging, ENTRY_BIN))) { throw new Error(`build-exe-for-python-sdk: staged bootstrap ${join(this.staging, ENTRY_BIN)} is missing.`) } - await bundle({ config: false, tsconfig: false, inputOptions: { tsconfig: false }, target: 'es2024', entry: { 'primary-runtime': join(root, 'scripts/primary-runtime/prepare.ts'), - 'office-sidecar': join(root, 'scripts/build-exe-for-python-sdk-office.ts') }, - outDir: this.staging, clean: false, format: 'esm', platform: 'node', dts: false, - deps: { neverBundle: [/^@deepseek-ai\//u], alwaysBundle: [/.*/u] }, shims: true, outExtensions: () => ({ js: '.mjs' }), - define: { 'import.meta.main': 'false' } }) + const helpers = [ + { name: 'primary-runtime', source: 'scripts/primary-runtime/prepare.ts', exports: 'downloadNodeRuntime, preparePrimaryRuntime' }, + { name: 'office-sidecar', source: 'scripts/build-exe-for-python-sdk-office.ts', exports: 'downloadOfficeSidecar' }, + ] + const entries: Record = {} + for (const helper of helpers) { + const entry = join(this.staging, `.${helper.name}-entry.ts`) + entries[helper.name] = entry + await writeFile(entry, `export { ${helper.exports} } from ${JSON.stringify(join(root, helper.source).replaceAll('\\', '/'))}\n`) + } + try { + await bundle({ config: false, tsconfig: false, inputOptions: { tsconfig: false }, target: 'es2024', entry: entries, + outDir: this.staging, clean: false, format: 'esm', platform: 'node', dts: false, + deps: { neverBundle: [/^@deepseek-ai\//u], alwaysBundle: [/.*/u] }, shims: true, outExtensions: () => ({ js: '.mjs' }), + define: { 'import.meta.main': 'false' } }) + } finally { + for (const entry of Object.values(entries)) await rm(entry, { force: true }) + } const manifest = JSON.parse(await readFile(manifestPath, 'utf8')) as Record await writeFile(manifestPath, `${JSON.stringify({ ...manifest, ...patch }, null, 2)}\n`) await this.run('resource helper imports', process.execPath, ['--input-type=module', '--eval', diff --git a/scripts/primary-runtime/prepare.spec.ts b/scripts/primary-runtime/prepare.spec.ts index 6100bc2b5b..11b5b7a6a8 100644 --- a/scripts/primary-runtime/prepare.spec.ts +++ b/scripts/primary-runtime/prepare.spec.ts @@ -1,6 +1,6 @@ import { readFileSync } from 'node:fs' import { createHash } from 'node:crypto' -import { mkdir, mkdtemp, readFile, rm, writeFile } from 'node:fs/promises' +import { mkdir, mkdtemp, readFile, rm, stat, writeFile } from 'node:fs/promises' import { tmpdir } from 'node:os' import { basename, join } from 'node:path' import { zipSync } from 'fflate' @@ -61,7 +61,11 @@ it('extracts a hash-verified cached library without a Python installer or networ const hash = createHash('sha256').update(libraryWheel).digest('hex') const archive = join(root, hash) await writeFile(archive, libraryWheel) + const original = await stat(archive) expect(await downloadPrimaryRuntimeAsset('https://unused.invalid/library.whl', hash, root)).toBe(archive) + const reused = await stat(archive) + expect(reused.ino).toBe(original.ino) + expect(reused.mtimeMs).toBe(original.mtimeMs) await unpackPrimaryRuntimeWheel(archive, join(root, 'site-packages')) expect(await readFile(join(root, 'site-packages/sample.py'), 'utf8')).toBe('sample = 42\n') await writeFile(archive, 'corrupt archive') diff --git a/scripts/primary-runtime/prepare.ts b/scripts/primary-runtime/prepare.ts index 1b290fb909..b42158bb7e 100644 --- a/scripts/primary-runtime/prepare.ts +++ b/scripts/primary-runtime/prepare.ts @@ -25,8 +25,10 @@ export async function downloadPrimaryRuntimeAsset(url: string, sha256: string, c mkdirSync(cache, { recursive: true }) const destination = join(cache, sha256) let bytes: Buffer + let cached = true try { bytes = readFileSync(destination) } catch (error) { if ((error as NodeJS.ErrnoException).code !== 'ENOENT') throw error + cached = false const timeout = Number(process.env.DSH_RESOURCE_DOWNLOAD_TIMEOUT_MS ?? '300000') if (!Number.isSafeInteger(timeout) || timeout < 1 || timeout > 2_147_483_647) throw new Error('DSH_RESOURCE_DOWNLOAD_TIMEOUT_MS must be an integer from 1 to 2147483647') const response = await fetch(url, { signal: AbortSignal.timeout(timeout) }) @@ -37,6 +39,7 @@ export async function downloadPrimaryRuntimeAsset(url: string, sha256: string, c rmSync(destination, { force: true }) throw new Error(`primary runtime download: checksum mismatch for ${url}`) } + if (cached) return destination const temporary = join(cache, `.${randomUUID()}`) try { writeFileSync(temporary, bytes) diff --git a/scripts/python-runtime-download.spec.ts b/scripts/python-runtime-download.spec.ts index 26db34943e..7d3b1bcfa4 100644 --- a/scripts/python-runtime-download.spec.ts +++ b/scripts/python-runtime-download.spec.ts @@ -23,15 +23,17 @@ async function fixture() { await file(join(root, 'node/node_modules/@deepseek-ai/dsh-sandbox-windows-acl/package.json'), '{"type":"module","exports":{"./runner":"./runner.mjs"}}') await file(join(root, 'node/node_modules/@deepseek-ai/dsh-sandbox-windows-acl/runner.mjs'), '') await file(join(root, 'node/node_modules/@deepseek-ai/dsh/package.json'), '{"type":"module"}') - await file(join(root, 'node/node_modules/@deepseek-ai/dsh/lib/bin.js'), 'export async function runCli() { console.log(JSON.stringify({ primary: process.env.DSH_BUNDLED_PRIMARY_RUNTIME, office: process.env.DSH_BUNDLED_OFFICE_CLI })) }') + await file(join(root, 'node/node_modules/@deepseek-ai/dsh/lib/bin.js'), 'export async function runCli() { if (process.argv[2]) { console.log(JSON.stringify({ argv: process.argv.slice(2) })); return } console.log(JSON.stringify({ primary: process.env.DSH_BUNDLED_PRIMARY_RUNTIME, office: process.env.DSH_BUNDLED_OFFICE_CLI })) }') await file(join(root, 'node/primary-runtime.mjs'), ` import { mkdir, writeFile } from 'node:fs/promises'; import { join } from 'node:path'; export async function downloadNodeRuntime(target, destination) { + if (process.env.DSH_RUNTIME_DOWNLOAD !== undefined) throw new Error("private selector leaked"); await mkdir(join(destination, 'bin'), { recursive: true }); await writeFile(join(destination, 'bin', process.platform === 'win32' ? 'node.exe' : 'node'), 'node'); } export async function preparePrimaryRuntime({ output }) { + if (process.env.DSH_RUNTIME_DOWNLOAD !== undefined) throw new Error("private selector leaked"); await mkdir(join(output, 'primary-runtime'), { recursive: true }); await writeFile(join(output, 'primary-runtime/runtime.json'), '{}'); } @@ -51,9 +53,9 @@ async function fixture() { } } `) - function run(flag?: string, extra: NodeJS.ProcessEnv = {}) { - const child = spawn(process.execPath, [bootstrap, ...flag === undefined ? [] : [flag]], { - env: { ...process.env, DSH_RESOURCE_CACHE: cache, DSH_BUNDLED_PRIMARY_RUNTIME: undefined, + function run(resource?: string, extra: NodeJS.ProcessEnv = {}, args: string[] = []) { + const child = spawn(process.execPath, [bootstrap, ...args], { + env: { ...process.env, DSH_RESOURCE_CACHE: cache, DSH_RUNTIME_DOWNLOAD: resource, DSH_BUNDLED_PRIMARY_RUNTIME: undefined, DSH_OFFICE_SIDECAR: undefined, DSH_BUNDLED_OFFICE_CLI: undefined, ...extra }, stdio: ['pipe', 'pipe', 'pipe'], }) @@ -78,32 +80,32 @@ it('starts without downloading, independently downloads either resource, and reu try { expect(await run().done).toMatchObject({ code: 0, stdout: '{}\n' }) await expect(readdir(cache)).rejects.toMatchObject({ code: 'ENOENT' }) - const primary = await run('--download-primary-runtime').done + const primary = await run('primary').done expect(primary.code, primary.stderr).toBe(0) expect(JSON.parse(primary.stdout)).toBe(join(cache, 'a'.repeat(64), 'primary', 'primary-runtime')) await expect(readdir(join(cache, 'a'.repeat(64), 'office'))).rejects.toMatchObject({ code: 'ENOENT' }) - const office = await run('--download-office').done + const office = await run('office').done expect(office.code, office.stderr).toBe(0) const location = JSON.parse(office.stdout) as string expect(await readFile(join(location, 'node_modules/@deepseek-ai/libreoffice-kit/lib/cli.js'), 'utf8')).toBe('cli') - expect(await run('--download-office', { DSH_FIXTURE_FAIL: '1' }).done).toMatchObject({ code: 0, stdout: office.stdout }) + expect(await run('office', { DSH_FIXTURE_FAIL: '1' }).done).toMatchObject({ code: 0, stdout: office.stdout }) } finally { await rm(root, { recursive: true, force: true }) } }) it('cleans failed staging and allows an explicit retry', async () => { const { root, cache, run } = await fixture() try { - const failed = await run('--download-office', { DSH_FIXTURE_FAIL: '1' }).done + const failed = await run('office', { DSH_FIXTURE_FAIL: '1' }).done expect(failed.code).toBe(1) expect(failed.stderr).toContain('fixture download failed') expect(await readdir(join(cache, 'a'.repeat(64)))).toEqual([]) - expect((await run('--download-office').done).code).toBe(0) + expect((await run('office').done).code).toBe(0) } finally { await rm(root, { recursive: true, force: true }) } }) it('publishes one complete cache when two processes download concurrently', async () => { const { root, cache, run } = await fixture() - const jobs = [run('--download-office', { DSH_FIXTURE_BARRIER: '1' }), run('--download-office', { DSH_FIXTURE_BARRIER: '1' })] + const jobs = [run('office', { DSH_FIXTURE_BARRIER: '1' }), run('office', { DSH_FIXTURE_BARRIER: '1' })] try { await Promise.all(jobs.map(job => job.ready)) for (const job of jobs) job.child.stdin.end('continue') @@ -117,3 +119,17 @@ it('publishes one complete cache when two processes download concurrently', asyn await rm(root, { recursive: true, force: true }) } }) + + +it('leaves ordinary CLI arguments to the CLI and rejects an invalid private download selector', async () => { + const { root, cache, run } = await fixture() + try { + expect(await run(undefined, {}, ['--download-office']).done).toMatchObject({ + code: 0, stdout: JSON.stringify({ argv: ['--download-office'] }) + '\n', + }) + const rejected = await run('invalid').done + expect(rejected.code).toBe(1) + expect(rejected.stderr).toContain('DSH_RUNTIME_DOWNLOAD must select office or primary') + await expect(readdir(cache)).rejects.toMatchObject({ code: 'ENOENT' }) + } finally { await rm(root, { recursive: true, force: true }) } +}) diff --git a/scripts/smoke-python-runtime.py b/scripts/smoke-python-runtime.py index 7515596bdf..417edb67d2 100644 --- a/scripts/smoke-python-runtime.py +++ b/scripts/smoke-python-runtime.py @@ -884,9 +884,16 @@ def main() -> None: def smoke_sdk_authoring(base_url: str, executable: Path, update_snapshots: bool) -> None: """Query the bundled Python and switch skills without replacing that environment.""" + + with tempfile.TemporaryDirectory(prefix="dsh-sdk-authoring-cache-") as cache_directory: + _smoke_sdk_authoring_cached(base_url, executable, update_snapshots, Path(cache_directory)) + + +def _smoke_sdk_authoring_cached(base_url: str, executable: Path, update_snapshots: bool, cache: Path) -> None: from deepseek_harness import DeepSeekHarness - primary = Path(json.loads(subprocess.check_output([str(executable), "--download-primary-runtime"], text=True))) + cache_environment = {**os.environ, "DSH_RESOURCE_CACHE": str(cache), "DSH_RUNTIME_DOWNLOAD": "primary"} + primary = Path(json.loads(subprocess.check_output([str(executable)], encoding="utf-8", env=cache_environment))) resources = primary.parent manifest = json.loads((primary / "runtime.json").read_text()) for mode in ("default", "replacement", "disabled"): @@ -904,7 +911,7 @@ def smoke_sdk_authoring(base_url: str, executable: Path, update_snapshots: bool) provider="deepseek-official", model="smoke-model", cwd=str(root), dsh_bin=str(executable), dsh_home=str(home), patches=(str(patch),), api_key="sk-keyless-smoke", base_url=base_url, - env={"DSH_PERMISSION_MODE": "danger-full-access", "DSH_TELEMETRY_DISABLED": "1"}, + env={"DSH_PERMISSION_MODE": "danger-full-access", "DSH_TELEMETRY_DISABLED": "1", "DSH_RESOURCE_CACHE": str(cache)}, request_timeout_seconds=60, ) as harness: result = harness.run(AUTHORING_PROMPT, session_id="authoring") @@ -960,7 +967,7 @@ def smoke_sdk_office(executable: Path) -> None: else: shutil.copy2(source, destination) cache_environment = {**os.environ, "DSH_RESOURCE_CACHE": str(root / "cache")} - office = Path(json.loads(subprocess.check_output([str(relocated), "--download-office"], text=True, env=cache_environment))) + office = Path(json.loads(subprocess.check_output([str(relocated)], encoding="utf-8", env={**cache_environment, "DSH_RUNTIME_DOWNLOAD": "office"}))) assert not (office.parent / "primary").exists(), "Office download unexpectedly prepared the authoring environment" adapter = office / "node_modules/@deepseek-ai/libreoffice-kit/package.json" native = stem.removeprefix("deepseek-harness-sdk-runtime-").replace("win-", "win32-").replace("macos-", "darwin-") @@ -1034,7 +1041,7 @@ recalculate(root / 'input.xlsx', root / 'calculated.xlsx') with zipfile.ZipFile(root / 'calculated.xlsx') as archive: sheet = ElementTree.fromstring(archive.read('xl/worksheets/sheet1.xml')) assert sheet.find('.//{*}c[@r="A1"]/{*}v').text == '3' -""", str(root), expected_backend], env={**cache_environment, "DSH_OFFICE_SIDECAR": str(office)}, check=True, timeout=180) +""", str(root), expected_backend], env={name: value for name, value in cache_environment.items() if name != "DSH_OFFICE_SIDECAR"}, check=True, timeout=180) assert not (office.parent / "primary").exists(), "Python Office API unexpectedly downloaded authoring resources" print(f"smoke-python-runtime: relocated Office {result['backend']} DOCX produced {len(pdf)} PDF bytes")