diff --git a/apps/desktop/README.i18n.yaml b/apps/desktop/README.i18n.yaml index f412464ae6..844f45a374 100644 --- a/apps/desktop/README.i18n.yaml +++ b/apps/desktop/README.i18n.yaml @@ -15,8 +15,8 @@ en: 069fdfd72d2ea26a zh: 76ae549d43c99bb4 /deepseek-harness-desktop/key-technical-decisions/bundled-workspace-dependencies: - en: 320733f93ede61a4 - zh: 33206c3c4753c802 + en: f9405b181784a24c + zh: 678483f4626112bf /deepseek-harness-desktop/bundled-command-runtime: en: beecf076aa2f0796 zh: c9001ce0f22d2379 diff --git a/apps/desktop/README.md b/apps/desktop/README.md index fa00c27e08..07950f4409 100644 --- a/apps/desktop/README.md +++ b/apps/desktop/README.md @@ -70,6 +70,8 @@ The private Desktop `runtime/bin` directory is added only to package-installatio Node prepares the bundled interpreters and Python libraries without a system Python or pip. [The download lock](../../scripts/primary-runtime/lock.json) pins interpreter archives, Python distribution versions and target-specific wheel URLs and hashes; the shared builder resolves pnpm from the root development dependency pin. The root package-manager version and the Desktop pin are checked for agreement. Wheel filenames and distribution versions must agree for every target. Preserve key order within the selected target, wheel records and distribution map, plus wheel-entry order; these affect payload identity, while top-level lock key order does not. Library wheels unpack into site-packages, retaining auxiliary files under each wheel's `.data/scripts` directory without generating command wrappers. Other installation schemes are rejected. Native-target checks verify the locked wheel set and versions, permit the interpreter's bundled pip, and check the Python version, Office document read/write operations and dependency completeness without writing bytecode after staging cleanup and again after macOS signing. The standalone Node executable receives the JIT entitlement required by V8; the x64 target also receives `com.apple.security.cs.allow-unsigned-executable-memory` for executable code pages, including under Rosetta. ARM64 Node and Office helpers retain JIT-only permissions. Cross-target execution and signed installation require the target release host. Both `dev:desktop` and `start:desktop` prepare `.desktop-build/targets//runtime/primary-runtime` before launching Electron; first use may download locked dependencies. An unfinished preparation cannot report a successful launcher exit. +The shared Desktop/SDK runtime builder omits directories named `tests` beneath NumPy and pandas before copying or signing the payload, and removes their rows from each distribution’s `RECORD`. It retains `numpy.testing`, `pandas.testing`, `pandas._testing`, native extensions and licenses. Complete upstream suites are unavailable; native checks verify the retained testing APIs, absence of test directories and existence of recorded files. + | Decision | Why | Direct consequence | |---|---|---| | Release identity | The shell API, Web client, backend, and plugin graph are qualified as one combination; independent versions would create untested combinations and ambiguous update availability. | Electron and `@deepseek-ai/dsh` always have the same exact version. A dsh upgrade is a Desktop release, even when the shell code is unchanged. | diff --git a/apps/desktop/README.zh.md b/apps/desktop/README.zh.md index 61caaf038d..eb5157866c 100644 --- a/apps/desktop/README.zh.md +++ b/apps/desktop/README.zh.md @@ -72,6 +72,8 @@ Desktop 私有的 `runtime/bin` 目录仅添加到包安装进程，不进入 PT Node 准备内置解释器和 Python 库，无需系统 Python 或 pip。[下载锁](../../scripts/primary-runtime/lock.json)固定解释器压缩包、Python 分发包版本及目标平台 wheel 的 URL 和哈希；共享构建器从根开发依赖中解析 pnpm 固定版本。测试检查根 package-manager 版本和 Desktop 固定版本保持一致。每个目标的 wheel 文件名必须与分发包版本一致。所选目标、wheel 记录及分发包映射内部的键顺序，以及 wheel 条目顺序都会影响产物身份，编辑时须保留；锁文件顶层键的顺序不影响该身份。库 wheel 解压到 site-packages，各 wheel 的 `.data/scripts` 目录保留辅助文件，不生成命令行包装器。其他安装方案会被拒绝。本机目标检查在清理暂存目录后以及 macOS 签名后验证锁定 wheel 的集合与版本，允许解释器自带的 pip，并检查 Python 版本、Office 文档读写和依赖完整性，不写入字节码。独立 Node 可执行文件获得 V8 所需的 JIT 权限；x64 目标还获得 `com.apple.security.cs.allow-unsigned-executable-memory`，允许可执行代码页，包括在 Rosetta 下运行时。ARM64 Node 和 Office 辅助程序保留仅 JIT 权限。跨目标执行和签名安装需要对应的发布主机。`dev:desktop` 和 `start:desktop` 都会在启动 Electron 前准备 `.desktop-build/targets//runtime/primary-runtime`；首次准备可能需要下载锁定的依赖。准备未完成时，启动命令不能报告成功退出。 +Desktop/SDK 共享运行时构建器在复制或签名产物前，裁剪 NumPy 和 pandas 下名为 `tests` 的目录，并移除各分发包 `RECORD` 中对应的记录。它保留 `numpy.testing`、`pandas.testing`、`pandas._testing`、原生扩展和许可证。完整上游测试套件不可用；本机检查验证保留的 testing API、测试目录已移除，以及清单所列文件存在。 + | 决策 | 原因 | 直接结果 | |---|---|---| | 发布身份 | 桌面壳 API、Web 客户端、后端与插件依赖图作为一个组合完成验证；独立版本会产生未经验证的组合，并让更新可用性含糊不清。 | Electron 与 `@deepseek-ai/dsh` 始终使用同一精确版本。即使桌面壳代码不变，升级 dsh 也必须发布新 Desktop 版本。 | diff --git a/docs/upgrade-guide/v0.2.0-rc.2/bundled-python-tests/guide.i18n.yaml b/docs/upgrade-guide/v0.2.0-rc.2/bundled-python-tests/guide.i18n.yaml new file mode 100644 index 0000000000..50af2cc791 --- /dev/null +++ b/docs/upgrade-guide/v0.2.0-rc.2/bundled-python-tests/guide.i18n.yaml @@ -0,0 +1,16 @@ +# Bilingual-pair consistency record for guide.md (docs/i18n/README.md): per heading +# section, a hash of its English and Chinese blocks outside code blocks and generated regions. +# After editing either side, bring the other along and re-record with: +# pnpm run verify-translation-pairing --write docs/upgrade-guide/v0.2.0-rc.2/bundled-python-tests/guide.md +/: + en: 985656162a55b14a + zh: c32bda15938879e5 +/numpy-and-pandas-upstream-tests-require-a-separate-environment: + en: ec0ae29022f8ce8c + zh: f5e49682b842544d +/numpy-and-pandas-upstream-tests-require-a-separate-environment/change: + en: 41410ee875be7975 + zh: b052e7b3c8fb096b +/numpy-and-pandas-upstream-tests-require-a-separate-environment/migration: + en: d0602fe057e3270a + zh: 1556ec03855019a0 diff --git a/docs/upgrade-guide/v0.2.0-rc.2/bundled-python-tests/guide.md b/docs/upgrade-guide/v0.2.0-rc.2/bundled-python-tests/guide.md new file mode 100644 index 0000000000..64de05c4ec --- /dev/null +++ b/docs/upgrade-guide/v0.2.0-rc.2/bundled-python-tests/guide.md @@ -0,0 +1,17 @@ +--- +kind: upgrade-guide +description: "Bundled NumPy and pandas omit their upstream test suites." +--- + +# NumPy and pandas upstream tests require a separate environment + +English | [中文](guide.zh.md) + +## Change + +Desktop and Python SDK runtime distributions previously included NumPy and pandas test directories. Bundled distributions now omit those directories, so scripts importing their test modules or running their complete upstream suites need a separate installation. Data processing, Office authoring, `numpy.testing`, `pandas.testing`, and `pandas._testing` remain available. + +## Migration + +1. If your scripts import `numpy.*.tests` or `pandas.tests`, install the original NumPy and pandas distributions and their upstream testing dependencies in a separate Python environment. Keep that environment outside the managed Harness runtime directory, which upgrades replace. +2. Run the affected scripts or upstream suites with that environment's interpreter. Scripts that use only library functionality or retained testing helpers require no changes. diff --git a/docs/upgrade-guide/v0.2.0-rc.2/bundled-python-tests/guide.zh.md b/docs/upgrade-guide/v0.2.0-rc.2/bundled-python-tests/guide.zh.md new file mode 100644 index 0000000000..22eb6db9c9 --- /dev/null +++ b/docs/upgrade-guide/v0.2.0-rc.2/bundled-python-tests/guide.zh.md @@ -0,0 +1,17 @@ +--- +kind: upgrade-guide +description: "内置 NumPy 和 pandas 不包含其上游测试套件。" +--- + +# NumPy 和 pandas 上游测试需要独立环境 + +[English](guide.md) | 中文 + +## 变更 + +Desktop 和 Python SDK 运行时分发包此前包含 NumPy 和 pandas 测试目录。内置分发包现在裁剪这些目录，因此导入其测试模块或运行其完整上游测试套件的脚本需要独立安装环境。数据处理、Office 创作、`numpy.testing`、`pandas.testing` 和 `pandas._testing` 仍然可用。 + +## 迁移 + +1. 如果脚本导入 `numpy.*.tests` 或 `pandas.tests`，请在独立 Python 环境中安装原始 NumPy 和 pandas 分发包及其上游测试依赖。请将该环境放在 Harness 托管运行时目录之外，升级会替换托管目录。 +2. 使用该环境的解释器运行受影响的脚本或上游测试套件。仅使用库功能或保留的 testing 辅助 API 的脚本无需修改。 diff --git a/package.json b/package.json index f5ce66dc9e..1d5c8fe130 100644 --- a/package.json +++ b/package.json @@ -228,6 +228,7 @@ "@types/jsdom": "^28.0.3", "@types/mdast": "^4.0.4", "@types/node": "^22.20.0", + "@types/papaparse": "^5.5.2", "@types/spdx-expression-parse": "^4.0.0", "@vitest/coverage-v8": "^4.1.8", "@yao-pkg/pkg": "6.21.0", @@ -249,6 +250,7 @@ "micromark-extension-gfm": "^3.0.0", "oxlint": "1.76.0", "oxlint-tsgolint": "7.0.2001", + "papaparse": "5.5.3", "pnpm": "11.7.0", "publint": "^0.3.21", "smol-toml": "^1.7.1", diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index da91191c1a..055ddf94ca 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -88,6 +88,9 @@ importers: '@types/node': specifier: ^22.20.0 version: 22.20.0 + '@types/papaparse': + specifier: ^5.5.2 + version: 5.5.2 '@types/spdx-expression-parse': specifier: ^4.0.0 version: 4.0.0 @@ -151,6 +154,9 @@ importers: oxlint-tsgolint: specifier: 7.0.2001 version: 7.0.2001 + papaparse: + specifier: 5.5.3 + version: 5.5.3 pnpm: specifier: 11.7.0 version: 11.7.0 diff --git a/python/sdk-runtime/README.i18n.yaml b/python/sdk-runtime/README.i18n.yaml index 5e935c8d47..d0014662db 100644 --- a/python/sdk-runtime/README.i18n.yaml +++ b/python/sdk-runtime/README.i18n.yaml @@ -6,8 +6,8 @@ en: 1ea1d8e6d2c14b02 zh: 8e0b9903158667a1 /deepseek-harness-runtime-bin/installed-commands-and-artifacts: - en: af4e7919327e1991 - zh: b4114823b4f1a3fa + en: 1a80f487c7d2286b + zh: 81cb14d2c646a906 /deepseek-harness-runtime-bin/python-module-api: en: a88b7574e0e00a28 zh: 3fd4fe8c0b338e79 diff --git a/python/sdk-runtime/README.md b/python/sdk-runtime/README.md index 39d1ce71c4..67c5040c27 100644 --- a/python/sdk-runtime/README.md +++ b/python/sdk-runtime/README.md @@ -12,7 +12,7 @@ Production executables are named `deepseek-harness-sdk-runtime-- Each target also requires `-office/`, where the stem excludes `.exe`. This directory contains the complete installed Office packages and their dependencies, preserving engine resources, manifests, licenses, source inventories, and helper permissions. Copy this directory together with the executable. A missing target engine fails the sidecar build with its npm package name and target platform/architecture. -Each wheel also includes `-/primary-runtime/` (CPython, locked Office Python libraries, standalone Node, and pnpm) and sibling `office-skills/` (three default workflows and their shared checker). These are ordinary relocatable files, not bytes embedded in the executable. The shared builder selects target-native archives for all five wheel targets and executes its smoke on the native build host. Packaging and installed-runtime lookup reject missing resources, wrong-platform metadata, and lost Python or Node executable permissions. The short platform directory avoids repeating the executable name in Python DLL paths on Windows. +Each wheel also includes `-/primary-runtime/` (CPython, locked Office Python libraries, standalone Node, and pnpm) and sibling `office-skills/` (three default workflows and their shared checker). These are ordinary relocatable files, not bytes embedded in the executable. The shared builder selects target-native archives for all five wheel targets and executes its smoke on the native build host. Packaging and installed-runtime lookup reject missing resources, wrong-platform metadata, and lost Python or Node executable permissions. The short platform directory avoids repeating the executable name in Python DLL paths on Windows. The shared builder applies the [NumPy/pandas test-file exclusions](../../apps/desktop/README.md#bundled-workspace-dependencies). The packaged bootstrap supplies `DSH_BUNDLED_PRIMARY_RUNTIME` as a carrier default. The `sdk` profile uses it when `DSH_PRIMARY_RUNTIME` is unset; an explicit path overrides it, and an empty string disables the query and Office provider. External payloads retain the `primary-runtime/` plus sibling `office-skills/` layout. Loaded skills expose the bundled Node and adjacent Office CLI as absolute paths; custom Python-only payloads must patch `skill-office.config.cli: false` or supply `skill-office.config.node`. The SDK reads Python in place without copying it into `DSH_HOME`. Source and dev-only Node carriers have no bundled default; they use an explicit `DSH_PRIMARY_RUNTIME`. diff --git a/python/sdk-runtime/README.zh.md b/python/sdk-runtime/README.zh.md index 4dd2b363cc..80e7aafe81 100644 --- a/python/sdk-runtime/README.zh.md +++ b/python/sdk-runtime/README.zh.md @@ -12,7 +12,7 @@ wheel 包会安装 `dsh` 控制台命令和 `deepseek_harness_runtime` Python 每个目标还要求 `-office/`，其中 stem 不含 `.exe`。该目录包含完整的已安装 Office 包及其依赖，保留引擎资源、清单、许可证、源码清单与辅助程序权限。复制可执行文件时必须一并复制此目录。缺少目标引擎会使 sidecar 构建失败，错误会指出其 npm 包名与目标平台／架构。 -每个 wheel 还包含 `-/primary-runtime/`（CPython、锁定版本的 Office Python 库、独立 Node 和 pnpm）及同级 `office-skills/`（三个默认工作流与共用检查脚本）。它们是可重定位的普通文件，不嵌入可执行文件。共享构建器为全部五个 wheel 目标选择原生归档，并在对应构建主机上执行冒烟检查。打包与已安装运行时定位会拒绝缺失资源、平台不符的元数据及 Python 或 Node 执行权限丢失。 较短的平台目录避免在 Windows Python DLL 路径中重复可执行文件名称。 +每个 wheel 还包含 `-/primary-runtime/`（CPython、锁定版本的 Office Python 库、独立 Node 和 pnpm）及同级 `office-skills/`（三个默认工作流与共用检查脚本）。它们是可重定位的普通文件，不嵌入可执行文件。共享构建器为全部五个 wheel 目标选择原生归档，并在对应构建主机上执行冒烟检查。打包与已安装运行时定位会拒绝缺失资源、平台不符的元数据及 Python 或 Node 执行权限丢失。 较短的平台目录避免在 Windows Python DLL 路径中重复可执行文件名称。共享构建器采用 [NumPy/pandas 测试文件裁剪规则](../../apps/desktop/README.zh.md#bundled-workspace-dependencies)。 打包启动器通过 `DSH_BUNDLED_PRIMARY_RUNTIME` 提供载体默认路径。`sdk` profile 在未设置 `DSH_PRIMARY_RUNTIME` 时使用该路径；显式路径覆盖它，空字符串禁用查询与 Office 提供方。外部 payload 保持 `primary-runtime/` 与同级 `office-skills/` 布局。加载后的技能给出内置 Node 与伴随 Office CLI 的绝对路径；自定义的纯 Python payload 必须配置 `skill-office.config.cli: false` 或提供 `skill-office.config.node`。SDK 原位读取 Python，不复制到 `DSH_HOME`。源码与仅供开发的 Node 载体没有随包默认路径，需显式设置 `DSH_PRIMARY_RUNTIME`。 diff --git a/scripts/primary-runtime/prepare.ts b/scripts/primary-runtime/prepare.ts index a156a7564a..635b0c2aa4 100644 --- a/scripts/primary-runtime/prepare.ts +++ b/scripts/primary-runtime/prepare.ts @@ -12,6 +12,7 @@ import extractZip from 'extract-zip' import { x as extractTar } from 'tar' import { parsePrimaryRuntime, workspaceDependencyPaths, type PrimaryRuntimeManifest } from '../../packages/skill/tool-workspace-dependencies/src/index.ts' import lock from './lock.json' with { type: 'json' } +import { prunePrimaryRuntimePythonTests } from './prune-python-tests.ts' /** * Download or reuse an archive only when its bytes match the release lock. @@ -54,7 +55,7 @@ export function primaryRuntimePayloadDigest( // Identity preserves key order within the selected target, wheel records and distribution map, plus wheel-entry order. // Bump format when extraction or assembly changes payload bytes without changing locked inputs. return createHash('sha256').update(JSON.stringify({ - format: 4, target, pythonVersion, pythonRelease, nodeVersion: pnpmVersion === undefined ? undefined : nodeVersion, + format: 6, target, pythonVersion, pythonRelease, nodeVersion: pnpmVersion === undefined ? undefined : nodeVersion, artifact: runtimeLock.targets[target], wheels, pythonPackages, pnpm: pnpmVersion, })).digest('hex') } @@ -155,6 +156,7 @@ export async function preparePrimaryRuntime(options: PreparePrimaryRuntimeOption for (const wheel of [...artifact.wheels, ...lock.wheels]) { await unpackPrimaryRuntimeWheel(await downloadPrimaryRuntimeAsset(wheel.url, wheel.sha256, paths.downloads), entries.pythonPackages) } + prunePrimaryRuntimePythonTests(entries.pythonPackages) writeFileSync(join(output, 'runtime.json'), `${JSON.stringify(manifest, undefined, 2)}\n`) const destination = join(paths.runtime, 'primary-runtime') rmSync(destination, { recursive: true, force: true }) diff --git a/scripts/primary-runtime/prune-python-tests.spec.ts b/scripts/primary-runtime/prune-python-tests.spec.ts new file mode 100644 index 0000000000..25b59f3e3c --- /dev/null +++ b/scripts/primary-runtime/prune-python-tests.spec.ts @@ -0,0 +1,85 @@ +import { createHash } from 'node:crypto' +import { mkdir, mkdtemp, readFile, rm, writeFile } from 'node:fs/promises' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import { zipSync } from 'fflate' +import { expect, it } from 'vitest' +import lock from './lock.json' with { type: 'json' } +import { unpackPrimaryRuntimeWheel } from './prepare.ts' +import { prunePrimaryRuntimePythonTests } from './prune-python-tests.ts' + +const records = { + numpy: `numpy-${lock.pythonPackages.numpy}.dist-info/RECORD`, + pandas: `pandas-${lock.pythonPackages.pandas}.dist-info/RECORD`, +} + +it('removes only NumPy/pandas test directories and keeps retained wheel files and RECORD hashes', async () => { + const root = await mkdtemp(join(tmpdir(), 'primary-python-pruning-')) + try { + const removed = [ + 'numpy/tests/test_numpy.py', 'numpy/_core/tests/test_core.py', 'numpy/_core/tests/data/values,small.csv', + 'numpy/testing/tests/test_utils.py', 'pandas/tests/io/test_excel.py', 'pandas/tests/io/data/workbook.xlsx', + ] + const retained = [ + 'numpy/__init__.py', 'numpy/_core/_multiarray_tests.so', 'numpy/_core/_multiarray_tests.pyd', + 'numpy/testing/__init__.py', 'numpy/_core/tests.py', 'numpy/data/values,small.csv', + 'pandas/__init__.py', 'pandas/testing.py', 'pandas/_testing/__init__.py', 'pandas/_libs/testing.pyd', + `numpy-${lock.pythonPackages.numpy}.dist-info/licenses/LICENSE.txt`, + `pandas-${lock.pythonPackages.pandas}.dist-info/LICENSE`, + 'other/tests/test_other.py', 'numpy_extra/tests/test_extra.py', + ] + const files = Object.fromEntries([...removed, ...retained].map(path => [path, Buffer.from(`contents of ${path}\n`)])) + // Quoted paths exercise RECORD's CSV encoding even when a filename contains a comma. + const row = (path: string): string => `"${path}",sha256=${createHash('sha256').update(files[path]!).digest('base64url')},${files[path]!.length}\n` + for (const [name, record] of Object.entries(records)) { + const paths = [...removed, ...retained].filter(path => path.startsWith(`${name}/`) || path.startsWith(`${name}-`)) + files[record] = Buffer.from(`${paths.map(row).join('')}${record},,\n`) + } + const archive = join(root, 'libraries.whl') + const sitePackages = join(root, 'site-packages') + const wheel = Buffer.from(zipSync(files)) + await writeFile(archive, wheel) + await unpackPrimaryRuntimeWheel(archive, sitePackages) + await mkdir(join(sitePackages, 'numpy/empty/tests'), { recursive: true }) + + prunePrimaryRuntimePythonTests(sitePackages) + + for (const path of [...removed, 'numpy/tests', 'numpy/empty/tests', 'pandas/tests']) { + await expect(readFile(join(sitePackages, path))).rejects.toMatchObject({ code: 'ENOENT' }) + } + for (const path of retained) expect(await readFile(join(sitePackages, path))).toEqual(files[path]) + for (const [name, record] of Object.entries(records)) { + const lines = (await readFile(join(sitePackages, record), 'utf8')).trimEnd().split('\n') + const expected = retained.filter(path => path.startsWith(`${name}/`) || path.startsWith(`${name}-`)) + expect(lines).toHaveLength(expected.length + 1) + for (const path of expected) { + const expectedRow = row(path).trimEnd() + expect(lines).toContain(path.includes(',') ? expectedRow : expectedRow.replace(`"${path}"`, path)) + } + expect(lines).toContain(`${record},,`) + } + expect(await readFile(archive)).toEqual(wheel) + const before = await Promise.all(Object.values(records).map(record => readFile(join(sitePackages, record)))) + prunePrimaryRuntimePythonTests(sitePackages) + expect(await Promise.all(Object.values(records).map(record => readFile(join(sitePackages, record))))).toEqual(before) + } finally { + await rm(root, { recursive: true, force: true }) + } +}) + +it.each(['"unterminated,,\n', 'numpy/tests/test_numpy.py,\n', '', ',,\n'])( + 'rejects malformed installed-file records before deleting that distribution\'s tests: %j', async (record) => { + const root = await mkdtemp(join(tmpdir(), 'primary-python-record-')) + try { + await mkdir(join(root, `numpy-${lock.pythonPackages.numpy}.dist-info`)) + await writeFile(join(root, records.numpy), record) + await mkdir(join(root, 'numpy/tests'), { recursive: true }) + const testFile = join(root, 'numpy/tests/test_numpy.py') + await writeFile(testFile, 'test fixture\n') + expect(() => { prunePrimaryRuntimePythonTests(root) }).toThrow('invalid wheel RECORD') + expect(await readFile(testFile, 'utf8')).toBe('test fixture\n') + } finally { + await rm(root, { recursive: true, force: true }) + } + }, +) diff --git a/scripts/primary-runtime/prune-python-tests.ts b/scripts/primary-runtime/prune-python-tests.ts new file mode 100644 index 0000000000..a953ca9afb --- /dev/null +++ b/scripts/primary-runtime/prune-python-tests.ts @@ -0,0 +1,41 @@ +/** Remove upstream Python test suites from the staged primary runtime. */ + +import { readFileSync, readdirSync, rmSync, writeFileSync } from 'node:fs' +import { join } from 'node:path' +import Papa from 'papaparse' +import lock from './lock.json' with { type: 'json' } + +function removeTestDirectories(root: string, directory: string): string[] { + const removed: string[] = [] + for (const entry of readdirSync(join(root, directory), { withFileTypes: true })) { + if (!entry.isDirectory()) continue + const path = `${directory}/${entry.name}` + if (entry.name === 'tests') { + rmSync(join(root, path), { recursive: true }) + removed.push(path) + } else { + removed.push(...removeTestDirectories(root, path)) + } + } + return removed +} + +/** + * Prune NumPy/pandas tests and their RECORD rows, preserving testing helpers and native extensions. + * @param sitePackages - Staged site-packages directory containing the locked wheels. + * @throws If a locked distribution's RECORD is missing or malformed. + */ +export function prunePrimaryRuntimePythonTests(sitePackages: string): void { + for (const name of ['numpy', 'pandas'] as const) { + const record = join(sitePackages, `${name}-${lock.pythonPackages[name]}.dist-info`, 'RECORD') + const { data, errors } = Papa.parse(readFileSync(record, 'utf8'), { delimiter: ',', skipEmptyLines: true }) + if (errors.length > 0 || data.length === 0 + || !data.every((row): row is [string, string, string] => row.length === 3 && Boolean(row[0]))) { + throw new Error(`primary runtime: invalid wheel RECORD: ${record}`) + } + const removed = removeTestDirectories(sitePackages, name) + if (removed.length === 0) continue + const retained = data.filter(([path]) => !removed.some(directory => path.startsWith(`${directory}/`))) + writeFileSync(record, `${Papa.unparse(retained, { newline: '\n' })}\n`) + } +} diff --git a/scripts/primary-runtime/smoke.py b/scripts/primary-runtime/smoke.py index 232d67e108..73d7f4418f 100644 --- a/scripts/primary-runtime/smoke.py +++ b/scripts/primary-runtime/smoke.py @@ -1,6 +1,8 @@ """Exercise the relocated Office payload with its own isolated interpreter.""" +import csv import importlib.metadata +import io import json from pathlib import Path import re @@ -9,6 +11,7 @@ import sys import tempfile import numpy +import numpy._core._multiarray_tests import pandas from docx import Document from docx.shared import Inches as DocxInches @@ -38,6 +41,16 @@ def main(): assert actual == expected, (name, actual, expected) assert numpy.arange(4).sum() == 6 assert pandas.DataFrame({"n": [1, 2]}).n.sum() == 3 + numpy.testing.assert_allclose(numpy.arange(4), [0, 1, 2, 3]) + pandas.testing.assert_frame_equal(pandas.DataFrame({"n": [1, 2]}), pandas.DataFrame({"n": [1, 2]})) + for library in (numpy, pandas): + root = Path(library.__file__).parent + assert not any(path.is_dir() for path in root.rglob("tests")), root + distribution = importlib.metadata.distribution(library.__name__) + record = distribution.read_text("RECORD") + assert record, library.__name__ + for file, _hash, _size in csv.reader(io.StringIO(record)): + assert distribution.locate_file(file).is_file(), file with tempfile.TemporaryDirectory(prefix="dsh-office-smoke-") as directory: root = Path(directory)