diff --git a/.agents/notes/implemented/process/2026-10-05-pr-artifact-typechecks.i18n.yaml b/.agents/notes/implemented/process/2026-10-05-pr-artifact-typechecks.i18n.yaml new file mode 100644 index 0000000000..0569fcc474 --- /dev/null +++ b/.agents/notes/implemented/process/2026-10-05-pr-artifact-typechecks.i18n.yaml @@ -0,0 +1,19 @@ +# Bilingual-pair consistency record for 2026-10-05-pr-artifact-typechecks.md (docs/i18n/README.md): per heading +# section, a hash of its English and Chinese blocks outside code blocks and generated regions. +# After editing either side, bring the other along and re-record with: +# pnpm run verify-translation-pairing --write .agents/notes/implemented/process/2026-10-05-pr-artifact-typechecks.md +/agent-note-pr-artifact-builds-and-aggregate-typescript-checks: + en: a25e7cdcad03934f + zh: 48e31429730e0885 +/agent-note-pr-artifact-builds-and-aggregate-typescript-checks/problem: + en: e5c984fa32cc5e69 + zh: 25d5742b2106d830 +/agent-note-pr-artifact-builds-and-aggregate-typescript-checks/decision: + en: 184aa812c4c857fe + zh: 953c2ec2f4e4031f +/agent-note-pr-artifact-builds-and-aggregate-typescript-checks/alternatives-considered: + en: f61da066f9ddcd62 + zh: 816fd6e0063017cd +/agent-note-pr-artifact-builds-and-aggregate-typescript-checks/consequences: + en: 3244bf7352ba0471 + zh: 484da0d4dc861655 diff --git a/.agents/notes/implemented/process/2026-10-05-pr-artifact-typechecks.md b/.agents/notes/implemented/process/2026-10-05-pr-artifact-typechecks.md new file mode 100644 index 0000000000..bfe998c2a3 --- /dev/null +++ b/.agents/notes/implemented/process/2026-10-05-pr-artifact-typechecks.md @@ -0,0 +1,31 @@ +# Agent Note: PR artifact builds and aggregate TypeScript checks + +Status: implemented + +English | [中文](2026-10-05-pr-artifact-typechecks.zh.md) + +## Problem + +PR jobs need the same compiled packages for benchmarks, compatibility, packaging, and runtime checks. Building the repository-wide test and script programs in every artifact job repeats expensive diagnostics. In the allowed #3661 baseline, benchmark build preparation takes about 304 seconds before 432 seconds of serial measurements. A cold macOS arm64 Host compiler profile on Node 24.19.0 takes 84.01 seconds, including 55.92 seconds in the aggregate program; its 282 emitted projects take 24.33 seconds combined. Increasing concurrency consumes more shared-runner resources. + +## Decision + +Required Linux consumer and native Windows build jobs retain normal builds, including both complete aggregate programs. Public `build` and `typecheck` keep the same checks. [Node version ownership](2026-07-06-node-engine-floor.md) and [native Windows ownership](2026-08-08-native-windows-pull-request-ci.md) remain in force. + +PR artifact builds compile every existing project reference from each compiler face with one serial `tsc -b` process, then run the ordinary Typert, bundling, Desktop, and Web stages. The helper reads the existing Host and Client aggregates rather than maintaining another package list. It omits only each root's additional test and script program; package diagnostics, emitted outputs, and Host-before-Client ordering remain required. Non-PR release and real-API workflow builds retain their normal checks. + +Job inventory, runner selectors, gate budgets, worker counts, and benchmark sample counts stay fixed. Coverage partitions reuse compact file-cost metadata within their platform and runner pool. Parallel release packing uses pnpm's existing bounded scheduler; serial publication keeps family order. + +## Alternatives considered + +**Raise job or worker concurrency.** Rejected because the shared runner must keep its current resource ceiling. + +**Remove aggregate checks from every build.** Rejected because package projects do not typecheck all repository tests and scripts; required Linux and Windows jobs must retain those diagnostics. + +**Cache compiled outputs across revisions.** Deferred because cache correctness and invalidation add obligations that removing duplicate aggregate work does not need. + +## Consequences + +The cold Host reference build takes 23.49 seconds with 1.91 GiB peak RSS, compared with 84.01 seconds and 3.87 GiB for the complete compiler pass. All 5,932 emitted files match by path and SHA-256. These are single local compiler samples, excluding bundling and tests; current-head CI owns the PR latency result. The benchmark scenarios and their budgets remain unchanged. + +Artifact builds deliberately do not diagnose aggregate-only test or script errors. Required normal builds reject those errors; fixture checks also reject package errors on both paths and compare emitted JavaScript, declarations, maps, and build information. Removing or changing a required full build requires transferring both aggregate checks to another blocking owner first. Master serial references remain complete under their [existing policy](2026-07-21-serial-cross-platform-ci-reference.md). diff --git a/.agents/notes/implemented/process/2026-10-05-pr-artifact-typechecks.zh.md b/.agents/notes/implemented/process/2026-10-05-pr-artifact-typechecks.zh.md new file mode 100644 index 0000000000..2f40ec42a5 --- /dev/null +++ b/.agents/notes/implemented/process/2026-10-05-pr-artifact-typechecks.zh.md @@ -0,0 +1,31 @@ +# Agent Note: PR 产物构建与 aggregate TypeScript 检查 + +Status: implemented + +[English](2026-10-05-pr-artifact-typechecks.md) | 中文 + +## Problem + +PR job 需要相同的编译包来运行 benchmark、兼容性、打包和运行时检查。每个产物 job 都构建仓库级测试和脚本 program，会重复执行昂贵的诊断。在允许使用的 #3661 baseline 中，benchmark 在 432 秒的串行测量前花费约 304 秒准备构建。在 macOS arm64、Node 24.19.0 上，一次冷启动 Host 编译器 profile 耗时 84.01 秒，其中 aggregate program 占 55.92 秒；282 个有 emit 的 project 合计耗时 24.33 秒。提高并发会消耗更多共享运行器资源。 + +## Decision + +必需的 Linux consumer 与原生 Windows build job 保留普通构建，包括两个完整 aggregate program。公开的 `build` 和 `typecheck` 保留相同检查。[Node 版本职责](2026-07-06-node-engine-floor.zh.md)与[原生 Windows 职责](2026-08-08-native-windows-pull-request-ci.zh.md)仍然有效。 + +PR 产物构建通过一个串行 `tsc -b` 进程编译各 compiler face 的全部现有 project reference，然后运行普通 Typert、bundling、Desktop 和 Web 阶段。helper 读取现有 Host 与 Client aggregate，而不维护另一份包清单。它只省略各 root 附加的测试和脚本 program；包诊断、emit 输出及 Host 先于 Client 的顺序仍然必需。非 PR 的 release 和真实 API 工作流构建保留普通检查。 + +job 清单、runner 选择、gate 预算、worker 数量和 benchmark sample 数量保持固定。覆盖率分区在各自的平台和 runner pool 内复用精简的文件成本元数据。并行 release 打包使用 pnpm 现有的有界调度器；串行发布保持 family 顺序。 + +## Alternatives considered + +**提高 job 或 worker 并发。** 否决，因为共享运行器必须保持现有资源上限。 + +**从所有构建中移除 aggregate 检查。** 否决，因为包 project 不会检查所有仓库测试和脚本的类型；必需的 Linux 与 Windows job 必须保留这些诊断。 + +**跨 revision 缓存编译输出。** 暂缓，因为缓存正确性与失效规则增加了额外义务，而移除重复 aggregate 工作无需承担这些义务。 + +## Consequences + +冷启动 Host reference 构建耗时 23.49 秒、峰值 RSS 为 1.91 GiB；完整编译器阶段为 84.01 秒和 3.87 GiB。全部 5,932 个 emit 文件的路径与 SHA-256 相同。这些是单次本地编译器样本，不含 bundling 和测试；PR 延迟结果以当前 head 的 CI 为准。benchmark 场景与预算保持不变。 + +产物构建有意不诊断仅属于 aggregate 的测试或脚本错误。必需的普通构建拒绝这些错误；fixture 检查还验证两个路径都会拒绝包错误，并比较 emit 的 JavaScript、声明、map 和 build 信息。删除或改变必需的完整构建前，必须先把两个 aggregate 检查转交另一处阻塞型 owner。master 串行 reference 按[现有政策](2026-07-21-serial-cross-platform-ci-reference.zh.md)保持完整。 diff --git a/.github/workflows/build-exe-for-python-sdk.yml b/.github/workflows/build-exe-for-python-sdk.yml index e53d524ead..e5056f2d1a 100644 --- a/.github/workflows/build-exe-for-python-sdk.yml +++ b/.github/workflows/build-exe-for-python-sdk.yml @@ -256,10 +256,18 @@ jobs: exit 1 } + - name: Build PR runtime libraries and Web artifacts + if: inputs.ci && github.event_name == 'pull_request' + env: + DSH_BUILD_CLIENT_PROFILE: official + run: pnpm run build:artifacts + - name: Build single-exe env: DSH_BUILD_CLIENT_PROFILE: official - run: pnpm exec tsx scripts/build-exe-for-python-sdk.ts --targets=${{ matrix.target }} + run: >- + pnpm exec tsx scripts/build-exe-for-python-sdk.ts --targets=${{ matrix.target }} + ${{ inputs.ci && github.event_name == 'pull_request' && '--skip-build' || '' }} - name: Resolve platform outputs (POSIX) id: runtime-posix diff --git a/.github/workflows/build-preview-cloudflare.yml b/.github/workflows/build-preview-cloudflare.yml index 80caebdb76..7787fa3c93 100644 --- a/.github/workflows/build-preview-cloudflare.yml +++ b/.github/workflows/build-preview-cloudflare.yml @@ -71,7 +71,7 @@ jobs: # build:preview packs the image through the packer's installed bin # (lib/bin.js), so neither half exists before the full build runs. - name: Build workspace - run: pnpm run build + run: pnpm run build:artifacts - name: Build the preview page and pack the VFS image env: diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index fdcec2540d..da803ffe1a 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -141,6 +141,14 @@ jobs: with: persist-credentials: false + - name: Restore coverage duration history + uses: actions/cache/restore@v4 + with: + path: .coverage-times.json + key: coverage-times-Linux-${{ runner.environment }}-${{ vars.DSH_CI_FAILOVER_LINUX || 'enterprise' }}-${{ github.run_id }}-${{ github.run_attempt }} + restore-keys: | + coverage-times-Linux-${{ runner.environment }}-${{ vars.DSH_CI_FAILOVER_LINUX || 'enterprise' }}- + # Redirect the Node compile cache (enabled by pnpm and TypeScript) off # the root partition's /tmp before the first pnpm call in this lane. - name: Redirect Node compile cache to runner temp @@ -188,6 +196,13 @@ jobs: - name: Run exhaustive coverage run: pnpm run check:ci:coverage + - name: Save coverage duration history + if: ${{ !cancelled() }} + uses: actions/cache/save@v4 + with: + path: .coverage-times.json + key: coverage-times-Linux-${{ runner.environment }}-${{ vars.DSH_CI_FAILOVER_LINUX || 'enterprise' }}-${{ github.run_id }}-${{ github.run_attempt }} + node-24-bench: if: github.event_name == 'pull_request' runs-on: ubuntu-24.04 @@ -592,9 +607,9 @@ jobs: uses: actions/cache/restore@v4 with: path: .coverage-times.json - key: coverage-times-${{ github.run_id }} + key: coverage-times-Windows-${{ runner.environment }}-${{ vars.DSH_CI_FAILOVER_WINDOWS || 'enterprise' }}-${{ github.run_id }}-${{ github.run_attempt }} restore-keys: | - coverage-times- + coverage-times-Windows-${{ runner.environment }}-${{ vars.DSH_CI_FAILOVER_WINDOWS || 'enterprise' }}- - name: Enable Developer Mode (symlink support) shell: pwsh run: >- @@ -635,7 +650,7 @@ jobs: uses: actions/cache/save@v4 with: path: .coverage-times.json - key: coverage-times-${{ github.run_id }} + key: coverage-times-Windows-${{ runner.environment }}-${{ vars.DSH_CI_FAILOVER_WINDOWS || 'enterprise' }}-${{ github.run_id }}-${{ github.run_attempt }} windows-native-tests: if: github.event_name == 'pull_request' diff --git a/.github/workflows/e2e.yml b/.github/workflows/e2e.yml index 8b3bfc1400..ddd29a425b 100644 --- a/.github/workflows/e2e.yml +++ b/.github/workflows/e2e.yml @@ -107,7 +107,7 @@ jobs: # the built artifact under plain Node, resolving plugins through real package # exports — the shape a real consumer runs. That requires a prior build. - name: Build (lib for the e2e example bins) - run: pnpm run build:official + run: pnpm run ${{ github.event_name == 'pull_request' && 'build:artifacts --profile official' || 'build:official' }} - name: Prepare Office runtime run: pnpm run prepare:primary-runtime --target linux-x64 --output .cache/e2e-office diff --git a/.github/workflows/release-vendor.yml b/.github/workflows/release-vendor.yml index 75fb865767..bb5f40be6f 100644 --- a/.github/workflows/release-vendor.yml +++ b/.github/workflows/release-vendor.yml @@ -89,7 +89,7 @@ jobs: # The vendored packages publish their own sources and build outputs; the # host build produces what their manifests select. - name: Build - run: pnpm run build:lib:host + run: pnpm run ${{ github.event_name == 'pull_request' && 'build:lib:host:artifacts' || 'build:lib:host' }} # Concurrency here is rehearsal-only: the credentialed publish workflows # invoke release:pack without the flag and keep the strictly serial path. diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 633186c8f1..3857f569a0 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -146,7 +146,7 @@ jobs: run: pnpm run release:verify --family dsh - name: Build - run: pnpm run build:official + run: pnpm run ${{ github.event_name == 'pull_request' && 'build:artifacts --profile official' || 'build:official' }} # Concurrency here is rehearsal-only: the credentialed publish workflows # invoke release:pack without the flag and keep the strictly serial path. diff --git a/docs/development.i18n.yaml b/docs/development.i18n.yaml index 904fdd17cb..2b7e3c5e46 100644 --- a/docs/development.i18n.yaml +++ b/docs/development.i18n.yaml @@ -21,8 +21,8 @@ en: 3221d1076554001f zh: 29f166be6a5c04d9 /development-guide/contributor-reference/typescript-project-layout: - en: 42c03e89df4379e3 - zh: 9df43176aa64d047 + en: b89173a1dde2f502 + zh: e57bf1d7cf1e6490 /development-guide/contributor-reference/environment-variables: en: dba2c72ee6ffa930 zh: b65c27fbb9129ec8 @@ -30,8 +30,8 @@ en: dc98eda4f4021feb zh: b08766708803cf10 /development-guide/contributor-reference/ci-gates: - en: 8fa1aca14c548969 - zh: a2bd1199fa7f7a8b + en: b64a914e41b1d285 + zh: 6c031b507c3c0e84 /development-guide/contributor-reference/daily-commands: en: 08aa6ec1cdabfeb7 zh: dcce5ae9d3c829d0 diff --git a/docs/development.md b/docs/development.md index d2f3ff337f..b58c82cef3 100644 --- a/docs/development.md +++ b/docs/development.md @@ -85,6 +85,8 @@ tsdown --env.DSH_BUILD_FACE client pnpm run build:web ``` +`pnpm run build:artifacts` emits the same project and application artifacts while compiling only the existing package references of each compiler face. Required Linux and Windows normal builds and public `typecheck` also check the repository-wide test and script programs; PR artifact jobs reuse that ownership ([rationale](../.agents/notes/implemented/process/2026-10-05-pr-artifact-typechecks.md)). + Both tsdown passes match `vendor/*`, `packages/*/*`, and `apps/cli`; the Host pass also matches `apps/desktop-host`. They neither scan build artifacts to discover Client packages nor maintain a Host/Client package filter list. Package-local tsdown configs select entries for the current phase through `DSH_BUILD_FACE`: an ordinary Client plugin produces both its Node loader and browser bundle during the Client phase; `api-remotes` uses `hostPhase: true` to produce its Host entry early and only its browser bundle during the Client phase. Tsdown consumes only the JavaScript emitted to `lib/types` by the preceding tsc phase. Tsdown builds the matched workspace members concurrently, so `apps/desktop`, whose main bundle inlines workspace devDependencies from their `lib/` output, bundles in its own step after the Host pass ([Desktop README](../apps/desktop/README.md#bundled-workspace-dependencies)). Typert runs only during Host tsdown, seeded by `tsconfig.host.json`. It analyzes Host types and generates both Host reflection artifacts and the Host-for-Client Remote projection; Client tsdown does not start Typert. Consequently, `pnpm run typecheck` runs the complete Host lib phase before Client tsc, while `pnpm run build` continues through Client tsdown and the Web build. @@ -132,7 +134,7 @@ Contributors can opt into the comprehensive local gate set with `pnpm run check: ### CI gates -The keyless [CI workflow](../.github/workflows/ci.yml) groups independent gates into broad lanes and runs a smaller compatibility signal across supported Node versions. Artifact consumers wait for one build within their lane. Required benchmarks run separately on standard GitHub-hosted Linux; the [benchmark runner reference](../benchmarks/AGENTS.md) owns routing and the job timeout. The separate real-API workflow runs `pnpm run test:e2e` with its configured worker bound. See [scripts/run-gates.ts](../scripts/run-gates.ts) and the workflow files for the current gate and job inventory. +The keyless [CI workflow](../.github/workflows/ci.yml) groups independent gates into broad lanes and runs a smaller compatibility signal across supported Node versions. Artifact consumers wait for one build within their lane. Coverage partitions reuse file timings scoped to their platform and runner pool to balance work within the configured worker limit. Required benchmarks run separately on standard GitHub-hosted Linux; the [benchmark runner reference](../benchmarks/AGENTS.md) owns routing and the job timeout. The separate real-API workflow runs `pnpm run test:e2e` with its configured worker bound. See [scripts/run-gates.ts](../scripts/run-gates.ts) and the workflow files for the current gate and job inventory. The credential-free dsh dependency-layout and dsh/vendor pack rehearsals use the existing Linux self-hosted pool only when `DSH_CI_FAILOVER_LINUX=selfhosted` and the event is a trusted master push or same-repository, non-fork, non-Dependabot pull request. All other cases, including manual dispatch, use `ubuntu-24.04`; manual publication stays hosted. See the [release rehearsal runner reference](../.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.md) for persistent-store isolation and fallback limits. diff --git a/docs/development.zh.md b/docs/development.zh.md index 0ad92e883d..4bfaba369f 100644 --- a/docs/development.zh.md +++ b/docs/development.zh.md @@ -89,6 +89,8 @@ tsdown --env.DSH_BUILD_FACE client pnpm run build:web ``` +`pnpm run build:artifacts` 产出相同的 project 和应用产物，但只编译各 compiler face 的现有包 reference。必需的 Linux 与 Windows 普通构建以及公开 `typecheck` 还检查仓库级测试和脚本 program；PR 产物 job 复用这一职责划分（[理由](../.agents/notes/implemented/process/2026-10-05-pr-artifact-typechecks.zh.md)）。 + 两次 tsdown 都匹配 `vendor/*`、`packages/*/*` 与 `apps/cli`，Host 阶段另外匹配 `apps/desktop-host`；两者都不扫描构建产物来发现 Client 包，也不维护 Host/Client 包过滤表。包内 tsdown 配置根据 `DSH_BUILD_FACE` 决定当前阶段的入口：普通 Client 插件在 Client 阶段同时生成 Node loader 与 browser bundle；`api-remotes` 通过 `hostPhase: true` 提前生成 Host 入口，再在 Client 阶段只生成 browser bundle。tsdown 只消费 `lib/types` 中由前置 tsc 发射的 JavaScript。tsdown 并发构建匹配到的 workspace 成员；`apps/desktop` 的主进程 bundle 需要从工作区 devDependencies 的 `lib/` 产物内联这些包，因此在 Host 阶段之后单独一步打 bundle（[Desktop README](../apps/desktop/README.zh.md#bundled-workspace-dependencies)）。 Typert 只在 Host tsdown 中以 `tsconfig.host.json` 为种子运行。它分析 Host 类型并生成 Host 反射产物及 Host-for-Client Remote 投影；Client tsdown 不启动 Typert。`pnpm run typecheck` 因此先执行完整 Host lib 阶段，再运行 Client tsc；`pnpm run build` 继续执行 Client tsdown 和 Web 构建。 @@ -136,7 +138,7 @@ vendor manifest 守卫检查 `vendor/*/src` 下的改动是否连同对应的 `v ### CI 门禁 -keyless [CI 工作流](../.github/workflows/ci.yml) 将独立门禁分组到若干宽粒度 lane，并在受支持的 Node 版本上运行一组较小的兼容性检查。产物消费方在各自 lane 内等待一次 build。必需 benchmark 在标准 GitHub 托管 Linux 上独立运行；[benchmark 运行器说明](../benchmarks/AGENTS.md)拥有路由及 job 超时。单独的真实 API 工作流按其配置的 worker 上限运行 `pnpm run test:e2e`。当前门禁和 job 清单以 [scripts/run-gates.ts](../scripts/run-gates.ts) 和工作流文件为准。 +keyless [CI 工作流](../.github/workflows/ci.yml) 将独立门禁分组到若干宽粒度 lane，并在受支持的 Node 版本上运行一组较小的兼容性检查。产物消费方在各自 lane 内等待一次 build。覆盖率分区复用按平台和运行器池隔离的文件耗时，在配置的 worker 上限内均衡工作量。必需 benchmark 在标准 GitHub 托管 Linux 上独立运行；[benchmark 运行器说明](../benchmarks/AGENTS.md)拥有路由及 job 超时。单独的真实 API 工作流按其配置的 worker 上限运行 `pnpm run test:e2e`。当前门禁和 job 清单以 [scripts/run-gates.ts](../scripts/run-gates.ts) 和工作流文件为准。 不带凭据的 dsh 依赖布局检查与 dsh/vendor 打包演练仅在 `DSH_CI_FAILOVER_LINUX=selfhosted`，且事件为受信任的 master 推送或同仓库、非 fork、非 Dependabot 拉取请求时使用现有 Linux 自托管池。其余情况（包括手动触发）均使用 `ubuntu-24.04`；手动发布仍使用托管运行器。持久化存储隔离与回退限制见[发布演练运行器说明](../.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.zh.md)。 diff --git a/package.json b/package.json index f4970a1730..323ede5cdb 100644 --- a/package.json +++ b/package.json @@ -19,11 +19,15 @@ "scripts": { "prepare:primary-runtime": "tsx scripts/primary-runtime/prepare.ts", "build": "tsx scripts/build.ts", - "build:bench": "npm run build:native-system && npm run build:lib && tsdown --config-loader native --config benchmarks/tsdown.config.ts", + "build:artifacts": "tsx scripts/build.ts --artifacts-only", + "build:bench": "npm run build:native-system && npm run build:lib:artifacts && tsdown --config-loader native --config benchmarks/tsdown.config.ts", "build:official": "tsx scripts/build.ts --profile official", "build:lib": "pnpm run build:lib:host && pnpm run build:lib:client", "build:lib:host": "node --max-old-space-size=4096 ./node_modules/typescript/bin/tsc -b tsconfig.host.json && tsdown --config-loader native --env.DSH_BUILD_FACE host && pnpm --filter @deepseek-ai/dsh-desktop run bundle", "build:lib:client": "tsc -b tsconfig.client.json && tsdown --config-loader native --env.DSH_BUILD_FACE client", + "build:lib:artifacts": "pnpm run build:lib:host:artifacts && pnpm run build:lib:client:artifacts", + "build:lib:host:artifacts": "tsx scripts/compile-referenced-projects.ts host && tsdown --config-loader native --env.DSH_BUILD_FACE host && pnpm --filter @deepseek-ai/dsh-desktop run bundle", + "build:lib:client:artifacts": "tsx scripts/compile-referenced-projects.ts client && tsdown --config-loader native --env.DSH_BUILD_FACE client", "build:web": "pnpm --filter @deepseek-ai/dsh-web-frontend run build", "build:desktop": "pnpm --filter @deepseek-ai/dsh-desktop run build", "dev:desktop": "pnpm --filter @deepseek-ai/dsh-desktop run dev", diff --git a/scripts/build.spec.ts b/scripts/build.spec.ts index 07f5f8b98f..ecb6420061 100644 --- a/scripts/build.spec.ts +++ b/scripts/build.spec.ts @@ -15,4 +15,14 @@ describe('repository build CLI', () => { expect(result.status).not.toBe(0) expect(result.stderr).toContain('build: Node.js TypeScript type stripping is unavailable') }) + + it('rejects a value for the artifacts-only flag before any build step', () => { + const result = spawnSync( + process.execPath, + ['--import', 'tsx/esm', resolve(root, 'scripts/build.ts'), '--artifacts-only=yes'], + { cwd: root, encoding: 'utf8' }, + ) + expect(result.status).not.toBe(0) + expect(result.stderr).toContain('does not take an argument') + }) }) diff --git a/scripts/build.ts b/scripts/build.ts index a7b06bda65..e90b4cf4a4 100644 --- a/scripts/build.ts +++ b/scripts/build.ts @@ -1,4 +1,4 @@ -/** Run the complete repository build and bind its client artifacts to their public environment. */ +/** Build repository artifacts and bind client outputs to their public environment. */ import { spawnSync } from 'node:child_process' import { rmSync } from 'node:fs' @@ -28,14 +28,17 @@ function runScript(script: string, environment: NodeJS.ProcessEnv): void { } } -/** Run the full build selected by `--profile` or `DSH_BUILD_CLIENT_PROFILE`. */ +/** Build client artifacts and optionally omit the repository test and script typechecks. */ function main(): void { // tsdown.config.ts loads only through Node type stripping (`--config-loader native`); this names the cause before tsdown fails. if (!process.features.typescript) { throw new Error('build: Node.js TypeScript type stripping is unavailable in this Node.js process; remove --no-experimental-strip-types from NODE_OPTIONS or use a Node.js build with TypeScript support') } const { values } = parseArgs({ - options: { profile: { type: 'string' } }, + options: { + profile: { type: 'string' }, + 'artifacts-only': { type: 'boolean' }, + }, allowPositionals: false, }) const root = resolve(import.meta.dirname, '..') @@ -46,7 +49,7 @@ function main(): void { rmSync(resolve(root, CLIENT_BUILD_RECORD_PATH), { force: true }) runScript('build:native-system', buildEnvironment) - runScript('build:lib', buildEnvironment) + runScript(values['artifacts-only'] ? 'build:lib:artifacts' : 'build:lib', buildEnvironment) runScript('build:web', buildEnvironment) const record = writeClientBuildRecord(root, clientEnvironment) console.log( diff --git a/scripts/ci-workflow.spec.ts b/scripts/ci-workflow.spec.ts index dd52510c86..5b885a9b74 100644 --- a/scripts/ci-workflow.spec.ts +++ b/scripts/ci-workflow.spec.ts @@ -14,6 +14,73 @@ const runnerPrivatePnpmDestination = /^\$\{\{ runner\.temp \}\}\/setup-pnpm-\$\{ const nativeWindowsPnpmDestination = '${{ runner.temp }}/setup-pnpm-js-${{ github.run_id }}-${{ github.run_attempt }}-${{ github.job }}' describe('CI workflow', () => { + it('retains aggregate diagnostics in required builds and protected publication', () => { + const ci = loadWorkflow('.github/workflows/ci.yml') + for (const [name, command] of [ + ['node-24-consumers', 'pnpm run check:ci:consumers'], + ['windows-build', 'pnpm run check:ci:windows-blocking'], + ] as const) { + const job = workflowJob(ci, name) + expect(job.steps).toContainEqual(expect.objectContaining({ run: command })) + } + for (const [file, command] of [ + ['release-publish.yml', 'pnpm run build:official'], + ['release-vendor-publish.yml', 'pnpm run build:lib:host'], + ] as const) { + expect(workflowJob(loadWorkflow('.github/workflows/' + file), 'pack').steps) + .toContainEqual(expect.objectContaining({ name: 'Build', run: command })) + } + const scripts = JSON.parse(readFileSync(resolve(root, 'package.json'), 'utf8')) as { scripts: Record } + expect(scripts.scripts['build:lib:host']).toContain('tsc -b tsconfig.host.json') + expect(scripts.scripts['build:lib:client']).toContain('tsc -b tsconfig.client.json') + expect(scripts.scripts.typecheck).toBe('npm run build:lib:host && npm run typecheck:contracts-ready') + }) + + it('selects artifact builds only for PR release and API rehearsals', () => { + const official = "pnpm run ${{ github.event_name == 'pull_request' && 'build:artifacts --profile official' || 'build:official' }}" + const vendor = "pnpm run ${{ github.event_name == 'pull_request' && 'build:lib:host:artifacts' || 'build:lib:host' }}" + for (const [file, jobName, expected] of [ + ['release.yml', 'pack', official], + ['release-vendor.yml', 'pack', vendor], + ['e2e.yml', 'e2e', official], + ] as const) { + const job = workflowJob(loadWorkflow('.github/workflows/' + file), jobName) + if (!Array.isArray(job.steps)) throw new TypeError(`${jobName} must define steps`) + const step = job.steps.filter(isRecord).find(candidate => typeof candidate.run === 'string' + && (candidate.run.includes('build:artifacts') || candidate.run.includes('build:lib:host:artifacts'))) + expect(step?.run).toBe(expected) + for (const event of ['pull_request', 'push', 'workflow_dispatch']) { + const expression = expected.slice('pnpm run ${{ '.length, -' }}'.length) + const selected: unknown = runInNewContext(expression, { github: { event_name: event } }, { timeout: 1000 }) + expect(selected).toBe(event === 'pull_request' + ? file === 'release-vendor.yml' ? 'build:lib:host:artifacts' : 'build:artifacts --profile official' + : file === 'release-vendor.yml' ? 'build:lib:host' : 'build:official') + } + } + }) + + it('builds PR Python artifacts before reusing them in the executable builder', () => { + const job = workflowJob(loadWorkflow('.github/workflows/build-exe-for-python-sdk.yml'), 'build') + if (!Array.isArray(job.steps)) throw new TypeError('Python runtime builder must define steps') + const steps = job.steps.filter(isRecord) + const prepare = steps.findIndex(step => step.run === 'pnpm run build:artifacts') + const execute = steps.findIndex(step => step.name === 'Build single-exe') + expect(prepare).toBeGreaterThanOrEqual(0) + expect(execute).toBeGreaterThan(prepare) + expect(steps[prepare]).toMatchObject({ + if: "inputs.ci && github.event_name == 'pull_request'", + env: { DSH_BUILD_CLIENT_PROFILE: 'official' }, + }) + expect(steps[execute]?.run).toContain("${{ inputs.ci && github.event_name == 'pull_request' && '--skip-build' || '' }}") + for (const ci of [false, true]) { + for (const event of ['pull_request', 'push', 'workflow_dispatch']) { + const context = { inputs: { ci }, github: { event_name: event } } + const selected: unknown = runInNewContext("inputs.ci && github.event_name == 'pull_request'", context, { timeout: 1000 }) + expect(selected).toBe(ci && event === 'pull_request') + } + } + }) + it('prepares confinement before Node compatibility smokes', () => { const job = workflowJob(loadWorkflow('.github/workflows/ci.yml'), 'node-compat') if (!Array.isArray(job.steps)) throw new TypeError('Node compatibility job must define steps') @@ -62,6 +129,34 @@ describe('CI workflow', () => { expect(wine.steps).toContainEqual(expect.objectContaining({ name: 'Shut down wineserver', if: 'always()' })) }) + it.each([ + ['node-24-coverage', 'Linux', 'LINUX', 'pnpm run check:ci:coverage'], + ['windows-coverage', 'Windows', 'WINDOWS', 'pnpm run check:ci:coverage'], + ])('reuses %s coverage timings only within its platform and runner pool', (jobName, platform, variable, command) => { + const job = workflowJob(loadWorkflow('.github/workflows/ci.yml'), jobName) + if (!Array.isArray(job.steps)) throw new TypeError(`${jobName} must define steps`) + const steps = job.steps.filter(isRecord) + const checkout = steps.findIndex(step => step.uses === 'actions/checkout@v6') + const restore = steps.findIndex(step => step.name === 'Restore coverage duration history') + const coverage = steps.findIndex(step => step.run === command) + const save = steps.findIndex(step => step.name === 'Save coverage duration history') + const prefix = `coverage-times-${platform}-\${{ runner.environment }}-\${{ vars.DSH_CI_FAILOVER_${variable} || 'enterprise' }}-` + const key = prefix + '${{ github.run_id }}-${{ github.run_attempt }}' + expect(checkout).toBeGreaterThanOrEqual(0) + expect(restore).toBeGreaterThan(checkout) + expect(coverage).toBeGreaterThan(restore) + expect(save).toBeGreaterThan(coverage) + expect(steps[restore]).toMatchObject({ + uses: 'actions/cache/restore@v4', + with: { path: '.coverage-times.json', key, 'restore-keys': `${prefix}\n` }, + }) + expect(steps[save]).toMatchObject({ + uses: 'actions/cache/save@v4', + if: '${{ !cancelled() }}', + with: { path: '.coverage-times.json', key }, + }) + }) + it('isolates every pnpm action setup destination per runner', () => { const files = ['.github/workflows/ci.yml', '.github/workflows/ci-master.yml'] const setups: Array<{ jobName: string; step: unknown }> = [] diff --git a/scripts/compile-referenced-projects.spec.ts b/scripts/compile-referenced-projects.spec.ts new file mode 100644 index 0000000000..4b8aa9b84e --- /dev/null +++ b/scripts/compile-referenced-projects.spec.ts @@ -0,0 +1,156 @@ +import { spawnSync } from 'node:child_process' +import { globSync, mkdirSync, mkdtempSync, readFileSync, rmSync, statSync, writeFileSync } from 'node:fs' +import { tmpdir } from 'node:os' +import { join, resolve } from 'node:path' +import { fileURLToPath } from 'node:url' +import { describe, expect, it, onTestFinished } from 'vitest' +import { compileReferencedProjects } from './compile-referenced-projects.ts' + +const compiler = fileURLToPath(import.meta.resolve('typescript/bin/tsc')) + +function fixture(): { root: string; leaf: string } { + const root = mkdtempSync(join(tmpdir(), 'dsh-referenced-projects-')) + onTestFinished(() => { rmSync(root, { recursive: true, force: true }) }) + const leaf = join(root, 'leaf package') + mkdirSync(join(leaf, 'src'), { recursive: true }) + writeFileSync(join(root, 'package.json'), JSON.stringify({ type: 'module' })) + writeFileSync(join(leaf, 'src/index.ts'), 'export const value: number = 42\n') + const compilerOptions = { + target: 'ES2024', + module: 'NodeNext', + moduleResolution: 'NodeNext', + lib: ['ES2024'], + types: [], + composite: true, + strict: true, + } + writeFileSync(join(leaf, 'tsconfig.json'), JSON.stringify({ + compilerOptions: { + ...compilerOptions, + rootDir: 'src', + outDir: 'lib/types', + declaration: true, + declarationMap: true, + sourceMap: true, + }, + include: ['src'], + })) + writeFileSync(join(root, 'aggregate.ts'), 'import { value } from "./leaf package/src/index.js"\nexport const aggregate: number = value\n') + for (const face of ['host', 'client']) { + writeFileSync(join(root, `tsconfig.${face}.json`), JSON.stringify({ + compilerOptions: { ...compilerOptions, noEmit: true }, + include: ['aggregate.ts'], + references: [{ path: './leaf package' }], + })) + } + return { root, leaf } +} + +function artifacts(leaf: string): Record { + const output = join(leaf, 'lib') + return Object.fromEntries(globSync('**/*', { cwd: output }) + .filter(path => statSync(join(output, path)).isFile()) + .sort() + .map(path => [path, readFileSync(join(output, path)).toString('base64')])) +} + +describe('package project compilation', () => { + it.each(['host', 'client'] as const)('preserves every %s package artifact byte', (face) => { + const { root, leaf } = fixture() + const complete = spawnSync(process.execPath, [compiler, '-b', `tsconfig.${face}.json`], { + cwd: root, + encoding: 'utf8', + }) + expect(complete.status, complete.stdout + complete.stderr).toBe(0) + const expected = artifacts(leaf) + expect(Object.keys(expected)).toEqual(expect.arrayContaining([ + 'types/index.js', 'types/index.js.map', 'types/index.d.ts', 'types/index.d.ts.map', + ])) + rmSync(join(leaf, 'lib'), { recursive: true }) + + compileReferencedProjects(root, face) + + expect(artifacts(leaf)).toEqual(expected) + }) + + it('keeps aggregate-only errors in the complete compiler check', () => { + const { root } = fixture() + writeFileSync(join(root, 'aggregate.ts'), 'export const aggregate: number = "invalid"\n') + + expect(() => { compileReferencedProjects(root, 'host') }).not.toThrow() + const complete = spawnSync(process.execPath, [compiler, '-b', 'tsconfig.host.json'], { + cwd: root, + encoding: 'utf8', + }) + expect(complete.status).not.toBe(0) + expect(complete.stdout).toContain('aggregate.ts(1,14): error TS2322') + }) + + it('rejects package source errors', () => { + const { root, leaf } = fixture() + writeFileSync(join(leaf, 'src/index.ts'), 'export const value: number = "invalid"\n') + + expect(() => { compileReferencedProjects(root, 'host') }).toThrow('host compiler exited with') + }) + + it.each([undefined, []])('rejects an aggregate with missing or empty references: %j', (references) => { + const { root } = fixture() + writeFileSync(join(root, 'tsconfig.host.json'), JSON.stringify({ files: [], references })) + + expect(() => { compileReferencedProjects(root, 'host') }).toThrow('requires non-empty project references') + }) + + it('rejects a missing referenced project', () => { + const { root } = fixture() + writeFileSync(join(root, 'tsconfig.host.json'), JSON.stringify({ + compilerOptions: { noEmit: true }, + files: [], + references: [{ path: './absent' }], + })) + + expect(() => { compileReferencedProjects(root, 'host') }).toThrow('host compiler exited with') + }) + + it.each([undefined, false])('rejects an aggregate capable of emitting files: noEmit=%s', (noEmit) => { + const { root } = fixture() + writeFileSync(join(root, 'tsconfig.host.json'), JSON.stringify({ + compilerOptions: { noEmit }, + include: ['aggregate.ts'], + references: [{ path: './leaf package' }], + })) + + expect(() => { compileReferencedProjects(root, 'host') }).toThrow( + `${join(root, 'tsconfig.host.json')} requires noEmit: true`, + ) + }) + + it('rejects a missing aggregate config', () => { + const { root } = fixture() + rmSync(join(root, 'tsconfig.host.json')) + + expect(() => { compileReferencedProjects(root, 'host') }).toThrow('Cannot read file') + }) + + it('rejects invalid compiler options', () => { + const { root } = fixture() + writeFileSync(join(root, 'tsconfig.host.json'), JSON.stringify({ + compilerOptions: { target: 'invalid' }, + files: [], + references: [{ path: './leaf package' }], + })) + + expect(() => { compileReferencedProjects(root, 'host') }).toThrow("Argument for '--target' option must be") + }) + + it('rejects an unsupported compiler face before building', () => { + const repository = resolve(import.meta.dirname, '..') + const result = spawnSync(process.execPath, [ + '--import', 'tsx/esm', + resolve(repository, 'scripts/compile-referenced-projects.ts'), + 'unsupported', + ], { cwd: repository, encoding: 'utf8' }) + + expect(result.status).not.toBe(0) + expect(result.stderr).toContain('expected exactly one compiler face: host or client') + }) +}) diff --git a/scripts/compile-referenced-projects.ts b/scripts/compile-referenced-projects.ts new file mode 100644 index 0000000000..9551c88f69 --- /dev/null +++ b/scripts/compile-referenced-projects.ts @@ -0,0 +1,57 @@ +/** Compile package projects with their existing diagnostics and outputs, excluding the aggregate test program. */ + +import { spawnSync } from 'node:child_process' +import { relative, resolve } from 'node:path' +import { fileURLToPath } from 'node:url' +import { parseArgs } from 'node:util' +import ts from 'typescript' + +/** + * Compile one aggregate's project references with the ordinary serial TypeScript builder. + * @param root - Repository root containing the aggregate tsconfig. + * @param face - Independent compiler face whose package projects are emitted. + * @throws If the aggregate emits files, its configuration is invalid, or package compilation fails. + */ +export function compileReferencedProjects(root: string, face: 'host' | 'client'): void { + const configPath = resolve(root, `tsconfig.${face}.json`) + const config = ts.getParsedCommandLineOfConfigFile(configPath, {}, { + ...ts.sys, + onUnRecoverableConfigFileDiagnostic(diagnostic) { + throw new Error(ts.flattenDiagnosticMessageText(diagnostic.messageText, '\n')) + }, + }) + if (config === undefined) throw new Error(`compile-referenced-projects: cannot read ${configPath}`) + const references = config.projectReferences + if (references === undefined || references.length === 0) { + throw new Error(`compile-referenced-projects: ${configPath} requires non-empty project references`) + } + if (config.errors.length > 0) { + throw new Error(config.errors + .map(diagnostic => ts.flattenDiagnosticMessageText(diagnostic.messageText, '\n')) + .join('\n')) + } + if (config.options.noEmit !== true) { + throw new Error(`compile-referenced-projects: ${configPath} requires noEmit: true to exclude its aggregate program`) + } + const compiler = fileURLToPath(import.meta.resolve('typescript/bin/tsc')) + const result = spawnSync(process.execPath, [ + ...face === 'host' ? ['--max-old-space-size=4096'] : [], + compiler, + '-b', + // Relative arguments keep the complete workspace graph below Windows' command-line limit. + ...references.map(reference => relative(root, reference.path)), + ], { cwd: root, stdio: 'inherit' }) + if (result.error !== undefined) throw result.error + if (result.status !== 0) { + throw new Error(`compile-referenced-projects: ${face} compiler exited with ${String(result.status ?? result.signal)}`) + } +} + +if (import.meta.main) { + const { positionals } = parseArgs({ allowPositionals: true }) + const [face] = positionals + if (positionals.length !== 1 || (face !== 'host' && face !== 'client')) { + throw new Error('compile-referenced-projects: expected exactly one compiler face: host or client') + } + compileReferencedProjects(resolve(import.meta.dirname, '..'), face) +} diff --git a/scripts/coverage-file-times.spec.ts b/scripts/coverage-file-times.spec.ts new file mode 100644 index 0000000000..be412f49a0 --- /dev/null +++ b/scripts/coverage-file-times.spec.ts @@ -0,0 +1,67 @@ +import { mkdtemp, readFile, rm } from 'node:fs/promises' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import type { ModuleDiagnostic, ResolvedConfig } from 'vitest/node' +import { afterEach, describe, expect, it } from 'vitest' +import CoverageFileTimesReporter from './coverage-file-times.ts' + +const roots: string[] = [] +afterEach(async () => { + for (const root of roots.splice(0)) await rm(root, { recursive: true, force: true }) +}) + +async function temporaryRoot(): Promise { + const root = await mkdtemp(join(tmpdir(), 'dsh-coverage-file-times-')) + roots.push(root) + return root +} + +describe('coverage file timing report', () => { + it('includes import and hook costs once without serializing coverage or test objects', async () => { + const root = await temporaryRoot() + const reporter = new CoverageFileTimesReporter() + reporter.onInit({ config: { root, outputFile: { fileTimes: 'times.json' } } }) + const diagnostic: ModuleDiagnostic = { + environmentSetupDuration: 3, + prepareDuration: 5, + setupDuration: 7, + collectDuration: 100, + duration: 200, + heap: undefined, + // Per-dependency import times are already included in collectDuration. + importDurations: { dependency: { selfTime: 100, totalTime: 100 } }, + } + const module = { + moduleId: join(root, 'packages', 'a', 'tests', 'x.spec.ts'), + diagnostic: () => diagnostic, + // Vitest modules retain tests and coverage through their project. The + // timing report must read only their paths and diagnostics. + toJSON: () => { throw new Error('test module was serialized') }, + } + await reporter.onTestRunEnd([module]) + + const output = await readFile(join(root, 'times.json'), 'utf8') + expect(JSON.parse(output)).toEqual({ 'packages/a/tests/x.spec.ts': 315 }) + expect(output.endsWith('\n')).toBe(true) + expect(output).not.toContain('coverageMap') + expect(output).not.toContain('assertionResults') + }) + + it('writes an empty report when no modules finished', async () => { + const root = await temporaryRoot() + const reporter = new CoverageFileTimesReporter() + reporter.onInit({ config: { root, outputFile: { fileTimes: 'times.json' } } }) + await reporter.onTestRunEnd([]) + expect(await readFile(join(root, 'times.json'), 'utf8')).toBe('{}\n') + }) + + const missingOutputs: ResolvedConfig['outputFile'][] = ['report.json', {}, { fileTimes: '' }] + it.each(missingOutputs)( + 'rejects missing timing output %j', + (outputFile) => { + const reporter = new CoverageFileTimesReporter() + expect(() => { reporter.onInit({ config: { root: '/repo', outputFile } }) }) + .toThrow('outputFile.fileTimes is required') + }, + ) +}) diff --git a/scripts/coverage-file-times.ts b/scripts/coverage-file-times.ts new file mode 100644 index 0000000000..7d9646636e --- /dev/null +++ b/scripts/coverage-file-times.ts @@ -0,0 +1,43 @@ +/** Record coverage partition file costs without retaining test results or coverage maps. */ +import { writeFile } from 'node:fs/promises' +import { relative, resolve, sep } from 'node:path' +import type { Reporter, ResolvedConfig, TestModule } from 'vitest/node' + +/** Vitest reporter that writes repository-relative file durations to `outputFile.fileTimes`. */ +export default class CoverageFileTimesReporter implements Reporter { + private root = '' + private outputFile = '' + + /** + * Resolve the timing output owned by the coverage coordinator. + * @param vitest - repository root and configured reporter output files. + * @throws when the file-times output is absent. + */ + public onInit(vitest: { config: Pick }): void { + const output = vitest.config.outputFile + if (typeof output !== 'object' || typeof output.fileTimes !== 'string' || output.fileTimes === '') { + throw new Error('coverage-file-times: outputFile.fileTimes is required.') + } + this.root = vitest.config.root + this.outputFile = resolve(this.root, output.fileTimes) + } + + /** + * Write each file's distinct environment, preparation, setup, collection, and execution costs. + * @param testModules - finished files, including failed and skipped files. + * @returns after the compact timing report has been written. + */ + public async onTestRunEnd(testModules: readonly Pick[]): Promise { + const durations: Record = {} + for (const module of testModules) { + const diagnostic = module.diagnostic() + const file = relative(this.root, module.moduleId).split(sep).join('/') + durations[file] = diagnostic.environmentSetupDuration + + diagnostic.prepareDuration + + diagnostic.setupDuration + + diagnostic.collectDuration + + diagnostic.duration + } + await writeFile(this.outputFile, `${JSON.stringify(durations)}\n`, 'utf8') + } +} diff --git a/scripts/coverage-partitions.spec.ts b/scripts/coverage-partitions.spec.ts index 2060608248..853207bc77 100644 --- a/scripts/coverage-partitions.spec.ts +++ b/scripts/coverage-partitions.spec.ts @@ -269,17 +269,26 @@ describe('coverage file inventory', () => { expect(durations.get('packages/a/tests/y.spec.ts')).toBe(7) }) - it('extracts per-file durations from partition json reports', async () => { + it('extracts per-file durations from compact partition timing reports', async () => { const root = await temporaryRoot() - const report = join(root, 'partition-1.report.json') + const report = join(root, 'partition-1.times.json') await writeFile(report, JSON.stringify({ - testResults: [ - { name: join(root, 'packages/a/tests/x.spec.ts'), startTime: 1000, endTime: 1500 }, - { name: 'not-a-spec', startTime: 1, endTime: 2 }, - ], + 'packages/a/tests/x.spec.ts': 500, + 'packages/a/tests/y.spec.ts': 0, + 'packages/a/tests/invalid.spec.ts': -1, + 'packages/a/tests/malformed.spec.ts': 'unknown', })) - const durations = collectPartitionDurations([report], root) + const durations = collectPartitionDurations([report]) expect(durations.get('packages/a/tests/x.spec.ts')).toBe(500) + expect(durations.get('packages/a/tests/y.spec.ts')).toBe(0) + expect(durations.size).toBe(2) + }) + + it.each(['null', '[]', '42', '"invalid"', '{'])('ignores corrupt timing metadata %s', async (contents) => { + const root = await temporaryRoot() + const report = join(root, 'partition.times.json') + await writeFile(report, contents) + expect(collectPartitionDurations([report])).toEqual(new Map()) }) }) @@ -511,9 +520,10 @@ describe('coverage partition coordinator', () => { '--maxWorkers=1', '--reporter=default', '--reporter=blob', - '--reporter=json', '--testTimeout=30000', ])) + expect(command.args).not.toContain('--reporter=json') + expect(command.args).toContain('--reporter=./scripts/coverage-file-times.ts') expect(command.args).not.toContain('--shard=1/3') expect(command.args.some(argument => argument.startsWith('--config='))).toBe(true) expect(command.env).toEqual({ diff --git a/scripts/coverage-partitions.ts b/scripts/coverage-partitions.ts index 575e475564..aafef76a60 100644 --- a/scripts/coverage-partitions.ts +++ b/scripts/coverage-partitions.ts @@ -27,6 +27,7 @@ export const COVERAGE_TEST_TIMEOUT_ENV = 'DSH_COVERAGE_TEST_TIMEOUT_MS' * working directory on every platform. */ const CANONICAL_LOCATIONS_REPORTER = './scripts/coverage-canonical-locations.ts' +const FILE_TIMES_REPORTER = './scripts/coverage-file-times.ts' /** One child command owned by the coverage coordinator. */ export interface CoverageCommand { @@ -299,23 +300,23 @@ export function writeFileDurations( } /** - * Extract per-file durations from Vitest JSON reporter outputs (one per - * partition). Each `testResults` entry names an absolute spec path and carries - * `startTime`/`endTime`; the difference is the file's recorded duration. + * Extract per-file durations from compact timing reports, one per partition. + * @param reportFiles - reports mapping repository-relative file paths to milliseconds. + * @returns finite, non-negative file costs from readable reports. */ -export function collectPartitionDurations(reportFiles: readonly string[], root: string): Map { +export function collectPartitionDurations(reportFiles: readonly string[]): Map { const durations = new Map() for (const file of reportFiles) { - let report: { testResults?: Array<{ name?: unknown; startTime?: number; endTime?: number }> } + let report: unknown try { - report = JSON.parse(readFileSync(file, 'utf8')) as { testResults?: Array<{ name?: unknown; startTime?: number; endTime?: number }> } + report = JSON.parse(readFileSync(file, 'utf8')) } catch { continue } - for (const result of report.testResults ?? []) { - if (typeof result.name !== 'string' || typeof result.startTime !== 'number' || typeof result.endTime !== 'number') continue - const relativePath = relative(root, result.name).split(sep).join('/') - durations.set(relativePath, Math.max(0, result.endTime - result.startTime)) + if (report === null || typeof report !== 'object' || Array.isArray(report)) continue + for (const [path, duration] of Object.entries(report)) { + if (typeof duration !== 'number' || !Number.isFinite(duration) || duration < 0) continue + durations.set(path, duration) } } return durations @@ -532,10 +533,10 @@ export class CoveragePartitionCoordinator { private persistDurations(partitionCount: number): void { const reportFiles = Array.from( { length: partitionCount }, - (_, index) => join(this.temporaryRoot, `partition-${index + 1}.report.json`), + (_, index) => join(this.temporaryRoot, `partition-${index + 1}.times.json`), ) const currentFiles = this.projectOf.size > 0 ? [...this.projectOf.keys()] : undefined - writeFileDurations(this.root, collectPartitionDurations(reportFiles, this.root), currentFiles) + writeFileDurations(this.root, collectPartitionDurations(reportFiles), currentFiles) } /** @@ -584,7 +585,7 @@ export class CoveragePartitionCoordinator { private partitionCommand(index: number, configPath: string): CoverageCommand { const blobPath = join(this.blobsRoot, `partition-${index}.json`) const reportsDirectory = join(this.temporaryRoot, `coverage-${index}`) - const jsonReportPath = join(this.temporaryRoot, `partition-${index}.report.json`) + const timingReportPath = join(this.temporaryRoot, `partition-${index}.times.json`) const invocation = pnpmInvocation([ 'exec', 'vitest', @@ -595,10 +596,10 @@ export class CoveragePartitionCoordinator { `--config=${this.relativePath(configPath)}`, '--reporter=default', '--reporter=blob', - '--reporter=json', + `--reporter=${FILE_TIMES_REPORTER}`, `--reporter=${CANONICAL_LOCATIONS_REPORTER}`, `--outputFile.blob=${this.relativePath(blobPath)}`, - `--outputFile.json=${this.relativePath(jsonReportPath)}`, + `--outputFile.fileTimes=${this.relativePath(timingReportPath)}`, `--coverage.reportsDirectory=${this.relativePath(reportsDirectory)}`, ...this.vitestArgs, ], { npm_execpath: this.pnpmEntrypoint }) diff --git a/scripts/release/pack.spec.ts b/scripts/release/pack.spec.ts new file mode 100644 index 0000000000..65fb196527 --- /dev/null +++ b/scripts/release/pack.spec.ts @@ -0,0 +1,241 @@ +/** Real-pnpm regression coverage for release packing, selection, hooks, and payload checks. */ + +import { existsSync, mkdirSync, mkdtempSync, readFileSync, readdirSync, writeFileSync } from 'node:fs' +import { createRequire } from 'node:module' +import { tmpdir } from 'node:os' +import { dirname, join, resolve } from 'node:path' +import { afterEach, describe, expect, it, vi } from 'vitest' +import { officialClientBuildEnvironment, writeClientBuildRecord } from '../client-build-environment.ts' +import { runGate, type Gate } from '../run-gates.ts' +import { removeFixtureSafely } from '../test-fixture-cleanup.ts' +import { releaseFamily, tarballName } from './families.ts' +import { packedIdentity, PUBLISH_ORDER_FILE, readPublishOrder } from './tarball.ts' + +const require = createRequire(import.meta.url) +const packScript = resolve(import.meta.dirname, 'pack.ts') +const tsxHook = require.resolve('tsx/esm') +const pnpmEntry = join(dirname(require.resolve('pnpm')), 'bin/pnpm.cjs') +const roots: string[] = [] +const commit = 'abcdef0' + +function write(path: string, content: string): void { + mkdirSync(dirname(path), { recursive: true }) + writeFileSync(path, content) +} + +function fixture(): string { + const root = mkdtempSync(join(tmpdir(), 'dsh-release-pack-')) + roots.push(root) + write(join(root, 'package.json'), JSON.stringify({ name: '@deepseek-ai/pack-root', private: true, version: '1.2.3' })) + write(join(root, 'pnpm-workspace.yaml'), 'packages:\n - vendor/*\n - native/*\n - packages/*/*\n - apps/*\n') + return root +} + +function packageFixture(root: string, directory: string, name: string, additions: Record = {}): void { + write(join(root, directory, 'package.json'), JSON.stringify({ + name, version: '1.2.3', files: ['lib/index.js', 'hook.cjs'], + scripts: { + prepack: 'node hook.cjs prepack', + prepare: 'node hook.cjs prepare', + postpack: 'node hook.cjs postpack', + }, + ...additions, + })) + write(join(root, directory, 'lib/index.js'), 'export const value = 1;\n') + write(join(root, directory, 'hook.cjs'), [ + "const fs = require('node:fs');", + "const path = require('node:path');", + "const manifest = require('./package.json');", + "fs.appendFileSync(path.join(process.env.DSH_PACK_FIXTURE, 'hooks.log'), `${manifest.name}:${process.argv[2]}\\n`);", + '', + ].join('\n')) +} + +function packGate(root: string, args: string[] = []): Gate { + return { + id: 'release-pack-fixture', + label: 'release pack fixture', + displayCommand: 'pnpm exec node release pack fixture', + command: process.execPath, + args: [pnpmEntry, '--dir', root, 'exec', process.execPath, '--import', tsxHook, packScript, '--family', 'vendor', '--out', 'packed', ...args], + env: { npm_execpath: pnpmEntry, DSH_PACK_FIXTURE: root, DSH_CLIENT_COMMIT_HASH: commit }, + } +} + +async function runPack(root: string, args: string[] = []) { + const controller = new AbortController() + const timer = setTimeout(() => { controller.abort() }, 90_000) + try { + const result = await runGate(packGate(root, args), controller.signal) + expect(result.error).toBeUndefined() + expect(result.aborted).toBe(false) + expect(result.signalCode).toBeNull() + return { + status: result.exitCode, + stderr: result.output.filter(chunk => chunk.stream === 'stderr').map(chunk => chunk.text).join(''), + } + } finally { + clearTimeout(timer) + } +} + +function stopped(pid: number): boolean { + if (process.platform === 'linux') { + try { + return /\)\s+Z\s/.test(readFileSync(`/proc/${pid}/stat`, 'utf8')) + } catch (error) { + if ((error as NodeJS.ErrnoException).code === 'ENOENT') return true + throw error + } + } + try { + process.kill(pid, 0) + return false + } catch (error) { + if ((error as NodeJS.ErrnoException).code === 'ESRCH') return true + throw error + } +} + +function readyHookPids(root: string): number[] { + const ready: unknown = JSON.parse(readFileSync(join(root, 'hook-ready.json'), 'utf8')) + if (ready === null || typeof ready !== 'object' || !('hook' in ready) || !('child' in ready) + || typeof ready.hook !== 'number' || !Number.isSafeInteger(ready.hook) || ready.hook < 1 + || typeof ready.child !== 'number' || !Number.isSafeInteger(ready.child) || ready.child < 1) { + throw new Error('pack fixture hook readiness requires positive process ids') + } + return [ready.hook, ready.child] +} + +afterEach(() => { + for (const root of roots.splice(0)) removeFixtureSafely(root) +}) + +describe('release pack CLI', () => { + it.each([undefined, '1', '2'])('packs only public family members and records their publish order (concurrency=%s)', async (concurrency) => { + const root = fixture() + // Publication places zebra before alpha; pnpm's workspace dev edge points in the opposite direction. + packageFixture(root, 'vendor/alpha', '@deepseek-ai/alpha', { peerDependencies: { '@deepseek-ai/zebra': '*' } }) + packageFixture(root, 'vendor/zebra', '@deepseek-ai/zebra', { devDependencies: { '@deepseek-ai/alpha': 'workspace:*' } }) + write(join(root, 'vendor/zebra/node_modules/@deepseek-ai/alpha/package.json'), JSON.stringify({ name: '@deepseek-ai/alpha', version: '1.2.3' })) + packageFixture(root, 'vendor/private', '@deepseek-ai/private', { private: true }) + packageFixture(root, 'native/unselected', '@deepseek-ai/unselected') + const family = releaseFamily('vendor') + const members = family.publishOrder(family.members(root)).order + const expectedOrder = members.map(tarballName) + + const result = await runPack(root, concurrency === undefined ? [] : ['--concurrency', concurrency]) + + expect(result.status, result.stderr).toBe(0) + expect(readPublishOrder(join(root, 'packed'))).toEqual(expectedOrder) + expect(readdirSync(join(root, 'packed')).sort()).toEqual([...expectedOrder, PUBLISH_ORDER_FILE].sort()) + for (const member of members) { + expect(packedIdentity(join(root, 'packed', tarballName(member)))) + .toEqual({ name: member.name, version: member.version }) + } + const hooks = readFileSync(join(root, 'hooks.log'), 'utf8').trim().split('\n') + const expectedHooks = members.flatMap(member => ['prepack', 'prepare', 'postpack'].map(phase => `${member.name}:${phase}`)) + expect([...hooks].sort()).toEqual([...expectedHooks].sort()) + if (concurrency !== '2') expect(hooks).toEqual(expectedHooks) + }, 120_000) + + it('does not broaden an empty parallel family to the whole workspace', async () => { + const root = fixture() + packageFixture(root, 'vendor/private', '@deepseek-ai/private', { private: true }) + packageFixture(root, 'native/unselected', '@deepseek-ai/unselected') + + const result = await runPack(root, ['--concurrency', '8']) + + expect(result.status, result.stderr).toBe(0) + expect(readFileSync(join(root, 'packed', PUBLISH_ORDER_FILE), 'utf8')).toBe('\n') + expect(readdirSync(join(root, 'packed'))).toEqual([PUBLISH_ORDER_FILE]) + expect(existsSync(join(root, 'hooks.log'))).toBe(false) + }) + + it('leaves no publish order after a pack hook fails', async () => { + const root = fixture() + packageFixture(root, 'vendor/failing', '@deepseek-ai/failing', { scripts: { prepack: 'node failing.cjs' } }) + write(join(root, 'vendor/failing/failing.cjs'), "throw new Error('pack fixture failure')\n") + + const result = await runPack(root, ['--concurrency', '8']) + + expect(result.status).not.toBe(0) + expect(result.stderr).toContain('pack fixture failure') + expect(existsSync(join(root, 'packed', PUBLISH_ORDER_FILE))).toBe(false) + }) + + it('rejects a successful command that did not produce the expected member tarball', async () => { + const root = fixture() + packageFixture(root, 'vendor/expected', '@deepseek-ai/expected', { scripts: { prepack: 'node rename.cjs' } }) + write(join(root, 'vendor/expected/rename.cjs'), [ + "const fs = require('node:fs');", + "const manifest = require('./package.json');", + "manifest.name = '@deepseek-ai/renamed';", + "fs.writeFileSync('package.json', JSON.stringify(manifest));", + '', + ].join('\n')) + + const result = await runPack(root, ['--concurrency', '8']) + + expect(result.status).not.toBe(0) + expect(result.stderr).toContain('@deepseek-ai/expected produced no tarball') + expect(existsSync(join(root, 'packed', 'deepseek-ai-renamed-1.2.3.tgz'))).toBe(true) + expect(existsSync(join(root, 'packed', PUBLISH_ORDER_FILE))).toBe(false) + }) + + it('validates every parallel tarball before recording the publish order', async () => { + const root = fixture() + packageFixture(root, 'packages/util/alpha', '@deepseek-ai/alpha') + packageFixture(root, 'packages/util/zebra', '@deepseek-ai/zebra', { files: ['lib/index.js', 'src/index.ts'] }) + write(join(root, 'packages/util/zebra/src/index.ts'), 'export const source = 1\n') + write(join(root, 'apps/web/dist/index.html'), '
') + writeClientBuildRecord(root, officialClientBuildEnvironment(root, { DSH_CLIENT_COMMIT_HASH: commit })) + + const result = await runPack(root, ['--family', 'dsh', '--concurrency', '8']) + + expect(result.status).not.toBe(0) + expect(result.stderr).toContain('@deepseek-ai/zebra publishes source file package/src/index.ts') + expect(readdirSync(join(root, 'packed')).filter(name => name.endsWith('.tgz'))).toHaveLength(2) + expect(existsSync(join(root, 'packed', PUBLISH_ORDER_FILE))).toBe(false) + }) + + it('stops the ready pack hook and its stalled descendant before fixture cleanup', async () => { + const root = fixture() + packageFixture(root, 'vendor/stalled', '@deepseek-ai/stalled', { scripts: { prepack: 'node stalled.cjs' } }) + write(join(root, 'vendor/stalled/stalled.cjs'), [ + "const { spawn } = require('node:child_process');", + "spawn(process.execPath, ['descendant.cjs'], { stdio: 'inherit' });", + '', + ].join('\n')) + write(join(root, 'vendor/stalled/descendant.cjs'), [ + "const fs = require('node:fs');", + "const path = require('node:path');", + "process.on('SIGTERM', () => {});", + 'setInterval(() => {}, 1000);', + "fs.writeFileSync(path.join(process.env.DSH_PACK_FIXTURE, 'hook-ready.json'), JSON.stringify({ hook: process.ppid, child: process.pid }));", + '', + ].join('\n')) + const controller = new AbortController() + const timer = setTimeout(() => { controller.abort() }, 90_000) + const pending = runGate(packGate(root, ['--concurrency', '8']), controller.signal) + try { + let pids: number[] = [] + await vi.waitFor(() => { + pids = readyHookPids(root) + expect(pids.every(pid => !stopped(pid))).toBe(true) + }, { timeout: 30_000, interval: 25 }) + + controller.abort() + const result = await pending + + expect(result.aborted).toBe(true) + expect(result.status).toBe('failed') + await vi.waitFor(() => { expect(pids.every(stopped)).toBe(true) }, { timeout: 10_000, interval: 25 }) + expect(existsSync(join(root, 'packed', PUBLISH_ORDER_FILE))).toBe(false) + } finally { + clearTimeout(timer) + controller.abort() + await pending + } + }, 120_000) +}) diff --git a/scripts/release/pack.ts b/scripts/release/pack.ts index 6f187e83af..c840f8e762 100644 --- a/scripts/release/pack.ts +++ b/scripts/release/pack.ts @@ -1,6 +1,7 @@ /** - * Pack one release family's whole publish set into a single directory, in - * publish order, and record that order for the publish step. + * Pack one release family's whole publish set into a single directory and + * record its publish order for the publish step. Default serial packs execute + * in that order; explicitly parallel packs use pnpm's bounded scheduler. * * The pack step is the release boundary: it runs without credentials, produces * every tarball from one commit, and hands the publish step exactly those bytes @@ -19,16 +20,13 @@ import { PUBLISH_ORDER_FILE, tarballFiles } from './tarball.ts' const DEFAULT_OUTPUT = 'dist/npm' /** - * Pack one member and check what its tarball carries. + * Check one member's expected tarball and its published files. * @param family - the release family being packed. * @param member - the member to pack. * @param destination - absolute output directory. * @returns The tarball filename. */ -async function packMember(family: ReleaseFamily, member: ReleaseMember, destination: string): Promise { - const invocation = pnpmInvocation(['--dir', member.directory, 'pack', '--pack-destination', destination]) - await runConcurrent(invocation.command, invocation.args) - +function validatePackedMember(family: ReleaseFamily, member: ReleaseMember, destination: string): string { const filename = tarballName(member) const tarball = join(destination, filename) if (!existsSync(tarball)) throw new Error(`${member.name} produced no tarball at ${tarball}`) @@ -36,6 +34,38 @@ async function packMember(family: ReleaseFamily, member: ReleaseMember, destinat return filename } +/** + * Pack serial members in publish order, or batch an explicitly parallel family. + * @param family - Release family whose payload checks apply to every tarball. + * @param members - Exact public members in the recorded publish order. + * @param destination - Absolute output directory. + * @param concurrency - Maximum active package packs; one preserves serial publication order. + * @returns Validated tarball filenames in publish order. + */ +async function packMembers( + family: ReleaseFamily, members: readonly ReleaseMember[], destination: string, concurrency: number, +): Promise { + if (concurrency === 1) { + const order: string[] = [] + for (const member of members) { + const invocation = pnpmInvocation(['--dir', member.directory, 'pack', '--pack-destination', destination]) + await runConcurrent(invocation.command, invocation.args) + order.push(validatePackedMember(family, member, destination)) + } + return order + } + if (members.length === 0) return [] + const invocation = pnpmInvocation([ + 'pack', + '--recursive', + `--workspace-concurrency=${String(concurrency)}`, + ...members.map(member => `--filter=${member.name}`), + '--pack-destination', destination, + ]) + await runConcurrent(invocation.command, invocation.args) + return members.map(member => validatePackedMember(family, member, destination)) +} + /** * @returns The validated `--concurrency` value; 1 (the default) packs the * members one at a time, exactly as the credentialed publish workflows run it. @@ -68,20 +98,7 @@ async function main(): Promise { rmSync(destination, { recursive: true, force: true }) mkdirSync(destination, { recursive: true }) - // Members pack in a bounded pool; the recorded publish order stays the - // members' order regardless of completion order, because each worker writes - // its result at the member's own position. - const order = new Array(members.length) - let cursor = 0 - await Promise.all(Array.from({ length: Math.min(concurrency, members.length) }, async () => { - while (cursor < members.length) { - const index = cursor - cursor += 1 - const member = members[index] - if (member === undefined) break - order[index] = await packMember(family, member, destination) - } - })) + const order = await packMembers(family, members, destination, concurrency) writeFileSync(join(destination, PUBLISH_ORDER_FILE), `${order.join('\n')}\n`) console.log(`release pack: family ${family.id}, ${String(order.length)} tarball(s) in ${values.out ?? DEFAULT_OUTPUT}`) diff --git a/scripts/run-gates.spec.ts b/scripts/run-gates.spec.ts index 76bcafb3ed..a125e9db3b 100644 --- a/scripts/run-gates.spec.ts +++ b/scripts/run-gates.spec.ts @@ -156,6 +156,13 @@ function withPlatform(platform: NodeJS.Platform, action: () => T): T { try { return action() } finally { Object.defineProperty(process, 'platform', original) } } +// Mutates worker-global state: only use for synchronous, non-concurrent graph inspection. +function withNodeVersion(version: string, action: () => T): T { + const original = Object.getOwnPropertyDescriptor(process.versions, 'node')! + Object.defineProperty(process.versions, 'node', { ...original, value: version }) + try { return action() } finally { Object.defineProperty(process.versions, 'node', original) } +} + describe('CI worker allocation', () => { it.each([1, 2, 4, 8, 16, 64])('shares a %i CPU coverage budget without multiplying pools', (cpus) => { const env = ciWorkerEnvironment('ci-coverage', {}, cpus) @@ -258,7 +265,7 @@ describe('gate graph validation', () => { }) expect(scripts['test:bench']).toBe('npm run build:bench && npm run build:web && npm run test:bench:built') expect(scripts['build:bench']).toBe( - 'npm run build:native-system && npm run build:lib && tsdown --config-loader native --config benchmarks/tsdown.config.ts', + 'npm run build:native-system && npm run build:lib:artifacts && tsdown --config-loader native --config benchmarks/tsdown.config.ts', ) expect(scripts['build:native-system']).toBe('tsx native/system/scripts/build.ts --host-addon-only') expect(scripts['test:bench:built']).toBe('vitest run --config vitest.bench.config.ts') @@ -828,6 +835,42 @@ describe('Node compatibility graph', () => { ], }) }) + + it.each([undefined, '1'])('builds the complete Node 22 artifacts once with skip-typecheck=%s', (skipTypecheck) => { + const subject = withNodeVersion('22.19.0', () => + withEnv('DSH_NODE_COMPAT_SKIP_TYPECHECK', skipTypecheck, () => + withPnpmEntrypoint(() => gatesForMode('node-compat')))) + + expect(subject.filter(item => item.displayCommand === 'pnpm run build:artifacts')).toHaveLength(1) + expect(subject.map(item => item.id)).not.toContain('build:web') + expect(subject.find(item => item.id === 'cli-lazy-search-startup-smoke')).toMatchObject({ + needs: ['build'], + env: { DSH_REQUIRE_BUILT_CLI_SMOKE: '1' }, + }) + }) + + it.each(['passed', 'failed'] as const)('runs the Node 22 lazy-search smoke only after a passed build (%s)', async (buildStatus) => { + const subject = withNodeVersion('22.19.0', () => + withEnv('DSH_NODE_COMPAT_SKIP_TYPECHECK', '1', () => + withPnpmEntrypoint(() => gatesForMode('node-compat')))) + let buildCompleted = false + let smokeStarted = false + const results = await runGates(subject, 2, async (item) => { + if (item.id === 'build') { + buildCompleted = true + return resultFor(item, buildStatus) + } + if (item.id === 'cli-lazy-search-startup-smoke') { + expect(buildCompleted).toBe(true) + smokeStarted = true + } + return resultFor(item) + }) + + expect(smokeStarted).toBe(buildStatus === 'passed') + expect(results.find(item => item.gate.id === 'cli-lazy-search-startup-smoke')?.status) + .toBe(buildStatus === 'passed' ? 'passed' : 'skipped') + }) }) describe('Node 24 lane ownership', () => { diff --git a/scripts/run-gates.ts b/scripts/run-gates.ts index f4af0f5a1b..2132c877e7 100644 --- a/scripts/run-gates.ts +++ b/scripts/run-gates.ts @@ -407,13 +407,9 @@ function nodeCompatGates(): Gate[] { } return [ ...typecheck, - pnpmScript('build', 'build', { + pnpmScript('build', 'build:artifacts', { ...typecheck.length === 0 ? {} : { needs: ['typecheck'] }, }), - pnpmScript('build:web', 'build:web', { - label: 'Web frontend build', - needs: ['build'], - }), ...nodeCompatSmokeGates({ cliSmoke: true }), ] } @@ -457,7 +453,7 @@ function nodeCompatSmokeGates(options: { cliSmoke?: boolean } = {}): Gate[] { ], { label: 'CLI lazy-search startup smoke', env: { DSH_REQUIRE_BUILT_CLI_SMOKE: '1' }, - needs: ['build:web'], + needs: ['build'], }), ) } diff --git a/scripts/tests/ci-release-selfhosted.spec.ts b/scripts/tests/ci-release-selfhosted.spec.ts index b7da434bbb..668efe22df 100644 --- a/scripts/tests/ci-release-selfhosted.spec.ts +++ b/scripts/tests/ci-release-selfhosted.spec.ts @@ -146,7 +146,9 @@ for (const [file, jobIds] of [['release.yml', ['dependencies', 'pack']], ['relea const output = family === 'dsh' ? 'dist/npm' : 'dist/npm-vendor' expect(job.steps[0]?.with?.['fetch-depth']).toBe(0) expect(commands).toContain('pnpm run release:verify --family ' + family) - expect(commands).toContain('pnpm run ' + (family === 'dsh' ? 'build:official' : 'build:lib:host')) + expect(commands).toContain(family === 'dsh' + ? "pnpm run ${{ github.event_name == 'pull_request' && 'build:artifacts --profile official' || 'build:official' }}" + : "pnpm run ${{ github.event_name == 'pull_request' && 'build:lib:host:artifacts' || 'build:lib:host' }}") expect(commands).toContain('pnpm run release:pack --family ' + family + ' --out ' + output + ' --concurrency 8') expect(commands).toContain('pnpm run release:verify-packed-install --family ' + family + ' --from ' + output + (family === 'dsh' ? ' --from dist/npm-vendor --from dist/npm-landlock' : '')) diff --git a/scripts/translation-links.spec.ts b/scripts/translation-links.spec.ts index 35efde120e..bad098e23e 100644 --- a/scripts/translation-links.spec.ts +++ b/scripts/translation-links.spec.ts @@ -4,6 +4,7 @@ import { mkdirSync, mkdtempSync, writeFileSync } from 'node:fs' import { tmpdir } from 'node:os' import { join } from 'node:path' import { afterEach, describe, expect, it } from 'vitest' +import { parseMarkdown } from './markdown.ts' import { normalizeTranslationMarkdownLinks, rewriteTranslationLinkLocales, @@ -183,6 +184,33 @@ describe('translation link locale validation', () => { }) describe('translation link rewriting and normalization', () => { + it('reuses an authored tree without changing its offsets, switcher exemption, or reference resolution', () => { + const root = fixture() + const input = '# 指南\n\n[English](guide.md) | 中文\n\n[行内](reference.md?view=full&x=1#overview "标题")\n\n[引用][ref]\n\n[ref]: "标注"\n[ref]: unpaired.md\n' + const context = linkContext(root, 'docs/guide.zh.md') + const tree = parseMarkdown(input) + const originalTree = structuredClone(tree) + + expect(normalizeTranslationMarkdownLinks(input, context, ['guide.md'], tree)).toBe( + '# 指南\n\n[English](guide.md) | 中文\n\n[行内](dsh-translation-target:docs/reference.md?view=full&x=1#overview "标题")\n\n[引用][ref]\n\n[ref]: "标注"\n[ref]: unpaired.md\n', + ) + expect(translationLinkLocaleViolations(input, context, ['guide.md'], tree)).toEqual([ + { + sourcePath: 'docs/guide.zh.md', + line: 5, + url: 'reference.md?view=full&x=1#overview', + expectedUrl: 'reference.zh.md?view=full&x=1#overview', + }, + { + sourcePath: 'docs/guide.zh.md', + line: 9, + url: 'reference.md#tail', + expectedUrl: 'reference.zh.md#tail', + }, + ]) + expect(tree).toEqual(originalTree) + }) + it('rewrites only the destination while preserving the suffix and title', () => { const root = fixture() const input = '[概览](reference.md?view=full&mode=all#overview "reference.md title")\n' diff --git a/scripts/translation-links.ts b/scripts/translation-links.ts index af7c991fd4..d149d44372 100644 --- a/scripts/translation-links.ts +++ b/scripts/translation-links.ts @@ -229,8 +229,8 @@ function visitDocumentLinkNodes( markdown: string, skipTargets: readonly string[], visitor: (node: LinkNode) => void, + tree: Nodes = parseMarkdown(markdown), ): void { - const tree = parseMarkdown(markdown) const switcherOffset = languageSwitcherLinkOffset(tree, markdown, skipTargets) const referencedIdentifiers = new Set() const visitedDefinitions = new Set() @@ -255,20 +255,29 @@ function visitResolvedDocumentLinks( context: TranslationLinkContext, skipTargets: readonly string[], visitor: (node: LinkNode, destination: MarkdownDestination, resolved: ResolvedTranslationLink) => void, + tree?: Nodes, ): void { visitDocumentLinkNodes(markdown, skipTargets, (node) => { if (isExternalOrAbsoluteMarkdownUrl(node.url)) return const destination = markdownDestination(markdown, node) const resolved = resolveTranslationLink(node.url, context, destination.url) if (resolved !== undefined) visitor(node, destination, resolved) - }) + }, tree) } -/** Return one violation per wrong-locale link or link definition. */ +/** + * Return one violation per wrong-locale link or link definition. + * @param markdown - Exact authored Markdown text used for link offsets. + * @param context - Repository and locale used to resolve relative links. + * @param skipTargets - Accepted language-switcher destinations. + * @param tree - Optional GFM parse of this exact Markdown text; never a normalized or edited document. + * @returns Wrong-locale links in document order. + */ export function translationLinkLocaleViolations( markdown: string, context: TranslationLinkContext, skipTargets: readonly string[] = [], + tree?: Nodes, ): TranslationLinkLocaleViolation[] { const violations: TranslationLinkLocaleViolation[] = [] visitResolvedDocumentLinks(markdown, context, skipTargets, (node, destination, resolved) => { @@ -279,7 +288,7 @@ export function translationLinkLocaleViolations( url: destination.url, expectedUrl: resolved.expectedUrl, }) - }) + }, tree) return violations } @@ -297,11 +306,19 @@ export function rewriteTranslationLinkLocales( return { content: applyReplacements(markdown, replacements), rewritten: replacements.length } } -/** Normalize only paired-document locale paths while retaining every other byte and URL suffix. */ +/** + * Normalize only paired-document locale paths while retaining every other byte and URL suffix. + * @param markdown - Exact authored Markdown text used for link offsets. + * @param context - Repository and locale used to resolve relative links. + * @param skipTargets - Accepted language-switcher destinations. + * @param tree - Optional GFM parse of this exact Markdown text; never a normalized or edited document. + * @returns Markdown with paired-document destinations replaced by semantic targets. + */ export function normalizeTranslationMarkdownLinks( markdown: string, context: TranslationLinkContext, skipTargets: readonly string[] = [], + tree?: Nodes, ): string { const replacements: Replacement[] = [] visitResolvedDocumentLinks(markdown, context, skipTargets, (_node, destination, resolved) => { @@ -309,7 +326,7 @@ export function normalizeTranslationMarkdownLinks( destination, `dsh-translation-target:${resolved.pair.source}${resolved.suffix}`, )) - }) + }, tree) return applyReplacements(markdown, replacements) } diff --git a/scripts/translation-pairing-record.ts b/scripts/translation-pairing-record.ts index 3990ec11c2..09dd1c8f41 100644 --- a/scripts/translation-pairing-record.ts +++ b/scripts/translation-pairing-record.ts @@ -68,11 +68,10 @@ function headingText(node: Nodes): string { * Fenced code blocks and generated regions, which the pairing gate requires to * be identical on both sides, are left out of each section's blocks. */ -function sections(markdown: string): Section[] { +function sections(markdown: string, tree: Nodes = parseTranslationMarkdown(markdown)): Section[] { const frontmatter = /^---\n[\s\S]*?\n---(?:\n|$)/.exec(markdown)?.[0].length ?? 0 const result: Section[] = [{ blocks: frontmatter > 0 ? [markdown.slice(0, frontmatter)] : [] }] const regionLines = generatedRegions(markdown).map(region => [region.begin + 1, region.end + 1] as const) - const tree = parseTranslationMarkdown(markdown) if (tree.type !== 'root') throw new Error('Markdown parser returned a non-root tree') for (const node of tree.children) { if ((node.position?.start.offset ?? 0) < frontmatter) continue @@ -106,6 +105,7 @@ function sectionHash(blocks: string[]): string { * @param en - English document text. * @param zh - Chinese document text. * @param context - Link-resolution inputs for both sides. + * @param trees - Optional GFM parses of the exact English and Chinese inputs; normalized text is parsed separately when it changes. * @returns Entries in document order. * @throws Error when the two sides have different heading counts or malformed generated regions. */ @@ -114,9 +114,12 @@ export function computeTranslationPairingRecord( en: string, zh: string, context: TranslationPairingRecordContext, + trees?: { readonly en: Nodes; readonly zh: Nodes }, ): TranslationPairingRecord { - const enSections = sections(normalizeTranslationMarkdownLinks(en, { ...context, sourcePath: paths.source })) - const zhSections = sections(normalizeTranslationMarkdownLinks(zh, { ...context, sourcePath: paths.zh })) + const normalizedEn = normalizeTranslationMarkdownLinks(en, { ...context, sourcePath: paths.source }, [], trees?.en) + const normalizedZh = normalizeTranslationMarkdownLinks(zh, { ...context, sourcePath: paths.zh }, [], trees?.zh) + const enSections = sections(normalizedEn, normalizedEn === en ? trees?.en : undefined) + const zhSections = sections(normalizedZh, normalizedZh === zh ? trees?.zh : undefined) if (enSections.length !== zhSections.length) { throw new Error(`${paths.source} has ${enSections.length - 1} heading(s) but ${paths.zh} has ${zhSections.length - 1}`) } diff --git a/scripts/translation-pairing.spec.ts b/scripts/translation-pairing.spec.ts index 89ab41d09d..2e18c15175 100644 --- a/scripts/translation-pairing.spec.ts +++ b/scripts/translation-pairing.spec.ts @@ -250,6 +250,31 @@ describe('translation pairing records', () => { .toEqual(record(`${en}\nSee [bar](bar.zh.md).\n`, `${zh}\nSee [bar](bar.md).\n`)) }) + it('retains section hashes after normalized links move later heading offsets', () => { + const english = '# Guide\n\n[Reference](bar.md?x=1&y=2#tail)\n\n## Tail\n\nTail body.\n' + const chinese = '# 指南\n\n[参考](bar.zh.md?x=1&y=2#tail)\n\n## 尾部\n\n尾部正文。\n' + const trees = { en: parseTranslationMarkdown(english), zh: parseTranslationMarkdown(chinese) } + const originalTrees = structuredClone(trees) + const computed = computeTranslationPairingRecord(paths, english, chinese, context, trees) + + expect([...computed]).toEqual([ + ['/guide', { en: '3f43f06d1244d874', zh: '021cfe53b05ba084' }], + ['/guide/tail', { en: '945d8537bf1f2b5f', zh: '0a851cb395705d76' }], + ]) + expect(computed).toEqual(record(english, chinese)) + expect(trees).toEqual(originalTrees) + }) + + it('retains section hashes when the authored tree needs no link normalization', () => { + const english = '# Guide\n\nBody.\n' + const chinese = '# 指南\n\n正文。\n' + const trees = { en: parseTranslationMarkdown(english), zh: parseTranslationMarkdown(chinese) } + + expect([...computeTranslationPairingRecord(paths, english, chinese, context, trees)]).toEqual([ + ['/guide', { en: '5e1bfa97a822af7a', zh: '3409c92bef7306bb' }], + ]) + }) + it('refuses sides with different heading counts', () => { expect(() => record(en, `${zh}\n## 额外\n`)).toThrow('docs/foo.md has 3 heading(s) but docs/foo.zh.md has 4') }) diff --git a/scripts/verify-translation-pairing.ts b/scripts/verify-translation-pairing.ts index fe2f1b494b..43489c6b3c 100644 --- a/scripts/verify-translation-pairing.ts +++ b/scripts/verify-translation-pairing.ts @@ -239,8 +239,12 @@ for (const source of [...pairAnchors].sort()) { const sourceText = sourceContent.toString('utf8') const zhText = zhContent.toString('utf8') let current: ReturnType + let sourceTree: ReturnType + let zhTree: ReturnType try { - current = computeTranslationPairingRecord(paths, sourceText, zhText, recordContext) + sourceTree = parseTranslationMarkdown(sourceText) + zhTree = parseTranslationMarkdown(zhText) + current = computeTranslationPairingRecord(paths, sourceText, zhText, recordContext, { en: sourceTree, zh: zhTree }) } catch (error) { errors.push(`${source} ↔ ${zh}: ${error instanceof Error ? error.message : String(error)}`) state.set(source, 'out-of-sync') @@ -265,13 +269,13 @@ for (const source of [...pairAnchors].sort()) { sourcePath: source, isTranslationPairSource, repositoryFileExists, - }, zhSwitcherTargets), + }, zhSwitcherTargets, sourceTree), ...translationLinkLocaleViolations(zhText, { repoRoot: root, sourcePath: zh, isTranslationPairSource, repositoryFileExists, - }, sourceSwitcherTargets), + }, sourceSwitcherTargets, zhTree), ]) { errors.push(`${violation.sourcePath}:${violation.line}: link target ${JSON.stringify(violation.url)} uses the wrong locale; expected ${JSON.stringify(violation.expectedUrl)}`) state.set(source, 'out-of-sync') @@ -309,8 +313,6 @@ for (const source of [...pairAnchors].sort()) { state.set(source, 'out-of-sync') } - const sourceTree = parseTranslationMarkdown(sourceText) - const zhTree = parseTranslationMarkdown(zhText) if (!hasLanguageSwitcher(zhTree, zhText, sourceSwitcherTargets)) { errors.push(`${zh}: missing language switcher — no link to ${basename(source)}`) }