docs(notes): record the local gate speed-ups
The new Agent Note records why type checks generate Typert declarations without bundling, why pre-commit leaves third-party notices to the test lane, how the local worker budget is chosen, and how the documentation gates and type-model.spec avoid repeated work, with the measurements and the evidence that outputs stay the same. Notes whose facts these changes superseded now describe the source-only Typert step and the test-lane ownership of notices freshness. Refs #5852
This commit is contained in:
imccyu committed
1 parent
703c8f659c
commit
fbcb8b0306
15 files changed
+393
-28
No files matched your search
+2
-2
@@ -33,8 +33,8 @@
|
||||
en: a244f6a88fdabf3d
|
||||
zh: e4feedb7297f4865
|
||||
/agent-note-typert-gateway-targeted-method-calls/three-artifact-kinds-and-two-typescript-programs:
|
||||
en: a5134745815f762f
|
||||
zh: 99b6a905abd58e6a
|
||||
en: c5bcd6eb76bf3d42
|
||||
zh: 9df2fbb364643186
|
||||
/agent-note-typert-gateway-targeted-method-calls/the-remote-package-entry:
|
||||
en: 75e977fe0fb71d59
|
||||
zh: 00760c79da173cb7
|
||||
|
||||
@@ -215,7 +215,7 @@ Host lib build
|
||||
→ Vite 构建 Web
|
||||
```
|
||||
|
||||
The top-level `build` runs the step graph in `scripts/build.ts`, which completes the Host bundle and the Remote artifacts before starting Client TypeScript compilation. `build:lib` remains the complete serial lib build for `build:bench`, and its Host half `build:lib:host` serves `typecheck` and the vendor release workflows. A clean build must not depend on stale `.d.ts` files from an earlier build.
|
||||
The top-level `build` runs the step graph in `scripts/build.ts`, which completes the Host bundle and the Remote artifacts before starting Client TypeScript compilation. `build:lib` remains the complete serial lib build for `build:bench`, and its Host half `build:lib:host` serves the vendor release workflows. `typecheck` writes the Remote artifacts through `scripts/generate-typert.ts`, beside Host compilation, before Client TypeScript compilation. A clean build must not depend on stale `.d.ts` files from an earlier build.
|
||||
|
||||
Compiler-backed repository gates that resolve the consumer surface have the same prerequisite even when their primary inputs are source files. The public `typecheck`, `lint`, and `doc-typecheck` commands run the Host contract pass first. The gate scheduler may use their `*:contracts-ready` variants only after an explicit Typert-contract or complete-build dependency, so parallel lanes neither read missing declarations nor run concurrent generators against the same outputs.
|
||||
|
||||
|
||||
@@ -215,7 +215,7 @@ Host lib build
|
||||
→ Vite 构建 Web
|
||||
```
|
||||
|
||||
顶层 `build` 运行 `scripts/build.ts` 中的步骤图，先完成 Host bundle 与 Remote artifact，再启动 Client TypeScript 编译。`build:lib` 仍是 `build:bench` 使用的完整串行 lib 构建，其 Host 部分 `build:lib:host` 供 `typecheck` 与 vendor 发布 workflow 使用。一次干净构建不能依赖上次残留的 `.d.ts`。
|
||||
顶层 `build` 运行 `scripts/build.ts` 中的步骤图，先完成 Host bundle 与 Remote artifact，再启动 Client TypeScript 编译。`build:lib` 仍是 `build:bench` 使用的完整串行 lib 构建，其 Host 部分 `build:lib:host` 供 vendor 发布 workflow 使用。`typecheck` 与 Host 编译并行，经 `scripts/generate-typert.ts` 写出 Remote artifact，再启动 Client TypeScript 编译。一次干净构建不能依赖上次残留的 `.d.ts`。
|
||||
|
||||
即使主要输入是源文件，需要通过编译器解析消费方 surface 的仓库门禁也有相同的前置条件。公共 `typecheck`、`lint` 和 `doc-typecheck` 命令会先执行 Host 约定 pass。门禁调度器仅可在显式的 Typert 约定依赖或完整构建依赖完成后使用对应的 `*:contracts-ready` 变体，使并行 lane 既不会读取缺失的声明，也不会针对同一输出并发运行多个生成器。
|
||||
|
||||
|
||||
+2
-2
@@ -15,8 +15,8 @@
|
||||
en: 194f87f780991068
|
||||
zh: 58096d2339576d20
|
||||
/agent-note-generated-third-party-notices/alternatives-considered:
|
||||
en: 093ddc027e296bf0
|
||||
zh: 6b9e64d99dcefa11
|
||||
en: 7e9d8d7864503e46
|
||||
zh: 5b93a1130cfab260
|
||||
/agent-note-generated-third-party-notices/consequences:
|
||||
en: ed2210ac6c133e6c
|
||||
zh: 7957cb33da842642
|
||||
@@ -36,7 +36,7 @@ The Claude distribution tests prove that only the exact direct SDK identity bypa
|
||||
|
||||
**Keep the hand-written file and review it at release time.** Reviewing a hundred derived rows by eye is exactly the work a generator does correctly, and the file's own claim — that it lists every direct dependency — would be unverified between releases.
|
||||
|
||||
**Verify through a dedicated `doc-sync` gate.** That is how every other generated artifact here is checked, and it was the first shape of this change. It costs a gate process and a scheduler slot in a matrix that is already long, and — worse — its only failure mode is telling a contributor, minutes after they pushed an unrelated dependency bump, to go rerun a generator. Regenerating at commit time removes the interruption, and the assertion inside a spec the test lane already runs keeps the guarantee at no additional CI cost.
|
||||
**Verify through a dedicated `doc-sync` gate.** That is how every other generated artifact here is checked, and it was the first shape of this change. It costs a gate process and a scheduler slot in a matrix that is already long, while the assertion inside a spec the test lane already runs reports the same staleness at no additional CI cost.
|
||||
|
||||
**Enumerate the full transitive closure.** The closure is thousands of packages, already recorded in the lock files with exact versions, and would bury the direct dependencies that a reader actually evaluates. The file points at the lock files and the `pnpm licenses list` renderer instead.
|
||||
|
||||
|
||||
@@ -36,7 +36,7 @@ Claude 分发测试证明：只有精确匹配的直接 SDK 身份会绕过通
|
||||
|
||||
**保留手写文件，发版时人工过一遍。** 用肉眼审阅上百行推导数据，恰恰是生成器能做对的活；而且在两次发版之间，文件自称「列出全部直接依赖」这句话无人验证。
|
||||
|
||||
**用专门的 `doc-sync`（文档同步门禁）校验。** 仓库里其他生成产物都是这么把关的，本次改动最初也是这个形态。但它要在本已冗长的矩阵里再占一个门禁进程和一个调度位；更糟的是，它唯一的失败方式，就是在别人推完一个无关的依赖升级几分钟后，通知对方回去重跑一次生成器。改为提交时重新生成消除了这次打断，而把断言放进测试 lane 本就会跑的 spec 里，则以零额外 CI 成本保住了这项保证。
|
||||
**用专门的 `doc-sync`（文档同步门禁）校验。** 仓库里其他生成产物都是这么把关的，本次改动最初也是这个形态。但它要在本已冗长的矩阵里再占一个门禁进程和一个调度位，而测试 lane 本就会跑的 spec 里的断言能以零额外 CI 成本报告同样的过期。
|
||||
|
||||
**列出完整传递闭包。** 闭包有数千个包，锁文件里已带精确版本，铺开只会淹没读者真正要评估的直接依赖。文件转而指向锁文件与 `pnpm licenses list`。
|
||||
|
||||
|
||||
@@ -15,8 +15,8 @@
|
||||
en: 174241330463841d
|
||||
zh: 7fe361c07e634053
|
||||
/agent-note-split-build-artifacts-and-build-check-with-build-running-both-concurrently/decision/steps-and-dependencies:
|
||||
en: 40f21a0941b9a23a
|
||||
zh: c9f1d702b804ecf2
|
||||
en: 43e78e255306cd43
|
||||
zh: a741d5ef1c7379f3
|
||||
/agent-note-split-build-artifacts-and-build-check-with-build-running-both-concurrently/decision/scheduling-failure-and-output:
|
||||
en: 4b324f92307d3691
|
||||
zh: e0cf4411a747f7e5
|
||||
@@ -27,14 +27,14 @@
|
||||
en: a5d74335200d1476
|
||||
zh: e817deb711f98504
|
||||
/agent-note-split-build-artifacts-and-build-check-with-build-running-both-concurrently/decision/completeness-and-lossless-evidence:
|
||||
en: 7308ab88d2b0926f
|
||||
zh: 773ce413da90bcbc
|
||||
en: 3cd40e0db9a2c89a
|
||||
zh: 3f708be239779d0a
|
||||
/agent-note-split-build-artifacts-and-build-check-with-build-running-both-concurrently/decision/intentional-trade-offs-not-lossless:
|
||||
en: f3a5621dbfcd602d
|
||||
zh: f5beabfa81272ffb
|
||||
en: 14370bc5b54e1278
|
||||
zh: 7ac933092d3e9954
|
||||
/agent-note-split-build-artifacts-and-build-check-with-build-running-both-concurrently/alternatives-considered:
|
||||
en: 54905b912d17c74e
|
||||
zh: 86a298004c8024e3
|
||||
/agent-note-split-build-artifacts-and-build-check-with-build-running-both-concurrently/consequences:
|
||||
en: ce9135e424811303
|
||||
zh: 1737f084156211c3
|
||||
en: 083467f68d0f03ed
|
||||
zh: 93400e9994b8c645
|
||||
@@ -44,7 +44,7 @@ native → host-src ─┬─ host-check ─────────────
|
||||
└─ client-bundle → web ┘
|
||||
```
|
||||
|
||||
- client-src must follow host-bundle: Client packages import `@deepseek-ai/dsh-*/remote`, whose types `lib/typert.remote-client.d.ts` only Typert inside Host tsdown writes. host-check reads no Typert output; clean serial builds have always finished the Host aggregate check before Host tsdown.
|
||||
- client-src must follow host-bundle: Client packages import `@deepseek-ai/dsh-*/remote`, whose types `lib/typert.remote-client.d.ts` Typert writes, inside Host tsdown in these scopes and through `scripts/generate-typert.ts` in the typecheck scope. host-check reads no Typert output; clean serial builds have always finished the Host aggregate check before Host tsdown.
|
||||
- The artifact chain keeps the two non-data edges native → host-src and desktop-bundle → client-src; only the two check steps attach to the chain in parallel.
|
||||
- [`compile-referenced-projects.ts`](../../../../scripts/compile-referenced-projects.ts) owns the package-library step list `libraryBuildSteps`, the references argument construction `referencedProjectsCompilerArgs` (relative paths keep the command below Windows' command-line limit), and the aggregate-check arguments `aggregateCheckArgs`. The vendor release rehearsal (`host-libraries`) and CI benchmarks (`--noCheck`) run the same step list serially.
|
||||
- The check steps use project mode (`tsc -p`): it reads the declarations that references already emitted and compiles no reference, writing incremental state to the aggregates' existing `tsconfig.{host,client}.tsbuildinfo`. `build` and `build:check` run the same command, so their diagnostics are identical.
|
||||
@@ -114,7 +114,7 @@ The serial branch sample overlapped other work on the host (742 s of foreign CPU
|
||||
| Package source type error | host-src / client-src | Not reported (reads existing declarations) | host-src / client-src | Reported | Reported |
|
||||
| Type error only in an aggregate root program's tests, scripts, benchmarks, or website | Not reported | host-check / client-check | host-check / client-check | Reported | consumers, windows-build |
|
||||
| Package tsconfig configuration error | host-src / client-src | Not reported (`tsc -p` does not validate referenced projects' tsconfig options or syntax) | host-src / client-src | Reported | Reported |
|
||||
| Typert analysis error | host-bundle | Not reported | host-bundle | Reported (through the Host bundle) | Reported |
|
||||
| Typert analysis error | host-bundle | Not reported | host-bundle | Reported (through host-typert) | Reported |
|
||||
| Missing prerequisite output | — | Compiler diagnostic (TS6305, TS2307) | — | — | — |
|
||||
|
||||
Negative controls: after a type error (TS2322) is added to `scripts/pnpm-invocation.spec.ts`, which only the Host aggregate root program contains, `build:artifacts` exits 0; `build:check` exits 1 with the Host aggregate check failing and the Client aggregate check passing; `build` exits 1 with 8 of 9 steps passing and 1 failing, and writes no client build record. In a scratch fixture whose referenced project's tsconfig has an invalid `target`, a JSON syntax error, or an unknown compiler option, `tsc -p` exits 0, while `tsc -b` reports TS6046, TS5092 with TS1012, and TS5023 respectively and exits 1; when the referenced project lacks `composite`, both report TS6306.
|
||||
@@ -123,7 +123,7 @@ CI evidence: on PR #6005 both required lanes that run the concurrent `pnpm run b
|
||||
|
||||
### Intentional trade-offs (not lossless)
|
||||
|
||||
- **Some commands skip the aggregate checks:** `build:artifacts`, the launch builds of `dev:web` and `dev:desktop`, and the prerequisite steps of `lint`, `lint:fix`, `doc-typecheck`, and the Typert contracts gate do not type-check the two aggregate root programs, and these prerequisite steps do not produce `apps/desktop/lib`. `build`, `build:check`, `typecheck` (pre-push hook), and the two required CI lanes own completeness; `dev:web` and `dev:desktop` do not refuse to launch because of test type errors.
|
||||
- **Some commands skip the aggregate checks:** `build:artifacts`, the launch builds of `dev:web` and `dev:desktop`, and the prerequisite steps of `lint`, `lint:fix`, `doc-typecheck`, and the Typert contracts gate do not type-check the two aggregate root programs, and these prerequisite steps produce declarations and Typert outputs but no bundle. `build`, `build:check`, `typecheck` (pre-push hook), and the two required CI lanes own completeness; `dev:web` and `dev:desktop` do not refuse to launch because of test type errors.
|
||||
- **`build:check` does not check freshness:** When sources change after `build:artifacts`, `build:check` checks against stale declarations, including Typert Remote declarations; rerun `build:artifacts` first. The complete `build` is unaffected.
|
||||
- **Concurrent `build` raises the memory peak:** This branch's complete `build` peaks at 7.53 GiB of process-tree RSS against 6.11 GiB for the master serial build (+1.42 GiB, medians of 2) and 6.27 GiB for the branch's serial build (n=1); CPU meanwhile falls from 423.1 s to 374.4 s because [the lossless speed-ups note](2026-10-11-lossless-build-speedups.md) removes more work than concurrent contention adds. In the prototype (without the Typert changes), concurrent scheduling alone raises the RSS peak by about 2.1 GiB (5.9 → 8.0 GiB) and CPU by about 3.7%. `DSH_GATE_CONCURRENCY` bounds it; under memory pressure use `DSH_GATE_CONCURRENCY=1 pnpm run build`, or run `build:artifacts` and `build:check` separately. In the windows-build lane, `build` runs concurrently with `docs:build`.
|
||||
- **Passing steps' output is hidden by default:** Only failed steps print their complete output.
|
||||
@@ -150,7 +150,7 @@ CI evidence: on PR #6005 both required lanes that run the concurrent `pnpm run b
|
||||
|
||||
Development iterations can skip the two aggregate root programs with `build:artifacts` (51.6% faster cold than master's `build`) and complete the checks with `build:check` or `build` before declaring work complete. `build` stays complete: the Host aggregate check is off the critical path, the Client aggregate check runs alongside the Client bundle and web, and a cold build is 46.7% faster than master (including the lossless speed-ups note's changes; medians of 2). The costs are a higher memory peak, a little extra CPU from concurrent contention, less passing output by default, and several development and prerequisite commands that do not check test types (see the trade-offs above).
|
||||
|
||||
Removing or changing `build` in the two required lanes, consumers and windows-build, requires first transferring both aggregate checks to another blocking owner. If Typert later gains its own generation step, the step table adds it and client-src depends on it; command semantics and scheduling stay the same.
|
||||
Removing or changing `build` in the two required lanes, consumers and windows-build, requires first transferring both aggregate checks to another blocking owner. The typecheck scope and the lint prerequisites generate Typert contracts without bundling ([local gate speed-ups](2026-10-12-local-gate-speedups.md)); the artifact scopes keep generating them inside Host tsdown.
|
||||
|
||||
client-src's dependency on host-bundle guarantees the "Host and Remote artifacts complete before Client compilation" ordering that [Typert Remote method calls](../architecture/2026-08-02-typert-remote-method-calls.md) describes.
|
||||
|
||||
|
||||
@@ -44,7 +44,7 @@ native → host-src ─┬─ host-check ─────────────
|
||||
└─ client-bundle → web ┘
|
||||
```
|
||||
|
||||
- client-src 必须在 host-bundle 之后：Client 包导入 `@deepseek-ai/dsh-*/remote`，其类型 `lib/typert.remote-client.d.ts` 只由 Host tsdown 中的 Typert 写出。host-check 不读取 Typert 输出，干净的串行构建一直在 Host tsdown 之前完成 Host 聚合检查。
|
||||
- client-src 必须在 host-bundle 之后：Client 包导入 `@deepseek-ai/dsh-*/remote`，其类型 `lib/typert.remote-client.d.ts` 由 Typert 写出：这些 scope 在 Host tsdown 中生成，typecheck scope 经 `scripts/generate-typert.ts` 生成。host-check 不读取 Typert 输出，干净的串行构建一直在 Host tsdown 之前完成 Host 聚合检查。
|
||||
- 产物链保留 native → host-src 与 desktop-bundle → client-src 这两条非数据依赖；只有两个 check 步骤并行挂接到链上。
|
||||
- [`compile-referenced-projects.ts`](../../../../scripts/compile-referenced-projects.ts) 拥有包库步骤列表 `libraryBuildSteps`、references 参数构造 `referencedProjectsCompilerArgs`（相对路径传参，避开 Windows 命令行长度上限）与聚合检查参数 `aggregateCheckArgs`。vendor 发布演练（`host-libraries`）与 CI benchmark（`--noCheck`）串行执行同一步骤列表。
|
||||
- check 步骤使用 project mode（`tsc -p`）：它读取 references 已有的声明而不编译任何 reference，incremental 状态写入聚合配置已有的 `tsconfig.{host,client}.tsbuildinfo`。`build` 与 `build:check` 使用同一条命令，因此诊断相同。
|
||||
@@ -114,7 +114,7 @@ wall 中位数从 249.4 s 降到 172.1 s（−77.3 s，−31%）。各步骤单
|
||||
| 包源码类型错误 | host-src / client-src | 不报告（读取已有声明） | host-src / client-src | 报告 | 报告 |
|
||||
| 只属于聚合根程序的测试、脚本、benchmark、website 类型错误 | 不报告 | host-check / client-check | host-check / client-check | 报告 | consumers、windows-build |
|
||||
| 包 tsconfig 配置错误 | host-src / client-src | 不报告（`tsc -p` 不校验被引用 project 的 tsconfig 选项与语法） | host-src / client-src | 报告 | 报告 |
|
||||
| Typert 分析错误 | host-bundle | 不报告 | host-bundle | 报告（经 Host bundle） | 报告 |
|
||||
| Typert 分析错误 | host-bundle | 不报告 | host-bundle | 报告（经 host-typert） | 报告 |
|
||||
| 前置产物缺失 | — | 编译器诊断（TS6305、TS2307） | — | — | — |
|
||||
|
||||
负对照：在只属于 Host 聚合根程序的 `scripts/pnpm-invocation.spec.ts` 中加入类型错误（TS2322）后，`build:artifacts` 以 0 退出；`build:check` 以 1 退出，其中 Host 聚合检查失败、Client 聚合检查通过；`build` 以 1 退出，9 个步骤中 8 个通过、1 个失败，且不写 client build record。在临时 fixture 中，被引用 project 的 tsconfig 含无效 `target`、JSON 语法错误或未知编译选项时，`tsc -p` 以 0 退出，`tsc -b` 分别报告 TS6046、TS5092 与 TS1012、TS5023 并以 1 退出；被引用 project 缺少 `composite` 时，两者都报告 TS6306。
|
||||
@@ -123,7 +123,7 @@ CI 证据：PR #6005 上，两个运行并发 `pnpm run build` 的必需 lane
|
||||
|
||||
### 有意的取舍（非无损）
|
||||
|
||||
- **部分命令跳过聚合检查：** `build:artifacts`、`dev:web` 与 `dev:desktop` 的启动构建，以及 `lint`、`lint:fix`、`doc-typecheck` 与 Typert contracts gate 的前置步骤，都不类型检查两个聚合根程序，这些前置步骤也不产出 `apps/desktop/lib`。完整性由 `build`、`build:check`、`typecheck`（pre-push hook）与两个必需 CI lane 负责；`dev:web` 与 `dev:desktop` 不会因测试类型错误而拒绝启动。
|
||||
- **部分命令跳过聚合检查：** `build:artifacts`、`dev:web` 与 `dev:desktop` 的启动构建，以及 `lint`、`lint:fix`、`doc-typecheck` 与 Typert contracts gate 的前置步骤，都不类型检查两个聚合根程序，这些前置步骤只产出声明与 Typert 输出，不打包。完整性由 `build`、`build:check`、`typecheck`（pre-push hook）与两个必需 CI lane 负责；`dev:web` 与 `dev:desktop` 不会因测试类型错误而拒绝启动。
|
||||
- **`build:check` 不检查新鲜度：** 在 `build:artifacts` 之后修改源码时，`build:check` 基于旧声明（包括 Typert Remote 声明）检查；需要先重跑 `build:artifacts`。完整 `build` 不受影响。
|
||||
- **并发 `build` 提高内存峰值：** 本分支完整 `build` 的进程树 RSS 峰值为 7.53 GiB，master 串行构建为 6.11 GiB（+1.42 GiB，2 个样本的中位数），本分支串行构建为 6.27 GiB（n=1）；同时 CPU 从 423.1 s 降到 374.4 s，因为[无损提速 Note](2026-10-11-lossless-build-speedups.zh.md) 省去的工作多于并发争用增加的工作。原型（不含 Typert 改动）中，单独的并发调度使 RSS 峰值增加约 2.1 GiB（5.9 → 8.0 GiB）、CPU 增加约 3.7%。上限由 `DSH_GATE_CONCURRENCY` 控制；内存紧张时使用 `DSH_GATE_CONCURRENCY=1 pnpm run build`，或分别运行 `build:artifacts` 与 `build:check`。windows-build lane 中 `build` 与 `docs:build` 同时运行。
|
||||
- **成功步骤的输出默认不显示：** 只有失败步骤打印完整输出。
|
||||
@@ -150,7 +150,7 @@ CI 证据：PR #6005 上，两个运行并发 `pnpm run build` 的必需 lane
|
||||
|
||||
开发迭代可以用 `build:artifacts` 跳过两个聚合根程序（冷启动比 master 的 `build` 快 51.6%），在声明完成前用 `build:check` 或 `build` 补齐检查。`build` 保持完整：Host 聚合检查不在关键路径上，Client 聚合检查与 Client bundle、web 并行，冷启动比 master 快 46.7%（含无损提速 Note 的改动，2 个样本的中位数）。代价是更高的内存峰值、并发争用带来的少量额外 CPU、默认更少的成功输出，以及多个开发与前置命令不检查测试类型（见上文取舍）。
|
||||
|
||||
删除或改变 consumers 与 windows-build 两个必需 lane 的 `build`，必须先把两个聚合检查转交另一处阻塞型 owner。若 Typert 以后获得独立的生成步骤，步骤表新增该步骤，client-src 改为依赖它；命令语义与调度不变。
|
||||
删除或改变 consumers 与 windows-build 两个必需 lane 的 `build`，必须先把两个聚合检查转交另一处阻塞型 owner。typecheck scope 与 lint 类前置步骤不打包即生成 Typert 约定（[本地门禁提速](2026-10-12-local-gate-speedups.zh.md)）；产物 scope 仍在 Host tsdown 中生成。
|
||||
|
||||
[Typert Remote 方法调用](../architecture/2026-08-02-typert-remote-method-calls.zh.md)描述的“Client 编译前先完成 Host 与 Remote 产物”由 client-src 对 host-bundle 的依赖保证。
|
||||
|
||||
|
||||
@@ -39,8 +39,8 @@
|
||||
en: 80c2ddc5fdff0b5f
|
||||
zh: f5ce309a2e88ff4f
|
||||
/agent-note-build-speed-ups-that-keep-outputs-unchanged/alternatives-considered:
|
||||
en: 186e42f2f5481c8d
|
||||
zh: 4a1c79a391a268b0
|
||||
en: 4372c82ef0c944dd
|
||||
zh: 5fb70d499baadea7
|
||||
/agent-note-build-speed-ups-that-keep-outputs-unchanged/consequences:
|
||||
en: 93a96d3ed398f348
|
||||
zh: 93b7c7405be981a6
|
||||
@@ -118,7 +118,7 @@ All numbers come from one shared Linux host (256 vCPU, 4 TB memory, load average
|
||||
- **Narrow `paths` to each face's reference closure (item 1):** The 328-entry version that keeps wildcard keys checks in 55.73 s against 59.08 s (one sample), within noise; the version without wildcard keys produces 18 diagnostics.
|
||||
- **Split the Client test aggregate program (item 1):** Every sub-program must inject the 242 files carrying module augmentations to pass, and the visible location count stays at 33–49; splitting into 2, 4, or 8 groups leaves the Check sum nearly flat while user CPU rises by 9%, 30%, and 57%, lowering only wall, and the 4- and 8-group splits each produce 1–2 diagnostics.
|
||||
- **Cache resolution results and make the discovery queue O(1) (Typert):** After items 2–4 only 28 surface walks remain, about 0.18 s, below noise.
|
||||
- **Move Typert into a separate process or build step (Typert):** The generator would have to be compiled first, and `@deepseek-ai/dsh-client-file-upload/types` would need a `paths` entry (otherwise Typert reads `lib/types/types.d.ts` while `tsc` writes it); after item 6 the worker already overlaps bundling, leaving at most about 1 s.
|
||||
- **Move Typert into a separate build step in artifact builds (Typert):** After item 6 the worker already overlaps bundling, leaving at most about 1 s. Type checks, which need no bundle, run a source-only generation step ([local gate speed-ups](2026-10-12-local-gate-speedups.md)).
|
||||
- **Filter `transform` by code content (item 7):** rolldown interprets `^` and `\s` with Rust regex semantics, which differ from JavaScript on lone CR, U+2028, and BOM, so a decorated module could be missed.
|
||||
- **Build a process-wide manifest index on `globalThis` (item 8):** The prototype saves about as much time as item 8, but it is process-global state without an owner, violating the shared-state ownership rule in the [package rules](../../../../packages/AGENTS.md).
|
||||
- **Disable tsdown's gzip report (item 8):** `--no-report` lowers Client tsdown by 0.95 s, within the 1.56 s baseline spread.
|
||||
|
||||
@@ -118,7 +118,7 @@ Status: implemented
|
||||
- **把 `paths` 收窄为各 face 的 reference closure（第 1 项）：** 保留通配键的 328 条版本 Check 为 55.73 s 对 59.08 s（1 个样本），在噪声内；去掉通配键的版本产生 18 个诊断。
|
||||
- **拆分 Client 测试聚合程序（第 1 项）：** 每个子程序都必须注入 242 个含模块增强的文件才能通过检查，可见位置数仍为 33–49；拆成 2、4、8 组时 Check 总和基本不变，user CPU 分别增加 9%、30%、57%，只降低 wall，且 4 组与 8 组各出现 1–2 个诊断。
|
||||
- **缓存解析结果并把发现队列改为 O(1)（Typert）：** 第 2–4 项之后只剩 28 次表面遍历，约 0.18 s，低于噪声。
|
||||
- **把 Typert 移到独立进程或独立构建步骤（Typert）：** 需要先编译 generator 自身，并为 `@deepseek-ai/dsh-client-file-upload/types` 补充 `paths` 条目（否则它读取 `tsc` 正在写入的 `lib/types/types.d.ts`）；第 6 项之后 worker 已与打包重叠，剩余收益不超过约 1 s。
|
||||
- **在产物构建中把 Typert 移到独立构建步骤（Typert）：** 第 6 项之后 worker 已与打包重叠，剩余收益不超过约 1 s。不需要 bundle 的类型检查运行只读源码的生成步骤（[本地门禁提速](2026-10-12-local-gate-speedups.zh.md)）。
|
||||
- **按代码内容过滤 `transform`（第 7 项）：** rolldown 以 Rust 正则语义解释 `^` 与 `\s`，在孤立 CR、U+2028 与 BOM 上与 JavaScript 不同，可能漏掉含装饰器的模块。
|
||||
- **在 `globalThis` 上建立进程级 manifest 索引（第 8 项）：** 原型收益与第 8 项相当，但它是没有 owner 的进程全局状态，违反[包规则](../../../../packages/AGENTS.md)中的共享状态归属要求。
|
||||
- **关闭 tsdown 的 gzip 报告（第 8 项）：** `--no-report` 使 Client tsdown 降低 0.95 s，在 1.56 s 的基线跨度内。
|
||||
|
||||
@@ -0,0 +1,49 @@
|
||||
# Bilingual-pair consistency record for 2026-10-12-local-gate-speedups.md (docs/i18n/README.md): per heading
|
||||
# section, a hash of its English and Chinese blocks outside code blocks and generated regions.
|
||||
# After editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write .agents/notes/implemented/process/2026-10-12-local-gate-speedups.md
|
||||
/agent-note-local-gate-hook-and-test-speed-ups:
|
||||
en: d8edb2c26730e9af
|
||||
zh: fb69bf2e7b602d4c
|
||||
/agent-note-local-gate-hook-and-test-speed-ups/problem:
|
||||
en: 1e2163ec378c44ff
|
||||
zh: d93cb5970896fccb
|
||||
/agent-note-local-gate-hook-and-test-speed-ups/decision:
|
||||
en: 9984b4ad646bfaf2
|
||||
zh: 753287ae67e29605
|
||||
/agent-note-local-gate-hook-and-test-speed-ups/decision/measurement-conditions:
|
||||
en: 7430353afba96352
|
||||
zh: 6d2a09619a67ecba
|
||||
/agent-note-local-gate-hook-and-test-speed-ups/decision/1-type-checks-generate-typert-declarations-without-bundling:
|
||||
en: b3929f2d2f4d1942
|
||||
zh: 98116a29bd25cea8
|
||||
/agent-note-local-gate-hook-and-test-speed-ups/decision/2-the-pre-commit-hook-does-not-regenerate-third-party-notices:
|
||||
en: e2e344f184621172
|
||||
zh: 97d8785aaf9105bf
|
||||
/agent-note-local-gate-hook-and-test-speed-ups/decision/3-local-default-worker-budget:
|
||||
en: 35288da41d222db0
|
||||
zh: 1ace882f8e7bd213
|
||||
/agent-note-local-gate-hook-and-test-speed-ups/decision/4-repository-reference-candidates-resolve-against-one-object-listing:
|
||||
en: 2c945250337c88cf
|
||||
zh: d4bfa34cc858782b
|
||||
/agent-note-local-gate-hook-and-test-speed-ups/decision/5-documentation-gates-parse-each-document-once:
|
||||
en: c22d38399a2bb87f
|
||||
zh: 9ae97e5a6e3fbf48
|
||||
/agent-note-local-gate-hook-and-test-speed-ups/decision/6-repository-file-walks-list-each-directory-once:
|
||||
en: 2f0d35b0a688c938
|
||||
zh: 7184fd260c066a01
|
||||
/agent-note-local-gate-hook-and-test-speed-ups/decision/7-cli-help-reads-run-concurrently-doc-graph-probes-are-memoized:
|
||||
en: 965503ba8a31c039
|
||||
zh: fe02a394a677d018
|
||||
/agent-note-local-gate-hook-and-test-speed-ups/decision/8-type-model-spec-ts-runs-against-an-in-memory-workspace:
|
||||
en: d519ec362903aa2f
|
||||
zh: f49ee6547cb954f6
|
||||
/agent-note-local-gate-hook-and-test-speed-ups/decision/repository-wide-comparison:
|
||||
en: 7c9388dc859e0f93
|
||||
zh: 5df1fc58eb7a46e8
|
||||
/agent-note-local-gate-hook-and-test-speed-ups/alternatives-considered:
|
||||
en: 763d94734b8967cf
|
||||
zh: abe980d486e1d7da
|
||||
/agent-note-local-gate-hook-and-test-speed-ups/consequences:
|
||||
en: ca00b04567d09408
|
||||
zh: 97011043c60f4bc9
|
||||
@@ -0,0 +1,158 @@
|
||||
# Agent Note: Local gate, hook, and test speed-ups
|
||||
|
||||
Status: implemented
|
||||
|
||||
English | [中文](2026-10-12-local-gate-speedups.zh.md)
|
||||
|
||||
## Problem
|
||||
|
||||
On the baseline, master at the merge of #6013, local runs of the Git hooks, `pnpm run test`, `test:docs`, and `doc-sync` spend most of their wall time on work that their results do not depend on:
|
||||
|
||||
- **Typert declarations from the bundler.** `typecheck`, the command the pre-push hook runs, executes `build:lib:host` (Host `tsc -b tsconfig.host.json`, the Host tsdown pass, and the Desktop bundle) before `tsc -b tsconfig.client.json`. The prerequisite of `lint`, `lint:fix`, `doc-typecheck`, and the CI `typert-contracts` gate runs Host package compilation and then the Host tsdown pass (`compile-referenced-projects.ts host-libraries`). Type checks read only package declarations and Typert declarations, yet `typecheck` with nothing to recheck takes 34.1/35.0 s and the prerequisite takes 52.2/29.6 s.
|
||||
- **Third-party notices on every commit.** The pre-commit hook regenerates `THIRD_PARTY_NOTICES.md` whenever a staged file is a generator input. To find browser-side third-party imports, the generator test-bundles 75 Client packages and runs one Vite build, so every package source file is an input; on a commit that stages one file under a package `src/` subdirectory, the hooks take 16.5/17.8 s, of which the notices job takes 12.4/13.4 s.
|
||||
- **Host CPUs as the local worker count.** `os.availableParallelism()` reports the host's CPUs (256 on the measurement host) rather than the container's CPU quota (64). With that default, `pnpm run test` starts 255 Vitest forks, peaks at 55.3 GiB RSS, and fails 12 additional files (14 timeouts) intermittently, while `doc-sync`, `test:docs`, `hygiene`, and `check:all` run at most 4 gates concurrently.
|
||||
- **One Git prefix lookup per candidate.** `verify-repository-references` resolves each of about 26,000 abbreviated hexadecimal candidates with a Git prefix lookup through `git cat-file --batch-check`; the lookups take 135.7 s of the gate's 166.8 s.
|
||||
- **Repeated Markdown parsing.** markdown links parses each anchor target twice more and probes target existence without a cache; translation pairing globs four times and parses each document again after link normalization; the package README model-experience and limitations checks each parse every README three times; type equivalence parses all 1,835 documents; Session format references parses each pair four times.
|
||||
- **Repeated directory reads.** `uniqueRepoFiles`, the walker behind markdown wrap, doc refs, and package paths, reads each directory once per pattern and `**` level and resolves every matched file with `realpath`.
|
||||
- **Serial help launches and repeated compiler probes.** `gen-cli-help` launches its 12 top-level source-mode `dsh … --help` processes one after another, each waiting mostly on profile boot I/O (49.9/53.9 s); `gen-doc-graphs` builds several compiler hosts and programs that repeat the same `ts.sys` file-system probes (48.7/44.7 s).
|
||||
- **A test file bound by file-system probes.** Each of the 61 cases of the Typert generator's `type-model.spec.ts` copies its fixture to disk and runs a full analysis whose four TypeScript programs each load the DOM library, `@types/node`, and zod v4, about 22,000 stat calls per analysis. The file takes 316.3/329.1 s and is the long tail of `pnpm run test`.
|
||||
|
||||
## Decision
|
||||
|
||||
The eight decisions below shorten those runs. Each item states the evidence that its outputs and diagnostics stay the same, or the intended difference it accepts.
|
||||
|
||||
| # | Decision | Affected commands | Measured wall | Outputs and diagnostics | Intended behavior difference |
|
||||
|---|---|---|---|---|---|
|
||||
| 1 | Type checks generate Typert declarations without bundling | `typecheck`, `lint`, `lint:fix`, `doc-typecheck`, CI `typert-contracts` | `typecheck` 34.1/35.0 → 21.9/20.0 s; prerequisite 52.2/29.6 → 12.5/11.3 s | 140 Typert files byte-identical; the same programs are checked | No runnable bundle and no client build record |
|
||||
| 2 | The pre-commit hook does not regenerate third-party notices | `pre-commit` | 16.5/17.8 → 4.3/4.5 s | Freshness asserted in the test lane | Stale notices fail in the test lane, not at commit |
|
||||
| 3 | Local default worker budget | run-gates aggregates, `build`, `typecheck`, `pnpm run test`, Oxlint, publint | `test` at 255 workers: 243.7 s, 55.3 GiB; `test:docs` 74.7 → 50.9 s | Same checks | Local defaults ignore CPUs above 12 or 48 |
|
||||
| 4 | Repository-reference candidates resolve against one object listing | `verify-repository-references` | 166.8 → 15.7 s | 0 classification differences over 27,050 candidates | None |
|
||||
| 5 | Documentation gates parse each document once | markdown links, translation pairing, README model experience and limitations, type equivalence, Session format references | markdown links 53.4/51.6 → 25.9/25.1 s; type equivalence 23.0/22.5 → 6.0/5.5 s | Byte-identical gate output | None |
|
||||
| 6 | Repository file walks list each directory once | markdown wrap, doc refs, package paths | doc refs 8.8/7.0 → 3.2/3.4 s | Identical file lists | doc refs does not read nested `lib` directories |
|
||||
| 7 | CLI help reads run concurrently; doc-graph probes are memoized | `verify-cli-help`, `verify-doc-graphs` | 49.9/53.9 → 6.9/7.2 s; 48.7/44.7 → 32.7/34.2 s | Byte-identical pages | CLI help uses about 3.6 GB at peak |
|
||||
| 8 | `type-model.spec.ts` runs against an in-memory workspace | `pnpm run test` | 316.3/329.1 → 14.65/15.04 s | Same 61 cases and assertions; one snapshot line differs | Stubs replace zod, `@types/node`, and the DOM library outside the Emitter case |
|
||||
|
||||
### Measurement conditions
|
||||
|
||||
All numbers are wall times from one shared Linux container with a 64-CPU quota on a host that reports 256 CPUs, at a one-minute load average of 20–150, so samples carry noise. The working tree lives on a network filesystem, where each file-system probe costs more than on a local disk. `a/b` lists two samples, and `x → y` compares the baseline with this implementation.
|
||||
|
||||
### 1. Type checks generate Typert declarations without bundling
|
||||
|
||||
- **Purpose:** Give `typecheck` and the prerequisite of `lint`, `lint:fix`, `doc-typecheck`, and the CI `typert-contracts` gate the declarations they read, package declarations from `tsc` and Typert declarations, without the Host tsdown pass or the Desktop bundle.
|
||||
- **Change:** `typecheck` is `tsx scripts/build.ts --scope typecheck`, which runs the steps below through the run-gates scheduler ([`scripts/build.ts`](../../../../scripts/build.ts)). The scope builds no native addon, tsdown bundle, Desktop bundle, or Web output, writes no client build record, and leaves runnable `lib/` output as the last `build:artifacts` left it; run `build:artifacts` before launching or testing built output. The prerequisite of `lint`, `lint:fix`, `doc-typecheck`, and `typert-contracts` is `compile-referenced-projects.ts host-contracts`, which runs host-src beside host-typert ([`compile-referenced-projects.ts`](../../../../scripts/compile-referenced-projects.ts)). `build`, `build:artifacts`, benchmark builds, and the PR rehearsal in `release-vendor.yml` run Typert inside the Host tsdown pass.
|
||||
|
||||
| Step | Command | `needs` |
|
||||
|---|---|---|
|
||||
| host-src | `node --max-old-space-size=4096 <tsc> -b <references of tsconfig.host.json>` | — |
|
||||
| host-typert | `node --import tsx/esm scripts/generate-typert.ts` | — |
|
||||
| client-src | `<tsc> -b <references of tsconfig.client.json>` | host-src, host-typert |
|
||||
| host-check | `node --max-old-space-size=4096 <tsc> -b tsconfig.host.json` | host-src |
|
||||
| client-check | `<tsc> -b tsconfig.client.json` | client-src |
|
||||
|
||||
- **Shared generation:** [`scripts/generate-typert.ts`](../../../../scripts/generate-typert.ts) calls `WorkspaceTypertGenerator.generateContributors()` for `WORKSPACE_TYPERT_FACES` ([`scripts/typert-faces.ts`](../../../../scripts/typert-faces.ts)), the faces the Host tsdown plugin also generates, and writes the result with `writeWorkspaceArtifacts()` from [`artifact-files.ts`](../../../../packages/typert/generator/src/artifact-files.ts), the writer the plugin uses. Like the plugin, it skips TypeScript diagnostics, which Host package compilation reports; a Typert analysis error fails host-typert as it fails the Host bundle in `build`. host-typert runs under Node with tsx's ESM hook rather than through `pnpm exec`, whose startup would lengthen the path to Client compilation.
|
||||
- **Aggregate checks in build mode:** host-check and client-check start after their own face's package compilation, so build mode finds every reference up to date, checks only the aggregate root program, and skips a root program whose tsbuildinfo is current.
|
||||
- **Source-only analysis:** host-typert can run beside host-src because its Host program reads no workspace build output. `tsconfig.base.json` maps `@deepseek-ai/dsh-client-file-upload/types` to source; without that alias the specifier resolves through package `exports` to `lib/types/types.d.ts`. With it, the Host program loads 1,289 files, none of them under a workspace package's `lib/`. `generate-typert.ts` enables `rejectBuildOutputs`: [`build-output-guard.ts`](../../../../packages/typert/generator/src/build-output-guard.ts) fails analysis with a `TypertAnalysisError` when a face program loads a file from a workspace package's `lib/` directory, naming each import specifier that resolved there and the `paths` entry that maps it to source. The tsdown plugin leaves the option off because in `build` Typert always runs after Host package compilation. The [Typert generator README](../../../../packages/typert/generator/README.md) documents the writers and the option.
|
||||
- **Why it is lossless:** No workspace tsconfig sets `allowJs`, every `exports` `types` condition points into `lib/types/` or at `lib/typert.*`, and every tsdown config sets `dts: false`, so no type check reads tsdown output. The 140 Typert files (28 packages × 5 outputs) are byte-identical across the baseline tsdown pass, the current tsdown pass, `generate-typert.ts` after every Typert output is deleted, and `generate-typert.ts` with all 344 `lib/types` directories hidden; with one Remote method temporarily added to `packages/goal/goal`, the tsdown pass and `generate-typert.ts` also write identical files. `typecheck` checks every package project of both faces and both aggregate root programs, the programs that `build` checks.
|
||||
- **Performance:** `typecheck` with nothing to recheck takes 34.1/35.0 → 21.9/20.0 s. When both aggregate root programs must be rechecked, it takes 83.5 → 76.0/72.1 s, and its critical path is the Host aggregate check at about 66–70 s. The prerequisite of `lint` and `doc-typecheck` takes 52.2/29.6 → 12.5/11.3 s.
|
||||
- **Trade-off:** `typecheck`, `lint`, `lint:fix`, `doc-typecheck`, and `typert-contracts` write Typert declarations but no bundle, so runnable `lib/` JavaScript can lag behind those declarations until the next `build:artifacts`.
|
||||
- **Relation to the build:** [The lossless build speed-ups note](2026-10-11-lossless-build-speedups.md) rejects moving Typert out of the Host tsdown pass for builds, where its Worker Thread already overlaps bundling and a separate step saves at most about 1 s. Type-check commands run no bundler, so generating without one removes the whole Host tsdown pass and the Desktop bundle from their critical path. [The build split note](2026-10-11-build-artifacts-check-split.md) lists every caller and its command.
|
||||
- **Known gap:** Incremental `tsc -b` does not record `lib/typert.remote-client.d.ts`, which Client sources reach through package `exports`, as an input of the consuming project; when a Remote signature changes and no Client source does, an incremental check does not recheck the consumers. The gap is independent of which pass writes the declarations; Issue #6022 tracks it.
|
||||
|
||||
### 2. The pre-commit hook does not regenerate third-party notices
|
||||
|
||||
- **Change:** The `pre-commit` hook in [`lefthook.yml`](../../../../lefthook.yml) has no third-party notices job. [`gen-third-party-notices.spec.ts`](../../../../scripts/gen-third-party-notices.spec.ts) asserts the committed bytes of `THIRD_PARTY_NOTICES.md` in the test lane, including the required `node 24 / coverage` lane, and names `pnpm run gen-third-party-notices` when they are stale. [The generated third-party notices note](2026-07-30-generated-third-party-notices.md) owns this freshness decision.
|
||||
- **Performance:** A commit that stages one file under a package `src/` subdirectory spends 4.3/4.5 s in the hooks instead of 16.5/17.8 s, of which the notices job took 12.4/13.4 s.
|
||||
- **Trade-off:** A commit that leaves the notices stale fails in the test lane after push rather than being corrected at commit time.
|
||||
|
||||
### 3. Local default worker budget
|
||||
|
||||
- **Purpose:** Size default parallelism for a shared workstation or container, where Node reports host CPUs rather than the CPU quota.
|
||||
- **Change:** [`scripts/parallelism.ts`](../../../../scripts/parallelism.ts) owns the default. With `CI=true` it is the CPU count Node reports. Otherwise it is capped at 12 (`LOCAL_PARALLELISM_CAP`) for the run-gates scheduler, `build`, `typecheck`, Oxlint, and publint, and at 48 (`LOCAL_TEST_PARALLELISM_CAP`) for Vitest forks and the local coverage budget. `doc-sync`, `test:docs`, `hygiene`, and `check:all`, whose gates each build a full `ts.Program`, run at most 8 gates concurrently. Explicit `DSH_GATE_CONCURRENCY`, `DSH_OXLINT_THREADS`, `DSH_COVERAGE_MAX_WORKERS`, and Vitest `--maxWorkers` take precedence; in CI, Vitest and Oxlint keep their own defaults unless the run-gates CI environment sets these variables. The [development guide](../../../../docs/development.md) states the defaults for contributors.
|
||||
|
||||
`pnpm run test` at four fork counts, with the disk-backed `type-model.spec.ts` that item 8 replaces, which sets the wall of the 32-worker run:
|
||||
|
||||
| Vitest workers | Wall | Peak RSS | Additional intermittent failures | Every file except the slowest finished at |
|
||||
|---|---|---|---|---|
|
||||
| 255 (the reported CPU count) | 243.7 s | 55.3 GiB | 12 files, 14 timeouts | about 135–170 s |
|
||||
| 32 | 310.4 s | 15.7 GiB | 4 files, 4 timeouts | 193 s |
|
||||
| 24 | 337.2 s | 13.2 GiB | 5 files, 5 timeouts | 233 s |
|
||||
| 12 | 410.5 s | 8.4 GiB | 2 files, 3 timeouts | 406 s |
|
||||
|
||||
- **Oxlint:** Oxlint's own default is 64 threads, read from the cgroup quota; at 12 threads its wall stays about 87–90 s with about 15% less CPU, and at 8 threads it takes about 4 s longer.
|
||||
- **Documentation aggregates:** `test:docs` takes 50.9 s at 8 concurrent gates against 74.7 s at 4. `doc-sync`, measured without item 4, takes 341.8, 295.0, and 253.2 s at 4, 8, and 12 concurrent gates, with peak RSS of 10.0, 13.5, and 16.3 GiB.
|
||||
- **Trade-off:** A machine whose CPU quota exceeds the caps gets no more default parallelism than 12 or 48 and needs the override variables; a container whose quota is below a cap still oversubscribes, because Node reports host CPUs. At 8 concurrent gates `doc-sync` peaks at 13.5 GiB RSS against 10.0 GiB at 4; 12 concurrent gates finish 41.8 s sooner at 16.3 GiB.
|
||||
|
||||
### 4. Repository-reference candidates resolve against one object listing
|
||||
|
||||
- **Change:** [`verify-repository-references.ts`](../../../../scripts/verify-repository-references.ts) lists every object name once with `git cat-file --batch-all-objects --batch-check=%(objectname)`, about 1.2 million objects in 0.6 s, sorts the list, and binary-searches each lowercase candidate prefix in memory; one `--batch-check=%(objectname) %(objecttype)` call then reads the types of the matched objects. A candidate is a commit reference only when its prefix matches exactly one available object and that object is a commit, the rule Git's own prefix lookup applies; the listing covers unreachable and alternate objects, as Git's lookup does.
|
||||
- **Performance:** 166.8 → 15.7 s.
|
||||
- **Why it is lossless:** Git's per-candidate prefix lookup and the in-memory search classify all 27,050 real and synthetic candidates identically, including ambiguous prefixes, blob and tree prefixes, and random strings.
|
||||
- **Trade-off:** The listing is about 49 MiB of Git output for 1.2 million objects, read under a 1 GiB output limit.
|
||||
|
||||
### 5. Documentation gates parse each document once
|
||||
|
||||
- **Change:** Each gate below parses a Markdown document once and passes the tree to every consumer in that run.
|
||||
|
||||
| Gate | Single-parse mechanism | Wall |
|
||||
|---|---|---|
|
||||
| markdown links ([`verify-md-links.ts`](../../../../scripts/verify-md-links.ts)) | One per-run cache parses each file once for its destinations and anchors and probes each resolved target once | 53.4/51.6 → 25.9/25.1 s |
|
||||
| translation pairing ([`verify-translation-pairing.ts`](../../../../scripts/verify-translation-pairing.ts)) | One glob covers every scope pattern; the record reuses the authored tree under the guard below | 51.0/47.1 → 30.3/29.3 s |
|
||||
| package README model experience, package README limitations | Each check parses a README once and passes the tree to `markdownProseLines()` and `markdownHeadingLines()` | about 12 → 5.9 s each |
|
||||
| type equivalence ([`verify-type-equiv.ts`](../../../../scripts/verify-type-equiv.ts)) | A text prefilter skips documents that cannot hold a source-equivalence fence; each manifest TypeScript source is parsed once | 23.0/22.5 → 6.0/5.5 s |
|
||||
| Session format references | The persistence renderer passes the trees it parses to the pairing record | 10.2 → 5.3 s |
|
||||
|
||||
- **Pairing reuse guard:** `normalizedTopLevelBlocks()` in [`translation-pairing-record.ts`](../../../../scripts/translation-pairing-record.ts) reuses the authored tree for normalized text only when every link-normalization edit replaces a complete, title-free destination that does not start a line, the authored text equals the node's `url`, and both the authored and the replacement text contain only `[A-Za-z0-9._/:#%-]`. Those characters cannot end a destination or start or end another construct, so the normalized text parses to the same nodes with shifted offsets. Any other edit falls back to a full parse of the normalized text.
|
||||
- **Type-equivalence prefilter:** [`type-equiv-prefilter.ts`](../../../../scripts/type-equiv-prefilter.ts) keeps a document for parsing when it contains `type-equiv` or `public-api`, or when a line containing `` ``` `` or `~~~` also contains `&` or `\`. Parsing changes an info string only by decoding character references and backslash escapes, so `ts type-equiv` and `ts type\-equiv` keep their documents; [`type-equiv-prefilter.spec.ts`](../../../../scripts/type-equiv-prefilter.spec.ts) covers both forms.
|
||||
- **Why it is lossless:** Every gate's output on the repository is byte-identical, and seeded faults are reported identically. For pairing, the top-level blocks of all 1,886 corpus documents and the records of all 943 pairs match those of a full reparse, as do randomized destinations.
|
||||
|
||||
### 6. Repository file walks list each directory once
|
||||
|
||||
- **Change:** `uniqueRepoFiles()` in [`scripts/repo-files.ts`](../../../../scripts/repo-files.ts) shares one directory-listing cache across all patterns of a call, so each directory is read at most once, and calls `realpath` only for matches reached through a symbolic link; a regular file reached through real directories canonicalizes to the repository's realpath joined with its relative path. Its optional `skipDirectory` predicate keeps the walker out of a directory whose every path the caller excludes; [`verify-doc-refs.ts`](../../../../scripts/verify-doc-refs.ts) skips every `lib` directory below the first path segment, whose files it excludes in any case.
|
||||
- **Performance:** markdown wrap 25.8/25.0 → 21.3/21.2 s; doc refs 8.8/7.0 → 3.2/3.4 s; package paths 9.9/8.1 → 7.2/5.5 s.
|
||||
- **Why it is lossless:** All three gates receive identical file lists, and a broken or cyclic symbolic link still fails through `realpath`.
|
||||
- **Behavior difference:** doc refs does not list a skipped `lib` directory, so an unreadable one does not fail the gate with EACCES.
|
||||
|
||||
### 7. CLI help reads run concurrently; doc-graph probes are memoized
|
||||
|
||||
- **CLI help:** `prefetchHelp()` in [`scripts/gen-cli-help.ts`](../../../../scripts/gen-cli-help.ts) starts the 12 top-level `dsh … --help` reads (the launcher, `plugin`, and each profile's shorthand and `--profile` forms) within the default worker budget, each with its own temporary `DSH_HOME` and `DSH_AGENTS_HOME`, and the in-order traversal consumes them; nested reads share the same limit. A failed read reaches the traversal when it requests that invocation, and `close()` aborts queued and running reads and waits for every started child. The gate takes 49.9/53.9 → 6.9/7.2 s.
|
||||
- **Doc graphs:** `withMemoizedFileSystemProbes()` in [`scripts/gen-doc-graphs.ts`](../../../../scripts/gen-doc-graphs.ts) memoizes `ts.sys` `fileExists`, `directoryExists`, `realpath`, `readDirectory`, and `getDirectories` while `renderDocs` runs, so every compiler host and config parse in one run shares one view of the file system. The workspace must not change during the run, and the original methods are restored before pages are written. The gate takes 48.7/44.7 → 32.7/34.2 s.
|
||||
- **Why it is lossless:** Both generators produce byte-identical pages.
|
||||
- **Trade-off:** The concurrent help processes use about 3.6 GB of memory combined at peak.
|
||||
|
||||
### 8. `type-model.spec.ts` runs against an in-memory workspace
|
||||
|
||||
- **Change:** [`type-model.spec.ts`](../../../../packages/typert/generator/tests/type-model.spec.ts) mocks `node:fs` with `memoryFileSystem()` and the `ts.sys` host from [`type-model-workspace.ts`](../../../../packages/typert/generator/tests/type-model-workspace.ts), so fixture copies, case edits, write-mode annotations, tsconfig expansion, and module resolution under the in-memory root never touch the disk. Each workspace mounts minimal `zod` and `@types/node` declaration stubs; only the Emitter case keeps real zod. The fixture's `tsconfig.base.json` sets `lib: ["ES2024"]`, which excludes the DOM library, and cases that read the unmodified fixture share one deep-frozen analysis. The generator source has no test-specific path.
|
||||
- **Performance:** The file alone takes 316.3/329.1 → 14.65/15.04 s.
|
||||
- **Why the behavior coverage stays:** The 61 case names and their assertions are the same. The snapshot differs in one line, the external zod symbol path, which names the stub (`<external>:node_modules/zod/index.d.ts#ZodType`) rather than a pnpm store version.
|
||||
- **Trade-off:** Outside the Emitter case, the spec does not exercise the analyzer against the real file system, the DOM library, or the real zod and `@types/node` declarations. The memory file system implements only the `node:fs` calls the analyzer makes (`existsSync`, `readFileSync`, `realpathSync`, and `writeFileSync`); another call on an in-memory path reaches the real file system, where the path does not exist.
|
||||
|
||||
### Repository-wide comparison
|
||||
|
||||
Both trees were measured in the same session, alternating baseline and implementation runs of each command after `pnpm run build:artifacts` (`pnpm run test` fails the same environment-dependent files on both trees, 42 on the baseline and a subset of 28 here):
|
||||
|
||||
| Command | Baseline | This implementation | Change | Items |
|
||||
|---|---|---|---|---|
|
||||
| `pre-commit` hooks, one staged file under a package `src/` subdirectory | 16.5/17.8 s | 4.3/4.5 s | −75% | 2 |
|
||||
| `pnpm run typecheck` (pre-push), nothing to recheck | 29.7/23.3 s | 14.2/16.1 s | −43% | 1, 3 |
|
||||
| `pnpm run lint` | 110.0 s | 106.7 s | −3% | 1 |
|
||||
| `pnpm run doc-typecheck` | 45.2 s | 36.1 s | −20% | 1 |
|
||||
| `pnpm run test:docs` | 102.4/104.0 s | 26.8/29.3 s | −73% | 3–6 |
|
||||
| `pnpm run doc-sync` | 186.8 s | 70.3 s | −62% | 3–7 |
|
||||
| `pnpm run test` | 222.7 s (255 workers) | 148.1 s (48 workers) | −34% | 3, 8 |
|
||||
|
||||
`lint` time is Oxlint itself; its prerequisite falls as item 1 states. CPU time falls with wall: `typecheck` user time 39.5 → 14.2 s, `test:docs` 212.6 → 126.3 s, `doc-sync` 628.2 → 503.2 s, and `test` 3,417 → 3,014 s.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
- **Cap every local default at 12, including Vitest forks (item 3):** `pnpm run test` takes 410.5 s at 12 workers against 243.7 s at 255, 68% longer.
|
||||
- **Use half of the cgroup CPU quota as the local budget (item 3):** Reading the quota is a larger change and helps only inside containers.
|
||||
- **Parse documents in parallel `worker_threads` (item 5):** It adds more complexity than parsing each document once, which removes the repeated work itself.
|
||||
- **Regenerate third-party notices in the pre-commit hook (item 2):** Browser inputs make every package source file a generator input, so the hook runs the 16-second generator on nearly every commit while few commits change its output; [the generated third-party notices note](2026-07-30-generated-third-party-notices.md) owns the decision.
|
||||
|
||||
## Consequences
|
||||
|
||||
The pre-push `typecheck` with nothing to recheck takes about 14–22 s, the `lint` and `doc-typecheck` prerequisite about 12 s, and the hooks of a one-file commit about 4 s. `pnpm run test` starts at most 48 forks locally, and `test:docs` and `doc-sync` run up to 8 gates at once. Typert outputs, generated pages, file lists, and reference classifications are byte-identical to the baseline.
|
||||
|
||||
Costs: type-check commands do not refresh runnable bundles; a stale notices file fails in the test lane after push; local defaults ignore CPUs above the caps and still oversubscribe a container whose quota is below them; 8 concurrent documentation gates raise the `doc-sync` RSS peak; CLI help uses about 3.6 GB at peak; doc graphs and markdown links assume the tree does not change during a run; `type-model.spec.ts` exercises stubs rather than real zod, `@types/node`, the DOM library, and disk I/O outside the Emitter case; and incremental `typecheck` inherits the Remote-declaration gap that Issue #6022 tracks.
|
||||
|
||||
Verification: `generate-typert.spec.ts` pins byte-identical files against the workspace-mode plugin, rejection of a Host program that loads a compiled workspace declaration with the missing alias named, and each failure stage; `compile-referenced-projects.spec.ts` pins the `host-contracts` steps and their failure reporting; `build.spec.ts` pins the typecheck scope's steps; `parallelism.spec.ts`, `run-gates.spec.ts`, and `run-oxlint.spec.ts` pin the CI and local budgets and the overrides; `repo-files.spec.ts`, `markdown.spec.ts`, `translation-pairing.spec.ts`, `type-equiv-prefilter.spec.ts`, `verify-md-links.spec.ts`, `gen-cli-help.spec.ts`, and `gen-doc-graphs.spec.ts` pin single directory listing, parse reuse, the pairing guard, the prefilter, the prefetch order and failure path, and probe memoization with restoration.
|
||||
@@ -0,0 +1,158 @@
|
||||
# Agent Note: 本地门禁、钩子与测试提速
|
||||
|
||||
Status: implemented
|
||||
|
||||
[English](2026-10-12-local-gate-speedups.md) | 中文
|
||||
|
||||
## 问题
|
||||
|
||||
在基线（合并 #6013 时的 master）上，本地运行 Git 钩子、`pnpm run test`、`test:docs` 与 `doc-sync`（文档同步门禁）时，大部分 wall 时间花在结果并不依赖的工作上：
|
||||
|
||||
- **经打包器获得 Typert 声明。** pre-push 钩子运行的 `typecheck` 先运行 `build:lib:host`（Host `tsc -b tsconfig.host.json`、Host tsdown 阶段与 Desktop bundle），再运行 `tsc -b tsconfig.client.json`。`lint`、`lint:fix`、`doc-typecheck` 与 CI `typert-contracts` 门禁的前置步骤先运行 Host 包编译，再运行 Host tsdown 阶段（`compile-referenced-projects.ts host-libraries`）。类型检查只读取包声明与 Typert 声明，但无需重查时 `typecheck` 仍需 34.1/35.0 s，前置步骤需 52.2/29.6 s。
|
||||
- **每次提交都生成第三方声明。** 只要暂存文件是生成器的输入，pre-commit 钩子就重新生成 `THIRD_PARTY_NOTICES.md`。为找出浏览器侧的第三方 import，生成器对 75 个 Client 包试打包并运行一次 Vite 构建，因此每个包源码文件都是它的输入；暂存一个位于包 `src/` 子目录下的文件的提交，钩子合计 16.5/17.8 s，其中第三方声明任务占 12.4/13.4 s。
|
||||
- **以宿主 CPU 数作为本地 worker 数。** `os.availableParallelism()` 报告的是宿主 CPU 数（测量主机上为 256），而不是容器的 CPU 配额（64）。按此默认值，`pnpm run test` 启动 255 个 Vitest fork，RSS 峰值 55.3 GiB，并偶发地多失败 12 个文件（14 次超时）；同时 `doc-sync`、`test:docs`、`hygiene` 与 `check:all` 最多只并发 4 个门禁。
|
||||
- **每个候选一次 Git 前缀查找。** `verify-repository-references` 通过 `git cat-file --batch-check` 对约 26,000 个缩写十六进制候选逐个做 Git 前缀查找；这些查找占该门禁 166.8 s 中的 135.7 s。
|
||||
- **重复解析 Markdown。** markdown links 对每个锚点目标再多解析两次，且探测目标是否存在时不缓存；translation pairing 做四次 glob，并在链接归一化后再次解析每个文档；包 README 的 model experience 与 limitations 检查各自把每个 README 解析三次；type equivalence 解析全部 1,835 个文档；Session format references 把每个配对解析四次。
|
||||
- **重复读取目录。** markdown wrap、doc refs 与 package paths 背后的遍历器 `uniqueRepoFiles` 对每个 pattern 与每层 `**` 各读取一次每个目录，并对每个匹配文件调用 `realpath`。
|
||||
- **串行启动 help 与重复的编译器探测。** `gen-cli-help` 依次启动 12 个顶层源码模式的 `dsh … --help` 进程，每个进程大部分时间在等待 profile 启动的 I/O（49.9/53.9 s）；`gen-doc-graphs` 构建多个编译器 host 与 program，重复执行同样的 `ts.sys` 文件系统探测（48.7/44.7 s）。
|
||||
- **受文件系统探测拖累的测试文件。** Typert generator 的 `type-model.spec.ts` 的 61 个用例各自把 fixture（测试前置数据）复制到磁盘并运行一次完整分析，分析中的四个 TypeScript program 各自加载 DOM lib、`@types/node` 与 zod v4，每次分析约 22,000 次 stat 调用。该文件耗时 316.3/329.1 s，是 `pnpm run test` 的长尾。
|
||||
|
||||
## 决策
|
||||
|
||||
以下八项决策缩短上述运行。每一项都给出其产物与诊断保持不变的依据，或它接受的有意差异。
|
||||
|
||||
| # | 决策 | 受影响的命令 | 实测 wall | 产物与诊断 | 有意的行为差异 |
|
||||
|---|---|---|---|---|---|
|
||||
| 1 | 类型检查不打包地生成 Typert 声明 | `typecheck`、`lint`、`lint:fix`、`doc-typecheck`、CI `typert-contracts` | `typecheck` 34.1/35.0 → 21.9/20.0 s；前置步骤 52.2/29.6 → 12.5/11.3 s | 140 个 Typert 文件逐字节一致；检查的程序相同 | 不产出可运行 bundle，不写 client build record |
|
||||
| 2 | pre-commit 钩子不重新生成第三方声明 | `pre-commit` | 16.5/17.8 → 4.3/4.5 s | 新鲜度由测试 lane 断言 | 第三方声明过期时在测试 lane 失败，而不是在提交时 |
|
||||
| 3 | 本地默认 worker 预算 | run-gates 聚合、`build`、`typecheck`、`pnpm run test`、Oxlint、publint | 255 个 worker 时 `test` 为 243.7 s、55.3 GiB；`test:docs` 74.7 → 50.9 s | 检查不变 | 本地默认值不使用超过 12 或 48 的 CPU |
|
||||
| 4 | 仓库引用候选对照一次对象列表解析 | `verify-repository-references` | 166.8 → 15.7 s | 27,050 个候选的判定 0 差异 | 无 |
|
||||
| 5 | 文档门禁每个文档只解析一次 | markdown links、translation pairing、README model experience 与 limitations、type equivalence、Session format references | markdown links 53.4/51.6 → 25.9/25.1 s；type equivalence 23.0/22.5 → 6.0/5.5 s | 门禁输出逐字节一致 | 无 |
|
||||
| 6 | 仓库文件遍历每个目录只列一次 | markdown wrap、doc refs、package paths | doc refs 8.8/7.0 → 3.2/3.4 s | 文件列表一致 | doc refs 不读取嵌套的 `lib` 目录 |
|
||||
| 7 | CLI（命令行界面）help 读取并发运行；doc graphs 的探测做 memo | `verify-cli-help`、`verify-doc-graphs` | 49.9/53.9 → 6.9/7.2 s；48.7/44.7 → 32.7/34.2 s | 页面逐字节一致 | CLI help 峰值约用 3.6 GB 内存 |
|
||||
| 8 | `type-model.spec.ts` 在内存工作区上运行 | `pnpm run test` | 316.3/329.1 → 14.65/15.04 s | 61 个用例与断言不变；快照有一行不同 | 除 Emitter 用例外，桩替代 zod、`@types/node` 与 DOM lib |
|
||||
|
||||
### 测量条件
|
||||
|
||||
所有数字都是 wall 时间，取自同一个共享 Linux 容器：CPU 配额 64 核，宿主报告 256 个 CPU，一分钟 load average 为 20–150，因此样本带有噪声。工作树位于网络文件系统上，每次文件系统探测的代价都高于本地磁盘。`a/b` 表示两个样本，`x → y` 比较基线与本实现。
|
||||
|
||||
### 1. 类型检查不打包地生成 Typert 声明
|
||||
|
||||
- **目的：** 让 `typecheck` 以及 `lint`、`lint:fix`、`doc-typecheck` 与 CI `typert-contracts` 门禁的前置步骤得到它们读取的声明，即 `tsc` 产出的包声明与 Typert 声明，而不运行 Host tsdown 阶段或 Desktop bundle。
|
||||
- **改动：** `typecheck` 是 `tsx scripts/build.ts --scope typecheck`，它通过 run-gates 调度器运行下表中的步骤（[`scripts/build.ts`](../../../../scripts/build.ts)）。该 scope 不构建 native addon、tsdown bundle、Desktop bundle 或 Web 产物，不写 client build record，可运行的 `lib/` 产物保持上一次 `build:artifacts` 的状态；启动或测试构建产物之前先运行 `build:artifacts`。`lint`、`lint:fix`、`doc-typecheck` 与 `typert-contracts` 的前置步骤是 `compile-referenced-projects.ts host-contracts`，它让 host-src 与 host-typert 并行（[`compile-referenced-projects.ts`](../../../../scripts/compile-referenced-projects.ts)）。`build`、`build:artifacts`、benchmark 构建与 `release-vendor.yml` 的 PR（Pull Request）演练在 Host tsdown 阶段内运行 Typert。
|
||||
|
||||
| 步骤 | 命令 | `needs` |
|
||||
|---|---|---|
|
||||
| host-src | `node --max-old-space-size=4096 <tsc> -b <references of tsconfig.host.json>` | — |
|
||||
| host-typert | `node --import tsx/esm scripts/generate-typert.ts` | — |
|
||||
| client-src | `<tsc> -b <references of tsconfig.client.json>` | host-src、host-typert |
|
||||
| host-check | `node --max-old-space-size=4096 <tsc> -b tsconfig.host.json` | host-src |
|
||||
| client-check | `<tsc> -b tsconfig.client.json` | client-src |
|
||||
|
||||
- **共用生成：** [`scripts/generate-typert.ts`](../../../../scripts/generate-typert.ts) 对 `WORKSPACE_TYPERT_FACES`（[`scripts/typert-faces.ts`](../../../../scripts/typert-faces.ts)，即 Host tsdown 插件同样生成的 face）调用 `WorkspaceTypertGenerator.generateContributors()`，并用 [`artifact-files.ts`](../../../../packages/typert/generator/src/artifact-files.ts) 中插件同样使用的 `writeWorkspaceArtifacts()` 写出结果。与插件一样，它跳过 TypeScript 诊断，这些诊断由 Host 包编译报告；Typert 分析错误使 host-typert 失败，正如它在 `build` 中使 Host bundle 失败。host-typert 由 Node 加 tsx 的 ESM 钩子运行，而不经 `pnpm exec`，因为后者的启动开销会拉长通往 Client 编译的路径。
|
||||
- **以 build mode 运行聚合检查：** host-check 与 client-check 在本侧 face 的包编译之后才开始，因此 build mode 发现所有 references 都是最新的，只检查聚合根程序，并在根程序的 tsbuildinfo 为最新时跳过它。
|
||||
- **只读源码的分析：** host-typert 能与 host-src 并行，是因为它的 Host program 不读取任何工作区构建产物。`tsconfig.base.json` 把 `@deepseek-ai/dsh-client-file-upload/types` 映射到源码；没有这条别名时，该说明符经包 `exports` 解析到 `lib/types/types.d.ts`。有了它，Host program 加载 1,289 个文件，其中没有一个位于工作区包的 `lib/` 下。`generate-typert.ts` 开启 `rejectBuildOutputs`：face program 加载工作区包 `lib/` 目录下的文件时，[`build-output-guard.ts`](../../../../packages/typert/generator/src/build-output-guard.ts) 以 `TypertAnalysisError` 使分析失败，并指出解析到那里的每个 import 说明符，以及把它映射到源码的 `paths` 条目。tsdown 插件不开启该选项，因为在 `build` 中 Typert 总在 Host 包编译之后运行。[Typert generator README](../../../../packages/typert/generator/README.zh.md) 记录了这些写入函数与该选项。
|
||||
- **无损依据：** 没有任何工作区 tsconfig 设置 `allowJs`，每个 `exports` 的 `types` 条件都指向 `lib/types/` 内或 `lib/typert.*`，每个 tsdown 配置都设置 `dts: false`，因此没有类型检查读取 tsdown 产物。140 个 Typert 文件（28 个包 × 5 个输出）在以下场景中逐字节一致：基线 tsdown 阶段、当前 tsdown 阶段、删除全部 Typert 输出后运行 `generate-typert.ts`、隐藏全部 344 个 `lib/types` 目录后运行 `generate-typert.ts`；临时给 `packages/goal/goal` 加一个 Remote 方法时，tsdown 阶段与 `generate-typert.ts` 写出的文件也相同。`typecheck` 检查两个 face 的全部包 project 与两个聚合根程序，即 `build` 检查的程序。
|
||||
- **性能：** 无需重查时 `typecheck` 为 34.1/35.0 → 21.9/20.0 s。两个聚合根程序都需要重查时为 83.5 → 76.0/72.1 s，其关键路径是约 66–70 s 的 Host 聚合检查。`lint` 与 `doc-typecheck` 的前置步骤为 52.2/29.6 → 12.5/11.3 s。
|
||||
- **取舍：** `typecheck`、`lint`、`lint:fix`、`doc-typecheck` 与 `typert-contracts` 写出 Typert 声明但不写 bundle，因此在下一次 `build:artifacts` 之前，可运行的 `lib/` JavaScript 可能落后于这些声明。
|
||||
- **与构建的关系：** [无损构建提速 Note](2026-10-11-lossless-build-speedups.zh.md) 否决了在构建中把 Typert 移出 Host tsdown 阶段：构建中它的 Worker Thread 已与打包重叠，独立步骤最多只能省约 1 s。类型检查命令不运行打包器，因此不打包地生成会把整个 Host tsdown 阶段与 Desktop bundle 移出它们的关键路径。[构建拆分 Note](2026-10-11-build-artifacts-check-split.zh.md) 列出每个调用方及其命令。
|
||||
- **已知缺口：** 增量 `tsc -b` 不把 Client 源码经包 `exports` 访问到的 `lib/typert.remote-client.d.ts` 记为消费方 project 的输入；Remote 签名改变而 Client 源码未变时，增量检查不会重查消费方。这一缺口与由哪一种生成写出声明无关，由 Issue #6022 跟踪。
|
||||
|
||||
### 2. pre-commit 钩子不重新生成第三方声明
|
||||
|
||||
- **改动：** [`lefthook.yml`](../../../../lefthook.yml) 中的 `pre-commit` 钩子没有第三方声明任务。[`gen-third-party-notices.spec.ts`](../../../../scripts/gen-third-party-notices.spec.ts) 在测试 lane（包括必过的 `node 24 / coverage` lane）中断言 `THIRD_PARTY_NOTICES.md` 已提交的字节，过期时指出 `pnpm run gen-third-party-notices`。[生成的第三方声明 Note](2026-07-30-generated-third-party-notices.zh.md) 拥有这一新鲜度决策。
|
||||
- **性能：** 暂存一个位于包 `src/` 子目录下的文件的提交，钩子耗时由 16.5/17.8 s 降为 4.3/4.5 s，其中第三方声明任务占 12.4/13.4 s。
|
||||
- **取舍：** 使第三方声明过期的提交会在推送后于测试 lane 失败，而不是在提交时被修正。
|
||||
|
||||
### 3. 本地默认 worker 预算
|
||||
|
||||
- **目的：** 按共享工作站或容器确定默认并行度；在这类环境中 Node 报告的是宿主 CPU 数而不是 CPU 配额。
|
||||
- **改动：** [`scripts/parallelism.ts`](../../../../scripts/parallelism.ts) 拥有默认值。`CI=true` 时它是 Node 报告的 CPU 数。否则，run-gates 调度器、`build`、`typecheck`、Oxlint 与 publint 的上限为 12（`LOCAL_PARALLELISM_CAP`），Vitest fork 与本地覆盖率预算的上限为 48（`LOCAL_TEST_PARALLELISM_CAP`）。`doc-sync`、`test:docs`、`hygiene` 与 `check:all` 的每个门禁都构建完整的 `ts.Program`，它们最多并发 8 个门禁。显式的 `DSH_GATE_CONCURRENCY`、`DSH_OXLINT_THREADS`、`DSH_COVERAGE_MAX_WORKERS` 与 Vitest `--maxWorkers` 优先；CI 中，除非 run-gates 的 CI 环境设置了这些变量，Vitest 与 Oxlint 使用各自的默认值。[开发指南](../../../../docs/development.zh.md)向贡献者说明这些默认值。
|
||||
|
||||
四种 fork 数下的 `pnpm run test`，运行的是第 8 项所替换的落盘版 `type-model.spec.ts`，32 个 worker 那次的 wall 由它决定：
|
||||
|
||||
| Vitest worker 数 | wall | RSS 峰值 | 额外的偶发失败 | 除最慢文件外其余文件完成于 |
|
||||
|---|---|---|---|---|
|
||||
| 255（报告的 CPU 数） | 243.7 s | 55.3 GiB | 12 个文件、14 次超时 | 约 135–170 s |
|
||||
| 32 | 310.4 s | 15.7 GiB | 4 个文件、4 次超时 | 193 s |
|
||||
| 24 | 337.2 s | 13.2 GiB | 5 个文件、5 次超时 | 233 s |
|
||||
| 12 | 410.5 s | 8.4 GiB | 2 个文件、3 次超时 | 406 s |
|
||||
|
||||
- **Oxlint：** Oxlint 自身的默认值是 64 线程（读取 cgroup 配额）；12 线程时其 wall 保持约 87–90 s，CPU 少约 15%；8 线程时慢约 4 s。
|
||||
- **文档聚合：** 并发 8 个门禁时 `test:docs` 为 50.9 s，并发 4 个时为 74.7 s。不含第 4 项时测得的 `doc-sync` 在并发 4、8、12 个门禁时分别为 341.8、295.0、253.2 s，RSS 峰值分别为 10.0、13.5、16.3 GiB。
|
||||
- **取舍：** CPU 配额超过上限的机器，默认并行度也不超过 12 或 48，需要使用覆盖变量；配额低于上限的容器仍会超额订阅，因为 Node 报告的是宿主 CPU 数。并发 8 个门禁时 `doc-sync` 的 RSS 峰值为 13.5 GiB，并发 4 个时为 10.0 GiB；并发 12 个门禁快 41.8 s，但峰值为 16.3 GiB。
|
||||
|
||||
### 4. 仓库引用候选对照一次对象列表解析
|
||||
|
||||
- **改动：** [`verify-repository-references.ts`](../../../../scripts/verify-repository-references.ts) 用 `git cat-file --batch-all-objects --batch-check=%(objectname)` 一次列出全部对象名（约 120 万个对象，0.6 s），排序后在内存中对每个小写候选前缀做二分查找；随后一次 `--batch-check=%(objectname) %(objecttype)` 调用读取命中对象的类型。只有当候选前缀恰好匹配一个可用对象且该对象是 commit 时，它才算 commit 引用，这与 Git 自身前缀查找的规则相同；与 Git 的查找一样，该列表也包含不可达对象与 alternate 对象。
|
||||
- **性能：** 166.8 → 15.7 s。
|
||||
- **无损依据：** 在 27,050 个真实与合成候选上（包括歧义前缀、blob 与 tree 前缀以及随机字符串），Git 的逐候选前缀查找与内存查找的判定完全一致。
|
||||
- **取舍：** 120 万个对象的列表约为 49 MiB 的 Git 输出，读取时的输出上限为 1 GiB。
|
||||
|
||||
### 5. 文档门禁每个文档只解析一次
|
||||
|
||||
- **改动：** 下表中的每个门禁对一个 Markdown 文档只解析一次，并把语法树传给本次运行中的所有使用方。
|
||||
|
||||
| 门禁 | 单次解析机制 | wall |
|
||||
|---|---|---|
|
||||
| markdown links（[`verify-md-links.ts`](../../../../scripts/verify-md-links.ts)） | 一个按运行划分的缓存对每个文件只解析一次，同时取得其 destination 与锚点，并对每个解析后的目标只探测一次 | 53.4/51.6 → 25.9/25.1 s |
|
||||
| translation pairing（[`verify-translation-pairing.ts`](../../../../scripts/verify-translation-pairing.ts)） | 一次 glob 覆盖所有范围 pattern；record 在下述守卫条件下复用原文语法树 | 51.0/47.1 → 30.3/29.3 s |
|
||||
| package README model experience、package README limitations | 每项检查对一个 README 只解析一次，并把语法树传给 `markdownProseLines()` 与 `markdownHeadingLines()` | 各约 12 → 5.9 s |
|
||||
| type equivalence（[`verify-type-equiv.ts`](../../../../scripts/verify-type-equiv.ts)） | 文本预筛跳过不可能含源码等价围栏的文档；manifest（元数据清单）中的每个 TypeScript 源文件只解析一次 | 23.0/22.5 → 6.0/5.5 s |
|
||||
| Session format references | persistence 渲染器把它解析出的语法树传给配对 record | 10.2 → 5.3 s |
|
||||
|
||||
- **配对复用守卫：** [`translation-pairing-record.ts`](../../../../scripts/translation-pairing-record.ts) 中的 `normalizedTopLevelBlocks()` 仅在以下条件同时成立时，才为归一化后的文本复用原文语法树：每处链接归一化编辑都替换一个完整、无 title 且不在行首开始的 destination，原文等于该节点的 `url`，且原文与替换文本都只含 `[A-Za-z0-9._/:#%-]`。这些字符既不能结束 destination，也不能开始或结束其他语法结构，因此归一化后的文本解析出相同的节点，只是偏移量随之移动。其他任何编辑都回退为对归一化文本的完整解析。
|
||||
- **type equivalence 预筛：** [`type-equiv-prefilter.ts`](../../../../scripts/type-equiv-prefilter.ts) 在文档含 `type-equiv` 或 `public-api`，或者某个含 `` ``` `` 或 `~~~` 的行同时含 `&` 或 `\` 时，保留该文档进行解析。解析只会通过解码字符引用与反斜杠转义改变信息字符串，因此 `ts type-equiv` 与 `ts type\-equiv` 所在的文档都会被保留；[`type-equiv-prefilter.spec.ts`](../../../../scripts/type-equiv-prefilter.spec.ts) 覆盖这两种写法。
|
||||
- **无损依据：** 每个门禁在仓库上的输出逐字节一致，人为注入的错误也被同样报告。对配对而言，语料中全部 1,886 个文档的顶层块与全部 943 个配对的 record 都与完整重新解析的结果一致，随机 destination 的结果也一致。
|
||||
|
||||
### 6. 仓库文件遍历每个目录只列一次
|
||||
|
||||
- **改动：** [`scripts/repo-files.ts`](../../../../scripts/repo-files.ts) 中的 `uniqueRepoFiles()` 在一次调用的所有 pattern 之间共享一个目录列表缓存，使每个目录最多读取一次，并且只对经符号链接到达的匹配调用 `realpath`；只经真实目录到达的普通文件，规范路径为仓库 realpath 拼接其相对路径。它的可选谓词 `skipDirectory` 让遍历器不进入调用方排除其下全部路径的目录；[`verify-doc-refs.ts`](../../../../scripts/verify-doc-refs.ts) 跳过第一段路径之后的每个 `lib` 目录，其下文件本来就全部被它排除。
|
||||
- **性能：** markdown wrap 25.8/25.0 → 21.3/21.2 s；doc refs 8.8/7.0 → 3.2/3.4 s；package paths 9.9/8.1 → 7.2/5.5 s。
|
||||
- **无损依据：** 三个门禁得到的文件列表逐项相同，损坏或循环的符号链接仍会经 `realpath` 失败。
|
||||
- **行为差异：** doc refs 不列出被跳过的 `lib` 目录，因此不可读的此类目录不会以 EACCES 使门禁失败。
|
||||
|
||||
### 7. CLI help 读取并发运行；doc graphs 的探测做 memo
|
||||
|
||||
- **CLI help：** [`scripts/gen-cli-help.ts`](../../../../scripts/gen-cli-help.ts) 中的 `prefetchHelp()` 在默认 worker 预算内启动 12 个顶层 `dsh … --help` 读取（启动器、`plugin`，以及每个 profile 的简写与 `--profile` 形式），每个读取使用独立的临时 `DSH_HOME` 与 `DSH_AGENTS_HOME`，并由按原顺序进行的遍历消费；嵌套读取共享同一上限。失败的读取在遍历请求该调用时才交给遍历；`close()` 中止排队与运行中的读取，并等待每个已启动的子进程结束。该门禁为 49.9/53.9 → 6.9/7.2 s。
|
||||
- **doc graphs：** [`scripts/gen-doc-graphs.ts`](../../../../scripts/gen-doc-graphs.ts) 中的 `withMemoizedFileSystemProbes()` 在 `renderDocs` 运行期间对 `ts.sys` 的 `fileExists`、`directoryExists`、`realpath`、`readDirectory` 与 `getDirectories` 做 memo，使一次运行中的所有编译器 host 与配置解析共享同一个文件系统视图。运行期间工作区不得改变，写出页面之前恢复原方法。该门禁为 48.7/44.7 → 32.7/34.2 s。
|
||||
- **无损依据：** 两个生成器生成的页面逐字节一致。
|
||||
- **取舍：** 并发的 help 进程合计峰值约占 3.6 GB 内存。
|
||||
|
||||
### 8. `type-model.spec.ts` 在内存工作区上运行
|
||||
|
||||
- **改动：** [`type-model.spec.ts`](../../../../packages/typert/generator/tests/type-model.spec.ts) 用 [`type-model-workspace.ts`](../../../../packages/typert/generator/tests/type-model-workspace.ts) 中的 `memoryFileSystem()` mock `node:fs` 与 `ts.sys` host，使内存根目录下的 fixture 复制、用例编辑、写模式注解、tsconfig 展开与模块解析都不触及磁盘。每个工作区挂载最小的 `zod` 与 `@types/node` 声明桩；只有 Emitter 用例保留真实 zod。fixture 的 `tsconfig.base.json` 设置 `lib: ["ES2024"]`，不含 DOM lib；读取未修改 fixture 的用例共享一份深度冻结的分析结果。generator 源码没有任何测试专用路径。
|
||||
- **性能：** 单独运行该文件为 316.3/329.1 → 14.65/15.04 s。
|
||||
- **行为覆盖保持不变的依据：** 61 个用例名及其断言相同。快照只有一行不同，即外部 zod 符号路径，它指向桩（`<external>:node_modules/zod/index.d.ts#ZodType`），而不是某个 pnpm store 版本。
|
||||
- **取舍：** 除 Emitter 用例外，该 spec 不针对真实文件系统、DOM lib 以及真实 zod 与 `@types/node` 声明检验分析器。内存文件系统只实现分析器使用的 `node:fs` 调用（`existsSync`、`readFileSync`、`realpathSync` 与 `writeFileSync`）；对内存路径的其他调用会落到真实文件系统，而该路径并不存在。
|
||||
|
||||
### 全仓对比
|
||||
|
||||
两棵树在同一时段测量，每个命令在 `pnpm run build:artifacts` 之后交替运行基线与本实现（`pnpm run test` 在两棵树上失败的都是依赖环境的文件，基线 42 个，本实现是其中的 28 个）：
|
||||
|
||||
| 命令 | 基线 | 本实现 | 变化 | 对应项 |
|
||||
|---|---|---|---|---|
|
||||
| `pre-commit` 钩子，暂存一个位于包 `src/` 子目录下的文件 | 16.5/17.8 s | 4.3/4.5 s | −75% | 2 |
|
||||
| `pnpm run typecheck`（pre-push），无需重查 | 29.7/23.3 s | 14.2/16.1 s | −43% | 1、3 |
|
||||
| `pnpm run lint` | 110.0 s | 106.7 s | −3% | 1 |
|
||||
| `pnpm run doc-typecheck` | 45.2 s | 36.1 s | −20% | 1 |
|
||||
| `pnpm run test:docs` | 102.4/104.0 s | 26.8/29.3 s | −73% | 3–6 |
|
||||
| `pnpm run doc-sync` | 186.8 s | 70.3 s | −62% | 3–7 |
|
||||
| `pnpm run test` | 222.7 s（255 个 worker） | 148.1 s（48 个 worker） | −34% | 3、8 |
|
||||
|
||||
`lint` 的耗时主要是 Oxlint 本身，其前置步骤的下降见第 1 项。CPU 时间随 wall 一同下降：`typecheck` 的 user 时间 39.5 → 14.2 s，`test:docs` 212.6 → 126.3 s，`doc-sync` 628.2 → 503.2 s，`test` 3,417 → 3,014 s。
|
||||
|
||||
## 考虑过的替代方案
|
||||
|
||||
- **所有本地默认值一律以 12 为上限，包括 Vitest fork（第 3 项）：** 12 个 worker 时 `pnpm run test` 为 410.5 s，255 个时为 243.7 s，慢 68%。
|
||||
- **以 cgroup CPU 配额的一半作为本地预算（第 3 项）：** 读取配额的改动更大，且只在容器内有意义。
|
||||
- **用 `worker_threads` 并行解析文档（第 5 项）：** 它带来的复杂度高于每个文档只解析一次，而后者直接消除了重复工作。
|
||||
- **在 pre-commit 钩子中重新生成第三方声明（第 2 项）：** 浏览器输入使每个包源码文件都成为生成器输入，钩子会让几乎每次提交都运行约 16 秒的生成器，而改变输出的提交很少；[生成的第三方声明 Note](2026-07-30-generated-third-party-notices.zh.md) 拥有该决策。
|
||||
|
||||
## 后果
|
||||
|
||||
无需重查时 pre-push `typecheck` 约 14–22 s，`lint` 与 `doc-typecheck` 的前置步骤约 12 s，改动 1 个文件的提交的钩子约 4 s。`pnpm run test` 在本地最多启动 48 个 fork，`test:docs` 与 `doc-sync` 最多同时运行 8 个门禁。Typert 输出、生成的页面、文件列表与引用判定都与基线逐字节一致。
|
||||
|
||||
代价：类型检查命令不刷新可运行的 bundle；第三方声明过期时在推送后于测试 lane 失败；本地默认值不使用超过上限的 CPU，而配额低于上限的容器仍会超额订阅；并发 8 个文档门禁抬高 `doc-sync` 的 RSS 峰值；CLI help 峰值约用 3.6 GB 内存；doc graphs 与 markdown links 假定运行期间工作树不变；除 Emitter 用例外，`type-model.spec.ts` 检验的是桩，而不是真实 zod、`@types/node`、DOM lib 与磁盘 I/O；增量 `typecheck` 继承 Issue #6022 跟踪的 Remote 声明缺口。
|
||||
|
||||
验证：`generate-typert.spec.ts` 固定与 workspace 模式插件逐字节一致的文件、对加载已编译工作区声明的 Host program 的拒绝（并指出缺失的别名），以及每个失败阶段；`compile-referenced-projects.spec.ts` 固定 `host-contracts` 的步骤及其失败报告；`build.spec.ts` 固定 typecheck scope 的步骤；`parallelism.spec.ts`、`run-gates.spec.ts` 与 `run-oxlint.spec.ts` 固定 CI 与本地预算以及覆盖变量；`repo-files.spec.ts`、`markdown.spec.ts`、`translation-pairing.spec.ts`、`type-equiv-prefilter.spec.ts`、`verify-md-links.spec.ts`、`gen-cli-help.spec.ts` 与 `gen-doc-graphs.spec.ts` 固定目录只列一次、语法树复用、配对守卫、预筛、预取顺序与失败路径，以及探测 memo 及其恢复。
|
||||
Reference in new issue
Block a user