diff --git a/.agents/notes/implemented/process/2026-08-10-npm-release-sequences.i18n.yaml b/.agents/notes/implemented/process/2026-08-10-npm-release-sequences.i18n.yaml index f40cecc02e..1158ff8785 100644 --- a/.agents/notes/implemented/process/2026-08-10-npm-release-sequences.i18n.yaml +++ b/.agents/notes/implemented/process/2026-08-10-npm-release-sequences.i18n.yaml @@ -17,12 +17,12 @@ /agent-note-private-npm-publication-as-three-independent-sequences/decision/versions-land-in-the-repository-from-a-local-command-ci-only-checks-and-uploads: en: 71d4cda384013bfb zh: 909d853c18496925 -/agent-note-private-npm-publication-as-three-independent-sequences/decision/vendor-publish-what-changed-and-let-tags-be-the-ledger: - en: 7f578bb18f7f8693 - zh: 49f25210f7c1a61f +/agent-note-private-npm-publication-as-three-independent-sequences/decision/vendor-advance-the-complete-family-with-per-package-tags: + en: 321f48c7e4afac24 + zh: 974a7fa1a75c810d /agent-note-private-npm-publication-as-three-independent-sequences/decision/publication-runs-only-on-github-and-the-registry-decides-what-goes-out: - en: 4f399390f58d7a34 - zh: 9df67248acfc61c8 + en: 8c8cd90e6eabe07d + zh: 7035b71a6eeeb379 /agent-note-private-npm-publication-as-three-independent-sequences/decision/workspace-internal-references-use-the-workspace-protocol: en: 82a7925d449fc283 zh: 653358a1ffbad80a @@ -42,8 +42,8 @@ en: c505237cad2c4128 zh: 55364ba462536680 /agent-note-private-npm-publication-as-three-independent-sequences/alternatives-considered: - en: 11877d09e5a84f6e - zh: ea372c779254be93 + en: ed22fa8a47b5c4b6 + zh: 7b367971a51a67ff /agent-note-private-npm-publication-as-three-independent-sequences/consequences: - en: f961ebb927e30b5c - zh: 019f1628fb1f2c56 + en: 46f82110175ea3f7 + zh: 1cdd1c99bfb88440 diff --git a/.agents/notes/implemented/process/2026-08-10-npm-release-sequences.md b/.agents/notes/implemented/process/2026-08-10-npm-release-sequences.md index 5de43e3720..711b01154c 100644 --- a/.agents/notes/implemented/process/2026-08-10-npm-release-sequences.md +++ b/.agents/notes/implemented/process/2026-08-10-npm-release-sequences.md @@ -36,9 +36,9 @@ Each sequence has one bump-and-commit command: it derives the target version, wr For equal release numbers, SemVer compares alphanumeric prerelease identifiers lexically: `alpha` is lower than `canary`, `canary` is lower than `rc`, and every prerelease is lower than the stable version. npm dist-tags are mutable aliases and do not participate in version precedence. -### vendor: publish what changed, and let tags be the ledger +### vendor: advance the complete family with per-package tags -The vendored packages are decoupled from upstream by their scope but keep their own version lines. The published version is the higher of the manifest version and the last published version, with the patch incremented — which also drops an upstream prerelease segment. The first published versions: +The vendored packages are decoupled from upstream by their scope but keep their own version lines. The next version uses the higher of the manifest version and the latest tagged version, incrementing the patch after a stable version. The first published versions: | Package | Upstream version | First published version | |---|---|---| @@ -52,13 +52,13 @@ The vendored packages are decoupled from upstream by their scope but keep their | `@deepseek-ai/cordis-plugin-group` | 1.0.0 | 1.0.1 | | `@deepseek-ai/cordis-plugin-logger-console` | 1.0.0 | 1.0.1 | -Taking the last published version as the baseline is what survives a re-sync: upstream restoring `4.0.0-rc.8` after this repository published `4.0.1` would otherwise compute `4.0.1` again and collide. `--prerelease rc.1` publishes a rehearsal instead, which takes `--tag next` and leaves the release numbers free: a prerelease has lower precedence than the release it precedes, so `4.0.1` still follows `4.0.1-rc.1`. That ordering is computed here rather than read from `git tag --sort=v:refname`, which places a prerelease above its release. +Taking the latest tagged version as the baseline prevents reuse after a re-sync: upstream restoring `4.0.0-rc.8` after this repository tagged `4.0.1` would otherwise compute `4.0.1` again. `release:vendor --prerelease rc.1` or `--prerelease alpha.1` selects a prerelease, whose default npm dist-tag is `next`. Later prereleases and the stable release reuse its release numbers: `4.0.1` follows `4.0.1-rc.1`. The script compares SemVer precedence rather than using `git tag --sort=v:refname`, which places a prerelease above its release. -Only changed packages publish, and the change judgement adds no state file: **each package has its own tag, and that tag records the commit it last published from**. For each package, bump reads the newest `vendor--v*` tag and diffs the package directory against it. A path counts when the manifest's `files` selects it, when npm publishes it regardless (`package.json`, `README*`, `LICENSE*`), or — for a package whose `files` selects `lib/` — when it is a build input (`src/**`, `tsconfig*.json`, a build config). That last rule exists because a built payload is not tracked by git: without it, a real source change reads as "nothing changed" and the next publication fails on a version whose bytes moved. +Every vendor release advances all nine packages, each with its own version and `vendor--v` tag. Repacking from another repository state can change a member's dependency ranges or built payload even when its directory has no source changes. Advancing the complete family avoids reusing an existing version for those different bytes. -A tag is a commit pointer, not proof of publication. Bump asks the registry whether the version its newest tag names exists and fails for a human to resolve when it does not, because a tag pushed for a publication that then failed would otherwise read as "already published" and skip the package indefinitely. Querying a private package needs credentials, so an unauthenticated machine reports the gap instead of failing. +A tag reserves a package version and identifies its commit; it does not establish successful publication. Bump uses the latest tag even if its publication failed. The publisher checks registry versions and integrity, and the release manager verifies completion for the complete family. -`vendor/cordis` publishes `src` as well. Its export map declares `"./src/*"`, so a tarball without those files points consumers at absent paths, and `files` selecting only build output left the change judgement with no tracked path to match. +`vendor/cordis` publishes `src` as well. Its export map declares `"./src/*"`, so a tarball without those files points consumers at absent paths. ### Publication runs only on GitHub, and the registry decides what goes out @@ -76,6 +76,10 @@ All three sequences decide this way, including the native one: it publishes thro Two registry behaviours shape how a publish is attempted. Writes are spaced by at least two seconds and retried with a backoff, because publishing several packages back to back outruns the registry's own processing and earns `E409 Failed to save packument`. And every retry re-reads the registry first: a reported failure can answer a write that landed anyway, so a version that now exists with this tarball's integrity counts as published rather than as a version to place again. +For dsh and vendor, `release:publish --dist-tag ` overrides the family's default publication channel. The vendor publish workflow accepts the optional `dist-tag` input; omitting it preserves the family default. An override must be a valid npm dist-tag. Before publishing any tarball, the publisher checks every packed member: the selected tag must already name the intended version, or both that tag and the version must be absent. A tag bound elsewhere, or an already published version without the selected tag, rejects the entire attempt before any upload. Existing integrity checks and retries still apply. Skipped versions are never retagged. + +The preflight does not reserve npm tags atomically. The release manager coordinates external publishers that could change a binding after the check. A dedicated dist-tag also does not exclude stable versions from existing dependency ranges: isolation requires prerelease versions and an audit of the actual packed dependency ranges. + ### Workspace-internal references use the `workspace:` protocol Every reference to a workspace member uses the `workspace:` protocol. Every dependency section and consumer uses exact `workspace:*` DSH references and `workspace:~` vendor/native references, including the native entry's optional platform packages. Vendor and native patch releases must preserve consumer-facing APIs and binary interfaces. Local workspace linking is unchanged. @@ -146,9 +150,9 @@ The installed-consumer probe captures npm's HTTP diagnostics and includes them w **Event-level tags (`vendor-r1`, `vendor-r2`).** Prepared for one release event carrying several package versions. Once the registry decides what publishes, the workflow no longer infers the set from the tag, so per-package tags suffice — and each one names its own package's real version. -**Putting the nine vendored packages on one `4.0.x` line.** It removes change detection, but cosmokit would jump from `1.8.1` to `4.0.1` and lose its upstream lineage; the upstream ranges inside the nine (`^1.8.1` and friends) would stop matching immediately, forcing a rewrite of the vendored manifests. +**Putting the nine vendored packages on one `4.0.x` line.** Cosmokit would jump from `1.8.1` to `4.0.1` and lose its upstream lineage; the upstream ranges inside the nine (`^1.8.1` and friends) would stop matching immediately, forcing a rewrite of the vendored manifests. -**Incrementing every vendored package on every vendor release, with no change detection.** The least machinery, at the cost of new version numbers for packages whose content is byte-identical to the previous release. Tags reduce change detection to reading one tag and running one diff, which is not worth trading for inflated version numbers. +**Publishing only changed vendor directories.** A directory diff does not establish that repacking from another repository state produces identical bytes. Advancing all nine packages avoids integrity collisions for unchanged directories at the cost of additional versions. **Deciding "already published" from the version alone, without comparing content.** The reference flow queries no registry: publish uploads each tarball and npm rejects a duplicate version. Skipping on the version alone misses code that changed without a bump, which is the only failure that quietly leaves stale bytes on the registry. The cost is a registry query and a dependency on reproducible builds. @@ -166,14 +170,14 @@ The installed-consumer probe captures npm's HTTP diagnostics and includes them w ## Consequences -The release scripts are importable modules behind a guarded entry point, and their judgements carry unit tests: tag naming, publish order and cycle reporting, version-baseline arithmetic, the payload change judgement, and each family's payload policy. Two defects the first draft carried — a publish command that ran the pack command on import, and a change judgement blind to `vendor/cordis` source edits — are exactly what a test at that seam catches. +The release scripts are importable modules behind a guarded entry point, and their judgements carry unit tests: tag naming, publish order and cycle reporting, version-baseline arithmetic, and each family's payload policy. The entry guard prevents an import from executing a release command. A pull request runs the full pack for both sequences without credentials and installs the packed dsh tarballs into a throwaway consumer, where plain Node drives `dsh --version`. That probe is deliberately one command: it proves `files` selected a complete payload and that the published ranges resolve, and says nothing about interactive behavior. What this costs: -- **Tags can drift from the registry.** A tag pushed for a publication that then failed is caught by bump's registry check, but only where credentials exist; an unauthenticated machine reports the gap and continues. -- **The change judgement depends on visible tags.** A shallow clone, or a checkout without tags, degrades the vendored judgement to "publish everything for the first time". `fetch-depth: 0` is a precondition, not an optimization. +- **Tags can drift from the registry.** Tags reserve versions even when publication fails; the release manager verifies registry completion separately. +- **The version baseline depends on visible tags.** A shallow clone or a checkout without tags can reuse a reserved vendor version. `fetch-depth: 0` is a precondition, not an optimization. - **The protocol rewrite touched 1504 dependency declarations.** It does not change local resolution — pnpm already resolves from the workspace — but it changes the ranges that go out. - **Private packages need credentials to install.** Every consumer — CI, sandbox e2e, outside users — needs scope credentials, including for the Landlock packages, which have never been published and so cut off no existing anonymous path. - **`repository` names a different organization than the one running the workflows.** Token-based publication is unaffected; npm's OIDC attestation requires the two to agree, so adopting it means either repointing `repository` or publishing from the organization it names. diff --git a/.agents/notes/implemented/process/2026-08-10-npm-release-sequences.zh.md b/.agents/notes/implemented/process/2026-08-10-npm-release-sequences.zh.md index e0dec2439a..871221f86e 100644 --- a/.agents/notes/implemented/process/2026-08-10-npm-release-sequences.zh.md +++ b/.agents/notes/implemented/process/2026-08-10-npm-release-sequences.zh.md @@ -36,9 +36,9 @@ Status: implemented 基础版本号相同时,SemVer 按字典序比较字母数字型预发布标识:`alpha` 小于 `canary`,`canary` 小于 `rc`,所有预发布版本都小于稳定版本。npm dist-tag 是可变别名,不参与版本优先级比较。 -### vendor:谁改了谁发版,tag 就是账本 +### vendor:整族递增,每包保留自己的 tag -vendor 九包加了 scope 之后与上游脱钩,但保留各自的版本线。发布版本取「manifest 版本」与「上次发布版本」中较高的那个,再递增 patch——这一步同时去掉上游的预发布段。首发版本: +vendor 九包加了 scope 之后与上游脱钩,但保留各自的版本线。下一版本取 manifest 版本与最新 tag 版本中较高的那个,稳定版本之后递增 patch。首发版本: | 包 | 上游版本 | 首发版本 | |---|---|---| @@ -52,13 +52,13 @@ vendor 九包加了 scope 之后与上游脱钩,但保留各自的版本线。 | `@deepseek-ai/cordis-plugin-group` | 1.0.0 | 1.0.1 | | `@deepseek-ai/cordis-plugin-logger-console` | 1.0.0 | 1.0.1 | -以「上次发布版本」为基线才扛得住重同步:本仓发过 `4.0.1` 之后上游把版本恢复成 `4.0.0-rc.8`,只看 manifest 会再算出 `4.0.1` 并撞上已发版本。加 `--prerelease rc.1` 则发一次排练版:它进 `--tag next`,而且不占用那组数字——预发布的优先级低于它所先行的正式版,所以 `4.0.1` 仍然接在 `4.0.1-rc.1` 之后。这个次序由脚本自己算,不读 `git tag --sort=v:refname`——git 会把预发布排在正式版之前。 +以最新 tag 的版本为基线可避免重同步后复用版本:本仓标记 `4.0.1` 之后,上游若将版本恢复成 `4.0.0-rc.8`,只看 manifest 会再次算出 `4.0.1`。`release:vendor --prerelease rc.1` 或 `--prerelease alpha.1` 选择预发布版本,默认 npm dist-tag 为 `next`。后续预发布和稳定版本复用其基础版本号:`4.0.1` 接在 `4.0.1-rc.1` 之后。脚本按 SemVer 优先级比较,不使用会将预发布排在稳定版本之前的 `git tag --sort=v:refname`。 -只发改动过的包,而变更判据不引入新的状态文件:**每包一个 tag,tag 就是「上次发布到哪个 commit」的记录**。bump 对每个包取最新的 `vendor-<包名>-v*` tag,拿包目录与它做 diff。一条路径算命中的条件是:manifest 的 `files` 选中它,或 npm 无论如何都会发布它(`package.json`、`README*`、`LICENSE*`),或者——当该包的 `files` 选中 `lib/` 时——它是构建输入(`src/**`、`tsconfig*.json`、构建配置)。最后那条规则的存在理由是构建产物不在 git 里:没有它,真实的源码改动会读成「没变化」,而下一次发布会在一个字节已变的版本上失败。 +每次 vendor 发布都递增全部九个包,每包保留自己的版本和 `vendor--v` tag。即使包目录没有源码变更,从不同仓库状态重新打包也可能改变其依赖范围或构建产物。整族递增可避免为这些不同的字节复用已有版本。 -tag 只是 commit 指针,不是发布成功的证明。bump 会向 registry 核对「最新 tag 指向的版本是否真的存在」,不存在就明确失败交人处理——否则一个为失败发布而推的 tag 会被读成「已发布」,从此永远跳过该包。查询私有包需要凭据,因此未鉴权的机器只报告这道核对被跳过,不失败。 +tag 预留包版本并标识其 commit,不代表发布成功。即使对应发布失败,bump 仍使用最新 tag。publisher 核对 registry 中的版本和 integrity,发布负责人核实整族发布完成。 -`vendor/cordis` 现在也发布 `src`。它的 exports 声明了 `"./src/*"`,tarball 里没有这些文件就等于把消费方指向不存在的路径;而 `files` 只选构建产物,也让变更判据没有任何受 git 跟踪的路径可匹配。 +`vendor/cordis` 也发布 `src`。它的 exports 声明了 `"./src/*"`,tarball 中缺少这些文件就会将消费方指向不存在的路径。 ### 发布只在 GitHub 执行,由 registry 状态决定发什么 @@ -76,6 +76,10 @@ tag 只是 commit 指针,不是发布成功的证明。bump 会向 registry registry 的两个行为决定了「怎么尝试一次发布」。写入之间至少间隔两秒并带退避重试,因为连续背靠背发多个包会超出 registry 自身的处理速度,换来 `E409 Failed to save packument`。而每次重试都先重查 registry:报出来的失败可能对应一次其实已经落地的写入,所以「该版本现在存在且 integrity 与本 tarball 相同」算作已发布,而不是又一个待放置的版本。 +对 dsh 和 vendor,`release:publish --dist-tag ` 覆盖该族默认发布通道。vendor 发布 workflow 接受可选的 `dist-tag` 输入;省略时保留该族默认值。覆盖值必须是合法的 npm dist-tag。发布任何 tarball 前,publisher 检查每个打包成员:所选 tag 必须已指向预期版本,或者该 tag 和版本都不存在。tag 指向其他版本,或版本已经发布但缺少所选 tag,都会在上传前拒绝整次发布。原有 integrity 检查与重试继续生效。跳过的版本不会重新绑定 tag。 + +预检不会原子性地预留 npm tag。发布负责人协调可能在检查后更改绑定的外部发布者。专用 dist-tag 也不会让稳定版本被已有依赖范围排除:隔离需要预发布版本,并核查实际打包产物中的依赖范围。 + ### workspace 内部引用走 `workspace:` 协议 所有指向 workspace 成员的引用都用 `workspace:` 协议。每个消费者的所有依赖区段对 DSH 使用精确的 `workspace:*` 引用,对 vendor/native 使用 `workspace:~` 引用,包括原生入口的可选平台包。vendor 与 native 的补丁版本必须保持面向消费者的 API 和二进制接口兼容。本地 workspace 链接方式不变。 @@ -146,9 +150,9 @@ dsh 的验证会一并安装 vendored 族的 pack 产物。harness 的包把 ven **事件级 tag(`vendor-r1`、`vendor-r2`)。** 为「一次发布事件携带多个包版本」准备。既然由 registry 决定发什么,workflow 就不再从 tag 推断集合,per-package tag 够用,而且每个 tag 携带的是它自己那个包的真实版本。 -**把九个 vendored 包统一到一条 `4.0.x` 版本线。** 省掉变更检测,但 cosmokit 会从 `1.8.1` 跳到 `4.0.1`、丢失上游血缘;九包内部的上游范围(`^1.8.1` 之类)会立刻失配,必须改写 vendored manifest。 +**把九个 vendored 包统一到一条 `4.0.x` 版本线。** Cosmokit 会从 `1.8.1` 跳到 `4.0.1`、丢失上游血缘;九包内部的上游范围(`^1.8.1` 之类)会立刻失配,必须改写 vendored manifest。 -**每次 vendor 发布把九包全部 patch+1,不做变更检测。** 机制最少,代价是内容与上一版逐字节相同的包也拿到新版本号。tag 把变更检测的成本压到「读一个 tag、跑一次 diff」,不值得为省这点让版本号虚涨。 +**只发布目录有变更的 vendor 包。** 目录 diff 不能证明从不同仓库状态重新打包会产出相同字节。递增全部九个包可避免未改目录的 integrity 冲突,代价是增加版本号。 **只按版本号判断「是否已发布」,不比对内容。** 参照流程根本不查 registry:publish 逐个上传,重复版本由 npm 拒绝。只按版本号跳过会漏掉「改了代码没 bump」,而这是唯一会安静地把旧字节留在 registry 上的错误。代价是引入一次 registry 查询和对构建可复现性的依赖。 @@ -166,14 +170,14 @@ dsh 的验证会一并安装 vendored 族的 pack 产物。harness 的包把 ven ## 后果 -发布脚本是带入口守卫的可 import 模块,其判断都有单测覆盖:tag 命名、发布顺序与环报告、版本基线运算、payload 变更判据,以及各族的 payload 策略。第一版带过的两个缺陷——publish 命令在 import 时执行了 pack 命令、变更判据对 `vendor/cordis` 的源码改动失明——正是这类测试在对应接缝上能抓住的。 +发布脚本是带入口守卫的可 import 模块,其判断都有单测覆盖:tag 命名、发布顺序与环报告、版本基线运算,以及各族的 payload 策略。入口守卫阻止 import 执行发布命令。 一个 pull request 会为两条序列跑完整的 pack(无凭据),并把打包好的 dsh tarball 装进一次性 consumer,用普通 Node 驱动 `dsh --version`。这个探针刻意只有一条命令:它证明 `files` 选出了完整 payload、发布出去的范围可解析,不涉及任何交互行为。 代价: -- **tag 可能与 registry 漂移。** 为失败发布而推的 tag 由 bump 的 registry 核对拦下,但只在有凭据的地方;未鉴权的机器只报告这道核对被跳过。 -- **变更判据依赖 tag 可见。** shallow clone 或未拉 tag 会把 vendored 族的判据退化成「全部首发」。`fetch-depth: 0` 是前提,不是优化。 +- **tag 可能与 registry 漂移。** 即使发布失败,tag 仍预留版本;发布负责人另行核实 registry 中的完成状态。 +- **版本基线依赖 tag 可见。** shallow clone 或未拉取 tag 的 checkout 可能复用已预留的 vendor 版本。`fetch-depth: 0` 是前提,不是优化。 - **协议改写触及 1504 处依赖声明。** 它不改变本机解析(pnpm 本来就从 workspace 解析),但改变了发布出去的范围写法。 - **私有包需要凭据才能安装。** 任何消费方——CI、沙箱 e2e、外部使用者——都要持有 scope 凭据,Landlock 三包也在其中;它们从未发布过,所以没有切断既有的匿名安装路径。 - **`repository` 指向的组织与运行 workflow 的组织不同。** 用 token 发布不受影响;npm 的 OIDC attestation 要求二者一致,届时要么把 `repository` 改指过去,要么从它指向的组织发布。 diff --git a/.github/workflows/release-vendor-publish.yml b/.github/workflows/release-vendor-publish.yml index a50a408025..a7c620fad3 100644 --- a/.github/workflows/release-vendor-publish.yml +++ b/.github/workflows/release-vendor-publish.yml @@ -7,6 +7,12 @@ name: Release publish (vendor) on: workflow_dispatch: + inputs: + dist-tag: + description: Optional unused npm channel. Leave empty for latest (stable) or next (prerelease). + required: false + type: string + default: '' permissions: contents: read @@ -111,4 +117,10 @@ jobs: - name: Publish tarballs env: NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }} - run: pnpm run release:publish --family vendor --from dist/npm-vendor + RELEASE_DIST_TAG: ${{ inputs.dist-tag }} + run: | + args=() + if [[ -n "$RELEASE_DIST_TAG" ]]; then + args+=(--dist-tag "$RELEASE_DIST_TAG") + fi + pnpm run release:publish --family vendor --from dist/npm-vendor "${args[@]}" diff --git a/package.json b/package.json index 8e768b69e5..4e3703067c 100644 --- a/package.json +++ b/package.json @@ -225,6 +225,7 @@ "@types/jsdom": "^28.0.3", "@types/mdast": "^4.0.4", "@types/node": "^22.20.0", + "@types/semver": "^7.8.0", "@types/spdx-expression-parse": "^4.0.0", "@vitest/coverage-v8": "^4.1.8", "@yao-pkg/pkg": "6.21.0", @@ -248,6 +249,7 @@ "oxlint-tsgolint": "7.0.2001", "pnpm": "11.7.0", "publint": "^0.3.21", + "semver": "^7.8.5", "smol-toml": "^1.7.1", "spdx-expression-parse": "^5.0.0", "tar": "^7.5.0", diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 30f07acff0..0ee904dcb3 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -91,6 +91,9 @@ importers: '@types/node': specifier: ^22.20.0 version: 22.20.0 + '@types/semver': + specifier: ^7.8.0 + version: 7.8.0 '@types/spdx-expression-parse': specifier: ^4.0.0 version: 4.0.0 @@ -160,6 +163,9 @@ importers: publint: specifier: ^0.3.21 version: 0.3.21 + semver: + specifier: ^7.8.5 + version: 7.8.5 smol-toml: specifier: ^1.7.1 version: 1.7.1 diff --git a/scripts/ci-workflow.spec.ts b/scripts/ci-workflow.spec.ts index 615aedcb2c..dd52510c86 100644 --- a/scripts/ci-workflow.spec.ts +++ b/scripts/ci-workflow.spec.ts @@ -1132,6 +1132,37 @@ describe('Issue lifecycle workflow', () => { }) describe('npm release workflows', () => { + it('passes the optional vendor channel as a quoted argument while preserving default publication', () => { + const workflow = loadWorkflow('.github/workflows/release-vendor-publish.yml') + expect(workflow.on).toMatchObject({ + workflow_dispatch: { + inputs: { + 'dist-tag': { + required: false, + type: 'string', + default: '', + }, + }, + }, + }) + const publish = workflowJob(workflow, 'publish') + expect(publish.steps).toContainEqual({ + name: 'Publish tarballs', + env: { + NODE_AUTH_TOKEN: '${{ secrets.NPM_TOKEN }}', + RELEASE_DIST_TAG: '${{ inputs.dist-tag }}', + }, + run: [ + 'args=()', + 'if [[ -n "$RELEASE_DIST_TAG" ]]; then', + ' args+=(--dist-tag "$RELEASE_DIST_TAG")', + 'fi', + 'pnpm run release:publish --family vendor --from dist/npm-vendor "${args[@]}"', + '', + ].join('\n'), + }) + }) + it('keeps publication dispatch-only and pack in the PR workflow', () => { // pack stays in the PR/master release workflows so a PR proves the set packs. for (const file of ['release.yml', 'release-vendor.yml']) { diff --git a/scripts/release/publish.spec.ts b/scripts/release/publish.spec.ts new file mode 100644 index 0000000000..338df6aa7b --- /dev/null +++ b/scripts/release/publish.spec.ts @@ -0,0 +1,312 @@ +/** npm channel selection, collision prevention, and immutable-artifact retries. */ + +import { createHash } from 'node:crypto' +import { mkdtempSync, rmSync, writeFileSync } from 'node:fs' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import { setTimeout as sleep } from 'node:timers/promises' +import { afterEach, beforeEach, describe, expect, it, onTestFinished, vi } from 'vitest' +import { attempt, attemptEchoed, type CommandResult } from './process.ts' +import { publishRelease } from './publish.ts' +import { packedIdentity, readPublishOrder } from './tarball.ts' + +vi.mock('./process.ts', async importOriginal => ({ + ...await importOriginal(), + attempt: vi.fn(), + attemptEchoed: vi.fn(), + isEntry: () => false, +})) + +vi.mock('./tarball.ts', async importOriginal => ({ + ...await importOriginal(), + packedIdentity: vi.fn(), + readPublishOrder: vi.fn(), +})) + +vi.mock('node:timers/promises', async importOriginal => ({ + ...await importOriginal(), + setTimeout: vi.fn(), +})) + +const CHANNEL = 'dsh-0-2-1-alpha-1' +const ABSENT: CommandResult = { status: 1, stdout: '', stderr: 'npm error code E404' } +const SUCCESS: CommandResult = { status: 0, stdout: '', stderr: '' } +const CONFLICT: CommandResult = { status: 1, stdout: '', stderr: 'npm error code E409 Failed to save packument' } + +interface PackedFixture { + readonly name: string + readonly version: string + readonly path: string + readonly integrity: string +} + +function packedRelease(versions: readonly string[]): { directory: string; packages: PackedFixture[] } { + const directory = mkdtempSync(join(tmpdir(), 'dsh-publish-channel-')) + onTestFinished(() => { rmSync(directory, { recursive: true, force: true }) }) + const packages = versions.map((version, index) => { + const name = `@deepseek-ai/release-fixture-${String(index)}` + const path = join(directory, `${String(index)}.tgz`) + const bytes = Buffer.from(`${name}@${version}\n`) + writeFileSync(path, bytes) + return { + name, + version, + path, + integrity: `sha512-${createHash('sha512').update(bytes).digest('base64')}`, + } + }) + vi.mocked(readPublishOrder).mockReturnValue(packages.map((_, index) => `${String(index)}.tgz`)) + vi.mocked(packedIdentity).mockImplementation((path) => { + const entry = packages.find(entry => entry.path === path) + if (entry === undefined) throw new Error(`Unexpected tarball ${path}`) + return { name: entry.name, version: entry.version } + }) + return { directory, packages } +} + +function registry( + tags: ReadonlyMap = new Map(), + integrities: ReadonlyMap = new Map(), +): void { + vi.mocked(attempt).mockImplementation((command, args) => { + expect(command).toBe('npm') + if (args[0] === 'dist-tag') { + expect(args).toHaveLength(3) + expect(args[1]).toBe('ls') + const name = args[2] + if (name === undefined) throw new Error('Missing npm dist-tag package') + if (!tags.has(name)) return ABSENT + const payload = tags.get(name) + const stdout = payload !== null && typeof payload === 'object' && !Array.isArray(payload) + ? Object.entries(payload).map(([tag, version]) => `${tag}: ${String(version)}`).join('\n') + : JSON.stringify(payload) ?? 'undefined' + return { ...SUCCESS, stdout } + } + expect(args).toHaveLength(4) + expect(args[0]).toBe('view') + expect(args[3]).toBe('--json') + const selector = args[1] + if (selector === undefined) throw new Error('Missing npm view selector') + if (args[2] === 'dist.integrity') { + const integrity = integrities.get(selector) + return integrity === undefined ? ABSENT : { ...SUCCESS, stdout: JSON.stringify(integrity) } + } + throw new Error(`Unexpected npm view field ${String(args[2])}`) + }) +} + +beforeEach(() => { + vi.resetAllMocks() + vi.mocked(attemptEchoed).mockReturnValue(SUCCESS) + vi.mocked(sleep).mockResolvedValue(undefined) + vi.spyOn(console, 'log').mockImplementation(() => {}) +}) + +afterEach(() => { + vi.restoreAllMocks() + vi.resetAllMocks() +}) + +describe('release publication channels', () => { + it.each([ + ['vendor', '1.0.6', undefined], + ['vendor', '1.0.6-alpha.1', 'next'], + ['dsh', '0.2.1', undefined], + ['dsh', '0.2.1-alpha.1', 'alpha'], + ['dsh', '0.2.1-canary.1', 'canary'], + ['dsh', '0.2.1-rc.1', 'next'], + ] as const)('uses the default %s channel for %s when no override is supplied', async (family, version, tag) => { + const fixture = packedRelease([version]) + const [entry] = fixture.packages + expect(entry).toBeDefined() + registry() + + await publishRelease(family, fixture.directory) + + expect(attempt).toHaveBeenCalledExactlyOnceWith('npm', ['view', `${entry!.name}@${version}`, 'dist.integrity', '--json']) + expect(attemptEchoed).toHaveBeenCalledExactlyOnceWith('npm', [ + 'publish', entry!.path, ...tag === undefined ? [] : ['--tag', tag], + ]) + }) + + it('checks every custom channel before publishing in the recorded order', async () => { + const fixture = packedRelease(['1.0.6-alpha.1', '4.0.5-alpha.1']) + const [first, second] = fixture.packages + expect(first).toBeDefined() + expect(second).toBeDefined() + registry(new Map([[first!.name, { latest: '1.0.5', next: '1.0.5-rc.1' }]])) + vi.mocked(attemptEchoed).mockImplementation(() => { + expect(vi.mocked(attempt).mock.calls.filter(([, args]) => args[0] === 'dist-tag')).toEqual([ + ['npm', ['dist-tag', 'ls', first!.name]], + ['npm', ['dist-tag', 'ls', second!.name]], + ]) + return SUCCESS + }) + + await publishRelease('vendor', fixture.directory, CHANNEL) + + expect(vi.mocked(attemptEchoed).mock.calls).toEqual(fixture.packages.map(entry => [ + 'npm', ['publish', entry.path, '--tag', CHANNEL], + ])) + }) + + it('uses an explicit channel instead of the dsh prerelease default', async () => { + const fixture = packedRelease(['0.2.1-alpha.1']) + registry() + + await publishRelease('dsh', fixture.directory, CHANNEL) + + expect(attemptEchoed).toHaveBeenCalledExactlyOnceWith('npm', ['publish', fixture.packages[0]!.path, '--tag', CHANNEL]) + }) + + it.each(['', ' ', ' next', '-release-preview', 'release/channel', '1.2.3', '^1.2', 'v1', 'x'])( + 'rejects invalid channel %j before reading artifacts or querying npm', + async (tag) => { + await expect(publishRelease('vendor', '/unused-release-artifacts', tag)).rejects.toThrow() + + expect(readPublishOrder).not.toHaveBeenCalled() + expect(packedIdentity).not.toHaveBeenCalled() + expect(attempt).not.toHaveBeenCalled() + expect(attemptEchoed).not.toHaveBeenCalled() + }, + ) + + it('stops before publishing when a later package already assigns the custom channel elsewhere', async () => { + const fixture = packedRelease(['1.0.6-alpha.1', '4.0.5-alpha.1']) + const [first, second] = fixture.packages + registry(new Map([ + [first!.name, { latest: '1.0.5' }], + [second!.name, { [CHANNEL]: '4.0.4' }], + ])) + + await expect(publishRelease('vendor', fixture.directory, CHANNEL)).rejects.toThrow() + + expect(vi.mocked(attempt).mock.calls.filter(([, args]) => args[0] === 'dist-tag')).toEqual([ + ['npm', ['dist-tag', 'ls', first!.name]], + ['npm', ['dist-tag', 'ls', second!.name]], + ]) + expect(attemptEchoed).not.toHaveBeenCalled() + }) + + it.each([null, [], {}, 'not an object', { [CHANNEL]: 42 }])('rejects malformed registry channels %j before publishing', async (payload) => { + const fixture = packedRelease(['1.0.6-alpha.1']) + registry(new Map([[fixture.packages[0]!.name, payload]])) + + await expect(publishRelease('vendor', fixture.directory, CHANNEL)).rejects.toThrow() + + expect(attemptEchoed).not.toHaveBeenCalled() + }) + + it('stops on a registry authorization failure before publishing', async () => { + const fixture = packedRelease(['1.0.6-alpha.1']) + vi.mocked(attempt).mockReturnValue({ status: 1, stdout: '', stderr: 'npm error code E403' }) + + await expect(publishRelease('vendor', fixture.directory, CHANNEL)).rejects.toThrow(/E403/u) + + expect(attemptEchoed).not.toHaveBeenCalled() + }) + + it('rejects repeated channel bindings before publishing', async () => { + const fixture = packedRelease(['1.0.6-alpha.1']) + vi.mocked(attempt).mockReturnValue({ + ...SUCCESS, + stdout: `${CHANNEL}: 1.0.6-alpha.1\n${CHANNEL}: 1.0.6-alpha.1\n`, + }) + + await expect(publishRelease('vendor', fixture.directory, CHANNEL)).rejects.toThrow('invalid dist-tags') + + expect(attemptEchoed).not.toHaveBeenCalled() + }) + + it('skips an identical artifact with its custom channel and no latest tag', async () => { + const fixture = packedRelease(['1.0.6-alpha.1']) + const entry = fixture.packages[0]! + registry( + new Map([[entry.name, { [CHANNEL]: entry.version }]]), + new Map([[`${entry.name}@${entry.version}`, entry.integrity]]), + ) + + await publishRelease('vendor', fixture.directory, CHANNEL) + + expect(attempt).toHaveBeenCalledWith('npm', ['dist-tag', 'ls', entry.name]) + expect(attemptEchoed).not.toHaveBeenCalled() + expect(sleep).not.toHaveBeenCalled() + }) + + it('rejects an already published version whose custom channel is absent', async () => { + const fixture = packedRelease(['1.0.6-alpha.1']) + const entry = fixture.packages[0]! + registry(new Map([[entry.name, { latest: '1.0.5' }]]), new Map([[`${entry.name}@${entry.version}`, entry.integrity]])) + + await expect(publishRelease('vendor', fixture.directory, CHANNEL)).rejects.toThrow() + + expect(attemptEchoed).not.toHaveBeenCalled() + expect(attempt).toHaveBeenCalledWith('npm', ['view', `${entry.name}@${entry.version}`, 'dist.integrity', '--json']) + }) + + it('stops before the first upload when a later published version has no custom channel', async () => { + const fixture = packedRelease(['1.0.6-alpha.1', '4.0.5-alpha.1']) + const [first, second] = fixture.packages + registry( + new Map([[first!.name, { latest: '1.0.5' }], [second!.name, { latest: '4.0.4' }]]), + new Map([[`${second!.name}@${second!.version}`, second!.integrity]]), + ) + + await expect(publishRelease('vendor', fixture.directory, CHANNEL)).rejects.toThrow() + + expect(attempt).toHaveBeenCalledWith('npm', ['view', `${second!.name}@${second!.version}`, 'dist.integrity', '--json']) + expect(attemptEchoed).not.toHaveBeenCalled() + expect(sleep).not.toHaveBeenCalled() + }) + + it('preserves the explicit channel when retrying an E409 publication', async () => { + const fixture = packedRelease(['1.0.6-alpha.1']) + const entry = fixture.packages[0]! + registry(new Map([[entry.name, { latest: '1.0.5', next: '1.0.5-rc.1' }]])) + vi.mocked(attemptEchoed).mockReturnValueOnce(CONFLICT).mockReturnValueOnce(SUCCESS) + + await publishRelease('vendor', fixture.directory, CHANNEL) + + expect(vi.mocked(attemptEchoed).mock.calls).toEqual([ + ['npm', ['publish', entry.path, '--tag', CHANNEL]], + ['npm', ['publish', entry.path, '--tag', CHANNEL]], + ]) + expect(sleep).toHaveBeenCalledExactlyOnceWith(2_000) + }) + + it('keeps the explicit channel when an E409 publication already landed', async () => { + const fixture = packedRelease(['1.0.6-alpha.1']) + const entry = fixture.packages[0]! + const tags = new Map>([[entry.name, { latest: '1.0.5', next: '1.0.5-rc.1' }]]) + const integrities = new Map() + registry(tags, integrities) + vi.mocked(attemptEchoed).mockImplementationOnce(() => { + integrities.set(`${entry.name}@${entry.version}`, entry.integrity) + tags.set(entry.name, { ...tags.get(entry.name)!, [CHANNEL]: entry.version }) + return CONFLICT + }) + + await publishRelease('vendor', fixture.directory, CHANNEL) + + expect(attemptEchoed).toHaveBeenCalledExactlyOnceWith('npm', ['publish', entry.path, '--tag', CHANNEL]) + const lookupIndex = vi.mocked(attempt).mock.calls.findLastIndex(([, args]) => args[2] === 'dist.integrity') + expect(lookupIndex).toBeGreaterThanOrEqual(0) + expect(vi.mocked(attempt).mock.invocationCallOrder[lookupIndex]).toBeGreaterThan( + vi.mocked(attemptEchoed).mock.invocationCallOrder[0]!, + ) + expect(sleep).not.toHaveBeenCalled() + }) + + it('rejects different bytes already published at the requested version', async () => { + const fixture = packedRelease(['1.0.6-alpha.1']) + const entry = fixture.packages[0]! + registry( + new Map([[entry.name, { [CHANNEL]: entry.version }]]), + new Map([[`${entry.name}@${entry.version}`, 'sha512-different']]), + ) + + await expect(publishRelease('vendor', fixture.directory, CHANNEL)).rejects.toThrow('already published with different content') + + expect(attemptEchoed).not.toHaveBeenCalled() + }) +}) diff --git a/scripts/release/publish.ts b/scripts/release/publish.ts index 87427ac816..732ff46943 100644 --- a/scripts/release/publish.ts +++ b/scripts/release/publish.ts @@ -17,9 +17,10 @@ import { readFileSync } from 'node:fs' import { join, resolve } from 'node:path' import { setTimeout as sleep } from 'node:timers/promises' import { parseArgs } from 'node:util' +import { valid, validRange } from 'semver' import { releaseFamily } from './families.ts' import { attempt, attemptEchoed, isEntry } from './process.ts' -import { packedIdentity, readPublishOrder } from './tarball.ts' +import { packedIdentity, readPublishOrder, type PackedIdentity } from './tarball.ts' /** * Registry codes that answer a write which did not settle, rather than a @@ -84,6 +85,44 @@ function registryState(name: string, version: string): RegistryState { return { kind: 'present', integrity: parsed } } +/** Reject empty tags, npm option prefixes, characters requiring URI encoding, and version ranges. */ +function validateDistTag(distTag: string): void { + if (distTag === '' || distTag.startsWith('-') || encodeURIComponent(distTag) !== distTag || validRange(distTag) !== null) { + throw new Error(`Invalid npm dist-tag ${JSON.stringify(distTag)}: use a non-empty tag without a leading hyphen, characters requiring URI encoding, or a version range.`) + } +} + +/** An explicit channel may be new or already name the same version after a partial publication. */ +function verifyDistTag(member: PackedIdentity, distTag: string): void { + const result = attempt('npm', ['dist-tag', 'ls', member.name]) + if (result.status !== 0) { + const output = `${result.stdout}${result.stderr}` + if (output.includes('E404') || output.includes('404 Not Found')) return + throw new Error(`npm dist-tag ls ${member.name} failed:\n${output}`) + } + const tags = new Map() + for (const line of result.stdout.split(/\r?\n/u).filter(line => line !== '')) { + const match = /^(\S+): (\S+)$/u.exec(line) + const tag = match?.[1] + const version = match?.[2] + if (tag === undefined || version === undefined || valid(version) === null || tags.has(tag)) { + throw new Error(`npm reported invalid dist-tags for ${member.name}: ${JSON.stringify(line)}`) + } + tags.set(tag, version) + } + if (tags.size === 0) throw new Error(`npm reported no dist-tags for ${member.name}`) + const version = tags.get(distTag) + if (version === undefined) { + if (registryState(member.name, member.version).kind === 'present') { + throw new Error(`${member.name}@${member.version} is already published without dist-tag ${distTag}; this run cannot bind an existing version to a new channel.`) + } + return + } + if (version !== member.version) { + throw new Error(`${member.name}@${distTag} already points to ${version}; choose an unused dist-tag instead of replacing it with ${member.version}.`) + } +} + /** * Publish one tarball, retrying a registry write that did not settle. * @@ -127,30 +166,32 @@ async function publishTarball( } } -/** Publish the family named by `--family` from the directory named by `--from`. */ -async function main(): Promise { - const { values } = parseArgs({ - options: { family: { type: 'string' }, from: { type: 'string' } }, - allowPositionals: false, - }) - if (values.family === undefined || values.from === undefined) { - throw new Error('usage: publish.ts --family --from ') - } - - const family = releaseFamily(values.family) - const directory = resolve(process.cwd(), values.from) +/** + * Publish packed artifacts, preserving occupied explicit channels and existing version integrity. + * @param familyId - release family whose default channels apply when no override is supplied. + * @param directory - packed artifact directory containing the publication order. + * @param distTag - optional explicit npm channel, checked across the whole batch before uploading. + * @returns Resolves after every artifact has been published or verified as already present. + */ +export async function publishRelease(familyId: string, directory: string, distTag?: string): Promise { + if (distTag !== undefined) validateDistTag(distTag) + const family = releaseFamily(familyId) // Every entry in the order settles as either published or already present, so // one counter answers "how far along is this run" for whoever is watching a // release that takes minutes per family. - const order = readPublishOrder(directory) + const order = readPublishOrder(directory).map((filename) => { + const tarball = join(directory, filename) + return { tarball, ...packedIdentity(tarball) } + }) + if (distTag !== undefined) { + for (const member of order) verifyDistTag(member, distTag) + } const total = String(order.length) let published = 0 let skipped = 0 - for (const [index, filename] of order.entries()) { + for (const [index, { tarball, name, version }] of order.entries()) { const progress = `[${String(index + 1)}/${total}]` - const tarball = join(directory, filename) - const { name, version } = packedIdentity(tarball) const state = registryState(name, version) if (state.kind === 'present') { const local = integrityOf(tarball) @@ -168,7 +209,7 @@ async function main(): Promise { // Space out the writes: the gap belongs between publishes, so a run that // only skips does not wait at all. if (published > 0) await sleep(PUBLISH_SPACING_MS) - await publishTarball(tarball, name, version, family.distTagForVersion(version)) + await publishTarball(tarball, name, version, distTag ?? family.distTagForVersion(version)) console.log(`release publish: ${progress} ${name}@${version} published`) published += 1 } @@ -179,4 +220,16 @@ async function main(): Promise { ) } +/** Publish the family named by `--family` from the directory named by `--from`. */ +async function main(): Promise { + const { values } = parseArgs({ + options: { family: { type: 'string' }, from: { type: 'string' }, 'dist-tag': { type: 'string' } }, + allowPositionals: false, + }) + if (values.family === undefined || values.from === undefined) { + throw new Error('usage: publish.ts --family --from [--dist-tag ]') + } + await publishRelease(values.family, resolve(process.cwd(), values.from), values['dist-tag']) +} + if (isEntry(import.meta.url)) await main() diff --git a/scripts/tests/ci-release-selfhosted.spec.ts b/scripts/tests/ci-release-selfhosted.spec.ts index f228f711b3..b7da434bbb 100644 --- a/scripts/tests/ci-release-selfhosted.spec.ts +++ b/scripts/tests/ci-release-selfhosted.spec.ts @@ -161,6 +161,6 @@ for (const [file, jobIds] of [['release.yml', ['dependencies', 'pack']], ['relea it.each(['release-publish.yml', 'release-vendor-publish.yml'])('keeps %s manual and entirely hosted', (file) => { const publish = workflow(file) - expect(publish.on).toEqual({ workflow_dispatch: null }) + expect(Object.keys(publish.on)).toEqual(['workflow_dispatch']) for (const job of Object.values(publish.jobs)) expect(job['runs-on']).toBe(hosted) })