fix(ci): harden Python runtime builds against transient failures
Master's windows-python-console-spawn-wait fix removed the plugin-add flake this branch previously retried, so that retry is dropped entirely; the branch keeps the changes that remain useful. - Retry the immutable install step up to three attempts total with a ten-second pause (bash on every platform): node-gyp's Node-header download from nodejs.org intermittently times out, as observed on the hosted node24-macos-x64 cell. - Keep checkout credentials out of the build tree (persist-credentials: false) so dependency-install scripts cannot read the embedded token. - Failover runbook: drop the stray leading '#' before the Dependabot paragraph on both sides. - Hosted-runtime note: name the PR/commit ownership and UTC timestamps of the cited runs and record the constraints a future self-hosted attempt must satisfy; new note records the install retry decision.
This commit is contained in:
1 parent
6ac3258390
commit
1dc50c492e
10 files changed
+79
-11
No files matched your search
@@ -2,5 +2,5 @@
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write .agents/notes/implemented/process/2026-07-26-ci-failover-runbook.md
|
||||
2026-07-26-ci-failover-runbook.md: d5c12492671941c45cf3ccab255dd76fb53773bd
|
||||
2026-07-26-ci-failover-runbook.zh.md: b42a14dc9a3e4f40c59b27633752f6969173777e
|
||||
2026-07-26-ci-failover-runbook.md: a3c824cb54f5f24d02cee256c1e384061ac457f8
|
||||
2026-07-26-ci-failover-runbook.zh.md: 114b14dd5edcd8dab6bda40b78342738e3513392
|
||||
@@ -38,7 +38,7 @@ The two switches are independent: flip only the one whose platform is degraded.
|
||||
2. Retrigger the required jobs so they re-resolve their pool. Jobs already **queued** for the hosted labels do not retarget and cannot be re-run in place, so for the documented indefinite-queue outage, cancel the stuck run and re-run all jobs, or push a new commit; "Re-run failed jobs" only helps once a job has actually failed rather than queued.
|
||||
3. That is the entire switch. Under Linux failover the workflow also drops `DSH_SNAPSHOT_MAX_CONCURRENCY` to 12 for the shared VM and skips the hosted-path pnpm cache restores because the VM's persistent store serves warm installs. Coverage uses the same four single-worker instrumented partitions and two exempt workers on both Linux pools. The Windows switch has no concurrency or cache branches; it only retargets the native Windows jobs' pool.
|
||||
|
||||
#**Dependabot exception.** Both switches' selectors deliberately exclude `dependabot[bot]`: under failover, Dependabot PRs stay queued for the hosted pool rather than executing dependency-supplied code on the persistent VMs. A Dependabot PR that remains queued during an outage is expected behavior, not a failed switch; it completes when the hosted pool recovers.
|
||||
**Dependabot exception.** Both switches' selectors deliberately exclude `dependabot[bot]`: under failover, Dependabot PRs stay queued for the hosted pool rather than executing dependency-supplied code on the persistent VMs. A Dependabot PR that remains queued during an outage is expected behavior, not a failed switch; it completes when the hosted pool recovers.
|
||||
|
||||
**Who can flip the variable.** GitHub's API lets any collaborator with write access manage repository variables, so each switch is writer-level, not strictly admin-only. In this repository's trust model that is not an escalation: the runner groups admit all workflows of this private, fork-disabled repository (a deliberate trade to make PR-ref failover possible at all), so any writer could already reach the VMs by pushing a branch workflow. The boundary against untrusted code is repository membership; the variables only route work for members.
|
||||
|
||||
|
||||
@@ -38,7 +38,7 @@ Status: implemented
|
||||
2. 重新触发必需作业,使其重新解析运行器池。已经为托管标签**排队**的作业不会重定向,也无法原地 re-run,因此对于本手册所述的无限排队故障,应取消卡住的运行并 re-run all jobs,或推送一个新提交;“Re-run failed jobs”只有在作业真正失败(而非仍在排队)时才有用。
|
||||
3. 切换到此完成。Linux 故障切换状态下,工作流还会把 `DSH_SNAPSHOT_MAX_CONCURRENCY` 降为 12,以限制共享虚拟机上的争抢,并跳过托管路径的 pnpm 缓存恢复,因为虚拟机的持久 store 会直接提供热安装。覆盖率在两个 Linux 池上都使用 4 个单 worker 插桩分区与 2 个豁免 worker。Windows 开关没有并发或缓存分支;它只重定向原生 Windows 作业的运行器池。
|
||||
|
||||
#**Dependabot 例外。**两个开关的选择器都刻意排除了 `dependabot[bot]`:故障切换期间,Dependabot 拉取请求继续在托管池排队,而不是把依赖项提供的代码放到持久化虚拟机上执行。故障期间 Dependabot PR 持续排队是预期行为而非切换失败;托管池恢复后它会自行完成。
|
||||
**Dependabot 例外。**两个开关的选择器都刻意排除了 `dependabot[bot]`:故障切换期间,Dependabot 拉取请求继续在托管池排队,而不是把依赖项提供的代码放到持久化虚拟机上执行。故障期间 Dependabot PR 持续排队是预期行为而非切换失败;托管池恢复后它会自行完成。
|
||||
|
||||
**谁能扳动这个变量。**GitHub 的 API 允许任何具有写权限的协作者管理仓库变量,因此每个开关实际是写者级而非严格的管理员级。在本仓库的信任模型下这并不构成升权:runner group 接纳本私有、禁 fork 仓库的全部工作流(这是让 PR 引用的故障切换得以成立的刻意取舍),因此任何写者本就可以通过推送分支工作流触达这台虚拟机。抵御不可信代码的边界是仓库成员资格;变量只是为成员路由工作。
|
||||
|
||||
|
||||
+2
-2
@@ -2,5 +2,5 @@
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write .agents/notes/implemented/process/2026-09-06-python-runtime-windows-hosted.md
|
||||
2026-09-06-python-runtime-windows-hosted.md: ca2f02e8bac8a90be2b10bd6d7ae0b68215152ae
|
||||
2026-09-06-python-runtime-windows-hosted.zh.md: e1d2ca1a65de19a6604f0848de23fe5cc100e87f
|
||||
2026-09-06-python-runtime-windows-hosted.md: 8ae69d9836a07d9760c856d12bea29b4e09d1461
|
||||
2026-09-06-python-runtime-windows-hosted.zh.md: ea0b07c8131402efb60e226a6583b1c0e8faf4f0
|
||||
@@ -6,7 +6,7 @@ English | [中文](2026-09-06-python-runtime-windows-hosted.zh.md)
|
||||
|
||||
## Problem
|
||||
|
||||
The Windows x64 target in [build-exe-for-python-sdk.yml](../../../../.github/workflows/build-exe-for-python-sdk.yml) started resolving through `DSH_CI_FAILOVER_WINDOWS=selfhosted` for trusted pull-request CI when #3629 added the failover selector and the job-private Windows toolchain. The shared `dsh-win-ci` pool did not make the lane more reliable. On 2026-09-06 the installed-wheel smoke passed at 09:12 on `dsh-win-ci-16` for [an earlier commit of the same pull request](https://github.com/deepseek-harness/deepseek-harness/actions/runs/34023970384), then failed at 10:06 on `dsh-win-ci-21` for [another pull request](https://github.com/deepseek-harness/deepseek-harness/actions/runs/34026500701) and at 10:46 on `dsh-win-ci-04` for [the same pull request](https://github.com/deepseek-harness/deepseek-harness/actions/runs/34028339888/job/101473395734), where `smoke_sdk_profile_plugin`'s packaged `dsh plugin add` child exited without output while the Linux and macOS cells of that run passed. The migration proposal ([#3629](https://github.com/deepseek-harness/deepseek-harness/pull/3629)) remained `proposed` because its throughput and shared-load acceptance criteria were never measured.
|
||||
The Windows x64 target in [build-exe-for-python-sdk.yml](../../../../.github/workflows/build-exe-for-python-sdk.yml) started resolving through `DSH_CI_FAILOVER_WINDOWS=selfhosted` for trusted pull-request CI when #3629 added the failover selector and the job-private Windows toolchain. The shared `dsh-win-ci` pool did not make the lane more reliable. On 2026-09-06 (all times UTC; every run executed the #3629 migration workflow's selector, which was live from the 07:56 merge) the installed-wheel smoke passed at 09:12 on `dsh-win-ci-16` for [commit `ca3ffe95` of PR #3640 (`ci/benchmark-standard-runner`)](https://github.com/deepseek-harness/deepseek-harness/actions/runs/34023970384), then failed at 10:06 on `dsh-win-ci-21` for [PR #3337 (`feat/visualizer-host-plugin`)](https://github.com/deepseek-harness/deepseek-harness/actions/runs/34026500701) and at 10:46 on `dsh-win-ci-04` for [PR #3640 at its final head `c5ba873f`](https://github.com/deepseek-harness/deepseek-harness/actions/runs/34028339888/job/101473395734), where `smoke_sdk_profile_plugin`'s packaged `dsh plugin add` child exited without output while the Linux and macOS cells of that run passed; a job rerun at 11:29 repeated the same silent death. The migration proposal ([#3629](https://github.com/deepseek-harness/deepseek-harness/pull/3629)) remained `proposed` because its throughput and shared-load acceptance criteria were never measured.
|
||||
|
||||
## Decision
|
||||
|
||||
@@ -22,4 +22,4 @@ The Windows x64 target always uses its hosted `matrix.runner` — `windows-2025`
|
||||
|
||||
## Consequences
|
||||
|
||||
Every qualifying pull request again pays GitHub-hosted Windows capacity for the runtime build, and the job-private setup and cleanup machinery — including the bounded filesystem retries — is gone with the lane. In exchange each build runs on a disposable host with the proven toolchain and hosted caches, and the Windows failover switch covers only the native Windows jobs as documented before the migration. A future self-hosted attempt must re-validate throughput and failure reproducibility on the actual pool before any routing change.
|
||||
Every qualifying pull request again pays GitHub-hosted Windows capacity for the runtime build, and the job-private setup and cleanup machinery — including the bounded filesystem retries — is gone with the lane. In exchange each build runs on a disposable host with the proven toolchain and hosted caches, and the Windows failover switch covers only the native Windows jobs as documented before the migration. A future self-hosted attempt must re-validate throughput and failure reproducibility on the actual pool before any routing change, and must re-establish the constraints the retired #3629 proposal recorded: the setup-python Windows installer removes matching machine/current-user records and installs for all users (a private toolcache does not isolate that registry state), every build cache and temporary test root must be job-owned with copy imports rather than shared-store links, cleanup must run on success, failure, and cancellation with bounded Windows filesystem retries, and only the Windows x64 target is portable — the Linux target's manylinux checks need Docker.
|
||||
@@ -6,7 +6,7 @@ Status: implemented
|
||||
|
||||
## 问题
|
||||
|
||||
当 #3629 加入故障切换选择器与作业私有的 Windows 工具链后,[build-exe-for-python-sdk.yml](../../../../.github/workflows/build-exe-for-python-sdk.yml) 中的 Windows x64 目标开始对受信任的 PR CI 通过 `DSH_CI_FAILOVER_WINDOWS=selfhosted` 解析运行器。共享的 `dsh-win-ci` 池并未让该通道更可靠。2026-09-06,安装后 wheel 冒烟测试在 09:12 于 `dsh-win-ci-16` 上为[同一拉取请求的较早提交](https://github.com/deepseek-harness/deepseek-harness/actions/runs/34023970384)通过,随后 10:06 在 `dsh-win-ci-21` 上为[另一个拉取请求](https://github.com/deepseek-harness/deepseek-harness/actions/runs/34026500701)失败,10:46 在 `dsh-win-ci-04` 上为[同一拉取请求](https://github.com/deepseek-harness/deepseek-harness/actions/runs/34028339888/job/101473395734)失败——`smoke_sdk_profile_plugin` 打包的 `dsh plugin add` 子进程无输出即退出,而该次运行的 Linux 与 macOS 单元均通过。迁移提案([#3629](https://github.com/deepseek-harness/deepseek-harness/pull/3629))保持 `proposed`,因为其吞吐量与共享负载验收标准从未实测。
|
||||
当 #3629 加入故障切换选择器与作业私有的 Windows 工具链后,[build-exe-for-python-sdk.yml](../../../../.github/workflows/build-exe-for-python-sdk.yml) 中的 Windows x64 目标开始对受信任的 PR CI 通过 `DSH_CI_FAILOVER_WINDOWS=selfhosted` 解析运行器。共享的 `dsh-win-ci` 池并未让该通道更可靠。2026-09-06(所有时间均为 UTC;每次运行都执行 #3629 迁移工作流的选择器,该选择器自 07:56 合并起生效):安装后 wheel 冒烟测试在 09:12 于 `dsh-win-ci-16` 上为[PR #3640(`ci/benchmark-standard-runner`)的提交 `ca3ffe95`](https://github.com/deepseek-harness/deepseek-harness/actions/runs/34023970384)通过,随后 10:06 在 `dsh-win-ci-21` 上为[PR #3337(`feat/visualizer-host-plugin`)](https://github.com/deepseek-harness/deepseek-harness/actions/runs/34026500701)失败,10:46 在 `dsh-win-ci-04` 上为[PR #3640 的最终 head `c5ba873f`](https://github.com/deepseek-harness/deepseek-harness/actions/runs/34028339888/job/101473395734)失败——`smoke_sdk_profile_plugin` 打包的 `dsh plugin add` 子进程无输出即退出,而该次运行的 Linux 与 macOS 单元均通过;11:29 的作业重试再次出现相同的无声死亡。迁移提案([#3629](https://github.com/deepseek-harness/deepseek-harness/pull/3629))保持 `proposed`,因为其吞吐量与共享负载验收标准从未实测。
|
||||
|
||||
## 决策
|
||||
|
||||
@@ -22,4 +22,4 @@ Windows x64 目标始终使用托管的 `matrix.runner`——PR CI 为 `windows-
|
||||
|
||||
## 后果
|
||||
|
||||
每个符合条件的拉取请求再次为 runtime 构建支付 GitHub 托管 Windows 容量,作业私有准备与清理机制(包括有界文件系统重试)随通道一同移除。交换来的是每次构建运行在带标准工具链与托管缓存的一次性主机上,且 Windows 故障切换开关只覆盖迁移前文档所述的原生 Windows 作业。未来的自托管尝试必须在任何路由变更前,对实际池重新验证吞吐量与失败可复现性。
|
||||
每个符合条件的拉取请求再次为 runtime 构建支付 GitHub 托管 Windows 容量,作业私有准备与清理机制(包括有界文件系统重试)随通道一同移除。交换来的是每次构建运行在带标准工具链与托管缓存的一次性主机上,且 Windows 故障切换开关只覆盖迁移前文档所述的原生 Windows 作业。未来的自托管尝试必须在任何路由变更前,对实际池重新验证吞吐量与失败可复现性,并且必须重建已退役 #3629 提案记录的约束:setup-python 的 Windows 安装器会删除匹配的机器/当前用户安装记录并为所有用户安装(私有工具缓存无法隔离这些注册表状态),每个构建缓存与临时测试根目录必须作业私有并使用复制导入而非共享 store 链接,清理必须在成功、失败与取消路径上以有界 Windows 文件系统重试执行,且只有 Windows x64 目标可移植——Linux 目标的 manylinux 检查需要 Docker。
|
||||
@@ -0,0 +1,6 @@
|
||||
# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write .agents/notes/implemented/testing/2026-09-06-python-runtime-install-retry.md
|
||||
2026-09-06-python-runtime-install-retry.md: c52e87c2d18393113c0e884d7e6d1e56cc6e6145
|
||||
2026-09-06-python-runtime-install-retry.zh.md: 7a7943f43fcec536542c5aef0e608e177f64907f
|
||||
@@ -0,0 +1,25 @@
|
||||
# Agent Note: Bounded retry for Python runtime dependency install
|
||||
|
||||
Status: implemented
|
||||
|
||||
English | [中文](2026-09-06-python-runtime-install-retry.zh.md)
|
||||
|
||||
## Problem
|
||||
|
||||
The Python runtime lane's `Install (immutable)` step runs `pnpm install` on every target, and install-time native build downloads fetch Node headers from nodejs.org. That endpoint stalls intermittently: on 2026-09-06 the hosted `node24-macos-x64` cell failed when the `fs-ext` build's node-gyp download raised `ConnectTimeoutError` against nodejs.org after a 10-second connect timeout, aborting the immutable install. The stall is external and transient; the lane previously had no recovery beyond a human job rerun.
|
||||
|
||||
## Decision
|
||||
|
||||
The install step retries `pnpm install --frozen-lockfile` up to three attempts total with a ten-second pause between failures, running under `bash` on every platform (Git Bash is on the hosted Windows images). Success on any attempt ends the step immediately; a file-lock check or native-build error that would fail every attempt still fails the step after the bounded budget. This mirrors the Wine lane's documented bounded-transfer policy without pulling in a mirror, because these installs also resolve native addons whose second-download provenance matters.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
**Increase the connect or job timeout.** Rejected: the observed stall is a connect timeout after 10 seconds, and retrying the whole operation with a fresh connection is the recovery the failure mode calls for; a longer timeout still fails when the endpoint is down.
|
||||
|
||||
**Use a mirror for Node header downloads.** Deferred: the Wine lane's mirror resumes its own archive; the Python runtime lane would need a per-target mirror and its own checksum authority, which the retry does not require for a transient outage.
|
||||
|
||||
**Rerun failed jobs by hand.** Rejected as the lane's standing remediation: it costs a full lane cycle and stays manual; the bounded retry absorbs the transient while a sustained outage still fails loudly.
|
||||
|
||||
## Consequences
|
||||
|
||||
A transient nodejs.org stall costs at most two extra install attempts (about twenty seconds), while a deterministic install defect still fails after the budget. All targets share the same retry path, and install diagnostics remain the pnpm output captured inside the step.
|
||||
@@ -0,0 +1,25 @@
|
||||
# Agent Note: Python runtime 依赖安装的有界重试
|
||||
|
||||
Status: implemented
|
||||
|
||||
[English](2026-09-06-python-runtime-install-retry.md) | 中文
|
||||
|
||||
## 问题
|
||||
|
||||
Python runtime 通道的 `Install (immutable)` 步骤在每个目标上运行 `pnpm install`,安装期原生构建下载会从 nodejs.org 获取 Node 头文件。该端点会间歇性停滞:2026-09-06,托管的 `node24-macos-x64` 单元在 `fs-ext` 构建的 node-gyp 下载对 nodejs.org 抛出 10 秒连接超时后的 `ConnectTimeoutError` 时失败,使不可变安装中止。该停滞属于外部、瞬时问题;此前通道除人工重跑作业外没有任何恢复手段。
|
||||
|
||||
## 决策
|
||||
|
||||
安装步骤对 `pnpm install --frozen-lockfile` 最多重试三次(共三次尝试,失败间隔十秒),并在每个平台上以 `bash` 运行(托管 Windows 镜像自带 Git Bash)。任意一次成功立即结束步骤;每次尝试都会失败的文件锁检查或原生构建错误仍会在有界预算后使步骤失败。这借鉴了 Wine 通道已记录的有界传输策略,而不引入镜像,因为这些安装还会解析原生 addon,其二次下载来源同样重要。
|
||||
|
||||
## 已考虑的替代方案
|
||||
|
||||
**提高连接或作业超时。** 不采用:观察到的停滞是 10 秒后的连接超时,而对该失败模式适用的恢复是携带新连接的整体重试;端点故障时更长的超时仍然失败。
|
||||
|
||||
**为 Node 头文件下载使用镜像。** 推迟:Wine 通道的镜像只续传自己的归档;Python runtime 通道需要按目标配置镜像及自己的校验权威,而瞬态故障的重试并不需要这些。
|
||||
|
||||
**人工重跑失败作业。** 不采用为该通道的常设补救:它消耗一整个通道周期且停留在手动层面;有界重试吸收瞬态故障,持续性故障仍会响亮失败。
|
||||
|
||||
## 后果
|
||||
|
||||
一次 nodejs.org 瞬态停滞最多付出两次额外安装尝试(约二十秒),确定性的安装缺陷仍会在预算后失败。所有目标共享同一重试路径,安装诊断仍为步骤内捕获的 pnpm 输出。
|
||||
@@ -158,6 +158,8 @@ jobs:
|
||||
include: ${{ fromJSON(needs.plan.outputs.matrix) }}
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
with:
|
||||
persist-credentials: false
|
||||
|
||||
- uses: pnpm/action-setup@v4
|
||||
with:
|
||||
@@ -194,7 +196,17 @@ jobs:
|
||||
pkg-fetch-${{ matrix.target }}-
|
||||
|
||||
- name: Install (immutable)
|
||||
run: pnpm install --frozen-lockfile
|
||||
# node-gyp fetches Node headers from nodejs.org during install; that
|
||||
# endpoint stalls intermittently (observed 10 s connect timeouts).
|
||||
# Bounded retries mirror the wine lane's transfer policy without a
|
||||
# mirror, since these installs also resolve native addons.
|
||||
shell: bash
|
||||
run: |
|
||||
for attempt in 1 2 3; do
|
||||
if pnpm install --frozen-lockfile; then exit 0; fi
|
||||
[ "$attempt" -lt 3 ] && sleep 10
|
||||
done
|
||||
exit 1
|
||||
|
||||
- name: Rebuild Linux node-pty against manylinux 2.28
|
||||
if: runner.os == 'Linux'
|
||||
|
||||
Reference in new issue
Block a user