6.3 KiB
Agent Note: Blacksmith as a failover leg for CI pools
Status: implemented
English | 中文
Problem
The failover runbook routes degraded enterprise-pool traffic onto the in-house vm-backup and dsh-win-ci pools, which are proven but bounded standbys. Blacksmith sells GitHub-hosted-style runners on demand, and its migration wizard proposed replacing the repository's runs-on labels wholesale, which would put credential-holding, benchmark, and spec-pinned jobs on third-party infrastructure as the default. The wholesale replacement is not acceptable as a default: it breaks the workflow contract specs, distorts benchmark and performance-budget measurements, and moves trust-boundary jobs without a decision record.
Decision
DSH_CI_FAILOVER_LINUX and DSH_CI_FAILOVER_WINDOWS additionally accept the value blacksmith. When any repository writer sets it, the platform's participating jobs retarget onto Blacksmith's hosted runners at each job's configured vCPU size; any other value (or unset) keeps today's runner, and selfhosted keeps routing to the in-house pools. Like the existing values this is writer-manageable repository state — no merge, re-run the affected jobs — and it costs money only while set, which is why it is an explicit outage or experiment choice, never the default.
Participating jobs under blacksmith: the three enterprise Linux workers, the node-compat legs, the all-checks-passed verdict, and the three Windows jobs in ci.yml, using the per-job runner sizes declared there; expected-filenames; the sandbox bwrap leg; and the two manual benchmark matrices, whose 4-32 vCPU tiers map to Blacksmith equivalents while the 64/96-core rows keep the fleet labels because Blacksmith offers no such tier. The blacksmith branches carry no Dependabot exclusion: Blacksmith runners are ephemeral, so the persistent-VM rationale that keeps Dependabot on the hosted pool does not apply. The python-runtime builder stays hosted in full: its PR and master callers (ci.yml's and ci-master.yml's python-runtime jobs) pass real-API credentials (DEEPSEEK_API_KEY_EXTERNAL) into the reusable workflow, its macOS and arm64 single-exe legs need GitHub's images, and its wheel output feeds the PyPI publication chain. The docs-pages workflow is a tag-dispatched release publication, so it stays hosted too.
Everything else keeps its own runner under every value: jobs pinned to standard hosted runners by workflow spec (node-24-bench, python-sdk, the master Wine gate, release-publish and release-vendor-publish, the Cloudflare preview, the weighted-approval pair); jobs that hold credentials (the real-API e2e lanes, the issue App-token jobs, the npm and PyPI publishers); jobs that need a host capability Blacksmith lacks (the Landlock legs and their pack verification, the Seatbelt leg, the linux-arm64 single-exe leg); and the release-shaped chains — the npm publishers, the PyPI publishers, the node-addon-system release chain, and the Pages deployment workflow — which stay entirely hosted because their artifacts feed published registry output. The release rehearsal jobs in release.yml and release-vendor.yml (dependencies, pack) keep their existing pre-change routing unchanged: they carry only the pre-existing selfhosted branch, so under blacksmith they fall back to their hosted ubuntu-24.04 default exactly as under an unset variable, and they gain no blacksmith branch.
Alternatives considered
Adopting the migration wizard wholesale. Rejected because it breaks the workflow contract specs, changes benchmark semantics and performance-budget calibration, and moves credential-holding and trust-boundary jobs to third-party infrastructure as the default; the earlier wizard migration (closed PR #3053) was rejected for cost and the same review findings.
Keeping the in-house pools as the only failover target. Rejected as the sole option because they are a bounded standby; Blacksmith adds elastic capacity for the outage and experiment cases where the fleet itself is the degraded resource.
Consequences
The blacksmith legs stay dormant until the value is set, so they are exercised only during an actual failover or an explicit experiment. The mirror trial (PR #3841, closed) ran the participating set hardcoded onto the Blacksmith labels and confirmed that blacksmith-4vcpu-ubuntu-2404, blacksmith-16vcpu-ubuntu-2404, and blacksmith-16vcpu-windows-2025 schedule and execute. Blacksmith registers its runners as self-hosted, so under the blacksmith value the node-compat legs run their toolcache-isolation and isolated-installation verification steps (gated on runner.environment == 'self-hosted') rather than skipping them; all three legs passed on the trial image (run 34322356689). The same trial surfaced one environment finding that this change fixes: while expanding **, node 24.13's internal fs.glob lstat-probes <matched>/<next segment> for a symlinked file and throws ENOTDIR instead of skipping (run 34319926270), failing verify-md-wrap inside check:ci:static on Blacksmith until repo-files.ts replaced globSync with a dirent walker; the fixed static lane is green on the same image. The trigger is the lstat-probe of a path beneath a matched symlink, not ** over symlinked trees in general: the other globSync ** call sites that scan symlink-bearing trees stayed green on the same run and needed no change. The benchmark matrices' 8 and 32 vCPU labels (ubuntu-2404 and windows-2025) were not part of the trial and are inferred by naming symmetry from the verified 4 and 16 vCPU labels; confirm with a manual dispatch before first use. The Landlock, Seatbelt, and linux-arm64 exclusions still rest on the prose dispatch measurements of the earlier migration attempt (closed PR #3053), recorded here as a known gap; the trial could not exercise those legs by design. Workflow specs pin the new branches — ci-workflow.spec.ts asserts the Blacksmith labels appear in the extended selectors, and ci-compatible-selfhosted.spec.ts evaluates the blacksmith routing mode — so a future rewrite of the selectors fails the unit gates. Default cost posture is unchanged: nothing runs on Blacksmith unless the variable says so.