test(bench): reject swap-enabled cgroups and record the memory-limit decision

This commit is contained in:
Turtle committed 2026-10-07 21:20:08 +08:00
1 parent 51e2e5630f
commit 59fd47e451
6 files changed
+168 -13

No files matched your search

@@ -0,0 +1,28 @@
# Bilingual-pair consistency record for 2026-10-07-memory-limited-benchmarks.md (docs/i18n/README.md): per heading
# section, a hash of its English and Chinese blocks outside code blocks and generated regions.
# After editing either side, bring the other along and re-record with:
# pnpm run verify-translation-pairing --write .agents/notes/implemented/testing/2026-10-07-memory-limited-benchmarks.md
/agent-note-memory-limited-benchmark-reruns:
en: 08979e022b49c675
zh: b0c12b1f992ac681
/agent-note-memory-limited-benchmark-reruns/problem:
en: b82519a7026a1a19
zh: a72c4417426ec95f
/agent-note-memory-limited-benchmark-reruns/decision:
en: 3cf2db98939b3994
zh: f80a1d3d6049381a
/agent-note-memory-limited-benchmark-reruns/decision/cases:
en: 9502c61025d27f23
zh: f743a56404327f92
/agent-note-memory-limited-benchmark-reruns/decision/limit-and-endpoint:
en: a314408967c5a450
zh: 2b6ed3e1a96fe272
/agent-note-memory-limited-benchmark-reruns/calibration-evidence:
en: ffe31b42aa199614
zh: 29e4a3297ab55853
/agent-note-memory-limited-benchmark-reruns/alternatives-considered:
en: 436516beabc2394f
zh: 7cabbecb5a05f77c
/agent-note-memory-limited-benchmark-reruns/consequences:
en: cb213ef0e7506c8f
zh: 3da65a3981c2518d
@@ -0,0 +1,42 @@
# Agent Note: Memory-limited benchmark reruns
Status: implemented
English | [中文](2026-10-07-memory-limited-benchmarks.zh.md)
## Problem
Benchmarks run on hosted runners with 16 GB of memory and on developer machines with more. Neither shows how DSH behaves on a machine with 4 GB or 8 GB available, where the Host, Chromium, and page cache share one memory budget.
## Decision
[run-memory-pressure-bench.ts](../../../../scripts/run-memory-pressure-bench.ts) builds the benchmark artifacts unconfined, then reruns a fixed list of existing benchmark files inside a cgroup v2 whose `memory.max` is 4 or 8 GiB and whose `memory.swap.max` is zero. The files keep their own workloads and budgets; the scenario fails when a case misses its normal budget or the cgroup kills a process.
### Cases
The list holds the files with the largest measured resident memory or the long-history UI path: [session-corpus](../../../../benchmarks/session-corpus/README.md) (list of 3,000 Sessions, content search over 1,000, fork), its projection-list file, [session-open](2026-09-04-session-open-performance-gate.md) (127,400-event Session), and [long-session-browser](../../../../benchmarks/long-session-browser/README.md) (Chromium and its Host). Terminal, fold, reconnect, and continuation files measure small in-process workloads and stay out.
### Limit and endpoint
The cgroup contains Vitest, every benchmark worker, the scaffold Host, and Chromium. Node 24 sizes its default heap from the cgroup limit, and page cache from Session logs is charged to the same limit, so eviction and rereads appear in the timings. On Linux hosts a `systemd-run --user --scope` provides a fresh cgroup. In a container that already enforces the limit, `--scoped` uses the container cgroup. Before running, the launcher requires `memory.max`, `memory.swap.max`, and `process.constrainedMemory()` to match the machine. After the run it reports the cgroup's `memory.peak` and `oom_kill`; both cover the cgroup's lifetime.
## Calibration evidence
On 2026-10-07, Docker on Apple silicon (Linux VM, `--cpus=4`, Node 24.21) ran the list once per limit and once without a limit. All 38 cases passed their budgets under both limits, with no OOM kill.
| Measurement | No limit | 8 GiB | 4 GiB |
|---|---:|---:|---:|
| cgroup peak (MB) | 3,538 | 3,649 | 3,895 |
| List 3,000 first / repeat (ms) | 1,235 / 1,023 | 1,365 / 1,072 | 2,613 / 1,994 |
| Search 1,000 first (ms) | 54,015 | 50,766 | 61,662 |
| Browser slowest page / input (ms) | 152 / 95 | 144 / 91 | 187 / 123 |
## Alternatives considered
- `--max-old-space-size` per worker: the list and search workers use about 100 MB of JavaScript heap while their resident memory reaches 0.7–1.2 GB, mostly native memory, so a heap flag does not model the machine. Node already derives the same 2 GB old space for 4 GB and 8 GB cgroups.
- `RLIMIT_AS`: V8 reserves address space far beyond its resident memory and fails under a virtual-memory limit.
- A pull-request CI job: the PR workflow keeps at most ten concurrent worker jobs, and each memory-limited run takes as long as the benchmark lane, so the scenario is manual.
## Consequences
macOS has no per-process-tree memory cap; developers there run `--scoped` inside a Linux container started with `--memory` and `--memory-swap` equal to the limit. A file added to the list must keep its budgets valid under both limits.
@@ -0,0 +1,42 @@
# Agent Note: 内存受限的基准重跑
Status: implemented
[English](2026-10-07-memory-limited-benchmarks.md) | 中文
## 问题
基准运行在 16 GB 内存的托管 runner 和内存更大的开发机上。两者都无法显示 DSH 在可用内存为 4 GB 或 8 GB 的机器上的表现,而这类机器上 Host、Chromium 和页缓存共享同一份内存预算。
## 决定
[run-memory-pressure-bench.ts](../../../../scripts/run-memory-pressure-bench.ts) 先在无约束环境中构建基准产物,再在 `memory.max` 为 4 或 8 GiB、`memory.swap.max` 为零的 cgroup v2 中,重跑一份固定的现有基准文件列表。这些文件保留各自的负载和预算;某个用例未达到其常规预算或 cgroup 杀死进程时,该场景失败。
### 用例
列表包含常驻内存实测最大的文件和长历史 UI 路径:[session-corpus](../../../../benchmarks/session-corpus/README.zh.md)(3,000 个 Session 的列表、1,000 个 Session 的内容搜索、fork)、其 projection-list 文件、[session-open](2026-09-04-session-open-performance-gate.zh.md)(127,400 个事件的 Session)和 [long-session-browser](../../../../benchmarks/long-session-browser/README.zh.md)(Chromium 及其 Host)。终端、fold、重连和续写文件测量的是较小的进程内负载,不在列表中。
### 上限与端点
cgroup 包含 Vitest、每个基准 worker、脚手架 Host 和 Chromium。Node 24 按 cgroup 上限确定默认堆大小,Session 日志的页缓存也计入同一上限,因此淘汰和重读会体现在耗时中。在 Linux 主机上,`systemd-run --user --scope` 提供一个新的 cgroup。在已经执行该上限的容器中,`--scoped` 使用容器的 cgroup。运行前,启动脚本要求 `memory.max`、`memory.swap.max` 和 `process.constrainedMemory()` 与机器一致。运行后,它报告该 cgroup 的 `memory.peak` 和 `oom_kill`;两者覆盖 cgroup 的整个生命周期。
## 校准证据
2026-10-07,Apple silicon 上的 Docker(Linux VM,`--cpus=4`,Node 24.21)对每个上限各运行一次该列表,并在无上限时运行一次。两个上限下全部 38 个用例都通过了预算,没有 OOM kill。
| 测量项 | 无上限 | 8 GiB | 4 GiB |
|---|---:|---:|---:|
| cgroup 峰值(MB) | 3,538 | 3,649 | 3,895 |
| 列出 3,000 个:首次 / 重复(ms) | 1,235 / 1,023 | 1,365 / 1,072 | 2,613 / 1,994 |
| 搜索 1,000 个:首次(ms) | 54,015 | 50,766 | 61,662 |
| 浏览器最慢一页 / 输入(ms) | 152 / 95 | 144 / 91 | 187 / 123 |
## 考虑过的替代方案
- 为每个 worker 设置 `--max-old-space-size`:列表和搜索 worker 使用约 100 MB JavaScript 堆,而常驻内存达到 0.7–1.2 GB,主要是原生内存,因此堆参数无法模拟机器。Node 已为 4 GB 和 8 GB cgroup 推导出相同的 2 GB old space。
- `RLIMIT_AS`:V8 预留的地址空间远超其常驻内存,在虚拟内存上限下会失败。
- PR CI 作业:PR 工作流最多保持十个并发 worker 作业,而每次内存受限运行与基准 lane 耗时相当,因此该场景为手动运行。
## 影响
macOS 没有按进程树的内存上限;那里的开发者在以等于上限的 `--memory` 和 `--memory-swap` 启动的 Linux 容器中运行 `--scoped`。加入列表的文件必须在两个上限下都保持预算有效。
+1 -1
View File
@@ -14,4 +14,4 @@ This tree owns required, repository-level performance gates whose measured user
- Exercise production entry points. Do not copy product algorithms, add production exports solely for measurement, or turn benchmark completion into duplicate semantic assertions.
- A compiled worker may bundle a private integration adapter when no public Node export exposes the measured user path. Keep package imports external so product services resolve through their built package exports.
- Record the workload, timing boundary, memory endpoint, calibration reference, alternatives, and known exclusions in the owning Agent Note.
- `pnpm run test:bench:memory 4g|8g` reruns the files listed in [run-memory-pressure-bench.ts](../scripts/run-memory-pressure-bench.ts) inside a swapless cgroup capped at 4 or 8 GiB, with unchanged budgets. Keep a file's budgets valid under both caps when it joins that list.
- `pnpm run test:bench:memory 4g|8g` reruns the files listed in [run-memory-pressure-bench.ts](../scripts/run-memory-pressure-bench.ts) inside a swapless cgroup capped at 4 or 8 GiB, with unchanged budgets. Keep a file's budgets valid under both caps when it joins that list ([Agent Note](../.agents/notes/implemented/testing/2026-10-07-memory-limited-benchmarks.md)).
+15
View File
@@ -3,6 +3,7 @@ import { resolve } from 'node:path'
import { describe, expect, it } from 'vitest'
import {
MEMORY_PRESSURE_FILES,
memoryLimitViolation,
memoryPressureMachine,
memoryScopeCommand,
} from './run-memory-pressure-bench.ts'
@@ -26,4 +27,18 @@ describe('memory-pressure benchmark launcher', () => {
'systemd-run', '--user', '--scope', '--quiet', '-p', 'MemoryMax=4G', '-p', 'MemorySwapMax=0', '--', 'node', 'x.js',
])
})
it('accepts only a cgroup capped at the machine size without swap', () => {
const capped = { max: '4294967296', swapMax: '0', constrained: 4 * 1024 ** 3 }
expect(memoryLimitViolation('4g', capped)).toBeUndefined()
for (const limits of [
{ ...capped, max: 'max' },
{ ...capped, max: '8589934592' },
{ ...capped, swapMax: 'max' },
{ ...capped, swapMax: '4294967296' },
{ ...capped, constrained: 2 * 1024 ** 3 },
]) {
expect(memoryLimitViolation('4g', limits)).toContain('does not cap 4g without swap')
}
})
})
+40 -12
View File
@@ -4,8 +4,8 @@ import { spawnSync } from 'node:child_process'
import { readFileSync } from 'node:fs'
import { join, resolve } from 'node:path'
import { getHeapStatistics } from 'node:v8'
import { buildCiBenchArtifacts } from './run-ci-bench.ts'
import { pnpmCommand } from './release/process.ts'
import { buildCiBenchArtifacts } from './run-ci-bench.ts'
/** Modeled machines: memory available to the whole benchmark process tree, without swap. */
export const MEMORY_PRESSURE_MACHINES = { '4g': 4, '8g': 8 } as const
@@ -66,26 +66,54 @@ function cgroupFile(name: string): string {
return readFileSync(join('/sys/fs/cgroup', line.slice(3), name), 'utf8').trim()
}
/** Memory controls of the cgroup that contains the benchmark process tree. */
export interface CgroupMemoryLimits {
/** `memory.max` contents. */
readonly max: string
/** `memory.swap.max` contents. */
readonly swapMax: string
/** `process.constrainedMemory()` in bytes, which also reflects ancestor cgroups. */
readonly constrained: number
}
/**
* Run the selected benchmark files inside the current scope and report the scope's memory facts.
* Describe why a cgroup does not model the machine.
* @param machine - Modeled machine.
* @param limits - Memory controls of the current cgroup.
* @returns The violation, or `undefined` when memory is capped at the machine's size without swap.
*/
export function memoryLimitViolation(machine: MemoryPressureMachine, limits: CgroupMemoryLimits): string | undefined {
const limitBytes = String(MEMORY_PRESSURE_MACHINES[machine] * 1024 ** 3)
if (limits.max === limitBytes && limits.swapMax === '0' && String(limits.constrained) === limitBytes) return undefined
return `the cgroup does not cap ${machine} without swap: memory.max=${limits.max}, `
+ `memory.swap.max=${limits.swapMax}, constrainedMemory=${String(limits.constrained)}`
}
/**
* Run the selected benchmark files inside the current cgroup and report its memory facts.
* `memory.peak` and `oom_kill` cover the cgroup's lifetime: a fresh systemd scope contains only this run, while a
* container cgroup also contains earlier commands in that container. `memory.peak` needs Linux 5.19 or later.
* @param root - Repository root with built benchmark artifacts.
* @param machine - Modeled machine that the current scope must enforce.
* @param machine - Modeled machine that the current cgroup must enforce.
* @returns The benchmark exit status.
* @throws If the current cgroup does not enforce the machine's memory.
* @throws If the current cgroup does not cap memory at the machine's size without swap.
*/
export function runScopedMemoryPressureBench(root: string, machine: MemoryPressureMachine): number {
const limitBytes = MEMORY_PRESSURE_MACHINES[machine] * 1024 ** 3
if (Number(cgroupFile('memory.max')) !== limitBytes || process.constrainedMemory() !== limitBytes) {
throw new Error(`run-memory-pressure-bench: the scope does not enforce ${machine}: memory.max=${cgroupFile('memory.max')}`)
}
const violation = memoryLimitViolation(machine, {
max: cgroupFile('memory.max'),
swapMax: cgroupFile('memory.swap.max'),
constrained: process.constrainedMemory(),
})
if (violation !== undefined) throw new Error(`run-memory-pressure-bench: ${violation}`)
const status = run(root, [...pnpmCommand(), 'exec', 'vitest', 'run', '--config', 'vitest.bench.config.ts', ...MEMORY_PRESSURE_FILES])
const oomKills = /^oom_kill (\d+)$/m.exec(cgroupFile('memory.events'))?.[1]
if (oomKills === undefined) throw new Error('run-memory-pressure-bench: memory.events has no oom_kill counter')
console.log(JSON.stringify({
benchmark: `memory-pressure/${machine}`,
limitMb: limitBytes / 1024 ** 2,
swapLimit: cgroupFile('memory.swap.max'),
nodeHeapLimitMb: Math.round(getHeapStatistics().heap_size_limit / 1024 ** 2),
peakMb: Math.round(Number(cgroupFile('memory.peak')) / 1024 ** 2),
limitMb: MEMORY_PRESSURE_MACHINES[machine] * 1024,
// Heap limit Node derives from the cgroup for a process without an explicit `--max-old-space-size`.
defaultNodeHeapLimitMb: Math.round(getHeapStatistics().heap_size_limit / 1024 ** 2),
cgroupPeakMb: Math.round(Number(cgroupFile('memory.peak')) / 1024 ** 2),
oomKills: Number(oomKills),
files: MEMORY_PRESSURE_FILES,
status,