test(bench): address Session corpus review findings
- Fork the measured p99 at rank 988, the highest top-level rank of that anchor; rank 990 fell in the 99.3% anchor. Derive fork ranks from the corpus size and pin their shapes in a unit test. - Seed separate search and fork roots so fork children never reach the search index. - Bound child deadlines by the file limit and give every test a deadline that outlasts its sequential children, so failures reap children before cleanup. - Budget peak RSS of the list, search, and fork processes, and check every recorded median against its budget. - Name a missing folded record, correct the list-order comment, state that search models an opt-in deployment, and keep new dependencies sorted.
This commit is contained in:
turtle1999 committed
1 parent
74fcbdd625
commit
2778be7375
10 files changed
+182
-119
No files matched your search
@@ -15,11 +15,11 @@
|
||||
en: ae4682fdb06d3192
|
||||
zh: 6a48d412f15bedfa
|
||||
/agent-note-performance-baselines-for-large-session-corpora/decision/corpus-construction:
|
||||
en: f96ecfd1d8754c9d
|
||||
zh: f2ef6f437a4a5030
|
||||
en: 1310a89c351e5bfe
|
||||
zh: e6386ae10b893c2d
|
||||
/agent-note-performance-baselines-for-large-session-corpora/decision/measured-endpoints:
|
||||
en: d9406e015ec714f3
|
||||
zh: f9cf3fbf2f0f2f7c
|
||||
en: b4d81575ac574f6f
|
||||
zh: f4f291ab7ce83094
|
||||
/agent-note-performance-baselines-for-large-session-corpora/calibration-evidence:
|
||||
en: 293174ed71f075d7
|
||||
zh: 8745652547f72a66
|
||||
@@ -27,5 +27,5 @@
|
||||
en: 4eaa4666262a1aee
|
||||
zh: 7e2eb44b94bad05d
|
||||
/agent-note-performance-baselines-for-large-session-corpora/consequences:
|
||||
en: f07509e956d3c017
|
||||
zh: e23aefb878138555
|
||||
en: f68218a31c17e0e6
|
||||
zh: 2ffd8943e41774bf
|
||||
@@ -29,7 +29,7 @@ Other measured aggregates also set the fixture: 19 Sessions per project director
|
||||
|
||||
### Corpus construction
|
||||
|
||||
Two preparation processes, one per CPU of a standard hosted runner, split the anchors into byte-balanced groups. Each authors one event body per anchor, compresses it once, and saves it with the anchor's projection record. A seed process then writes each Session as its own header frame followed by its anchor's body. Frame compression and file writes run eight at a time: each pending compression holds a native Zstandard context, and unbounded compression peaked at 7.7 GB RSS, more than a standard hosted runner provides. Sessions sharing an anchor differ only in header identity, creation time, project directory, and parent link. Creation time is a fixed permutation of rank, so list order does not follow length. Events carry the fields that `Session.append` records, compact streams come from the production `AssistantStreamAccumulator`, and rows are encoded with the persistence package's `eventLines`. Before saving, each preparation process replays its authored bodies through `Session.create`, as `sessionQuery.readSession` does.
|
||||
Two preparation processes, one per CPU of a standard hosted runner, split the anchors into byte-balanced groups. Each authors one event body per anchor, compresses it once, and saves it with the anchor's projection record. A seed process then writes each Session as its own header frame followed by its anchor's body. Frame compression and file writes run eight at a time: each pending compression holds a native Zstandard context, and unbounded compression peaked at 7.7 GB RSS, more than a standard hosted runner provides. Sessions sharing an anchor differ only in header identity, creation time, project directory, and parent link. Creation time is a fixed permutation of rank. List order follows each body's last prompt time, which Sessions sharing an anchor share; it does not affect list cost. Events carry the fields that `Session.append` records, compact streams come from the production `AssistantStreamAccumulator`, and rows are encoded with the persistence package's `eventLines`. Before saving, each preparation process replays its authored bodies through `Session.create`, as `sessionQuery.readSession` does.
|
||||
|
||||
The projection cache is part of the list endpoint: each cold row carries cached projection values. Preparation folds each anchor once through the public `sessionProjectionCache.coldSnapshot` write-back. The seed then writes per-record documents for every Session with `serializeRecord`, rebinding identity to each header. Writing through the domain API costs one fsync per record, which took 89 s for 10,000 Sessions. The list worker requires every returned row to carry projections, so a storage layout change fails the benchmark instead of silently removing that work.
|
||||
|
||||
@@ -41,9 +41,9 @@ Workers are compiled plain-Node processes, each started fresh against a seeded r
|
||||
|---|---|---:|---|
|
||||
| List | 3,000 | 3 | Host boot; first `session.list`; repeated `session.list` in the same process |
|
||||
| Content search | 1,000 | 1 | First `session.search`, including the index build; a second query with a different term |
|
||||
| Fork | 1,000 | 3, then 1 | `session.fork` return for rank midpoints of ten equal length strata and rank 990; then rank 999 once |
|
||||
| Fork | 1,000, separate root | 3, then 1 | `session.fork` return for rank midpoints of ten equal length strata and rank 988; then rank 999 once |
|
||||
|
||||
The file must finish within five minutes on standard hosted CI, including seeding, and its last case asserts that limit. Search over 1,000 Sessions and one fork of the longest Session take about 170 s of it, so each has one sample. The list corpus is the largest that fits the remainder. Budgets enforce medians of list boot, first, and repeat, and of the fork median over strata and rank 990; search and the longest fork enforce their single samples. Reports include peak RSS and heap after the endpoint, but no memory budget applies.
|
||||
The file must finish within five minutes on standard hosted CI, including seeding, and its last case asserts that limit. Search over 1,000 Sessions and one fork of the longest Session take about 170 s of it, so each has one sample. The list corpus is the largest that fits the remainder. Budgets enforce medians of list boot, first, and repeat, and of the fork median over strata and rank 990; search and the longest fork enforce their single samples. Search and fork use separate 1,000-Session roots, so fork children never reach the search index. Rank 988 is the highest top-level rank at the measured p99 (7,229 events); forking a subagent child adds a lineage lookup, and rank 990 already falls in the 99.3% anchor. Peak RSS of the list, search, and fork processes has a budget with the shared 1.25 headroom and no time scale. Each child's deadline equals the file limit, and each test's deadline outlasts its sequential children, so a failed case reaps every child before the scratch root is removed.
|
||||
|
||||
## Calibration evidence
|
||||
|
||||
@@ -74,6 +74,6 @@ On Apple M5 Pro with Node 26.5, isolated runs measured list 1,000 at 311–358 m
|
||||
|
||||
## Consequences
|
||||
|
||||
The gate covers corpus-scale list, search, and fork costs that single-Session gates cannot see. On standard hosted CI it already shows that content search spends 139 s building its index on first use, that the first list took 5.1 s at 5,000 Sessions in calibration, and that forking the longest Session takes 17.5 to 20.3 s. Seeding writes about 1.2 GB.
|
||||
The gate covers corpus-scale list, search, and fork costs that single-Session gates cannot see. On standard hosted CI it already shows that content search spends 139 s building its index on first use, that the first list took 5.1 s at 5,000 Sessions in calibration, and that forking the longest Session takes 17.5 to 20.3 s. Seeding writes about 1.5 GB.
|
||||
|
||||
Exclusions: the corpus is current-generation only, although most real Sessions are stored in older generations, which list revision hashing and migration-on-read would add. Only 22 distinct bodies exist, so search indexes repeated text. Fork runs without Agent presets or Workspace attachment. Calls are in-process, without Typert transport or a browser. Filesystem caches are not evicted, so cold means a fresh process, not cold storage.
|
||||
@@ -29,7 +29,7 @@ Status: implemented
|
||||
|
||||
### 语料构建
|
||||
|
||||
两个准备进程（对应标准托管运行器的每个 CPU）把锚点分成字节均衡的两组。每个进程为每个锚点生成一份事件主体，只压缩一次，并与该锚点的投影记录一起保存。随后播种进程把每个 Session 写成独立的 header 帧加上其锚点的主体。帧压缩和文件写入每次并发八个：每个待完成的压缩都持有一个原生 Zstandard 上下文，不加限制的压缩峰值 RSS 达到 7.7 GB，超过标准托管运行器的内存。共享锚点的 Session 只在 header 身份、创建时间、项目目录和父链接上不同。创建时间是排名的固定置换，因此列表顺序不跟随长度。事件携带 `Session.append` 记录的字段，紧凑流来自生产 `AssistantStreamAccumulator`，行通过持久化包的 `eventLines` 编码。保存之前，每个准备进程按 `sessionQuery.readSession` 的方式，通过 `Session.create` 回放其生成的主体。
|
||||
两个准备进程（对应标准托管运行器的每个 CPU）把锚点分成字节均衡的两组。每个进程为每个锚点生成一份事件主体，只压缩一次，并与该锚点的投影记录一起保存。随后播种进程把每个 Session 写成独立的 header 帧加上其锚点的主体。帧压缩和文件写入每次并发八个：每个待完成的压缩都持有一个原生 Zstandard 上下文，不加限制的压缩峰值 RSS 达到 7.7 GB，超过标准托管运行器的内存。共享锚点的 Session 只在 header 身份、创建时间、项目目录和父链接上不同。创建时间是排名的固定置换。列表顺序跟随各主体的最后提示时间，共享锚点的 Session 也共享该时间；这不影响列表成本。事件携带 `Session.append` 记录的字段，紧凑流来自生产 `AssistantStreamAccumulator`，行通过持久化包的 `eventLines` 编码。保存之前，每个准备进程按 `sessionQuery.readSession` 的方式，通过 `Session.create` 回放其生成的主体。
|
||||
|
||||
投影缓存属于列表端点：每个冷行都携带缓存的投影值。准备过程通过公开的 `sessionProjectionCache.coldSnapshot` 回写，为每个锚点折叠一次。之后播种进程用 `serializeRecord` 为每个 Session 写入逐记录文档，并把身份重新绑定到各自的 header。通过 domain API 写入时每条记录一次 fsync，10,000 个 Session 耗时 89 s。列表 worker 要求返回的每一行都携带投影，因此存储布局变化会使基准失败，而不是静默去掉这部分工作。
|
||||
|
||||
@@ -41,9 +41,9 @@ worker 是编译后的纯 Node 进程，每次都针对已播种根目录全新
|
||||
|---|---|---:|---|
|
||||
| 列表 | 3,000 | 3 | Host 启动；首次 `session.list`；同一进程中重复的 `session.list` |
|
||||
| 内容搜索 | 1,000 | 1 | 首次 `session.search`，包括索引构建；使用不同词项的第二次查询 |
|
||||
| Fork | 1,000 | 3，然后 1 | 十个等长度分层的中点排名和排名 990 的 `session.fork` 返回；然后排名 999 一次 |
|
||||
| Fork | 1,000，独立根目录 | 3，然后 1 | 十个等长度分层的中点排名和排名 988 的 `session.fork` 返回；然后排名 999 一次 |
|
||||
|
||||
此文件（包括播种）必须在标准托管 CI 上五分钟内完成，其最后一个用例断言该上限。1,000 个 Session 的搜索和最长 Session 的一次 fork 约占其中 170 s，因此两者各取一个样本。列表语料是剩余时间内能容纳的最大规模。预算约束列表启动、首次和重复的中位数，以及 fork 分层中位数和排名 990 的中位数；搜索和最长 fork 约束各自的单个样本。报告包含端点之后的峰值 RSS 和堆，但不设内存预算。
|
||||
此文件（包括播种）必须在标准托管 CI 上五分钟内完成，其最后一个用例断言该上限。1,000 个 Session 的搜索和最长 Session 的一次 fork 约占其中 170 s，因此两者各取一个样本。列表语料是剩余时间内能容纳的最大规模。预算约束列表启动、首次和重复的中位数，以及 fork 分层中位数和排名 990 的中位数；搜索和最长 fork 约束各自的单个样本。搜索和 fork 使用各自独立的 1,000 个 Session 根目录，因此 fork 子会话不会进入搜索索引。排名 988 是实测 p99（7,229 个事件）上最高的顶层排名；fork subagent 子会话会增加一次谱系查找，而排名 990 已落入 99.3% 锚点。列表、搜索和 fork 进程的峰值 RSS 设有预算，使用共享的 1.25 余量，不做时间缩放。每个子进程的截止时间等于文件上限，每个测试的截止时间长于其顺序子进程之和，因此失败的用例会在删除临时根目录之前回收所有子进程。
|
||||
|
||||
## 校准证据
|
||||
|
||||
@@ -74,6 +74,6 @@ worker 是编译后的纯 Node 进程，每次都针对已播种根目录全新
|
||||
|
||||
## 后果
|
||||
|
||||
该门禁覆盖单 Session 门禁看不到的语料规模列表、搜索和 fork 成本。在标准托管 CI 上，它已经显示：内容搜索在首次使用时花 139 s 构建索引，校准时 5,000 个 Session 的首次列表耗时 5.1 s，fork 最长 Session 耗时 17.5 到 20.3 s。播种写入约 1.2 GB。
|
||||
该门禁覆盖单 Session 门禁看不到的语料规模列表、搜索和 fork 成本。在标准托管 CI 上，它已经显示：内容搜索在首次使用时花 139 s 构建索引，校准时 5,000 个 Session 的首次列表耗时 5.1 s，fork 最长 Session 耗时 17.5 到 20.3 s。播种写入约 1.5 GB。
|
||||
|
||||
排除项：语料只包含当前代，而大多数真实 Session 以旧代存储，这会增加列表的修订哈希和读取时迁移。只有 22 份不同主体，因此搜索索引的是重复文本。Fork 不挂载 Agent preset，也不附加 Workspace。调用在进程内完成，不经过 Typert 传输或浏览器。不驱逐文件系统缓存，因此“冷”指全新进程，而不是冷存储。
|
||||
@@ -23,26 +23,26 @@
|
||||
"@deepseek-ai/dsh-llm": "workspace:*",
|
||||
"@deepseek-ai/dsh-sdk-client": "workspace:*",
|
||||
"@deepseek-ai/dsh-session": "workspace:*",
|
||||
"@deepseek-ai/dsh-session-format": "workspace:*",
|
||||
"@deepseek-ai/dsh-session-format-catalog": "workspace:*",
|
||||
"@deepseek-ai/dsh-session-format-v3-to-v4": "workspace:*",
|
||||
"@deepseek-ai/dsh-session-persistence": "workspace:*",
|
||||
"@deepseek-ai/dsh-session-persistence-jsonl": "workspace:*",
|
||||
"@deepseek-ai/dsh-session-projection": "workspace:*",
|
||||
"@deepseek-ai/dsh-session-projection-cache": "workspace:*",
|
||||
"@deepseek-ai/dsh-session-query": "workspace:*",
|
||||
"@deepseek-ai/dsh-session-query-sqlite": "workspace:*",
|
||||
"@deepseek-ai/dsh-session-stats": "workspace:*",
|
||||
"@deepseek-ai/dsh-session-title": "workspace:*",
|
||||
"@deepseek-ai/dsh-session-turn-outline": "workspace:*",
|
||||
"@deepseek-ai/dsh-storage": "workspace:*",
|
||||
"@deepseek-ai/dsh-storage-domain": "workspace:*",
|
||||
"@deepseek-ai/dsh-storage-json": "workspace:*",
|
||||
"@deepseek-ai/dsh-subagent": "workspace:*",
|
||||
"@deepseek-ai/dsh-tools": "workspace:*",
|
||||
"@deepseek-ai/dsh-token-meter": "workspace:*",
|
||||
"@deepseek-ai/dsh-typert-protocol": "workspace:*",
|
||||
"@deepseek-ai/dsh-util-values": "workspace:*",
|
||||
"@deepseek-ai/dsh-session-projection-cache": "workspace:*",
|
||||
"@deepseek-ai/dsh-session-query-sqlite": "workspace:*",
|
||||
"@deepseek-ai/dsh-storage": "workspace:*",
|
||||
"@deepseek-ai/dsh-storage-domain": "workspace:*",
|
||||
"@deepseek-ai/dsh-storage-json": "workspace:*",
|
||||
"@deepseek-ai/dsh-session-format": "workspace:*",
|
||||
"@deepseek-ai/dsh-session-format-catalog": "workspace:*",
|
||||
"@deepseek-ai/dsh-session-format-v3-to-v4": "workspace:*"
|
||||
"@deepseek-ai/dsh-util-values": "workspace:*"
|
||||
},
|
||||
"peerDependencies": {
|
||||
"@deepseek-ai/cordis": "workspace:~"
|
||||
|
||||
@@ -6,14 +6,14 @@
|
||||
en: 28ebbfcfa3ea913a
|
||||
zh: 3ebad56e2c97c7c8
|
||||
/session-corpus-benchmarks/summary:
|
||||
en: 4bc34acd1959a4b5
|
||||
zh: 6f7932de565a5a8f
|
||||
en: 651a02571a0f6f42
|
||||
zh: dcd33bbc5baa0977
|
||||
/session-corpus-benchmarks/table-of-contents:
|
||||
en: 66a769ce60d49ff3
|
||||
zh: 7e0bfa7c598157fa
|
||||
/session-corpus-benchmarks/run:
|
||||
en: 491f186c7f0bd9f0
|
||||
zh: eff8ca4ff9537e0d
|
||||
en: 2560e832041ae481
|
||||
zh: 2fbbff71135e52ab
|
||||
/session-corpus-benchmarks/measurements:
|
||||
en: c357c231c3bfb588
|
||||
zh: 3d901a2caccf2587
|
||||
|
||||
@@ -4,7 +4,7 @@ English | [中文](README.zh.md)
|
||||
|
||||
## Summary
|
||||
|
||||
Measure the Web Host's Session list, content search, and fork over synthetic corpora in which every Session is at least as long as the same quantile of a measured local DSH corpus. Search and fork run over 1,000 Sessions; the list runs over 3,000, the largest count that keeps this file within five minutes on standard hosted CI. No case uses network services, recorded Sessions, or a browser.
|
||||
Measure the Web Host's Session list, content search, and fork over synthetic corpora in which every Session is at least as long as the same quantile of a measured local DSH corpus. Search and fork run over separate 1,000-Session corpora; the list runs over 3,000, the largest count that keeps this file within five minutes on standard hosted CI. Content search models a deployment that opts in with `openAt: first-search`; shipped profiles disable it. No case uses network services, recorded Sessions, or a browser.
|
||||
|
||||
## Table of Contents
|
||||
|
||||
@@ -16,7 +16,7 @@ Measure the Web Host's Session list, content search, and fork over synthetic cor
|
||||
|
||||
## Run
|
||||
|
||||
From the repository root, build the libraries and workers with `pnpm run build:bench`, then run `pnpm exec vitest run --config vitest.bench.config.ts benchmarks/session-corpus/session-corpus.bench.ts`. Do not overlap timing runs with builds or other benchmarks. Seeding writes about 1.2 GB of Zstandard logs under a private temporary root, which the test removes after success or failure.
|
||||
From the repository root, build the libraries and workers with `pnpm run build:bench`, then run `pnpm exec vitest run --config vitest.bench.config.ts benchmarks/session-corpus/session-corpus.bench.ts`. Do not overlap timing runs with builds or other benchmarks. Seeding writes about 1.5 GB of Zstandard logs under a private temporary root, which the test removes after success or failure.
|
||||
|
||||
The test reports every fresh-process sample, the CPU model, available parallelism, platform, and Node version, and enforces median budgets. Its last case fails when the file, including seeding, exceeds five minutes. A failed worker reports its exit, signal, timeout, and stderr tail. The required benchmark lane discovers this file automatically.
|
||||
|
||||
|
||||
@@ -4,7 +4,7 @@
|
||||
|
||||
## 概述
|
||||
|
||||
在合成语料上测量 Web Host 的 Session 列表、内容搜索和 fork；语料中每个 Session 的长度都不短于一份实测本地 DSH 语料在同一分位上的长度。搜索和 fork 用例运行 1,000 个 Session；列表用例运行 3,000 个 Session，这是在标准托管 CI 上让此文件五分钟内完成的最大数量。所有用例均不使用网络服务、录制的 Session 或浏览器。
|
||||
在合成语料上测量 Web Host 的 Session 列表、内容搜索和 fork；语料中每个 Session 的长度都不短于一份实测本地 DSH 语料在同一分位上的长度。搜索和 fork 用例分别运行在独立的 1,000 个 Session 语料上；列表用例运行 3,000 个 Session，这是在标准托管 CI 上让此文件五分钟内完成的最大数量。内容搜索模拟通过 `openAt: first-search` 启用该功能的部署；已发布的 profile 默认禁用它。所有用例均不使用网络服务、录制的 Session 或浏览器。
|
||||
|
||||
## 目录
|
||||
|
||||
@@ -16,7 +16,7 @@
|
||||
|
||||
## 运行
|
||||
|
||||
在仓库根目录使用 `pnpm run build:bench` 构建库和 worker，然后运行 `pnpm exec vitest run --config vitest.bench.config.ts benchmarks/session-corpus/session-corpus.bench.ts`。不要让计时运行与构建或其他基准重叠。播种过程会在私有临时根目录下写入约 1.2 GB 的 Zstandard 日志，测试成功或失败后都会删除该目录。
|
||||
在仓库根目录使用 `pnpm run build:bench` 构建库和 worker，然后运行 `pnpm exec vitest run --config vitest.bench.config.ts benchmarks/session-corpus/session-corpus.bench.ts`。不要让计时运行与构建或其他基准重叠。播种过程会在私有临时根目录下写入约 1.5 GB 的 Zstandard 日志，测试成功或失败后都会删除该目录。
|
||||
|
||||
测试报告每个新进程样本、CPU 型号、可用并行度、平台和 Node 版本，并约束中位数预算。当整个文件（包括播种）超过五分钟时，最后一个用例失败。worker 失败时报告退出状态、信号、超时和 stderr 末尾。必需基准通道自动发现此文件。
|
||||
|
||||
|
||||
@@ -8,6 +8,7 @@ import { afterAll, beforeAll, describe, expect, it } from 'vitest'
|
||||
import { runBuiltBenchmarkWorker } from '../support/built-worker.ts'
|
||||
import { PERFORMANCE_BUDGET_HEADROOM } from '../support/calibration.ts'
|
||||
import { anchorShape, ANCHOR_COUNT, sessionShape } from './corpus-shape.ts'
|
||||
import { subagentRank } from './synthetic-corpus.ts'
|
||||
import type {
|
||||
AnchorsReport,
|
||||
ForkReport,
|
||||
@@ -17,28 +18,39 @@ import type {
|
||||
SessionCorpusReport,
|
||||
} from './session-corpus.worker.ts'
|
||||
|
||||
/** Session counts: search and fork share the query corpus; the list corpus is the largest that fits the time limit. */
|
||||
const CORPUS = { query: 1_000, list: 3_000 } as const
|
||||
/**
|
||||
* Sessions per corpus. Search and fork use separate roots, so fork children never reach the search index;
|
||||
* the list corpus is the largest that keeps this file within its time limit.
|
||||
*/
|
||||
const CORPUS = { search: 1_000, fork: 1_000, list: 3_000 } as const
|
||||
/** Upper bound on this file's hosted CI wall time, from seeding through the last case. */
|
||||
const FILE_LIMIT_MS = 300_000
|
||||
/** Per-child deadline: an endpoint slower than its budget still reports a measurement instead of a kill. */
|
||||
const WORKER_TIMEOUT_MS = FILE_LIMIT_MS
|
||||
/** Fresh processes per list and fork-strata scenario; the median enforces each budget. */
|
||||
const ATTEMPTS = { list: 3, fork: 3 } as const
|
||||
/** Midpoints of ten equal length strata of the query corpus, then its p99 Session. */
|
||||
const FORK_STRATA = [50, 150, 250, 350, 450, 550, 650, 750, 850, 950] as const
|
||||
const FORK_P99_RANK = 990
|
||||
/** The longest Session; one sample, because one fork takes about 20 s on standard hosted CI. */
|
||||
const FORK_LONGEST_RANK = 999
|
||||
const WORKER_TIMEOUT_MS = 240_000
|
||||
/** Concurrent anchor-preparation processes; standard hosted runners have two CPUs. */
|
||||
const PREPARATION_PROCESSES = 2
|
||||
/** Midpoints of ten equal length strata of the fork corpus. */
|
||||
const FORK_STRATA = Array.from({ length: 10 }, (_, stratum) => Math.floor(((stratum + 0.5) * CORPUS.fork) / 10))
|
||||
/** The highest top-level rank at the measured p99; forking a subagent child adds a lineage lookup. */
|
||||
const FORK_P99_RANK = topLevelRank(Math.ceil(0.99 * CORPUS.fork) - 1)
|
||||
/** The longest Session; one sample, because one fork takes about 20 s on standard hosted CI. */
|
||||
const FORK_LONGEST_RANK = CORPUS.fork - 1
|
||||
const WORKER = join(import.meta.dirname, '..', '.dsh-build', 'session-corpus', 'session-corpus.worker.js')
|
||||
|
||||
/**
|
||||
* Standard two-CPU hosted CI expectations, rounded above the higher recorded median of the EPYC 7763 and
|
||||
* EPYC 9V74 runners before variance headroom. Recorded 7763 / 9V74 medians: list 2,317.8 / 2,204.5,
|
||||
* 3,228.7 / 2,633.6, and 2,636.4 / 2,235.1 ms; search 144,359 / 135,065 and 3,417 / 2,704 ms;
|
||||
* fork 94.0 / 73.5, 1,522.7 / 1,485.5, and 18,871.9 / 20,326.7 ms.
|
||||
*/
|
||||
/** Higher recorded median of the standard EPYC 7763 and EPYC 9V74 hosted runners, per budgeted endpoint. */
|
||||
const RECORDED_CI_MS = {
|
||||
listBoot: 2_317.8,
|
||||
listFirst: 3_228.7,
|
||||
listRepeat: 2_636.4,
|
||||
searchFirst: 144_359,
|
||||
searchRepeat: 3_417,
|
||||
forkStratumMedian: 94,
|
||||
forkP99: 1_522.7,
|
||||
forkLongest: 20_326.7,
|
||||
} as const
|
||||
/** Hosted CI expectations, rounded above {@link RECORDED_CI_MS} before variance headroom. */
|
||||
const EXPECTED_CI_MS = {
|
||||
listBoot: 2_400,
|
||||
listFirst: 3_300,
|
||||
@@ -48,16 +60,32 @@ const EXPECTED_CI_MS = {
|
||||
forkStratumMedian: 100,
|
||||
forkP99: 1_600,
|
||||
forkLongest: 20_400,
|
||||
} as const
|
||||
|
||||
function budget(expectedCiMs: number): number {
|
||||
return Math.ceil(expectedCiMs * PERFORMANCE_BUDGET_HEADROOM)
|
||||
}
|
||||
} as const satisfies Record<keyof typeof RECORDED_CI_MS, number>
|
||||
/** Higher recorded peak RSS of the two runner models; memory takes headroom but no time scale. */
|
||||
const RECORDED_CI_PEAK_RSS_MB = { list: 742.2, search: 1_197.8, fork: 737.8 } as const
|
||||
const EXPECTED_CI_PEAK_RSS_MB = {
|
||||
list: 750,
|
||||
search: 1_200,
|
||||
fork: 750,
|
||||
} as const satisfies Record<keyof typeof RECORDED_CI_PEAK_RSS_MB, number>
|
||||
|
||||
type CorpusName = keyof typeof CORPUS
|
||||
|
||||
async function run<Report extends SessionCorpusReport>(args: readonly string[], timeoutMs = WORKER_TIMEOUT_MS): Promise<Report> {
|
||||
const outcome = await runBuiltBenchmarkWorker<Report>({ worker: WORKER, args, timeoutMs, exposeGc: true })
|
||||
function topLevelRank(rank: number): number {
|
||||
return subagentRank(rank) ? rank - 1 : rank
|
||||
}
|
||||
|
||||
function budget(expected: number): number {
|
||||
return Math.ceil(expected * PERFORMANCE_BUDGET_HEADROOM)
|
||||
}
|
||||
|
||||
/** Test deadline for sequential workers; it outlasts their deadlines, so every child is reaped before cleanup. */
|
||||
function serialDeadline(workers: number): number {
|
||||
return workers * WORKER_TIMEOUT_MS + 30_000
|
||||
}
|
||||
|
||||
async function run<Report extends SessionCorpusReport>(args: readonly string[]): Promise<Report> {
|
||||
const outcome = await runBuiltBenchmarkWorker<Report>({ worker: WORKER, args, timeoutMs: WORKER_TIMEOUT_MS, exposeGc: true })
|
||||
if (outcome.report === undefined) {
|
||||
const stderr = outcome.stderr.trim().split('\n').slice(-20).join('\n')
|
||||
throw new Error(`session-corpus worker ${args[1] ?? ''} failed: exit=${String(outcome.exitCode)}, `
|
||||
@@ -104,7 +132,7 @@ function environment() {
|
||||
}
|
||||
|
||||
describe('Session corpus workload', () => {
|
||||
it('is at least the measured distribution at every rank of both corpora', () => {
|
||||
it('is at least the measured distribution at every rank of every corpus', () => {
|
||||
for (const count of Object.values(CORPUS)) {
|
||||
const shapes = Array.from({ length: count }, (_, rank) => sessionShape(rank, count))
|
||||
expect(new Set(shapes.map(shape => shape.anchor)).size).toBe(ANCHOR_COUNT)
|
||||
@@ -116,50 +144,61 @@ describe('Session corpus workload', () => {
|
||||
expect(current?.logicalBytes).toBeGreaterThanOrEqual(previous?.logicalBytes ?? 0)
|
||||
}
|
||||
}
|
||||
expect(() => sessionShape(CORPUS.query, CORPUS.query)).toThrow('outside')
|
||||
expect(() => sessionShape(CORPUS.fork, CORPUS.fork)).toThrow('outside')
|
||||
})
|
||||
|
||||
it('accepts recorded hosted medians and rejects medians a quarter above their expectation', () => {
|
||||
const recorded = [
|
||||
[median([2_317.8, 2_322.2, 2_264.6]), EXPECTED_CI_MS.listBoot],
|
||||
[median([3_294.7, 3_228.7, 3_144.8]), EXPECTED_CI_MS.listFirst],
|
||||
[144_359, EXPECTED_CI_MS.searchFirst],
|
||||
[20_326.7, EXPECTED_CI_MS.forkLongest],
|
||||
[median([92.5, 94, 106.7]), EXPECTED_CI_MS.forkStratumMedian],
|
||||
] as const
|
||||
for (const [value, expected] of recorded) {
|
||||
expectWithinBudget(value, budget(expected))
|
||||
it('forks top-level Sessions at the strata midpoints, the measured p99, and the maximum', () => {
|
||||
const ranks = [...FORK_STRATA, FORK_P99_RANK, FORK_LONGEST_RANK]
|
||||
expect(ranks.filter(subagentRank)).toEqual([])
|
||||
expect(FORK_STRATA).toEqual([50, 150, 250, 350, 450, 550, 650, 750, 850, 950])
|
||||
expect(sessionShape(FORK_P99_RANK, CORPUS.fork).events).toBe(7_229)
|
||||
expect(sessionShape(FORK_LONGEST_RANK, CORPUS.fork).events).toBe(84_467)
|
||||
})
|
||||
|
||||
it('accepts every recorded hosted median and rejects one a quarter above its expectation', () => {
|
||||
const endpoints = [
|
||||
...(Object.keys(EXPECTED_CI_MS) as (keyof typeof EXPECTED_CI_MS)[])
|
||||
.map(key => [RECORDED_CI_MS[key], EXPECTED_CI_MS[key]] as const),
|
||||
...(Object.keys(EXPECTED_CI_PEAK_RSS_MB) as (keyof typeof EXPECTED_CI_PEAK_RSS_MB)[])
|
||||
.map(key => [RECORDED_CI_PEAK_RSS_MB[key], EXPECTED_CI_PEAK_RSS_MB[key]] as const),
|
||||
]
|
||||
expect(endpoints).toHaveLength(11)
|
||||
for (const [recorded, expected] of endpoints) {
|
||||
expect(recorded).toBeLessThanOrEqual(expected)
|
||||
expectWithinBudget(recorded, budget(expected))
|
||||
expect(() => expectWithinBudget(Math.ceil(expected * 1.26), budget(expected))).toThrow()
|
||||
}
|
||||
expect(budget(EXPECTED_CI_MS.forkLongest)).toBe(25_500)
|
||||
})
|
||||
})
|
||||
|
||||
describe('Session corpus operations', () => {
|
||||
let scratch = ''
|
||||
let started = 0
|
||||
let seeded: SeedReport | undefined
|
||||
const root = (name: CorpusName): string => join(scratch, `corpus-${String(CORPUS[name])}`)
|
||||
let seeded: readonly SeedReport[] = []
|
||||
const root = (name: CorpusName): string => join(scratch, `corpus-${name}`)
|
||||
|
||||
beforeAll(async () => {
|
||||
started = performance.now()
|
||||
scratch = await mkdtemp(join(tmpdir(), 'dsh-session-corpus-bench-'))
|
||||
const prepared = await Promise.all(anchorGroups().map(group => run<AnchorsReport>([scratch, 'anchors', ...group.map(String)])))
|
||||
seeded = await run<SeedReport>([scratch, 'seed', String(CORPUS.query), String(CORPUS.list)])
|
||||
console.log(JSON.stringify({ benchmark: 'session-corpus/seed', prepared, ...seeded, environment: environment() }))
|
||||
}, WORKER_TIMEOUT_MS)
|
||||
seeded = await Promise.all((Object.keys(CORPUS) as CorpusName[])
|
||||
.map(name => run<SeedReport>([scratch, 'seed', name, String(CORPUS[name])])))
|
||||
console.log(JSON.stringify({ benchmark: 'session-corpus/seed', prepared, seeded, environment: environment() }))
|
||||
}, serialDeadline(2))
|
||||
|
||||
afterAll(async () => {
|
||||
if (scratch !== '') await rm(scratch, { recursive: true, force: true })
|
||||
})
|
||||
|
||||
it('writes corpora at least as long as the measured distribution', () => {
|
||||
for (const count of [CORPUS.query, CORPUS.list]) {
|
||||
const facts = seeded?.corpora.find(corpus => corpus.sessions === count)
|
||||
expect(seeded.map(report => report.name).sort()).toEqual(Object.keys(CORPUS).sort())
|
||||
for (const { name, corpus } of seeded) {
|
||||
const count = CORPUS[name as CorpusName]
|
||||
const shapes = Array.from({ length: count }, (_, rank) => sessionShape(rank, count))
|
||||
expect(facts?.distinctBodies).toBe(ANCHOR_COUNT)
|
||||
expect(facts?.events).toBeGreaterThanOrEqual(shapes.reduce((sum, shape) => sum + shape.events, 0))
|
||||
expect(facts?.logicalBytes).toBeGreaterThanOrEqual(shapes.reduce((sum, shape) => sum + shape.logicalBytes, 0))
|
||||
expect(corpus.sessions).toBe(count)
|
||||
expect(corpus.distinctBodies).toBe(ANCHOR_COUNT)
|
||||
expect(corpus.events).toBeGreaterThanOrEqual(shapes.reduce((sum, shape) => sum + shape.events, 0))
|
||||
expect(corpus.logicalBytes).toBeGreaterThanOrEqual(shapes.reduce((sum, shape) => sum + shape.logicalBytes, 0))
|
||||
}
|
||||
})
|
||||
|
||||
@@ -168,6 +207,7 @@ describe('Session corpus operations', () => {
|
||||
bootMs: budget(EXPECTED_CI_MS.listBoot),
|
||||
firstMs: budget(EXPECTED_CI_MS.listFirst),
|
||||
repeatMs: budget(EXPECTED_CI_MS.listRepeat),
|
||||
peakRssMb: budget(EXPECTED_CI_PEAK_RSS_MB.list),
|
||||
}
|
||||
const reports: ListReport[] = []
|
||||
for (let attempt = 0; attempt < ATTEMPTS.list; attempt++) {
|
||||
@@ -185,28 +225,35 @@ describe('Session corpus operations', () => {
|
||||
expectWithinBudget(median(samples.bootMs), budgets.bootMs)
|
||||
expectWithinBudget(median(samples.firstMs), budgets.firstMs)
|
||||
expectWithinBudget(median(samples.repeatMs), budgets.repeatMs)
|
||||
})
|
||||
expectWithinBudget(median(samples.peakRssMb), budgets.peakRssMb)
|
||||
}, serialDeadline(ATTEMPTS.list))
|
||||
|
||||
it(`searches ${String(CORPUS.query)} Sessions with a cold and then a built content index`, async () => {
|
||||
const budgets = { firstMs: budget(EXPECTED_CI_MS.searchFirst), repeatMs: budget(EXPECTED_CI_MS.searchRepeat) }
|
||||
it(`searches ${String(CORPUS.search)} Sessions with a cold and then a built content index`, async () => {
|
||||
const budgets = {
|
||||
firstMs: budget(EXPECTED_CI_MS.searchFirst),
|
||||
repeatMs: budget(EXPECTED_CI_MS.searchRepeat),
|
||||
peakRssMb: budget(EXPECTED_CI_PEAK_RSS_MB.search),
|
||||
}
|
||||
// One sample: the cold index build dominates this file's time.
|
||||
const report = await run<SearchReport>([root('query'), 'search'])
|
||||
const report = await run<SearchReport>([root('search'), 'search'])
|
||||
console.log(JSON.stringify({ benchmark: 'session-corpus/search', report, budgets, environment: environment() }))
|
||||
expectWithinBudget(report.firstMs, budgets.firstMs)
|
||||
expectWithinBudget(report.repeatMs, budgets.repeatMs)
|
||||
}, WORKER_TIMEOUT_MS)
|
||||
expectWithinBudget(report.memory.peakRssMb, budgets.peakRssMb)
|
||||
}, serialDeadline(1))
|
||||
|
||||
it(`forks Sessions across the length distribution of ${String(CORPUS.query)} Sessions`, async () => {
|
||||
it(`forks Sessions across the length distribution of ${String(CORPUS.fork)} Sessions`, async () => {
|
||||
const budgets = {
|
||||
stratumMedianMs: budget(EXPECTED_CI_MS.forkStratumMedian),
|
||||
p99Ms: budget(EXPECTED_CI_MS.forkP99),
|
||||
longestMs: budget(EXPECTED_CI_MS.forkLongest),
|
||||
peakRssMb: budget(EXPECTED_CI_PEAK_RSS_MB.fork),
|
||||
}
|
||||
const reports: ForkReport[] = []
|
||||
for (let attempt = 0; attempt < ATTEMPTS.fork; attempt++) {
|
||||
reports.push(await run<ForkReport>([root('query'), 'fork', ...[...FORK_STRATA, FORK_P99_RANK].map(String)]))
|
||||
reports.push(await run<ForkReport>([root('fork'), 'fork', ...[...FORK_STRATA, FORK_P99_RANK].map(String)]))
|
||||
}
|
||||
const longest = await run<ForkReport>([root('query'), 'fork', String(FORK_LONGEST_RANK)])
|
||||
const longest = await run<ForkReport>([root('fork'), 'fork', String(FORK_LONGEST_RANK)])
|
||||
const forkMs = (report: ForkReport, rank: number): number => {
|
||||
const fork = report.forks.find(entry => entry.rank === rank)
|
||||
if (fork === undefined) throw new Error(`fork report omits rank ${String(rank)}`)
|
||||
@@ -223,7 +270,8 @@ describe('Session corpus operations', () => {
|
||||
expectWithinBudget(median(samples.stratumMedianMs), budgets.stratumMedianMs)
|
||||
expectWithinBudget(median(samples.p99Ms), budgets.p99Ms)
|
||||
expectWithinBudget(samples.longestMs, budgets.longestMs)
|
||||
}, WORKER_TIMEOUT_MS)
|
||||
expectWithinBudget(Math.max(...samples.peakRssMb), budgets.peakRssMb)
|
||||
}, serialDeadline(ATTEMPTS.fork + 1))
|
||||
|
||||
it(`completes within ${String(FILE_LIMIT_MS / 60_000)} minutes`, () => {
|
||||
const elapsedMs = performance.now() - started
|
||||
|
||||
@@ -52,10 +52,11 @@ export interface AnchorsReport {
|
||||
readonly prepareMs: number
|
||||
}
|
||||
|
||||
/** Seeding report: facts of every corpus written from prepared anchors, and the time spent writing them. */
|
||||
/** Seeding report: facts of one named corpus written from prepared anchors, and the time spent writing it. */
|
||||
export interface SeedReport {
|
||||
readonly mode: 'seed'
|
||||
readonly corpora: readonly (SyntheticCorpusFacts & { readonly writeMs: number })[]
|
||||
readonly name: string
|
||||
readonly corpus: SyntheticCorpusFacts
|
||||
readonly seedMs: number
|
||||
}
|
||||
|
||||
@@ -221,8 +222,8 @@ async function readPreparedAnchor(root: string, anchor: number): Promise<AnchorB
|
||||
}
|
||||
}
|
||||
|
||||
/** Write every corpus from prepared anchors, rebinding each projection record to its Session header. */
|
||||
async function seed(root: string, counts: readonly number[]): Promise<SeedReport> {
|
||||
/** Write one named corpus from prepared anchors, rebinding each projection record to its Session header. */
|
||||
async function seed(root: string, name: string, count: number): Promise<SeedReport> {
|
||||
const started = performance.now()
|
||||
const writer = new SyntheticCorpusWriter()
|
||||
const records = new Map<number, CheckpointRecord>()
|
||||
@@ -231,27 +232,23 @@ async function seed(root: string, counts: readonly number[]): Promise<SeedReport
|
||||
await writer.loadBody(join(root, 'bodies'), anchor, facts)
|
||||
records.set(anchor, record)
|
||||
}
|
||||
const corpora: SeedReport['corpora'][number][] = []
|
||||
for (const count of counts) {
|
||||
const corpusRoot = join(root, `corpus-${String(count)}`)
|
||||
const writeStarted = performance.now()
|
||||
const written = await writer.writeCorpus(join(corpusRoot, 'sessions'), count)
|
||||
// Sessions of one anchor share events, so their records differ only in the header-bound identity.
|
||||
// Per-record documents are written directly: the domain's one-fsync-per-put chain would dominate seeding,
|
||||
// and the list endpoint rejects a corpus whose rows the cache does not serve.
|
||||
const table = join(corpusRoot, 'storages', projectionCacheDomainSpec.name, 'sessions')
|
||||
await mkdir(table, { recursive: true })
|
||||
await forEachConcurrently(written.sessions.length, async (rank) => {
|
||||
const { header, anchor } = written.sessions[rank] as WrittenSession
|
||||
const source = records.get(anchor) as CheckpointRecord
|
||||
await writeFile(join(table, `${header.id}.json`), serializeRecord(projectionCacheDomainSpec.version, {
|
||||
identity: { ...source.identity, createdAt: header.createdAt, cwd: header.cwd },
|
||||
rows: source.rows,
|
||||
}))
|
||||
})
|
||||
corpora.push({ ...written.facts, writeMs: performance.now() - writeStarted })
|
||||
}
|
||||
return { mode: 'seed', corpora, seedMs: performance.now() - started }
|
||||
const corpusRoot = join(root, `corpus-${name}`)
|
||||
const written = await writer.writeCorpus(join(corpusRoot, 'sessions'), count)
|
||||
// Sessions of one anchor share events, so their records differ only in the header-bound identity.
|
||||
// Per-record documents are written directly: the domain's one-fsync-per-put chain would dominate seeding,
|
||||
// and the list endpoint rejects a corpus whose rows the cache does not serve.
|
||||
const table = join(corpusRoot, 'storages', projectionCacheDomainSpec.name, 'sessions')
|
||||
await mkdir(table, { recursive: true })
|
||||
await forEachConcurrently(written.sessions.length, async (rank) => {
|
||||
const { header, anchor } = written.sessions[rank] as WrittenSession
|
||||
const source = records.get(anchor)
|
||||
if (source === undefined) throw new Error(`corpus anchor ${String(anchor)} has no folded projection record`)
|
||||
await writeFile(join(table, `${header.id}.json`), serializeRecord(projectionCacheDomainSpec.version, {
|
||||
identity: { ...source.identity, createdAt: header.createdAt, cwd: header.cwd },
|
||||
rows: source.rows,
|
||||
}))
|
||||
})
|
||||
return { mode: 'seed', name, corpus: written.facts, seedMs: performance.now() - started }
|
||||
}
|
||||
|
||||
async function measureList(root: string, expected: number): Promise<ListReport> {
|
||||
@@ -335,27 +332,35 @@ assertBuiltBenchmarkRuntime(import.meta.url, Object.fromEntries([
|
||||
'@deepseek-ai/dsh-session-projection-cache',
|
||||
].map(name => [name, import.meta.resolve(name)])))
|
||||
|
||||
const USAGE = 'usage: session-corpus.worker.js <root> <anchors anchors...|seed name count|list count|search|fork ranks...>'
|
||||
const [root, mode, ...rest] = process.argv.slice(2)
|
||||
const numbers = rest.map(Number)
|
||||
if (root === undefined || numbers.some(value => !Number.isSafeInteger(value) || value < 0)) {
|
||||
throw new Error('usage: session-corpus.worker.js <root> <anchors anchors...|seed counts...|list count|search|fork ranks...>')
|
||||
if (root === undefined) throw new Error(USAGE)
|
||||
|
||||
function counts(values: readonly string[]): number[] {
|
||||
const parsed = values.map(Number)
|
||||
if (parsed.some(value => !Number.isSafeInteger(value) || value < 0)) throw new Error(USAGE)
|
||||
return parsed
|
||||
}
|
||||
|
||||
let report: SessionCorpusReport
|
||||
switch (mode) {
|
||||
case 'anchors':
|
||||
report = await prepareAnchors(root, numbers)
|
||||
report = await prepareAnchors(root, counts(rest))
|
||||
break
|
||||
case 'seed':
|
||||
report = await seed(root, numbers)
|
||||
case 'seed': {
|
||||
const [name, count] = rest
|
||||
if (name === undefined || !/^[a-z]+$/.test(name) || count === undefined) throw new Error(USAGE)
|
||||
report = await seed(root, name, counts([count])[0] as number)
|
||||
break
|
||||
}
|
||||
case 'list':
|
||||
report = await measureList(root, numbers[0] ?? 0)
|
||||
report = await measureList(root, counts(rest)[0] ?? 0)
|
||||
break
|
||||
case 'search':
|
||||
report = await measureSearch(root)
|
||||
break
|
||||
case 'fork':
|
||||
report = await measureFork(root, numbers)
|
||||
report = await measureFork(root, counts(rest))
|
||||
break
|
||||
default:
|
||||
throw new Error(`unknown Session corpus mode ${String(mode)}`)
|
||||
|
||||
@@ -305,6 +305,15 @@ export interface WrittenCorpus {
|
||||
readonly sessions: readonly WrittenSession[]
|
||||
}
|
||||
|
||||
/**
|
||||
* Tell whether one corpus rank stores a subagent child.
|
||||
* @param rank - zero-based ascending length rank.
|
||||
* @returns whether the rank is a child of the preceding rank.
|
||||
*/
|
||||
export function subagentRank(rank: number): boolean {
|
||||
return rank % CHILD_EVERY === CHILD_EVERY - 1
|
||||
}
|
||||
|
||||
/**
|
||||
* Build the stored header of one corpus rank.
|
||||
* @param rank - zero-based ascending length rank.
|
||||
@@ -313,8 +322,9 @@ export interface WrittenCorpus {
|
||||
*/
|
||||
export function corpusHeader(rank: number, count: number): SessionHeader {
|
||||
if (count % CREATION_STRIDE === 0) throw new Error('corpus size must be coprime with the creation stride')
|
||||
const child = rank % CHILD_EVERY === CHILD_EVERY - 1
|
||||
// Creation order is a fixed permutation of length rank, so list order does not track length.
|
||||
const child = subagentRank(rank)
|
||||
// Creation time is a fixed permutation of length rank. List order follows each body's last prompt time,
|
||||
// which Sessions sharing an anchor also share.
|
||||
const createdAt = TIME_ZERO - (((rank * CREATION_STRIDE) % count) + 1) * 60_000
|
||||
return {
|
||||
version: SESSION_FORMAT_VERSION,
|
||||
|
||||
Reference in new issue
Block a user