Merge pull request #6055 from deepseek-harness/fix/user-text-benchmark-sampling

test(benchmarks): stabilize user-text timing samples

.agents/notes/implemented/testing/2026-10-11-user-text-credential-matching.i18n.yaml

@@ -21,11 +21,11 @@
   en: f07bb6fbbdd01fe4
   zh: 9313a0ef2a22af12
 /agent-note-linear-credential-matching-and-performance-gates-for-user-text/decision/performance-gate:
-  en: 6a303115f85ed4e8
-  zh: e894f3e9d630bf57
+  en: 29b8f8c1ad73c77b
+  zh: 8293b21ec08d3896
 /agent-note-linear-credential-matching-and-performance-gates-for-user-text/alternatives-considered:
-  en: 1e21f5f9ca959f7e
-  zh: 401e303071b8d426
+  en: 7600c1671b5dc73c
+  zh: b38e3ae76a5a32d1
 /agent-note-linear-credential-matching-and-performance-gates-for-user-text/consequences:
   en: 8f9d25b72d3e7660
   zh: e5c6ae0194905876

.agents/notes/implemented/testing/2026-10-11-user-text-credential-matching.md

@@ -43,15 +43,18 @@ English | [中文](2026-10-11-user-text-credential-matching.zh.md)
 | Item | Content |
 |---|---|
 | Workloads | Without credentials: ordinary messages, and long prose, code, base64, Markdown, log, JSON, and shell text. With credentials: messages each carrying one kind (model key, GitHub token, cloud key ID, JWT, address password, private-key block), and batches of `.env` keys. Adversarial: repeated input shaped against each format's prefix, length, and separator rules, at two sizes. All generated from constants; credential samples are assembled at run time |
-| Timing | Each sample builds a fresh registry, document, and matcher instance and times evaluation only; each workload is measured with and without the credential matcher, keeping the fastest of several samples and the process CPU time over the same interval |
+| Timing | Ten unmeasured pairs warm each workload with the same evaluation count as measured pairs. Five measured pairs alternate baseline-first and matcher-first order; each evaluation builds a fresh registry, document, and matcher instance and times evaluation only. A single-document sample averages twenty evaluations; a 5,000-message sample evaluates the batch once. Each side selects its minimum sample mean, with the mean process CPU time from that sample. The worker JSON in CI logs includes every measured pair and averaging count; the HTML report shows the selected with-credential mean. Registry disposal completes outside the timer before the next evaluation, preventing cleanup and retained registries from accumulating. Warm-up retains no matcher memo |
 | Allocation | A separate process with `--expose-gc` and a fixed semi-space larger than any evaluation; after warm-up and GC, the matcher alone answers every query once and the young-generation growth is taken; GC event timestamps confirm no GC ran inside the interval |
 | Assertions | Without credentials: absolute budgets and a cap on the with/without time ratio; with credentials: absolute budgets and the time ratio when the key count doubles; adversarial: absolute budgets and doubling ratios; allocation: a floor plus caps per credential, per line break inside masks, and per masked character |
-| Excluded | React rendering, browser paint and layout; allocation results are asserted and printed to the console, not added to the HTML report |
+| Excluded | Cold code compilation, React rendering, browser paint and layout; allocation results are asserted and printed to the console, not added to the HTML report |
 
 Expectations come from an x64 development machine. CI time budgets are converted with the shared scale and headroom; time ratios, doubling ratios, and allocation budgets do not scale with machine speed. Negative controls: the same benchmark fails when pointed at the regular-expression implementation or at an implementation that rebuilds its memo per query. Equivalence tests compare this implementation with the regular-expression one ([regex-oracle.client.ts](../../../../packages/client/ui-secret-mask/tests/regex-oracle.client.ts)) node by node through the production scheduler and from every query offset.
 
+The reference constants are calibrated for minima of five single evaluations. Minimum-of-five sample means use the existing absolute headroom without recalibration. Averaging can move the ratio of independently selected minima in either direction; a threefold matcher-work negative control verifies that the JSON ratio limit still rejects extra scanning.
+
 ## Alternatives considered
 
+- **Unwarmed, separate baseline and matcher batches**: sub-millisecond baseline measurements can reach optimized code before the matcher’s newly exercised paths do, and separate batches amplify clock-speed drift. Per-workload warm-up, adjacent pairs, and fixed batches compare warmed code while limiting noise in short intervals and preserving fresh documents and matcher state.
 - **One quantified pattern per format, run from every query**: patterns without a hit scan to the end of the text each time, so cost grows quadratically with credentials, and length quantifiers backtrack on failure.
 - **One large quantified pattern for all formats**: one scan finds the leftmost match, but branch quantifiers still backtrack on failure.
 - **Scanning for rare characters first, then looking back for prefixes**: faster on ordinary text, but cost depends on character distribution, and text dense in `-` or `_` becomes clearly slower.

.agents/notes/implemented/testing/2026-10-11-user-text-credential-matching.zh.md

@@ -43,15 +43,18 @@ Status: implemented
 | 项 | 内容 |
 |---|---|
 | 负载 | 无凭据：普通消息，以及散文、代码、base64、Markdown、日志、JSON、shell 长文本。含凭据：每条消息含一种凭据（模型密钥、GitHub 令牌、云密钥 ID、JWT、地址密码、私钥块），以及成批的 `.env` 密钥。对抗：针对每种格式的前缀、长度与分隔规则构造的重复输入，两种长度。全部由常量生成，凭据样本运行时拼接 |
-| 计时 | 每个样本新建注册表、文档和匹配器实例，只计求值；每个负载分别测有无凭据匹配器，取多个样本中最快的一个，并记录同一区间的进程 CPU 时间 |
+| 计时 | 每个负载先用十组未计入结果的配对预热，每组的求值次数与测量配对相同。五组测量交替以基线或凭据匹配器开头；每次求值新建注册表、文档和匹配器实例，只计求值。单文档样本取二十次求值的平均值，5,000 条消息的样本则整批求值一次。两侧各取样本均值的最小值，并保留对应样本的平均进程 CPU 时间。CI 日志中的 worker JSON 包含全部测量配对和求平均值的次数，HTML 报告展示选中的带凭据匹配器均值。每次求值在计时结束后等待注册表清理完成，再开始下一次，避免清理工作和未释放注册表积累。预热不保留匹配器记忆 |
 | 分配 | 单独进程，`--expose-gc`，semi-space 固定为大于任何一次求值的大小；预热后 GC，再单独驱动匹配器回答全部查询一遍，取新生代增量；用 GC 事件的时间戳确认区间内没有 GC |
 | 断言 | 无凭据：绝对预算，以及"有/无凭据匹配器"的耗时比上限；含凭据：绝对预算，密钥数翻倍时的耗时比；对抗：绝对预算与翻倍耗时比；分配：底噪加每个凭据、每个遮挡内换行和每个遮挡字符的上限 |
-| 排除 | React 渲染、浏览器绘制与布局；分配结果只用于断言并输出到控制台，不进 HTML 报告 |
+| 排除 | 冷代码编译、React 渲染、浏览器绘制与布局；分配结果只用于断言并输出到控制台，不进 HTML 报告 |
 
 预期值来自 x64 开发机。CI 时间预算按共享比例与余量换算；耗时比、翻倍比和分配预算不随机器缩放。负对照：同一 benchmark 指向正则实现或"每次查询新建记忆"的实现时都会失败。等价性测试经生产调度器逐节点比较本实现与正则实现（[regex-oracle.client.ts](../../../../packages/client/ui-secret-mask/tests/regex-oracle.client.ts)），并对每个查询起点单独比较。
 
+参考常量按五次单独求值的最小值校准；五组样本均值的最小值使用既有绝对耗时余量，不重新校准。求平均值可能使两侧独立最小值的比值向任一方向变化；将匹配器工作量增至三倍的负对照验证 JSON 比值上限仍能拒绝额外扫描。
+
 ## 考虑过的替代方案
 
+- **不预热、分批测量基线和匹配器**：亚毫秒基线测量可能先进入优化后的代码，而匹配器新触及的路径还未完成优化；分批测量也会放大时钟频率漂移。逐负载预热、相邻配对和固定批次比较预热后的代码，降低短区间测量的噪声，同时保持文档和匹配器状态全新。
 - **每次查询对每个格式各跑一个带量词的正则**：没有命中的正则每次都扫到文末，凭据多时平方增长；长度量词在失败时回溯。
 - **把全部格式合成一个带量词的大正则**：一次扫描能找到最左匹配，但各分支的量词失败时仍回溯。
 - **先找稀有字符再回看前缀**：普通文本扫描更快，但成本取决于字符分布，`-`、`_` 密集的文本明显变慢。

benchmarks/user-text/user-text.bench.client.ts

@@ -9,7 +9,7 @@ import type {
   UserTextAllocationReport, UserTextTimeReport, UserTextWorkloadAllocation, UserTextWorkloadTiming,
 } from './user-text.worker.client.ts'
 
-/** Fresh registries and documents per sample in one compiled worker; the fastest sample removes scheduler delay. */
+/** Fresh registries and documents after warm-up; independent minima of adjacent pairs limit scheduler delay. */
 const ATTEMPTS = 5
 /** A stuck worker is reaped before the outer benchmark deadline. */
 const WORKER_TIMEOUT_MS = 180_000
@@ -103,7 +103,10 @@ describe('sent user text with the credential matcher', () => {
     return found
   }
   const record = (benchmark: BenchmarkCase, measured: UserTextWorkloadTiming, budgetMs: number): void => {
-    recordTimings(benchmark, { withSecretMs: { ms: measured.withSecretMs, cpuMs: measured.withSecretCpuMs } }, { withSecretMs: budgetMs })
+    recordTimings({
+      ...benchmark,
+      measures: `${benchmark.measures} Minimum of ${ATTEMPTS} per-sample means of ${measured.evaluationsPerSample} fresh evaluations after warm-up.`,
+    }, { withSecretMs: { ms: measured.withSecretMs, cpuMs: measured.withSecretCpuMs } }, { withSecretMs: budgetMs })
   }
 
   it.each(Object.entries(UNMATCHED_EXPECTED))('evaluates %s without credentials within its budget and ratio', (id, expected) => {

benchmarks/user-text/user-text.worker.client.ts

@@ -17,20 +17,34 @@ import { assertBuiltBenchmarkRuntime } from '../support/built-worker.ts'
 /** What a workload contains: no credential, real credentials, or input shaped against one format. */
 export type UserTextWorkloadGroup = 'unmatched' | 'matched' | 'adversarial'
 
+/** Adjacent baseline and credential measurements; even-indexed pairs measure the baseline first. */
+export interface UserTextTimingPair {
+  readonly baseMs: number
+  readonly baseCpuMs: number
+  readonly withSecretMs: number
+  readonly withSecretCpuMs: number
+}
+
 /** Timing of one workload evaluated by the shipped rule set with and without the credential matcher. */
 export interface UserTextWorkloadTiming {
   readonly id: string
   readonly group: UserTextWorkloadGroup
   readonly documents: number
   readonly characters: number
+  /** Fresh evaluations averaged into each timed sample; counts and characters describe one evaluation. */
+  readonly evaluationsPerSample: number
   readonly secrets: number
   /** Characters inside credential ranges, whose masks the matcher builds. */
   readonly maskedCharacters: number
   readonly baseMs: number
+  /** Mean process CPU time per evaluation in the fastest baseline sample. */
+  readonly baseCpuMs: number
   readonly withSecretMs: number
-  /** Process CPU time over the fastest with-credential sample. */
+  /** Mean process CPU time per evaluation in the fastest with-credential sample. */
   readonly withSecretCpuMs: number
   readonly ratio: number
+  /** All measured pairs in execution order, excluding warm-up. */
+  readonly samples: readonly UserTextTimingPair[]
 }
 
 /** Heap allocation of the credential matcher over one workload, with the size of what it returns. */
@@ -45,8 +59,9 @@ export interface UserTextWorkloadAllocation {
   readonly maskedLineBreaks: number
 }
 
-/** Result of `time <attempts>`: the fastest of `attempts` fresh samples per workload. */
+/** Result of `time <attempts>`: warmed code, fresh instances, and independent minima from adjacent sample pairs. */
 export interface UserTextTimeReport {
+  readonly warmupPairs: number
   readonly attempts: number
   readonly workloads: readonly UserTextWorkloadTiming[]
 }
@@ -65,6 +80,10 @@ const BASE_RULES: readonly UserTextRule[] = [
 const FACTS: Readonly<Partial<UserTextFacts>> = { sessionLabels: ['会话一'], skillNames: ['plan', 'review'] }
 
 const KIB = 1024
+/** Exercise each workload's code paths before measuring sub-millisecond baseline evaluations. */
+const WARMUP_PAIRS = 10
+/** Batch every single-document workload to limit noise in its shortest evaluation intervals. */
+const SINGLE_DOCUMENT_EVALUATIONS = 20
 const PROSE = '帮我看看这个函数为什么慢，顺便把 README 也更新一下。The quick brown fox jumps over the lazy dog, 1234 times. '
 const CODE = 'const value = await fetch(`https://api.example.com/v1/items?id=${id}`); if (!value.ok) throw new Error("bad: " + value.status);\n'
 const BASE64 = 'QUJDREVGR0hJSktMTU5PUFFSU1RVVldYWVphYmNkZWZnaGlqa2xtbm9wcXJzdHV2d3h5ejAxMjM0NTY3ODkrLw'
@@ -168,48 +187,84 @@ interface Sample {
   readonly maskedCharacters: number
 }
 
+interface SamplePair {
+  readonly base: Sample
+  readonly secret: Sample
+}
+
 /** Rules for one sample; a fresh credential matcher per sample carries no state from an earlier sample. */
 type RuleSet = () => readonly UserTextRule[]
 const withoutSecret: RuleSet = () => BASE_RULES
 const withSecret: RuleSet = () => [...BASE_RULES, createSecretRule()]
 
-function registryWith(rules: RuleSet): { readonly registry: UserTextRegistry; readonly scope: Context } {
-  const scope = new Context()
-  const registry = new UserTextRegistry(scope)
-  for (const rule of rules()) registry.register(rule)
-  return { registry, scope }
-}
-
 /**
- * Evaluate every text once as a fresh document, as a newly rendered message does.
+ * Evaluate fresh documents; await registry disposal outside the measured interval before returning.
  * @returns elapsed and CPU milliseconds, credential nodes produced, and their characters.
  */
-function evaluateAll(rules: RuleSet, texts: readonly string[]): Sample {
-  const { registry, scope } = registryWith(rules)
-  let secrets = 0
-  let maskedCharacters = 0
-  const cpuStart = process.cpuUsage()
-  const start = performance.now()
-  for (const text of texts) {
-    for (const node of registry.createEvaluator()({ text, facts: FACTS })) {
-      if (node.id !== 'secret') continue
-      secrets++
-      maskedCharacters += node.end - node.start
+async function evaluateAll(rules: RuleSet, texts: readonly string[]): Promise<Sample> {
+  const scope = new Context()
+  try {
+    const registry = new UserTextRegistry(scope)
+    for (const rule of rules()) registry.register(rule)
+    let secrets = 0
+    let maskedCharacters = 0
+    const cpuStart = process.cpuUsage()
+    const start = performance.now()
+    for (const text of texts) {
+      for (const node of registry.createEvaluator()({ text, facts: FACTS })) {
+        if (node.id !== 'secret') continue
+        secrets++
+        maskedCharacters += node.end - node.start
+      }
     }
+    const ms = performance.now() - start
+    const cpu = process.cpuUsage(cpuStart)
+    return { ms, cpuMs: (cpu.user + cpu.system) / 1_000, secrets, maskedCharacters }
+  } finally {
+    await scope.fiber.dispose()
   }
-  const ms = performance.now() - start
-  const cpu = process.cpuUsage(cpuStart)
-  void scope.fiber.dispose()
-  return { ms, cpuMs: (cpu.user + cpu.system) / 1_000, secrets, maskedCharacters }
 }
 
-function fastest(rules: RuleSet, texts: readonly string[], attempts: number): Sample {
-  let best = evaluateAll(rules, texts)
-  for (let attempt = 1; attempt < attempts; attempt++) {
-    const sample = evaluateAll(rules, texts)
-    if (sample.ms < best.ms) best = sample
+async function sample(rules: RuleSet, texts: readonly string[], evaluations: number): Promise<Sample> {
+  const first = await evaluateAll(rules, texts)
+  let ms = first.ms
+  let cpuMs = first.cpuMs
+  for (let index = 1; index < evaluations; index++) {
+    const next = await evaluateAll(rules, texts)
+    ms += next.ms
+    cpuMs += next.cpuMs
+  }
+  return { ...first, ms: ms / evaluations, cpuMs: cpuMs / evaluations }
+}
+
+async function evaluatePair(texts: readonly string[], index: number, evaluations: number): Promise<SamplePair> {
+  if (index % 2 === 0) {
+    const base = await sample(withoutSecret, texts, evaluations)
+    return { base, secret: await sample(withSecret, texts, evaluations) }
+  }
+  const secret = await sample(withSecret, texts, evaluations)
+  return { base: await sample(withoutSecret, texts, evaluations), secret }
+}
+
+async function measure(texts: readonly string[], attempts: number, evaluations: number): Promise<SamplePair & {
+  readonly samples: readonly UserTextTimingPair[]
+}> {
+  for (let warm = 0; warm < WARMUP_PAIRS; warm++) await evaluatePair(texts, warm, evaluations)
+  const pairs: SamplePair[] = []
+  for (let attempt = 0; attempt < attempts; attempt++) {
+    pairs.push(await evaluatePair(texts, attempt, evaluations))
+  }
+  const best = pairs.reduce((min, pair) => ({
+    base: pair.base.ms < min.base.ms ? pair.base : min.base,
+    secret: pair.secret.ms < min.secret.ms ? pair.secret : min.secret,
+  }))
+  return {
+    ...best,
+    samples: pairs.map(pair => ({
+      baseMs: pair.base.ms, baseCpuMs: pair.base.cpuMs,
+      withSecretMs: pair.secret.ms, withSecretCpuMs: pair.secret.cpuMs,
+    })),
   }
-  return best
 }
 
 declare const gc: (() => void) | undefined
@@ -307,26 +362,28 @@ async function allocations(): Promise<UserTextAllocationReport> {
   return { workloads: results }
 }
 
-function timings(attempts: number): UserTextTimeReport {
-  return {
-    attempts,
-    workloads: workloads().map(({ id, group, texts }) => {
-      const base = fastest(withoutSecret, texts, attempts)
-      const secret = fastest(withSecret, texts, attempts)
-      return {
-        id,
-        group,
-        documents: texts.length,
-        characters: texts.reduce((sum, text) => sum + text.length, 0),
-        secrets: secret.secrets,
-        maskedCharacters: secret.maskedCharacters,
-        baseMs: base.ms,
-        withSecretMs: secret.ms,
-        withSecretCpuMs: secret.cpuMs,
-        ratio: secret.ms / base.ms,
-      }
-    }),
+async function timings(attempts: number): Promise<UserTextTimeReport> {
+  const results: UserTextWorkloadTiming[] = []
+  for (const { id, group, texts } of workloads()) {
+    const evaluationsPerSample = texts.length === 1 ? SINGLE_DOCUMENT_EVALUATIONS : 1
+    const { base, secret, samples } = await measure(texts, attempts, evaluationsPerSample)
+    results.push({
+      id,
+      group,
+      documents: texts.length,
+      characters: texts.reduce((sum, text) => sum + text.length, 0),
+      evaluationsPerSample,
+      secrets: secret.secrets,
+      maskedCharacters: secret.maskedCharacters,
+      baseMs: base.ms,
+      baseCpuMs: base.cpuMs,
+      withSecretMs: secret.ms,
+      withSecretCpuMs: secret.cpuMs,
+      ratio: secret.ms / base.ms,
+      samples,
+    })
   }
+  return { warmupPairs: WARMUP_PAIRS, attempts, workloads: results }
 }
 
 assertBuiltBenchmarkRuntime(import.meta.url, {
@@ -335,4 +392,4 @@ assertBuiltBenchmarkRuntime(import.meta.url, {
 })
 const [mode, count] = process.argv.slice(2)
 if (mode !== 'time' && mode !== 'allocation') throw new Error('usage: user-text.worker.js time <attempts> | allocation')
-console.log(JSON.stringify(mode === 'time' ? timings(positiveInteger(count, 'attempts')) : await allocations()))
+console.log(JSON.stringify(mode === 'time' ? await timings(positiveInteger(count, 'attempts')) : await allocations()))