fix(code-runtime-python): raise the output-budget worst-case multiple to 12 and reject the boundary
The load-time output-budget/addressSpaceMb gate used a worst-case multiple of 8, assuming two simultaneous ~4x astral copies (the built string and its encode). Three are live at the peak: on the newline path a single write holds the caller's text argument, the line slice handed to push, and push's encode copy; the settlement flush_line path held the pending chunks, their join, and that encode copy. A budget admitted at 8x (e.g. maxLogBytes 48 MiB against addressSpaceMb 512) could still OOM the child. The multiple is now 12, the strict `>` is `>=` so a budget whose peak exactly equals the room left after the interpreter baseline is rejected (that peak plus the baseline is the whole address space), and flush_line drops the pending chunks before its push to match the newline path's join-clear-push order. The child re-check mirror and both note sides move in step; config-catalog is regenerated from the updated field JSDoc.
This commit is contained in:
Chinesezjc authored and Tianyi Cui committed
1 parent
ce91c70f9a
commit
9d9525549d
7 files changed
+119
-82
No files matched your search
+2
-2
@@ -2,5 +2,5 @@
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write .agents/notes/implemented/bug-fix/2026-07-31-code-runtime-python-settlement-fixes.md
|
||||
2026-07-31-code-runtime-python-settlement-fixes.md: 8dc9a1e991f89efe0607f858dbbcc77929f582ce
|
||||
2026-07-31-code-runtime-python-settlement-fixes.zh.md: beb6e1bf4381f542ae33ed36a301ae31e3eabc10
|
||||
2026-07-31-code-runtime-python-settlement-fixes.md: 088b26397765b908cfbf3514fe020301a0a19235
|
||||
2026-07-31-code-runtime-python-settlement-fixes.zh.md: 1d45ceffda823f8cc1fb15f6cfb0a3bcef1688ad
|
||||
+2
-2
@@ -58,7 +58,7 @@ Also in `src/index.ts`, native stdout/stderr bytes (C-extension writes, `os.writ
|
||||
|
||||
### An incompatible output-budget/addressSpaceMb pair is rejected at load
|
||||
|
||||
The child ([`py/bootstrap.py`](../../../../packages/code-runtime/code-runtime-python/py/bootstrap.py)) builds, charges, and frames a `maxLogBytes` log entry or a `maxValueBytes` completion value under `RLIMIT_AS`, and both ledgers trigger on CHARACTER count against a serialized-BYTE budget. An astral character is one character but four bytes of CPython `str` storage and four UTF-8 bytes, so a budget's worth of astral characters is ~4× the budget in the built string and ~4× again in the `encode` copy taken to measure or ship it, live at once — a peak of several times the budget. When a budget approaches `addressSpaceMb`, a LEGITIMATE near-budget output breaches the address space during that build-and-encode and dies as `worker-exit` instead of truncating (log) or failing as `output-limit` (value). Metering every child write against the address space at runtime is the wrong fix: an exact serialized-cost check on the hot path is either a full `encode` (the allocation being avoided) or a per-character Python loop (which burns the CPU budget — a 10 MB legitimate write hits SIGXCPU under `cpuSeconds: 1`). Both trade one resource bound for another. Instead [`src/index.ts`](../../../../packages/code-runtime/code-runtime-python/src/index.ts) rejects the incompatible pair at LOAD: each budget times `OUTPUT_BUDGET_WORST_CASE_ADDRESS_SPACE_MULTIPLE` (eight — the two simultaneous ~4× copies) must fit the address space LEFT after a fixed `INTERPRETER_BASELINE_BYTES` reservation for the interpreter's own footprint, with a strict `>` so a budget whose worst-case peak exactly equals that room is rejected. The baseline is reserved SEPARATELY from the multiple because it is a fixed cost, not one that scales with the budget: folding it into the multiple would leave a budget sized right at `addressSpaceMb / 8` admitted while its peak plus the interpreter still overran. Both `maxLogBytes` and `maxValueBytes` are gated symmetrically; the value path's build-and-encode is the same shape. The check runs on every platform, not just where `RLIMIT_AS` is enforced: the incompatibility is a property of the config values, so a Linux deployment OOMs regardless of the host that assembled the config, and a uniform load-time rejection is the fail-loud contract (Darwin skips only the runtime `setrlimit`). This eliminates the class at the config seam rather than patching the write path, so `_LogStream` keeps its original character-count buffering (a valid lower bound on serialized cost, memory-safe once the budget fits the address space).
|
||||
The child ([`py/bootstrap.py`](../../../../packages/code-runtime/code-runtime-python/py/bootstrap.py)) builds, charges, and frames a `maxLogBytes` log entry or a `maxValueBytes` completion value under `RLIMIT_AS`, and both ledgers trigger on CHARACTER count against a serialized-BYTE budget. An astral character is one character but four bytes of CPython `str` storage and four UTF-8 bytes, and THREE such copies are live at the peak: on the newline path a single `sys.stdout.write(line + "\n")` holds the caller's `text` argument (alive for the whole `write` call, ~4×), the line slice handed to `LogBuffer.push` (~4×), and the `text.encode("utf-8")` copy `_push_locked` takes to charge and ship it (~4×); the settlement `flush_line` path holds the pending chunks, their `"".join(...)`, and that same encode copy — a peak of ~12× the budget. When a budget approaches `addressSpaceMb`, a LEGITIMATE near-budget output breaches the address space during that build-and-encode and dies as `worker-exit` instead of truncating (log) or failing as `output-limit` (value). Metering every child write against the address space at runtime is the wrong fix: an exact serialized-cost check on the hot path is either a full `encode` (the allocation being avoided) or a per-character Python loop (which burns the CPU budget — a 10 MB legitimate write hits SIGXCPU under `cpuSeconds: 1`). Both trade one resource bound for another. Instead [`src/index.ts`](../../../../packages/code-runtime/code-runtime-python/src/index.ts) rejects the incompatible pair at LOAD: each budget times `OUTPUT_BUDGET_WORST_CASE_ADDRESS_SPACE_MULTIPLE` (twelve — the three simultaneous ~4× copies) must fit the address space LEFT after a fixed `INTERPRETER_BASELINE_BYTES` reservation for the interpreter's own footprint, with a `>=` so a budget whose worst-case peak exactly equals that room is rejected (that peak plus the reserved baseline is the whole address space, the `RLIMIT_AS` edge). `flush_line` was also made to drop the pending chunks BEFORE its push, matching the newline path's join-clear-push order, so that path holds at most the join and its encode rather than three copies. The baseline is reserved SEPARATELY from the multiple because it is a fixed cost, not one that scales with the budget: folding it into the multiple would leave a budget sized right at `addressSpaceMb / 12` admitted while its peak plus the interpreter still overran. Both `maxLogBytes` and `maxValueBytes` are gated symmetrically; the value path's build-and-encode is the same shape. The check runs on every platform, not just where `RLIMIT_AS` is enforced: the incompatibility is a property of the config values, so a Linux deployment OOMs regardless of the host that assembled the config, and a uniform load-time rejection is the fail-loud contract (Darwin skips only the runtime `setrlimit`). This eliminates the class at the config seam rather than patching the write path, so `_LogStream` keeps its original character-count buffering (a valid lower bound on serialized cost, memory-safe once the budget fits the address space).
|
||||
|
||||
The host gate validates against the CONFIGURED `addressSpaceMb`, but a launch environment can inherit a STRICTER `RLIMIT_AS` (a `ulimit -v` wrapper below `addressSpaceMb`), which the bootstrap's `_clamped` correctly lowers the EFFECTIVE limit to — leaving the budgets sized for a ceiling the child never gets. So `bootstrap.py` re-checks both budgets against the effective clamped soft limit after applying it, mirroring the host gate's multiple and baseline, and raises at boot (caught by the setrlimit-phase handler and reported as `exception`, the same class as any other resource-limit-application failure) rather than letting a near-budget output OOM mid-run. The two child constants are kept in step with the host's by the shared reasoning, not a wire field.
|
||||
|
||||
@@ -68,7 +68,7 @@ One residual write-path copy is fixed alongside, independent of the config gate:
|
||||
|
||||
- `tests/boot-write-failure.spec.ts` mocks `spawn` so the fd-3 pipe throws on the boot write — the one path a real subprocess cannot be coerced into — and asserts `run()` resolves a `worker-exit` rather than rejecting. A sibling case makes the mocked `spawn` throw SYNCHRONOUSLY and asserts `run()` still resolves a `worker-exit` and removes its staging directory, keyed off the exact bootstrap path the mocked `spawn` received in its argv so a sibling worker's concurrent staging cannot flake it. Both are isolated in this spec so the real-subprocess suite is untouched.
|
||||
- `tests/residual-detach.spec.ts` unit-tests `detachResidual`: the carried copy equals the residual, owns a backing store sized to its own length (fixture kept above Node's Buffer pool threshold), and does not share the source frame's `ArrayBuffer`.
|
||||
- `tests/runtime.spec.ts` — the output-cap case asserts the `ceiling - envelope` bound (268435392) and its message. A daemon-thread case drives four threads emitting unterminated writes through settlement's flush. A native-write case writes 200 KiB with no newline via `os.write` under a raised `maxLogBytes` and asserts it reads back as EXACTLY one log entry (proving stray output is aggregated by line, not split at pipe-chunk boundaries); a companion writes `b"one\ntwo\nthree"` and asserts three entries (proving real newlines still delimit). A newline-free-flood case writes 2 MiB under a 4 KiB `maxLogBytes` and asserts the capture ends at the truncation marker and stays under budget (proving the residual is bounded by the ledger, not buffered whole); a NUL-flood companion writes 4000 newline-free NULs under the same budget and asserts truncation (proving the residual is charged by SERIALIZED cost, ~6× raw, measured without allocating the escaped copy); an illegal-UTF-8 case paces single-byte `\xff` writes under a 3072-byte budget with `Buffer.concat` wrapped to measure the peak merged buffer, asserting it stays under 2048 (charged at the U+FFFD width 3 the residual flushes near 1024 raw bytes; a raw-byte undercount would let it reach ~3072, so the bound discriminates); a CESU-8/overlong case paces the structurally-well-formed but illegal `ED A0 80` one byte at a time and asserts the same peak bound (charged at the true 9 per sequence it flushes early; charging the structural width 3 triples the peak, so reverting the per-lead range check turns it red); a broken-multibyte case writes a 3-byte lead then a fresh ASCII byte in separate chunks and asserts both a captured `A` and a U+FFFD (exercising `accrueStrayCost`'s cross-chunk broken-sequence branch); a post-truncation case writes a 108-byte payload (under the smallest PIPE_BUF, so one atomic write) whose first line exhausts a 64-byte budget and asserts the second line is dropped (exercising the post-truncation admit no-op in one `data` callback, no v8-ignore); a short-escape case writes a line mixing a tab, quote, backslash, a `\uXXXX` control, a multibyte character, and ASCII, asserting it round-trips verbatim (exercising every branch of `jsonStringCostUpTo`); a reassembly case writes a payload spanning every valid multibyte lead class (E0-range, plain 3-byte, F0, and F4) past the pipe buffer and asserts it round-trips with no U+FFFD (exercising `accrueStrayCost`'s per-lead ranges and cross-chunk reassembly); a lone-surrogate case forges an fd-3 `log` frame flooding 1000 `\ud800` escapes under a 4 KiB budget and asserts truncation (the count sits in the window where charging 3 bytes would admit and 6 bytes truncates, proving the surrogate is charged its full escaped width); a stray-sealing case paces 60000 single-byte newline-free `os.write(1, …)` calls under a raised budget with `Buffer.concat` wrapped to measure copy volume, asserting the trickle coalesces to one entry and the cumulative copy stays under a measured 256 KiB threshold (the sealed shape copies ~120 KB, the re-merge shape ~538 KB, so reverting the seal to a re-merge turns the assertion red — proving the fragment list seals into blocks past `MAX_PENDING_CHUNKS`). A closeDeadline-flush case has the leader write a newline-free diagnostic then spawn a `setsid` orphan holding the pipes open, and asserts the diagnostic survives in `logs` (proving the residual is flushed before the deadline destroys the streams). The same-group reap case spawns a SIGTERM-ignoring same-group descendant that releases the pipes and bumps a heartbeat file; the test asserts the heartbeat STOPS after the grace-window SIGKILL — an assertion robust whether the killed descendant is reaped or lingers as a zombie, so it holds where PID 1 does not wait() orphans. A dispose-after-resolve case asserts `dispose()` of a completed run with a same-group survivor returns only after the survivor stops executing (proving the run stays in `live` until its group is reaped), with an `expect(afterDispose).toBeGreaterThan(0)` guard so the frozen-heartbeat assertion cannot pass vacuously when the file was never written. A deadline case busy-blocks the event loop past both timers and asserts the survivor's heartbeat freezes (proving the poll's deadline arm sends SIGKILL itself rather than cancelling the unfired escalation). The cross-loop case runs a binding from a worker thread's own `asyncio.run` loop while the main coroutine yields with `await asyncio.sleep`, asserting the reply round-trips instead of timing out; a companion case abandons a thread's call so its loop closes, then answers it before a later binding — asserting the pump survives the closed-loop `call_soon_threadsafe` (host-gated ordering makes it deterministic, fail-before hangs the later binding to the wall clock). The inherited-soft-limit case runs the interpreter through a `ulimit -S -t` wrapper that sets a CPU soft limit below `cpuSeconds` and asserts the applieLine truncated
|
||||
- `tests/runtime.spec.ts` — the output-cap case asserts the `ceiling - envelope` bound (268435392) and its message. A daemon-thread case drives four threads emitting unterminated writes through settlement's flush. A native-write case writes 200 KiB with no newline via `os.write` under a raised `maxLogBytes` and asserts it reads back as EXACTLY one log entry (proving stray output is aggregated by line, not split at pipe-chunk boundaries); a companion writes `b"one\ntwo\nthree"` and asserts three entries (proving real newlines still delimit). A newline-free-flood case writes 2 MiB under a 4 KiB `maxLogBytes` and asserts the capture ends at the truncation marker and stays under budget (proving the residual is bounded by the ledger, not buffered whole); a NUL-flood companion writes 4000 newline-free NULs under the same budget and asserts truncation (proving the residual is charged by SERIALIZED cost, ~6× raw, measured without allocating the escaped copy); an illegal-UTF-8 case paces single-byte `\xff` writes under a 3072-byte budget with `Buffer.concat` wrapped to measure the peak merged buffer, asserting it stays under 2048 (charged at the U+FFFD width 3 the residual flushes near 1024 raw bytes; a raw-byte undercount would let it reach ~3072, so the bound discriminates); a CESU-8/overlong case paces the structurally-well-formed but illegal `ED A0 80` one byte at a time and asserts the same peak bound (charged at the true 9 per sequence it flushes early; charging the structural width 3 triples the peak, so reverting the per-lead range check turns it red); a broken-multibyte case writes a 3-byte lead then a fresh ASCII byte in separate chunks and asserts both a captured `A` and a U+FFFD (exercising `accrueStrayCost`'s cross-chunk broken-sequence branch); a post-truncation case writes a 108-byte payload (under the smallest PIPE_BUF, so one atomic write) whose first line exhausts a 64-byte budget and asserts the second line is dropped (exercising the post-truncation admit no-op in one `data` callback, no v8-ignore); a short-escape case writes a line mixing a tab, quote, backslash, a `\uXXXX` control, a multibyte character, and ASCII, asserting it round-trips verbatim (exercising every branch of `jsonStringCostUpTo`); a reassembly case writes a payload spanning every valid multibyte lead class (E0-range, plain 3-byte, F0, and F4) past the pipe buffer and asserts it round-trips with no U+FFFD (exercising `accrueStrayCost`'s per-lead ranges and cross-chunk reassembly); a lone-surrogate case forges an fd-3 `log` frame flooding 1000 `\ud800` escapes under a 4 KiB budget and asserts truncation (the count sits in the window where charging 3 bytes would admit and 6 bytes truncates, proving the surrogate is charged its full escaped width); a stray-sealing case paces 60000 single-byte newline-free `os.write(1, …)` calls under a raised budget with `Buffer.concat` wrapped to measure copy volume, asserting the trickle coalesces to one entry and the cumulative copy stays under a measured 256 KiB threshold (the sealed shape copies ~120 KB, the re-merge shape ~538 KB, so reverting the seal to a re-merge turns the assertion red — proving the fragment list seals into blocks past `MAX_PENDING_CHUNKS`). A closeDeadline-flush case has the leader write a newline-free diagnostic then spawn a `setsid` orphan holding the pipes open, and asserts the diagnostic survives in `logs` (proving the residual is flushed before the deadline destroys the streams). The same-group reap case spawns a SIGTERM-ignoring same-group descendant that releases the pipes and bumps a heartbeat file; the test asserts the heartbeat STOPS after the grace-window SIGKILL — an assertion robust whether the killed descendant is reaped or lingers as a zombie, so it holds where PID 1 does not wait() orphans. A dispose-after-resolve case asserts `dispose()` of a completed run with a same-group survivor returns only after the survivor stops executing (proving the run stays in `live` until its group is reaped), with an `expect(afterDispose).toBeGreaterThan(0)` guard so the frozen-heartbeat assertion cannot pass vacuously when the file was never written. A deadline case busy-blocks the event loop past both timers and asserts the survivor's heartbeat freezes (proving the poll's deadline arm sends SIGKILL itself rather than cancelling the unfired escalation). The cross-loop case runs a binding from a worker thread's own `asyncio.run` loop while the main coroutine yields with `await asyncio.sleep`, asserting the reply round-trips instead of timing out; a companion case abandons a thread's call so its loop closes, then answers it before a later binding — asserting the pump survives the closed-loop `call_soon_threadsafe` (host-gated ordering makes it deterministic, fail-before hangs the later binding to the wall clock). The inherited-soft-limit case runs the interpreter through a `ulimit -S -t` wrapper that sets a CPU soft limit below `cpuSeconds` and asserts the applieLine truncated
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
|
||||
+2
-2
@@ -58,7 +58,7 @@ Status: implemented
|
||||
|
||||
### An incompatible output-budget/addressSpaceMb pair is rejected at load
|
||||
|
||||
子进程（[`py/bootstrap.py`](../../../../packages/code-runtime/code-runtime-python/py/bootstrap.py)）在 `RLIMIT_AS` 之下构建、计费并分帧一条 `maxLogBytes` 的日志条目或一个 `maxValueBytes` 的完成值，而两个账本都是按字符计数、对照一个序列化字节预算触发的。一个星芒面字符是一个字符，但占 CPython `str` 存储的四个字节以及四个 UTF-8 字节，因此一个预算份额的星芒面字符在构建出的字符串中约为预算的 4 倍，在为度量或发送它而取的 `encode` 副本中再约 4 倍，两者同时存活——峰值为预算的数倍。当一项预算逼近 `addressSpaceMb` 时，一次合法的、接近预算的输出会在那次构建加编码期间突破地址空间，并作为 `worker-exit`（日志）而不是截断而终止，或作为 `output-limit`（值）而失败。在运行时对每次子进程写入按地址空间计量是错误的修复：热路径上一次精确的序列化开销检查，要么是一次完整的 `encode`（正是要避免的那次分配），要么是一个逐字符的 Python 循环（它会烧掉 CPU 预算——一次 10 MB 的合法写入会在 `cpuSeconds: 1` 之下触发 SIGXCPU）。两者都是拿一种资源界限换另一种。取而代之，[`src/index.ts`](../../../../packages/code-runtime/code-runtime-python/src/index.ts) 在加载期（LOAD）拒绝这个不兼容的组合：每项预算乘以 `OUTPUT_BUDGET_WORST_CASE_ADDRESS_SPACE_MULTIPLE`（八——两份同时存在的约 4 倍副本）必须放得进为解释器自身占用预留一份固定的 `INTERPRETER_BASELINE_BYTES` 之后剩下的地址空间，并用一个严格的 `>`，使得一项其最坏情况峰值恰好等于那片余量的预算也会被拒绝。该基线是与倍数分开（SEPARATELY）预留的，因为它是一项固定开销，而非随预算伸缩的开销：把它折进倍数会让一项恰好为 `addressSpaceMb / 8` 大小的预算被放行，而其峰值加上解释器仍会越界。`maxLogBytes` 和 `maxValueBytes` 都被对称地门控；值路径的构建加编码是同一形态。该检查在每个平台上都运行，而不仅在强制 `RLIMIT_AS` 的平台上：这种不兼容是那些配置值的属性，因此一个 Linux 部署无论由哪个宿主组装配置都会 OOM，而一致的加载期拒绝正是 fail-loud 契约（Darwin 仅跳过运行时的 `setrlimit`）。这在配置 seam 处消除了这一类问题，而不是给写入路径打补丁，因此 `_LogStream` 保留它原有的按字符计数的缓冲（一个对序列化开销有效的下界，一旦预算放进地址空间就是内存安全的）。
|
||||
子进程（[`py/bootstrap.py`](../../../../packages/code-runtime/code-runtime-python/py/bootstrap.py)）在 `RLIMIT_AS` 之下构建、计费并分帧一条 `maxLogBytes` 的日志条目或一个 `maxValueBytes` 的完成值，而两个账本都是按字符计数、对照一个序列化字节预算触发的。一个星芒面字符是一个字符，但占 CPython `str` 存储的四个字节以及四个 UTF-8 字节，且峰值时有三份这样的副本同时存活：换行路径上一次 `sys.stdout.write(line + "\n")` 会持有调用方的 `text` 实参（在整个 `write` 调用期间存活，约 4 倍）、交给 `LogBuffer.push` 的行切片（约 4 倍）、以及 `_push_locked` 为计费和发送而取的 `text.encode("utf-8")` 副本（约 4 倍）；结算期的 `flush_line` 路径则持有 pending 分块、它们的 `"".join(...)` 以及同一份 encode 副本——峰值约为预算的 12 倍。当一项预算逼近 `addressSpaceMb` 时，一次合法的、接近预算的输出会在那次构建加编码期间突破地址空间，并作为 `worker-exit`（日志）而不是截断而终止，或作为 `output-limit`（值）而失败。在运行时对每次子进程写入按地址空间计量是错误的修复：热路径上一次精确的序列化开销检查，要么是一次完整的 `encode`（正是要避免的那次分配），要么是一个逐字符的 Python 循环（它会烧掉 CPU 预算——一次 10 MB 的合法写入会在 `cpuSeconds: 1` 之下触发 SIGXCPU）。两者都是拿一种资源界限换另一种。取而代之，[`src/index.ts`](../../../../packages/code-runtime/code-runtime-python/src/index.ts) 在加载期（LOAD）拒绝这个不兼容的组合：每项预算乘以 `OUTPUT_BUDGET_WORST_CASE_ADDRESS_SPACE_MULTIPLE`（十二——三份同时存在的约 4 倍副本）必须放得进为解释器自身占用预留一份固定的 `INTERPRETER_BASELINE_BYTES` 之后剩下的地址空间，并用一个 `>=`，使得一项其最坏情况峰值恰好等于那片余量的预算也会被拒绝（该峰值加上预留的基线正好是整个地址空间，即 `RLIMIT_AS` 边界）。`flush_line` 也被改为在 push 之前先丢弃 pending 分块，与换行路径的 join-清空-push 顺序一致，使该路径至多只持有 join 及其 encode 副本，而非三份副本。该基线是与倍数分开（SEPARATELY）预留的，因为它是一项固定开销，而非随预算伸缩的开销：把它折进倍数会让一项恰好为 `addressSpaceMb / 12` 大小的预算被放行，而其峰值加上解释器仍会越界。`maxLogBytes` 和 `maxValueBytes` 都被对称地门控；值路径的构建加编码是同一形态。该检查在每个平台上都运行，而不仅在强制 `RLIMIT_AS` 的平台上：这种不兼容是那些配置值的属性，因此一个 Linux 部署无论由哪个宿主组装配置都会 OOM，而一致的加载期拒绝正是 fail-loud 契约（Darwin 仅跳过运行时的 `setrlimit`）。这在配置 seam 处消除了这一类问题，而不是给写入路径打补丁，因此 `_LogStream` 保留它原有的按字符计数的缓冲（一个对序列化开销有效的下界，一旦预算放进地址空间就是内存安全的）。
|
||||
|
||||
宿主门控是对照配置的（CONFIGURED）`addressSpaceMb` 校验的，但一个启动环境可能继承一个更严格的（STRICTER）`RLIMIT_AS`（一个低于 `addressSpaceMb` 的 `ulimit -v` 包装层），而 bootstrap 的 `_clamped` 会正确地把有效（EFFECTIVE）限制降到该值——从而让这些预算是按一个子进程永远得不到的上限来定尺寸的。因此 `bootstrap.py` 在应用该有效被夹紧的软限制之后，会对照它重新检查两项预算，镜像宿主门控的倍数与基线，并在引导期抛出（被 setrlimit 阶段的处理器捕获，并作为 `exception` 上报——与任何其他资源限制应用失败同属一类），而不是任由一次接近预算的输出在运行途中 OOM。这两个子进程侧常量与宿主侧的保持一致，靠的是共享的推理，而不是一个 wire 字段。
|
||||
|
||||
@@ -68,7 +68,7 @@ Status: implemented
|
||||
|
||||
- `tests/boot-write-failure.spec.ts` 对 `spawn` 做 mock，使 fd-3 管道在引导写入时抛出异常（这是真实子进程无法被迫进入的唯一路径），并断言 `run()` resolve 出一个 `worker-exit` 而非 reject。一个同级用例让被 mock 的 `spawn` 同步抛出，并断言 `run()` 仍然 resolve 出一个 `worker-exit`，且会移除它的暂存目录——以被 mock 的 `spawn` 在其 argv 中收到的确切引导路径为准，因此一个同级 worker 的并发暂存不会让它变得不稳定。两者都被隔离在这个 spec 中，因此真实子进程测试套件不受影响。
|
||||
- `tests/residual-detach.spec.ts` 对 `detachResidual` 做单元测试：向前传递的副本与残余数据相等、拥有一个大小与其自身长度一致的底层存储（fixture 保持在 Node 的 Buffer 池阈值之上），并且不与源帧的 `ArrayBuffer` 共享。
|
||||
- `tests/runtime.spec.ts`：output-cap 用例断言 `ceiling - envelope` 上界（268435392）及其消息。一个 daemon 线程用例驱动四个线程穿过结算的 flush 发出未结束的写入。一个 native-write 用例在抬高后的 `maxLogBytes` 之下，通过 `os.write` 写入 200 KiB 且不含换行符，断言它回读时恰好是一条日志条目（证明散逸输出是按行聚合的，而不是在管道分片边界处被切开）；一个配套用例写入 `b"one\ntwo\nthree"`，断言得到三条条目（证明真正的换行符仍然起分隔作用）。一个 newline-free-flood 用例在一个 4 KiB 的 `maxLogBytes` 之下写入 2 MiB，断言捕获终止于截断标记且保持在预算之内（证明残余数据受账本约束，而不是被整体缓冲）；一个 NUL-flood 配套用例在同一预算之下写入 4000 个不含换行符的 NUL，断言发生截断（证明残余数据是按序列化开销计费的，约为原始的 6 倍，且在度量时不分配转义后的副本）；一个 illegal-UTF-8 用例在一个 3072 字节的预算之下控速发出单字节 `\xff` 写入，并对 `Buffer.concat` 做包装以度量峰值合并缓冲区，断言它保持在 2048 之下（按 U+FFFD 宽度 3 计费时残余数据在约 1024 原始字节处冲刷；一次原始字节的少计会让它达到约 3072，因此该界限具有区分力）；一个 CESU-8/overlong 用例把结构良构但非法的 `ED A0 80` 一次一个字节地控速发出，断言同样的峰值界限（按每序列真实的 9 计费时它提前冲刷；按结构宽度 3 计费会使峰值增至三倍，因此把逐前导字节范围检查回退会使它变红）；一个 broken-multibyte 用例在分开的分片里先写入一个 3 字节的前导字节、再写入一个新的 ASCII 字节，断言同时捕获到一个 `A` 和一个 U+FFFD（覆盖 `accrueStrayCost` 的跨分片断裂序列分支）；一个 post-truncation 用例写入一个 108 字节的载荷（小于最小的 PIPE_BUF，因此是一次原子写入），其首行耗尽一个 64 字节的预算，断言第二行被丢弃（覆盖单次 `data` 回调中的截断后准入空操作，无需 v8-ignore）；一个 short-escape 用例写入一行混合了制表符、引号、反斜杠、一个 `\uXXXX` 控制字符、一个多字节字符和 ASCII 的内容，断言它原样完成往返（覆盖 `jsonStringCostUpTo` 的每一条分支）；一个 reassembly 用例写入一个跨越每个合法多字节前导字节类别（E0 范围、普通 3 字节、F0 和 F4）、越过管道缓冲区的载荷，断言它原样完成往返且不含 U+FFFD（覆盖 `accrueStrayCost` 的逐前导字节范围与跨分片重组）；一个 lone-surrogate 用例在一个 4 KiB 预算之下伪造一个以 1000 个 `\ud800` 转义洪泛的 fd-3 `log` 帧，断言发生截断（该计数正落在计 3 字节会放行、计 6 字节则截断的窗口内，证明该代理项是按其完整转义宽度计费的）；一个 stray-sealing 用例在抬高后的预算之下控速发出 60000 次单字节、不含换行符的 `os.write(1, …)` 调用，并对 `Buffer.concat` 做包装以度量复制量，断言这股细流合并为一条条目、且累积复制量保持在一个实测的 256 KiB 阈值之下（封存后的形态复制约 120 KB，重新合并的形态复制约 538 KB，因此把封存回退成重新合并会使该断言变红——证明分片列表在越过 `MAX_PENDING_CHUNKS` 后封存为块）。一个 closeDeadline-flush 用例让 leader 写入一段不含换行符的诊断，随后 spawn 一个持有管道不放的 `setsid` 孤儿进程，断言该诊断在 `logs` 中存留下来（证明残余数据在截止时间销毁流之前被冲刷）。same-group 回收用例 spawn 一个忽略 SIGTERM 的同进程组后代，它释放管道并递增一个心跳文件；该测试断言在宽限窗口的 SIGKILL 之后心跳停止：无论被杀死的后代是被回收还是作为僵尸进程滞留，这个断言都成立，因此它在 PID 1 不 wait() 孤儿进程的环境下同样成立。一个 dispose-after-resolve 用例断言，对一个已完成、且存在同进程组存活者的运行调用 `dispose()`，只有在该存活者停止执行之后才返回（证明该运行会一直留在 `live` 中，直到它的进程组被回收），并带有一个 `expect(afterDispose).toBeGreaterThan(0)` 守卫，使得当心跳文件从未被写入时，冻结心跳的断言不会被空洞地通过。一个 deadline 用例忙阻塞事件循环越过两个定时器，断言该存活者的心跳冻结（证明轮询的截止时间分支自身发送 SIGKILL，而不是取消尚未触发的升级）。cross-loop 用例在主协程通过 `await asyncio.sleep` 让出时，从一个工作线程自己的 `asyncio.run` 事件循环运行一个绑定，断言该回复完成往返而不是超时；一个配套用例放弃某个线程的调用，使其事件循环关闭，Line truncated
|
||||
- `tests/runtime.spec.ts`：output-cap 用例断言 `ceiling - envelope` 上界（268435392）及其消息。一个 daemon 线程用例驱动四个线程穿过结算的 flush 发出未结束的写入。一个 native-write 用例在抬高后的 `maxLogBytes` 之下，通过 `os.write` 写入 200 KiB 且不含换行符，断言它回读时恰好是一条日志条目（证明散逸输出是按行聚合的，而不是在管道分片边界处被切开）；一个配套用例写入 `b"one\ntwo\nthree"`，断言得到三条条目（证明真正的换行符仍然起分隔作用）。一个 newline-free-flood 用例在一个 4 KiB 的 `maxLogBytes` 之下写入 2 MiB，断言捕获终止于截断标记且保持在预算之内（证明残余数据受账本约束，而不是被整体缓冲）；一个 NUL-flood 配套用例在同一预算之下写入 4000 个不含换行符的 NUL，断言发生截断（证明残余数据是按序列化开销计费的，约为原始的 6 倍，且在度量时不分配转义后的副本）；一个 illegal-UTF-8 用例在一个 3072 字节的预算之下控速发出单字节 `\xff` 写入，并对 `Buffer.concat` 做包装以度量峰值合并缓冲区，断言它保持在 2048 之下（按 U+FFFD 宽度 3 计费时残余数据在约 1024 原始字节处冲刷；一次原始字节的少计会让它达到约 3072，因此该界限具有区分力）；一个 CESU-8/overlong 用例把结构良构但非法的 `ED A0 80` 一次一个字节地控速发出，断言同样的峰值界限（按每序列真实的 9 计费时它提前冲刷；按结构宽度 3 计费会使峰值增至三倍，因此把逐前导字节范围检查回退会使它变红）；一个 broken-multibyte 用例在分开的分片里先写入一个 3 字节的前导字节、再写入一个新的 ASCII 字节，断言同时捕获到一个 `A` 和一个 U+FFFD（覆盖 `accrueStrayCost` 的跨分片断裂序列分支）；一个 post-truncation 用例写入一个 108 字节的载荷（小于最小的 PIPE_BUF，因此是一次原子写入），其首行耗尽一个 64 字节的预算，断言第二行被丢弃（覆盖单次 `data` 回调中的截断后准入空操作，无需 v8-ignore）；一个 short-escape 用例写入一行混合了制表符、引号、反斜杠、一个 `\uXXXX` 控制字符、一个多字节字符和 ASCII 的内容，断言它原样完成往返（覆盖 `jsonStringCostUpTo` 的每一条分支）；一个 reassembly 用例写入一个跨越每个合法多字节前导字节类别（E0 范围、普通 3 字节、F0 和 F4）、越过管道缓冲区的载荷，断言它原样完成往返且不含 U+FFFD（覆盖 `accrueStrayCost` 的逐前导字节范围与跨分片重组）；一个 lone-surrogate 用例在一个 4 KiB 预算之下伪造一个以 1000 个 `\ud800` 转义洪泛的 fd-3 `log` 帧，断言发生截断（该计数正落在计 3 字节会放行、计 6 字节则截断的窗口内，证明该代理项是按其完整转义宽度计费的）；一个 stray-sealing 用例在抬高后的预算之下控速发出 60000 次单字节、不含换行符的 `os.write(1, …)` 调用，并对 `Buffer.concat` 做包装以度量复制量，断言这股细流合并为一条条目、且累积复制量保持在一个实测的 256 KiB 阈值之下（封存后的形态复制约 120 KB，重新合并的形态复制约 538 KB，因此把封存回退成重新合并会使该断言变红——证明分片列表在越过 `MAX_PENDING_CHUNKS` 后封存为块）。一个 closeDeadline-flush 用例让 leader 写入一段不含换行符的诊断，随后 spawn 一个持有管道不放的 `setsid` 孤儿进程，断言该诊断在 `logs` 中存留下来（证明残余数据在截止时间销毁流之前被冲刷）。same-group 回收用例 spawn 一个忽略 SIGTERM 的同进程组后代，它释放管道并递增一个心跳文件；该测试断言在宽限窗口的 SIGKILL 之后心跳停止：无论被杀死的后代是被回收还是作为僵尸进程滞留，这个断言都成立，因此它在 PID 1 不 wait() 孤儿进程的环境下同样成立。一个 dispose-after-resolve 用例断言，对一个已完成、且存在同进程组存活者的运行调用 `dispose()`，只有在该存活者停止执行之后才返回（证明该运行会一直留在 `live` 中，直到它的进程组被回收），并带有一个 `expect(afterDispose).toBeGreaterThan(0)` 守卫，使得当心跳文件从未被写入时，冻结心跳的断言不会被空洞地通过。一个 deadline 用例忙阻塞事件循环越过两个定时器，断言该存活者的心跳冻结（证明轮询的截止时间分支自身发送 SIGKILL，而不是取消尚未触发的升级）。cross-loop 用例在主协程通过 `await asyncio.sleep` 让出时，从一个工作线程自己的 `asyncio.run` 事件循环运行一个绑定，断言该回复完成往返而不是超时；一个配套用例放弃某个线程的调用，使其事件循环关闭，Line truncated
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
|
||||
+12
-8
@@ -379,24 +379,28 @@ export interface Config {
|
||||
* cleanly. Not applied on Darwin, where the dyld shared cache mapped into
|
||||
* every process at exec exceeds any practical cap and the kernel rejects
|
||||
* the call; `cpuSeconds` and `maxWallMs` still bound the run there. Bounds
|
||||
* `maxLogBytes`/`maxValueBytes` at load on EVERY platform (not just where the
|
||||
* limit is enforced): each budget times a worst-case Unicode expansion must
|
||||
* fit this byte count, so a near-budget output cannot breach the address space
|
||||
* during the child's build-and-encode.
|
||||
* `maxLogBytes`/`maxValueBytes` at load on EVERY platform (this static check
|
||||
* runs on Darwin too, where only the runtime `setrlimit` is skipped): each
|
||||
* budget times a worst-case Unicode expansion must fit this byte count minus a
|
||||
* fixed interpreter baseline, so a near-budget output cannot breach the address
|
||||
* space during the child's build-and-encode.
|
||||
*/
|
||||
addressSpaceMb?: number
|
||||
/**
|
||||
* Shared byte budget for captured log text (host-side ledger). Bounded at load
|
||||
* against `addressSpaceMb`: the child builds and encodes a near-budget entry
|
||||
* under RLIMIT_AS, so this cap times the worst-case Unicode expansion must fit
|
||||
* the address space (see `addressSpaceMb`).
|
||||
* under RLIMIT_AS with several copies live at once, so this cap times the
|
||||
* worst-case Unicode expansion must fit the address space left after the
|
||||
* interpreter baseline (see `addressSpaceMb`) — a load-time rejection, not a
|
||||
* runtime clamp.
|
||||
*/
|
||||
maxLogBytes?: number
|
||||
/**
|
||||
* Byte cap for the completion value. Bounded at load against `addressSpaceMb`
|
||||
* the same way `maxLogBytes` is: the child builds and encodes a near-budget
|
||||
* value under RLIMIT_AS, so this cap times the worst-case Unicode expansion
|
||||
* must fit the address space.
|
||||
* value under RLIMIT_AS with several copies live at once, so this cap times the
|
||||
* worst-case Unicode expansion must fit the address space left after the
|
||||
* interpreter baseline.
|
||||
*/
|
||||
maxValueBytes?: number
|
||||
/** SIGTERM→SIGKILL grace period on kill, matching bash-local's default. */
|
||||
|
||||
@@ -51,11 +51,13 @@ _MAX_FALLBACK_NAME_CHARS = 200
|
||||
# Mirror of the host's output-budget/address-space gate (src/index.ts's
|
||||
# OUTPUT_BUDGET_WORST_CASE_ADDRESS_SPACE_MULTIPLE and INTERPRETER_BASELINE_BYTES),
|
||||
# re-applied against the EFFECTIVE RLIMIT_AS after inheritance clamping. An astral
|
||||
# character is one character but ~4 bytes of str storage and ~4 UTF-8 bytes, live
|
||||
# at once while the ledger charges and frames it, so a budget's worst-case peak is
|
||||
# eight times its byte count; the interpreter's own footprint is reserved on top.
|
||||
# Kept in sync with the host constants by the shared reasoning, not a wire field.
|
||||
_OUTPUT_BUDGET_WORST_CASE_MULTIPLE = 8
|
||||
# character is one character but ~4 bytes of str storage and ~4 UTF-8 bytes, and
|
||||
# three such copies are live at the peak — the caller's write argument, the line
|
||||
# slice or joined pending handed to push, and the encode copy push takes — so a
|
||||
# budget's worst-case peak is twelve times its byte count; the interpreter's own
|
||||
# footprint is reserved on top. Kept in sync with the host constants by the shared
|
||||
# reasoning, not a wire field.
|
||||
_OUTPUT_BUDGET_WORST_CASE_MULTIPLE = 12
|
||||
_INTERPRETER_BASELINE_BYTES = 64 * 1024 * 1024
|
||||
|
||||
|
||||
@@ -350,9 +352,16 @@ class _LogStream(io.TextIOBase):
|
||||
# against them.
|
||||
with self._logs.lock:
|
||||
if self._pending:
|
||||
self._logs.push("".join(self._pending))
|
||||
# Join, drop the chunks, THEN push — the same order the newline
|
||||
# path uses (:232-235). Pushing before the clear would keep the
|
||||
# pending chunks alive through `_push_locked`'s `text.encode`, so
|
||||
# the chunks, their join, and the encode copy would all be live at
|
||||
# once; dropping the chunks first leaves only the join and its
|
||||
# encode, matching that path's peak.
|
||||
line = "".join(self._pending)
|
||||
self._pending = []
|
||||
self._pending_chars = 0
|
||||
self._logs.push(line)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -664,7 +673,7 @@ async def _run(channel: ProtocolChannel) -> None:
|
||||
if effective_soft != resource.RLIM_INFINITY:
|
||||
budgetable = effective_soft - _INTERPRETER_BASELINE_BYTES
|
||||
for _budget_key in ("maxLogBytes", "maxValueBytes"):
|
||||
if int(boot[_budget_key]) * _OUTPUT_BUDGET_WORST_CASE_MULTIPLE > budgetable:
|
||||
if int(boot[_budget_key]) * _OUTPUT_BUDGET_WORST_CASE_MULTIPLE >= budgetable:
|
||||
raise ValueError(
|
||||
"config.%s is too large for the inherited RLIMIT_AS of %d bytes "
|
||||
"(a near-budget output would breach it during encode); "
|
||||
|
||||
@@ -57,24 +57,28 @@ export interface Config {
|
||||
* cleanly. Not applied on Darwin, where the dyld shared cache mapped into
|
||||
* every process at exec exceeds any practical cap and the kernel rejects
|
||||
* the call; `cpuSeconds` and `maxWallMs` still bound the run there. Bounds
|
||||
* `maxLogBytes`/`maxValueBytes` at load on EVERY platform (not just where the
|
||||
* limit is enforced): each budget times a worst-case Unicode expansion must
|
||||
* fit this byte count, so a near-budget output cannot breach the address space
|
||||
* during the child's build-and-encode.
|
||||
* `maxLogBytes`/`maxValueBytes` at load on EVERY platform (this static check
|
||||
* runs on Darwin too, where only the runtime `setrlimit` is skipped): each
|
||||
* budget times a worst-case Unicode expansion must fit this byte count minus a
|
||||
* fixed interpreter baseline, so a near-budget output cannot breach the address
|
||||
* space during the child's build-and-encode.
|
||||
*/
|
||||
addressSpaceMb?: number
|
||||
/**
|
||||
* Shared byte budget for captured log text (host-side ledger). Bounded at load
|
||||
* against `addressSpaceMb`: the child builds and encodes a near-budget entry
|
||||
* under RLIMIT_AS, so this cap times the worst-case Unicode expansion must fit
|
||||
* the address space (see `addressSpaceMb`).
|
||||
* under RLIMIT_AS with several copies live at once, so this cap times the
|
||||
* worst-case Unicode expansion must fit the address space left after the
|
||||
* interpreter baseline (see `addressSpaceMb`) — a load-time rejection, not a
|
||||
* runtime clamp.
|
||||
*/
|
||||
maxLogBytes?: number
|
||||
/**
|
||||
* Byte cap for the completion value. Bounded at load against `addressSpaceMb`
|
||||
* the same way `maxLogBytes` is: the child builds and encodes a near-budget
|
||||
* value under RLIMIT_AS, so this cap times the worst-case Unicode expansion
|
||||
* must fit the address space.
|
||||
* value under RLIMIT_AS with several copies live at once, so this cap times the
|
||||
* worst-case Unicode expansion must fit the address space left after the
|
||||
* interpreter baseline.
|
||||
*/
|
||||
maxValueBytes?: number
|
||||
/** SIGTERM→SIGKILL grace period on kill, matching bash-local's default. */
|
||||
@@ -236,20 +240,24 @@ const CLOSE_REAP_MARGIN_MS = 2_000
|
||||
* as a multiple of the budget. The child's ledgers trigger on CHARACTER count
|
||||
* against a serialized-BYTE budget, and an astral character is one character but
|
||||
* four bytes of CPython `str` storage and four UTF-8 bytes — so a budget's worth
|
||||
* of astral characters is ~4x the budget in the built string and ~4x again in
|
||||
* the `encode` copy taken to measure or ship it, live at the same time (the
|
||||
* concat that briefly holds both is bounded by those two). Eight covers that
|
||||
* simultaneous pair. The interpreter baseline is NOT in this multiple — it is
|
||||
* reserved separately as {@link INTERPRETER_BASELINE_BYTES} — because it is a
|
||||
* fixed cost, not one that scales with the budget. Used to bound
|
||||
* `maxLogBytes`/`maxValueBytes` against `addressSpaceMb` at load, with a STRICT
|
||||
* `>` so a budget whose worst-case peak exactly equals the room left after the
|
||||
* baseline is rejected, so a legitimate near-budget output truncates (log) or
|
||||
* fails as `output-limit` (value) rather than breaching `RLIMIT_AS` as
|
||||
* `worker-exit`. A fixed safety invariant tying the budgets to the address
|
||||
* space, not a knob.
|
||||
* of astral characters is ~4x the budget in each string that holds it. THREE
|
||||
* such copies are live at the peak: on the newline path a single
|
||||
* `sys.stdout.write(line + "\n")` holds the caller's `text` argument (alive for
|
||||
* the whole `write` call, ~4x), the line slice `text[pos:newline]` handed to
|
||||
* `LogBuffer.push` (~4x), and the `text.encode("utf-8")` copy `_push_locked`
|
||||
* takes to charge and ship it (~4x); the settlement `flush_line` path holds the
|
||||
* pending chunks, their `"".join(...)`, and that same encode copy. Twelve covers
|
||||
* those three simultaneous ~4x copies. The interpreter baseline is NOT in this
|
||||
* multiple — it is reserved separately as {@link INTERPRETER_BASELINE_BYTES} —
|
||||
* because it is a fixed cost, not one that scales with the budget. Used to bound
|
||||
* `maxLogBytes`/`maxValueBytes` against `addressSpaceMb` at load, with a `>=` so
|
||||
* a budget whose worst-case peak exactly equals the room left after the baseline
|
||||
* is rejected (that peak plus the baseline is the whole address space, the
|
||||
* RLIMIT_AS edge), so a legitimate near-budget output truncates (log) or fails
|
||||
* as `output-limit` (value) rather than breaching `RLIMIT_AS` as `worker-exit`.
|
||||
* A fixed safety invariant tying the budgets to the address space, not a knob.
|
||||
*/
|
||||
const OUTPUT_BUDGET_WORST_CASE_ADDRESS_SPACE_MULTIPLE = 8
|
||||
const OUTPUT_BUDGET_WORST_CASE_ADDRESS_SPACE_MULTIPLE = 12
|
||||
|
||||
/**
|
||||
* Fixed address-space headroom reserved for the CPython interpreter itself
|
||||
@@ -728,20 +736,21 @@ export class PythonCodeRuntime extends CodeRuntime {
|
||||
// `maxValueBytes` completion value under `RLIMIT_AS`, and both paths trigger
|
||||
// on CHARACTER count against a serialized-BYTE budget. An astral character is
|
||||
// one character but four bytes of `str` storage and four UTF-8 bytes, so a
|
||||
// budget's worth of them peaks at several simultaneous ~4x copies (the built
|
||||
// string, the concat that still references it, and the encode taken to
|
||||
// measure or ship it). A budget approaching `addressSpaceMb` therefore makes
|
||||
// a LEGITIMATE near-budget output breach the address space and die as
|
||||
// `worker-exit` instead of truncating (log) or failing as `output-limit`
|
||||
// (value). Metering every child write against the address space at runtime is
|
||||
// the wrong fix — an exact serialized-cost check is either a full encode (the
|
||||
// allocation being avoided) or a per-character Python loop that burns the CPU
|
||||
// budget — so the incompatible pair is rejected at load: each budget times the
|
||||
// worst-case multiple must fit the address space. Checked on every platform,
|
||||
// not just where `RLIMIT_AS` is enforced: the incompatibility is a property of
|
||||
// the config values, and the child OOMs on a Linux deployment regardless of
|
||||
// the host that assembled the config, so a uniform load-time rejection is the
|
||||
// fail-loud contract (Darwin skips only the runtime `setrlimit`).
|
||||
// budget's worth of them peaks at three simultaneous ~4x copies (the caller's
|
||||
// write argument, the line slice or joined pending handed to push, and the
|
||||
// encode push takes to charge and ship it). A budget approaching
|
||||
// `addressSpaceMb` therefore makes a LEGITIMATE near-budget output breach the
|
||||
// address space and die as `worker-exit` instead of truncating (log) or
|
||||
// failing as `output-limit` (value). Metering every child write against the
|
||||
// address space at runtime is the wrong fix — an exact serialized-cost check
|
||||
// is either a full encode (the allocation being avoided) or a per-character
|
||||
// Python loop that burns the CPU budget — so the incompatible pair is rejected
|
||||
// at load: each budget times the worst-case multiple must fit the address
|
||||
// space. Checked on every platform, not just where `RLIMIT_AS` is enforced:
|
||||
// the incompatibility is a property of the config values, and the child OOMs
|
||||
// on a Linux deployment regardless of the host that assembled the config, so a
|
||||
// uniform load-time rejection is the fail-loud contract (Darwin skips only the
|
||||
// runtime `setrlimit`).
|
||||
const addressSpaceBytes = this.config.addressSpaceMb * 1024 * 1024
|
||||
// Room left for the peak output allocation after the interpreter's own fixed
|
||||
// footprint. A budget must fit MULTIPLE times over into THIS, not the whole
|
||||
@@ -749,10 +758,15 @@ export class PythonCodeRuntime extends CodeRuntime {
|
||||
// the multiple alone would admit — cannot leave the peak plus the interpreter
|
||||
// over the limit.
|
||||
const budgetableBytes = addressSpaceBytes - INTERPRETER_BASELINE_BYTES
|
||||
const admissibleBudget = Math.floor(budgetableBytes / OUTPUT_BUDGET_WORST_CASE_ADDRESS_SPACE_MULTIPLE)
|
||||
// The largest budget that fits: the peak (budget * MULTIPLE) must leave room,
|
||||
// so a budget whose peak exactly equals `budgetableBytes` is rejected — that
|
||||
// peak plus the reserved baseline is the whole address space, the RLIMIT_AS
|
||||
// edge. `ceil(budgetableBytes / MULTIPLE) - 1` is the last integer strictly
|
||||
// under `budgetableBytes / MULTIPLE`.
|
||||
const admissibleBudget = Math.ceil(budgetableBytes / OUTPUT_BUDGET_WORST_CASE_ADDRESS_SPACE_MULTIPLE) - 1
|
||||
for (const key of ['maxLogBytes', 'maxValueBytes'] as const) {
|
||||
if (this.config[key] * OUTPUT_BUDGET_WORST_CASE_ADDRESS_SPACE_MULTIPLE > budgetableBytes) {
|
||||
throw new Error(`dsh-code-runtime-python: config.${key} times the ${OUTPUT_BUDGET_WORST_CASE_ADDRESS_SPACE_MULTIPLE}x worst-case Unicode expansion must fit the ${budgetableBytes} bytes left after the ${INTERPRETER_BASELINE_BYTES}-byte interpreter baseline within the ${addressSpaceBytes}-byte addressSpaceMb, so a near-budget output truncates rather than breaching RLIMIT_AS as worker-exit; got ${String(this.config[key])} against a limit of ${admissibleBudget}`)
|
||||
if (this.config[key] * OUTPUT_BUDGET_WORST_CASE_ADDRESS_SPACE_MULTIPLE >= budgetableBytes) {
|
||||
throw new Error(`dsh-code-runtime-python: config.${key} times the ${OUTPUT_BUDGET_WORST_CASE_ADDRESS_SPACE_MULTIPLE}x worst-case Unicode expansion must fit within the ${budgetableBytes} bytes left after the ${INTERPRETER_BASELINE_BYTES}-byte interpreter baseline within the ${addressSpaceBytes}-byte addressSpaceMb, so a near-budget output truncates rather than breaching RLIMIT_AS as worker-exit; got ${String(this.config[key])} against a limit of ${admissibleBudget}`)
|
||||
}
|
||||
}
|
||||
ctx.effect(() => () => this.teardown(), 'python code-runtime teardown')
|
||||
|
||||
@@ -119,7 +119,7 @@ describe('PythonCodeRuntime — seam descriptors and misuse', () => {
|
||||
// The boundary value itself loads: the bound is the largest cap a frame can
|
||||
// still carry, not one below it. It needs an address space large enough to
|
||||
// clear the separate maxValueBytes/addressSpaceMb worst-case gate (the cap
|
||||
// times the 8x Unicode expansion must fit), so this pairs it with a 4 GiB
|
||||
// times the 12x Unicode expansion must fit), so this pairs it with a 4 GiB
|
||||
// addressSpaceMb — the two load-time bounds are independent.
|
||||
const boundary = await ctx.plugin(PythonCodeRuntime, { maxValueBytes: admissible, addressSpaceMb: 4096 })
|
||||
await boundary.dispose()
|
||||
@@ -418,11 +418,11 @@ describe('PythonCodeRuntime — inherited resource limits', () => {
|
||||
// never gets, so a near-budget output would OOM mid-run as an opaque
|
||||
// worker-exit. The bootstrap re-checks both budgets against the EFFECTIVE
|
||||
// clamped limit and fails loud at boot instead. A 128 MiB inherited limit
|
||||
// leaves 64 MiB budgetable (8 MiB admissible), under which a 32 MiB
|
||||
// maxLogBytes — admitted by the 512 MiB configured default — is rejected. The
|
||||
// rejection surfaces as an 'exception' (bootstrap's setrlimit-phase failure
|
||||
// class), not a mid-run OOM. The repro is Linux-only (macOS ignores
|
||||
// `ulimit -v`); there the run proceeds.
|
||||
// leaves 64 MiB budgetable (~5 MiB admissible under the 12x multiple), under
|
||||
// which a 32 MiB maxLogBytes — admitted by the 512 MiB configured default — is
|
||||
// rejected. The rejection surfaces as an 'exception' (bootstrap's
|
||||
// setrlimit-phase failure class), not a mid-run OOM. The repro is Linux-only
|
||||
// (macOS ignores `ulimit -v`); there the run proceeds.
|
||||
const dir = await mkdtemp(join(tmpdir(), 'dsh-rlimit-'))
|
||||
const wrapper = join(dir, 'python3-tight')
|
||||
await writeFile(wrapper, '#!/bin/sh\nulimit -v 131072\nexec python3 "$@"\n', { mode: 0o755 })
|
||||
@@ -783,21 +783,30 @@ describe('PythonCodeRuntime — programs and bindings', () => {
|
||||
// The child builds, charges, and encodes a `maxLogBytes` log entry or a
|
||||
// `maxValueBytes` completion value under RLIMIT_AS, and both trigger on
|
||||
// character count against a serialized-byte budget — an astral character is
|
||||
// one character but ~4 bytes stored and ~4 encoded, so a budget approaching
|
||||
// the address space lets a legitimate near-budget output breach it and die as
|
||||
// worker-exit. The incompatible pair is rejected at load: each budget times
|
||||
// the worst-case multiple (8) must fit the address space LEFT after the fixed
|
||||
// interpreter baseline. Against a 256 MiB address space that leaves 192 MiB
|
||||
// budgetable (24 MiB admissible), so a 50 MB cap is far over; the default caps
|
||||
// against 512 MiB are not. Both budgets are gated symmetrically — the value
|
||||
// case sets a default-fitting maxLogBytes so the maxValueBytes check is what
|
||||
// fires.
|
||||
// one character but ~4 bytes stored and ~4 encoded, and THREE such copies are
|
||||
// live at the peak (the caller's write argument, the slice/join handed to
|
||||
// push, and the encode copy), so a budget approaching the address space lets a
|
||||
// legitimate near-budget output breach it and die as worker-exit. The
|
||||
// incompatible pair is rejected at load: each budget times the worst-case
|
||||
// multiple (12) must fit the address space LEFT after the fixed interpreter
|
||||
// baseline. Against a 256 MiB address space that leaves 192 MiB budgetable
|
||||
// (~16 MiB admissible), so a 50 MB cap is far over; the default caps against
|
||||
// 512 MiB are not. Both budgets are gated symmetrically — the value case sets
|
||||
// a default-fitting maxLogBytes so the maxValueBytes check is what fires.
|
||||
const ctxLog = new Context()
|
||||
await expect(ctxLog.plugin(PythonCodeRuntime, { maxLogBytes: 50_000_000, addressSpaceMb: 256 }))
|
||||
.rejects.toThrow(/maxLogBytes times the 8x worst-case Unicode expansion must fit/)
|
||||
.rejects.toThrow(/maxLogBytes times the 12x worst-case Unicode expansion must fit/)
|
||||
const ctxValue = new Context()
|
||||
await expect(ctxValue.plugin(PythonCodeRuntime, { maxValueBytes: 50_000_000, addressSpaceMb: 256 }))
|
||||
.rejects.toThrow(/maxValueBytes times the 8x worst-case Unicode expansion must fit/)
|
||||
.rejects.toThrow(/maxValueBytes times the 12x worst-case Unicode expansion must fit/)
|
||||
// Discriminates 12 from 8: a 48 MiB maxLogBytes against a 512 MiB address
|
||||
// space leaves 448 MiB budgetable. 48*8 = 384 MiB fits (the old 8x multiple
|
||||
// wrongly ADMITTED this), but 48*12 = 576 MiB does not — and this is exactly
|
||||
// the config that OOMs, since a settlement flush holds the pending chunks,
|
||||
// their join, and the encode copy at once (~12x). The 12x gate rejects it.
|
||||
const ctxTwelve = new Context()
|
||||
await expect(ctxTwelve.plugin(PythonCodeRuntime, { maxLogBytes: 48 * 1024 * 1024, addressSpaceMb: 512 }))
|
||||
.rejects.toThrow(/maxLogBytes times the 12x worst-case Unicode expansion must fit/)
|
||||
// The default caps against the default 512 MiB address space load.
|
||||
const ok = new Context()
|
||||
const fiber = await ok.plugin(PythonCodeRuntime, { maxLogBytes: 65536, maxValueBytes: 32768, addressSpaceMb: 512 })
|
||||
@@ -3588,15 +3597,16 @@ describe('PythonCodeRuntime — hostile peer', () => {
|
||||
expect(serialized).toBeLessThan(1024)
|
||||
})
|
||||
|
||||
it('charges the serialized cost child-side, so a control-heavy line truncates instead of breaching the address space', async () => {
|
||||
it('charges the serialized cost child-side, so a control-heavy line truncates instead of being admitted whole', async () => {
|
||||
// The child's ledger must charge what the entry costs on the wire, not its
|
||||
// raw UTF-8 length: a NUL is one raw byte but six as its escape. A 24 MiB NUL
|
||||
// line clears the cheap char-count lower bound (24 MiB < 32 MiB budget), so
|
||||
// charging raw bytes would ADMIT it and then encode a ~144 MiB escaped
|
||||
// payload plus its UTF-8 copy — past the 384 MiB address space, killing the
|
||||
// child (surfaced host-side as `worker-exit`) instead of truncating.
|
||||
// Charging the serialized cost rejects it before any encode.
|
||||
const { runtime } = await setup({ maxLogBytes: 32 * 1024 * 1024, addressSpaceMb: 384, maxWallMs: 20_000 })
|
||||
// charging raw bytes would ADMIT it and emit a ~144 MiB escaped entry;
|
||||
// charging the serialized cost (~144 MiB > the 32 MiB budget) rejects it
|
||||
// before any encode and emits the marker instead. The address space (512 MiB,
|
||||
// clearing the 12x load gate for a 32 MiB budget) is sized so the run loads;
|
||||
// the gate separately guarantees a correctly-charged near-budget entry fits.
|
||||
const { runtime } = await setup({ maxLogBytes: 32 * 1024 * 1024, addressSpaceMb: 512, maxWallMs: 20_000 })
|
||||
const result = await runtime.run({
|
||||
program: [
|
||||
'print("\\x00" * (24 * 1024 * 1024))',
|
||||
|
||||
Reference in new issue
Block a user