[WaterrrForever]

Problem

Auto worker sizing capped capture workers at (V8 heap − 1024 MB) / 640 MB. In a 10 GB container Node's default heap is 2,096 MB, so every render captured with one worker on any number of cores.

The 640 MB figure came from one field OOM. Measured, the parent heap needs far less (below), so the heap was never the real limit.

Change (engine only)

• HEAP_PER_WORKER_MB 640 → 320, from measured peaks. In the 10 GB container with the default 2,096 MB heap that allows 3 workers, with no relaunch or second process. Every caller of computeWorkerSizing gets it, not only hyperframes render.
• Desktop app keeps 640 MB. Inside Electron the heap is a hard 4,192 MB cage, and that is where the field OOM behind the old figure happened (fix(engine): auto worker count stays within the heap budget #5149: 5 auto workers on a heavy export). The 320 figure is measured only outside the app, so the app stays at 640 (4 workers) until a heavy desktop export is measured. A packaged app runs the CLI on its own plain Node, so the engine knows the app by process.versions.electron or by the HYPERFRAMES_CLIENT=desktop/… tag it sets on every CLI call.
• CPU bound reads the container quota. getCgroupCpuLimit() reads cpu.max (cgroup v2) or cpu.cfs_quota_us / cpu.cfs_period_us (v1) beside the memory limit; with a quota, CPU-bound workers are min(cores − 2, floor(quota)). Without one, unchanged.

The relaunch, signal forwarding, child telemetry skip and HF_RENDER_HEAP_MB from the first version are gone.

Measured

Peak parent V8 heap from node --trace-gc (max heap-used before a GC), in the bench container: --memory 10g --cpus 3.5, default heap 2,096 MB, --workers N.

Film	1 worker	3 workers	4 workers	Per extra worker
ig03 (1080×1920, 36.7 s, video-heavy)	427 MB · 219 s	879 MB · 213 s	1,110 MB · 209 s	~228 MB
johnny (short, map)	79 MB · 137 s	122 MB · 99 s	—	~21 MB

320 MB keeps 1.4× over the heaviest measured slope, and the 1,024 MB reserve sits well above the measured base (~200 MB).

CPU, same container, johnny: 1 worker 138 s, 2 → 125 s, 3 → 96 s, 4 → 132 s, so floor(3.5) = 3 is the right bound there.

Not exercised: the field-OOM composition behind the old 640 figure (not available), and cgroup v1 on a real host (unit-tested only).

Tests

• systemMemory.test.ts: parseCgroupCpus (v2, v1, unlimited, malformed) and getCgroupCpuLimit (quota, max, non-Linux).
• parallelCoordinator.test.ts: the heap-bound case is now the container case (2,096 MB → 3 workers); a new case checks the desktop app's 4,192 MB heap still gives 4 inside Electron and 4 on plain Node tagged desktop/… (9 otherwise). Engine 88 pass with systemMemory.
• Producer render unit lane 1,117 pass. oxlint, oxfmt, typecheck, comment checks pass.

[github-actions]

Edit accuracy: accurate 271 (base branch 271), smooth 137 of those

The gate passes.
Smoothness is reported in the artifact, not gated. A case fails only if it fails 2 of 3 runs.

Quarantined, measured but not gated (0)

[jrusso1020]

Looks good. We checked the parts that could go wrong:

• Memory: getSystemTotalMb reads the cgroup limit, so the half-memory cap uses the container's 10 GB, not the host's.
• Signals: a terminal Ctrl-C reaches both parent and child, so the child sees SIGINT twice. That's safe because the cancellation abort only fires once. The parent's listeners also stop it from dying before the child exits, and they are removed on exit.
• Relaunch guard: HF_RENDER_HEAP_MB plus the hand-set check (both flag spellings, NODE_OPTIONS and execArgv) rule out a double relaunch. Telemetry and notices are skipped in the child.
• preHeapWorkers only adds a field. Existing sizing is unchanged.

Nit, fine as a follow-up: heapMbForAutoWorkers sizes for an unbounded frame count, so a short render that will only ever use 1 worker still relaunches in a small container. That costs one extra Node startup and produces the same output. To skip it, render.ts could only relaunch when the composition is long enough to get more than one worker, but that needs the frame count before the plan is built, so it isn't worth blocking on.

[miguel-heygen]

Requesting changes. The relaunch works around the per-worker heap figure instead of fixing it, and that figure is what is wrong.

Blocker: fix the budget, not the heap size. HEAP_PER_WORKER_MB = 640 (packages/engine/src/services/parallelCoordinator.ts:155) was inferred from one field OOM. The only measured figure in the same file is ~15-25 MB of parent heap per worker (APPLE_DISK_HEAP_PER_WORKER_MB, line 157). Your two runs already record the real number: please post peakHeapUsedMb from each run's perf summary next to its worker count. If it sits well under 2,096 MB at 2-3 workers, the heap was never the limit, and a per-worker figure derived from that data gives the same 2-3 workers in a 10 GB container with no second process. That fix also reaches every caller of computeWorkerSizing (the producer server and other embedders), while the relaunch only covers hyperframes render.

Blocker: the CPU bound reads the host, not the container. cpuBasedWorkers is cpus().length - 2 (lines 324, 338), which ignores a cgroup CPU quota, while memory already reads the cgroup limit (line 328). In a container started with --cpus 4 on a 32-core host, the CPU bound says 30. Today the heap clamp hides this. Lifting the heap, by relaunch or by a better figure, exposes it and can oversubscribe a CPU-limited container: that shows up as CFS throttling, not speed. Read cpu.max (cgroup v2) or cpu.cfs_quota_us / cpu.cfs_period_us (v1) next to the memory read, with a test for each.

With both fixed, I expect nothing in this PR is still needed: the relaunch, signal forwarding, child telemetry skip and HF_RENDER_HEAP_MB guard all exist to satisfy a budget that looks several times too high.

Important, only if the relaunch stays: heapMbForAutoWorkers always multiplies by HEAP_PER_WORKER_MB (line 431 on this branch), ignoring the Apple-disk figure the sizer itself uses, so the two disagree on that path. The branch also conflicts with main, and Comments is red.

The speedups in the body are real for that container. The point is to get them from the right bound, for every caller.

Verdict: REQUEST CHANGES
Reasoning: The worker cap is a wrong per-worker heap constant plus a CPU bound that ignores container quotas; fixing those two in the sizer is smaller, covers every caller, and makes the relaunch unnecessary.

[WaterrrForever]

@miguel-heygen thanks, you were right on both. Reworked at 55a40c0d:

1. Budget, not heap size. Measured peak parent heap with node --trace-gc in the 10 GB / 3.5-CPU container, default 2,096 MB heap: ig03 427 / 879 / 1,110 MB at 1 / 3 / 4 workers (~228 MB per extra worker), johnny 79 → 122 MB at 1 → 3 (~21 MB). The heap was never the limit. HEAP_PER_WORKER_MB is now 320, which gives 3 workers there with no second process. The relaunch, signal forwarding, child telemetry skip and HF_RENDER_HEAP_MB are all removed.
2. CPU quota. getCgroupCpuLimit() reads cpu.max (v2) or cfs quota/period (v1) next to the memory read; with a quota the CPU bound is min(cores − 2, floor(quota)). In the same container, johnny ran fastest at 3 workers (96 s, against 125 s at 2 and 132 s at 4), which matches floor(3.5). Tests cover each format.

The Apple-disk mismatch disappears with heapMbForAutoWorkers. Merged main; Comments should pass now. The old 640 figure's field OOM composition isn't something I could rerun, so that's the one case not measured.

[jrusso1020]

Re-reviewed at 55a40c0. This version is simpler and we like it better than the relaunch.

• parseCgroupCpus / getCgroupCpuLimit: v2 cpu.max first, then v1 quota over period, and max, -1, zero and malformed values all come back as null. A fractional quota under 1 floors to 0 and Math.max(1, ...) keeps one worker. Good.
• min(cores - 2, floor(quota)) only ever lowers the CPU bound, so hosts without a quota are unchanged.
• The 3.5 CPU bench (3 workers beat 2 and 4) backs the floor.

One thing to watch, not blocking. Going from 640 to 320 also lifts the heap bound on bigger hosts. With a ~4 GB default heap it goes from 4 workers to 9, and on a 48+ core box auto can pick 6 again. 6 workers on a ~4 GB heap is the field OOM the 640 figure came from. Your ig03 slope says 6 workers need about 1.6 GB, so that film fits fine. The field composition used over 500 MB per worker, though, and the PR notes it wasn't re-run. If it comes back, a cheap guard is to stop adding workers when heap_used passes about 75% of heap_size_limit during the first chunk, or to log heapBasedWorkers and peak heap per render so we can see it. Fine as a follow-up.

— Rames

[miguel-heygen]

Re-review at 55a40c0d: this is the right fix. The relaunch is gone, the per-worker figure now comes from a measurement, and the CPU bound reads the container's quota.

Strengths

• systemMemory.ts: parseCgroupCpus handles v2 cpu.max (including max) and v1 quota/period, and returns null when there's no quota, so hosts without one are unchanged. Tests cover each format.
• parallelCoordinator.ts: HEAP_PER_WORKER_MB = 320 cites the measurement it came from (427 to 1,110 MB going from 1 to 4 workers on the heavy film).

Important, fine as a follow-up: two other terms in the same sizer still read the host's cores instead of the quota:

• the contention clamp (cpuScaledMax = floor(cpuCount / weightedCoresPerWorker), line 419);
• the auto max (defaultSafeMaxWorkers, line 175).

With a 6-CPU quota on a 32-core host, the clamp sees 32 cores. Routing them through the quota would change worker counts on CPU-limited hosts, so that change should come with its own measurement. We're benchmarking exactly that on 6-CPU containers now and will follow up in a separate PR that builds on this one.

Nit: the 640 figure came from a field OOM that couldn't be rerun, as you noted. The workers_heap_* telemetry will show whether 320 holds across real compositions.

Verdict: APPROVE
Reasoning: Both earlier blockers are fixed at the root, with measurements and tests; what's left is making the rest of the sizer use the quota, which needs its own data.

[jerrai-bot-heygen]

The desktop client tag reaches the packaged plain-Node render and preserves the prior 640 MB worker allowance; the new test covers that sizing branch. The original composition and the vendored Node heap remain unmeasured. — Jerrai