[Muzzy5150]

What

Give each hyperframes benchmark invocation its own OS temporary render directory and remove those render files after their measurements are collected.

Why

Benchmark render files are implementation details, but separate invocations currently share deterministic paths under renders/.benchmark. Concurrent benchmarks or benchmarks launched for different projects from the same working directory can overwrite one another, and completed runs leave generated media behind.

Root cause

The command resolves one fixed renders/.benchmark directory from the caller's working directory and derives each output filename only from the benchmark configuration and run index.

Related work

No related issue or overlapping PR was found. This is a small self-contained correction, so no separate issue was created.

How

• Create a unique hyperframes-benchmark-* directory under the operating system temporary directory for each command invocation.
• Keep the existing render-size measurement and result calculation behavior.
• Remove the invocation's render directory after all benchmark results have been collected.
• Add regression coverage in both JSON and human-output modes.

Before and after

Before:

• Two benchmark invocations used the same directory and deterministic filenames.
• Completed benchmark media remained under renders/.benchmark.

After:

• Each invocation receives a different output directory, preventing cross-process filename collisions.
• Temporary renders are removed after their sizes are measured.
• Reported benchmark results and output formats are unchanged.

Test plan

• Unit tests added/updated
• Manual testing performed
• Documentation updated (not applicable: the temporary paths are not public output)
• Comments follow CONTRIBUTING.md "Comments": they say why, not what, and a bug fix says what the code must do and how to reproduce the bug

Regression tests:

• Baseline with the new regression assertions: 10 tests passed and 2 failed because both invocations shared the same directory.
• bun run --cwd packages/cli test src/commands/benchmark.test.ts: 1 file passed, 12 tests passed.
• bun run --cwd packages/cli typecheck: passed.
• bun run build: passed on current main.
• bun run lint: passed with 0 warnings and 0 errors.
• COMMENT_CHECK_BASE=upstream/main node scripts/check-comment-citations.mjs: passed, 2 files examined.
• COMMENT_CHECK_BASE=upstream/main node scripts/comment-ratchet.mjs: passed, 1 changed source file.

A full CLI test run was also attempted locally: 5,923 tests passed, 105 skipped, and 8 unrelated environment-sensitive tests failed. An untouched upstream worktree reproduced seven stable failures involving macOS /var path canonicalization, an installed system whisper-cli, unavailable WebGL contexts, and Node deprecation output. One filesystem-watcher failure and one proxy teardown ECONNRESET were intermittent under the concurrent full run.

Compatibility

This uses Node's cross-platform tmpdir, mkdtempSync, and rmSync APIs. Result calculation, JSON shape, render configurations, and user-selected project paths are unchanged. A force-killed process can still leave its unique directory for normal OS temporary-file cleanup, but it cannot collide with another benchmark invocation.

[miguel-heygen]

Thanks for this. The fix is small and lands in the right place: the output directory comes from a single line, and it now uses the same mkdtempSync(join(tmpdir(), ...)) pattern as compare and grade-compare.

What I checked:

• On main, benchmark.ts resolves renders/.benchmark from the working directory, and output names only depend on the config label and run index, so concurrent runs overwrite each other and the files are never removed. Nothing else in the repo reads that directory, so moving it doesn't break anything.
• The producer creates its work directory next to the output path and removes it itself, so its intermediate files also land in the per-run temp directory.
• Sizes are measured inside the loop, before the directory is removed, so the reported results don't change.
• I reverted each half of the fix and reran the tests. Going back to a fixed directory fails the new test ("not to be renders/.benchmark"), and dropping the rmSync fails it too. On the PR head the file passes 12/12, three runs in a row.

Two small, non-blocking suggestions:

• nit: the temp directory is created before loadProducer() (benchmark.ts:75 vs :79-94). If the producer fails to load, failCommand() throws before the rmSync at :163, which leaves an empty hyperframes-benchmark-* directory behind. Creating the directory after the producer loads, or wrapping the loop in try/finally like compare.ts does, would cover that.
• nit: every render is kept until all configs finish, so with --runs 20 there are 100 files in the temp directory at once, and on systems where /tmp is in memory that adds up. Deleting each file right after its statSync would keep only one at a time. Happy either way.

CI has finished green at this head.

Verdict: APPROVE
Reasoning: Correct root-cause fix with tests that fail without it, and all checks pass at this head.