Skip to content

Benchmark TypeScript prototype against Rust - #1852

Merged
JacobStephens2 merged 4 commits into
issue-1831from
issue-1840
Nov 7, 2026
Merged

JacobStephens2 merged 4 commits into
issue-1831from
issue-1840

Conversation

@JacobStephens2

@JacobStephens2 JacobStephens2 commented Nov 6, 2026 •

Copy link
Copy Markdown
Owner

Summary

prototypes/typescript/
├── benchmark.ts                 fresh committed Worktree; paired cold/repeat measurements
└── tests/benchmark.test.ts       CLI execution/failure cleanup and report completeness
docs/research/
└── typescript-vs-rust.{md,json}  empirical comparison and recorded commands/output/inventories

Measures typechecking, slice tests, standalone compilation, executable bytes,
source inventories and ADR-0017 model execution. The report compares subprocess
cleanup, exhaustiveness and shared Console contracts, and states the workload
and cache limits. ADR-0001 is unchanged.

Closes #1840

Evidence

Execution evidence

  • Before: bun test --cwd prototypes/typescript tests/benchmark.test.ts
    failed with ENOENT: no such file or directory for
    docs/research/typescript-vs-rust.md (1 pass, 1 fail). The initial CLI test
    also failed with Module not found for benchmark.ts (0 pass, 1 fail).
    After: the same command passes (2 pass, 0 fail, 125 expect() calls).
  • bun prototypes/typescript/benchmark.ts --ref 8700be5d --output /tmp/issue-1840-benchmark-final.json
    completed all 16 measurements. TypeScript's bundle passed 54 tests; Rust's
    selected bundle passed 183 with three ignored. The committed JSON retains
    actual command output, tool/host metadata and per-file line counts.
  • bun run --cwd prototypes/typescript typecheck passed.
  • With Python 3.12.12 selected as python3,
    scripts/affected-tests 01ecf28cba640c766ce2bfa008fdbc63fbe7c163 passed
    typechecking (5.02s) and all prototype tests (56 passed, 443 assertions,
    99.70s). Its full Rust selection stopped at the unchanged Console cache test
    (unchanged request reread 156409 bytes): 1657 unit tests passed, and Console
    had 235 passed, one failed and one ignored. Rust wall time was 851.50s with
    cached compilation. The earlier Python tomllib setup failure was corrected;
    the affected Bug origins regression also passed in isolation and in this run.

Seams under test: the benchmark CLI's JSON/log output, failure propagation and
Worktree cleanup; the published report and its recorded measurement artifact.

The unchanged Rust Console baseline also fails in isolation:
cargo test --locked --test console idle::unchanged_requests_reuse_retained_records_but_replacements_are_visible -- --exact --nocapture
reports unchanged request reread 156041 bytes against a 48 KiB limit. The
237-test Console diagnostic had 235 passed, one failed and one ignored.
Production code, Rust tests, Cargo files and embedded skills have no diff from
the pinned Base branch. The report explicitly discloses this limitation.

Review coverage

Independent Standards and Spec reviews against issue-1831 at 01ecf28c:
0 Standards findings; 0 Spec findings. Each axis read all six changed files,
including the complete raw JSON data. Generated-output exclusions: none.
Unread changed files: none.

Unaddressed findings

Standards

None.

Spec

None.

Security

None.

Security review outcome

No unaddressed introduced findings.

Merge Danger

Door: two-way

Blast Radius: tooling

The change adds prototype benchmarking and research evidence; reverting it
removes those artifacts without a data migration.

Built with codex codex-cli 0.160.0 · gpt-6.1-sol · high

CI notes

  • console-test run 42314231567 intermittently timed out at console/tests/inbox.test.cjs:1584 waiting for the live CI red badge; the test, Console sources, Cargo dependencies and CI job have no diff between head 0bfaa8b0 and base issue-1831 (01ecf28c), the focused test passes on both checkouts, and cargo build --locked plus npm test --prefix console passes all 150 browser tests locally without changes.
  • scripts/affected-tests origin/issue-1831 passes prototype typechecking and all 56 prototype tests (443 assertions), then stops at the previously documented, unchanged Rust Console cache assertion: idle::unchanged_requests_reuse_retained_records_but_replacements_are_visible, unchanged request reread 157557 bytes (tests/console/idle.rs:43); 1,657 Rust unit tests pass, and the Console integration result is 235 passed, one failed, one ignored. src/, Rust tests and Cargo files are identical to 01ecf28c; this is unrelated to the benchmark/research diff.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant