Skip to content

Land benchmarks in their own PR before the change they measure - #11549

Merged
joseph-isaacs merged 3 commits into
developfrom
ji/agents-bench-then-impl
Nov 17, 2026
Merged

joseph-isaacs merged 3 commits into
developfrom
ji/agents-bench-then-impl

Conversation

@joseph-isaacs

@joseph-isaacs joseph-isaacs commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

A benchmark added in the same PR as the change it measures has no baseline, so CodSpeed reports it only as new. This tells agents to land the benchmark first and stack the change on it. It also records two pitfalls from #11548 and #11545: the benchmark PR must register its count in scripts/codspeed_bench_counts.json, and the stacked change must not be merged while its base is still the benchmark branch.

🤖 Generated with Claude Code

A benchmark added in the same pull request as the change it measures has no
baseline, so CodSpeed reports it only as new. Tell agents to split such work
into a benchmark pull request into `develop` and an implementation pull
request stacked on it.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UeniBE6jGwH9sRoqTmLZFz
@joseph-isaacs
joseph-isaacs enabled auto-merge (squash) November 17, 2026 16:59
joseph-isaacs and others added 2 commits November 17, 2026 17:02
Fold it into the Pull Requests section that develop added, and add what
landing #11548 and #11545 showed: the benchmark PR must also count its
benchmarks in `scripts/codspeed_bench_counts.json`, and merging the stacked
change while its base is still the benchmark branch folds it into the
benchmark PR.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UeniBE6jGwH9sRoqTmLZFz
@joseph-isaacs joseph-isaacs changed the title Land benchmarked changes as a benchmark pull request, then the change Land benchmarks in their own PR before the change they measure Nov 17, 2026
@codspeed

codspeed Bot commented Nov 17, 2026

Copy link
Copy Markdown

Merging this PR will improve performance by 10.86%

1 warning

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚡ 4 improved benchmarks
✅ 1961 untouched benchmarks
⏩ 53 skipped benchmarks1

Performance Changes

Mode Benchmark BASE HEAD Efficiency
⚡ Walltime add_i64_nullable_neon 20.3 µs 18.1 µs +12.24%
⚡ Walltime add_constant_shapes_neon[(32768, PerRowNullableConstant)] 18.2 µs 16.4 µs +10.64%
⚡ Walltime mul_u32_nonnull_neon 19.9 µs 18 µs +10.36%
⚡ Walltime decode_chunked_neon[i64, 524288] 15.2 µs 13.7 µs +10.21%

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.


Comparing ji/agents-bench-then-impl (c060196) with develop (3111483)

Open in CodSpeed

Footnotes

  1. 53 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

@joseph-isaacs
joseph-isaacs merged commit a57e53a into develop Nov 17, 2026
103 of 105 checks passed
@joseph-isaacs
joseph-isaacs deleted the ji/agents-bench-then-impl branch November 17, 2026 17:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants