Skip to content

ci: add focused PR benchmark labels - #9018

Merged
connortsui20 merged 5 commits into
developfrom
ct/benchmark-labels
Aug 5, 2026
Merged

connortsui20 merged 5 commits into
developfrom
ct/benchmark-labels

Conversation

@connortsui20

@connortsui20 connortsui20 commented Jul 27, 2026 •

Copy link
Copy Markdown
Member

Adds focused PR benchmark triggers:

  • action/bench-random-access
  • action/bench-compress
  • action/bench-string (benchmarks: add string compression benchmark #9060)
  • action/bench-sql (excludes Vortex Compact)
  • action/bench-sql-compact (Vortex Compact plus Parquet controls)
  • action/bench-all (random access, compression, string encoding, and SQL, while GPU remains separate)

Each trigger dispatches a role-specific reusable workflow. PR benchmarks check out github.event.pull_request.head.sha. SQL coverage stays in the Python catalog and is resolved into a matrix at runtime. The all-CPU path uses a single pr-all preset (the exact union of pr and pr-compact), so shared SQL jobs run once instead of twice.

Benchmark reporting now:

  • Keeps the latest rerun for each logical timing or file-size result.
  • Includes Parquet controls in pr-compact to distinguish code changes from runner drift.
  • Namespaces remote datasets by preset to prevent concurrent jobs from clobbering each other.

TODO: After merge, delete the legacy action/benchmark* labels once develop uses the new dispatcher.

@connortsui20
connortsui20 force-pushed the ct/benchmark-labels branch from 89eebfa to 77a7e5d Compare July 27, 2026 16:42
@connortsui20
connortsui20 requested review from AdamGS and myrrc July 27, 2026 17:39
@connortsui20

Copy link
Copy Markdown
Member Author

note that we probably dont want all of these labels, but I feel like a bit more than what we have on develop now would be nice?

@connortsui20

Copy link
Copy Markdown
Member Author

actually Im going to make them all disjoint, lets see how that works

@connortsui20
connortsui20 force-pushed the ct/benchmark-labels branch from 77a7e5d to 9cbc6da Compare July 27, 2026 17:49
Base automatically changed from ct/declarative-bench to develop July 27, 2026 18:00
@connortsui20
connortsui20 force-pushed the ct/benchmark-labels branch 2 times, most recently from 2d3d02c to c5173aa Compare July 27, 2026 18:06
@connortsui20
connortsui20 marked this pull request as ready for review July 27, 2026 18:36
@connortsui20
connortsui20 requested a review from onursatici July 27, 2026 18:37
@codspeed

codspeed Bot commented Jul 27, 2026 •

Copy link
Copy Markdown

Merging this PR will improve performance by 47.71%

⚡ 1 improved benchmark
✅ 1829 untouched benchmarks
⏩ 43 skipped benchmarks1

Performance Changes

Mode Benchmark BASE HEAD Efficiency
⚡ Simulation decompress[u64, (10000, 1024)] 175 µs 118.4 µs +47.71%

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.


Comparing ct/benchmark-labels (75556e6) with develop (3884b98)

Open in CodSpeed

Footnotes

  1. 43 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

myrrc
myrrc previously approved these changes Jul 28, 2026
@myrrc

myrrc commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Let's use bench-random-access, ra isn't straight obvious

@connortsui20
connortsui20 force-pushed the ct/benchmark-labels branch from c5173aa to 0b4eb99 Compare July 28, 2026 14:01
@connortsui20
connortsui20 requested a review from myrrc July 28, 2026 14:03
@connortsui20
connortsui20 force-pushed the ct/benchmark-labels branch from 0b4eb99 to dac3fb7 Compare July 28, 2026 14:34
@connortsui20
connortsui20 requested review from 0ax1 and removed request for onursatici July 28, 2026 14:34

@0ax1 0ax1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

couple of q's

# v4 (Postgres) ingest -- REQUIRED (see develop-bench.yml rationale). Empty records:
# post-ingest.py --postgres upserts the commit row only. Gated on the
# ingest-role ARN var (the assume-role input that MUST exist for OIDC to
# succeed).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably need a action/bench-gpu-compress label for GPU compress?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah I am going to run this once this merges:

gh label edit "action/benchmark-gpu-compress" --name "action/bench-gpu-compress" \
  --description "Run only the GPU compression benchmark on this PR"
gh label delete action/benchmark --yes
gh label delete action/benchmark-sql --yes
gh label delete action/benchmark-sql-full --yes

"pr-compact": COMPACT,
"pr-full": FULL_PR,
"develop": FULL_LOCAL,
},

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is there any label reaching pr full?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no since I think we should just keep everything disjoint, if someone wants to run more benchmarks I think they should just add all 4 (or 5 for gpu) labels.

@connortsui20
connortsui20 force-pushed the ct/benchmark-labels branch 2 times, most recently from 4bbe37b to fb5f8aa Compare July 28, 2026 15:29
cat comment.md >> "$GITHUB_STEP_SUMMARY"
- name: Comment PR
if: github.event.pull_request.head.repo.fork == false
if: github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork == false

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think this is a qol change

@connortsui20
connortsui20 requested a review from 0ax1 July 28, 2026 15:31
@connortsui20
connortsui20 force-pushed the ct/benchmark-labels branch from fb5f8aa to e332145 Compare July 28, 2026 16:45
@connortsui20 connortsui20 added action/bench-random-access Run only the random-access benchmark on this PR action/bench-sql-compact Run only Vortex Compact SQL benchmarks on this PR labels Jul 29, 2026
@github-actions github-actions Bot removed action/bench-sql-compact Run only Vortex Compact SQL benchmarks on this PR action/bench-random-access Run only the random-access benchmark on this PR labels Jul 29, 2026
@myrrc
myrrc removed their request for review July 31, 2026 15:00
@connortsui20
connortsui20 force-pushed the ct/benchmark-labels branch from 1247c00 to b1b5ca5 Compare July 31, 2026 17:24
@0ax1
0ax1 removed their request for review August 3, 2026 13:22
@connortsui20
connortsui20 requested a review from myrrc August 3, 2026 14:48
Comment thread .github/workflows/sql-bench-matrix.yml Resolved

@joseph-isaacs joseph-isaacs left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we just move some stuff around?

@connortsui20

Copy link
Copy Markdown
Member Author

ok codex is finding other issues that are tangentially related to this (not caused by this PR) but I think we might as well fix them now

@connortsui20

Copy link
Copy Markdown
Member Author

@AdamGS there are a few more changes in here now that you might want to take a look at, happy to try and split out the changes (though that will be kind of painful)

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
@@ -0,0 +1,20 @@
# Runs the string encoding benchmark for a pull request.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is this one different from random-access and compression?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It feels strange that we would put random access, compression, and string encoding all in the same bucket? I also mainly wanted to split out compression because its by far the slowest benchmark we have

@connortsui20 connortsui20 added action/bench-random-access Run only the random-access benchmark on this PR action/bench-sql-compact Run only Vortex Compact SQL benchmarks on this PR action/bench-string Run only the string encoding benchmark on this PR labels Aug 5, 2026
@github-actions github-actions Bot removed action/bench-sql-compact Run only Vortex Compact SQL benchmarks on this PR action/bench-random-access Run only the random-access benchmark on this PR action/bench-string Run only the string encoding benchmark on this PR labels Aug 5, 2026

@AdamGS AdamGS left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just one question re the new strings benchmark.

overall LGTM, once this merges, can you send a message in the slack channel so its more visible?

@github-actions

github-actions Bot commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Benchmarks: String Encoding 📖


vortex / vortex-file-compressed / ms (1.010x ➖, 0↑ 0↓)
name PR 75556e6 (ms) base 3884b98 (ms) ratio (PR/base)
read/clickbench/URL/shard-0/fsst 47.862028 48.348287 0.99
read/clickbench/URL/shard-0/onpair-12 37.137094 37.459657 0.99
read/tpch/l_comment/fsst 94.750145 95.006134 1.00
read/tpch/l_comment/onpair-12 105.449846 107.052801 0.99
write/clickbench/URL/shard-0/fsst 916.63642 873.941068 1.05
write/clickbench/URL/shard-0/onpair-12 2102.91248 2111.65836 1.00
write/tpch/l_comment/fsst 3437.39275 3169.16642 1.08
write/tpch/l_comment/onpair-12 3948.73565 3972.97392 0.99
vortex / vortex-file-compressed / % (1.000x ➖, 0↑ 0↓)
name PR 75556e6 (%) base 3884b98 (%) ratio (PR/base)
size/clickbench/URL/shard-0/fsst 47.4688631 47.4688631 1.00
size/clickbench/URL/shard-0/onpair-12 29.7509552 29.7509552 1.00
size/tpch/l_comment/fsst 30.5060886 30.5060886 1.00
size/tpch/l_comment/onpair-12 23.2389905 23.2389905 1.00

@connortsui20
connortsui20 dismissed joseph-isaacs’s stale review August 5, 2026 16:12

everything resolved

@connortsui20
connortsui20 merged commit 69661e0 into develop Aug 5, 2026
163 of 165 checks passed
@connortsui20
connortsui20 deleted the ct/benchmark-labels branch August 5, 2026 16:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants