Repository navigation
Store SQL benchmark file sizes in the benchmarks database - #11443
Conversation
PR benchmark comments lost their file-size section when the baseline moved from S3 to the benchmarks Postgres, because file sizes were never ingested. `capture-file-sizes.py` now also writes `file_size` ingest records, which the SQL bench job appends to `results.ingest.jsonl`. Develop runs upsert them into the new `file_sizes` table, and PR runs fetch them back as the baseline, one lookup per (benchmark, scale factor, format) so files a PR stops writing still show up. The size-change column now prints the minus sign on shrinking files. Requires the `file_sizes` table (vortex.dev `db/migrations/014_file_sizes.sql`) to be applied before this merges: the develop ingest is required and writes each file in one transaction. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniB9quRiv5fCx8AhzH9s Signed-off-by: Claude <noreply@anthropic.com>
- Choose the baseline commit from the measurements, not the file-size scopes. A scope has one row per file per commit, so paging back through it took seconds once a PR was a hundred commits behind develop. - Skip a kind whose table the database does not have yet, so a missing `file_sizes` table costs the file sizes and not the whole baseline. - Stop crashing when the baseline holds only file sizes: the empty benchmark frame now keeps the columns the comparison reads. - Say "No baseline file sizes found" instead of silently dropping the section. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniB9quRiv5fCx8AhzH9s Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (0.995x ➖, 1↑ 0↓)
No baseline file sizes found for base commit. |
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.979x ➖, 1↑ 0↓)
datafusion / parquet / ns (0.995x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.991x ➖, 1↑ 0↓)
duckdb / parquet / ns (1.016x ➖, 0↑ 0↓)
No baseline file sizes found for base commit. |
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.004x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.003x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.998x ➖, 0↑ 0↓)
No baseline file sizes found for base commit. |
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.000x ➖, 0↑ 1↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.999x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.004x ➖, 0↑ 0↓)
No baseline file sizes found for base commit. |
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.002x ➖, 0↑ 1↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.006x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.009x ➖, 0↑ 0↓)
No baseline file sizes found for base commit. |
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.995x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.991x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.000x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.998x ➖, 0↑ 0↓)
No baseline file sizes found for base commit. |
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.107x ➖, 0↑ 3↓)
datafusion / parquet / ns (0.993x ➖, 1↑ 1↓)
duckdb / vortex-file-compressed / ns (1.032x ➖, 1↑ 1↓)
duckdb / parquet / ns (0.973x ➖, 1↑ 0↓)
|
Benchmarks: TPC-DS SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.996x ➖, 1↑ 0↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.000x ➖, 0↑ 1↓)
duckdb / parquet / ns (0.997x ➖, 0↑ 0↓)
No baseline file sizes found for base commit. |
develop (#11443, #11444) made "Capture file sizes" append file-size records to results.ingest.jsonl, which "Fetch baseline" reads, and added the sampled file tree dump and upload. The benchmark job now runs the capture alongside the dump, then the tree upload alongside the baseline fetch, so the fetch still sees the file-size records. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBAMeLpyH1yASJhrK1N Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
## Summary #11443 started storing SQL benchmark file sizes in the benchmarks database, but only for `vortex-file-compressed` and `vortex-compact`. This PR stores every on-disk format, so bench can chart Parquet, DuckDB, Lance and the rest beside Vortex (vortex-data/vortex.dev#222). PR comments still compare only the Vortex formats. ## Changes - `scripts/capture-file-sizes.py` records every format directory that `Format::name()` in `vortex-bench/src/lib.rs` names: `arrow-ipc`, `csv`, `duckdb`, `lance`, `litdata`, `parquet`, `vortex-compact`, `vortex-file-compressed` and `vortex-spatial-native`. Each develop commit adds roughly one row per Parquet file; ClickBench has about 100. - **DuckDB catalogs:** DuckDB keeps a catalog of views (`duckdb.db`, and `duckdb.db.wal` while open) in every format directory it queries (`duckdb-bench/src/lib.rs`). That isn't the format's data, so it's now skipped everywhere except the `duckdb` directory, where the database is the data. This also takes the catalog out of the Vortex totals. Rows recorded today include it, so PR comments based on today's develop commits list `duckdb.db` as a removed file, once. - `scripts/compare-benchmark-jsons.py`: the "File Size Changes" section compares only formats starting with `vortex`. The other formats' files come from the data generators and don't change with a Vortex PR. New tests check that capture records every format, skips the DuckDB catalogs outside `duckdb`, and ignores unknown directories. Another checks that the report leaves out non-Vortex formats. ### Checks - `uv run --no-project --with pytest --with pandas --with numpy --with orjson --with tabulate --with xxhash pytest scripts/tests/test_measurement_id.py scripts/tests/test_benchmark_reporting.py scripts/tests/test_post_ingest.py`: 49 passed. - `uvx ruff format --check` and `uvx ruff check` on the changed files pass, as does `git diff --check`. - Not run: a CI benchmark run. `action/bench-sql` on this PR would exercise capture and the comment. The new formats only reach the database from develop runs. --- _Generated by [Claude Code](https://claude.ai/code/session_01UeniB9quRiv5fCx8AhzH9s)_ Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Summary
The file-size section disappeared from SQL benchmark PR comments in #10987, which moved the baseline from S3 to the benchmarks Postgres. File sizes were never stored there, so
compare-benchmark-jsons.pyfound no baseline rows and dropped the section without a message. This PR stores the sizes on develop runs and reads them back on PR runs.