Repository navigation
Add a TSBS cpu-only time-series benchmark suite - #10218
Closed
joseph-isaacs wants to merge 3 commits into
Closed
joseph-isaacs wants to merge 3 commits into
joseph-isaacs wants to merge 3 commits into
Conversation
Add `tsbs`, which runs the cpu-only use case of the Time Series Benchmark Suite at its README configuration: 4,000 hosts over three days, 103,680,000 rows of 10-second CPU metrics. The data is generated with `tsbs_generate_data` at a pinned commit via `go run`, then parsed and written as zstd level 3 Parquet. The 15 DevOps queries were generated with `tsbs_generate_queries` at the same commit and seed, and rewritten from TimescaleDB SQL so the same text runs on DataFusion and DuckDB. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0168W9zatdzbCo2A9kw7sJLJ Signed-off-by: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0168W9zatdzbCo2A9kw7sJLJ Signed-off-by: Claude <noreply@anthropic.com>
Add TSBS to the `pr-full` preset alone, and install Go for its matrix entry, since its data is generated with `go run`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0168W9zatdzbCo2A9kw7sJLJ Signed-off-by: Claude <noreply@anthropic.com>
Contributor
|
This PR has been marked as stale because it has been open for 14 days with no activity. Please comment or remove the stale label if you wish to keep it active, otherwise it will be closed in 7 days |
This was referenced Oct 20, 2026
Contributor
Author
Contributor
|
This PR has been marked as stale because it has been open for 14 days with no activity. Please comment or remove the stale label if you wish to keep it active, otherwise it will be closed in 7 days |
Contributor
|
This PR was closed because it has been inactive for 7 days since being marked as stale. |
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
tsbs, which runs thecpu-onlyuse case of the Time Series Benchmark Suite (TSBS). TSBS is the benchmark most time-series databases publish results on. The suite runs only under theaction/bench-sql-extendedlabel. Stacked on #10217.The data uses the configuration from the TSBS README: 4,000 hosts over three days with one reading every 10 seconds, which is 103,680,000 rows. Each row has ten text tags and ten CPU metrics stored as doubles. The data is synthetic, as it is in every published time-series benchmark.
Changes
vortex-bench/src/tsbs: runstsbs_generate_dataat a pinned commit throughgo run, parses its output and writes zstd level 3 Parquet. It checks the row count and the generator's header lines. Columns are typed the way TSBS's TimescaleDB loader creates them.vortex-bench/sql/tsbs.sql: one query of each of the 15 DevOps query types. They were produced bytsbs_generate_queriesat the same commit, seed and scale, in its TimescaleDB format without a tags table. They were then changed so the same text runs on DataFusion and DuckDB:time_bucketbecamedate_trunc, which gives the same 1-minute and 1-hour buckets.+0000offset becameTIMESTAMPliterals.timecolumn is quoted.vortex-bench/sql/tsbs.md: explains the suite and lists those changes.bench-orchestrator/README.mdand the bench-performance skill now name both time-series suites.bench-orchestrator/.../catalog.py: addstsbs-nvmeto thepr-fullpreset only, running DataFusion and DuckDB on Parquet and Vortex..github/workflows/sql-bench-matrix.yml: installs Go throughactions/setup-gov7.0.0, pinned by commit hash, only for the TSBS matrix entry.Checks run:
cargo clippy -p vortex-bench --all-targets --all-features -- -D warningspasses.cargo test -p vortex-benchpasses, including new tests for the generator-output parser, header validation and the query file.cargo +nightly-2026-09-10 fmt -p vortex-bench -- --checkis clean.ruffandyamllint --strictare clean.vx-bench matrixliststsbs-nvmeonly forpr-full.data-gen tsbs --formats parquet,vortexbuilt the dataset in about 7 minutes: 984 MB as Parquet and 1.7 GB as Vortex.datafusion-bench tsbs --formats parquet,vortexran all 15 queries.Two things for reviewers:
lastpointis slow. It takes about a minute per run per target, because DataFusion sorts every row forDISTINCT ON. Its iterations could be reduced if the extended run takes too long.The Go setup step has not run on the bench runners yet. The first extended run will exercise it.
🤖 Generated with Claude Code
https://claude.ai/code/session_0168W9zatdzbCo2A9kw7sJLJ
Generated by Claude Code