Repository navigation
Reorganize the compressor - #8745
Conversation
|
actually I want to move some of the logic out of mod.rs into a cascade.rs file |
Polar Signals Profiling ResultsLatest Run
Previous Runs (1)
Powered by Polar Signals Cloud |
Benchmarks: PolarSignals ProfilingVortex (geomean): 0.980x ➖ How to read Verdict and Engines
datafusion / vortex-file-compressed (0.980x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: FineWeb NVMeVerdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (1.059x ➖, 0↑ 4↓)
datafusion / vortex-compact (1.014x ➖, 0↑ 2↓)
datafusion / parquet (0.945x ➖, 1↑ 0↓)
duckdb / vortex-file-compressed (0.967x ➖, 0↑ 1↓)
duckdb / vortex-compact (0.975x ➖, 0↑ 0↓)
duckdb / parquet (0.998x ➖, 0↑ 0↓)
File Size Changes (1 files changed, +0.0% overall, 1↑ 0↓)
Totals:
|
Benchmarks: TPC-H SF=1 on NVMEVerdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (1.001x ➖, 0↑ 0↓)
datafusion / vortex-compact (0.920x ➖, 9↑ 0↓)
datafusion / parquet (0.975x ➖, 2↑ 0↓)
datafusion / arrow (0.916x ➖, 9↑ 0↓)
duckdb / vortex-file-compressed (0.948x ➖, 4↑ 0↓)
duckdb / vortex-compact (0.980x ➖, 1↑ 0↓)
duckdb / parquet (1.020x ➖, 0↑ 4↓)
duckdb / duckdb (0.978x ➖, 0↑ 0↓)
File Size Changes (9 files changed, -0.2% overall, 1↑ 8↓)
Totals:
|
Benchmarks: TPC-DS SF=1 on NVMEVerdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (0.960x ➖, 3↑ 1↓)
datafusion / vortex-compact (0.960x ➖, 1↑ 0↓)
datafusion / parquet (0.954x ➖, 8↑ 2↓)
duckdb / vortex-file-compressed (0.939x ➖, 14↑ 0↓)
duckdb / vortex-compact (0.963x ➖, 4↑ 0↓)
duckdb / parquet (0.971x ➖, 1↑ 0↓)
duckdb / duckdb (0.963x ➖, 6↑ 1↓)
File Size Changes (7 files changed, -0.0% overall, 2↑ 5↓)
Totals:
|
751ef7c to
37463ff
Compare
Benchmarks: FineWeb S3Verdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (0.956x ➖, 1↑ 0↓)
datafusion / vortex-compact (1.080x ➖, 0↑ 1↓)
datafusion / parquet (0.937x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed (0.921x ➖, 0↑ 0↓)
duckdb / vortex-compact (0.910x ➖, 0↑ 0↓)
duckdb / parquet (0.915x ➖, 0↑ 0↓)
|
Benchmarks: Clickbench Sorted on NVMEVerdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (1.086x ➖, 0↑ 4↓)
datafusion / parquet (0.982x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed (1.216x ❌, 0↑ 8↓)
duckdb / parquet (1.106x ❌, 0↑ 6↓)
duckdb / duckdb (0.991x ➖, 0↑ 0↓)
File Size Changes (201 files changed, +0.0% overall, 104↑ 97↓)
Totals:
|
Merging this PR will improve performance by 10.98%
Performance Changes
Tip Curious why this is faster? Comment Comparing Footnotes
|
Benchmarks: Statistical and Population GeneticsVerdict: No clear signal (low confidence) How to read Verdict and Engines
duckdb / vortex-file-compressed (1.009x ➖, 0↑ 0↓)
duckdb / vortex-compact (1.013x ➖, 0↑ 0↓)
duckdb / parquet (0.995x ➖, 0↑ 0↓)
File Size Changes (1 files changed, -0.0% overall, 0↑ 1↓)
Totals:
|
Benchmarks: TPC-H SF=10 on NVMEVerdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (1.012x ➖, 0↑ 0↓)
datafusion / vortex-compact (0.996x ➖, 0↑ 0↓)
datafusion / parquet (1.009x ➖, 0↑ 0↓)
datafusion / arrow (1.009x ➖, 1↑ 0↓)
duckdb / vortex-file-compressed (0.994x ➖, 0↑ 0↓)
duckdb / vortex-compact (0.988x ➖, 0↑ 0↓)
duckdb / parquet (0.987x ➖, 0↑ 0↓)
duckdb / duckdb (0.988x ➖, 0↑ 0↓)
File Size Changes (27 files changed, +0.0% overall, 14↑ 13↓)
Totals:
|
Benchmarks: Clickbench on NVMEVerdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (0.933x ➖, 4↑ 0↓)
datafusion / parquet (0.930x ➖, 6↑ 0↓)
duckdb / vortex-file-compressed (0.945x ➖, 2↑ 0↓)
duckdb / parquet (0.969x ➖, 0↑ 0↓)
duckdb / duckdb (0.967x ➖, 0↑ 0↓)
File Size Changes (101 files changed, -0.0% overall, 51↑ 50↓)
Totals:
|
Benchmarks: TPC-H SF=1 on S3Verdict: No clear signal (environment too noisy confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (0.812x ➖, 6↑ 0↓)
datafusion / vortex-compact (0.828x ➖, 2↑ 0↓)
datafusion / parquet (0.976x ➖, 0↑ 1↓)
duckdb / vortex-file-compressed (0.903x ➖, 0↑ 0↓)
duckdb / vortex-compact (0.889x ➖, 0↑ 0↓)
duckdb / parquet (0.883x ➖, 1↑ 0↓)
|
Benchmarks: Random AccessVortex (geomean): 1.010x ➖ How to read Verdict and Engines
unknown / unknown (1.003x ➖, 1↑ 1↓)
|
Benchmarks: TPC-H SF=10 on S3Verdict: No clear signal (environment too noisy confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (1.041x ➖, 0↑ 0↓)
datafusion / vortex-compact (1.008x ➖, 0↑ 0↓)
datafusion / parquet (0.977x ➖, 0↑ 1↓)
duckdb / vortex-file-compressed (1.015x ➖, 0↑ 0↓)
duckdb / vortex-compact (0.997x ➖, 0↑ 0↓)
duckdb / parquet (0.973x ➖, 0↑ 0↓)
|
Benchmarks: Appian on NVMEVerdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (1.120x ❌, 0↑ 6↓)
datafusion / parquet (1.105x ❌, 0↑ 5↓)
duckdb / vortex-file-compressed (1.085x ➖, 0↑ 3↓)
duckdb / parquet (1.082x ➖, 0↑ 1↓)
duckdb / duckdb (1.075x ➖, 0↑ 2↓)
File Size Changes (4 files changed, -0.0% overall, 3↑ 1↓)
Totals:
|
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
37463ff to
e87eec5
Compare
Benchmarks: CompressionVortex (geomean): 1.003x ➖ How to read Verdict and Engines
unknown / unknown (1.011x ➖, 0↑ 0↓)
|
|
It seems like there are no logic changes, however some benchmark numbers changed (rerunning) |
|
@joseph-isaacs I don't believe any of the benchmark changes... |
Rationale for this change
Tracking issue: #7697
I'd like to make some improvements to the compressor logic, and having a better structured file system makes that a bit easier.
This is a purely cosmetic change (except for some paths changing, namely
vortex_compressor::ctx::*andvortex_compressor::estimate::*).What changes are included in this PR?
Moving around a lot of stuff, the tree is now: