Repository navigation
duckdb: keep pushed filters visible to the join planner - #10941
robert3005 wants to merge 1 commit into
Conversation
The Vortex scan took complex filters out of DuckDB's plan and reported a post-filter row count (total * 0.2) instead. DuckDB uses the scan's row count as the upper limit for the distinct values of a join key. With a post-filter count, joins on a filtered dimension table did not reduce the estimate, and filters that the scan consumed were invisible to the join order optimizer. When the TPC-DS tables were split into many files, DuckDB lost the per-file min/max statistics that had hidden this. Query 64 then got a plan with a nested loop join on `cd1.cd_marital_status != cd2.cd_marital_status` that makes about 718 billion rows at SF=10, and the query ran out of memory. Now the scan copies each complex filter and still evaluates it, but DuckDB keeps the filter in its plan. The scan reports its row count before filters. The q64 estimates now match DuckDB on Parquet. Signed-off-by: Robert Kruszewski <github@robertk.io>
Merging this PR will degrade performance by 10.67%2 warnings
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing Footnotes
|
| } | ||
| assert!(plan.contains("Read Vortex"), "missing Vortex scan:\n{plan}"); | ||
| assert!(!plan.contains("FILTER"), "filter was not pushed:\n{plan}"); | ||
| assert!(plan.contains("Filters"), "filter was not pushed:\n{plan}"); |
There was a problem hiding this comment.
Don't change the string
|
This fixes the underlying issue of tpcds sf=100 but it's not a good solution imho |
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨Benchmark |
🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨Benchmark |
🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨Benchmark |
|
We can't merge this PR because duckdb expects post-pushdown estimates in cardinality(). The point of pushing down filters is that we don't want to evaluate them twice as some are quite expensive. This is to some extent already fixed in 2.0 because some optimizer rules can query filters even if they were pushed |
🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨Benchmark |
🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨Benchmark |
Benchmarks: FineWeb S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.937x ➖, 1↑ 1↓)
datafusion / vortex-compact / ns (1.068x ➖, 0↑ 2↓)
datafusion / parquet / ns (1.069x ➖, 0↑ 1↓)
duckdb / vortex-file-compressed / ns (0.996x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (1.112x ➖, 0↑ 1↓)
duckdb / parquet / ns (0.966x ➖, 1↑ 0↓)
|
🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨Benchmark |
🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨Benchmark |
🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨Benchmark |
🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨Benchmark |
🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨Benchmark |
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.076x ➖, 1↑ 3↓)
datafusion / vortex-compact / ns (0.942x ➖, 3↑ 3↓)
datafusion / parquet / ns (1.101x ➖, 0↑ 3↓)
duckdb / vortex-file-compressed / ns (0.984x ➖, 1↑ 2↓)
duckdb / vortex-compact / ns (0.931x ➖, 0↑ 1↓)
duckdb / parquet / ns (1.050x ➖, 0↑ 2↓)
|
🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨Benchmark |
Benchmarks: TPC-H SF=10 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.874x ➖, 3↑ 1↓)
datafusion / vortex-compact / ns (0.904x ➖, 3↑ 2↓)
datafusion / parquet / ns (0.934x ➖, 2↑ 2↓)
duckdb / vortex-file-compressed / ns (0.948x ➖, 1↑ 1↓)
duckdb / vortex-compact / ns (0.919x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.000x ➖, 0↑ 1↓)
|
Fixes #10933.
Problem
Since #10519 split the TPC-DS tables into many files, DuckDB on Vortex runs out of memory on TPC-DS query 64 at SF=10, and does not finish at SF=100. Queries 1 to 63 did not change.
The Vortex scan took complex filters out of DuckDB's plan, and reported
total * 0.2rows to make up for them. DuckDB uses the scan's row count as the upper limit for the distinct values of a join key (relation_statistics_scan.cpp). So with a post-filter count, a join on a filtered dimension table did not reduce the estimate. Filters that the scan consumed, such asi_color IN (...), were also invisible to DuckDB's own selectivity estimates.With one file per table, the Vortex min/max statistics hid this. DuckDB drops per-file statistics for a multi-file scan, so after the split, the
store_salesside of q64 looked about 25 times larger than it is. DuckDB then picked a plan with a nested loop join oncd1.cd_marital_status != cd2.cd_marital_statusunder a hash aggregate. At SF=10 that join makes 717,764,544,000 rows (467,100 customer rows × 1,920,800customer_demographicsrows), and the hash aggregate runs out of memory and spill space.Change
pushdown_complex_filterstill copies each filter into the scan, so the scan prunes and filters with it. It now returnsfalse, so DuckDB also keeps the filter in its plan and can estimate it. It skips filters that it already holds, because DuckDB can offer a kept filter again.cardinalityreports the row count before filters. This removes the 0.2DEFAULT_SELECTIVITYfrom Mimic duckdb's post-filter cardinality estimates #7895, and thehas_non_optional_filterflag that only fed it.FILTERoperator is gone.With this change, every q64 estimate matches DuckDB on Parquet (for example
store_sales ⋈ date_dim~5,759,877 rows,cs_ui ⋈ item~72,826 rows), and the plan has no nested loop join.Results
TPC-DS SF=10, split files,
duckdb-benchon a 16-core Apple Silicon laptop.Some short queries got slower in both runs: q9 (1.4×), q28 (1.44×), q29 (1.5×), q74 (1.4×), q30 and q81 (1.8×, both under 35 ms). Laptop noise was up to 1.7× on single small queries. One likely cause is that DuckDB now evaluates each kept filter again on the rows that the scan returns; I have not confirmed this with a profile. The nightly benchmark should show if the slowdowns are real.
The DuckDB on Vortex TPC-DS SF=10 and SF=100 nightly jobs should pass again with this change.
Checks
cargo check -p vortex-duckdb --tests: no errors or warnings.duckdb-bench tpcdsat SF=10 as above: plan and timings.test_membership_filter_pushdown,spatial_filter_pushes_*) and the newcardinality_ignores_pushed_filterstest have not been run locally.🤖 Generated with Claude Code