Repository navigation
Description
Follow-up to #23, which made an unevaluable conjunct decline the scan rather than silently drop it. Declining is correct but blunt: a predicate that touches a nested column costs that scan the cache entirely. Supporting the conjunct instead would keep it.
What is refused today
PushdownChecker::check_single_column (src/datafusion/src/reader/plantime/row_filter.rs) sets non_primitive_columns for any Column whose file-schema type is nested, so st.a = 3 cannot be evaluated as an ArrowPredicate. unevaluable_conjunct then keeps the whole scan off the liquid path (convert_parquet_scan, src/datafusion/src/optimizers/mod.rs), and it reads as vanilla parquet.
Upstream DataFusion 54 does not refuse this. Its PushdownChecker (datafusion-datasource-parquet/src/row_filter.rs) recognises the get_field(Column("st"), "a") shape in f_down, records a StructFieldAccess { root_index, field_path }, and leaves non_primitive_columns clear. It then builds the predicate's ProjectionMask from those field paths — a leaf-level mask — instead of from root indices. A bare Column of struct type is still refused, there and here.
Why the liquid fork cannot just copy that
The fork masks by root:
// build_row_filter
let projection = ProjectionMask::roots(
metadata.file_metadata().schema_descr(),
candidate.projection.iter().copied(),
);and then reads the mask back out as cache column ids:
// get_predicate_column_id — row_filter.rs
let project_inner: &ProjectionMaskLiquid = unsafe { std::mem::transmute(projection) };
// ... returns the positions of the set bitsProjectionMask.mask is leaf-indexed (parquet 58.3.0, parquet/src/arrow/mod.rs: "a leaf column should be included if the value at the corresponding index is true"). The ids that come out are therefore parquet leaf positions.
Their consumers treat them as top-level file-schema field indices:
LiquidCacheFileRef::create_row_group(src/datafusion/src/cache/mod.rs) computesis_predicate_columnaspredicate_column_ids.contains(&idx)while enumeratingself.file_schema.fields().evaluate_selection_with_predicate(same file) passescolumn_ids[0]straight toself.get_column(column_id as u64).
The two index spaces coincide only when every file column is primitive. Supporting nested predicates means the cache has to address columns at leaf granularity, or the row filter has to translate leaf positions back to roots. That is a change to the cache's column-id model, not a row-filter change.
Also worth fixing on the way
get_predicate_column_id reaches its result by transmuteing ProjectionMask to a locally-declared struct with the same shape. That is UB the moment parquet-rs adds or reorders a field, and it is what ties the ids to leaf positions in the first place. ProjectionMask exposes leaf_included(leaf_idx) publicly; a supported accessor plus an explicit leaf-to-root mapping would remove the transmute and make the index space a deliberate choice rather than an accident.
Done when
- A conjunct of the form
get_field(col, "field") <op> ...is evaluated by the liquid row filter instead of declining the scan. - Predicate columns are identified without
transmute, and the leaf/root distinction is explicit at every boundary. - Refuse row-filter pushdown instead of dropping conjuncts that cannot be evaluated #23's
only_a_nested_predicate_costs_the_cacheis updated: the two bypassed queries in it should keep the cache once this lands. - A bare
Columnof nested type (no field access) still declines, matching upstream.
Blocked by (or at least entangled with) the leaf/field index confusion filed separately — that bug has to be settled before the id model can be widened.
Activity
zfarrell commentedon Aug 28, 2026
Blocked by #38: the cache's column-id model has to be corrected before it can be widened to leaf granularity.