[DISCUSSION] DataFusion Road Map: Q3-Q4 2025

I suggest we use this ticket to discuss and coordinate the "roadmap" for DataFusion over the next 3-6 months

DataFusion is a community project, and unlike many other open source projects there is no "DataFusion Company" that pays contributors and determines their priority. The project's evolution and priorities is determined by those community members able and willing to devote their time and energy to driving the project forward

In the past it has been helpful to have a public discussion about features in order to:

Find others interested in features you may be thinking of working on
See who is actively looking for certain features / capabilities

So please feel free to post comments about:

Features you plan to work on
Features you would like others to work on
Features you are willing / able / planning to help others with

See previous discussions:

@XiangpengHao

My plans:

🚀 performance (dynamic filter pushdown)

parquet predicate pushdown with @XiangpengHao @zhuqi-lucas Enable parquet filter pushdown (filter_pushdown) by default #3463
dynamic filter pushdown with @adriangb and @berkaysynnada: Dynamic pruning filters from TopK state (optimize ORDER BY LIMIT queries) #15037

Variant / semi structured data support

I plan to help @PinkCrow007, and others work on variant. This likely requires doing so in arrow-rs / parquet and then moving on to features in datafusion like expression pushdown and user defined types

I also plan to continue releases, help review code for important projects, blog writing, etc

My list (still mainly about aggregation performance, it has falled much behind duckdb on clickbench...):

Push forward Improve aggregate performance with adaptive sizing in accumulators / avoiding reallocations in accumulators #7065 , which has been proved to be really promising in poc (10%~20% faster for related queries in clickbench).
The sketch pr Intermediate result blocked approach to aggregation memory management #15591 is already ready for review, after this big one merged. The implementations of GroupValuess and GroupAccumulators can be worked in parallel, and push it forward fast.
Optimizations on optimizer side, such as Linear Aggregate Functions Optimization #15633 and POC: Eliminate unnecessary group by keys (q35 in clickbench 1.35x faster) #13617
Help push forward other working promising opmizations such as Support zero copy hash repartitioning for Hash Aggregate #15383 and Enable parquet filter pushdown (filter_pushdown) by default #3463

My list (still mainly about aggregation performance, it has falled much behind duckdb on clickbench...):

Welcome back!

@skyzh

I would like to help push forward the CMU effort for variant types.

I would also like to try to get this back up: support simple/cross lateral joins #14595 (@skyzh would you be interested in continuing to work on this?)
Implementing Merge statement: POC: Parse to Merge Logical Plan #15862 (would be great if anyone can take a look)

I think #14595 is in a decent shape and could be merged :) Though it cannot unnest all queries and might produce wrong result for some lateral joins, that would be a good starting point for us to slowly add lateral join and full unnesting support to the codebase.

I am also trying to help organize a join task force for planning joins / subqueries: [DISCUSSION] JOIN "task force" / project team #15885

@skyzh

I think #14595 is in a decent shape and could be merged :) Though it cannot unnest all queries and might produce wrong result for some lateral joins, that would be a good starting point for us to slowly add lateral join and full unnesting support to the codebase.

@skyzh -- I think the conversation on #5492 might be relevant to you as well

I'll try to start with Add Memory Profiling Functionality #14510

@suremarc

My lists:

Work with @suremarc @alamb and @wiedld to make Optimized SPM cross the finish line
Finish all my legacy PRs: https://github.com/apache/datafusion/pulls/xudong963
Start to work on the dynamic filter for join Push Dynamic Join Predicates into Scan ("Sideways Information Passing", etc) #7955, it's helpful for my team
Speed up logical optimizer, there is a tracking issue: Tracking: speed up the logical optimizer #15775
Rich statistics tests
Releases, help review code, etc

My list includes:

feat: Add ConfigOptions to ScalarFunctionArgs #13527 (finally finish this)
Introduce Async User Defined Functions #14837 (adopt my async UDF's to use this and validate)
Each Union branch should produce fields with nullable: true if any of the branches produces fields with nullable: true on the same position #15394
[EPIC] Additional Date/Time related open issues #8282 (Specifically allowing changing default tz via config and UDF's/etc properly handling tz data ).
look into adding an 'ANSI' mode that when disabled causes null to be returned instead of an error. Useful especially when parsing external date/timestamps from strings.

I am currently interested in the following subjects where I'll probably experiment with some things or help out others.

Window Functions (profiling, implementing improvements / optimizations)

** Vectorize window functions (GroupsAccumulator-style) #15607
Possible (logical plan) optimizations - limit pushdown?

Sorting performance ideas / helping out, examples:

Reuse Rows allocation in SortPreservingMergeStream / RowCursorStream #15720
Perf: Optimize in memory sort #15380 (review)
Create version of LexicographicalComparator that compares fixed number of columns (~ -15%) arrow-rs#7530
Improve multi-column sorting for primitive arrays by avoiding multi-column sort arrow-rs#7532 [ ] Streamlining arrow-rs kernels in terms of speed, consistency, reduce use of unsafe, etc.

Do we have blogs about these epics?

@irenjj

Here is a ticket tracking @irenjj 's work for correlated subqueries for this summer:

[Epic]: Google Summer of Code 2025 Correlated Subquery Support #16059

I filed a new roadmap discussion ticket

[DISCUSSION] DataFusion Road Map: Q1 2026 #18494

@niebayes

Do we have blogs about these epics?

Sorry I didn't see this before @niebayes -- I do not know of any blogs / writeups about epics that are not already listed on the epics themselves

The current list of EPICs can be found here.
The current list of PROPOSAL EPIC (that are not yet underway) can be found here.

Let's continue the conversation on #18494

Navigation Menu

Repository navigation

[DISCUSSION] DataFusion Road Map: Q3-Q4 2025 #15878

🚀 performance (dynamic filter pushdown)

Variant / semi structured data support

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[DISCUSSION] DataFusion Road Map: Q3-Q4 2025 #15878

Description

Activity

alamb commented on Apr 28, 2025

🚀 performance (dynamic filter pushdown)

Variant / semi structured data support

Rachelint commented on Apr 28, 2025

alamb commented on Apr 28, 2025

jonathanc-n commented on Apr 28, 2025

skyzh commented on Apr 28, 2025

alamb commented on Apr 28, 2025

alamb commented on Apr 28, 2025

comphead commented on Apr 28, 2025

xudong963 commented on Apr 29, 2025

Omega359 commented on Apr 30, 2025

Dandandan commented on May 1, 2025

niebayes commented on May 7, 2025

alamb commented on May 18, 2025

alamb commented on Nov 5, 2025

alamb commented on Nov 5, 2025

alamb commented on Nov 5, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions