Repository navigation
build(java): run the Spark compat tests on Spark 4.3 - #11405
Conversation
Spark 4.3.0 is released, and the Spark 4.0 module's tests ran against 4.1 and 4.2 only. Add 4.3 to the compat suites. Spark 4.3 asks a scan for row counts only when CBO is on, so three statistics tests now enable it. Signed-off-by: jackylee-ch <qcsd2011@gmail.com>
Merging this PR will improve performance by 27.57%2 warnings
Performance Changes
Tip Curious why performance improved? Comment Comparing Footnotes
|
Summary
The Spark 4.0 module's tests run against Spark 4.1 and 4.2 to catch binary breaks, as in #10792, but not against Spark 4.3 (on Maven Central since 2026-10-30). A break on 4.3 would ship unnoticed, as the 4.2 read failure (#10791) did.
Changes
spark43tolibs.versions.tomland4.3tosparkCompatVersions, which registersspark43CompatTestand adds it tocheck.DataSourceV2ScanRelation.computeStatsasks the scan for row counts only when CBO or plan stats are enabled, sonumRowsno longer reachesoptimizedPlan().stats()by default. Three tests read it that way (VortexDataSourceStatsTest,VortexAggregatePushdownTest,VortexFileSplitTest) and now setspark.sql.cbo.enabled. The connector is unchanged.Tested with:
./gradlew --parallel javadoc check: passes, including the 4.1, 4.2 and 4.3 compat suitesspark43CompatTestbefore the test change: 3 of 244 failed; Spark 4.2: all passedVortexDataSourceS3MockTestskipped locally (no Docker)AI assistance
Written with agentic AI assistance.