Skip to content

build(java): run the Spark compat tests on Spark 4.3 - #11405

Merged
robert3005 merged 1 commit into
vortex-data:developfrom
jackylee-ch:build/spark-compat-43
Nov 12, 2026
Merged

robert3005 merged 1 commit into
vortex-data:developfrom
jackylee-ch:build/spark-compat-43

Conversation

@jackylee-ch

Copy link
Copy Markdown
Contributor

Summary

The Spark 4.0 module's tests run against Spark 4.1 and 4.2 to catch binary breaks, as in #10792, but not against Spark 4.3 (on Maven Central since 2026-10-30). A break on 4.3 would ship unnoticed, as the 4.2 read failure (#10791) did.

Changes

  • Add spark43 to libs.versions.toml and 4.3 to sparkCompatVersions, which registers spark43CompatTest and adds it to check.
  • The suite found one difference: from Spark 4.3, DataSourceV2ScanRelation.computeStats asks the scan for row counts only when CBO or plan stats are enabled, so numRows no longer reaches optimizedPlan().stats() by default. Three tests read it that way (VortexDataSourceStatsTest, VortexAggregatePushdownTest, VortexFileSplitTest) and now set spark.sql.cbo.enabled. The connector is unchanged.

Tested with:

  • ./gradlew --parallel javadoc check: passes, including the 4.1, 4.2 and 4.3 compat suites
  • spark43CompatTest before the test change: 3 of 244 failed; Spark 4.2: all passed
  • VortexDataSourceS3MockTest skipped locally (no Docker)

AI assistance

Written with agentic AI assistance.

Spark 4.3.0 is released, and the Spark 4.0 module's tests ran against 4.1 and 4.2 only. Add 4.3 to the compat suites. Spark 4.3 asks a scan for row counts only when CBO is on, so three statistics tests now enable it.

Signed-off-by: jackylee-ch <qcsd2011@gmail.com>
@robert3005 robert3005 added the changelog/chore A trivial change label Nov 12, 2026
@robert3005
robert3005 enabled auto-merge (squash) November 12, 2026 10:12
@codspeed

codspeed Bot commented Nov 12, 2026

Copy link
Copy Markdown

Merging this PR will improve performance by 27.57%

2 warnings

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚠️ 1 benchmark is dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

⚡ 1 improved benchmark
✅ 1920 untouched benchmarks
⏩ 50 skipped benchmarks1

Performance Changes

Mode Benchmark BASE HEAD Efficiency
⚡ CPU Simulation chunked_bool_canonical_into[(1000, 10)] 38.5 µs 30.2 µs +27.57%

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.


Comparing jackylee-ch:build/spark-compat-43 (8f8d08f) with develop (b6cc108)

Open in CodSpeed

Footnotes

  1. 50 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

@robert3005
robert3005 merged commit bb1b744 into vortex-data:develop Nov 12, 2026
98 of 99 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/chore A trivial change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants