Skip to content

Read Vortex file column statistics through vortex-jni - #10863

Merged
connortsui20 merged 3 commits into
developfrom
rk/jni-file-statistics
Nov 2, 2026
Merged

connortsui20 merged 3 commits into
developfrom
rk/jni-file-statistics

Conversation

@robert3005

@robert3005 robert3005 commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Adds per-file statistics reads to the Java bindings. The statistics come from the file footer, so no column data is read.

  • NativeFiles.readStatistics(Session, String uri, Map<String, String> options) returns List<VortexColumnStatistics>.
  • NativeFiles.readStatistics(Session, NativeReadable) does the same through caller-provided I/O.
  • Each entry holds the column index, compressed size, value count, null count, NaN count, min, and max, in Arrow schema order. Statistics that the file does not store are empty.
  • Each statistic is resolved through the file's DataSource. The stored aggregate partial from field_statistic comes first. The finalized legacy statistics from field_statistics are used only when a statistic has no partial, for example in a file written before partials were stored.

@robert3005 robert3005 added the changelog/feature A new feature label Oct 27, 2026
robert3005 and others added 3 commits November 2, 2026 14:35
Add NativeFiles.readStatistics for a URI and for a NativeReadable. Both
return the per-column statistics stored in the file footer and read no
column data. Move the statistics-to-Java conversion out of the writer
into a shared module so writes and reads use the same code.

Signed-off-by: Robert Kruszewski <github@robertk.io>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Resolve each column statistic through the file's DataSource. Use the
stored aggregate partial from field_statistic first, and read the
finalized legacy statistics from field_statistics only when a statistic
has no partial, for example in a file written before partials were
stored.

Signed-off-by: Robert Kruszewski <github@robertk.io>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Robert Kruszewski <github@robertk.io>
@robert3005
robert3005 requested a review from myrrc November 2, 2026 14:36
@robert3005
robert3005 force-pushed the rk/jni-file-statistics branch from 0911e01 to 2b3c453 Compare November 2, 2026 14:36
@codspeed

codspeed Bot commented Nov 2, 2026

Copy link
Copy Markdown

Merging this PR will degrade performance by 22.48%

2 warnings

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚠️ 2 benchmarks are dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

❌ 4 regressed benchmarks
✅ 2314 untouched benchmarks

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
❌ Walltime filter_neon[64] 1.9 µs 2.7 µs -32.21%
❌ Walltime threshold_neon[64] 2.5 µs 3.5 µs -27.88%
❌ Walltime threshold_neon[80] 3.4 µs 4 µs -15.38%
❌ Walltime filter_neon[80] 2.6 µs 3 µs -12.72%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing rk/jni-file-statistics (2b3c453) with develop (23bb75b)

Open in CodSpeed

@connortsui20
connortsui20 merged commit a606596 into develop Nov 2, 2026
75 of 76 checks passed
@connortsui20
connortsui20 deleted the rk/jni-file-statistics branch November 2, 2026 15:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/feature A new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants