Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

### Fixed
- A `vortex.uuid` column's file statistics carry its min and max, as Rust writes them from 0.92, and `WriteSummary` reports them as `UUID`s. ([#730](https://github.com/dfa1/vortex-java/pull/730))

### Changed
- A struct column is written as Rust's `vortex.struct` layout, each field chunked and compressed on its own, so files are smaller and a reader can prune one field. ([#724](https://github.com/dfa1/vortex-java/issues/724))
- Tested against vortex-jni 0.93.0, up from 0.91.0. ([#730](https://github.com/dfa1/vortex-java/pull/730))

## [0.20.0] — 2026-11-10

Expand Down
6 changes: 3 additions & 3 deletions docs/compatibility.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Compatibility

Tested against the [Rust reference implementation](https://github.com/vortex-data/vortex) v0.91.0.
Tested against the [Rust reference implementation](https://github.com/vortex-data/vortex) v0.93.0.
For the rest of the API surface (reader, writer, scan, CLI), see [reference.md](reference.md).

## Read-only deployment
Expand Down Expand Up @@ -318,7 +318,7 @@ guarantee once frozen (ADR 0023) — a write-time/read-time policy, not part of
| `preview2026.08.0` | `preview` | nothing yet |
| `zstd2026.02.0` | `zstd` | `vortex.zstd_buffers` (read only; buffer-level Zstd for GPU decode; declared by Rust's `vortex-zstd` plugin, opt-in) |

Mirrors Rust's `vortex-edition` declarations at the pinned vortex-jni release (0.91.0);
Mirrors Rust's `vortex-edition` declarations at the pinned vortex-jni release (0.93.0);
`EditionCatalogParityIntegrationTest` fails the build on drift. `core` editions are frozen with a
forever read-compatibility guarantee; `preview` holds opt-in components not yet adopted by `core`;
`zstd` is declared by Rust's `vortex-zstd` plugin crate rather than the core declarations, and is
Expand Down Expand Up @@ -403,7 +403,7 @@ As in Rust, opening a file with any other extension id that has no registered de
(`VortexException`; Rust: "No such DType extension ID"), unless
`ReadRegistry.Builder#allowUnknown()` is set — then the id round-trips verbatim through
`DType.Extension`'s raw `String` field and the column reads as its storage array, Rust's
`ForeignExtDType`. vortex-jni 0.91.0 exposes no `allow_unknown`, so it cannot read a file with a
`ForeignExtDType`. vortex-jni 0.93.0 exposes no `allow_unknown`, so it cannot read a file with a
custom extension id at all. A blank id, or one with a control character, fails the read with
`VortexException` (Rust accepts any string).

Expand Down
2 changes: 1 addition & 1 deletion docs/explanation.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ Background reading on design decisions, architecture, and benchmarks.
The official Vortex ecosystem provides JVM bindings via JNI (bundled native `.so`/`.dylib`).
JNI bindings are fast but add deployment friction: platform-specific artifacts, native build
toolchains, and crash-domain coupling between the JVM and native code. The JAR for
vortex-jni 0.91.0 is **76.8MB**.
vortex-jni 0.93.0 is **77.3MB**.

This library takes a different approach — 100% Java, no JNI, no `sun.misc.Unsafe`.
It uses the Java FFM API (`MemorySegment` / `Arena`, Java 25+) for zero-copy memory-mapped
Expand Down
4 changes: 2 additions & 2 deletions docs/reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -145,7 +145,7 @@ Writes a Vortex file. Implements `Closeable`. The file is complete and readable
| `static create(WritableByteChannel, DType.Struct, WriteOptions, WriteRegistry)` | Custom [`WriteRegistry`](#writeregistry-iogithubdfa1vortexwriterwriteregistry) — encoders and extension encoders together |
| `writeChunk(Consumer<Chunk>)` | One batch of rows; typed builder validates column names + array types at each `.put`; missing columns throw `IllegalStateException` when the lambda returns. Preferred when columns are known at compile time. |
| `writeChunk(Map<ColumnName, Object>)` | One batch of rows by map. Validates that every schema column is present and that all columns share the same row count. Use when the column set is built dynamically (Parquet/JDBC importers, generic exporters). |
| `finish()` | Finalizes the file (footer, postscript, trailer) and returns a `WriteSummary`: `fileSize()`, `rowCount()` and per column (schema order) its `compressedSize()` and `stats()` — the footer's file statistics (min, max, null and NaN counts and uncompressed size, accumulated over the batches, as vortex-jni's `VortexWriter.finish()` reports them; a UTF-8 or binary min/max is a 64-byte bound, never exact). Later calls return the same summary; `writeChunk` afterwards throws `IllegalStateException` |
| `finish()` | Finalizes the file (footer, postscript, trailer) and returns a `WriteSummary`: `fileSize()`, `rowCount()` and per column (schema order) its `compressedSize()` and `stats()` — the footer's file statistics (min, max, null and NaN counts and uncompressed size, accumulated over the batches, as vortex-jni's `VortexWriter.finish()` reports them; a UTF-8 or binary min/max is a 64-byte bound, never exact; a `vortex.uuid` min/max is a `java.util.UUID`, in unsigned byte order). Later calls return the same summary; `writeChunk` afterwards throws `IllegalStateException` |
| `close()` | `finish()` unless it already ran |

Batches are not chunks: as Rust's writer does, each primitive, boolean, decimal, string and binary
Expand Down Expand Up @@ -362,7 +362,7 @@ always be written. See [how-to.md § Add a custom extension type](how-to.md#add-

Frozen, named, additive sets of encoding IDs, each carrying a forever read-compatibility
guarantee once frozen (see ADR 0023). vortex-java mirrors Rust's `vortex-edition` declarations at
the pinned vortex-jni release (0.91.0); `EditionCatalogParityIntegrationTest` fails the build if the
the pinned vortex-jni release (0.93.0); `EditionCatalogParityIntegrationTest` fails the build if the
two drift. Editions are a **write/read policy, not part of the wire format**: nothing about a
targeted edition is ever persisted into a `.vortex` file.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,6 @@
import java.util.stream.Stream;

import static org.assertj.core.api.Assertions.assertThat;
import static org.junit.jupiter.api.Assumptions.assumeFalse;

/// vortex-jni writes a low-cardinality column of every Arrow type it accepts, dict-encoding the ones
/// Rust supports; vortex-java must read each one back and rebuild the exact values.
Expand Down Expand Up @@ -233,11 +232,6 @@ static Stream<Case> cases() {
@ParameterizedTest(name = "{0}")
@MethodSource("cases")
void jniDictColumn_javaRebuildsValues(Case c, @TempDir Path tmp) throws IOException {
// vortex-jni 0.90 keeps min/max for nested columns (vortex-data/vortex#9775), and its
// writer's close() cannot hand a uuid's fixed_size_list(u8)[16] statistic back to Java
// ("cannot return nested scalar write statistic", vortex-jni/src/writer.rs)
assumeFalse(c.name().equals("uuid"), "vortex-jni 0.90 cannot close a writer with a uuid column");

// Given — ROWS rows cycling through 4 distinct values, written by vortex-jni
Path file = tmp.resolve("dict.vortex");
writeJni(file, c);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@
import io.github.dfa1.vortex.writer.encode.NullEncodingEncoder;
import io.github.dfa1.vortex.writer.encode.NullableData;
import io.github.dfa1.vortex.writer.encode.StructData;
import io.github.dfa1.vortex.writer.encode.UuidExtensionEncoder;
import org.apache.arrow.c.ArrowArray;
import org.apache.arrow.c.ArrowSchema;
import org.apache.arrow.c.Data;
Expand All @@ -39,6 +40,7 @@
import org.apache.arrow.vector.IntVector;
import org.apache.arrow.vector.UInt1Vector;
import org.apache.arrow.vector.UInt8Vector;
import org.apache.arrow.vector.UuidVector;
import org.apache.arrow.vector.VarBinaryVector;
import org.apache.arrow.vector.VarCharVector;
import org.apache.arrow.vector.VectorSchemaRoot;
Expand All @@ -48,6 +50,7 @@
import org.apache.arrow.vector.ipc.ArrowReader;
import org.apache.arrow.vector.types.DateUnit;
import org.apache.arrow.vector.types.FloatingPointPrecision;
import org.apache.arrow.vector.extension.UuidType;
import org.apache.arrow.vector.types.pojo.ArrowType;
import org.apache.arrow.vector.types.pojo.Field;
import org.apache.arrow.vector.types.pojo.FieldType;
Expand All @@ -70,6 +73,7 @@
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.UUID;

import static org.assertj.core.api.Assertions.assertThat;

Expand Down Expand Up @@ -153,6 +157,47 @@ void nullColumn_countsEveryRowNull(@TempDir Path tmp) throws Exception {
assertThat(result).isEqualTo(fileStatistics(jni));
}

@Test
void uuidColumn_writesTheStatisticsJniWrites(@TempDir Path tmp) throws Exception {
// Given — a nullable uuid column, unsorted with every fourth row null: from 0.92 Rust
// imports arrow.uuid as vortex.uuid and records its min and max (vortex-data/vortex#11299)
List<UUID> values = new ArrayList<>(ROWS);
for (int i = 0; i < ROWS; i++) {
values.add(i % 4 == 0 ? null : new UUID(i * 0x9E3779B97F4A7C15L, ~i * 31L));
}
Schema arrow = new Schema(List.of(Field.nullable("u", UuidType.INSTANCE)));
Path jni = tmp.resolve("jni.vortex");
try (dev.vortex.api.VortexWriter w = dev.vortex.api.VortexWriter.builder(SESSION,
jni.toAbsolutePath().toUri().toString(), arrow, ALLOCATOR).build();
VectorSchemaRoot root = VectorSchemaRoot.create(arrow, ALLOCATOR);
ArrowArray array = ArrowArray.allocateNew(ALLOCATOR); ArrowSchema schema = ArrowSchema.allocateNew(ALLOCATOR)) {
UuidVector vec = (UuidVector) root.getVector("u");
for (int i = 0; i < ROWS; i++) {
if (values.get(i) == null) {
vec.setNull(i);
} else {
vec.setSafe(i, values.get(i));
}
}
root.setRowCount(ROWS);
Data.exportVectorSchemaRoot(ALLOCATOR, root, null, array, schema);
w.writeBatch(array.memoryAddress(), schema.memoryAddress());
}
Path java = tmp.resolve("java.vortex");
List<ColumnName> names = List.of(ColumnName.of("u"));
DType.Struct schema = new DType.Struct(names, List.of(UuidExtensionEncoder.INSTANCE.dtype(true)), false);
try (FileChannel ch = FileChannel.open(java, StandardOpenOption.CREATE, StandardOpenOption.WRITE);
VortexWriter w = VortexWriter.create(ch, schema, WriteOptions.defaults())) {
w.writeChunk(Map.of(ColumnName.of("u"), values));
}

// When
List<String> result = fileStatistics(java, names);

// Then
assertThat(result).isEqualTo(fileStatistics(jni, names));
}

@Test
void noBatch_writesWhatJniWrites(@TempDir Path tmp) throws Exception {
// Given — a file without any batch
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -44,17 +44,16 @@
import org.apache.arrow.vector.BigIntVector;
import org.apache.arrow.vector.BitVector;
import org.apache.arrow.vector.DateDayVector;
import org.apache.arrow.vector.FixedSizeBinaryVector;
import org.apache.arrow.vector.Float2Vector;
import org.apache.arrow.vector.Float8Vector;
import org.apache.arrow.vector.IntVector;
import org.apache.arrow.vector.TimeMilliVector;
import org.apache.arrow.vector.TimeStampMilliVector;
import org.apache.arrow.vector.ValueVector;
import org.apache.arrow.vector.UuidVector;
import org.apache.arrow.vector.VectorSchemaRoot;
import org.apache.arrow.vector.complex.ListVector;
import org.apache.arrow.vector.ipc.ArrowReader;
import org.junit.jupiter.api.Disabled;
import org.junit.jupiter.api.Named;
import org.junit.jupiter.api.Tag;
import org.junit.jupiter.api.Test;
Expand Down Expand Up @@ -2537,12 +2536,9 @@ void javaWriter_rustReader_nullable_timestamp(@TempDir Path tmp) throws IOExcept
}

@Test
@Disabled("vortex-jni 0.74.0 uses the non-plugin-aware DType.to_arrow_schema() which bails "
+ "for any extension type other than parquet.variant — including vortex.uuid. "
+ "Re-enable when the JNI is updated to use ArrowSession.to_arrow_schema().")
void javaWriter_rustReader_nullable_uuid(@TempDir Path tmp) throws IOException {
// Given — nullable vortex.uuid column; row 1 is null.
// UUID stored as big-endian 16-byte FixedSizeBinary → Arrow FixedSizeBinary(16) on read.
// Given — nullable vortex.uuid column; row 1 is null. Rust exports vortex.uuid as Arrow's
// arrow.uuid extension (vortex-data/vortex#11224; before 0.92 it could not export it at all).
Path file = tmp.resolve("java_nullable_uuid.vtx");
UUID u0 = UUID.fromString("12345678-1234-5678-9abc-def012345678");
UUID u2 = UUID.fromString("00000000-0000-0000-0000-000000000001");
Expand All @@ -2554,7 +2550,7 @@ void javaWriter_rustReader_nullable_uuid(@TempDir Path tmp) throws IOException {
sut.writeChunk(Map.of(ColumnName.of("u"), Arrays.asList(u0, null, u2)));
}

// Then — Rust reads FixedSizeBinary(16) vector; row 1 null, rows 0/2 round-trip as UUID
// Then — Rust reads an arrow.uuid vector; row 1 null, rows 0/2 round-trip
String uri = file.toAbsolutePath().toUri().toString();
DataSource ds = DataSource.open(SESSION, uri);
Scan scan = ds.scan(ScanOptions.of());
Expand All @@ -2564,9 +2560,9 @@ void javaWriter_rustReader_nullable_uuid(@TempDir Path tmp) throws IOException {
try (ArrowReader reader = partition.scanArrow(ALLOCATOR)) {
while (reader.loadNextBatch()) {
VectorSchemaRoot root = reader.getVectorSchemaRoot();
FixedSizeBinaryVector vec = (FixedSizeBinaryVector) root.getVector("u");
UuidVector vec = (UuidVector) root.getVector("u");
for (int i = 0; i < root.getRowCount(); i++) {
uuids.add(vec.isNull(i) ? null : uuidFromBytes(vec.get(i)));
uuids.add(vec.getObject(i));
}
}
}
Expand Down Expand Up @@ -2799,14 +2795,4 @@ void prop_pco_i64_consecutive_roundTripsViaRust(long[] data) throws IOException
deleteDir(tmp);
}
}

private static UUID uuidFromBytes(byte[] b) {
long msb = 0;
long lsb = 0;
for (int k = 0; k < 8; k++) {
msb = (msb << 8) | (b[k] & 0xFFL);
lsb = (lsb << 8) | (b[8 + k] & 0xFFL);
}
return new UUID(msb, lsb);
}
}
Loading
Loading