diff --git a/CHANGELOG.md b/CHANGELOG.md index 549538c74..daa9d8092 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,8 +7,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Fixed +- A `vortex.uuid` column's file statistics carry its min and max, as Rust writes them from 0.92, and `WriteSummary` reports them as `UUID`s. ([#730](https://github.com/dfa1/vortex-java/pull/730)) + ### Changed - A struct column is written as Rust's `vortex.struct` layout, each field chunked and compressed on its own, so files are smaller and a reader can prune one field. ([#724](https://github.com/dfa1/vortex-java/issues/724)) +- Tested against vortex-jni 0.93.0, up from 0.91.0. ([#730](https://github.com/dfa1/vortex-java/pull/730)) ## [0.20.0] — 2026-11-10 diff --git a/docs/compatibility.md b/docs/compatibility.md index 794316f63..9600d558d 100644 --- a/docs/compatibility.md +++ b/docs/compatibility.md @@ -1,6 +1,6 @@ # Compatibility -Tested against the [Rust reference implementation](https://github.com/vortex-data/vortex) v0.91.0. +Tested against the [Rust reference implementation](https://github.com/vortex-data/vortex) v0.93.0. For the rest of the API surface (reader, writer, scan, CLI), see [reference.md](reference.md). ## Read-only deployment @@ -318,7 +318,7 @@ guarantee once frozen (ADR 0023) — a write-time/read-time policy, not part of | `preview2026.08.0` | `preview` | nothing yet | | `zstd2026.02.0` | `zstd` | `vortex.zstd_buffers` (read only; buffer-level Zstd for GPU decode; declared by Rust's `vortex-zstd` plugin, opt-in) | -Mirrors Rust's `vortex-edition` declarations at the pinned vortex-jni release (0.91.0); +Mirrors Rust's `vortex-edition` declarations at the pinned vortex-jni release (0.93.0); `EditionCatalogParityIntegrationTest` fails the build on drift. `core` editions are frozen with a forever read-compatibility guarantee; `preview` holds opt-in components not yet adopted by `core`; `zstd` is declared by Rust's `vortex-zstd` plugin crate rather than the core declarations, and is @@ -403,7 +403,7 @@ As in Rust, opening a file with any other extension id that has no registered de (`VortexException`; Rust: "No such DType extension ID"), unless `ReadRegistry.Builder#allowUnknown()` is set — then the id round-trips verbatim through `DType.Extension`'s raw `String` field and the column reads as its storage array, Rust's -`ForeignExtDType`. vortex-jni 0.91.0 exposes no `allow_unknown`, so it cannot read a file with a +`ForeignExtDType`. vortex-jni 0.93.0 exposes no `allow_unknown`, so it cannot read a file with a custom extension id at all. A blank id, or one with a control character, fails the read with `VortexException` (Rust accepts any string). diff --git a/docs/explanation.md b/docs/explanation.md index 6ace4d4d4..07e94cdb5 100644 --- a/docs/explanation.md +++ b/docs/explanation.md @@ -7,7 +7,7 @@ Background reading on design decisions, architecture, and benchmarks. The official Vortex ecosystem provides JVM bindings via JNI (bundled native `.so`/`.dylib`). JNI bindings are fast but add deployment friction: platform-specific artifacts, native build toolchains, and crash-domain coupling between the JVM and native code. The JAR for -vortex-jni 0.91.0 is **76.8MB**. +vortex-jni 0.93.0 is **77.3MB**. This library takes a different approach — 100% Java, no JNI, no `sun.misc.Unsafe`. It uses the Java FFM API (`MemorySegment` / `Arena`, Java 25+) for zero-copy memory-mapped diff --git a/docs/reference.md b/docs/reference.md index 34d6c43ab..d2bd79197 100644 --- a/docs/reference.md +++ b/docs/reference.md @@ -145,7 +145,7 @@ Writes a Vortex file. Implements `Closeable`. The file is complete and readable | `static create(WritableByteChannel, DType.Struct, WriteOptions, WriteRegistry)` | Custom [`WriteRegistry`](#writeregistry-iogithubdfa1vortexwriterwriteregistry) — encoders and extension encoders together | | `writeChunk(Consumer)` | One batch of rows; typed builder validates column names + array types at each `.put`; missing columns throw `IllegalStateException` when the lambda returns. Preferred when columns are known at compile time. | | `writeChunk(Map)` | One batch of rows by map. Validates that every schema column is present and that all columns share the same row count. Use when the column set is built dynamically (Parquet/JDBC importers, generic exporters). | -| `finish()` | Finalizes the file (footer, postscript, trailer) and returns a `WriteSummary`: `fileSize()`, `rowCount()` and per column (schema order) its `compressedSize()` and `stats()` — the footer's file statistics (min, max, null and NaN counts and uncompressed size, accumulated over the batches, as vortex-jni's `VortexWriter.finish()` reports them; a UTF-8 or binary min/max is a 64-byte bound, never exact). Later calls return the same summary; `writeChunk` afterwards throws `IllegalStateException` | +| `finish()` | Finalizes the file (footer, postscript, trailer) and returns a `WriteSummary`: `fileSize()`, `rowCount()` and per column (schema order) its `compressedSize()` and `stats()` — the footer's file statistics (min, max, null and NaN counts and uncompressed size, accumulated over the batches, as vortex-jni's `VortexWriter.finish()` reports them; a UTF-8 or binary min/max is a 64-byte bound, never exact; a `vortex.uuid` min/max is a `java.util.UUID`, in unsigned byte order). Later calls return the same summary; `writeChunk` afterwards throws `IllegalStateException` | | `close()` | `finish()` unless it already ran | Batches are not chunks: as Rust's writer does, each primitive, boolean, decimal, string and binary @@ -362,7 +362,7 @@ always be written. See [how-to.md § Add a custom extension type](how-to.md#add- Frozen, named, additive sets of encoding IDs, each carrying a forever read-compatibility guarantee once frozen (see ADR 0023). vortex-java mirrors Rust's `vortex-edition` declarations at -the pinned vortex-jni release (0.91.0); `EditionCatalogParityIntegrationTest` fails the build if the +the pinned vortex-jni release (0.93.0); `EditionCatalogParityIntegrationTest` fails the build if the two drift. Editions are a **write/read policy, not part of the wire format**: nothing about a targeted edition is ever persisted into a `.vortex` file. diff --git a/integration/src/test/java/io/github/dfa1/vortex/integration/DictAllTypesInteropIntegrationTest.java b/integration/src/test/java/io/github/dfa1/vortex/integration/DictAllTypesInteropIntegrationTest.java index 509fc0e18..22a47ee80 100644 --- a/integration/src/test/java/io/github/dfa1/vortex/integration/DictAllTypesInteropIntegrationTest.java +++ b/integration/src/test/java/io/github/dfa1/vortex/integration/DictAllTypesInteropIntegrationTest.java @@ -60,7 +60,6 @@ import java.util.stream.Stream; import static org.assertj.core.api.Assertions.assertThat; -import static org.junit.jupiter.api.Assumptions.assumeFalse; /// vortex-jni writes a low-cardinality column of every Arrow type it accepts, dict-encoding the ones /// Rust supports; vortex-java must read each one back and rebuild the exact values. @@ -233,11 +232,6 @@ static Stream cases() { @ParameterizedTest(name = "{0}") @MethodSource("cases") void jniDictColumn_javaRebuildsValues(Case c, @TempDir Path tmp) throws IOException { - // vortex-jni 0.90 keeps min/max for nested columns (vortex-data/vortex#9775), and its - // writer's close() cannot hand a uuid's fixed_size_list(u8)[16] statistic back to Java - // ("cannot return nested scalar write statistic", vortex-jni/src/writer.rs) - assumeFalse(c.name().equals("uuid"), "vortex-jni 0.90 cannot close a writer with a uuid column"); - // Given — ROWS rows cycling through 4 distinct values, written by vortex-jni Path file = tmp.resolve("dict.vortex"); writeJni(file, c); diff --git a/integration/src/test/java/io/github/dfa1/vortex/integration/FileStatisticsInteropIntegrationTest.java b/integration/src/test/java/io/github/dfa1/vortex/integration/FileStatisticsInteropIntegrationTest.java index d0ee8d4c1..4201d0690 100644 --- a/integration/src/test/java/io/github/dfa1/vortex/integration/FileStatisticsInteropIntegrationTest.java +++ b/integration/src/test/java/io/github/dfa1/vortex/integration/FileStatisticsInteropIntegrationTest.java @@ -25,6 +25,7 @@ import io.github.dfa1.vortex.writer.encode.NullEncodingEncoder; import io.github.dfa1.vortex.writer.encode.NullableData; import io.github.dfa1.vortex.writer.encode.StructData; +import io.github.dfa1.vortex.writer.encode.UuidExtensionEncoder; import org.apache.arrow.c.ArrowArray; import org.apache.arrow.c.ArrowSchema; import org.apache.arrow.c.Data; @@ -39,6 +40,7 @@ import org.apache.arrow.vector.IntVector; import org.apache.arrow.vector.UInt1Vector; import org.apache.arrow.vector.UInt8Vector; +import org.apache.arrow.vector.UuidVector; import org.apache.arrow.vector.VarBinaryVector; import org.apache.arrow.vector.VarCharVector; import org.apache.arrow.vector.VectorSchemaRoot; @@ -48,6 +50,7 @@ import org.apache.arrow.vector.ipc.ArrowReader; import org.apache.arrow.vector.types.DateUnit; import org.apache.arrow.vector.types.FloatingPointPrecision; +import org.apache.arrow.vector.extension.UuidType; import org.apache.arrow.vector.types.pojo.ArrowType; import org.apache.arrow.vector.types.pojo.Field; import org.apache.arrow.vector.types.pojo.FieldType; @@ -70,6 +73,7 @@ import java.util.LinkedHashMap; import java.util.List; import java.util.Map; +import java.util.UUID; import static org.assertj.core.api.Assertions.assertThat; @@ -153,6 +157,47 @@ void nullColumn_countsEveryRowNull(@TempDir Path tmp) throws Exception { assertThat(result).isEqualTo(fileStatistics(jni)); } + @Test + void uuidColumn_writesTheStatisticsJniWrites(@TempDir Path tmp) throws Exception { + // Given — a nullable uuid column, unsorted with every fourth row null: from 0.92 Rust + // imports arrow.uuid as vortex.uuid and records its min and max (vortex-data/vortex#11299) + List values = new ArrayList<>(ROWS); + for (int i = 0; i < ROWS; i++) { + values.add(i % 4 == 0 ? null : new UUID(i * 0x9E3779B97F4A7C15L, ~i * 31L)); + } + Schema arrow = new Schema(List.of(Field.nullable("u", UuidType.INSTANCE))); + Path jni = tmp.resolve("jni.vortex"); + try (dev.vortex.api.VortexWriter w = dev.vortex.api.VortexWriter.builder(SESSION, + jni.toAbsolutePath().toUri().toString(), arrow, ALLOCATOR).build(); + VectorSchemaRoot root = VectorSchemaRoot.create(arrow, ALLOCATOR); + ArrowArray array = ArrowArray.allocateNew(ALLOCATOR); ArrowSchema schema = ArrowSchema.allocateNew(ALLOCATOR)) { + UuidVector vec = (UuidVector) root.getVector("u"); + for (int i = 0; i < ROWS; i++) { + if (values.get(i) == null) { + vec.setNull(i); + } else { + vec.setSafe(i, values.get(i)); + } + } + root.setRowCount(ROWS); + Data.exportVectorSchemaRoot(ALLOCATOR, root, null, array, schema); + w.writeBatch(array.memoryAddress(), schema.memoryAddress()); + } + Path java = tmp.resolve("java.vortex"); + List names = List.of(ColumnName.of("u")); + DType.Struct schema = new DType.Struct(names, List.of(UuidExtensionEncoder.INSTANCE.dtype(true)), false); + try (FileChannel ch = FileChannel.open(java, StandardOpenOption.CREATE, StandardOpenOption.WRITE); + VortexWriter w = VortexWriter.create(ch, schema, WriteOptions.defaults())) { + w.writeChunk(Map.of(ColumnName.of("u"), values)); + } + + // When + List result = fileStatistics(java, names); + + // Then + assertThat(result).isEqualTo(fileStatistics(jni, names)); + } + @Test void noBatch_writesWhatJniWrites(@TempDir Path tmp) throws Exception { // Given — a file without any batch diff --git a/integration/src/test/java/io/github/dfa1/vortex/integration/JavaWritesRustReadsIntegrationTest.java b/integration/src/test/java/io/github/dfa1/vortex/integration/JavaWritesRustReadsIntegrationTest.java index 6b350cdcb..4c2b5a9bd 100644 --- a/integration/src/test/java/io/github/dfa1/vortex/integration/JavaWritesRustReadsIntegrationTest.java +++ b/integration/src/test/java/io/github/dfa1/vortex/integration/JavaWritesRustReadsIntegrationTest.java @@ -44,17 +44,16 @@ import org.apache.arrow.vector.BigIntVector; import org.apache.arrow.vector.BitVector; import org.apache.arrow.vector.DateDayVector; -import org.apache.arrow.vector.FixedSizeBinaryVector; import org.apache.arrow.vector.Float2Vector; import org.apache.arrow.vector.Float8Vector; import org.apache.arrow.vector.IntVector; import org.apache.arrow.vector.TimeMilliVector; import org.apache.arrow.vector.TimeStampMilliVector; import org.apache.arrow.vector.ValueVector; +import org.apache.arrow.vector.UuidVector; import org.apache.arrow.vector.VectorSchemaRoot; import org.apache.arrow.vector.complex.ListVector; import org.apache.arrow.vector.ipc.ArrowReader; -import org.junit.jupiter.api.Disabled; import org.junit.jupiter.api.Named; import org.junit.jupiter.api.Tag; import org.junit.jupiter.api.Test; @@ -2537,12 +2536,9 @@ void javaWriter_rustReader_nullable_timestamp(@TempDir Path tmp) throws IOExcept } @Test - @Disabled("vortex-jni 0.74.0 uses the non-plugin-aware DType.to_arrow_schema() which bails " - + "for any extension type other than parquet.variant — including vortex.uuid. " - + "Re-enable when the JNI is updated to use ArrowSession.to_arrow_schema().") void javaWriter_rustReader_nullable_uuid(@TempDir Path tmp) throws IOException { - // Given — nullable vortex.uuid column; row 1 is null. - // UUID stored as big-endian 16-byte FixedSizeBinary → Arrow FixedSizeBinary(16) on read. + // Given — nullable vortex.uuid column; row 1 is null. Rust exports vortex.uuid as Arrow's + // arrow.uuid extension (vortex-data/vortex#11224; before 0.92 it could not export it at all). Path file = tmp.resolve("java_nullable_uuid.vtx"); UUID u0 = UUID.fromString("12345678-1234-5678-9abc-def012345678"); UUID u2 = UUID.fromString("00000000-0000-0000-0000-000000000001"); @@ -2554,7 +2550,7 @@ void javaWriter_rustReader_nullable_uuid(@TempDir Path tmp) throws IOException { sut.writeChunk(Map.of(ColumnName.of("u"), Arrays.asList(u0, null, u2))); } - // Then — Rust reads FixedSizeBinary(16) vector; row 1 null, rows 0/2 round-trip as UUID + // Then — Rust reads an arrow.uuid vector; row 1 null, rows 0/2 round-trip String uri = file.toAbsolutePath().toUri().toString(); DataSource ds = DataSource.open(SESSION, uri); Scan scan = ds.scan(ScanOptions.of()); @@ -2564,9 +2560,9 @@ void javaWriter_rustReader_nullable_uuid(@TempDir Path tmp) throws IOException { try (ArrowReader reader = partition.scanArrow(ALLOCATOR)) { while (reader.loadNextBatch()) { VectorSchemaRoot root = reader.getVectorSchemaRoot(); - FixedSizeBinaryVector vec = (FixedSizeBinaryVector) root.getVector("u"); + UuidVector vec = (UuidVector) root.getVector("u"); for (int i = 0; i < root.getRowCount(); i++) { - uuids.add(vec.isNull(i) ? null : uuidFromBytes(vec.get(i))); + uuids.add(vec.getObject(i)); } } } @@ -2799,14 +2795,4 @@ void prop_pco_i64_consecutive_roundTripsViaRust(long[] data) throws IOException deleteDir(tmp); } } - - private static UUID uuidFromBytes(byte[] b) { - long msb = 0; - long lsb = 0; - for (int k = 0; k < 8; k++) { - msb = (msb << 8) | (b[k] & 0xFFL); - lsb = (lsb << 8) | (b[8 + k] & 0xFFL); - } - return new UUID(msb, lsb); - } } diff --git a/writer/src/main/java/io/github/dfa1/vortex/writer/FileStatsAccumulator.java b/writer/src/main/java/io/github/dfa1/vortex/writer/FileStatsAccumulator.java index a8a612ac9..d868f7dd9 100644 --- a/writer/src/main/java/io/github/dfa1/vortex/writer/FileStatsAccumulator.java +++ b/writer/src/main/java/io/github/dfa1/vortex/writer/FileStatsAccumulator.java @@ -2,21 +2,27 @@ import io.github.dfa1.vortex.core.model.ArrayStats; import io.github.dfa1.vortex.core.model.DType; +import io.github.dfa1.vortex.core.model.ExtensionId; import io.github.dfa1.vortex.core.model.PType; +import io.github.dfa1.vortex.core.proto.ProtoListValue; import io.github.dfa1.vortex.core.proto.ProtoScalarValue; import io.github.dfa1.vortex.core.simd.SimdOperationsSupport; import io.github.dfa1.vortex.writer.ZoneAccumulator.Extremes; import io.github.dfa1.vortex.writer.ZoneAccumulator.Kind; +import io.github.dfa1.vortex.writer.encode.FixedSizeListData; import io.github.dfa1.vortex.writer.encode.NodeStats; import io.github.dfa1.vortex.writer.encode.NullableData; import java.math.BigDecimal; import java.math.BigInteger; import java.nio.charset.StandardCharsets; +import java.util.ArrayList; import java.util.Arrays; +import java.util.List; +import java.util.UUID; /// One top-level column's file statistics, written into the postscript's `statistics` segment: -/// Rust 0.91.0's `FileStatsAccumulator` (`vortex-layout/src/layouts/file_stats.rs`) over its default +/// Rust 0.93.0's `FileStatsAccumulator` (`vortex-layout/src/layouts/file_stats.rs`) over its default /// aggregates (max, min, null count, NaN count, uncompressed size), as vortex-jni writes them. /// /// Rust accumulates every aggregate batch by batch, so this does too: @@ -27,7 +33,10 @@ /// - the uncompressed size is the sum of each batch's ([UncompressedSize]). /// /// An extension column has its storage's min and max but no NaN count, a struct, list or -/// fixed-size list only its null count and size, and a variant column no statistic at all. +/// fixed-size list only its null count and size, and a variant column no statistic at all. A +/// `vortex.uuid` column, though its storage is a `FixedSizeList(U8, 16)`, has an exact min and max: +/// its 16 bytes in unsigned lexicographic order, written as a list of `u8` scalars (Rust from 0.92, +/// vortex-data/vortex#11299). final class FileStatsAccumulator { private final DType dtype; @@ -37,6 +46,7 @@ final class FileStatsAccumulator { private final boolean extension; private final boolean nullDtype; private final boolean variant; + private final boolean uuid; private Object min; private Object max; private boolean maxDropped; @@ -53,6 +63,7 @@ final class FileStatsAccumulator { this.extension = dtype instanceof DType.Extension; this.nullDtype = storage instanceof DType.Null; this.variant = storage instanceof DType.Variant; + this.uuid = dtype instanceof DType.Extension ext && ext.extensionId() == ExtensionId.VORTEX_UUID; this.min = null; this.max = null; this.maxDropped = false; @@ -75,6 +86,10 @@ void add(Object batch, long rows) { nullCount += rows; return; } + if (uuid) { + addUuids(batch, n); + return; + } Object data = kind == Kind.NULL_COUNT_ONLY ? ZoneAccumulator.validityOnly(batch, n) : ZoneAccumulator.comparable(batch); if (ptype != null && !(data instanceof NullableData)) { // the common case, on the writer's critical path: the vectorized kernel, no widened copy @@ -104,7 +119,7 @@ NodeStats stats() { /// The folded stats as Java values: a `Long` for an integer (a `U64` as its bits), a `Float` for /// an `F16`/`F32` bound and a `Double` for an `F64` one, a `Boolean`, a `BigDecimal`, a `String` - /// for UTF-8 and a `byte[]` for binary. + /// for UTF-8, a `byte[]` for binary and a `UUID` for `vortex.uuid`. /// /// @return the same stats as [#stats()]; [ArrayStats#empty()] where that is [NodeStats#NONE] ArrayStats arrayStats() { @@ -144,10 +159,58 @@ private Object javaValue(Object value) { yield value; } case UTF8 -> new String((byte[]) value, StandardCharsets.UTF_8); + case NULL_COUNT_ONLY -> uuid((byte[]) value); default -> value; }; } + private static UUID uuid(byte[] bytes) { + long msb = 0; + long lsb = 0; + for (int b = 0; b < 8; b++) { + msb = (msb << 8) | (bytes[b] & 0xFFL); + lsb = (lsb << 8) | (bytes[8 + b] & 0xFFL); + } + return new UUID(msb, lsb); + } + + /// A batch of a `vortex.uuid` column: its null rows, and its valid rows' extremes as unsigned + /// 16-byte strings, the order Rust compares a `FixedSizeList(U8, 16)` scalar in. + private void addUuids(Object batch, int rows) { + boolean[] validity = null; + Object storage = batch; + if (batch instanceof NullableData nd) { + validity = nd.validity(); + storage = nd.values(); + } + byte[] flat = (byte[]) ((FixedSizeListData) storage).elements(); + int lo = -1; + int hi = -1; + for (int row = 0; row < rows; row++) { + if (validity != null && !validity[row]) { + nullCount++; + continue; + } + int off = row * 16; + if (lo < 0 || Arrays.compareUnsigned(flat, off, off + 16, flat, lo, lo + 16) < 0) { + lo = off; + } + if (hi < 0 || Arrays.compareUnsigned(flat, off, off + 16, flat, hi, hi + 16) > 0) { + hi = off; + } + } + if (lo >= 0) { + byte[] batchMin = Arrays.copyOfRange(flat, lo, lo + 16); + byte[] batchMax = Arrays.copyOfRange(flat, hi, hi + 16); + if (min == null || Arrays.compareUnsigned(batchMin, (byte[]) min) < 0) { + min = batchMin; + } + if (max == null || Arrays.compareUnsigned(batchMax, (byte[]) max) > 0) { + max = batchMax; + } + } + } + /// Whether the min and max are the column's exact extremes: they are, but for a UTF-8 or binary /// column, whose `BoundedMin`/`BoundedMax` Rust writes as inexact even when no bound was cut. /// @@ -278,10 +341,24 @@ private ProtoScalarValue scalar(Object value) { case DECIMAL -> ProtoScalarValue.ofBytesValue(littleEndian(((BigDecimal) value).unscaledValue(), decimal.byteWidth())); case UTF8 -> ProtoScalarValue.ofStringValue(new String((byte[]) value, StandardCharsets.UTF_8)); case BINARY -> ProtoScalarValue.ofBytesValue((byte[]) value); - case NULL_COUNT_ONLY -> throw new IllegalStateException("no min or max for " + kind); + case NULL_COUNT_ONLY -> { + if (!uuid) { + throw new IllegalStateException("no min or max for " + kind); + } + yield ProtoScalarValue.ofListValue(u8List((byte[]) value)); + } }; } + /// A `vortex.uuid` bound as Rust writes its `FixedSizeList(U8, 16)` scalar: one `u8` per byte. + private static ProtoListValue u8List(byte[] bytes) { + List elements = new ArrayList<>(bytes.length); + for (byte b : bytes) { + elements.add(ProtoScalarValue.ofUint64Value(Byte.toUnsignedLong(b))); + } + return new ProtoListValue(elements); + } + private static byte[] littleEndian(BigInteger unscaled, int width) { // toByteArray is big-endian and minimal: reverse it, then sign-extend to the width byte[] be = unscaled.toByteArray(); diff --git a/writer/src/test/java/io/github/dfa1/vortex/writer/FileStatsAccumulatorTest.java b/writer/src/test/java/io/github/dfa1/vortex/writer/FileStatsAccumulatorTest.java index 6052922a5..062ef793f 100644 --- a/writer/src/test/java/io/github/dfa1/vortex/writer/FileStatsAccumulatorTest.java +++ b/writer/src/test/java/io/github/dfa1/vortex/writer/FileStatsAccumulatorTest.java @@ -1,11 +1,19 @@ package io.github.dfa1.vortex.writer; +import io.github.dfa1.vortex.core.model.ArrayStats; import io.github.dfa1.vortex.core.model.DTypes; +import io.github.dfa1.vortex.core.proto.ProtoListValue; import io.github.dfa1.vortex.core.proto.ProtoScalarValue; import io.github.dfa1.vortex.writer.encode.NodeStats; import io.github.dfa1.vortex.writer.encode.NullableData; +import io.github.dfa1.vortex.writer.encode.UuidExtensionEncoder; import org.junit.jupiter.api.Test; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.List; +import java.util.UUID; + import static org.assertj.core.api.Assertions.assertThat; /// The accumulation rules of [FileStatsAccumulator], each on its own; `FileStatisticsInteropIntegrationTest` @@ -118,4 +126,79 @@ void nullDtype_countsEveryRowNull() { // Then assertThat(result).isEqualTo(new NodeStats(null, null, null, null, null, 4L, 0L, null)); } + + @Test + void uuid_minAndMaxInUnsignedByteOrder() { + // Given — 0x80… sorts above 0x7f… as unsigned bytes, though it is the smaller signed long, + // which is how UUID.compareTo would order them + UUID high = UUID.fromString("80000000-0000-0000-0000-000000000000"); + UUID low = UUID.fromString("7fffffff-ffff-ffff-ffff-ffffffffffff"); + FileStatsAccumulator sut = new FileStatsAccumulator(UuidExtensionEncoder.INSTANCE.dtype(false)); + sut.add(UuidExtensionEncoder.INSTANCE.encodeAll(UuidExtensionEncoder.INSTANCE.dtype(false), List.of(high, low)), 2); + + // When + ArrayStats result = sut.arrayStats(); + + // Then + assertThat(result.min()).isEqualTo(low); + assertThat(result.max()).isEqualTo(high); + assertThat(result.minExact()).isTrue(); + assertThat(result.maxExact()).isTrue(); + } + + @Test + void uuid_boundIsAListOfSixteenU8Scalars() { + // Given + UUID value = UUID.fromString("00010203-0405-0607-0809-0a0b0c0d0eff"); + FileStatsAccumulator sut = new FileStatsAccumulator(UuidExtensionEncoder.INSTANCE.dtype(false)); + sut.add(UuidExtensionEncoder.INSTANCE.encodeAll(UuidExtensionEncoder.INSTANCE.dtype(false), List.of(value)), 1); + + // When + NodeStats result = sut.stats(); + + // Then — Rust's FixedSizeList(U8, 16) scalar: one u8 per byte, big-endian as stored + List expected = new ArrayList<>(); + for (int b : new int[]{0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 255}) { + expected.add(ProtoScalarValue.ofUint64Value((long) b)); + } + assertThat(result.min()).isEqualTo(ProtoScalarValue.ofListValue(new ProtoListValue(expected))); + assertThat(result.max()).isEqualTo(result.min()); + } + + @Test + void uuid_nullsCountAndKeepTheOtherBatchesBounds() { + // Given — a nullable column: an all-null batch adds no bound + UUID a = UUID.fromString("00000000-0000-0000-0000-000000000002"); + UUID b = UUID.fromString("00000000-0000-0000-0000-000000000001"); + UUID c = UUID.fromString("ffffffff-ffff-ffff-ffff-ffffffffffff"); + var dtype = UuidExtensionEncoder.INSTANCE.dtype(true); + FileStatsAccumulator sut = new FileStatsAccumulator(dtype); + sut.add(UuidExtensionEncoder.INSTANCE.encodeAll(dtype, Arrays.asList(a, null)), 2); + sut.add(UuidExtensionEncoder.INSTANCE.encodeAll(dtype, Arrays.asList(null, null)), 2); + sut.add(UuidExtensionEncoder.INSTANCE.encodeAll(dtype, Arrays.asList(c, b)), 2); + + // When + ArrayStats result = sut.arrayStats(); + + // Then + assertThat(result.min()).isEqualTo(b); + assertThat(result.max()).isEqualTo(c); + assertThat(result.nullCount()).isEqualTo(3L); + } + + @Test + void uuidAllNull_hasNoBound() { + // Given + var dtype = UuidExtensionEncoder.INSTANCE.dtype(true); + FileStatsAccumulator sut = new FileStatsAccumulator(dtype); + sut.add(UuidExtensionEncoder.INSTANCE.encodeAll(dtype, Arrays.asList(null, null, null)), 3); + + // When + NodeStats result = sut.stats(); + + // Then + assertThat(result.min()).isNull(); + assertThat(result.max()).isNull(); + assertThat(result.nullCount()).isEqualTo(3L); + } }