Skip to content

test(spark): characterize ArrowUtils Arrow-to-Spark type mapping - #8770

Merged
robert3005 merged 1 commit into
vortex-data:developfrom
jackylee-ch:spark-arrowutils-tests
Jul 15, 2026
Merged

robert3005 merged 1 commit into
vortex-data:developfrom
jackylee-ch:spark-arrowutils-tests

Conversation

@jackylee-ch

Copy link
Copy Markdown
Contributor

Rationale for this change

ArrowUtils (java/vortex-spark/src/main/java/dev/vortex/spark/ArrowUtils.java) maps
Arrow types to Spark SQL DataTypes on the read path, and currently has no test
coverage
. It has several branches that deliberately reject type configurations
(unsigned ints, half floats, non-DAY dates, non-microsecond timestamps, unrecognized
types), and the mapping table itself is exactly the kind of code that regresses quietly.

What changes are included in this PR?

A new ArrowUtilsTest that characterizes both directions of the mapping:

  • Supported mappings: Bool; signed ints 8/16/32/64 → Byte/Short/Integer/Long;
    SINGLE/DOUBLE float → Float/Double; Decimal (precision + scale preserved);
    Utf8/LargeUtf8 → StringType; Binary/LargeBinary → BinaryType;
    Date(DAY) → DateType; Timestamp(MICROSECOND) → TimestampType with a timezone
    and TimestampNTZType without; Null → NullType; and the nested fromArrowField
    cases for Struct and List (carrying element nullability).
  • Rejected configurations: unsigned integers, half-precision float, non-DAY date
    units, and non-microsecond timestamp units all raise UnsupportedOperationException;
    an unrecognized Arrow type raises IllegalArgumentException.

This is a test-only change; no production code is modified. 16 tests, all passing
locally (:vortex-spark_2.12:test --tests 'dev.vortex.spark.ArrowUtilsTest').

What APIs are changed? Are there any user-facing changes?

None. No production code or public API is touched.

`ArrowUtils` maps Arrow types to Spark SQL `DataType`s and had no test
coverage. Add `ArrowUtilsTest` pinning down both directions of its behavior:

- supported mappings: Bool, signed ints (8/16/32/64), single/double float,
  decimal (precision+scale preserved), Utf8/LargeUtf8, Binary/LargeBinary,
  Date(DAY), Timestamp(MICROSECOND) with/without timezone, and Null; plus the
  nested `fromArrowField` cases for Struct and List (element nullability).
- rejected configurations: unsigned integers, half-precision float, non-DAY
  date units, and non-microsecond timestamp units all raise
  UnsupportedOperationException; an unrecognized type raises
  IllegalArgumentException.

Test-only change; no production code is modified.

Signed-off-by: jackylee <qcsd2011@gmail.com>
@robert3005 robert3005 added the changelog/chore A trivial change label Jul 15, 2026

@robert3005 robert3005 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, I will have a pr to improve the completeness of these cases

@robert3005
robert3005 merged commit 6b27c45 into vortex-data:develop Jul 15, 2026
79 of 80 checks passed
robert3005 pushed a commit that referenced this pull request Jul 16, 2026
…materialization (#8783)

## Rationale for this change

`PartitionPathUtils`
(`java/vortex-spark/src/main/java/dev/vortex/spark/read/PartitionPathUtils.java`)
discovers Hive-style partition columns from file paths and materializes
them as constant
column vectors on the read path. It currently has no test coverage,
despite encoding
several subtle behaviors: URL decoding of keys and values, the
`__HIVE_DEFAULT_PARTITION__` null sentinel, first-`=` splitting, type
inference
precedence (int → long → double → boolean → string), and per-type
parsing into
`ConstantColumnVector`. This continues the characterization-test series
from #8770
(read-path `ArrowUtils`) and #8782 (write-path `SparkToArrowSchema`).

## What changes are included in this PR?

A new `PartitionPathUtilsTest` covering all three public methods:

- **`parsePartitionValues`**: extracts ordered `key=value` segments from
paths; ignores
segments without the `key=value` shape (including empty keys or values);
URL-decodes
both keys and values (`New%20York` → `New York`); splits on the first
`=` so values
  may themselves contain `=`.
- **`inferPartitionColumnType`**: infers `Integer` for int-width values,
`Long` for
wider integrals, `Double` for decimal-point values, `Boolean` for
case-insensitive
  true/false, and falls back to `String` (including for null and the
  `__HIVE_DEFAULT_PARTITION__` sentinel).
- **`createConstantVector`**: null and the Hive default-partition
sentinel produce null
vectors; string/int/long/short/byte/boolean/float/double values are
parsed into their
target types; `DateType` parses as days-since-epoch (int) and both
timestamp types as
micros-since-epoch (long); unrecognized target types fall back to UTF8
strings.

This is a **test-only** change; no production code is modified. 19
tests, all passing
locally (`:vortex-spark_2.12:test --tests
'dev.vortex.spark.read.PartitionPathUtilsTest'`).

## What APIs are changed? Are there any user-facing changes?

None. No production code or public API is touched.

Signed-off-by: jackylee <qcsd2011@gmail.com>
robert3005 pushed a commit that referenced this pull request Jul 16, 2026
#8782)

## Rationale for this change

`SparkToArrowSchema`
(`java/vortex-spark/src/main/java/dev/vortex/spark/write/SparkToArrowSchema.java`)
converts Spark SQL schemas to Arrow schemas on the **write path**, and
currently has no
test coverage. It is the mirror image of `ArrowUtils` on the read path,
which gained
characterization tests in #8770. Like that mapping table, this
conversion is exactly the
kind of code that regresses quietly: a wrong bit width, a dropped
timezone, or lost
nullability would silently corrupt written files rather than fail
loudly.

## What changes are included in this PR?

A new `SparkToArrowSchemaTest` that characterizes the conversion:

- **Primitive mappings**: `Boolean` → `Bool`; `Byte/Short/Integer/Long`
→ signed `Int`
8/16/32/64; `Float/Double` → `SINGLE/DOUBLE` floating point; `String` →
`Utf8`;
`Binary` → `Binary`; `Decimal` → 128-bit Arrow `Decimal` preserving
precision and
scale; `Date` → `Date(DAY)`; `Timestamp` → `Timestamp(MICROSECOND,
"UTC")` and
  `TimestampNTZ` → `Timestamp(MICROSECOND, null)`.
- **Schema structure**: field names and ordering are preserved;
field-level nullability
  is carried over to the Arrow field.
- **Nested types**: `StructType` converts to a `Struct` field with
recursively converted
children (including per-child nullability); `ArrayType` converts to a
`List` whose
  `element` child carries `containsNull`.
- **Rejected types**: unsupported Spark types (e.g.
`CalendarIntervalType`) raise
  `UnsupportedOperationException` naming the offending type.

This is a **test-only** change; no production code is modified. 12
tests, all passing
locally (`:vortex-spark_2.12:test --tests
'dev.vortex.spark.write.SparkToArrowSchemaTest'`).

## What APIs are changed? Are there any user-facing changes?

None. No production code or public API is touched.

Signed-off-by: jackylee <qcsd2011@gmail.com>
robert3005 pushed a commit that referenced this pull request Aug 3, 2026
…8859)

## Rationale for this change

`VortexBatchExec`
(`java/vortex-spark/src/main/java/dev/vortex/spark/read/VortexBatchExec.java`)
turns the scan's resolved paths into Spark `InputPartition`s, and it
currently has no
direct test coverage. Its planning contract is easy to break silently:
each `.vortex`
file must become exactly one `VortexFilePartition`, each partition must
carry the
Hive-style partition values parsed from *its own* path (not a
neighbour's), and the
read schema and format options must survive into every partition — these
are what the
executor-side reader relies on to materialize partition columns
correctly. This
continues the characterization-test series from #8770, #8782, #8783,
#8850, and #8857.

## What changes are included in this PR?

A new `VortexBatchExecTest` with 5 tests covering `planInputPartitions`
for explicit
`.vortex` paths (which never touch native directory listing, keeping
this a pure unit
test):

- one `VortexFilePartition` per file, each holding exactly one path;
- partitions preserve the input path order;
- the requested read schema is carried into every partition;
- Hive-style partition values are parsed from each file's own path
(`year=2024/month=01`
vs `year=2025/month=02` vs an unpartitioned path yielding an empty map);
- format options are propagated to every partition.

This is a **test-only** change; no production code is modified. All 5
tests pass
locally on both matrix legs (`:vortex-spark_2.12:test` and
`:vortex-spark_2.13:test`
with `--tests 'dev.vortex.spark.read.VortexBatchExecTest'`).

## What APIs are changed? Are there any user-facing changes?

None. No production code or public API is touched.

Signed-off-by: jackylee <qcsd2011@gmail.com>
robert3005 pushed a commit that referenced this pull request Aug 3, 2026
…ing (#8850)

## Rationale for this change

`HadoopUtils`
(`java/vortex-spark/src/main/java/dev/vortex/spark/config/HadoopUtils.java`)
is the bridge that forwards S3 and Azure credentials from the Spark
session's Hadoop
`Configuration` into the Vortex object-store property keys, and it
currently has no test
coverage. A silently broken key mapping here doesn't fail loudly — it
surfaces as
authentication errors (or unintentionally anonymous access via the Azure
`skip_signature` fallback) only when running against real cloud storage.
This continues
the characterization-test series from #8770 (`ArrowUtils`), #8782
(`SparkToArrowSchema`),
and #8783 (`PartitionPathUtils`).

## What changes are included in this PR?

A new `HadoopUtilsTest` covering both public methods:

- **`s3PropertiesFromHadoopConf`**: maps `fs.s3a.access.key` /
`fs.s3a.secret.key` /
`fs.s3a.session.token` / `fs.s3a.endpoint.region` to the corresponding
`aws_*` keys;
qualifies a bare `fs.s3a.endpoint` with `https://` while preserving
endpoints that
already carry an `http://` or `https://` scheme; ignores unrelated
Hadoop keys and
  returns an empty map for an empty configuration.
- **`azurePropertiesFromHadoopConf`**: extracts the storage account key
from any
  `fs.azure.account.key`-prefixed entry and the SAS token from any
`fs.azure.sas.fixed.token.`-prefixed entry; sets
`azure_skip_signature=true` whenever
no account key is present (including the SAS-token-only case); ignores
unrelated
  `fs.azure.*` keys.

This is a **test-only** change; no production code is modified. 10
tests, passing
locally on both matrix legs (`:vortex-spark_2.12:test` and
`:vortex-spark_2.13:test`
with `--tests 'dev.vortex.spark.config.HadoopUtilsTest'`).

## What APIs are changed? Are there any user-facing changes?

None. No production code or public API is touched.

Signed-off-by: jackylee <qcsd2011@gmail.com>
robert3005 pushed a commit that referenced this pull request Aug 6, 2026
…ing (#8857)

## Rationale for this change

`VortexScanBuilder`
(`java/vortex-spark/src/main/java/dev/vortex/spark/read/VortexScanBuilder.java`)
decides which Spark V2 predicates are pushed into the Vortex scan and
which are returned
to Spark for post-scan evaluation, and it currently has no direct test
coverage. The
split logic encodes correctness-critical rules: predicates referencing
partition columns
must *not* be pushed (partition values live in directory paths, not
inside the Vortex
files), untranslatable operators must be returned to Spark, and
`pushedPredicates()`
must not expose internal state. A regression in any of these silently
drops filters or
scans the wrong data. This continues the characterization-test series
from #8770,
#8782, #8783, and #8850.

## What changes are included in this PR?

A new `VortexScanBuilderTest` with 9 tests covering:

- **`pushPredicates` on data columns**: a supported predicate on a data
column is pushed
and removed from the post-scan set; predicates referencing columns
absent from the
read schema, or using operators `SparkPredicateToVortexExpression`
cannot translate
(e.g. a raw `LIKE`), are returned to Spark and never recorded as pushed.
- **`pushPredicates` on partition columns**: a predicate on an
identity-partitioned
column is left to Spark; mixed batches split correctly (data-column
predicate pushed,
partition-column predicate post-scan); an `And` spanning a data column
and a partition
  column is kept whole on the Spark side.
- **`pushedPredicates`**: empty before any push, and returns a defensive
copy so callers
  cannot mutate the builder's internal state.
- **`build`**: fails fast with `IllegalStateException` when no paths
were added.

This is a **test-only** change; no production code is modified. All
tests pass locally
on both matrix legs (`:vortex-spark_2.12:test` and
`:vortex-spark_2.13:test` with
`--tests 'dev.vortex.spark.read.VortexScanBuilderTest'`).

## What APIs are changed? Are there any user-facing changes?

None. No production code or public API is touched.

Signed-off-by: jackylee <qcsd2011@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/chore A trivial change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants