Skip to content

test(spark): characterize SparkToArrowSchema Spark-to-Arrow conversion - #8782

Merged
robert3005 merged 1 commit into
vortex-data:developfrom
jackylee-ch:spark-to-arrow-schema-tests
Jul 16, 2026
Merged

robert3005 merged 1 commit into
vortex-data:developfrom
jackylee-ch:spark-to-arrow-schema-tests

Conversation

@jackylee-ch

Copy link
Copy Markdown
Contributor

Rationale for this change

SparkToArrowSchema (java/vortex-spark/src/main/java/dev/vortex/spark/write/SparkToArrowSchema.java)
converts Spark SQL schemas to Arrow schemas on the write path, and currently has no
test coverage. It is the mirror image of ArrowUtils on the read path, which gained
characterization tests in #8770. Like that mapping table, this conversion is exactly the
kind of code that regresses quietly: a wrong bit width, a dropped timezone, or lost
nullability would silently corrupt written files rather than fail loudly.

What changes are included in this PR?

A new SparkToArrowSchemaTest that characterizes the conversion:

  • Primitive mappings: Boolean → Bool; Byte/Short/Integer/Long → signed Int
    8/16/32/64; Float/Double → SINGLE/DOUBLE floating point; String → Utf8;
    Binary → Binary; Decimal → 128-bit Arrow Decimal preserving precision and
    scale; Date → Date(DAY); Timestamp → Timestamp(MICROSECOND, "UTC") and
    TimestampNTZ → Timestamp(MICROSECOND, null).
  • Schema structure: field names and ordering are preserved; field-level nullability
    is carried over to the Arrow field.
  • Nested types: StructType converts to a Struct field with recursively converted
    children (including per-child nullability); ArrayType converts to a List whose
    element child carries containsNull.
  • Rejected types: unsupported Spark types (e.g. CalendarIntervalType) raise
    UnsupportedOperationException naming the offending type.

This is a test-only change; no production code is modified. 12 tests, all passing
locally (:vortex-spark_2.12:test --tests 'dev.vortex.spark.write.SparkToArrowSchemaTest').

What APIs are changed? Are there any user-facing changes?

None. No production code or public API is touched.

@jackylee-ch
jackylee-ch force-pushed the spark-to-arrow-schema-tests branch from 071404d to fb19e1d Compare July 16, 2026 09:15
robert3005 pushed a commit that referenced this pull request Jul 16, 2026
…materialization (#8783)

## Rationale for this change

`PartitionPathUtils`
(`java/vortex-spark/src/main/java/dev/vortex/spark/read/PartitionPathUtils.java`)
discovers Hive-style partition columns from file paths and materializes
them as constant
column vectors on the read path. It currently has no test coverage,
despite encoding
several subtle behaviors: URL decoding of keys and values, the
`__HIVE_DEFAULT_PARTITION__` null sentinel, first-`=` splitting, type
inference
precedence (int → long → double → boolean → string), and per-type
parsing into
`ConstantColumnVector`. This continues the characterization-test series
from #8770
(read-path `ArrowUtils`) and #8782 (write-path `SparkToArrowSchema`).

## What changes are included in this PR?

A new `PartitionPathUtilsTest` covering all three public methods:

- **`parsePartitionValues`**: extracts ordered `key=value` segments from
paths; ignores
segments without the `key=value` shape (including empty keys or values);
URL-decodes
both keys and values (`New%20York` → `New York`); splits on the first
`=` so values
  may themselves contain `=`.
- **`inferPartitionColumnType`**: infers `Integer` for int-width values,
`Long` for
wider integrals, `Double` for decimal-point values, `Boolean` for
case-insensitive
  true/false, and falls back to `String` (including for null and the
  `__HIVE_DEFAULT_PARTITION__` sentinel).
- **`createConstantVector`**: null and the Hive default-partition
sentinel produce null
vectors; string/int/long/short/byte/boolean/float/double values are
parsed into their
target types; `DateType` parses as days-since-epoch (int) and both
timestamp types as
micros-since-epoch (long); unrecognized target types fall back to UTF8
strings.

This is a **test-only** change; no production code is modified. 19
tests, all passing
locally (`:vortex-spark_2.12:test --tests
'dev.vortex.spark.read.PartitionPathUtilsTest'`).

## What APIs are changed? Are there any user-facing changes?

None. No production code or public API is touched.

Signed-off-by: jackylee <qcsd2011@gmail.com>
@robert3005 robert3005 added the changelog/chore A trivial change label Jul 16, 2026
@robert3005
robert3005 merged commit 8d52fff into vortex-data:develop Jul 16, 2026
88 of 91 checks passed
robert3005 pushed a commit that referenced this pull request Aug 3, 2026
…8859)

## Rationale for this change

`VortexBatchExec`
(`java/vortex-spark/src/main/java/dev/vortex/spark/read/VortexBatchExec.java`)
turns the scan's resolved paths into Spark `InputPartition`s, and it
currently has no
direct test coverage. Its planning contract is easy to break silently:
each `.vortex`
file must become exactly one `VortexFilePartition`, each partition must
carry the
Hive-style partition values parsed from *its own* path (not a
neighbour's), and the
read schema and format options must survive into every partition — these
are what the
executor-side reader relies on to materialize partition columns
correctly. This
continues the characterization-test series from #8770, #8782, #8783,
#8850, and #8857.

## What changes are included in this PR?

A new `VortexBatchExecTest` with 5 tests covering `planInputPartitions`
for explicit
`.vortex` paths (which never touch native directory listing, keeping
this a pure unit
test):

- one `VortexFilePartition` per file, each holding exactly one path;
- partitions preserve the input path order;
- the requested read schema is carried into every partition;
- Hive-style partition values are parsed from each file's own path
(`year=2024/month=01`
vs `year=2025/month=02` vs an unpartitioned path yielding an empty map);
- format options are propagated to every partition.

This is a **test-only** change; no production code is modified. All 5
tests pass
locally on both matrix legs (`:vortex-spark_2.12:test` and
`:vortex-spark_2.13:test`
with `--tests 'dev.vortex.spark.read.VortexBatchExecTest'`).

## What APIs are changed? Are there any user-facing changes?

None. No production code or public API is touched.

Signed-off-by: jackylee <qcsd2011@gmail.com>
robert3005 pushed a commit that referenced this pull request Aug 3, 2026
…ing (#8850)

## Rationale for this change

`HadoopUtils`
(`java/vortex-spark/src/main/java/dev/vortex/spark/config/HadoopUtils.java`)
is the bridge that forwards S3 and Azure credentials from the Spark
session's Hadoop
`Configuration` into the Vortex object-store property keys, and it
currently has no test
coverage. A silently broken key mapping here doesn't fail loudly — it
surfaces as
authentication errors (or unintentionally anonymous access via the Azure
`skip_signature` fallback) only when running against real cloud storage.
This continues
the characterization-test series from #8770 (`ArrowUtils`), #8782
(`SparkToArrowSchema`),
and #8783 (`PartitionPathUtils`).

## What changes are included in this PR?

A new `HadoopUtilsTest` covering both public methods:

- **`s3PropertiesFromHadoopConf`**: maps `fs.s3a.access.key` /
`fs.s3a.secret.key` /
`fs.s3a.session.token` / `fs.s3a.endpoint.region` to the corresponding
`aws_*` keys;
qualifies a bare `fs.s3a.endpoint` with `https://` while preserving
endpoints that
already carry an `http://` or `https://` scheme; ignores unrelated
Hadoop keys and
  returns an empty map for an empty configuration.
- **`azurePropertiesFromHadoopConf`**: extracts the storage account key
from any
  `fs.azure.account.key`-prefixed entry and the SAS token from any
`fs.azure.sas.fixed.token.`-prefixed entry; sets
`azure_skip_signature=true` whenever
no account key is present (including the SAS-token-only case); ignores
unrelated
  `fs.azure.*` keys.

This is a **test-only** change; no production code is modified. 10
tests, passing
locally on both matrix legs (`:vortex-spark_2.12:test` and
`:vortex-spark_2.13:test`
with `--tests 'dev.vortex.spark.config.HadoopUtilsTest'`).

## What APIs are changed? Are there any user-facing changes?

None. No production code or public API is touched.

Signed-off-by: jackylee <qcsd2011@gmail.com>
robert3005 pushed a commit that referenced this pull request Aug 6, 2026
…ing (#8857)

## Rationale for this change

`VortexScanBuilder`
(`java/vortex-spark/src/main/java/dev/vortex/spark/read/VortexScanBuilder.java`)
decides which Spark V2 predicates are pushed into the Vortex scan and
which are returned
to Spark for post-scan evaluation, and it currently has no direct test
coverage. The
split logic encodes correctness-critical rules: predicates referencing
partition columns
must *not* be pushed (partition values live in directory paths, not
inside the Vortex
files), untranslatable operators must be returned to Spark, and
`pushedPredicates()`
must not expose internal state. A regression in any of these silently
drops filters or
scans the wrong data. This continues the characterization-test series
from #8770,
#8782, #8783, and #8850.

## What changes are included in this PR?

A new `VortexScanBuilderTest` with 9 tests covering:

- **`pushPredicates` on data columns**: a supported predicate on a data
column is pushed
and removed from the post-scan set; predicates referencing columns
absent from the
read schema, or using operators `SparkPredicateToVortexExpression`
cannot translate
(e.g. a raw `LIKE`), are returned to Spark and never recorded as pushed.
- **`pushPredicates` on partition columns**: a predicate on an
identity-partitioned
column is left to Spark; mixed batches split correctly (data-column
predicate pushed,
partition-column predicate post-scan); an `And` spanning a data column
and a partition
  column is kept whole on the Spark side.
- **`pushedPredicates`**: empty before any push, and returns a defensive
copy so callers
  cannot mutate the builder's internal state.
- **`build`**: fails fast with `IllegalStateException` when no paths
were added.

This is a **test-only** change; no production code is modified. All
tests pass locally
on both matrix legs (`:vortex-spark_2.12:test` and
`:vortex-spark_2.13:test` with
`--tests 'dev.vortex.spark.read.VortexScanBuilderTest'`).

## What APIs are changed? Are there any user-facing changes?

None. No production code or public API is touched.

Signed-off-by: jackylee <qcsd2011@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/chore A trivial change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants