Skip to content

test(spark): characterize PartitionPathUtils partition discovery and materialization - #8783

Merged
robert3005 merged 1 commit into
vortex-data:developfrom
jackylee-ch:spark-partition-path-utils-tests
Jul 16, 2026
Merged

robert3005 merged 1 commit into
vortex-data:developfrom
jackylee-ch:spark-partition-path-utils-tests

Conversation

@jackylee-ch

Copy link
Copy Markdown
Contributor

Rationale for this change

PartitionPathUtils (java/vortex-spark/src/main/java/dev/vortex/spark/read/PartitionPathUtils.java)
discovers Hive-style partition columns from file paths and materializes them as constant
column vectors on the read path. It currently has no test coverage, despite encoding
several subtle behaviors: URL decoding of keys and values, the
__HIVE_DEFAULT_PARTITION__ null sentinel, first-= splitting, type inference
precedence (int → long → double → boolean → string), and per-type parsing into
ConstantColumnVector. This continues the characterization-test series from #8770
(read-path ArrowUtils) and #8782 (write-path SparkToArrowSchema).

What changes are included in this PR?

A new PartitionPathUtilsTest covering all three public methods:

  • parsePartitionValues: extracts ordered key=value segments from paths; ignores
    segments without the key=value shape (including empty keys or values); URL-decodes
    both keys and values (New%20York → New York); splits on the first = so values
    may themselves contain =.
  • inferPartitionColumnType: infers Integer for int-width values, Long for
    wider integrals, Double for decimal-point values, Boolean for case-insensitive
    true/false, and falls back to String (including for null and the
    __HIVE_DEFAULT_PARTITION__ sentinel).
  • createConstantVector: null and the Hive default-partition sentinel produce null
    vectors; string/int/long/short/byte/boolean/float/double values are parsed into their
    target types; DateType parses as days-since-epoch (int) and both timestamp types as
    micros-since-epoch (long); unrecognized target types fall back to UTF8 strings.

This is a test-only change; no production code is modified. 19 tests, all passing
locally (:vortex-spark_2.12:test --tests 'dev.vortex.spark.read.PartitionPathUtilsTest').

What APIs are changed? Are there any user-facing changes?

None. No production code or public API is touched.

…materialization

Signed-off-by: jackylee <qcsd2011@gmail.com>
@jackylee-ch
jackylee-ch force-pushed the spark-partition-path-utils-tests branch from 8a4cb75 to 5b652ef Compare July 16, 2026 09:16
@robert3005 robert3005 added the changelog/chore A trivial change label Jul 16, 2026
@robert3005
robert3005 merged commit d54f70d into vortex-data:develop Jul 16, 2026
79 of 80 checks passed
robert3005 pushed a commit that referenced this pull request Aug 3, 2026
…8859)

## Rationale for this change

`VortexBatchExec`
(`java/vortex-spark/src/main/java/dev/vortex/spark/read/VortexBatchExec.java`)
turns the scan's resolved paths into Spark `InputPartition`s, and it
currently has no
direct test coverage. Its planning contract is easy to break silently:
each `.vortex`
file must become exactly one `VortexFilePartition`, each partition must
carry the
Hive-style partition values parsed from *its own* path (not a
neighbour's), and the
read schema and format options must survive into every partition — these
are what the
executor-side reader relies on to materialize partition columns
correctly. This
continues the characterization-test series from #8770, #8782, #8783,
#8850, and #8857.

## What changes are included in this PR?

A new `VortexBatchExecTest` with 5 tests covering `planInputPartitions`
for explicit
`.vortex` paths (which never touch native directory listing, keeping
this a pure unit
test):

- one `VortexFilePartition` per file, each holding exactly one path;
- partitions preserve the input path order;
- the requested read schema is carried into every partition;
- Hive-style partition values are parsed from each file's own path
(`year=2024/month=01`
vs `year=2025/month=02` vs an unpartitioned path yielding an empty map);
- format options are propagated to every partition.

This is a **test-only** change; no production code is modified. All 5
tests pass
locally on both matrix legs (`:vortex-spark_2.12:test` and
`:vortex-spark_2.13:test`
with `--tests 'dev.vortex.spark.read.VortexBatchExecTest'`).

## What APIs are changed? Are there any user-facing changes?

None. No production code or public API is touched.

Signed-off-by: jackylee <qcsd2011@gmail.com>
robert3005 pushed a commit that referenced this pull request Aug 3, 2026
…ing (#8850)

## Rationale for this change

`HadoopUtils`
(`java/vortex-spark/src/main/java/dev/vortex/spark/config/HadoopUtils.java`)
is the bridge that forwards S3 and Azure credentials from the Spark
session's Hadoop
`Configuration` into the Vortex object-store property keys, and it
currently has no test
coverage. A silently broken key mapping here doesn't fail loudly — it
surfaces as
authentication errors (or unintentionally anonymous access via the Azure
`skip_signature` fallback) only when running against real cloud storage.
This continues
the characterization-test series from #8770 (`ArrowUtils`), #8782
(`SparkToArrowSchema`),
and #8783 (`PartitionPathUtils`).

## What changes are included in this PR?

A new `HadoopUtilsTest` covering both public methods:

- **`s3PropertiesFromHadoopConf`**: maps `fs.s3a.access.key` /
`fs.s3a.secret.key` /
`fs.s3a.session.token` / `fs.s3a.endpoint.region` to the corresponding
`aws_*` keys;
qualifies a bare `fs.s3a.endpoint` with `https://` while preserving
endpoints that
already carry an `http://` or `https://` scheme; ignores unrelated
Hadoop keys and
  returns an empty map for an empty configuration.
- **`azurePropertiesFromHadoopConf`**: extracts the storage account key
from any
  `fs.azure.account.key`-prefixed entry and the SAS token from any
`fs.azure.sas.fixed.token.`-prefixed entry; sets
`azure_skip_signature=true` whenever
no account key is present (including the SAS-token-only case); ignores
unrelated
  `fs.azure.*` keys.

This is a **test-only** change; no production code is modified. 10
tests, passing
locally on both matrix legs (`:vortex-spark_2.12:test` and
`:vortex-spark_2.13:test`
with `--tests 'dev.vortex.spark.config.HadoopUtilsTest'`).

## What APIs are changed? Are there any user-facing changes?

None. No production code or public API is touched.

Signed-off-by: jackylee <qcsd2011@gmail.com>
robert3005 pushed a commit that referenced this pull request Aug 6, 2026
…ing (#8857)

## Rationale for this change

`VortexScanBuilder`
(`java/vortex-spark/src/main/java/dev/vortex/spark/read/VortexScanBuilder.java`)
decides which Spark V2 predicates are pushed into the Vortex scan and
which are returned
to Spark for post-scan evaluation, and it currently has no direct test
coverage. The
split logic encodes correctness-critical rules: predicates referencing
partition columns
must *not* be pushed (partition values live in directory paths, not
inside the Vortex
files), untranslatable operators must be returned to Spark, and
`pushedPredicates()`
must not expose internal state. A regression in any of these silently
drops filters or
scans the wrong data. This continues the characterization-test series
from #8770,
#8782, #8783, and #8850.

## What changes are included in this PR?

A new `VortexScanBuilderTest` with 9 tests covering:

- **`pushPredicates` on data columns**: a supported predicate on a data
column is pushed
and removed from the post-scan set; predicates referencing columns
absent from the
read schema, or using operators `SparkPredicateToVortexExpression`
cannot translate
(e.g. a raw `LIKE`), are returned to Spark and never recorded as pushed.
- **`pushPredicates` on partition columns**: a predicate on an
identity-partitioned
column is left to Spark; mixed batches split correctly (data-column
predicate pushed,
partition-column predicate post-scan); an `And` spanning a data column
and a partition
  column is kept whole on the Spark side.
- **`pushedPredicates`**: empty before any push, and returns a defensive
copy so callers
  cannot mutate the builder's internal state.
- **`build`**: fails fast with `IllegalStateException` when no paths
were added.

This is a **test-only** change; no production code is modified. All
tests pass locally
on both matrix legs (`:vortex-spark_2.12:test` and
`:vortex-spark_2.13:test` with
`--tests 'dev.vortex.spark.read.VortexScanBuilderTest'`).

## What APIs are changed? Are there any user-facing changes?

None. No production code or public API is touched.

Signed-off-by: jackylee <qcsd2011@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/chore A trivial change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants