Skip to content

Replace extension examples with independent conformance tests - #11361

Merged
connortsui20 merged 1 commit into
developfrom
ct/arrow-roundtrip-properties
Nov 17, 2026
Merged

connortsui20 merged 1 commit into
developfrom
ct/arrow-roundtrip-properties

Conversation

@connortsui20

@connortsui20 connortsui20 commented Nov 9, 2026 •

Copy link
Copy Markdown
Member

Summary

Builds on #11513.

Round trips can pass when import and export make matching mistakes. Independent Arrow references are needed to check each direction against the intended extension contract.

Changes

Generators stay in their owning crates to avoid adding plugin dependencies to vortex-arrow. Generated cases replace repetitive examples while retaining directed regressions and rejection tests. Native values, aggregates, and tensor layouts are checked separately so transport failures cannot hide failures in those contracts.

Variant remains excluded pending #11359.

Source checks passed. No builds or runtime tests were run for the split.

@connortsui20
connortsui20 added this pull request to stack #11362 November 9, 2026 18:16
@connortsui20 connortsui20 mentioned this pull request Nov 9, 2026
19 of 57 tasks
@connortsui20
connortsui20 force-pushed the ct/arrow-roundtrip-properties branch 5 times, most recently from fc51d62 to f2987e6 Compare November 9, 2026 22:12
@connortsui20
connortsui20 force-pushed the ct/arrow-roundtrip-properties branch from f2987e6 to 73f31f1 Compare November 10, 2026 14:52
@connortsui20
connortsui20 force-pushed the ct/arrow-roundtrip-properties branch 2 times, most recently from f32781e to 73a471c Compare November 10, 2026 16:16
@connortsui20
connortsui20 force-pushed the ct/arrow-roundtrip-properties branch from 73a471c to 53f31ab Compare November 10, 2026 16:52
@connortsui20
connortsui20 removed this pull request from stack #11362 November 10, 2026 16:55
@connortsui20
connortsui20 force-pushed the ct/arrow-roundtrip-properties branch from 53f31ab to d5c2d05 Compare November 10, 2026 18:18
Base automatically changed from ct/arrow-roundtrip-fixes to develop November 10, 2026 23:12
@connortsui20
connortsui20 force-pushed the ct/arrow-roundtrip-properties branch from d5c2d05 to 9e3f744 Compare November 10, 2026 23:12
@connortsui20
connortsui20 marked this pull request as ready for review November 11, 2026 13:26
@connortsui20

Copy link
Copy Markdown
Member Author

I think I want to also delete a lot of the existing tests too

@codspeed

codspeed Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

Merging this PR will not alter performance

1 warning

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

✅ 1965 untouched benchmarks
⏩ 53 skipped benchmarks1


Comparing ct/arrow-roundtrip-properties (fbae5ff) with develop (a59dc52)

Open in CodSpeed

Footnotes

  1. 53 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

@connortsui20
connortsui20 force-pushed the ct/arrow-roundtrip-properties branch 2 times, most recently from 8e2125f to 35520db Compare November 12, 2026 15:51
@connortsui20
connortsui20 force-pushed the ct/arrow-roundtrip-properties branch from 35520db to 0d6def1 Compare November 12, 2026 15:56
@connortsui20
connortsui20 marked this pull request as draft November 12, 2026 16:18
@connortsui20
connortsui20 force-pushed the ct/arrow-roundtrip-properties branch 2 times, most recently from 1ec883e to a65043b Compare November 16, 2026 19:09
@connortsui20
connortsui20 marked this pull request as ready for review November 16, 2026 20:36
@connortsui20
connortsui20 marked this pull request as draft November 16, 2026 20:37
@connortsui20 connortsui20 changed the title Add Hegel coverage for Arrow extension round trips Replace extension examples with independent conformance tests Nov 16, 2026
@connortsui20
connortsui20 force-pushed the ct/arrow-roundtrip-properties branch from fc848fb to a22b2f7 Compare November 16, 2026 22:02
@connortsui20
connortsui20 removed this pull request from stack #11376 November 16, 2026 22:02
@connortsui20
connortsui20 changed the base branch from develop to ct/spatial-filtered-list-export November 16, 2026 22:02
@connortsui20 connortsui20 added the changelog/skip Do not list PR in the changelog label Nov 16, 2026 — with ChatGPT Codex Connector
@connortsui20 connortsui20 removed the changelog/fix A bug fix label Nov 16, 2026
@connortsui20
connortsui20 marked this pull request as ready for review November 16, 2026 22:07
Base automatically changed from ct/spatial-filtered-list-export to develop November 17, 2026 12:43
@connortsui20
connortsui20 force-pushed the ct/arrow-roundtrip-properties branch from a22b2f7 to a3770f9 Compare November 17, 2026 12:43
@connortsui20
connortsui20 enabled stacked auto-merge (squash) November 17, 2026 14:46
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
@connortsui20
connortsui20 force-pushed the ct/arrow-roundtrip-properties branch from a3770f9 to fbae5ff Compare November 17, 2026 15:00
stacked auto-merge was automatically disabled November 17, 2026 15:12

Replaced by a newer auto-merge request

@connortsui20
connortsui20 enabled stacked auto-merge (squash) November 17, 2026 15:12
stacked auto-merge was automatically disabled November 17, 2026 15:27

Auto-merge was disabled for another pull request in the stack

@connortsui20
connortsui20 merged commit fdc113e into develop Nov 17, 2026
76 checks passed
@connortsui20
connortsui20 deleted the ct/arrow-roundtrip-properties branch November 17, 2026 15:27
connortsui20 added a commit that referenced this pull request Nov 17, 2026
## Summary

Tracking Issue: #11114. Builds on #11361.

Known extension importers can silently fall back to plain storage when
Arrow uses a different layout. For example, PyArrow marks list elements
nullable, which can make a vector column lose its `Vector` type. Valid
input needs normalization into the extension's declared storage, while
invalid input needs an error.

## Changes

Arrow permits arbitrary child values under a null parent, so masked
nulls must not cause a false rejection. Visible nulls still fail at
non-nullable levels. The name-only spatial fallback cannot preserve
dimensions and CRS, so spatial metadata must be read fully.

Visibility propagation currently covers structs and fixed-size lists.
List views, maps, dictionaries, and run-end arrays remain follow-ups in
#11114.

Source checks passed. No builds or runtime tests were run for the split.

## API Changes

- Known extension fields now fail on unsupported or malformed input
instead of silently becoming storage arrays.
- Nullable-marked vector elements and spatial coordinates, plus GeoArrow
`LargeList` input, import as their extension when the visible values are
valid.
- `ArrowSession` rejects visible nulls in non-nullable children,
including when the supplied schema is stricter than the Arrow array.
- Adds `ArrowSession::from_arrow_storage`.

---------

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/skip Do not list PR in the changelog

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants