Skip to content

fix(python): keep the scheme of object store paths in the Ray datasource - #11046

Merged
robert3005 merged 5 commits into
vortex-data:developfrom
jackylee-ch:fix/ray-s3-paths
Nov 2, 2026
Merged

robert3005 merged 5 commits into
vortex-data:developfrom
jackylee-ch:fix/ray-s3-paths

Conversation

@jackylee-ch

@jackylee-ch jackylee-ch commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Ray's path resolution returns object store paths without their scheme, so VortexDatasource(url="s3://bucket/data/") listed bucket/data/01.vortex and vortex.open looked for it on the local disk: the read failed with No such file or directory (os error 2).

The datasource now puts a scheme back on each listed path, escaped so a key with # or % reaches the right object. The scheme comes from the filesystem Ray resolved, pyarrow's type_name or an fsspec filesystem's protocol, so a gcs:// URL gets gs://, which vortex.open accepts. Local filesystems, including fsspec's for fsspec+file://, keep plain paths and read as before.

An Azure URL's account is not kept, and schemes pyarrow cannot list, such as oss://, still produce an empty dataset.

Tests

The new test lists and reads two files, one with # and % in its key, from a local server that answers the S3 requests pyarrow and Vortex make. It fails on develop. The local read test also runs with file:// and fsspec+file:// URLs, and a unit test covers the GCS scheme.

AI assistance

Written with agentic AI assistance.

Ray's path resolution returns object store paths without their scheme,
so VortexDatasource(url="s3://bucket/data/") passed bucket/data/01.vortex
to vortex.open, which looked for it on the local disk and failed. When
Ray resolves the URL to a non-local filesystem, put the URL's scheme back
on each listed path and escape it.

Signed-off-by: jackylee-ch <qcsd2011@gmail.com>
@robert3005 robert3005 added the changelog/fix A bug fix label Oct 30, 2026
@robert3005
robert3005 enabled auto-merge (squash) October 30, 2026 16:05
@codspeed

codspeed Bot commented Oct 30, 2026 •

Copy link
Copy Markdown

Merging this PR will not alter performance

1 warning

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

✅ 2318 untouched benchmarks


Comparing jackylee-ch:fix/ray-s3-paths (adadbb0) with develop (b9b02f6)

Open in CodSpeed

Signed-off-by: jackylee-ch <qcsd2011@gmail.com>
Since vortex-data#11011 the datasource wraps the filesystem Ray resolves in
RetryingPyFileSystem, so isinstance(fs, LocalFileSystem) was false for
local paths and they got a scheme. Check the filesystem it wraps.

Signed-off-by: jackylee-ch <qcsd2011@gmail.com>
auto-merge was automatically disabled October 30, 2026 18:44

Head branch was pushed to by a user without write access

@jackylee-ch

Copy link
Copy Markdown
Contributor Author

Python (test) failed after merging develop: #11011 wraps Ray's filesystem in RetryingPyFileSystem, so the local check missed and local paths got a scheme prefix. The check now unwraps it first. That line and a merge of develop are new since your approval; both failing tests reproduce locally on the merge and pass with the change on develop's lock (Ray 2.59). I also dropped hf:// from the description, because pyarrow resolves it through fsspec when huggingface_hub is installed.

Taking the scheme from the URL sent fsspec+file:// URLs, which develop
read as local paths, to vortex.open as fsspec+file:// paths it rejects,
and gave gcs:// paths a scheme Vortex does not accept. Take it from the
filesystem Ray resolved: its type_name, or the protocol of an fsspec
filesystem, with gcs mapped to gs. Local filesystems keep plain paths.

Signed-off-by: jackylee-ch <qcsd2011@gmail.com>
@jackylee-ch

Copy link
Copy Markdown
Contributor Author

Pushed 3a83919 after the approval: the scheme now comes from the filesystem Ray resolved instead of the URL. Taking it from the URL broke fsspec+file:// URLs, which develop reads as local paths: vortex.open rejected the rewritten fsspec+file:///... path. gcs:// URLs now get gs://, which vortex.open accepts. The local read test also runs with fsspec+file://, and a unit test covers GCS.

Signed-off-by: jackylee-ch <qcsd2011@gmail.com>
@robert3005
robert3005 merged commit 581ace7 into vortex-data:develop Nov 2, 2026
73 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/fix A bug fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants