Repository navigation
fix(python): keep the scheme of object store paths in the Ray datasource - #11046
Conversation
Ray's path resolution returns object store paths without their scheme, so VortexDatasource(url="s3://bucket/data/") passed bucket/data/01.vortex to vortex.open, which looked for it on the local disk and failed. When Ray resolves the URL to a non-local filesystem, put the URL's scheme back on each listed path and escape it. Signed-off-by: jackylee-ch <qcsd2011@gmail.com>
Merging this PR will not alter performance1 warning
Comparing |
Signed-off-by: jackylee-ch <qcsd2011@gmail.com>
Since vortex-data#11011 the datasource wraps the filesystem Ray resolves in RetryingPyFileSystem, so isinstance(fs, LocalFileSystem) was false for local paths and they got a scheme. Check the filesystem it wraps. Signed-off-by: jackylee-ch <qcsd2011@gmail.com>
Head branch was pushed to by a user without write access
|
Python (test) failed after merging develop: #11011 wraps Ray's filesystem in |
Taking the scheme from the URL sent fsspec+file:// URLs, which develop read as local paths, to vortex.open as fsspec+file:// paths it rejects, and gave gcs:// paths a scheme Vortex does not accept. Take it from the filesystem Ray resolved: its type_name, or the protocol of an fsspec filesystem, with gcs mapped to gs. Local filesystems keep plain paths. Signed-off-by: jackylee-ch <qcsd2011@gmail.com>
|
Pushed 3a83919 after the approval: the scheme now comes from the filesystem Ray resolved instead of the URL. Taking it from the URL broke |
Signed-off-by: jackylee-ch <qcsd2011@gmail.com>
Ray's path resolution returns object store paths without their scheme, so
VortexDatasource(url="s3://bucket/data/")listedbucket/data/01.vortexandvortex.openlooked for it on the local disk: the read failed withNo such file or directory (os error 2).The datasource now puts a scheme back on each listed path, escaped so a key with
#or%reaches the right object. The scheme comes from the filesystem Ray resolved, pyarrow'stype_nameor an fsspec filesystem's protocol, so agcs://URL getsgs://, whichvortex.openaccepts. Local filesystems, including fsspec's forfsspec+file://, keep plain paths and read as before.An Azure URL's account is not kept, and schemes pyarrow cannot list, such as
oss://, still produce an empty dataset.Tests
The new test lists and reads two files, one with
#and%in its key, from a local server that answers the S3 requests pyarrow and Vortex make. It fails ondevelop. The local read test also runs withfile://andfsspec+file://URLs, and a unit test covers the GCS scheme.AI assistance
Written with agentic AI assistance.