Repository navigation
[Java] new FileSystem is called repeatedly and cause performance degrade #166
Description
Describe the bug, including details regarding any error messages, version, and platform.
I am integrating Arrow to Spark using jni, it seems FileSystem is created repeatedly and causing performance degrade. like this:
where did the JNI callback to java method ? maybe this Object should be provided in the interface.
Component(s)
Java
Metadata
Metadata
Assignees
Labels
No labels
Activity
[-]new FileSystem is called repeatedly and cause performance degrade[/-][+][Java] new FileSystem is called repeatedly and cause performance degrade[/+]zinking commentedon Sep 11, 2023
it seems the dataset creation api will always create the filesystem from scratch when calling into libHdfs
hdfsBuilderConnectI'd suggest we create a config to pass that fileSystem in, rather than create it.