Skip to content

[Java] while using s3 FileSystemDatasetFactory getting this exception #36069

Closed

Description

@febinct

Describe the bug, including details regarding any error messages, version, and platform.

/Users/voltrondata/github-actions-runner/_work/crossbow/crossbow/arrow/cpp/src/arrow/filesystem/s3fs.cc:2598: arrow::fs::FinalizeS3 was not called even though S3 was initialized. This could lead to a segmentation fault at exit

i think the java client is not closing the s3 client gracefully because of that memory leak happening

Component(s)

Java

Activity

  1. changed the title [-]while using s3 FileSystemDatasetFactory getting this exception[/-] [+][Java] while using s3 FileSystemDatasetFactory getting this exception[/+] on Jun 14, 2023
  2. westonpace commented on Jun 14, 2023

    @westonpace
    Member

    CC @davisusanibar @lidavidm (note that this warning was newly added to the S3 filesystem in the previous release so it is very possible the Java implementation has never been calling finalize)

  3. davisusanibar commented on Jun 14, 2023

    @davisusanibar
    Contributor

    CC @davisusanibar @lidavidm (note that this warning was newly added to the S3 filesystem in the previous release so it is very possible the Java implementation has never been calling finalize)

    Just able to reproduce this warning with:

    import org.apache.arrow.dataset.file.FileFormat;
    import org.apache.arrow.dataset.file.FileSystemDatasetFactory;
    import org.apache.arrow.dataset.jni.NativeMemoryPool;
    import org.apache.arrow.dataset.scanner.ScanOptions;
    import org.apache.arrow.dataset.scanner.Scanner;
    import org.apache.arrow.dataset.source.Dataset;
    import org.apache.arrow.dataset.source.DatasetFactory;
    import org.apache.arrow.memory.BufferAllocator;
    import org.apache.arrow.memory.RootAllocator;
    import org.apache.arrow.vector.ipc.ArrowReader;
    import org.apache.arrow.vector.types.pojo.Schema;
    
    public class DatasetModule {
        public static void main(String[] args) {
            String uri = "s3://voltrondata-labs-datasets/nyc-taxi-tiny/year=2022/month=2/part-0.parquet"; // AWS S3
            // String uri = "hdfs://{hdfs_host}:{port}/nyc-taxi-tiny/year=2022/month=2/part-0.parquet"; // HDFS
            // String uri = "gs://voltrondata-labs-datasets/nyc-taxi-tiny/year=2022/month=2/part-0.parquet"; // Google Cloud Storage
            ScanOptions options = new ScanOptions(/*batchSize*/ 32768);
            try (
                BufferAllocator allocator = new RootAllocator();
                DatasetFactory datasetFactory = new FileSystemDatasetFactory(allocator, NativeMemoryPool.getDefault(), FileFormat.PARQUET, uri);
                Dataset dataset = datasetFactory.finish();
                Scanner scanner = dataset.newScan(options);
                ArrowReader reader = scanner.scanBatches()
            ) {
                Schema schema = scanner.schema();
                System.out.println(schema);
                while (reader.loadNextBatch()) {
                    System.out.println("RowCount: " + reader.getVectorSchemaRoot().getRowCount());
                }
            } catch (Exception e) {
                e.printStackTrace();
            }
        }
    }

    Output messages:

    RowCount: 2979
    /Users/runner/work/crossbow/crossbow/arrow/cpp/src/arrow/filesystem/s3fs.cc:2598:  arrow::fs::FinalizeS3 was not called even though S3 was initialized.  This could lead to a segmentation fault at exit
    

    Next step:

    1. Review reason of error messages
    2. Add Arrow Java cookbook to cover S3 integration
  4. febinsathar commented on Jun 15, 2023

    @febinsathar

    while running s3 file reader using java after for sometime the machine is crash looping due to this AWS Error NETWORK_CONNECTION during HeadObject operation: curlCode: 43, A libcurl function was given a bad argument
    this generally happens when aws s3 connection pool are not closed AFAIK

  5. febinct commented on Jul 21, 2023

    @febinct
    Author

    this is in java not python

  6. mapleFU commented on Jul 23, 2023

    @mapleFU
    Member

    I guess the problem in introduced in #33858 . The description contains:

    BREAKING CHANGE: S3 can only be initialized and finalized once.
    BREAKING CHANGE: S3 (the AWS SDK) will not be finalized until after all CPU & I/O threads are finished.

    A helpful way is calling FinalizeS3 when exit. This patch is in arrow-12.0

    However, I guess this patch might fixed it: #36442 . Maybe you can confirm it latter?

  7. kou commented on Jul 24, 2023

    @kou
    Member

    AFAIK, there is no pyarrow package for Java.

    Java's dataset API uses JNI. It means that Java's dataset API calls C++ implementation directly. (It doesn't use Python.)

    We need to implement a Java binding for arrow::fs::FinalizeS3() and users need to call it explicitly.
    Or we may be able to do it implicitly if Java provides an atexit() like hook.

  8. lidavidm commented on Jul 24, 2023

    @lidavidm
    Member

    Java has Runtime#addShutdownHook

  9. lidavidm commented on Jul 24, 2023

    @lidavidm
    Member

    @davisusanibar @danepitkin any interest in this?

  10. added this to the 14.0.0 milestone on Jul 24, 2023
  11. danepitkin commented on Jul 24, 2023

    @danepitkin
    Member

    Yes! We can take this on. Thanks for the ping.

  12. added a commit that references this issue on Jul 28, 2023
  13. added a commit that references this issue on Aug 2, 2023
  14. added a commit that references this issue on Nov 13, 2023
  15. added a commit that references this issue on Oct 24, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions