Skip to content

pq.read_table doesn't seem to be thread safe #33189

Closed as not planned
Closed as not planned

Description

@asfimport

Before PR: #13799 gets merged in master, I am using multithreading to improve read bandwidth from S3. Even after that PR gets merged, I probably will still try to use multithreading to some extent.

However pq.read_table from S3 doesn't seem to be thread safe. Seems like it uses the new dataset reader under the hood. I cannot provide a reproduction, not a stable one anyway. But this is roughly the script I have been using 


def get_next_batch(self, mapper_id, pos=None):

def download(file):
    return pq.read_table("s3://" + self.bucket + "/" + file, columns=self.columns, filters=self.filters)
 
executor = concurrent.futures.ThreadPoolExecutor(max_workers=self.workers)

futures= \{executor.submit(download, file): file for file in my_files}
for future inconcurrent.futures.as_completed(futures):
    yield  future.result()

The errors all have to do with malloc segfaults which makes me suspect the connection object is being reused across different pq.read_table invocations in different threads

(InputReaderNode pid=25001, ip=172.31.60.29) malloc_consolidate(): invalid chunk size
(InputReaderNode pid=25001, ip=172.31.60.29) \*\*\* SIGABRT received at time=1665464922 on cpu 9 \*\*\*
(InputReaderNode pid=25001, ip=172.31.60.29) PC: @     0x7f9a480a803b  (unknown)  raise
(InputReaderNode pid=25001, ip=172.31.60.29)     @     0x7f9a480a80c0       4160  (unknown)
(InputReaderNode pid=25001, ip=172.31.60.29)     @     0x7f9a480fa32c  (unknown)  (unknown)

Note, this multithreaded code is running inside a Ray actor process, but that shouldn't be a problem.

Reporter: Ziheng Wang / @marsupialtail

Original Issue Attachments:

Note: This issue was originally created as ARROW-17984. Please see the migration documentation for further details.

Activity

  1. asfimport commented on Oct 11, 2022

    @asfimport
    Author

    Antoine Pitrou / @pitrou:
    Can you enable core dumps and try to get a gdb backtrace of all threads?

  2. asfimport commented on Oct 11, 2022

    @asfimport
    Author

    Ziheng Wang / @marsupialtail:
    Unfortunately I cannot figure out how to get a core dump. However I can say with confidence that this is the line that triggers this:

    "/home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/parquet/init.py", line 2353 in init

      filesystem, path_or_paths = FileSystem.from_uri(path_or_paths)

    For context this will be trying to make different S3FileSystem objects in different threads. 

  3. asfimport commented on Oct 28, 2022

    @asfimport
    Author

    Ziheng Wang / @marsupialtail:
    I have attached a crash file. You can unpack using apport-unpack, and then gdb the CoreDump with gdb cat ExecutablePath CoreDump. Once inside the gdb do thread apply all bt

  4. asfimport commented on Oct 28, 2022

    @asfimport
    Author

    Weston Pace / @westonpace:
    Can you share the full output of thread apply all bt? The core dump itself isn't too portable without the binaries that generated it. E.g...

    
    warning: Can't open file /usr/bin/python3.8 during file-backed mapping note processing
    
    warning: Can't open file /usr/lib/x86_64-linux-gnu/libresolv-2.31.so during file-backed mapping note processing
    
    warning: Can't open file /home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow_python_flight.so.900 during file-backed mapping note processing
    

    So I just get:

    
    (gdb) thread apply all bt
    
    Thread 42 (LWP 10252):
    #0  0x00007fd4adf28f37 in ?? ()
    #1  0x0000000000000024 in ?? ()
    #2  0x00007fd4a5ff31e0 in ?? ()
    #3  0x0000000000000000 in ?? ()
    
    Thread 41 (LWP 10072):
    #0  0x00007fd4ae05676d in ?? ()
    #1  0x00007fd1f3864396 in ?? ()
    #2  0x0000000000000000 in ?? ()
    ...
    
  5. asfimport commented on Nov 3, 2022

    @asfimport
    Author

    Ziheng Wang / @marsupialtail:

    
    Thread 42 (Thread 0x7fd1d77fe700 (LWP 10252)):
    #0  futex_abstimed_wait (private=0, abstime=0x0, clockid=0, expected=2, futex_word=<optimized out>) at ../sysdeps/nptl/futex-internal.h:284
    #1  __pthread_rwlock_wrlock_full (abstime=0x0, clockid=0, rwlock=0x2fd02f0) at pthread_rwlock_common.c:830
    #2  __GI___pthread_rwlock_wrlock (rwlock=0x2fd02f0) at pthread_rwlock_wrlock.c:27
    #3  0x00007fd4a586ee59 in CRYPTO_THREAD_write_lock () from /home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
    #4  0x00007fd4a586436f in ossl_namemap_add_names () from /home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
    #5  0x00007fd4a58512c0 in construct_evp_method () from /home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
    #6  0x00007fd4a5863e15 in ossl_method_construct_this () from /home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
    #7  0x00007fd4a5863c8e in algorithm_do_this () from /home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
    #8  0x00007fd4a586e1db in ossl_provider_doall_activated () from /home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
    #9  0x00007fd4a5863d4f in ossl_algorithm_do_all () from /home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
    #10 0x00007fd4a5864015 in ossl_method_construct () from /home/ubuntu/.local/lib--~~Type <RET> for~~-Type <RET> for -~~Type~~----~~Ty~~ ~~Type <RET> for mo~~ ~~Type -~~-~~Type <RET> for more, q to quit, c to continue without paging~~-
    python3.8/site-packages/pyarrow/libarrow.so.900
    #11 0x00007fd4a5851876 in evp_generic_fetch () from /home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
    #12 0x00007fd4a584fd1a in EVP_CIPHER_fetch () from /home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
    #13 0x00007fd4a5c6ad22 in ssl_evp_cipher_fetch () from /home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
    #14 0x00007fd4a5c605b3 in ssl_load_ciphers () from /home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
    #15 0x00007fd4a5c6c1b1 in SSL_CTX_new_ex () from /home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
    #16 0x00007fd4a5c4e727 in ossl_connect_step1 () from /home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
    #17 0x00007fd4a5c51c7d in ossl_connect_common () from /home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
    #18 0x00007fd4a5c1a348 in Curl_ssl_connect_nonblocking () from /home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
    #19 0x00007fd4a5c32766 in Curl_http_connect () from /home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
    #20 0x00007fd4a5bfc2eb in multi_runsingle () from /home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
    #21 0x00007fd4a5bfd76b in curl_multi_perform () from /home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
    #22 0x00007fd4a5bec833 in easy_transfer () from /home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
    #23 0x00007fd4a5becde9 in curl_easy_perform () from /home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
    #24 0x00007fd4a5737984 in Aws::Http::CurlHttpClient::MakeRequest(std::shared_ptr<Aws::Http::HttpRequest> const&, Aws::Utils::RateLimits::RateLimiterInterface\*, Aws::Utils::RateLimits::RateLimiterInterface\*) const () from /home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
    #25 0x00007fd4a56fa581 in Aws::Client::AWSClient::AttemptOneRequest(std::shared_ptr<Aws::Http::HttpRequest> const&, Aws::AmazonWebServiceRequest const&, char const\*, char const\*, char const\*) const () from /home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
    #26 0x00007fd4a56fb3d0 in Aws::Client::AWSClient::AttemptExhaustively(Aws::Http::URI const&, Aws::AmazonWebServiceRequest const&, Aws::Http::HttpMethod, char const\*, char const\*, char const\*) const () from /home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
    #27 0x00007fd4a57072f1 in Aws::Client::AWSXMLClient::MakeRequest(Aws::Http::URI const&, Aws::AmazonWebServiceRequest const&, Aws::Http::HttpMethod, char const\*, char const\*, char const\*) const () from /home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
    #28 0x00007fd4a5219dd9 in arrow::fs::ResolveS3BucketRegion(std::string const&) () from /home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
    #29 0x00007fd4a521b6a3 in arrow::fs::S3Options::FromUri(arrow::internal::Uri const&, std::string\*) () from /home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
    #30 0x00007fd4a4528c4f in arrow::fs::(anonymous namespace)::FileSystemFromUriReal(arrow::internal::Uri const&, std::string const&, arrow::io::IOContext const&, std::string\*) () from /home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
    #31 0x00007fd4a45298c0 in arrow::fs::FileSystemFromUri(std::string const&, arrow::io::IOContext const&, std::string\*) () from /home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
    #32 0x00007fd4a4529aea in arrow::fs::FileSystemFromUriOrPath(std::string const&, arrow::io::IOContext const&, std::string\*) () from /home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
    #33 0x00007fd4a4529cfd in arrow::fs::FileSystemFromUriOrPath(std::string const&, std::string\*) () from /home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
    #34 0x00007fd488170b7d in __pyx_pw_7pyarrow_3_fs_10FileSystem_3from_uri(_object\*, _object\*, _object\*) () from /home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/_fs.cpython-38-x86_64-linux-gnu.so
    #35 0x00000000005f6929 in PyCFunction_Call ()
    #36 0x00000000005f74f6 in _PyObject_MakeTpCall ()
    #37 0x0000000000571164 in _PyEval_EvalFrameDefault ()
    #38 0x0000000000569dba in _PyEval_EvalCodeWithName ()
    #39 0x00000000005f6eb3 in _PyFunction_Vectorcall ()
    #40 0x000000000059ca27 in ?? ()
    #41 0x00000000005f745f in _PyObject_MakeTpCall ()
    #42 0x000000000057164c in _PyEval_EvalFrameDefault ()
    #43 0x0000000000569dba in _PyEval_EvalCodeWithName ()
    #44 0x00000000005f6eb3 in _PyFunction_Vectorcall ()
    #45 0x000000000056cc1f in _PyEval_EvalFrameDefault ()
    #46 0x0000000000569dba in _PyEval_EvalCodeWithName ()
    #47 0x00000000005f6eb3 in _PyFunction_Vectorcall ()
    #48 0x00000000005f6082 in PyObject_Call ()
    #49 0x000000000056d2d5 in _PyEval_EvalFrameDefault ()
    #50 0x00000000005f6cd6 in _PyFunction_Vectorcall ()
    #51 0x000000000056bbfa in _PyEval_EvalFrameDefault ()
    
    
  6. asfimport commented on Nov 4, 2022

    @asfimport
    Author

    Antoine Pitrou / @pitrou:
    @marsupialtail Can you post the backtrace for all threads? (using "thread apply all bt" as suggested by @westonpace).

    Since that will probably be quite long I suggest posting it on a site such as https://gist.github.com/ and posting the link here.

  7. thisisnic commented on Nov 10, 2026

    @thisisnic
    Member

    I tried to reproduce this with a reprex that starts 32 threads at once, each calling FileSystem.from_uri() on an s3:// URI and then pq.read_table(), in a fresh process each time. It ran cleanly 25 times out of 25 on both pyarrow 9.0.0 and 26.0.1.

    The backtrace above shows a thread waiting on a lock inside OpenSSL during the bucket region lookup rather than the thread that crashed, so it doesn't tell us where the heap corruption came from. The code on that path (EnsureS3Initialized(), RegionResolver::DefaultInstance(), ResolveS3BucketRegion()) is guarded by locks in both 9.0.0 and current main.

    Closing as we can't reproduce it and the full thread apply all bt output was never provided. If you still see this on a current release, please open a new issue with the backtrace for all threads.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions