You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
pq.read_table doesn't seem to be thread safe #33189
Before PR: #13799 gets merged in master, I am using multithreading to improve read bandwidth from S3. Even after that PR gets merged, I probably will still try to use multithreading to some extent.
However pq.read_table from S3 doesn't seem to be thread safe. Seems like it uses the new dataset reader under the hood. I cannot provide a reproduction, not a stable one anyway. But this is roughly the script I have been using
The errors all have to do with malloc segfaults which makes me suspect the connection object is being reused across different pq.read_table invocations in different threads
Ziheng Wang / @marsupialtail:
Unfortunately I cannot figure out how to get a core dump. However I can say with confidence that this is the line that triggers this:
"/home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/parquet/init.py", line 2353 in init
Ziheng Wang / @marsupialtail:
I have attached a crash file. You can unpack using apport-unpack, and then gdb the CoreDump with gdb cat ExecutablePath CoreDump. Once inside the gdb do thread apply all bt
Weston Pace / @westonpace:
Can you share the full output of thread apply all bt? The core dump itself isn't too portable without the binaries that generated it. E.g...
warning: Can't open file /usr/bin/python3.8 during file-backed mapping note processing
warning: Can't open file /usr/lib/x86_64-linux-gnu/libresolv-2.31.so during file-backed mapping note processing
warning: Can't open file /home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow_python_flight.so.900 during file-backed mapping note processing
So I just get:
(gdb) thread apply all bt
Thread 42 (LWP 10252):
#0 0x00007fd4adf28f37 in ?? ()
#1 0x0000000000000024 in ?? ()
#2 0x00007fd4a5ff31e0 in ?? ()
#3 0x0000000000000000 in ?? ()
Thread 41 (LWP 10072):
#0 0x00007fd4ae05676d in ?? ()
#1 0x00007fd1f3864396 in ?? ()
#2 0x0000000000000000 in ?? ()
...
Thread 42 (Thread 0x7fd1d77fe700 (LWP 10252)):
#0 futex_abstimed_wait (private=0, abstime=0x0, clockid=0, expected=2, futex_word=<optimized out>) at ../sysdeps/nptl/futex-internal.h:284
#1 __pthread_rwlock_wrlock_full (abstime=0x0, clockid=0, rwlock=0x2fd02f0) at pthread_rwlock_common.c:830
#2 __GI___pthread_rwlock_wrlock (rwlock=0x2fd02f0) at pthread_rwlock_wrlock.c:27
#3 0x00007fd4a586ee59 in CRYPTO_THREAD_write_lock () from /home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
#4 0x00007fd4a586436f in ossl_namemap_add_names () from /home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
#5 0x00007fd4a58512c0 in construct_evp_method () from /home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
#6 0x00007fd4a5863e15 in ossl_method_construct_this () from /home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
#7 0x00007fd4a5863c8e in algorithm_do_this () from /home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
#8 0x00007fd4a586e1db in ossl_provider_doall_activated () from /home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
#9 0x00007fd4a5863d4f in ossl_algorithm_do_all () from /home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
#10 0x00007fd4a5864015 in ossl_method_construct () from /home/ubuntu/.local/lib--~~Type <RET> for~~-Type <RET> for -~~Type~~----~~Ty~~ ~~Type <RET> for mo~~ ~~Type -~~-~~Type <RET> for more, q to quit, c to continue without paging~~-
python3.8/site-packages/pyarrow/libarrow.so.900
#11 0x00007fd4a5851876 in evp_generic_fetch () from /home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
#12 0x00007fd4a584fd1a in EVP_CIPHER_fetch () from /home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
#13 0x00007fd4a5c6ad22 in ssl_evp_cipher_fetch () from /home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
#14 0x00007fd4a5c605b3 in ssl_load_ciphers () from /home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
#15 0x00007fd4a5c6c1b1 in SSL_CTX_new_ex () from /home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
#16 0x00007fd4a5c4e727 in ossl_connect_step1 () from /home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
#17 0x00007fd4a5c51c7d in ossl_connect_common () from /home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
#18 0x00007fd4a5c1a348 in Curl_ssl_connect_nonblocking () from /home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
#19 0x00007fd4a5c32766 in Curl_http_connect () from /home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
#20 0x00007fd4a5bfc2eb in multi_runsingle () from /home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
#21 0x00007fd4a5bfd76b in curl_multi_perform () from /home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
#22 0x00007fd4a5bec833 in easy_transfer () from /home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
#23 0x00007fd4a5becde9 in curl_easy_perform () from /home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
#24 0x00007fd4a5737984 in Aws::Http::CurlHttpClient::MakeRequest(std::shared_ptr<Aws::Http::HttpRequest> const&, Aws::Utils::RateLimits::RateLimiterInterface\*, Aws::Utils::RateLimits::RateLimiterInterface\*) const () from /home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
#25 0x00007fd4a56fa581 in Aws::Client::AWSClient::AttemptOneRequest(std::shared_ptr<Aws::Http::HttpRequest> const&, Aws::AmazonWebServiceRequest const&, char const\*, char const\*, char const\*) const () from /home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
#26 0x00007fd4a56fb3d0 in Aws::Client::AWSClient::AttemptExhaustively(Aws::Http::URI const&, Aws::AmazonWebServiceRequest const&, Aws::Http::HttpMethod, char const\*, char const\*, char const\*) const () from /home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
#27 0x00007fd4a57072f1 in Aws::Client::AWSXMLClient::MakeRequest(Aws::Http::URI const&, Aws::AmazonWebServiceRequest const&, Aws::Http::HttpMethod, char const\*, char const\*, char const\*) const () from /home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
#28 0x00007fd4a5219dd9 in arrow::fs::ResolveS3BucketRegion(std::string const&) () from /home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
#29 0x00007fd4a521b6a3 in arrow::fs::S3Options::FromUri(arrow::internal::Uri const&, std::string\*) () from /home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
#30 0x00007fd4a4528c4f in arrow::fs::(anonymous namespace)::FileSystemFromUriReal(arrow::internal::Uri const&, std::string const&, arrow::io::IOContext const&, std::string\*) () from /home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
#31 0x00007fd4a45298c0 in arrow::fs::FileSystemFromUri(std::string const&, arrow::io::IOContext const&, std::string\*) () from /home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
#32 0x00007fd4a4529aea in arrow::fs::FileSystemFromUriOrPath(std::string const&, arrow::io::IOContext const&, std::string\*) () from /home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
#33 0x00007fd4a4529cfd in arrow::fs::FileSystemFromUriOrPath(std::string const&, std::string\*) () from /home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/libarrow.so.900
#34 0x00007fd488170b7d in __pyx_pw_7pyarrow_3_fs_10FileSystem_3from_uri(_object\*, _object\*, _object\*) () from /home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/_fs.cpython-38-x86_64-linux-gnu.so
#35 0x00000000005f6929 in PyCFunction_Call ()
#36 0x00000000005f74f6 in _PyObject_MakeTpCall ()
#37 0x0000000000571164 in _PyEval_EvalFrameDefault ()
#38 0x0000000000569dba in _PyEval_EvalCodeWithName ()
#39 0x00000000005f6eb3 in _PyFunction_Vectorcall ()
#40 0x000000000059ca27 in ?? ()
#41 0x00000000005f745f in _PyObject_MakeTpCall ()
#42 0x000000000057164c in _PyEval_EvalFrameDefault ()
#43 0x0000000000569dba in _PyEval_EvalCodeWithName ()
#44 0x00000000005f6eb3 in _PyFunction_Vectorcall ()
#45 0x000000000056cc1f in _PyEval_EvalFrameDefault ()
#46 0x0000000000569dba in _PyEval_EvalCodeWithName ()
#47 0x00000000005f6eb3 in _PyFunction_Vectorcall ()
#48 0x00000000005f6082 in PyObject_Call ()
#49 0x000000000056d2d5 in _PyEval_EvalFrameDefault ()
#50 0x00000000005f6cd6 in _PyFunction_Vectorcall ()
#51 0x000000000056bbfa in _PyEval_EvalFrameDefault ()
I tried to reproduce this with a reprex that starts 32 threads at once, each calling FileSystem.from_uri() on an s3:// URI and then pq.read_table(), in a fresh process each time. It ran cleanly 25 times out of 25 on both pyarrow 9.0.0 and 26.0.1.
The backtrace above shows a thread waiting on a lock inside OpenSSL during the bucket region lookup rather than the thread that crashed, so it doesn't tell us where the heap corruption came from. The code on that path (EnsureS3Initialized(), RegionResolver::DefaultInstance(), ResolveS3BucketRegion()) is guarded by locks in both 9.0.0 and current main.
Closing as we can't reproduce it and the full thread apply all bt output was never provided. If you still see this on a current release, please open a new issue with the backtrace for all threads.
Activity
asfimport commentedon Oct 11, 2022
Antoine Pitrou / @pitrou:
Can you enable core dumps and try to get a gdb backtrace of all threads?
asfimport commentedon Oct 11, 2022
Ziheng Wang / @marsupialtail:
Unfortunately I cannot figure out how to get a core dump. However I can say with confidence that this is the line that triggers this:
"/home/ubuntu/.local/lib/python3.8/site-packages/pyarrow/parquet/init.py", line 2353 in init
filesystem, path_or_paths = FileSystem.from_uri(path_or_paths)
For context this will be trying to make different S3FileSystem objects in different threads.
asfimport commentedon Oct 28, 2022
Ziheng Wang / @marsupialtail:
I have attached a crash file. You can unpack using apport-unpack, and then gdb the CoreDump with gdb
cat ExecutablePathCoreDump. Once inside the gdb do thread apply all btasfimport commentedon Oct 28, 2022
Weston Pace / @westonpace:
Can you share the full output of
thread apply all bt? The core dump itself isn't too portable without the binaries that generated it. E.g...So I just get:
asfimport commentedon Nov 3, 2022
Ziheng Wang / @marsupialtail:
asfimport commentedon Nov 4, 2022
Antoine Pitrou / @pitrou:
@marsupialtail Can you post the backtrace for all threads? (using "thread apply all bt" as suggested by @westonpace).
Since that will probably be quite long I suggest posting it on a site such as https://gist.github.com/ and posting the link here.
thisisnic commentedon Nov 10, 2026
I tried to reproduce this with a reprex that starts 32 threads at once, each calling
FileSystem.from_uri()on ans3://URI and thenpq.read_table(), in a fresh process each time. It ran cleanly 25 times out of 25 on both pyarrow 9.0.0 and 26.0.1.The backtrace above shows a thread waiting on a lock inside OpenSSL during the bucket region lookup rather than the thread that crashed, so it doesn't tell us where the heap corruption came from. The code on that path (
EnsureS3Initialized(),RegionResolver::DefaultInstance(),ResolveS3BucketRegion()) is guarded by locks in both 9.0.0 and current main.Closing as we can't reproduce it and the full
thread apply all btoutput was never provided. If you still see this on a current release, please open a new issue with the backtrace for all threads.