Repository navigation
No protection: import numpy in two different threads can lead to race-condition #91238
Closed as duplicate
Description
| BPO | 47082 |
|---|---|
| Nosy | @tiran, @seberg, @jhgoebbert |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2022-03-21.17:03:05.346>
labels = ['interpreter-core', 'type-bug', '3.11']
title = 'No protection: `import numpy` in two different threads can lead to race-condition'
updated_at = <Date 2022-03-21.23:04:38.800>
user = 'https://github.com/jhgoebbert'bugs.python.org fields:
activity = <Date 2022-03-21.23:04:38.800>
actor = 'seberg'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Interpreter Core']
creation = <Date 2022-03-21.17:03:05.346>
creator = 'jhgoebbert'
dependencies = []
files = []
hgrepos = []
issue_num = 47082
keywords = []
message_count = 4.0
messages = ['415687', '415690', '415718', '415722']
nosy_count = 3.0
nosy_names = ['christian.heimes', 'seberg', 'jhgoebbert']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue47082'
versions = ['Python 3.11']
Activity
jhgoebbert commentedon Mar 21, 2022
While using Xpra we came across a bug which might be a Python or a NumPy issue.
Perhaps some of you can help us understanding some internals.
Calling
import numpyat the same time in two different threads of the Python program can lead to a race-condition.This happens for example with Xpra when loading the encoder nvjpeg:
Here the environment variable OPENBLAS_MAIN_FREE is set in the
numpycode:https://github.com/numpy/numpy/blob/maintenance/1.21.x/numpy/core/__init__.py#L18
and short after that it is deleted
https://github.com/numpy/numpy/blob/maintenance/1.21.x/numpy/core/__init__.py#L51
But this deletion fails ... perhaps because the initialization runs at the same time in two threads 🤔
Shouldn't Python protect us by design?
@seberg comments HERE:
Can anyone comment on this?
seberg commentedon Mar 21, 2022
To add to this: it would seem to me that the side-effects of importing should be guaranteed to only be called once?
However, IO or other operations could be part of the import side-effects and release the GIL. So even a simple, pure-Python, package could run into this same issue and probably won't even realize that they can run into it.
(Assuming I understand how this is happening correctly.)
So it would seem to me that either:
sys.modulesdictionary?threadingmodule could somehow ensure safety by hooking intothe import machinery?
only be run once have to lock (i.e. NumPy).
(But it seems to me that many packages will not even be aware of
possible issues.)
tiran commentedon Mar 21, 2022
Python used to have a global import lock. The global import lock prevented recursive imports from other threads while the current thread was importing. The import lock was source of issues and dead locks. Antoine replaced it with a fine grained import lock in bpo-9260.
seberg commentedon Mar 21, 2022
Thanks, so there should already be a lock in place (sorry, I missed that). But somehow we seem to get around it?
Do you know what may cause the locking logic to fail in this case? Recursive imports in NumPy itself? Or Cython using low-level C-API?
I.e. can you think of something to investigate that may help NumPy/Cython to make sure that locking is successful?
fbrunodr commentedon Apr 15, 2022
@seberg can you provide a working example of that bug? I tried to simulate this race condition and could not reproduce the bug.
seberg commentedon Apr 15, 2022
Unfortunately not, it was @jhgoebbert who experienced it, but maybe I can try tomorrow or so. My hypothesis was cython got involved. But of course it could be that there is a deeper threading issue elsewhere. Another thing may be that you need a fairly new NumPy versions to run into the environment variable issue, because older versions probably do not mess with
OPENBLAS_MAIN_FREE.cg505 commentedon Apr 11, 2025
Hi! We have an extremely fragile reproduction case in the CI for SkyPilot here: https://github.com/skypilot-org/skypilot/actions/runs/14391450370/job/40359190320?pr=5095
Relevant stacktrace:
We have some custom code to lazily import
numpyas needed, and I guess it is running in one thread. It causes a crash in another thread - which is not doing an import, but is just trying to read fromos.environ. In fact, you can see that the bottom three frames of the stacktrace are in CPython, and they all look very innocuous.It seems that the numpy import is somehow changing os.environ out from under the thread that crashes. That's easy enough to understand, but what I can't understand is how this is not protected by GIL, since numpy's
os.environmodification is happening in Python:https://github.com/numpy/numpy/blob/v1.24.4/numpy/core/__init__.py#L14-L20
https://github.com/numpy/numpy/blob/v1.24.4/numpy/core/__init__.py#L50-L52
This is python 3.8.20 (I know EOL, but sharing anyway since it might help pin down the actual issue) with numpy 1.24.4. The threads are all part of a
multiprocessing.pool.ThreadPool, which you can see in the stacktrace. It's here in the SkyPilot source:https://github.com/skypilot-org/skypilot/blob/d01d496e66615656a2ddf57e4695151392b61c63/sky/utils/subprocess_utils.py#L136-L138
Unsurprisingly, it unfortunately seems very difficult to reproduce this in other environments.
gpshead commentedon Jan 11, 2026
I believe the root cause of this is #143650 - your numpy environment error matches the scenario that led me to debug this.
fix(startup): remove background preloader daemon to stop import race