Repository navigation
[subinterpreters] Make the PyGILState API compatible with subinterpreters #59956
Description
| BPO | 15751 |
|---|---|
| Nosy | @mhammond, @jcea, @ronaldoussoren, @ncoghlan, @pitrou, @vstinner, @phsilva, @asvetlov, @ericsnowcurrently, @soltysh, @seberg, @erlend-aasland, @Felk, @h-vetinari |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2012-08-21.05:18:00.735>
labels = ['expert-subinterpreters', 'expert-C-API', 'type-feature', '3.10']
title = '[subinterpreters] Make the PyGILState API compatible with subinterpreters'
updated_at = <Date 2022-01-07.18:32:22.492>
user = 'https://github.com/ncoghlan'bugs.python.org fields:
activity = <Date 2022-01-07.18:32:22.492>
actor = 'vstinner'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['C API', 'Subinterpreters']
creation = <Date 2012-08-21.05:18:00.735>
creator = 'ncoghlan'
dependencies = []
files = []
hgrepos = []
issue_num = 15751
keywords = []
message_count = 49.0
messages = ['168742', '168759', '168764', '168766', '168768', '168769', '168771', '168772', '168776', '168779', '168780', '168787', '168789', '168790', '168792', '168812', '168815', '168816', '168818', '168832', '169008', '169010', '169013', '169014', '169278', '169280', '169284', '169286', '169310', '169329', '169330', '169332', '169333', '169334', '169352', '169354', '169359', '198115', '206016', '206017', '206042', '206048', '261779', '368885', '368904', '368906', '375134', '380325', '409995']
nosy_count = 17.0
nosy_names = ['mhammond', 'jcea', 'ronaldoussoren', 'ncoghlan', 'pitrou', 'vstinner', 'phsilva', 'grahamd', 'asvetlov', 'python-dev', 'eric.snow', 'maciej.szulik', 'seberg', 'erlendaasland', 'felk', 'h-vetinari', 'gabrielhae']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'needs patch'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue15751'
versions = ['Python 3.10']Linked PRs
- gh-59956: Clarify GILState-related Code #101161
- gh-59956: Fix Function Groupings in pystate.c #101172
- gh-59956: Allow the "Trashcan" Mechanism to Work Without a Thread State #101209
- gh-59956: Clarify Runtime State Status Expectations #101308
- gh-59956: Partial Fix for GILState API Compatibility with Subinterpreters #101431
- gh-59956: Add a Test to Verify GILState Matches the "Current" Thread State #101625
Metadata
Metadata
Assignees
Projects
- Status
Activity
ncoghlan commentedon Aug 21, 2012
Currently, modules that use the PyGILState* APIs cannot be used with mod_wsgi, as mod_wsgi uses the subinterpreter support.
Graham Dumpleton and I spent some time discussing this at PyCon AU 2012, and we believe that the incompatibility can be resolved with a single API in the core: a function that mod_wsgi can call to set the interpreter used by the GIL state APIs to implicitly create new thread states.
This is still only a seed of an idea (and it's entirely possible we've missed something), but it's a place to start in resolving this longstanding incompatibility.
grahamd commentedon Aug 21, 2012
Just to clarify. One can still tell WSGI applications under mod_wsgi to run in the main interpreter and in that case modules using PyGILState* do work. By default though, sub interpreters are used as noted.
The mechanism for forcing use of main interpreter is the directive:
WSGIApplicationGroup %{GLOBAL}
Some details about this issue can be found in:
http://code.google.com/p/modwsgi/wiki/ApplicationIssues#Python_Simplified_GIL_State_API
pitrou commentedon Aug 21, 2012
How would it work?
ncoghlan commentedon Aug 21, 2012
It would twiddle the autoInterpreterState and autoTLSkey entries in the pystate.c global variables to point to a different subinterpreter.
As I understand the situation, mod_wsgi doesn't need arbitrary externally created threads to be able to call back into arbitrary subinterpreters, it just needs to be able to direct externally created threads in a process to a subinterpreter other than the main one.
Graham, looking at the current impl - have you experimented with just calling _PyGILState_Init() with the interpreter state and current thread state for the desired subinterpreter to see what happens?
I think the new method could just be a cleaner combination of _PyGILState_ReInit and _PyGILState_Init. If I'm right, then calling _PyGILState_Init should convert the current crashes and deadlocks into a relatively less harmful memory leak (since the old entry in the TLS won't get deleted properly).
ncoghlan commentedon Aug 21, 2012
Graham, even better would be if you could try the following combination:
_PyGILState_Fini();
_PyGILState_Init(si, st);
(where si and st are the interpreter state and thread state for the target subinterpreter)
If a new PyGILState_SwitchInterpreter API is going to be able to solve this in 3.4, then I believe those private APIs should be enough to make it possible in *current* versions.
If those private APIs *aren't* enough, then I'm missing something and this isn't going to be as easy as I thought.
pitrou commentedon Aug 21, 2012
But how does it know that the right externally created thread will point
to the right subinterpreter? The OS is free to schedule threads as it
desires.
ncoghlan commentedon Aug 21, 2012
Just as they do today, all externally created threads will still go to *one* interpreter when they hit PyGILState_Ensure(). It's just that interpreter won't be the main one.
Since the whole point of the PyGILState API is to support threads that don't have a previously created thread state, there's no getting around the requirement to have a single blessed interpreter that handles all externally created threads in a given process.
It will be up to mod_wsgi (and any other embedding application that uses the new function) to make sure it calls this at a time when there aren't any existing calls to PyGILState that would be disrupted. (Assuming we can't figure out a locking scheme that *ensures* no such threads are running when the switch occurs)
pitrou commentedon Aug 21, 2012
Uh but how does it solve the issue? (unless you create a mod_wsgi app
with only a single worker thread)
ncoghlan commentedon Aug 21, 2012
My understanding of the mod_wsgi architecture is that it uses subinterpreters to maintain a persistent process, while still providing a relatively pristine interpreter state to handle each new request. This means even when you're using multiple processes with a single request handling thread per process, you're running a subinterpreter unless you explicitly configure mod_wsgi to always run in the main interpreter (which, I believe, will result in additional state persistence across requests).
The proposed API change can only fix scenarios where *at a given point in time*, *all* PyGILState_Ensure calls should be directed to a particular subinterpreter. The target subinterpreter may change *later*, but there still cannot be two desired targets within the same process at the same time.
However, at the moment, PyGILState doesn't even allow that - all externally created threads are handled in the *main* interpreter even if that isn't what the embedding application wants.
ncoghlan commentedon Aug 21, 2012
Sorry, I mischaracterised the way mod_wsgi works slightly. However, my understanding is still that the scope of this particular fix is merely to allow all external threads to be redirected to a different subinterpreter at various times over the life of a process. It does not need to allow different external threads to be redirected to different subinterpreters.
(Note: I am assuming that any hooks Apache/mod_wsgi has into external thread creation could already just create an appropriate thread state if that was the desired behaviour. It may be I'm incorrect on this, and what Graham really wants is the ability to change the target interpreter state just for the current thread. However, if that's what he wants, then there's additional background info I need on mod_wsgi and its ability to influence thread creation within a process, because I didn't get the impression on the weekend that that is what he was after)
pitrou commentedon Aug 21, 2012
I don't think that's true. On hg.python.org, the hglookup application
keeps a cached mapping of changeset ids to repo URLs, which wouldn't be
possible if its interpreter was re-bootstrapped for each new request.
ncoghlan commentedon Aug 21, 2012
s/slightly/completely/ (I believe my description of the core problem was right, but my description of why that problem exists was wrong - it actually has to do with the way mod_wsgi handles virtual hosts and endpoints)
If we expose an official way to switch the destination of the PyGILState APIs, then what it means is that mod_wsgi can, over the lifecycle of a single process in the pool, *switch* the virtual host and WSGI endpoint that process is handling by changing the active interpreter. There are still some extension modules where that won't work (because they create persistent threads that periodically call back into Python, so they may still end up calling back in to the wrong interpreter), but it will allow those that just do callbacks from external worker threads (or other operations that are similarly bound by the lifecycle of a request) to start working properly.
pitrou commentedon Aug 21, 2012
I don't understand what that means. It's the OS which switches threads,
including interpreter threads.
(by the way, the real fix to the GILState vs. subinterpreters issue
would be to create new API functions which take an interpreter as
argument, so that you have a distinct TLS key per interpreter)
ncoghlan commentedon Aug 21, 2012
Umm, no. The whole point of the GILState API is that you can call it from a thread which knows *nothing* about Python.
It will then use the *process global* state in the pystate.c statics to initialise that thread with a Python thread state. Currently, that thread state will always be for the main Python interpreter for the process.
All Graham wants is an officially supported way to change which interpreter the pystate.c globals reference *without* shutting down and reinitialising Python completely.
There are going to be limitations on how effective this will be - it still won't support *all* extension modules that use the PyGILState APIs. It will, however, support many more of them than the status quo (which is zero, unless you force your WSGI app to use the main interpreter, which has its own problems).
And you absolutely can control when the OS switches threads - controlling that is what synchronisation primitives are *for*.
57 remaining items
pythongh-59956: Clarify Runtime State Status Expectations (pythongh-1…
gh-59956: Partial Fix for GILState API Compatibility with Subinterpre…
gh-59956: Add a Test to Verify GILState Matches the "Current" Thread …
yqs112358 commentedon Oct 5, 2023
It is known that Python3.12 has supported per-interpreter GIL. Considering that running different sub-interpreters in parallel in multiple threads becomes a reality, it is a necessity to make the PyGILState API compatible.
Hope this will be fixed later in 3.12.x. ❤
Or, is there any instructions to teach us how to safely call PyGILState_Ensure() and PyGILState_Release() when using sub-interpreters in Non-Python created threads?
ericsnowcurrently commentedon Oct 5, 2023
Just to be clear, this issue describes two distinct but related concerns:
PyGILState_Ensure()when it creates a new thread state1Presumably you are talking about the bug part.
The bug part should be fixed in 3.12, via gh-1014312 (and mostly tested via gh-101625). There should no longer be any crashes or inconsistent state when using the GILState API with subinterpreters. (The feature part of this issue is still unresolved.)
If the PR didn't actually solve the problem then we'll see if we can fix it in 3.12. However, a fix would have to wait until 3.13 if it requires adding new API.
Is there a specific case that isn't working for you on 3.12? If so, please provide an example that reproduces the failure.
It shouldn't need any special treatment. Use
PyGILState_Ensure()like normal. You don't get to pick which interpreter state any new thread state targets though. That's the feature part of this issue that still needs to be solved (in 3.13).Footnotes
it always uses the main interpreter ↩
Note that the main objective of gh-59956: Partial Fix for GILState API Compatibility with Subinterpreters #101431 was to preserve the existing behavior of the GILState API, which was impacted by various runtime changes needed for a per-interpreter GIL. However, my understanding was that it also solved the bug part of this issue. ↩
pitrou commentedon Oct 5, 2023
Not speaking for the OP, but if
PyGILState_Ensure()always selects the main interpreter it means that any extension using it will not be compatible with subinterpreters (unless the piece of code that's guarded byPyGILState_Ensureis stateless enough that it doesn't matter which interpreter it executes in).For example, if an extension gives access to a database and allows writing user-defined functions (UDFs) in Python, UDFs would then be always executed in the main interpreter which is not necessarily expected.
What would be needed is an additional API such as:
(
PyGILState_STATE_EXmight or might not be the same asPyGILState_STATE)... with one open question being: what happens if the given interpreter was destroyed in the meantime?
(should there instead be some kind of interpreter id pointing to a hash table of interpreters?)
terryjreedy commentedon Feb 24, 2025
@ericsnowcurrently Do your 6 patches sufficiently fix this issue to close it?
ncoghlan commentedon Feb 26, 2025
The feature request is still open (there's no API to make
PyGIL_StateEnsure()pick an interpreter other than the main one).It's a desirable feature (since it makes subinterpreters compatible with more existing extension modules), but not an easy one to add.
encukou commentedon Jul 10, 2025
PEP 788 proposes deprecating
PyGILState_Ensurein favour of a newPyThreadState_Ensurefunction, which takes aPyInterpreterRef(a strong reference also prevents the interpreter from being torn down, withPyInterpreterWeakRefavailable for cases where you don't want that).cc @ZeroIntensity