Repository navigation
[subinterpreters] PyObject statics exposed in the limited API break isolation. #87669
Description
| BPO | 43503 |
|---|---|
| Nosy | @gvanrossum, @nascheme, @vstinner, @encukou, @markshannon, @ericsnowcurrently, @mattip |
| PRs |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2021-03-15.18:53:43.710>
labels = ['interpreter-core', 'expert-subinterpreters', 'type-bug', '3.10', 'extension-modules', 'expert-C-API']
title = '[subinterpreters] PyObject statics exposed in the limited API break isolation.'
updated_at = <Date 2021-03-17.14:46:29.259>
user = 'https://github.com/ericsnowcurrently'bugs.python.org fields:
activity = <Date 2021-03-17.14:46:29.259>
actor = 'eric.snow'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Extension Modules', 'Interpreter Core', 'C API', 'Subinterpreters']
creation = <Date 2021-03-15.18:53:43.710>
creator = 'eric.snow'
dependencies = []
files = []
hgrepos = []
issue_num = 43503
keywords = ['patch']
message_count = 14.0
messages = ['388759', '388764', '388765', '388773', '388782', '388783', '388848', '388859', '388861', '388871', '388920', '388922', '388923', '388924']
nosy_count = 7.0
nosy_names = ['gvanrossum', 'nascheme', 'vstinner', 'petr.viktorin', 'Mark.Shannon', 'eric.snow', 'mattip']
pr_nums = ['24828']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue43503'
versions = ['Python 3.10']
Activity
ericsnowcurrently commentedon Mar 15, 2021
In the limited C-API we expose the following static PyObject variables:
Since they are part of the limited API, they have a direct effect on the stable ABI.
The problem is that these objects should not be shared between interpreters. There are a number of possible solutions for isolating the objects, but the big constraint is that the solution cannot break the stable ABI.
ericsnowcurrently commentedon Mar 15, 2021
Here are some solutions that I've considered:
a. make the objects truly immutable/const
b. make the objects effectively immutable
a. replace them with macros that do a per-interpreter lookup
b. replace them with simple placeholders and do a per-interpreter lookup internally
c. replace them with PyObject placeholders and do a per-interpreter lookup internally
As far as I'm aware, only (1b) and (2c) are realistic and won't break the stable ABI (i.e. preserve layout).
(FWIW, I think that even with (1b) we would still have per-interpreter objects.)
-- Regarding (1a) --
See see #69016 for an example implementation. This includes storing some state for the objects in PyInterpreterState and doing a lookup internally.
pros:
cons:
-- Regarding (2c) --
This involves adding an API to get the per-interpreter object for a given identity value (flag, global, etc.) and then mapping the limited API objects to the corresponding per-interpreter ones.
pros:
cons:
ericsnowcurrently commentedon Mar 15, 2021
If the stable ABI weren't an issue then we would probably:
It may make sense to do that for "#ifndef Py_LIMITED_API", regardless of how we handle the limited API.
1 remaining item
vstinner commentedon Mar 15, 2021
This issue is discussed in bpo-39511 "[subinterpreters] Per-interpreter singletons (None, True, False, etc.)".
This issue is discussed in bpo-40601: "[C API] Hide static types from the limited C API".
vstinner commentedon Mar 15, 2021
Include/README file is being written, have a look ;-)
https://github.com/python/cpython/pull/24884/files
ericsnowcurrently commentedon Mar 16, 2021
One simple solution is to explicitly state that the limited API does not support subinterpreters. This is already implied by the fact that the multi-phase init API (PEP-489) requires subinterpreter support but is not part of the limited API.
If we establish this constraint then the problem I originally described here goes away (and we can close this issue).
(Note: I'm pretty sure this is something someone suggested to me at some point, rather than my own idea.)
ericsnowcurrently commentedon Mar 16, 2021
FYI, I posted to capi-sig about this:
https://mail.python.org/archives/list/capi-sig@python.org/thread/INLCGPMTYFLRTWQL7RB4MUQZ37JAFRAU/
mattip commentedon Mar 16, 2021
I am confused. How can widening the usable number of functions (i.e. using the whole C-API rather than the limited API) help c-extension modules be usable in subinterpreters? Aren't the same singletons, exception types, and other types exposed in the full C-API?
encukou commentedon Mar 16, 2021
There seems to be much confusion here. Maybe on my side?
PEP-489 is *very much* part of the limited API.
vstinner commentedon Mar 17, 2021
Eric Snow proposes that C extensions which want to be compatible with subinterpreters must use an hypothetical variant of the C API which doesn't inherit flaws of the current C API. For example, static types like "&PyLong_Type" would be excluded.
To be clear, the limited C API does expose (indirectly) "&PyLong_Type". We are talking about a new variant of the C API.
The main interpreter would continue to use its static type "&PyLong_Type", whereas each subinterpreter would get its own "int" type allocated on the heap (heap type).
Someone has to write a PoC to ensure that this idea works in practice.
In bpo-40601, I proposed that all interpreters including the main interpreter only use heap types: remove "&PyLong_Type" from the C API which is a backward incompatible C API change.
ericsnowcurrently commentedon Mar 17, 2021
If Py_LIMITED_API is defined then things would stay the same. Otherwise we would replace the names with macros to do the appropriate lookup. (That isn't the whole story since the Py*_Type names are PyTypeObject and not PyObject*.)
ericsnowcurrently commentedon Mar 17, 2021
Gah, I missed that. That said, I don't think it matters; I just lose an easy point in the rationale. :)
ericsnowcurrently commentedon Mar 17, 2021
FYI, I'm going to focus discussion on the capi-sig thread.