Convert static types to heap types: use PyType_FromSpec()

@pitrou

BPO	40077
Nosy	@pitrou, @vstinner, @phsilva, @encukou, @skrah, @ericsnowcurrently, @serhiy-storchaka, @corona10, @pablogsal, @miss-islington, @shihai1991, @erlend-aasland, @h-vetinari, @nw0, @kumaraditya303, @CharlieZhao95
PRs	bpo-40077: Convert _jsonmodule to use PyType_FromSpec. #19177 bpo-40077: Convert _abc module to use PyType_FromSpec() #19202 [WIP] bpo-40077: Convert _bz2 extension module to use PyType_FromSpec() #19341 bpo-40077: Fix potential refleaks of _json #19344 bpo-40077: Remove redundant cast in _json extension module #19438 bpo-40077: Convert _bz2 module to use PyType_FromSpec #20960 bpo-40077: Convert _csv module to use PyType_FromSpec #20974 bpo-40077: Convert _operaator to use PyType_FromSpec #21953 bpo-40077: Convert _operator to use PyType_FromSpec #21954 bpo-40077: Convert mmap to use heap types #23108 bpo-40077: Convert arraymodule to use heap types and establish module state #23124 bpo-40077: Convert _queuemodule to use heap types #23136 bpo-40077: Convert _elementtree types to heap types #23428 bpo-42327: Add PyModule_Add() (smaller). #23443 bpo-40077: Fix typo in simplequeue_get_state_by_type() #23975 bpo-40077: Convert itertools types to heap types #24065 bpo-40077: Add traverse/clear/free to arraymodule #24066 bpo-40077: Convert zoneinfo type to heap type #24481 bpo-40077: Convert _pickle module to use heap types. #30884
Files	bench_isinstance_check.py bench_subclass_check.py bench_isinstance_check.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2020-03-26.16:04:44.633>
labels = ['extension-modules', 'type-feature', '3.10']
title = 'Convert static types to heap types: use PyType_FromSpec()'
updated_at = <Date 2022-02-02.05:50:10.693>
user = 'https://github.com/corona10'

bugs.python.org fields:

activity = <Date 2022-02-02.05:50:10.693>
actor = 'kumaraditya'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Extension Modules']
creation = <Date 2020-03-26.16:04:44.633>
creator = 'corona10'
dependencies = []
files = ['49007', '49008', '49009']
hgrepos = []
issue_num = 40077
keywords = ['patch']
message_count = 47.0
messages = ['365087', '365088', '365117', '365125', '365142', '365145', '365149', '365154', '365205', '365220', '365241', '365242', '365255', '365257', '365317', '365558', '365773', '366024', '366061', '368669', '371120', '371813', '371890', '372073', '372074', '372076', '372326', '375943', '380109', '380267', '380516', '380517', '380518', '383848', '383908', '384228', '384236', '384271', '384272', '384293', '410810', '411630', '411903', '411904', '411907', '411964', '412338']
nosy_count = 16.0
nosy_names = ['pitrou', 'vstinner', 'phsilva', 'petr.viktorin', 'skrah', 'eric.snow', 'serhiy.storchaka', 'corona10', 'pablogsal', 'miss-islington', 'shihai1991', 'erlendaasland', 'h-vetinari', 'nw0', 'kumaraditya', 'CharlieZhao']
pr_nums = ['19177', '19202', '19341', '19344', '19438', '20960', '20974', '21953', '21954', '23108', '23124', '23136', '23428', '23443', '23975', '24065', '24066', '24481', '30884']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue40077'
versions = ['Python 3.10']

Some of modules is not using PyType_FromSpec.
We need to convert them.

This changes can bring

allow to destroy types at exit!
allow subinterpreters to have their own "isolated" typ

We need to convert them.

Let me elaborate. Static types have multiple issues:

Their lifetime is not well defined.
It is not obvious when they are ready to be used.
They are not destroyed at exit.
They are incompatible with subinterpreters: each interpreter should have its own copy of a type, rather than static types are shared by all interpreters which cause problems with reference counting (require GIL or atomic operation).
They are causing issues with stable ABI (limited C API): PEP-384.

Wouldn't having less static types slow down startup time?

Pablo:

Wouldn't having less static types slow down startup time?

That's possible, even if I only expect a minor or non significant overhead.

But before starting talking about performances, we should focus on the correctness. Static types are causing issues with subinterpreters and the Python finalization.

New changeset 33f15a1 by Dong-hee Na in branch 'master':
bpo-40077: Convert _json module to use PyType_FromSpec() (GH-19177)
33f15a1

Wouldn't having less static types slow down startup time?

Yes, and not only startup time:

https://bugs.python.org/issue15722

In the _json module, PyModule_GetState() (called by get_json_state()) is only used by the garbage collector (traverse function) and to unload the module (clear and free functions). It's not used in "hot code" (let me consider that the GC is not part of the usual "hot code" :-)).

But maybe we should measure the overhead of future changes if PyModule_GetState() starts to be used in "hot code" by running a few microbenchmarks.

--

Stefan Krah:

Yes, and not only startup time:
https://bugs.python.org/issue15722

Aha, that's interesting. I didn't know that it could have an impact on runtime performance as well.

_decimal_pep3121-384_v1.patch attached to bpo-15722 seems to use:

#define _decimal_state_global ((_decimal_state *)PyModule_GetState(PyState_FindModule(&_decimal_module)))

whereas the commit 33f15a1 only uses:

static inline _jsonmodulestate*
get_json_state(PyObject *module)
{
    void *state = PyModule_GetState(module);
    assert(state != NULL);
    return (_jsonmodulestate *)state;
}

Maybe PyState_FindModule() adds a little overhead, even if this function is simple: in short, it gets the i-th item of a list (from PyInterpreterState.modules_by_index).

PyModule_GetState() function is really simple: it only reads PyModuleObject.md_state attribute.

Or maybe _decimal_state_global was used in "hot code".

If PyState_FindModule() or PyModule_GetState() is the source of the overhead, maybe we could try to optimise these functions, or pass directly the module state to inner functions.

--

PyState_FindModule() doesn't work for a module using PyModuleDef_Init(). The PEP-573 is going to give access to the module state in functions which didn't access to it previsouly.

The commit 33f15a1 removed a few assertions checking that "self" has the expected type. It wasn't possible to get the module state to get the types, because PEP-573 is not implemented yet and PyState_FindModule() doesn't work in _json (which uses PyModuleDef_Init()). I decided that it's ok to remove these assertions: it should not be possible to call these functions with another type in practice.

--

In his PR 19177, Dong-hee started by replacing direct access to PyTypeObject fields, like replacing:

type->free(self);

with:

freefunc free_func = PyType_GetSlot(tp, Py_tp_free);
free_func(self);

I asked him to revert these changes.

I'm interested to experiment a few C extension modules of the stdlib using the limited C API (PEP-384, stable ABI), but I don't think that it should done while converting modules to PyType_FromSpec().

We should separate the two changes. And I would prefer to first see the overhead of PyType_FromSpec(), and discuss the advantages and drawbacks.

Or maybe _decimal_state_global was used in "hot code".

Yes, _decimal has problems here that most modules don't have.
Modules like atexit are obviously fine. :)

I just posted it as an example why one should be a bit cautious.

The PEP-573 is going to give access to the module state in functions which didn't access to it previously.

That could help, we'll see.

And I would prefer to first see the overhead of PyType_FromSpec(), and discuss the advantages and drawbacks.

Should we stop the work until the overhead is measured?

Should we stop the work until the overhead is measured?

Can you try to measure the _abc._abc_instancecheck() and _abc._abc_subclasscheck() functions performance before/after your change? Functions like register() are rarely call, so I don't care much of their performance (I expect a small overhead or no overhead).

It shown 1% slower for performance.

+--------------------------+-----------------+----------------------------+
| Benchmark | master-subclass | bpo-40077-subclass |
+==========================+=================+============================+
| bench _abc_subclasscheck | 295 ns | 300 ns: 1.01x slower (+1%) |
+--------------------------+-----------------+----------------------------+

+--------------------------+-------------------+----------------------------+
| Benchmark | master-isinstance | bpo-40077-isinstance |
+==========================+===================+============================+
| bench _abc_instancecheck | 229 ns | 232 ns: 1.01x slower (+1%) |
+--------------------------+-------------------+----------------------------+

See also bpo-46417 "[subinterpreters] Clear static types in Py_Finalize()".

Converting further static types to heap types require a discussion. See what the Python Steering Council wrote at Feb 8, 2021:

"The Steering Council discussed the ongoing work on porting types in the standard library to heap-types and the subinterpreter-related changes. It was decided that through Pablo, the Steering Council will ask the core developers driving those changes to create an informational PEP and not to make any more changes in this area after beta 1, as per our general policy."

https://github.com/python/steering-council/blob/1d85eefdc5861a096c3859e9990dbc8527c5973b/updates/2021-02-steering-council-update.md

I used the following shell command to search remaining static types:
---
grep -E 'static PyTypeObject [a-zA-Z_0-9]+ *(;|= *{|) *$' $(find -name "*.c")
---

I found 86 static types in 17 files:

PC/_msi.c:
- msidb_Type
- msiview_Type
- record_Type
- summary_Type
Modules/_ctypes/_ctypes.c:
- DictRemover_Type
- PyComError_Type
- PyDec_Type
- Simple_Type
- StructParam_Type
- Struct_Type
- UnionType_Type
- Union_Type
Modules/_decimal/_decimal.c:
- PyDecContextManager_Type
- PyDecContext_Type
- PyDecSignalDict_Type
Modules/xxmodule.c:
- Null_Type
- Str_Type
- Xxo_Type
Modules/_testbuffer.c:
- NDArray_Type
- StaticArray_Type
Modules/itertoolsmodule.c (multiphase init):
- _grouper_type
- accumulate_type
- combinations_type
- compress_type
- count_type
- cwr_type
- cycle_type
- dropwhile_type
- filterfalse_type
- groupby_type
- pairwise_type
- permutations_type
- starmap_type
- takewhile_type
- tee_type
- teedataobject_type
Modules/_xxsubinterpretersmodule.c:
- ChannelIDtype
Modules/_datetimemodule.c:
- PyDateTime_DateTimeType
- PyDateTime_DateType
- PyDateTime_DeltaType
- PyDateTime_IsoCalendarDateType
- PyDateTime_TZInfoType
- PyDateTime_TimeType
- PyDateTime_TimeZoneType
Modules/_testcapimodule.c:
- ContainerNoGC_type
- GenericAlias_Type
- Generic_Type
- MethClass_Type
- MethInstance_Type
- MethStatic_Type
- MethodDescriptor2_Type
- MethodDescriptorBase_Type
- MethodDescriptorDerived_Type
- MethodDescriptorNopGet_Type
- MyList_Type
- PyRecursingInfinitelyError_Type
- _HashInheritanceTester_Type
- awaitType
- ipowType
- matmulType
- test_structmembersType
Modules/_zoneinfo.c:
- PyZoneInfo_ZoneInfoType
Modules/ossaudiodev.c:
- OSSAudioType
- OSSMixerType
Modules/socketmodule.c:
- sock_type
Modules/xxsubtype.c:
- spamdict_type
- spamlist_type
Modules/_collectionsmodule.c:
- defdict_type
- deque_type
- dequeiter_type
- dequereviter_type
- tuplegetter_type
Modules/_elementtree.c:
- ElementIter_Type
- Element_Type
- TreeBuilder_Type
- XMLParser_Type
Modules/_pickle.c:
- Pdata_Type
- PicklerMemoProxyType
- Pickler_Type
- UnpicklerMemoProxyType
- Unpickler_Type
Modules/_asynciomodule.c:
- FutureIterType
- FutureType
- PyRunningLoopHolder_Type
- TaskStepMethWrapper_Type
- TaskType

And I found 135 more static types with this command:
---
grep -E '^PyTypeObject [a-zA-Z_0-9]+ *(;|= *{|) *$' $(find -name "*.c")
---

Types:

And I found 135 more static types with this command

Of these 135 static types, most are cleared since bpo-46417 was implemented:

103 types are cleared by _PyTypes_FiniTypes()
15 types are cleared by _PyIO_Fini()
the remaining 17 types are not cleared at Python exit: types of _ctypes, _curses and _multiprocessing extensions

I marked bpo-23769 "valgrind reports leaks for test_zipimport" as duplicate of this issue. At exit, Python doesn't clear the static types of the _collections, itertools and _struct extensions:

itertools.accumulate
itertools.combinations
itertools.combinations_with_replacement
itertools.cycle
itertools.dropwhile
itertools.takewhile
itertools.islice
itertools.starmap
itertools.chain
itertools.compress
itertools.filterfalse
itertools.count
itertools.zip_longest
itertools.pairwise
itertools.permutations
itertools.product
itertools.repeat
itertools.groupby
itertools._grouper
itertools._tee
itertools._tee_dataobject
collections.deque
_collections._deque_iterator
_collections._deque_reverse_iterator
_collections._tuplegetter
_struct.Struct
_struct.unpack_iterator

See: https://bugs.python.org/issue23769#msg411963

On Windows PC/winreg.c has PyHKEY_Type static type which isn't cleared at exit too.

PEP 687 was accepted

PEP 687 was accepted

Note that this does not imply that all types shall be converted to heap types. This is also mentioned in the PEP; please read it carefully.

Note that this does not imply that all types shall be converted to heap types. This is also mentioned in the PEP; please read it carefully.

We are on the same page, every module needs to considered individually and I am against converting any built-in modules or types unless it's necessary.

Superseded by #103092

Navigation Menu

Repository navigation

Convert static types to heap types: use PyType_FromSpec() #84258

49 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Convert static types to heap types: use PyType_FromSpec() #84258

Description

Activity

corona10 commented on Mar 26, 2020

vstinner commented on Mar 26, 2020

pablogsal commented on Mar 26, 2020

vstinner commented on Mar 27, 2020

vstinner commented on Mar 27, 2020

skrah commented on Mar 27, 2020

vstinner commented on Mar 27, 2020

skrah commented on Mar 27, 2020

corona10 commented on Mar 28, 2020

vstinner commented on Mar 28, 2020

corona10 commented on Mar 29, 2020

49 remaining items

vstinner commented on Jan 17, 2022

vstinner commented on Jan 25, 2022

vstinner commented on Jan 27, 2022

vstinner commented on Jan 27, 2022

vstinner commented on Jan 27, 2022

vstinner commented on Jan 28, 2022

kumaraditya303 commented on Feb 2, 2022

kumaraditya303 commented on Jun 27, 2022

erlend-aasland commented on Jun 27, 2022

kumaraditya303 commented on Jun 28, 2022

kumaraditya303 commented on May 17, 2023

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions