Datetime NoneType after calling Py_Finalize and Py_Initialize

@ncoghlan

BPO	27400
Nosy	@ncoghlan, @abalkin, @tiran, @Fak3, @JimJJewett, @zooba, @MojoVampire, @ndjensen, @yan12125, @ammaraskar, @cschramm, @pganssle
Files	issue27400.patch 27400.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/abalkin'
closed_at = None
created_at = <Date 2016-06-27.13:49:18.523>
labels = ['interpreter-core', 'type-bug', '3.7']
title = 'Datetime NoneType after calling Py_Finalize and Py_Initialize'
updated_at = <Date 2018-07-05.17:14:18.266>
user = 'https://bugs.python.org/DennyWeinberg'

bugs.python.org fields:

activity = <Date 2018-07-05.17:14:18.266>
actor = 'pablogsal'
assignee = 'belopolsky'
closed = False
closed_date = None
closer = None
components = ['Interpreter Core']
creation = <Date 2016-06-27.13:49:18.523>
creator = 'Denny Weinberg'
dependencies = []
files = ['44678', '46478']
hgrepos = []
issue_num = 27400
keywords = ['patch']
message_count = 15.0
messages = ['269379', '269381', '269751', '269914', '269916', '276598', '276604', '276622', '276626', '276630', '276632', '277184', '285560', '286618', '286636']
nosy_count = 15.0
nosy_names = ['ncoghlan', 'belopolsky', 'christian.heimes', 'palm.kevin', 'Roman.Evstifeev', 'Jim.Jewett', 'steve.dower', 'josh.r', 'ndjensen', 'yan12125', 'Denny Weinberg', 'ammar2', 'cschramm', 'p-ganssle', 'FrankBlabu']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue27400'
versions = ['Python 3.6', 'Python 3.7']

Linked PRs

After calling Py_Finalize and Py_Initialize I get the message "attribute of type 'NoneType' is not callable" on the datetime.strptime method.

Example:

from datetime import datetime
s = '20160505 160000'
refdatim = datetime.strptime(s, '%Y%m%d %H%M%S')

The first call works fine but it crashes after the re initialization.

Workaround:

from datetime import datetime
s = '20160505 160000'
try:
    refdatim = datetime.strptime(s, '%Y%m%d %H%M%S')
except TypeError:
    import time
    refdatim = datetime.fromtimestamp(time.mktime(time.strptime(s, '%Y%m%d %H%M%S')))

Related Issue: bpo-17408 ("second python execution fails when embedding")

Just to be clear:

The error happens after these steps:

Call strptime
Call cpython function "Py_Finalize" and "Py_Initialize"
Call strptime again

Now we get the error "attribute of type 'NoneType' is not callable"

Thanks for the report Denny. Looking at https://hg.python.org/cpython/file/30099abdb3a4/Modules/datetimemodule.c#l3929, there's a problematic caching of the "_strptime" module that is almost certainly the cause of the problem - it will attempt to call _strptime._strptime from the already finalized interpreter rather than the new one.

It should be possible to adjust that logic to permit a check for _strptime._strptime being set to None, and reimporting _strptime in that case.

Is there any particular reason that datetime.strptime caches the imported module like that?

From a quick search, these two other examples don't bother with any caching:

cpython/Modules/timemodule.c

Line 709 in 2d26423

PyObject *strptime_module = PyImport_ImportModuleNoBlock("_strptime");

cpython/Python/traceback.c

Line 277 in 64fe35c

io = PyImport_ImportModuleNoBlock("io");

Aye, skipping the caching entirely would be an even simpler solution - the only thing it is saving in the typical case is a dictionary lookup in the modules cache.

Nick: Looks like it's quite a bit more work than just a dict lookup. That PyImport_ImportModuleNoBlock call (which seems odd; the implementation of NoBlock is just to wrap the blocking function; guess we don't allow non-blocking imports anymore and this is just to avoid changing all the names elsewhere?) involves a *lot* more work than just a dict lookup (it devolves to a PyImport_Import call https://hg.python.org/cpython/file/3.5/Python/import.c#l1743 , which basically does everything involved in the import process aside from actually reading/parsing the file unconditionally, because of how weird __import__ overrides can be, I guess).

While it's not a perfect comparison, compare:

>>> import _strptime  # It's now cached

# Cache globals dict for fair comparison without globals() call overhead
>>> g = globals()     

# Reimport (this might be *more* expensive at C layer, see notes below)
>>> %timeit -r5 import _strptime
1000000 loops, best of 5: 351 ns per loop

# Dict lookup (should be at least a bit cheaper at C layer if done equivalently, using GetAttrId to avoid temporary str)
>>> %timeit -r5 g['_strptime']
10000000 loops, best of 5: 33.1 ns per loop

# Cached reference (should be *much* cheaper at C layer)
>>> %timeit -r5 _strptime
100000000 loops, best of 5: 19.1 ns per loop

Note: I'm a little unclear on whether a Python function implemented in C has its own globals, or whether it's simulated as part of the C module initialization); if it lacks globals, then the work done for PyImport_Import looks like it roughly doubles (it has to do all sorts of work to simulate globals and the like), so that 351 ns per re-import might actually be costlier in C.

Either way, it's a >10x increase in cost to reimport compared to a dict lookup, and ~18x speedup over using a cached reference (and like I said, I think the real cost of the cheaper options would be much less in C, so the multiplier is higher). Admittedly, in tests, empty string calls to _strptime._strptime take around 7.4 microseconds (with realistic calls taking 8.5-13.5 microseconds), so caching is saving maybe a third of a microsecond overhead, maybe 2.5%-4.5% of the work involved in the strptime call.

Hmm... On checking down some of the code paths and realizing there were some issues in 3.5 (redundant code, and what looked like two memory leaks), I checked tip (to avoid opening bugs on stale code), and discovered that bpo-22557 rewrote the import code, reducing the cost of top level reimport by ~60%, so my microbenchmarks (run on Python 3.5.0) are already out of date for 3.6's faster re-import. Even so, caching wasn't a wholly unreasonable optimization before now, and undoing it now still has a cost, if a smaller one.

I am not sure this is possible to fix without refactoring the datetime module according to PEP-3121. See bpo-15390.

PEP-3121 is a big change. Can we use PyModuleDef->m_clear() for a clever hack?

Yes, I think something like the attached patch may do the trick.

Wouldn't it clear strptime_module when a subinterpreter shuts down, too? It's not a big deal because it can't cause a crash.

gh-71587: Drop local reference cache to `_strptime` module in `_datet…

pythongh-71587: Drop local reference cache to `_strptime` module in `…

This has been fixed by gh-120224. (Backports to 3.13 and 3.12 are in progress.)

[3.13] gh-71587: Drop local reference cache to `_strptime` module in …

[3.12] gh-71587: Drop local reference cache to `_strptime` module in …

pythongh-71587: Drop local reference cache to `_strptime` module in `…

pythongh-71587: Establish global state in `_datetime` (python#110475)

Navigation Menu

Datetime NoneType after calling Py_Finalize and Py_Initialize #71587

Linked PRs

31 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Datetime NoneType after calling Py_Finalize and Py_Initialize #71587

Description

Linked PRs

Activity

DennyWeinberg commented on Jun 27, 2016

DennyWeinberg commented on Jun 27, 2016

ncoghlan commented on Jul 3, 2016

ammaraskar commented on Jul 7, 2016

ncoghlan commented on Jul 7, 2016

MojoVampire commented on Sep 15, 2016

MojoVampire commented on Sep 15, 2016

abalkin commented on Sep 15, 2016

tiran commented on Sep 15, 2016

abalkin commented on Sep 15, 2016

tiran commented on Sep 15, 2016

31 remaining items

ericsnowcurrently commented on Jun 12, 2024

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions