Repository navigation
Non-deterministic bytecode generation for frozenset #78903
Description
Activity
peterebden commented
on Sep 18, 2018 peterebdenmannequinMannequinAuthorWe've found that the following code produces non-deterministic bytecode,
even post PEP-552:def test(x): if x in {'ONE', 'TWO', 'THREE'}: pass
It's not too hard to test it:
$ python3.7 -m compileall --invalidation-mode=unchecked-hash test.py Compiling 'test.py'... $ sha1sum __pycache__/test.cpython-37.pyc 61e5682ca95e8707b4ef2a79f64566664dafd800 __pycache__/test.cpython-37.pyc $ rm __pycache__/test.cpython-37.pyc $ python3.7 -m compileall --invalidation-mode=unchecked-hash test.py Compiling 'test.py'... $ sha1sum __pycache__/test.cpython-37.pyc 222a06621b491879e5317b34e9dd715bacd89b7d __pycache__/test.cpython-37.pyc
It looks like the peephole optimiser is converting the LOAD_CONST instructions
for the set into a single LOAD_CONST for a frozenset which then serialises in
nondeterministic order. One can hence work around it by setting PYTHONHASHSEED
to a known value.I'm happy to help out with this if needed, although I don't have a lot of
familiarity with the relevant code.Have a look at line 512 in Python/marshal.c which calls PyObject_GetIter(). We would need to add PySequence_List() and PyList_Sort(). This will slow down marshaling but would make the bytecode deterministic.
Not all types are orderable.
>>> sorted({'', None}) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: '<' not supported between instances of 'NoneType' and 'str'
peterebden commented
on Sep 19, 2018 peterebdenmannequinMannequinAuthorThanks for the pointer, I'll have a bit more of a dig into it (although Serhiy makes a good point too...).
Possibly we should just sort the individual marsahalled entries of the frozenset.
Bumping up the priority a bit on this one. It would be nice to get it in for 3.8.
yan12125 commented
on Jul 13, 2019 yan12125mannequinMannequinI encounter another case that leads to non-deterministic bytecode. For example, with commit e6b46aa + PR 9472, I got:
$ cat foobar.py _m = None $ PYTHONHASHSEED=0 ./python -m compileall --invalidation-mode=unchecked-hash foobar.py Compiling 'foobar.py'... $ sha256sum __pycache__/foobar.cpython-39.pyc 7f84b08d5536390d6ce4ccb2d65e259449c56549ee9cc67560f61771824f20ea __pycache__/foobar.cpython-39.pyc $ rm __pycache__/foobar.cpython-39.pyc $ PYTHONHASHSEED=1 ./python -m compileall --invalidation-mode=unchecked-hash foobar.py Compiling 'foobar.py'... $ sha256sum __pycache__/foobar.cpython-39.pyc 46dadbb92ad6e1e5b5f8abe9f107086cd231b2b80c15fe84f86e2081a6b8c428 __pycache__/foobar.cpython-39.pyc
In this case there are no sets. I guess the cause is different. Should I open a new issue?
yan12125 commented
on Sep 5, 2021 yan12125mannequinMannequinbpo-37596 merged a fix to enable deterministic frozensets. I think this issue can be closed?
Regarding my last comment msg347820 - it seems similar to one of https://bugs.python.org/issue34033 or https://bugs.python.org/issue34093. I followed those tickets instead.
- changed the title
[-]Non-deterministic bytecode generation[/-][+]Non-deterministic bytecode generation for frozenset[/+]on May 4, 2022
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields: