Repository navigation
struct and memoryview tests rely on undefined behavior (as revealed by clang 9) #83870
Description
Activity
stratakis commented
on Feb 19, 2020 stratakismannequinMannequinAuthorThe clang build was recently added for that buildbot and it seems on that particular architecture, test_struct fails with:
======================================================================
FAIL: test_bool (test.test_struct.StructTest)
----------------------------------------------------------------------Traceback (most recent call last): File "/home/dje/cpython-buildarea/3.x.edelsohn-fedora-rawhide-z.clang-ubsan/build/Lib/test/test_struct.py", line 520, in test_bool self.assertTrue(struct.unpack('>?', c)[0]) AssertionError: False is not true
https://buildbot.python.org/all/#/builders/488/builds/6
Fedora rawhide recently upgraded Clang to version 10. The rest of the architectures seem fine.
stratakis commented
on Feb 19, 2020 stratakismannequinMannequinAuthorFailed assertion here: https://github.com/python/cpython/blob/master/Lib/test/test_struct.py#L520
stratakis commented
on Feb 26, 2020 stratakismannequinMannequinAuthorOn this loop:
for c in [b'\x01', b'\x7f', b'\xff', b'\x0f', b'\xf0']: self.assertTrue(struct.unpack('>?', c)[0])
It fails for the b'\xf0' case
The call:
struct.unpack('>?', b'\xf0')
means to unpack a "native bool", i.e. native size and alignment. Internally, this does:static PyObject * nu_bool(const char *p, const formatdef *f) { _Bool x; memcpy((char *)&x, p, sizeof x); return PyBool_FromLong(x != 0); }
i.e., copies "sizeof x" (1 byte) of memory to a temporary buffer x, and then treats that as _Bool.
While I don't have access to the C standard, I believe it says that assignment of a true value to _Bool can coerce to a unique "true" value. It seems that if a char doesn't have the exact bit pattern for true or false, casting to _Bool is undefined behavior. Is that correct?
Clang 10 on s390x seems to take advantage of this: it probably only looks at the last bit(s) so a _Bool with a bit pattern of 0xf0 turns out false.
But the tests assume that 0xf0 should unpack to True.C compiler dev that it's indeed undefined behavior.
Quick and obvious fix:
static PyObject * nu_bool(const char \*p, const formatdef \*f) { char x; memcpy((char \*)&x, p, sizeof x); return PyBool_FromLong(x != 0); }Which is optimized to
static PyObject * nu_bool(const char \*p, const formatdef \*f) { return PyBool_FromLong(*p != 0); }I'm left with a question for CPython's struct experts:
The above would be my preferred fix, but the Python code is asking to convert a memory buffer to bool *using platform-specific semantics*.
Is this fix OK if C treats a \xf0 _Bool as falsey?(Also, this assumes size of _Bool is the same as size of char.
I guess we can add a build-time assertion for that, and say we don't support platforms where that's not the case.)maybe we should be raising an error if the bytes are not a valid platform _Bool pattern?
the concept of a native _Bool seems fuzzy. the important thing for the struct module is to consume sizeof _Bool bytes from the input stream. how those are interpreted is up to the platform. So if the platform says a bool is 8 bytes and it only ever looks at the lowest bit in those for bool-ness, good for it.
in that situation our unittest assuming that b'\xf0' should be true when interpreted as a bool is wrong.
just get rid of that value from the loop in the test?
24 remaining items
skrah commented
on Mar 12, 2020 skrahmannequinMannequinmemoryview only supports the native format, so I've disabled the
(wrong) test that casts arrays with arbitrary values to _Bool. So
memoryview is done.IMO the problem in _struct is that it swaps the x->unpack function
for the native one, which does not seem right for _Bool:/* Scan through the native table, find a matching entry in the endian table and swap in the native implementations whenever possible (64-bit platforms may not have "standard" sizes) */If one disables that swap, the tests pass here.
You are the one who wanted to *introduce* a hack by dereferencing
as char and then cast to _Bool. :-)Yes, I did change my mind after reading the documentation.
The docs say two contradicting things:
- The '?' conversion code corresponds to the _Bool type defined by C99
- ... any non-zero value will be True when unpacking.
So it's clear that something has to change. IMO, preserving (2) and relaxing (1) is the more useful choice.
skrah commented
on Mar 17, 2020 skrahmannequinMannequinSo it's clear that something has to change. IMO, preserving (2) and relaxing (1) is the more useful choice.
But not in this issue I think. #63169 is a minimal change that
*removes* UB for the standard sizes.UB for the native type is a direct consequence of using _Bool.
Native types should be left as is because that's what array
libraries expect. The docs could need a change (in another issue).Also, UB can only happen in a constructed example --- correctly
packed arrays don't have any incorrect values.So I think any fear of UB here is not warranted.
Note that the implementation of np_bool in _struct.c [1] is incorrect because this is supposed to access a boolean of a standard size, but uses _Bool. The size of _Bool is not prescribed, and IIRC sizeof(_Bool) was 4 with the compilers used for macOS/PPC.
[1] https://github.com/python/cpython/blob/master/Modules/_struct.c#L703
Sigh... never mind, I misread the code. Please ignore msg364472
I think we are speaking past each other.
In my (current) view, the semantics are spelled out in the documentation: "any non-zero value will be True when unpacking".
There's also a mention that this corresponds to the _Bool type in C. While this was the case with compilers in the past, it's no longer true with clang 9.In your view, the semantics are dictated by the correspondence to _Bool, and the "non-zero value will be True when unpacking" is the fluff to be ignored and removed.
The docs assume the two behaviors (_Bool and non-zero) are equivalent. In this bug we find out that they are not, so to fix the bug, we need to make a choice which one to keep and which one to throw out.
I see nothing that would make one view inherently better than the other.What "array libraries expect" is IMO not relevant: under any of the two views, libraries that are well-written (under that view) will be fine. Problems come when the library and Python choose different sides, e.g. when a non-C library can't use _Bool and thus packs arrays with the expectation that "any non-zero value will be True when unpacking".
What is a minimal change in *implementation* is a bigger change in *behavior*: unpacking of arrays will now depend greatly on details like the compiler used to build Python. I see that as the greater evil: since the data can be sharted across environments, languages and compilers, keeping the semantics well-defined seems better than leaving them to the compiler.
I don't see a compelling reason to choose _Bool semantics, but perhaps there is one.skrah commented
on Mar 18, 2020 skrahmannequinMannequinI think this issue should be about fixing the tests so that people
looking at the sanitizer buildbots can move on.#63169 fixes "<?", ">?" and "!?", which clearly used wrong
semantics with the new compiler behavior. This should be an
uncontroversial fix that also takes care of test_struct.Can we please discuss native _Bool in another issue?
There is no non-hackish way of unpacking _Bool if new compilers
essentially treat values outside [0, 1] as a trap representation.You could determine sizeof(_Bool), use the matching unsigned type,
unpack as that, then cast to _Bool. But do you really want to force
that procedure on all array libraries that want to be PEP-3118
compatible?I'd rather deprecate _Bool and use uint8_t, but that definitely
deserves a separate issue.I see. Thanks for your patience explaining this to me!
I will merge and continue in a different issue.
Moved to Discourse, IMO that's a better place for maintainers of other PEP-3118-compatible libraries to chime in:
https://discuss.python.org/t/behavior-of-struct-format-native-bool/3774Objects/memoryview.c uses memcpy() on _Bool which leads to undefined behavior with GCC 11: see bpo-42587.
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields: