Skip to content

test_binascii: test_b2a_roundtrip() failure with Hypothesis #109421

Closed as not planned
Closed as not planned
test_binascii: test_b2a_roundtrip() failure with Hypothesis#109421
Labels
testsTests in the Lib/test dirtype-bugAn unexpected behavior, bug, or error

Description

@vstinner
vstinner
opened on Sep 14, 2023

It seems like Hypothesis found an issue. Can someone have a look? How can it be reproduced locally?

ERROR: test_b2a_roundtrip (test.test_binascii.MemoryviewBinASCIITest.test_b2a_roundtrip)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_binascii.py", line 233, in test_b2a_roundtrip
    binary=hypothesis.strategies.binary(),
           ^^^^^^^
  File "/home/runner/work/cpython/cpython-builddir/hypovenv/lib/python3.13/site-packages/hypothesis/core.py", line 1386, in wrapped_test
    raise the_error_hypothesis_found
  File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_binascii.py", line 237, in test_b2a_roundtrip
    converted = binascii.b2a_uu(self.type2test(binary), backtick=backtick)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
binascii.Error: At most 45 bytes at once
Falsifying example: test_b2a_roundtrip(
    self=<test.test_binascii.MemoryviewBinASCIITest testMethod=test_b2a_roundtrip>,
    binary=b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00',
    backtick=False,
)

You can reproduce this example by temporarily adding @reproduce_failure('6.84.0', b'AXicY2RgYBwciIEBAA2TAC8=') as a decorator on your test case

----------------------------------------------------------------------
Ran 85 tests in 127.421s

FAILED (errors=1, skipped=1)
test test_binascii failed

build: https://github.com/python/cpython/actions/runs/6189519251/job/16803785370

Activity

  1. vstinner
    added
    type-bugAn unexpected behavior, bug, or error
    testsTests in the Lib/test dir
    on Sep 14, 2023

  2. vstinner commented on Sep 14, 2023

    @vstinner
    vstinner
    on Sep 14, 2023
    MemberAuthor

    I modified the test as requested to try to reproduce the issue:

    diff --git a/Lib/test/test_binascii.py b/Lib/test/test_binascii.py
index 3d3e0746e9..cec72c5c78 100644
--- a/Lib/test/test_binascii.py
+++ b/Lib/test/test_binascii.py
@@ -5,7 +5,7 @@
 import array
 import re
 from test.support import bigmemtest, _1G, _4G
-from test.support.hypothesis_helper import hypothesis
+import hypothesis
 
 
 # Note: "*_hex" functions are aliases for "(un)hexlify"
@@ -16,6 +16,7 @@
 all_functions = a2b_functions + b2a_functions + ['crc32', 'crc_hqx']
 
 
+@hypothesis.reproduce_failure('6.84.0', b'AXicY2RgYBwciIEBAA2TAC8=')
 class BinASCIITest(unittest.TestCase):
 
     type2test = bytes

    I install hypothesis in a venv:

    ./python -m venv env
env/bin/python -m pip install hypothesis

    When I try to reproduce the failure, I get:

    vstinner@mona$ env/bin/python -m test test_binascii -v
(...)
======================================================================
ERROR: test_b2a_qp_a2b_qp_round_trip (test.test_binascii.BinASCIITest.test_b2a_qp_a2b_qp_round_trip)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vstinner/python/main/Lib/test/test_binascii.py", line 400, in test_b2a_qp_a2b_qp_round_trip
    binary=hypothesis.strategies.binary(),
           ^^^^^^^
  File "/home/vstinner/python/main/env/lib/python3.13/site-packages/hypothesis/core.py", line 1263, in wrapped_test
    fail_health_check(settings, msg, HealthCheck.differing_executors)
  File "/home/vstinner/python/main/env/lib/python3.13/site-packages/hypothesis/internal/healthcheck.py", line 27, in fail_health_check
    raise FailedHealthCheck(message)
hypothesis.errors.FailedHealthCheck: The method BinASCIITest.test_b2a_qp_a2b_qp_round_trip was called from multiple different executors. This may lead to flaky tests and nonreproducible errors when replaying from database.
See https://hypothesis.readthedocs.io/en/latest/healthchecks.html for more information about this. If you want to disable just this health check, add HealthCheck.differing_executors to the suppress_health_check settings for this test.
(...)

    And then many tests fail with the same error message:

    • test_b2a_qp_a2b_qp_round_trip (test.test_binascii.BinASCIITest.test_b2a_qp_a2b_qp_round_trip)
    • test_b2a_roundtrip (test.test_binascii.BinASCIITest.test_b2a_roundtrip)
    • test_base64_roundtrip (test.test_binascii.BinASCIITest.test_base64_roundtrip)
    • test_hex_roundtrip (test.test_binascii.BinASCIITest.test_hex_roundtrip)
    • test_b2a_qp_a2b_qp_round_trip (test.test_binascii.BytearrayBinASCIITest.test_b2a_qp_a2b_qp_round_trip)
    • test_b2a_roundtrip (test.test_binascii.BytearrayBinASCIITest.test_b2a_roundtrip)
    • test_base64_roundtrip (test.test_binascii.BytearrayBinASCIITest.test_base64_roundtrip)
    • test_hex_roundtrip (test.test_binascii.BytearrayBinASCIITest.test_hex_roundtrip)
    • test_b2a_qp_a2b_qp_round_trip (test.test_binascii.MemoryviewBinASCIITest.test_b2a_qp_a2b_qp_round_trip)
    • test_b2a_roundtrip (test.test_binascii.MemoryviewBinASCIITest.test_b2a_roundtrip)
    • test_base64_roundtrip (test.test_binascii.MemoryviewBinASCIITest.test_base64_roundtrip)
    • test_hex_roundtrip (test.test_binascii.MemoryviewBinASCIITest.test_hex_roundtrip)

    I gave up at the FailedHealthCheck.

  3. vstinner commented on Sep 14, 2023

    @vstinner
    vstinner
    on Sep 14, 2023
    MemberAuthor

    cc @pganssle @Zac-HD

  4. hugovk commented on Sep 14, 2023

    @hugovk
    hugovk
    on Sep 14, 2023
    Member

    The good news is this is a duplicate of #109418 that was fixed moments ago by #109419.

    Perhaps @sobolevn can assist with reproducing, if you're still interested?

  5. sobolevn commented on Sep 14, 2023

    @sobolevn
    sobolevn
    on Sep 14, 2023
    Member

    @vstinner try this:

    » git patch
diff --git Lib/test/test_binascii.py Lib/test/test_binascii.py
index ef744f6b972..db7d25ab98e 100644
--- Lib/test/test_binascii.py
+++ Lib/test/test_binascii.py
@@ -233,6 +233,7 @@ def test_uu(self):
         binary=hypothesis.strategies.binary(max_size=45),
         backtick=hypothesis.strategies.booleans(),
     )
+    @hypothesis.reproduce_failure('6.84.0', b'AXicY2RgYBwciIEBAA2TAC8=')
     def test_b2a_roundtrip(self, binary, backtick):
         converted = binascii.b2a_uu(self.type2test(binary), backtick=backtick)
         restored = binascii.a2b_uu(self.type2test(converted))

    It produces a correct result on main (after my fix):

    ======================================================================
ERROR: test_b2a_roundtrip (test.test_binascii.MemoryviewBinASCIITest.test_b2a_roundtrip)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/sobolev/Desktop/cpython/Lib/test/test_binascii.py", line 233, in test_b2a_roundtrip
    binary=hypothesis.strategies.binary(max_size=45),
           ^^^^^^^
  File "/Users/sobolev/Desktop/cpython/.venv/lib/python3.13/site-packages/hypothesis/core.py", line 1290, in wrapped_test
    raise DidNotReproduce(
hypothesis.errors.DidNotReproduce: Expected the test to raise an error, but it completed successfully.

----------------------------------------------------------------------
  6. sobolevn
    closed this as not plannedon Sep 14, 2023

  7. sobolevn commented on Sep 14, 2023

    @sobolevn
    sobolevn
    on Sep 14, 2023
    Member

    Btw, if you have any questions about hypothesis, feel free to ping me :)

  8. vstinner commented on Sep 14, 2023

    @vstinner
    vstinner
    on Sep 14, 2023
    MemberAuthor

    hypothesis.errors.FailedHealthCheck: The method BinASCIITest.test_b2a_qp_a2b_qp_round_trip was called from multiple different executors. This may lead to flaky tests and nonreproducible errors when replaying from database.
    See https://hypothesis.readthedocs.io/en/latest/healthchecks.html for more information about this. If you want to disable just this health check, add HealthCheck.differing_executors to the suppress_health_check settings for this test.

    I don't understand this message. Why does it want to contact a remote server since I want to reproduce the issue locally? Maybe the hint in the first message is wrong?

    You can reproduce this example by temporarily adding @reproduce_failure('6.84.0', b'AXicY2RgYBwciIEBAA2TAC8=') as a decorator on your test case

  9. vstinner commented on Sep 14, 2023

    @vstinner
    vstinner
    on Sep 14, 2023
    MemberAuthor

    The good news is this is a duplicate of #109418 that was fixed moments ago by #109419.

    Oh, two hours ago, nice.

  10. sobolevn commented on Sep 14, 2023

    @sobolevn
    sobolevn
    on Sep 14, 2023
    Member

    I don't understand this message. Why does it want to contact a remote server since I want to reproduce the issue locally? Maybe the hint in the first message is wrong?

    This is interesting, it should not happen, because I disabled this check ~a week ago:

    cpython/Lib/test/support/hypothesis_helper.py

    Lines 13 to 16 in 68a6f21

    suppress_health_check=[
    hypothesis.HealthCheck.too_slow,
    hypothesis.HealthCheck.differing_executors,
    ],

    I will try to reproduce your case.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    testsTests in the Lib/test dirtype-bugAn unexpected behavior, bug, or error

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions