Skip to content

hashlib segmentation fault #79103

Closed
Closed

Description

@httpsgithubcomxcainiao
BPO 34922
Nosy @terryjreedy, @vstinner, @tiran, @ned-deily, @serhiy-storchaka, @tirkarthi, @https://github.com/xcainiao
PRs
  • bpo-34922: Fixed integer overflow in the digest() and hexdigest() methods #9751
  • [3.6] bpo-34922: Fix integer overflow in the digest() and hexdigest() methods (GH-9751) #9797
  • [3.7] bpo-34922: Fix integer overflow in the digest() and hexdigest() methods (GH-9751) #9798
  • [3.6] bpo-34922: Fix integer overflow in the digest() and hexdigest() methods (GH-9751) (GH-9798) #9801
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2020-01-14.22:56:24.735>
    created_at = <Date 2018-10-07.12:40:29.942>
    labels = ['extension-modules', '3.7', '3.8', 'type-crash']
    title = 'hashlib segmentation fault'
    updated_at = <Date 2020-01-14.22:56:24.734>
    user = 'https://github.com/httpsgithubcomxcainiao'

    bugs.python.org fields:

    activity = <Date 2020-01-14.22:56:24.734>
    actor = 'ned.deily'
    assignee = 'none'
    closed = True
    closed_date = <Date 2020-01-14.22:56:24.735>
    closer = 'ned.deily'
    components = ['Extension Modules']
    creation = <Date 2018-10-07.12:40:29.942>
    creator = 'shuoz'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 34922
    keywords = ['patch']
    message_count = 16.0
    messages = ['327277', '327283', '327285', '327288', '327294', '327306', '327312', '327495', '327515', '327518', '327519', '327604', '327606', '327619', '327677', '360009']
    nosy_count = 7.0
    nosy_names = ['terry.reedy', 'vstinner', 'christian.heimes', 'ned.deily', 'serhiy.storchaka', 'xtreak', 'shuoz']
    pr_nums = ['9751', '9797', '9798', '9801']
    priority = None
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'crash'
    url = 'https://bugs.python.org/issue34922'
    versions = ['Python 3.6', 'Python 3.7', 'Python 3.8']

    Activity

    1. httpsgithubcomxcainiao commented on Oct 7, 2018

      httpsgithubcomxcainiaomannequin
      MannequinAuthor

      python hashlib a signd overflow maybe cause a memory over read.

      python version:
      Python 3.6.7rc1+ (heads/3.6:cb0bec3, Oct 1 2018, 02:19:39)
      [GCC 7.3.0] on linux
      Type "help", "copyright", "credits" or "license" for more information.

      [----------------------------------registers-----------------------------------]
      RAX: 0x0
      RBX: 0x7fffffffd5f0 --> 0x41b58ab3
      RCX: 0x0
      RDX: 0x1ffffffffffffff6
      RSI: 0x7ffff35ae880 --> 0x0
      RDI: 0x7fffffffd650 --> 0x7d828fe8a42b9c7f
      RBP: 0xffffffffabe --> 0x0
      RSP: 0x7fffffffd5c8 --> 0x7ffff2a5f793 (<_sha3_shake_128_hexdigest+627>:	test   eax,eax)
      RIP: 0x7ffff2a5ec60 (<_PySHA3_KeccakWidth1600_SpongeSqueeze>:	push   r15)
      R8 : 0x65fc7ba985946aff
      R9 : 0xefbdaa140b587a16
      R10: 0x50573373c9b2b8dc
      R11: 0xfba4d93abbdabffc
      R12: 0x7fffffffd770 --> 0x7fffffffd7d0 --> 0xffffffffb00 --> 0x0
      R13: 0x7fffffffd650 --> 0x7d828fe8a42b9c7f
      R14: 0x7ffff35ae880 --> 0x0
      R15: 0xfffffffffffffff6
      EFLAGS: 0xa06 (carry PARITY adjust zero sign trap INTERRUPT direction OVERFLOW)
      [-------------------------------------code-------------------------------------]
         0x7ffff2a5ec50 <_PySHA3_KeccakP1600_ExtractBytes+160>:	jmp    0x7ffff2a54d10 <_PySHA3_KeccakP1600_ExtractBytesInLane@plt>
         0x7ffff2a5ec55:	nop
         0x7ffff2a5ec56:	nop    WORD PTR cs:[rax+rax*1+0x0]
      => 0x7ffff2a5ec60 <_PySHA3_KeccakWidth1600_SpongeSqueeze>:	push   r15
         0x7ffff2a5ec62 <_PySHA3_KeccakWidth1600_SpongeSqueeze+2>:	push   r14
         0x7ffff2a5ec64 <_PySHA3_KeccakWidth1600_SpongeSqueeze+4>:	push   r13
         0x7ffff2a5ec66 <_PySHA3_KeccakWidth1600_SpongeSqueeze+6>:	push   r12
         0x7ffff2a5ec68 <_PySHA3_KeccakWidth1600_SpongeSqueeze+8>:	mov    r13,rdx
      [------------------------------------stack-------------------------------------]
      0000| 0x7fffffffd5c8 --> 0x7ffff2a5f793 (<_sha3_shake_128_hexdigest+627>:	test   eax,eax)
      0008| 0x7fffffffd5d0 --> 0x7fffffffd5f0 --> 0x41b58ab3
      0016| 0x7fffffffd5d8 --> 0xffffefdb33b --> 0x0
      0024| 0x7fffffffd5e0 --> 0x7ffff7ed99d8 --> 0x0
      0032| 0x7fffffffd5e8 --> 0x7ffff3606910 --> 0x6190000096e5 --> 0x9000009828000000
      0040| 0x7fffffffd5f0 --> 0x41b58ab3
      0048| 0x7fffffffd5f8 --> 0x7ffff2a68c08 ("2 32 8 6 length 96 224 4 temp ")
      0056| 0x7fffffffd600 --> 0x7ffff2a5f520 (<_sha3_shake_128_hexdigest>:	push   r15)
      [------------------------------------------------------------------------------]
      Legend: code, data, rodata, value
      
      Breakpoint 2, _PySHA3_KeccakWidth1600_SpongeSqueeze (instance=0x7fffffffd650, data=0x7ffff35ae880 "", dataByteLen=0x1ffffffffffffff6) at /home/test/cpython/Modules/_sha3/kcp/KeccakSponge.inc:272
      ``````py
      dataByteLen=0x1ffffffffffffff6
      
      
      
      RAX: 0x7ffff3615f90 --> 0xfffffffffffffffa
      RBX: 0xa8
      RCX: 0x7ffff3616028 --> 0xf938000001a4
      RDX: 0x18
      RSI: 0x7fffffffd6e0 --> 0x6ab2a5fe4fe8efd
      RDI: 0x7ffff3615fe0 --> 0x44b6a41dfdc1a3df
      RBP: 0x7fffffffd510 --> 0xa8
      RSP: 0x7fffffffcc78 --> 0x7ffff6e936cf (mov    rcx,QWORD PTR [rbp-0x38])
      RIP: 0x7ffff6120786 (<__memmove_sse2_unaligned_erms+614>:	movntdq XMMWORD PTR [rdi+0x20],xmm2)
      R8 : 0xfffffffffffffff0
      R9 : 0x10007e6bac07 --> 0x0
      R10: 0x7ffff3616038 --> 0x0
      R11: 0x7ffff3615f90 --> 0xfffffffffffffffa
      R12: 0x7ffff3615f90 --> 0xfffffffffffffffa
      R13: 0x7fffffffd650 --> 0xa35bf3e9cd13e78e
      R14: 0x7ffff3615f90 --> 0xfffffffffffffffa
      R15: 0x0
      EFLAGS: 0x10206 (carry PARITY adjust zero sign trap INTERRUPT direction overflow)
      [-------------------------------------code-------------------------------------]
         0x7ffff6120779 <__memmove_sse2_unaligned_erms+601>:	sub    rdx,0x40
         0x7ffff612077d <__memmove_sse2_unaligned_erms+605>:	movntdq XMMWORD PTR [rdi],xmm0
         0x7ffff6120781 <__memmove_sse2_unaligned_erms+609>:	movntdq XMMWORD PTR [rdi+0x10],xmm1
      => 0x7ffff6120786 <__memmove_sse2_unaligned_erms+614>:	movntdq XMMWORD PTR [rdi+0x20],xmm2
         0x7ffff612078b <__memmove_sse2_unaligned_erms+619>:	movntdq XMMWORD PTR [rdi+0x30],xmm3
         0x7ffff6120790 <__memmove_sse2_unaligned_erms+624>:	add    rdi,0x40
         0x7ffff6120794 <__memmove_sse2_unaligned_erms+628>:	cmp    rdx,0x40
         0x7ffff6120798 <__memmove_sse2_unaligned_erms+632>:	ja     0x7ffff6120758 <__memmove_sse2_unaligned_erms+568>
      [------------------------------------stack-------------------------------------]
      0000| 0x7fffffffcc78 --> 0x7ffff6e936cf (mov    rcx,QWORD PTR [rbp-0x38])
      0008| 0x7fffffffcc80 --> 0x7fffffffccf0 --> 0x41b58ab3
      0016| 0x7fffffffcc88 --> 0x7fffffffcd90 --> 0x6
      0024| 0x7fffffffcc90 --> 0xffffffff99e --> 0x0
      0032| 0x7fffffffcc98 --> 0x7fffffffcd50 --> 0x0
      0040| 0x7fffffffcca0 --> 0x0
      0048| 0x7fffffffcca8 --> 0x7ffff3616038 --> 0x0
      0056| 0x7fffffffccb0 --> 0x7ffff358a068 --> 0x1
      [------------------------------------------------------------------------------]
      Legend: code, data, rodata, value
      Stopped reason: SIGSEGV
      __memmove_sse2_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:492
      492	../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S: No such file or directory.
      gdb-peda$ bt
      #0  __memmove_sse2_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:492
      #1  0x00007ffff6e936cf in ?? () from /usr/lib/x86_64-linux-gnu/libasan.so.4
      #2  0x00007ffff2a5eab4 in memcpy (__len=0xa8, __src=<optimized out>, __dest=<optimized out>) at /usr/include/x86_64-linux-gnu/bits/string_fortified.h:34
      #3  _PySHA3_KeccakP1600_ExtractLanes (state=<optimized out>, data=<optimized out>, laneCount=0x15) at /home/test/cpython/Modules/_sha3/kcp/KeccakP-1600-opt64.c:342
      #4  0x00007ffff2a5ec2c in _PySHA3_KeccakP1600_ExtractBytes (state=0x7fffffffd650, data=0x7ffff3615f90 "\372\377\377\377\377\377\377\377\002", offset=<optimized out>, length=0xa8)
          at /home/test/cpython/Modules/_sha3/kcp/KeccakP-1600-opt64.c:375
      #5  0x00007ffff2a5ee1d in _PySHA3_KeccakWidth1600_SpongeSqueeze (instance=0x7fffffffd650, data=<optimized out>, dataByteLen=0x1ffffffffffffff6)
          at /home/test/cpython/Modules/_sha3/kcp/KeccakSponge.inc:287
      #6  0x00007ffff2a5f793 in _SHAKE_digest (hex=0x1, digestlen=0xfffffffffffffff6, self=0x7ffff7ed98e8) at /home/test/cpython/Modules/_sha3/sha3module.c:620
      #7  _sha3_shake_128_hexdigest_impl (length=0xfffffffffffffff6, self=0x7ffff7ed98e8) at /home/test/cpython/Modules/_sha3/sha3module.c:669
      #8  _sha3_shake_128_hexdigest (self=0x7ffff7ed98e8, args=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>) at /home/test/cpython/Modules/_sha3/clinic/sha3module.c.h:149
      #9  0x000055555583eab6 in _PyCFunction_FastCallDict (kwargs=0x0, nargs=0x1, args=0x616000021518, func_obj=0x7ffff2e86f30) at Objects/methodobject.c:250
      #10 _PyCFunction_FastCallKeywords (func=func@entry=0x7ffff2e86f30, stack=0x616000021518, nargs=nargs@entry=0x1, kwnames=kwnames@entry=0x0) at Objects/methodobject.c:294
      #11 0x0000555555995945 in call_function (pp_stack=pp_stack@entry=0x7fffffffdc30, oparg=oparg@entry=0x1, kwnames=kwnames@entry=0x0) at Python/ceval.c:4837
      #12 0x000055555599feaa in _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:3335
      #13 0x0000555555994939 in PyEval_EvalFrameEx (throwflag=0x0, f=0x616000021398) at Python/ceval.c:754
      #14 _PyEval_EvalCodeWithName (_co=_co@entry=0x7ffff36088a0, globals=globals@entry=0x0, locals=locals@entry=0x7ffff355a9d8, args=args@entry=0x0, argcount=argcount@entry=0x0, kwnames=kwnames@entry=0x0,
          kwargs=0x0, kwcount=0x0, kwstep=0x2, defs=0x0, defcount=0x0, kwdefs=0x0, closure=0x0, name=0x0, qualname=0x0) at Python/ceval.c:4166
      #15 0x0000555555997b73 in PyEval_EvalCodeEx (closure=0x0, kwdefs=0x0, defcount=0x0, defs=0x0, kwcount=0x0, kws=0x0, argcount=0x0, args=0x0, locals=locals@entry=0x7ffff355a9d8, globals=globals@entry=0x0,
          _co=_co@entry=0x7ffff36088a0) at Python/ceval.c:4187
      #16 PyEval_EvalCode (co=co@entry=0x7ffff36088a0, globals=globals@entry=0x7ffff7e5a318, locals=locals@entry=0x7ffff7e5a318) at Python/ceval.c:731
      #17 0x00005555556b5b3b in run_mod (arena=0x7ffff7e75150, flags=<optimized out>, locals=0x7ffff7e5a318, globals=0x7ffff7e5a318, filename=0x7ffff358d270, mod=0x62500001e300) at Python/pythonrun.c:1025
      #18 PyRun_FileExFlags (fp=<optimized out>, filename_str=<optimized out>, start=<optimized out>, globals=<optimized out>, locals=<optimized out>, closeit=<optimized out>, flags=<optimized out>)
          at Python/pythonrun.c:978
      #19 0x00005555556b5fdc in PyRun_SimpleFileExFlags (fp=<optimized out>,
          filename=0x7ffff35c2680 "\314\070\064\302\227\a\254\bJf\331u\230N\273\022\355@\200\352\024`z[\267&\257+\022Q\324\017\310\nSyF2+\001{\327\354\355\245\275\002\064d-\235x\\\327O\230٧\036ތF\222\326\336\060\027q\220\037\217\b\364#=\366\224,\362\355\224i4h\030.c\377\225\360.׀M\033\066\251\ve'M=\261\t\365\307\016\267\203Q\316\313n\251]+\351H\222\244\266{\224FG\257\022\340\071\233r\300\220\065\031\236][\266\v\027\071#\354Ɣ\310\\\243M\243\251\250\372_\362^Φ\306ڝ\222\365\062O1nY\224pĥ\243IV\364\070\356\232\\\222z\242\321\v\027|\342\027\325\325O֬\300\252a0\250"..., closeit=0x1, flags=<optimized out>)
          at Python/pythonrun.c:419
      #20 0x00005555556f2704 in run_file (p_cf=0x7fffffffe2b0, filename=0x604000000010 L"crash.py", fp=0x616000034880) at Modules/main.c:340
      #21 Py_Main (argc=<optimized out>, argv=<optimized out>) at Modules/main.c:810
      #22 0x000055555569a293 in main (argc=argc@entry=0x2, argv=argv@entry=0x7fffffffe528) at ./Programs/python.c:69
      #23 0x00007ffff6086b97 in __libc_start_main (main=0x55555569a050 <main>, argc=0x2, argv=0x7fffffffe528, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffe518)
          at ../csu/libc-start.c:310
      #24 0x000055555569bb2a in _start ()
      

      x.py

      import hashlib
      hashlib.shake_128().hexdigest(-10)
      
    2. ned-deily commented on Oct 7, 2018

      @ned-deily
      Member

      See also bpo-33729. We need this addressed for 3.6.7.

    3. tirkarthi commented on Oct 7, 2018

      @tirkarthi
      Member

      Thanks for the report. Interesting, this is not reproducible on master and latest 3.7 branches though both have different errors but reproducible in latest 3.6 and v3.7.0 . As Ned noted this seems to have been fixed with bpo-33729 but still there is no decision on reverting/keeping the commits made with the linked issue.

      # master

      ./python.exe
      Python 3.8.0a0 (heads/master:7dfbd49671, Oct  7 2018, 16:00:31)
      [Clang 7.0.2 (clang-700.1.81)] on darwin
      Type "help", "copyright", "credits" or "license" for more information.
      >>> import hashlib
      >>> hashlib.shake_128().hexdigest(-10)
      Traceback (most recent call last):
        File "<stdin>", line 1, in <module>
      ValueError: value must be positive

      # upstream/3.7

      ./python.exe
      Python 3.7.1rc1+ (remotes/upstream/3.7:3b699932e5, Oct  7 2018, 21:44:03)
      [Clang 7.0.2 (clang-700.1.81)] on darwin
      Type "help", "copyright", "credits" or "license" for more information.
      >>> import hashlib
      >>> hashlib.shake_128().hexdigest(-10)
      Traceback (most recent call last):
        File "<stdin>", line 1, in <module>
      OverflowError: can't convert negative value to unsigned int

      # 3.7.0 segfaults

      ./python.exe
      Python 3.7.0 (tags/v3.7.0:1bf9cc5093, Oct  7 2018, 21:51:43)
      [Clang 7.0.2 (clang-700.1.81)] on darwin
      Type "help", "copyright", "credits" or "license" for more information.
      >>> import hashlib
      >>> hashlib.shake_128().hexdigest(-10)
      [1]    67585 bus error  ./python.exe

      # upstream/3.6 segfaults

      ./python.exe
      Python 3.6.7rc1+ (remotes/upstream/3.6:177254c96f, Oct  7 2018, 21:42:19)
      [GCC 4.2.1 Compatible Apple LLVM 7.0.2 (clang-700.1.81)] on darwin
      Type "help", "copyright", "credits" or "license" for more information.
      >>> import hashlib
      >>> hashlib.shake_128().hexdigest(-10)
      [1]    49096 bus error  ./python.exe

      Thanks

    4. tirkarthi commented on Oct 7, 2018

      @tirkarthi
      Member

      Sorry Ned, my comment seems to have changed the priority while submitting the comment. I would also propose adding the attached report as a unit test.

    5. ned-deily commented on Oct 7, 2018

      @ned-deily
      Member

      No problem; that's something to watch out for when you get an update conflict message from the bug tracker! Regarding this issue, I believe Serhiy is going to do a PR but perhaps you can work with him on providing the test case.

    6. added
      type-crashA hard crash of the interpreter, possibly with a core dump
      and removed on Oct 7, 2018
    7. 2 remaining items

    8. ned-deily commented on Oct 10, 2018

      @ned-deily
      Member

      We've reached the cutoff point for 3.7.1rc2 and 3.6.7rc2 and I don't see a PR or a resolution of this for either branch yet. If there's a chance for merged PRs in the next couple of hours, I'll wait a bit longer but otherwise these fixes will have to wait.

    9. serhiy-storchaka commented on Oct 11, 2018

      @serhiy-storchaka
      Member

      New changeset 9b8c2e7 by Serhiy Storchaka in branch 'master':
      bpo-34922: Fix integer overflow in the digest() and hexdigest() methods (GH-9751)
      9b8c2e7

    10. serhiy-storchaka commented on Oct 11, 2018

      @serhiy-storchaka
      Member

      New changeset 8b040e5 by Serhiy Storchaka in branch '3.7':
      [3.7] bpo-34922: Fix integer overflow in the digest() and hexdigest() methods (GH-9751) (GH-9798)
      8b040e5

    11. serhiy-storchaka commented on Oct 11, 2018

      @serhiy-storchaka
      Member

      New changeset 69e6ad6 by Serhiy Storchaka (Miss Islington (bot)) in branch '3.6':
      [3.6] bpo-34922: Fix integer overflow in the digest() and hexdigest() methods (GH-9751) (GH-9798) (GH-9801)
      69e6ad6

    12. terryjreedy commented on Oct 12, 2018

      @terryjreedy
      Member

      Should this be closed as fixed?

    13. serhiy-storchaka commented on Oct 12, 2018

      @serhiy-storchaka
      Member

      Since it is tagged as a release blocker, I think that only Ned can close it.

      Personally I don't think that this issue is a security issue. digest() and hexdigest() argument usually is a constant. It is unlikely that the crash can be triggered by user data.

    14. httpsgithubcomxcainiao commented on Oct 13, 2018

      httpsgithubcomxcainiaomannequin
      MannequinAuthor

      oh brother, maybe this worth open a cve.

    15. ned-deily commented on Oct 13, 2018

      @ned-deily
      Member

      Serhiy's fixes (thanks!) are now released in 3.7.0rc2 and 3.6.7rc2 so I'm removing the "release blocker" status. If there is nothing more to be done for this issue, can we close it?

      shuoz:

      oh brother, maybe this worth open a cve.

      Note that Serhiy believes that this is not a security issue since it is unlikely that the crash can be triggered by user data. Anyone can cause segfaults or do damage if they have unrestricted access to a Python interpreter; that's a threat model for any language that allows sometime like Python's os.system or subprocess. A better question is can a user of an application written in Python likely cause a DOS or create a privilege escalation. Is that the case here?

    16. ned-deily commented on Jan 14, 2020

      @ned-deily
      Member

      Since there has been no further discussion on this since the fixes were pushed over a year ago, I am declaring this issue resolved. Thanks for everyone's help!

    17. removed their assignment
      on Jan 14, 2020
    18. transferred this issue fromon Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      3.7 (EOL)end of life3.8 (EOL)end of lifeextension-modulesC modules in the Modules dirtype-crashA hard crash of the interpreter, possibly with a core dump

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions