Skip to content

CIFuzz build failures - _freeze_module segfaults during the build, fuzz testing doesn't start. #116886

Closed
Closed
CIFuzz build failures - _freeze_module segfaults during the build, fuzz testing doesn't start.#116886
Labels
type-bugAn unexpected behavior, bug, or error

Description

@gpshead
gpshead
opened on Mar 16, 2024

PR authors and core devs have been seeing CIFuzz failures on PRs not related to their changes of late. These are non-blocking, but confusing.

The CIFuzz build step is failing, it seems that _freeze_module crashes:

./Programs/_freeze_module importlib._bootstrap_external ./Lib/importlib/_bootstrap_external.py Python/frozen_modules/importlib._bootstrap_external.h
Segmentation fault (core dumped)

I believe _freeze_module got compiled with memory-sanitizer but no useful MSAN report seems to be generated by that crash.

Unclear where the problem lies, maybe this could be CIFuzz container or clang related?

CIFuzz background: read PR #107653 (discussion and links) and Issue #107652 (the pitch).

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Linked PRs

Activity

  1. gpshead
    added
    type-bugAn unexpected behavior, bug, or error
    on Mar 16, 2024

  2. gpshead commented on Mar 16, 2024

    @gpshead
    gpshead
    on Mar 16, 2024
    MemberAuthor

    @illia-v

  3. freakboy3742 commented on Mar 16, 2024

    @freakboy3742
    freakboy3742
    on Mar 16, 2024
    Contributor

    Looking at CI history, the last PR merged to main that I can find that passed that test was #116761; the first to fail it was #116528. However, the CIFuzz test isn't run on every PR, and PRs aren't necessarily up to date at time of merge, so the trigger for this failure isn't necessarily in that specific merge range - but I'm guessing it will be close.

  4. freakboy3742 commented on Mar 16, 2024

    @freakboy3742
    freakboy3742
    on Mar 16, 2024
    Contributor

    Another thought - the issue could be with the CI action itself. The CIFuzz job uses the master branch of the actions, so CPython's CI results are going to be dependent on the stability of the upstream repo.

    As an experiment, I modified the CIFuzz checkout on an older version of the action, and ran it on a copy of #116454.

    If I revert to the 10e6cbd7 commit to oss-fuzz from a couple of days ago, the CIFuzz test fails because of a clang issue failing to compile the code, which is at least a different failure, and demonstrates that there's at least some sensitivity to the oss-fuzz version.

  5. vstinner commented on Mar 16, 2024

    @vstinner
    vstinner
    on Mar 16, 2024
    Member

    When I tried to reproduce in an Ubuntu 22.04 container, I get these errors:

    root@ubuntu-dev:/cpython# make -j4 SHELL="bash -x"
(...)
./Programs/_freeze_module getpath ./Modules/getpath.py Python/frozen_modules/getpath.h
==11165==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x55d06e6084c0 in read_text /cpython/Programs/_freeze_module.c:109:9
    #1 0x55d06e6084c0 in main /cpython/Programs/_freeze_module.c:225:24
    #2 0x7f1097f42d8f  (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f) (BuildId: c289da5071a3399de893d2af81d6a30c62646e1e)
    #3 0x7f1097f42e3f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x29e3f) (BuildId: c289da5071a3399de893d2af81d6a30c62646e1e)
    #4 0x55d06e580f34 in _start (/cpython/Programs/_freeze_module+0x85f34) (BuildId: f91c6a29210dc0cd457ac4d39f134316a9608c08)

  Uninitialized value was created by an allocation of 'stat.i' in the stack frame of function 'main'
    #0 0x55d06e606f60 in main /cpython/Programs/_freeze_module.c:206

SUMMARY: MemorySanitizer: use-of-uninitialized-value /cpython/Programs/_freeze_module.c:109:9 in read_text
==11165==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x55d06e6084cd in read_text /cpython/Programs/_freeze_module.c:116:21
    #1 0x55d06e6084cd in main /cpython/Programs/_freeze_module.c:225:24
    #2 0x7f1097f42d8f  (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f) (BuildId: c289da5071a3399de893d2af81d6a30c62646e1e)
    #3 0x7f1097f42e3f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x29e3f) (BuildId: c289da5071a3399de893d2af81d6a30c62646e1e)
    #4 0x55d06e580f34 in _start (/cpython/Programs/_freeze_module+0x85f34) (BuildId: f91c6a29210dc0cd457ac4d39f134316a9608c08)

  Uninitialized value was created by an allocation of 'stat.i' in the stack frame of function 'main'
    #0 0x55d06e606f60 in main /cpython/Programs/_freeze_module.c:206

SUMMARY: MemorySanitizer: use-of-uninitialized-value /cpython/Programs/_freeze_module.c:116:21 in read_text
MemorySanitizer: 2 warnings reported.
make: *** [Makefile:1640: Python/frozen_modules/getpath.h] Error 1
make: *** Waiting for unfinished jobs....

    I can reproduce the issue by running _freeze_module manually:

    # ./Programs/_freeze_module zipimport ./Lib/zipimport.py Python/frozen_modules/zipimport.h
==11199==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x5622e273f4c0 in read_text /cpython/Programs/_freeze_module.c:109:9
    #1 0x5622e273f4c0 in main /cpython/Programs/_freeze_module.c:225:24
    #2 0x7fa50d770d8f  (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f) (BuildId: c289da5071a3399de893d2af81d6a30c62646e1e)
    #3 0x7fa50d770e3f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x29e3f) (BuildId: c289da5071a3399de893d2af81d6a30c62646e1e)
    #4 0x5622e26b7f34 in _start (/cpython/Programs/_freeze_module+0x85f34) (BuildId: f91c6a29210dc0cd457ac4d39f134316a9608c08)

  Uninitialized value was created by an allocation of 'stat.i' in the stack frame of function 'main'
    #0 0x5622e273df60 in main /cpython/Programs/_freeze_module.c:206

SUMMARY: MemorySanitizer: use-of-uninitialized-value /cpython/Programs/_freeze_module.c:109:9 in read_text
==11199==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x5622e273f4cd in read_text /cpython/Programs/_freeze_module.c:116:21
    #1 0x5622e273f4cd in main /cpython/Programs/_freeze_module.c:225:24
    #2 0x7fa50d770d8f  (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f) (BuildId: c289da5071a3399de893d2af81d6a30c62646e1e)
    #3 0x7fa50d770e3f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x29e3f) (BuildId: c289da5071a3399de893d2af81d6a30c62646e1e)
    #4 0x5622e26b7f34 in _start (/cpython/Programs/_freeze_module+0x85f34) (BuildId: f91c6a29210dc0cd457ac4d39f134316a9608c08)

  Uninitialized value was created by an allocation of 'stat.i' in the stack frame of function 'main'
    #0 0x5622e273df60 in main /cpython/Programs/_freeze_module.c:206

SUMMARY: MemorySanitizer: use-of-uninitialized-value /cpython/Programs/_freeze_module.c:116:21 in read_text
MemorySanitizer: 2 warnings reported.

    Commands used to build Python in a fresh Ubuntu 22.04 container:

    apt update && apt install --yes git make clang libssl-dev libreadline-dev libc6-dev
git clone https://github.com/python/cpython --depth=1 
cd cpython/
export ASAN_OPTIONS=detect_leaks=0
./configure --with-memory-sanitizer --without-ensurepip 'CFLAGS=-mllvm -msan-keep-going=1' --prefix /opt/python CC=clang LD=clang
make -j4 SHELL="bash -x"

    Then just to reproduce the issue:

    ./Programs/_freeze_module zipimport ./Lib/zipimport.py Python/frozen_modules/zipimport.h

  6. vstinner commented on Mar 16, 2024

    @vstinner
    vstinner
    on Mar 16, 2024
    Member

    Oh wait, stat.i is an old issue: #91043

  7. vstinner commented on Mar 16, 2024

    @vstinner
    vstinner
    on Mar 16, 2024
    Member

    The stat.i issue can be worked around by:

    # git diff
diff --git a/Python/fileutils.c b/Python/fileutils.c
index 882d329957..b1e273d810 100644
--- a/Python/fileutils.c
+++ b/Python/fileutils.c
@@ -1235,6 +1235,7 @@ _Py_stat_basic_info_to_stat(FILE_STAT_BASIC_INFORMATION *info,
 int
 _Py_fstat_noraise(int fd, struct _Py_stat_struct *status)
 {
+    memset(status, 0, sizeof(*status));
 #ifdef MS_WINDOWS
     BY_HANDLE_FILE_INFORMATION info;
     FILE_BASIC_INFO basicInfo;

  8. illia-v commented on Mar 17, 2024

    @illia-v
    illia-v
    on Mar 17, 2024
    Contributor

    I ran CIFuzz for a commit from 3 weeks ago in my fork and it failed whereas the job was succeeding this time ago for PRs continuously.

    According to https://oss-fuzz-build-logs.storage.googleapis.com/index.html#cpython3, Google haven't recently had any problems building CPython with the memory sanitizer for OSS-Fuzz.

    So, I believe the issue is not with CPython but with the environment created by the action.

  9. vstinner commented on Mar 18, 2024

    @vstinner
    vstinner
    on Mar 18, 2024
    Member

    It's annoying that pull request have a red "X" failure sign when "CIFuzz (memory)" is the only failing job, and it's a known failure. Can we disable the job or change GitHub configuration to not mark a whole PR as failed, when a CIFuzz job fails?

  10. ambv
    added a commit that references this issue on Mar 19, 2024

    pythongh-116886: Temporarily disable CIfuzz (memory)

    47b2dcf
  11. hugovk
    added a commit that references this issue on Mar 19, 2024

    gh-116886: Temporarily disable CIfuzz (memory) (#117018)

    1ab0d0b
  12. vstinner
    added a commit that references this issue on Mar 20, 2024

    pythongh-116886: Temporarily disable CIfuzz (memory) (python#117018)

    682ea4e
  13. adorilson
    added a commit that references this issue on Mar 25, 2024

    pythongh-116886: Temporarily disable CIfuzz (memory) (python#117018)

    89d140c
  14. illia-v
    added a commit that references this issue on Mar 27, 2024

    Revert "pythongh-116886: Temporarily disable CIfuzz (memory) (python#…

    77498f3

  15. hugovk commented on Mar 27, 2024

    @hugovk
    hugovk
    on Mar 27, 2024
    Member

    Should be fixed now: #117289, thanks for following up @illia-v.

  16. gpshead
    added a commit that references this issue on Mar 28, 2024

    Revert "gh-116886: Temporarily disable CIfuzz (memory) (GH-117018)" (G…

    2cedd25

  17. gpshead commented on Mar 28, 2024

    @gpshead
    gpshead
    on Mar 28, 2024
    MemberAuthor

    Reopen this if anyone sees failures. The revert was merged based on Github reporting the infrastructure/container-definition issue having been fixed to avoid the problem.

  18. gpshead
    closed this as completedon Mar 28, 2024
  19. NGRsoftlab
    added 2 commits that reference this issue on Apr 2, 2024

    Squashed commit of the following:

    1b5fe4c

    Squashed commit of the following:

    a016a54
  20. diegorusso
    added 2 commits that reference this issue on Apr 17, 2024

    pythongh-116886: Temporarily disable CIfuzz (memory) (python#117018)

    3980637

    Revert "pythongh-116886: Temporarily disable CIfuzz (memory) (pythonG…

    f38f531
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    type-bugAn unexpected behavior, bug, or error

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions