Can't initialize heap: [0x000001e7] Attempt to access invalid address random segfault on Windown x86 #9650

Activity

1. mvorisek

   added

   Bug

   Status: Needs Triage

   on Sep 30, 2022
2. mvorisek commented on Oct 1, 2022

   

   mvorisek

   on Oct 1, 2022

   Last edited by mvorisek

   ContributorAuthor

   More actions

   related to #8643 and https://bugs.php.net/bug.php?id=77194 (confirmed in https://bugs.php.net/bug.php?id=77194#1548144531)

   To reproduce, please run couple hundreds of full tests iterations on Windows /w x86 build locally and you should be able to reproduce it, in CI, it is reproducible after a few pushes /w random test failures.
3. mvorisek

   changed the title [-]`Zend/tests/function_arguments/argument_count_incorrect_internal_strict.phpt` results in segfault on Win 32 x86[/-] [+]`Can't initialize heap: [0x000001e7] Attempt to access invalid addresst` random segfault on Win 32 x86[/+] on Oct 1, 2022
4. mvorisek

   changed the title [-]`Can't initialize heap: [0x000001e7] Attempt to access invalid addresst` random segfault on Win 32 x86[/-] [+]`Can't initialize heap: [0x000001e7] Attempt to access invalid address` random segfault on Win 32 x86[/+] on Oct 1, 2022
5. mvorisek

   changed the title [-]`Can't initialize heap: [0x000001e7] Attempt to access invalid address` random segfault on Win 32 x86[/-] [+]`Can't initialize heap: [0x000001e7] Attempt to access invalid address` random segfault on Windown x86[/+] on Oct 1, 2022
6. TysonAndre commented on Oct 11, 2022

   

   TysonAndre

   on Oct 11, 2022

   Last edited by TysonAndre

   Contributor

   More actions

   I've seen similar spurious issues in Phan's for a while, using https://ci.appveyor.com/project/TysonAndre/phan/branch/v5/job/1f74k3nsurgxpdgb when php is running in paratest

   • I guess I can switch away from x86 to work around this, it's just a coincidence that the appveyor file I based Phan's appveyor setup on was using x86, and x64 is far, far more common

   .phan/plugins\RedundantAssignmentPlugin.php:1 PhanNativePHPSyntaxCheckPlugin means that https://github.com/phan/phan/blob/v5/.phan/plugins/InvokePHPNativeSyntaxCheckPlugin.php was running php.exe --syntax-check on the file RedundantAssignmentPlugin

   • This is essentially making multiple calls to php --syntax-check, up to max_incomplete_processes concurrently - knowing that the issue can also be reproduced with just --syntax-check may be of use in trying to diagnose or reproduce the issue

     (though php is also doing a lot of other things)
     (This is also using the x86 32-bit build for reasons I forget.)
     (https://github.com/phan/phan/blob/v5/.appveyor.yml - I believe it doesn't use opcache, but I guess the tests of php-src)

   • Many failures in the build history are related to InvokePHPNativeSyntaxCheckPlugin, probably because it calls php.exe extremely often, so it has the largest chance of encountering those failures

   Can't initialize heap: [0x000001e7] Attempt to access invalid address
   ░░░░░  54 / 619 (  8%) 409MB
   ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 108 / 619 ( 17%) 426MB
   ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 162 / 619 ( 26%) 436MB
   ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 216 / 619 ( 35%) 449MB
   ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 270 / 619 ( 43%) 456MB
   ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ [324](https://ci.appveyor.com/project/TysonAndre/phan/branch/v5/job/1f74k3nsurgxpdgb#L324) / 619 ( 52%) 458MB
   ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 378 / 619 ( 61%) 461MB
   ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 432 / 619 ( 69%) 462MB
   ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 486 / 619 ( 78%) 463MB
   ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 540 / 619 ( 87%) 466MB
   ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 594 / 619 ( 96%) 470MB
   ░░░░░░░░░░░░░░░░░░░░░░░░░                              619 / 619 (100%) 471MB
   .phan/plugins\RedundantAssignmentPlugin.php:1 PhanNativePHPSyntaxCheckPlugin Saw error or notice for php --syntax-check: "No output was detected. Is 'c:\\\\projects\\\\php\\\\php.exe' a relative or absolute path to an executable PHP binary?"
   
7. TysonAndre commented on Oct 11, 2022

   

   TysonAndre

   on Oct 11, 2022

   Contributor

   More actions

   It looks like initializing the memory manager ("Can't initialize heap", zend_mm_init) is failing. It's attempting to allocate a 2MB chunk (ZEND_MM_CHUNK_SIZE) but failing.

   If php was compiled for windows with ZEND_MM_ERROR it looks like it would output the error message with the failure reason for VirtualAlloc

   • Guesses: Not enough memory in appveyor (really unlikely if it's failing on the 2MB)? Or is the heap somehow fragmented?
   • Maybe the fact that hundreds of php processes have been started up has caused available memory to be fragmented

   https://learn.microsoft.com/en-us/windows/win32/memory/large-page-support This sounds somewhat related, but I'm unfamiliar with win32 programming

   When writing applications that use large-page memory, keep the following considerations in mind:

   • Large-page memory regions may be difficult to obtain after the system has been running for a long time because the physical space for each large page must be contiguous, but the memory may have become fragmented. Allocating large pages under these conditions can significantly affect system performance. Therefore, applications should avoid making repeated large-page allocations and instead allocate all large pages one time, at startup.

   // Zend/zend_alloc.c
   static zend_mm_heap *zend_mm_init(void)
   {
   	zend_mm_chunk *chunk = (zend_mm_chunk*)zend_mm_chunk_alloc_int(ZEND_MM_CHUNK_SIZE, ZEND_MM_CHUNK_SIZE);
   	zend_mm_heap *heap;
   
   	if (UNEXPECTED(chunk == NULL)) {
   #if ZEND_MM_ERROR
   #ifdef _WIN32
   		stderr_last_error("Can't initialize heap");
   #else
   		fprintf(stderr, "\nCan't initialize heap: [%d] %s\n", errno, strerror(errno));
   #endif
   #endif
   		return NULL;
   	}

   // Zend/zend_alloc.c
   static void *zend_mm_mmap(size_t size)
   {
   #ifdef _WIN32
   	void *ptr = VirtualAlloc(NULL, size, MEM_COMMIT | MEM_RESERVE, PAGE_READWRITE);
   
   	if (ptr == NULL) {
   #if ZEND_MM_ERROR
   		stderr_last_error("VirtualAlloc() failed");
   #endif
   		return NULL;
   	}
   	return ptr;
   #else
8. TysonAndre commented on Oct 11, 2022

   

   TysonAndre

   on Oct 11, 2022

   Contributor

   More actions

   If the issue is just in VirtualAlloc and is actually related to starting and stopping thousands of php processes in a short time period, causing virtual memory to be fragmented (not actually sure), then a possible workaround for applications that aren't php-src to in x86(32-bit builds) is to SET USE_ZEND_ALLOC=0 before running php to use malloc/free instead of the Zend memory manager, which will perform regular size malloc allocations rather than allocating blocks of size 2 MB.
9. TysonAndre

   added a commit that references this issue on Oct 11, 2022

   Switch to testing 64-bit Windows builds in appveyor

   ...

   9d7f592
10. mvorisek commented on Oct 11, 2022

    

    mvorisek

    on Oct 11, 2022

    ContributorAuthor

    More actions

    @TysonAndre thank you very much for looking into it!

    Guesses: Not enough memory in appveyor (really unlikely if it's failing on the 2MB)? Or is the heap somehow fragmented?

    I would expect consequent tests then to fail. Currently the failure is more or less random and next test is (p>99%) passing.

    The fragmentation can be closely related, as in 32-bit address space it is much easier to create some collision than in 64-bit space. Maybe there is some bug also in 64-bit builds, but they are only much more rare.

    I did some more reseach:

    https://forum.exetools.com/showthread.php?t=8963 - are we sure we do not increment the address over the alloved range?

    VirtualAlloc(NULL, ... is maybe fine, but VirtualAlloc(addr, ... can create collisions easily (in 64-bit address space however less likely)

    "mremp Windows" - https://stackoverflow.com/questions/17197615/no-mremap-for-windows#38502927 can this help?

    to test is memory remap is a problem, is there any easy way to compile php without zend_mm_mmap_fixed completely?

    or can we reserve (but not commit) like 1GB address space for php mremp on php start at least?

    based on https://stackoverflow.com/questions/31672069/is-the-mem-reserve-flag-of-virtualalloc-really-useful-today and https://stackoverflow.com/questions/21368429/error-code-487-error-invalid-address-when-using-virtualallocex we should always reserve a memory

    more links (for later, if the above does not help):

    https://www.youtube.com/watch?v=MPnsPlDJbhI - can someone with Windows check?

    using VirtualQuery https://forums.codeguru.com/showthread.php?258460-ERROR_INVALID_ADDRESS-with-VirtualAllocEx-and-Win2K it should be possible to check the problem before even VirtualAlloc call

    https://stackoverflow.com/questions/55119066/virtualalloc-failed

    golang/go#35465

    https://bugzilla.mozilla.org/show_bug.cgi?id=1662560
11. mvorisek commented on Oct 11, 2022

    

    mvorisek

    on Oct 11, 2022

    ContributorAuthor

    More actions

    I belive I found the problem: https://github.com/php/php-src/pull/9721/files#r992229663
12. cmb69

    added 2 commits that reference this issue on Nov 17, 2022

    Fix GH-9650: Can't initialize heap: [0x000001e7]

    ...

    This issue will close once commit 8d65c2f is merged into the 'master' branch.

    8d65c2f

    Merge branch 'PHP-8.1' into PHP-8.2

    ...

    This issue will close once commit c6204ac is merged into the 'master' branch.

    c6204ac
13. cmb69

    closed this as completedin 8d65c2fon Nov 17, 2022
14. cmb69

    added a commit that references this issue on Nov 17, 2022

    Merge branch 'PHP-8.2'

    ...

    This issue will close once commit 2cf03b6 is merged into the 'master' branch.

    2cf03b6