Repository navigation
Description
Description
Code using openssl_encrpyt() causes segmentation faults under high loads with static ZTS builds of PHP.
For instance, this part of Laravel triggers the bug: https://github.com/laravel/framework/blob/10.x/src/Illuminate/Encryption/Encrypter.php#L102-L105
Here is a backtrace gathered with GDB:
Backtrace
#1 0x0000783330654ec3 in __libc_free (p=0x7832da600cb0) at src/malloc/mallocng/free.c:105
#2 0x000078332fef89f0 in EVP_CIPHER_CTX_reset ()
#3 0x000078332f15bf83 in php_openssl_encrypt (data=0x7832d67946a8 "cdcfc94a44f1372ecc088f6e0abca9d4c1416db0|XFaOVGRiKdwkZpamJ3fb2fXpv4vvguazIfVNxK8R", data_len=81, method=0x7832d7995220 "aes-256-cbc", method_len=11,
password=0x7832d67acd88 "\036\216/a\363-\257e\332Wu㖂&\033%\341\307\036O\243hOX(\257\204\346j\334\t", password_len=32, options=0, iv=0x7832d6686558 "\\\226\024Y\204:#b\276+1b\332\034%\225", iv_len=16, tag=0x7832d7818a40, tag_len=16,
aad=0x783330d21ef5 "", aad_len=0) at /go/src/app/dist/static-php-cli/source/php-src/ext/openssl/openssl.c:7700
#4 0x000078332f15c126 in zif_openssl_encrypt (execute_data=0x7832d78189a0, return_value=0x7832d7818970) at /go/src/app/dist/static-php-cli/source/php-src/ext/openssl/openssl.c:7719
#5 0x000078332ef7351f in ZEND_DO_ICALL_SPEC_RETVAL_USED_HANDLER () at /go/src/app/dist/static-php-cli/source/php-src/Zend/zend_vm_execute.h:1337
#6 execute_ex (ex=0x7832d78171d0) at /go/src/app/dist/static-php-cli/source/php-src/Zend/zend_vm_execute.h:57216
#7 0x000078332edcfec2 in zend_call_function (fci=0x7832dafe9720, fci_cache=0x7832dafe96f0) at /go/src/app/dist/static-php-cli/source/php-src/Zend/zend_execute_API.c:957
#8 0x000078332eca9591 in zif_frankenphp_handle_request (execute_data=<optimized out>, return_value=0x7832d7817140) at frankenphp.c:371
#9 0x000078332ef7351f in ZEND_DO_ICALL_SPEC_RETVAL_USED_HANDLER () at /go/src/app/dist/static-php-cli/source/php-src/Zend/zend_vm_execute.h:1337
#10 execute_ex (ex=0x7832d7817020) at /go/src/app/dist/static-php-cli/source/php-src/Zend/zend_vm_execute.h:57216
#11 0x000078332ef908a7 in zend_execute (op_array=0x7832d7866000, return_value=0x0) at /go/src/app/dist/static-php-cli/source/php-src/Zend/zend_vm_execute.h:61604
#12 0x000078332edfca5c in zend_execute_scripts (type=8, retval=0x0, file_count=3) at /go/src/app/dist/static-php-cli/source/php-src/Zend/zend.c:1881
#13 0x000078332ed3d05a in php_execute_script (primary_file=0x7832dafef660) at /go/src/app/dist/static-php-cli/source/php-src/main/main.c:2523
#14 0x000078332ecaa4f1 in frankenphp_execute_script (file_name=0x7832d946ab00 "4b48bc1af71891ab467ffde5c30d1b724928ab8c9dee168d597ce7f9e1dc9498") at frankenphp.c:825
#15 0x000078332eca7c37 in _cgo_d49d0ae4a108_Cfunc_frankenphp_execute_script (v=0xc000810dd8) at /tmp/go-build/cgo-gcc-prolog:55
#16 0x000078332d966a01 in runtime.asmcgocall () at /usr/local/go/src/runtime/asm_amd64.s:918
#17 0x000000c0008016c0 in ?? ()
#18 0x00007832dafef7d0 in ?? ()
#19 0x00007832dafef834 in ?? ()
#20 0x0000000000000000 in ?? ()
Several Laravel users have confirmed the issue at https://github.com/laravel/octane/issues/791.
A reproducer is available in the linked PR.
Using a static build of FrankenPHP with debug symbols like the one available here https://github.com/dunglas/frankenphp/actions/runs/8206010369?pr=635 (frankenphp-linux-x86_64-debug) always triggers the error on Linux/amd64. I'm not able to reproduce on Apple Silicon (both on macOS and Linux).
PHP Version
PHP 8.3.3
Operating System
Ubuntu 22.04.4 LTS
Activity
dunglas commentedon Mar 11, 2024
Some extra information: FrankenPHP static builds are created using @crazywhalecc's static-php-cli, which uses Musl under the hood.
bukka commentedon Mar 11, 2024
does it build it statically with OpenSSL as well or is that linked as shared lib? What OpenSSL version is used?
dunglas commentedon Mar 11, 2024
I've done a bit of digging, I still don't know exactly what the underlying problem is, but here are my guesses so far:
EVP_CIPHER_CTX_reset()(php-src/ext/openssl/openssl.c
Line 7736 in 8595bea
EVP_CipherInit_ex()(php-src/ext/openssl/openssl.c
Line 7553 in 8595bea
php-src/ext/openssl/openssl.c
Line 7596 in 8595bea
php-src/ext/openssl/openssl.c
Line 7686 in 8595bea
EVP_CIPHER_CTX_reset().Under heavy loads (this crash is only triggered during load tests),
EVP_CipherInit_ex()probably fails for one reason or another.In the documentation, it is not clear whether it is safe to call
EVP_CIPHER_CTX_reset()ifEVP_CipherInit_ex()has not been called. It seems that the crash only occurs when using Musl's mallocng allocator, which may do some extra safety checks compared to glibc (not sure about that).Another unrelated thing:
EVP_CipherInit_ex()is called two times inphp_openssl_cipher_init(), shouldn't it be reset before being initialized again?Take all of this with a grain of salt as I know nothing about OpenSSL and this code is a bit complicated.
dunglas commentedon Mar 11, 2024
@bukka OpenSSL is compiled statically as well, the latest version if used.
bukka commentedon Mar 11, 2024
Calling
EVP_CipherInit_extwice is necessary to set some things (like AEAD tag and key length) so this should be fine. There might be something unusual in combination latest OpenSSL with musl but need to debug it to see.devnexen commentedon Mar 11, 2024
It might be irrelevant but how the toolchain has been built might count here too, see topic here.
dunglas commentedon Mar 11, 2024
@devnexen This build uses the default options for static-php-cli: https://github.com/dunglas/frankenphp/blob/main/build-static.sh#L140
And
-static-piefor the final FrankenPHP binary: https://github.com/dunglas/frankenphp/blob/main/build-static.sh#L140C107-L140C118bukka commentedon Mar 13, 2024
So I have been looking through this properly. It took me longest to actually create the build with OpenSSL debug symbols and recreate the issue. It was recreatable only on my old Intel laptop (couldn't re-create on my more powerful AMD desktop).
The backtrace with all symbols looked like this
It says that it is failing on de-allocation fetched cipher
type_namehere: https://github.com/openssl/openssl/blob/openssl-3.2.1/crypto/evp/evp_enc.c#L1735The cipher context is however reset after freeing which can be seen here: https://github.com/openssl/openssl/blob/openssl-3.2.1/crypto/evp/evp_enc.c#L46-L48 so there doesn't seem any way to keep the freed context cipher around. I have been also checking if there is a way to maybe just partially initialize the context in
evp_cipher_init_internalbut can't see anything. If this was possible somehow, then it would be OpenSSL bug as it would prevent deallocation of the context.The actual cipher type is set here: https://github.com/openssl/openssl/blob/openssl-3.2.1/crypto/evp/evp_enc.c#L1589 . The
ossl_algorithm_get1_first_name uses
OPENSSL_strndupwhich uses malloc to allocate the type. There does not seem anything unusual.The PHP code does not seem to have any issue either except the unnecessary EVP_CIPHER_CTX_reset in https://github.com/php/php-src/blob/php-8.3.3/ext/openssl/openssl.c#L7700-L7701 . Removing that won't change much though as internally
EVP_CIPHER_CTX_freecallsEVP_CIPHER_CTX_resetso the crash would still be there. We need to free the context as soon as it is created usingEVP_CIPHER_CTX_newso there is no way around that. So any issue keeping that inconsistent would be an OpenSSL issue.Another possibility is a problem in allocator. I would suggest to try to replace allocator as using MUSL one is not ideal anyway for highly multi-threaded application. It might be worth to put some effort into using jemalloc or simillar and see if it fixes the issue.
dunglas commentedon Mar 16, 2024
I created a build of FrankenPHP that uses https://github.com/microsoft/mimalloc instead of mallogng (php/frankenphp#666), and there is a similar problem (but not exactly the same).
This works, but under high load there are segmentation faults. Here is the stack trace:
0xdfdfdfdfdfdfdfdfis interesting because it seems to be used as a magic number by some allocators to detect "use after free" issues (not sure if it's the case of mimalloc).dunglas commentedon May 30, 2024
A user of FrankenPHP reported a crash that is a bit different but may be related (php/frankenphp#676 (comment)).
ThePooN commentedon Jun 4, 2024
I think I'm running into this as well but with a different stack trace, if that helps:
Running a FrankenPHP v1.1.5 amd64 static build with added
dsextension (following these debug instructions) on https://github.com/ppy/osu-webOf note, I cannot reproduce using
wrk -c1 -t1orab -c 1. Falls over easily as soon as I add concurrency.xepozz commentedon May 14, 2025
Any news?
bukka commentedon May 14, 2025
I got #18516 which might potentially resolve this but needs a bit more work and testing to see if it's really the solution...
bukka commentedon May 22, 2025
The #18516 is ready for review so if anyone wants to test it, that would be cool. I will be testing FrankenPHP with this changes later next week. I need to first see if I can still recreate it as I updated my laptop but if I do, I should be able to verify if that change fixes the problems.
andypost commentedon Jun 5, 2025
probably resolved via 2f5ef4d
henderkes commentedon May 15, 2026
@dunglas found this one still open, cannot reproduce the crash with 8.4/8.5 anymore while 8.2.15 crashes. So this can be closed and we can remove the known incompatibility.