Repository navigation
Segfault in php-fpm 8.1 on Linux #8731
Description
Description
Hello,
Since migrating a big(ish) app on PHP 8.1, we get repeated segfaults, a few times per day (no more than 25 segfaults per day per server, for a few million hits per day per server). Here is a sample stack trace:
#0 0x00007f3ed8800556 in zend_accel_class_hash_copy (source=0x46736bb0, source=0x46736bb0, target=0x55ffcca9c330) at ./Zend/zend_types.h:1173
1173 return p->refcount;
(gdb) bt
#0 0x00007f3ed8800556 in zend_accel_class_hash_copy (source=0x46736bb0, source=0x46736bb0, target=0x55ffcca9c330) at ./Zend/zend_types.h:1173
#1 zend_accel_load_script (persistent_script=persistent_script@entry=0x46736a80, from_shared_memory=from_shared_memory@entry=1)
at ./ext/opcache/zend_accelerator_util_funcs.c:230
#2 0x00007f3ed87eed26 in persistent_compile_file (type=<optimized out>, file_handle=<optimized out>) at ./ext/opcache/ZendAccelerator.c:2240
#3 persistent_compile_file (file_handle=<optimized out>, type=<optimized out>) at ./ext/opcache/ZendAccelerator.c:1951
#4 0x000055ffcc3edef8 in compile_filename (type=type@entry=2, filename=filename@entry=0x7f3ed210d820) at ./Zend/zend_language_scanner.c:707
#5 0x000055ffcc45ae40 in zend_include_or_eval (inc_filename_zv=<optimized out>, type=2) at ./Zend/zend_execute.c:4617
#6 0x000055ffcc4693ba in ZEND_INCLUDE_OR_EVAL_SPEC_CV_HANDLER () at ./Zend/zend_vm_execute.h:38713
#7 0x000055ffcc48f357 in execute_ex (ex=0x46736ce0) at ./Zend/zend_vm_execute.h:59122
#8 0x000055ffcc41b46e in zend_call_function (fci=<optimized out>, fci_cache=<optimized out>) at ./Zend/zend_execute_API.c:908
#9 0x000055ffcc41b785 in zend_call_known_function (fn=0x7f3ed8c05cd8, object=<optimized out>, called_scope=<optimized out>, retval_ptr=retval_ptr@entry=0x0,
param_count=param_count@entry=1, params=params@entry=0x7fff83b4f4e0, named_params=0x0) at ./Zend/zend_execute_API.c:997
#10 0x000055ffcc31b66a in spl_perform_autoload (class_name=0x40e9f288, lc_name=0x40e9f2c8) at ./ext/spl/php_spl.c:433
#11 0x000055ffcc41a6b6 in zend_lookup_class_ex (name=name@entry=0x40e9f288, key=0x40e9f2c8, flags=flags@entry=512) at ./Zend/zend_execute_API.c:1141
#12 0x000055ffcc41ba70 in zend_fetch_class_by_name (class_name=0x40e9f288, key=<optimized out>, fetch_type=fetch_type@entry=512)
at ./Zend/zend_execute_API.c:1601
#13 0x000055ffcc466dd7 in ZEND_INIT_STATIC_METHOD_CALL_SPEC_CONST_CONST_HANDLER () at ./Zend/zend_vm_execute.h:6778
#14 0x000055ffcc48fac0 in execute_ex (ex=0x46736ce0) at ./Zend/zend_vm_execute.h:56351
#15 0x000055ffcc41b46e in zend_call_function (fci=fci@entry=0x7fff83b4f770, fci_cache=<optimized out>, fci_cache@entry=0x7f3ed21b60d0)
at ./Zend/zend_execute_API.c:908
#16 0x00007f3ed8754a0d in curl_write_header (data=data@entry=0x55ffccb729e0 "\r\n", size=size@entry=1, nmemb=nmemb@entry=2, ctx=0x7f3ed8cc2a80)
at ./ext/curl/interface.c:1596
#17 0x00007f3ed86c572c in chop_write (conn=0x55ffcce9ea80, conn=0x55ffcce9ea80, olen=2, optr=0x55ffccb729e0 "\r\n", type=2) at sendf.c:623
#18 Curl_client_write (conn=0x55ffcce9ea80, type=2, ptr=0x55ffccb729e0 "\r\n", len=len@entry=2) at sendf.c:679
#19 0x00007f3ed86c3a72 in Curl_http_readwrite_headers (data=data@entry=0x55ffcce92f90, conn=conn@entry=0x55ffcce9ea80, nread=nread@entry=0x7fff83b4f960,
stop_reading=stop_reading@entry=0x7fff83b4f95f) at http.c:3405
#20 0x00007f3ed86d7778 in readwrite_data (comeback=0x7fff83b4f9eb, done=0x7fff83b4f9ea, didwhat=<synthetic pointer>, k=0x55ffcce93088, conn=0x55ffcce9ea80,
data=0x55ffcce92f90) at transfer.c:677
#21 Curl_readwrite (conn=0x55ffcce9ea80, data=data@entry=0x55ffcce92f90, done=done@entry=0x7fff83b4f9ea, comeback=comeback@entry=0x7fff83b4f9eb)
at transfer.c:1294
#22 0x00007f3ed86e1933 in multi_runsingle (multi=multi@entry=0x55ffcce98a20, now=..., data=data@entry=0x55ffcce92f90) at multi.c:1933
#23 0x00007f3ed86e2bd9 in curl_multi_perform (multi=multi@entry=0x55ffcce98a20, running_handles=running_handles@entry=0x7fff83b4fb64) at multi.c:2218
#24 0x00007f3ed86d9742 in easy_transfer (multi=0x55ffcce98a20) at easy.c:686
#25 easy_perform (events=false, data=0x55ffcce92f90) at easy.c:780
#26 curl_easy_perform (data=0x55ffcce92f90) at easy.c:799
#27 0x00007f3ed8755b24 in zif_curl_exec (execute_data=0x7f3ed8c16750, return_value=0x7fff83b4fcc0) at ./ext/curl/interface.c:3063
#28 0x000055ffcc490eba in ZEND_DO_ICALL_SPEC_RETVAL_UNUSED_HANDLER () at ./Zend/zend_vm_execute.h:1235
#29 execute_ex (ex=0x46736ce0) at ./Zend/zend_vm_execute.h:55752
#30 0x000055ffcc49773c in zend_execute (op_array=0x7f3ed8c02000, return_value=0x0) at ./Zend/zend_vm_execute.h:60123
#31 0x000055ffcc429875 in zend_execute_scripts (type=type@entry=8, retval=retval@entry=0x0, file_count=file_count@entry=3) at ./Zend/zend.c:1792
#32 0x000055ffcc3c601f in php_execute_script (primary_file=<optimized out>) at ./main/main.c:2538
#33 0x000055ffcc26f335 in main (argc=<optimized out>, argv=<optimized out>) at ./sapi/fpm/fpm/fpm_main.c:1914
I suspect the errors are opcache-related; here is our opcache configuration:
[opcache]
opcache.memory_consumption=512
opcache.max_accelerated_files=1000000
opcache.revalidate_freq=0
opcache.revalidate_path=1
opcache.huge_code_pages=0
opcache.file_update_protection=30
opcache.jit=disable
We are running Debian 10 and PHP 8.1.6, up-to-date from packages.sury.org.
PHP Version
PHP 8.1.6 (cli) (built: May 17 2022 16:49:19) (NTS)
Operating System
Debian 10
Activity
iluuu1994 commentedon Jun 9, 2022
Hi @nono-gdv! Unfortunately it's very hard to understand this problem without more context. Assuming the code is closed source and can't be shared, you could try the following:
zbacktracein gdb to get a PHP backtraceframe 1and thenp (char *)persistent_script->script.filename->valto get the name of the compiled scriptMaybe you can then share the given file if the problem can be isolated.
nono-gdv commentedon Jun 9, 2022
Here you are.
The file is from Guzzle, this version: https://github.com/guzzle/guzzle/blob/7.4.3/src/Handler/HeaderProcessor.php
This is not consistent, however; segfaults sometimes happen in other files, even though Guzzle is a usual offender.
We are in the process of downgrading to PHP 8.0.19; it is a little early to tell for sure, but it looks like the segfaults are gone.
nono-gdv commentedon Jun 9, 2022
I can confirm the segfaults are gone. Since last logrotate (about 5am, so 12h ago, give or take), we had 317 segfault with PHP 8.1, 300 of them in opcache.so. We downgraded to PHP 8.0 two hours ago, not a single segfault since.
xPaw commentedon Jun 24, 2022
I had the same crash yesterday:
It seems to be related to autoload, as OP's trace also has
spl_perform_autoload.The zbacktrace basically boils down to the
requirecall afterfile_existsinside ofspl_autoload_registercallback. The string (path) looks okay to me in the trace.brainexe commentedon Jul 21, 2022
we also have the same random segfault with 8.1.7...also quite rare but overall having many failing processed per day. PHP8.0 downgrade is no option and we tried already different opcache config changes (optimizatzion level, disable jit) without success.
jirkace commentedon Jul 22, 2022
@brainexe what about 8.1.8, did you try it? Did you try
opcache.jit=1205?brainexe commentedon Jul 22, 2022
jit is disabled and php8.1.8 will be used next week on prod, then we will see 🤞
joelvh commentedon Jul 28, 2022
@brainexe any luck with PHP 8.1.8?
brainexe commentedon Aug 1, 2022
yes, still failing on production with php8.1.8, sadly no core dumps on this production maschines.
wow-apps commentedon Apr 19, 2023
does anyone tried to set:
opcache.huge_code_pages=0?tsg1992 commentedon Jun 23, 2023
On board with same issue. We didn't mind a few SIGSEGV 11 here and there until it rendered a whole server pretty much down. Every request to a specific app kept crashing PHP-FPM threads producing 502 Bad Gateway results. The only remediation was restarting PHP-FPM and it's now back to the usual but rare segfaults. JIT is enabled.
Platform:
Ubuntu 22.04.2 LTS GNU/Linux 5.15.0-75-genericPHP/FPM:
PHP 8.1.2-1ubuntu2.11,OPcache v8.1.2-1ubuntu2.11opcache.blacklist_filename = no value
opcache.consistency_checks = 0
opcache.dups_fix = Off
opcache.enable = On
opcache.enable_cli = Off
opcache.enable_file_override = Off
opcache.error_log = no value
opcache.file_cache = no value
opcache.file_cache_consistency_checks = On
opcache.file_cache_only = Off
opcache.file_update_protection = 2
opcache.force_restart_timeout = 180
opcache.huge_code_pages = Off
opcache.interned_strings_buffer = 16
opcache.jit = 1235
opcache.jit_bisect_limit = 0
opcache.jit_blacklist_root_trace = 16
opcache.jit_blacklist_side_trace = 8
opcache.jit_buffer_size = 500000000
opcache.jit_debug = 0
opcache.jit_hot_func = 127
opcache.jit_hot_loop = 64
opcache.jit_hot_return = 8
opcache.jit_hot_side_exit = 8
opcache.jit_max_exit_counters = 8192
opcache.jit_max_loop_unrolls = 8
opcache.jit_max_polymorphic_calls = 2
opcache.jit_max_recursive_calls = 2
opcache.jit_max_recursive_returns = 2
opcache.jit_max_root_traces = 1024
opcache.jit_max_side_traces = 128
opcache.jit_prof_threshold = 0.005
opcache.lockfile_path = /tmp
opcache.log_verbosity_level = 1
opcache.max_accelerated_files = 10000
opcache.max_file_size = 0
opcache.max_wasted_percentage = 10
opcache.memory_consumption = 256
opcache.opt_debug_level = 0
opcache.optimization_level = 0x7FFEBFFF
opcache.preferred_memory_model = no value
opcache.preload = no value
opcache.preload_user = no value
opcache.protect_memory = Off
opcache.record_warnings = Off
opcache.restrict_api = no value
opcache.revalidate_freq = 0
opcache.revalidate_path = Off
opcache.save_comments = On
opcache.use_cwd = On
opcache.validate_permission = Off
opcache.validate_root = Off
opcache.validate_timestamps = On
bukka commentedon Jul 9, 2023
@tsg1992 Can you try to disable jit and see if it's still crashing?
maxsky commentedon Jun 24, 2024
This problem from remirepo install version.
I tried if use this command install
yum --enablerepo=remi-php81 install, it will be crash.But now I install though compile, its running normal! I spend more than 20 hours to solve this problem. T_T...
I had a same problem, if disable
opcacheall Apps work normal, this is my config:I'm also tried disable jit or set jit to
1205, it stillexited on signal 11 (SIGSEGV)and response502.Here is core file contents through gdb: