Repository navigation
Segmentation fault on AArch64 release build with opcache.jit=1112 #12596
Description
Description
This appears on AArch64 build with phpdbg disabled.
Build steps
$ bash buildconf
$ bash configure --disable-phpdbg
$ make -j 50
INI
opcache.enable_cli=1
opcache.jit=1112
opcache.jit_buffer_size=16M
The following code:
<?php
class test {
function __construct() {
if (empty($this->test[0][0])) { print "test1";}
if (!isset($this->test[0][0])) { print "test2";}
}
}
$test1 = new test();
?>Resulted in this output:
test1test2Segmentation fault (core dumped)
But I expected this output instead:
test1test2
GDB
(gdb) r
Starting program: /mnt/local/php-src/sapi/cli/php -c /mnt/local/www index.php
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
test1test2
Program received signal SIGSEGV, Segmentation fault.
0x0000aaaaaaeea798 in execute_ex (ex=0x0) at /home/penli01/php-src/Zend/zend_vm_execute.h:57064
57064 EG(current_execute_data) = EX(prev_execute_data);
(gdb) bt
#0 0x0000aaaaaaeea798 in execute_ex (ex=0x0) at /home/penli01/php-src/Zend/zend_vm_execute.h:57064
#1 0x0000aaaaaaef5d18 in zend_execute (op_array=0xfffff5681800, return_value=<optimized out>)
at /home/penli01/php-src/Zend/zend_vm_execute.h:61594
#2 0x0000aaaaaae71cf4 in zend_execute_scripts (type=type@entry=8, retval=retval@entry=0x0, file_count=file_count@entry=3)
at /home/penli01/php-src/Zend/zend.c:1881
#3 0x0000aaaaaae06060 in php_execute_script (primary_file=0xfffff7a87200 <__GI__IO_doallocbuf+108>, primary_file@entry=0xfffffffdb300)
at /home/penli01/php-src/main/main.c:2492
#4 0x0000aaaaaaf69570 in do_cli (argc=4, argv=0xaaaaab97bc60) at /home/penli01/php-src/sapi/cli/php_cli.c:966
#5 0x0000aaaaaabbc9bc in main (argc=4, argv=<optimized out>) at /home/penli01/php-src/sapi/cli/php_cli.c:1340
(gdb)PHP Version
PHP master @ 95c8ad2
Operating System
Ubuntu 22.04
Activity
dstogov commentedon Nov 2, 2023
I cannot reproduce this.
phpdbgshouldn't affect JIT directly. (I hope you didn't forgetmake install).Could you please configure PHP with
--with-capstone, then run the script with-d opcache.jit_debug=0x507and provide the output.pfustc commentedon Nov 2, 2023
Yes, I just checked that the issue also exists with
phpdbgenabled. But the SegFault disappears with--enable-debugwhen configuring the build. Have you tried without--enable-debug?Another guess: Have you run the test on real AArch64 hardware, or just QEMU? Perhaps there's some difference.
I can also reproduce this with
make test. A lot of unit tests fail on AArch64 withopcache.jit=1112.Sure, please see below. Another clue from GDB is that SegFault is caused by
(ex=0x0) at Zend/zend_vm_execute.h:57064, not in the JITed code.dstogov commentedon Nov 2, 2023
Yes. I run with
--enable-debugQEMU. I don't have a suitable AArch64 box.
I'll try the release build...
dstogov commentedon Nov 2, 2023
I can't reproduce the crash. My assembler code look identical, so I suspect, this may be related to memory model differences in QEMU and real CPU.
Can you try to debug this down?
The crash is most probably occur because
x27register reserved forexecute_datasomehow got invalid value.I recommend to run PHP under GDB with
-d opcache.jit_debug=0x100(this will expose JIT symbols to GDB), set breakpoint atzend_runtime_jit, continue after the first break, finish after the second, then execute si few times (while get into JIT-edtest::__consructor) code, thenrecordandcontinue. After the crash you may execute in reverse to understand wherex27got a wrong value, or what else went wrong.[-]Segmentation fault on AArch64 with phpdbg disabled and opcache.jit=1112[/-][+]Segmentation fault on AArch64 release build with opcache.jit=1112[/+]pfustc commentedon Nov 3, 2023
Thanks for your guide. I just followed your steps and got the entry of the JIT-ed code.
Unfortunately, GDB
recordfailed withrecord: failed to record execution log.. I turned to usewatchbut still didn't seex27changed anywhere until the segmentation fault appeared.From GDB I see
x27is still valid at the point of the SegFault. The instruction causes the SegFault looks likewhere
x0holds a zero. What does the previousldr x0, [x19, #1392]access? I see the memory around[x19, #1392]are all zeros.Please see below GDB output.
dstogov commentedon Nov 3, 2023
Thank you for the debugging. My assumption about
x27was wrong.It looks like
execute_ex()usesx19to store some temporary address (probably address ofexecute_globalstructure) and the JIT code clobbers this register.The inline asm
__asm__ __volatile__ (""::: "x19","x20","x21","x22","x23","x24","x25","x26");should prevent using persistent registers across calls to JIT code, but it seems it doesn't. It probably, should be moved to a better place(s). E.g. directly before JIT code call.I probably won't be able to dive deeper before Noveber 7.
pfustc commentedon Nov 8, 2023
Hi @dstogov,
I tracked all change points of
x19but don't see any JIT-ed code. All change points are in the Zend runtime, php-cli or system calls.But I find two mistakes in AArch64 register definition.
php-src/ext/opcache/jit/ir/ir_aarch64.h
Line 31 in 66e95d9
php-src/ext/opcache/jit/ir/ir_aarch64.h
Line 65 in 66e95d9
Unfortunately, after fixing these locally, the segmentation fault issue still exists. Could you try running
with AArch64 release build to reproduce the failures in your QEMU environment?
dstogov commentedon Nov 8, 2023
Thanks for the fixes and tests. I'll try to debug this issue, but it's at the end of my TODO list.
pfustc commentedon Nov 8, 2023
I haven't created any PR for the fix. Would you like to push a commit to fix them directly?
dstogov commentedon Nov 8, 2023
I already committed your fix.
3 remaining items
dstogov commentedon Nov 9, 2023
Thanks for the debugging.
Right. This is not a GCC bug. The different GCC version just makes the bug visible.
It would be great to reproduce it with my GCC version.
Yeah.
x19appear in the disassembler you already posted a week ago.I think the same.
In case I can reproduce the bug I will able to fix it.
In worst case, I may provide a patch and you will test it.
It's clear. Debug build uses
-O0and this prevents compiler using aggressive register allocation.Do you mean opcache.jit=xxx2?
Probably this optimization level leads to calling ZEND_RETURN VM handler that causes failure in case of register clobbering. Less aggressive JIT don't use persistent registers, more aggressive JIT inlines ZEND_RETURN handler, but this is just a guess. I think the bug is general, and may lead to failures with other JIT levels and on different platforms as well. So it would be great to fix it.
dstogov commentedon Nov 28, 2023
I made some more research and found that the difference in generated code may be caused by
-fpicor-fpieflags and common sub-expression elimination foradrp x19, _GLOBAL_OFFSET_TABLE_. Unfortunately, I wasn't able to reproduce the CSE effect.I hope you are still able to reproduce the failure with old GCC.
Could you please check if #12813 fixes this.
pfustc commentedon Nov 29, 2023
I just did some tests with #12813. I don't think the issue is fixed.
The segmentation fault in my previous case does not appear now. But in other test cases, I still see segmentation faults at the same point caused by
x19clobbered by JIT-ed code. Running below command with release build can reproduce all the issues.Below is another case that still causes segmentation fault with the same JIT option.
What do you think of making
x19tox26not allocatable again? The old JIT implementation does so by defining a register set ofZEND_REGSET_PRESERVED. And AArch64 still has enough numbers of allocatable GP registers even without them.dstogov commentedon Nov 29, 2023
This is not a solution. Only x19-x26 may be used across function calls without spilling.
pfustc commentedon Nov 30, 2023
Perhaps I don't quite understand your intention. What do you mean by "across function calls without spilling"?
AFAIK, if we conform to the C calling convention, we should save and restore x19-x26 if they are clobbered in a compiled function. I just tried CALL VM and found it works exactly in this way. The CALL VM generates
stpandldpsequences inir_emit_prologueandir_emit_epiloguerespectively, so it doesn't have this SegFault issue. But in HYBRID VM, we don't setctx.fixed_save_regsetinzend_jit_init_ctx()hence these callee-saved registers are NOT actually protected by the JIT-ed code.If I understand correctly, the goal of
__asm__ __volatile__()is to protect these registers, but it doesn't. This inline asm can save the clobbered registers before the asm code restore them afterwards. However, the real JIT-ed code is not at this place. So there is no code restoring clobbered registers after the JIT-ed code. Your patch #12813 moves the inline asm into the big HYBRIDwhileloop. But there is still no register restoring if the JIT-ed code directly branches to another handler.Perhaps it's not a good solution to protect the callee-saves by adding
__asm__ __volatile__()anywhere. Why not save and restore these registers in the prologue and epilogue just like the CALL VM does?dstogov commentedon Nov 30, 2023
In case we perform a CALL in JIT code and some value is used before and after the CALL then it can't be keept in scratch registers. Only in x19-x26 or in memory. This will reduce quality of generated code.
HYBRID VM executes JIT code in context of VM execite_ex(). It doesn't create additional stack frame.
I see.
It should be another solution to prevent CC from allocating registers across JIT transitions.
Somehow a register is used only to keep
_GLOBAL_OFFSET_TABLE_and this is done only by old GCC, however symbols may be linked statically (using ADRP).This will increase the cost of each JIT<->VM transition.
pfustc commentedon Dec 1, 2023
Thank you so much for your explanation. I have learned a lot from your reply.
An attempt to fix phpGH-12596: Segmentation fault on AArch64 release …
dstogov commentedon Dec 5, 2023
@pfustc I understood the difference between GCC 11.4.0 from Ubuntu 22.04 and my GCC. GCC 11.4.0 was configured with
--enable-default-pieand as result it silently adds-fpieflag that leads to PIC code to access external variables.You may check if your GCC 12.0.3 produces PIC code by
echo | cc -dM -E - | grep -i picAnyway, I updated #12813 and now it works for me even with GCC 11.4.0
pfustc commentedon Dec 5, 2023
I have run unit tests with HYBRID VM and all JIT option combinations on AArch64. It looks good.
Fixed phpGH-12596: Segmentation fault on AArch64 release build with o…