Repository navigation
crash on mipsel64 from v12.x #31118
Description
- Version: 12.14.0
- Platform: Linux localhost.localdomain 3.10.0 deps: update openssl to 1.0.1j #1 SMP PREEMPT Mon Sep 16 21:11:02 CST mips64 mips64 mips64 GNU/Linux
- Subsystem: gcc-4.9.3-3 gcc-7.3.0
I just build nodejs@12.14.0 on mips64el (loongson 3A3000) with --with-snapshot ，the result was successed. When I run nodejs, it crashed. The messages are as follows:
#
# Fatal error in ../deps/v8/src/execution/isolate.cc, line 232
# Embedded blob checksum verification failed. This indicates that the embedded blob has been modified since compilation time. A common cause is a debugging breakpoint set within builtin code.
#
#
#
#FailureMessage Object: 0xffffffacc0
Program received signal SIGTRAP, Trace/breakpoint trap.
v8::base::OS::Abort () at ../deps/v8/src/base/platform/platform-posix.cc:406
But nodejs@v10.x and nodejs@v11.x works fine. I found that when I build nodejs@v12.x with --without-snapshot，then nodejs works fine. Maybe mips64el has some problems with v8 snapshot.
The versions of v8 and nodejs are :
- nodejs@12.14.0 -> v8@7.7.299.13
- nodejs@11.15.1 -> v8@7.0.276.38
- nodejs@10.17.0 -> v8@6.8.275.32
According to the nodejs building document, have to build nodejs with gcc>=6.3，but I build it with gcc 4.9.3-3 without errors. after crash,I use gcc@7.3.0 to build nodejs,the result are same.
I also tried to build nodejs with crossing compiler gcc@4.9.4 and gcc@7.3.1, nodejs was crashed too.You can download them from
http://www.loongnix.org/index.php/Cross-compile , and direct download url gcc@4.9.4 and gcc@7.3.1 ,put the date directory to /usr/loca/
the crossing build script as follows(host: ubuntu 18.04 x86_64 and gcc@7.4.0):
#!/bin/bash
export PREFIX=/usr/local/mips-loongson-gcc4.9-linux-gnu/bin/mips-linux-gnu-
export CC=${PREFIX}"gcc -march=gs464e -mips64r2 -mabi=64"
export CXX=${PREFIX}"c++ -march=gs464e -mips64r2 -mabi=64"
export LINK=$CXX
export LD=${PREFIX}ld
export AR=${PREFIX}ar
export AS=${PREFIX}as
export RANLIB=${PREFIX}ranlib
export CROSS_COMPILE=mips-loongson
export ARCH=mips64el
# Native compilers
export AR_host="ar"
export CC_host="gcc"
export CXX_host="g++"
export LINK_host="g++"
export AR_HOST="ar"
export CC_HOST="gcc"
export CXX_HOST="g++"
export LINK_HOST="g++"
# extras for convenience.
export OBJD=${PREFIX}objdump
export GDB=${PREFIX}gdb
export RDE=${PREFIX}readelf
./configure --dest-cpu=mips64el --cross-compiling --with-mips-arch-variant=r2 --dest-os=linux --openssl-no-asm --verbose
make -j$(grep -c ^processor /proc/cpuinfo 2>/dev/null || 1)Maybe nodejs crashed in v8 builtin code,But when I use ./configure xxxx and --gdb, build failed on src\deps\v8\src\diagnostics\gdb-jit.cc@629 #error Unsupported target architecture.
void WriteHeader(Writer* w) {
DCHECK_EQ(w->position(), 0);
Writer::Slot<ELFHeader> header = w->CreateSlotHere<ELFHeader>();
#if (V8_TARGET_ARCH_IA32 || V8_TARGET_ARCH_ARM)
const uint8_t ident[16] = {0x7F, 'E', 'L', 'F', 1, 1, 1, 0,
0, 0, 0, 0, 0, 0, 0, 0};
#elif V8_TARGET_ARCH_X64 && V8_TARGET_ARCH_64_BIT || \
V8_TARGET_ARCH_PPC64 && V8_TARGET_LITTLE_ENDIAN
const uint8_t ident[16] = {0x7F, 'E', 'L', 'F', 2, 1, 1, 0,
0, 0, 0, 0, 0, 0, 0, 0};
#elif V8_TARGET_ARCH_PPC64 && V8_TARGET_BIG_ENDIAN && V8_OS_LINUX
const uint8_t ident[16] = {0x7F, 'E', 'L', 'F', 2, 2, 1, 0,
0, 0, 0, 0, 0, 0, 0, 0};
#elif V8_TARGET_ARCH_S390X
const uint8_t ident[16] = {0x7F, 'E', 'L', 'F', 2, 2, 1, 3,
0, 0, 0, 0, 0, 0, 0, 0};
#elif V8_TARGET_ARCH_S390
const uint8_t ident[16] = {0x7F, 'E', 'L', 'F', 1, 2, 1, 3,
0, 0, 0, 0, 0, 0, 0, 0};
#else
#error Unsupported target architecture. <--
#endif
Hi @bnoordhuis:
Thank you for help. Yes, mips(el) is not an officially supported
architecture, but less than nothing.
Activity
bnoordhuis commentedon Dec 29, 2019
That "Embedded blob checksum verification failed" error is a debug check, it's not enabled in release builds, so I'm kind of surprised you're getting that message. Where did you obtain the source code from?
whhif commentedon Dec 30, 2019
When nodejs crashed, I build nodejs with
--debugflag ,so I get the error message. But I can not built it with-gdbflag.bnoordhuis commentedon Jan 2, 2020
--gdbis for debugging jitted code (the machine code V8 emits when it compiles a JS method), you probably don't need that but if you do, you can probably make it work with minor tweaks to gdb-jit.cc.You mention release builds crash? What does the backtrace look like in gdb?
FlyGoat commentedon Feb 26, 2020
So I just dig the issue deeper. It crashed at node_mksnapshot.
With GDB, it told me
With no backtrace frame.
Disassembly at that point seems strange. It should be padding or something, not code.
So I tried to trace RA register, and got the real call routine.
Looks like it jumped into wrong address. Any further hint on debugging is appreciated.
Thanks.
@bnoordhuis
FlyGoat commentedon Feb 26, 2020
It appears to be a GCC only regression.
Clang build works fine, but there are some test failures with OpenSSL, like:
Will do some investigation.
bnoordhuis commentedon Feb 26, 2020
#31118 (comment) looks like a C++ -> JS function call gone wrong but I can't really tell more from the backtrace.
It looks like this function call:
node/src/api/environment.cc
Lines 462 to 471 in db125c5
cc @joyeecheung - that code seems to have undergone quite some changes recently so perhaps this is a known and already fixed issue?
@FlyGoat You might want to try out the HEAD of the v12.x-staging branch, see if that works better.
FlyGoat commentedon Feb 26, 2020
@bnoordhuis still happens on v12.x-staging with GCC.
But I can't understand why clang just works fine.
joyeecheung commentedon Feb 26, 2020
Not that I know of, but I can't tell from the stack trace which part of the snapshot building was particularly relevant - I believe the call on the stack frames was just the first time V8 in our builds would ever try to call from C++ to JS through the API? It might help to just create a regular v8 context, compile a simple JS function and call into it in
node::SnapshotBuilder::Generate()right after the registration of isolates to figure out if it's just general C++ -> JS call failure:node/tools/snapshot/snapshot_builder.cc
Line 71 in 24a4f76
See this test on how to do this:
node/deps/v8/test/cctest/test-compiler.cc
Line 418 in 6ca81ad
joyeecheung commentedon Feb 26, 2020
And, you can also try building with the configure option
--without-node-snapshotand see if the node binary fails when it startsFlyGoat commentedon Feb 27, 2020
Thanks for your suggestions, node binary without snapshot crashed at the same point.
It looks like a v8 issue. I'm going to run some v8 tests.
xen0n commentedon Mar 8, 2020
Bisected down to commit f579e11, v8 broke between 7.3.492.25 and 7.4.288.13. Need to investigate deeper.
xwafish commentedon Apr 14, 2020
on mips platform, we always build v8 use llvm before, so we do not notice this issue.
I will do some investigation about this.
xwafish commentedon Apr 15, 2020
This should be a bug of GCC assembler on mips.
v8 snapshot use file "embeded.S" as a input file, if v8 do not insert source location information into this file(".loc"), everything goes right, all code are writed as .octa xxxxx , but if v8 insert ".loc" to "embeded.S", the code will be writed by ".octa xxx" along with ".byte xxxx" see https://github.com/nodejs/node/blob/v12.x/deps/v8/src/snapshot/embedded/embedded-file-writer.cc#L165
looks like gcc assembler do not handle this situation successfully, but clang can handle successfully, I have report this to our compiler team.
do not insert source information to this file can workaround for this issue temporary, patch is attached.
xen0n commentedon Apr 15, 2020
It's very nice of you to finish the investigation! We'll see how to best fix things upstream.
FlyGoat commentedon Apr 15, 2020
Filled an v8 upstream issue
As it's currently restricted to Googlers, I'll paste content below.
FlyGoat commentedon Apr 15, 2020
@bnoordhuis
As now we've addressed the issue, after the workaround, NodeJS managed to pass most of the tests on mips64el, is it possible to push mips64el into an experimental or Tier 2 level supported architecture?
@xen0n and I can provide help with general MIPS issues, @xwafish is maintaining MIPS v8 upstream, and @wzssyqa from Debian can help with toolchain & system environment issues.
We can also provide mips64el Cl machine hosted in China.
Thanks.
bnoordhuis commentedon Apr 16, 2020
@FlyGoat If that machine can be set up in a way where it's managed by our Build WG in order that they can run Jenkins etc. on it, then promoting mips64el to experimental shouldn't be a problem.
Tier 2 status means test failures block releases but the MIPS user base isn't large enough to warrant that.
FlyGoat commentedon Apr 16, 2020
@bnoordhuis Who should I get in touch for that?
Should I open a new issue?
Thanks.
bnoordhuis commentedon Apr 16, 2020
@FlyGoat Can you open an issue over at https://github.com/nodejs/build/issues explaining you want to donate a machine, what specs it has, etc.? The build people will take it from there.
(Technically, I'm one of the build people but I'm no expert on how to provision machines.)
FlyGoat commentedon Apr 17, 2020
Fixed in upstream.
https://chromium-review.googlesource.com/c/v8/v8/+/2153187
bnoordhuis commentedon Apr 23, 2020
@FlyGoat You can open a back-port of the bug fix if you want. The process is outlined in https://github.com/nodejs/node/blob/master/doc/guides/maintaining-V8.md, specifically the "Backporting to Abandoned Branches" section.
whhif commentedon Apr 27, 2020
The v8 patch is working on v12.x. Thank you all. @FlyGoat should open a back-port of the bug.
kapouer commentedon May 31, 2020
about that ssl test failure: the tests are supposed to be run using openssl.cnf distributed with nodejs,
which doesn't happen if built against shared openssl, in which case one has to set:
OPENSSL_CONF=./deps/openssl/openssl/apps/openssl.cnf make test-js