[BUG] rimage segfault in build for TGL with xt-xcc

Describe the bug
The build stops due to segmentation fault in rimage. The error is:

Scanning dependencies of target run_rimage
Segmentation fault (core dumped)
make[3]: *** [src/arch/xtensa/CMakeFiles/run_rimage.dir/build.make:57: src/arch/xtensa/CMakeFiles/run_rimage] Error 139
make[2]: *** [CMakeFiles/Makefile2:2152: src/arch/xtensa/CMakeFiles/run_rimage.dir/all] Error 2
make[1]: *** [CMakeFiles/Makefile2:2102: src/arch/xtensa/CMakeFiles/bin.dir/rule] Error 2
make: *** [Makefile:697: bin] Error 2

To Reproduce

rm -rf build
mkdir -p build
cd build
cmake -DTOOLCHAIN=xt -DROOT_DIR=$XTENSA_ELF -DMEU_PATH=$MEU_PATH -DMEU_PRIVATE_KEY=$PRIVATE_KEY_PATH ..
make tigerlake_defconfig

Reproduction Rate
Always.

Expected behavior
Produce sof-tgl.ri and sof-tgl.ldc.

Impact
Showstopper.

Environment

Branch name and commit hash of the 2 repositories: sof (firmware/topology) and linux (kernel driver).
- Kernel: N/A
- SOF: fa974eb
- Rimage: 7317f2a
Name of the topology file
- Topology: N/A
Name of the platform(s) on which the bug is observed.
- Platform: TGL, build computer is Ubuntu 20.04.1 LTS, Xplorer RG-2017.8-linux, X6H3CNL_2017_8

Screenshots or console output
sof_tgl_xtxcc_build.txt
rimage_valgrind.txt
sof_tgl_gdb_run.txt
rimage_ldd.txt

Note: Based on another PR it looks the configuration should be cavs2x_LX6HiFi3_2017_8. I'll get it and re-test.

The build works with the new configuration. I'll change to P3 and no milestone. I didn't close because the tool should not crash even if obsolete configuration has been used.

Based on another PR it looks the configuration should be cavs2x_LX6HiFi3_2017_8.

Pretty sure you meant #3187

I didn't close because the tool should not crash even if obsolete configuration has been used.

You bet. I'm raising to P2 because it's hard crash that could happen in other conditions until we know better.

Adding the "rimage" keyword in the title and inlining the valgrind trace because I think it's the most useful piece. It will also make it searchable:

==385981== Memcheck, a memory error detector
==385981== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==385981== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info
==385981== Command: ../../../rimage_ep/build/rimage -o sof-tgl.ri -m tgl -s 1344 -k /home/singalsu/work/current/sof/sof/keys/otc_private_key.pem -i 3 -f 1.5 -b 1 -x 24 bootloader-tgl sof-tgl
==385981== 
==385981== Invalid write of size 1
==385981==    at 0x4842B63: memmove (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)


==385981==  Address 0x6737fa62 is not stack'd, malloc'd or (recently) free'd
==385981== 
==385981== 
==385981== Process terminating with default action of signal 11 (SIGSEGV)
==385981==  Access not within mapped region at address 0x6737FA62
==385981==    at 0x4842B63: memmove (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==385981==    by 0x4BD15DA: _IO_file_xsgetn (fileops.c:1296)
==385981==    by 0x4BC5062: fread (iofread.c:38)
==385981==    by 0x10D012: man_copy_sram (manifest.c:193)
==385981==    by 0x10D16D: man_copy_elf_section (manifest.c:225)
==385981==    by 0x10DB22: man_module_create (manifest.c:463)
==385981==    by 0x10E354: man_create_modules (manifest.c:678)
==385981==    by 0x10F270: man_write_fw_meu_v2_5 (manifest.c:1153)
==385981==    by 0x111E2D: main (rimage.c:195)
==385981==  If you believe this happened as a result of a stack
==385981==  overflow in your program's main thread (unlikely but
==385981==  possible), you can try to increase the size of the
==385981==  main thread stack using the --main-stacksize= flag.
==385981==  The main thread stack size used in this run was 8388608.

@singalsu

@singalsu have you tried my PR #3187 . You need meu for signing TGL

@singalsu

cmake -DTOOLCHAIN=xt -DROOT_DIR=$XTENSA_ELF -DMEU_PATH=$MEU_PATH -DMEU_PRIVATE_KEY=$PRIVATE_KEY_PATH ..

@singalsu I'm confused: AFAIK the build uses either MEU or rimage but never both in the same build.

No segfault after using the proper configuration. Lower the priority to P3 and remove from 1.6 blockers.

@marc-hb

@marc-hb I can keep this as very low priority activity since I reported this. I just checked that the segfault still happens with today's rimage version. Also I don't understand why cmake runs rimage in this case if meu should handle the non-OTC key based builds.

manifest.c: assert buffer overflow in man_copy_sram()

@singalsu

Thanks @singalsu for sending me the binaries produced with xcc. rimage crashes because the .module section of the bootloader-tgl file you sent me is either garbage or incorrectly parsed by rimage as garbage. Are you sure the -x 24 offset argument passed to the rimage command line is correct?

More specifically, manifest.c:man_get_module_manifest(bootloader-tgl) finds sof_mod.text_size to be 1650553970 (1.6G) which does not seem reasonable and causes the crash later. Other "interesting" values found by the same function in the same sof_mod:

sof_mod.module.segment[2].v_base_addr = 1650553965
sof_mod.module.segment[2].file_offset = 1953705472

I submitted one-line thesofproject/rimage#28 that converts the memory corruption and crash into an assert, could you please test it?

When dropping the bootloader-tgl argument from the command line, rimage goes straight to the next, sof-tgl binary input that you also sent me and then it crashes again - even with the assert above. I didn't investigate this other crash because I now feel like fuzzing rimage would demonstrate enough memory corruptions to fill a lifetime of debugging. Focusing on and fixing this particular bug requires understanding first the .module parsing issue I just described and addressing it.

And once again I have no idea why rimage runs when using MEU.

@marc-hb

@marc-hb Thanks for deep dive with this! I re-tested and got the assert. Build with proper configuration for TGL works OK. I wonder if the added assert is sufficient to close this issue?

Also in the working build there's rimage run:

Extended manifest saved to file sof-tgl.ri.xman size 0x0310 (784) bytes

Hello I'm rimage!
[100%] Built target run_rimage

$ head -57 src/arch/xtensa/CMakeFiles/run_rimage.dir/build.make | tail -1
	cd /home/singalsu/work/current/sof/sof/build/src/arch/xtensa && ../../../rimage_ep/build/rimage -o sof-tgl.ri -c /home/singalsu/work/current/sof/sof/rimage/config/tgl.toml -s 1344 -k /home/singalsu/work/current/sof/sof/keys/otc_private_key.pem -i 3 -f 1.5 -b 1 -e -x 24 bootloader-tgl sof-tgl; echo "Hello I'm rimage!"

The run is a mistake in makefiles apparently. Should another issue be created?

Build with proper configuration for TGL works OK [...] The run is a mistake in makefiles apparently. Should another issue be created?

I think creating another SOF issue focusing on CMake and user feedback when a "non-proper" configuration is used (which one?) is probably a good idea. Then before you close this one please transfer it to the "rimage" repo, see link on the right side (I'm not sure you can transfer a closed issue)

Closing this old issue.

Navigation Menu

Repository navigation

[BUG] rimage segfault in build for TGL with xt-xcc #3423

8 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[BUG] rimage segfault in build for TGL with xt-xcc #3423

Description

Activity

singalsu commented on Sep 16, 2020

singalsu commented on Sep 16, 2020

marc-hb commented on Sep 16, 2020

ranj063 commented on Sep 16, 2020

marc-hb commented on Sep 16, 2020

8 remaining items

mengdonglin commented on Sep 23, 2020

singalsu commented on Sep 25, 2020

marc-hb commented on Oct 15, 2020

singalsu commented on Oct 16, 2020

marc-hb commented on Oct 16, 2020

singalsu commented on May 31, 2021

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions