Repository navigation
XCC RG-2017.8-linux: generated object code is affected by -g2 level (the default level) + longer source paths #7114
Description
tl;dr: if you're failing to reproduce a build, try a longer or shorter west topdir source directory. That's the workaround. There seems to be a threshold somewhere around 27 characters.
Summary
A very strange and complex toolchain issue where the -g level and debug symbols affect the object code on some Intel CI build system(s) for a totally unknown reason.
Unexpected variations like this break build reproducibility = how this issue was discovered in the first place.
Variations like these could also make it harder to reproduce race conditions and other difficult bugs.
I focused on .text.k_ticks_to_us_floor64 and RG-2017.8-linux below because it just happened to be the first difference I found but I noticed that the -g level also affects .text.z_swap_irqlock in the same conditions and there could be unexpected differences in other files.
=> Don't trust RG-2017.8-linux to produce consistent object code
This is not a problem with the Zephyr SDK toochain: it produces identical object code across Linux and Windows, as routinely verified in CI across all targets: https://github.com/thesofproject/sof/actions/runs/4189511798/jobs/7262044804
I haven't observed any problem with RI-2020.5-linux (and MTL) yet which Zephyr still identifies as GNU 4.2.2. Note there is on-going work to switch to xt-clang.
Longer story
Reproduced on SOF commit 64fe2ea + west update + sof/scripts/xtensa-build-zephyr.py -p tgl
Reproduced with a couple other SOF commits too.
The code generated for the k_ticks_to_us_floor64() normally looks like this:
xt-objdump -s --section=.text.k_ticks_to_us_floor64 ../build-tgl/zephyr/kernel/CMakeFiles/kernel.dir/sched.c.obj
Contents of section .text.k_ticks_to_us_floor64:
0000 3661000c 05b10000 61000041 00000c07 6a......a..A....
0010 b08682b0 b6a2a2c8 fe87ba01 1bbb0c0d ................
0020 c2afff81 0000e008 00a0d4a2 a0e582a0 ................
0030 c482b0f4 827cfb0c 0afaddea dd16fd07 .....|..........
0040 9cea40a2 8250d282 40b2a240 c382dabb ..@..P..@..@....
0050 cabbdd07 cd068100 00e00800 3d0b2d0a ............=.-.
0060 1df0bd03 ad02dd07 cd068100 00e00800 ................
0070 b0d482a0 c582a0b4 a2a0a482 dabbcabb ................
0080 dd07cd06 810000e0 0800cd06 dd07a911 ................
0090 b901ad02 bd038100 00e00800 3801a095 ............8...
00a0 82e811b0 b482a084 a2a02482 ba882a2e ..........$...*.
00b0 9a888a33 e7b2151b 331df000 00000000 ...3....3....... <== padding / alignment
00c0 ceebb0c0 41f73f0d 0c1a86dc ff1df0 ....A.?........
=> Note the five 00 bytes of padding.
@andyross, who answered A LOT of questions during this investigation (thank you!), suspects xt-as adds this padding for cache alignment and performance reasons. This padding is there in most but unfortunately not all cases, which is a bug.
You don't need SOF to observe this code and padding, you can produce it with a pure upstream zephyr workspace:
ZEPHYR_TOOLCHAIN_VARIANT=xcc
XTENSA_TOOLCHAIN_PATH=$XCCLOC/install/tools
TOOLCHAIN_VER=RG-2017.8-linux
XTENSA_SYSTEM=$XCCLOC/install/builds/RG-2017.8-linux/cavs2x_LX6HiFi3_2017_8/config
export ZEPHYR_TOOLCHAIN_VARIANT XTENSA_TOOLCHAIN_PATH TOOLCHAIN_VER XTENSA_SYSTEM
cd zephyr/
git checkout d9c4ec31fc49e7eef3c
west build -p -b intel_adsp_cavs25 samples/hello_world/ -- -DCONFIG_SYS_CLOCK_TICKS_PER_SEC=15000 -DCONFIG_SPEED_OPTIMIZATIONS=y
xt-objdump -s --section=.text.k_ticks_to_us_floor64 zephyr/kernel/CMakeFiles/kernel.dir/sched.c.obj
Make sure you use ZEPHYR_TOOLCHAIN_VARIANT=xcc and other variables below. If .text.k_ticks_to_us_floor64 is missing then you probably forgot to switch to that toolchain.
I could unfortunately not reproduce this issue with hello_world in any configuration; so far reproduction requires SOF.
On at least two of our automated build systems (sofbld07 and 08, Ubuntu 20.04), the padding disappears in the default configuration. This breaks build reproducibility. There are other differences in sched.c.obj caused by -g (apparently not in other .c.obj files, which don't have 150+ ELF sections)
EDIT: this happens because these systems use longer source paths / debug symbols, that's what triggering this compiler bug.
When unspecified with -g, the default debug level is '-g2'. All other things equal, the high debug level set by default by Zephyr makes the usual padding disappear on these particular systems.
When decreasing the debug level in cmake/compiler/gcc/compiler_flags.cmake to -g1 or -g0 or no -g at all and making no other change whatsoever, the usual padding re-appears! In other words, a lower -g debug level makes the object code that runs on the DSP normal again.
--- a/cmake/compiler/gcc/compiler_flags.cmake
+++ b/cmake/compiler/gcc/compiler_flags.cmake
@@ -168,11 +168,11 @@ check_set_compiler_property(APPEND PROPERTY hosted -fno-freestanding)
check_set_compiler_property(PROPERTY freestanding -ffreestanding)
# Flag to enable debugging
-set_compiler_property(PROPERTY debug -g)
+set_compiler_property(PROPERTY debug -g1 )
# GCC 11 by default emits DWARF version 5 which cannot be parsed byFor quick testing use:
cd build-tgl/
touch ../zephyr/kernel/sched.c
ninja -j1 -v
... then copy/paste and edit the line that compiles sched.c
The difference appears during the -S sched.i -> sched.s compilation step.
The compiled sched-g1.s and sched-g2.s have no actual difference in the assembly code, they only have a lot of .byte lines which are different. Somehow these bytes affects the assembly phase and the object code on those systems.
The pre-processed files sched-g1.i and sched-g2.i are strictly identical.
Another proof that the -g level triggers the issue: "hiding" the zephyr source code during the -S sched.i -> sched.s step restores the usual padding. stracing with -e openat the compiler at that step shows that it reads hundreds pf source files when using -g (even at the -g1 level that preserves the usual padding).
I tried to reproduce on a few other systems including another Ubuntu 20.04 but did not: the padding is always there no matter what I tried, only a few systems are affected. I'm of course using the very same toolchain.
I could not reproduce on the "guilty" systems with samples/hello_world/ either: with hello_world the padding is always there too. Too few debug symbols in hello_world to trigger this bug?
I unfortunately couldn't find what's unusual about these CI systems. Whatever makes them special, I don't think that, all other things being equal, the generated object code should EVER differ between -g1 and -g2 in any circumstance. It should even less differ on some systems but not on others when using the very same toolchain. So this qualifies as a compiler bug IMHO and makes that toolchain untrustworthy for reproducible builds.
cc:
Activity
marc-hb commentedon Feb 17, 2023
Some relevant code
include/zephyr/sys/time_units.h:#define Z_HZ_ticks CONFIG_SYS_CLOCK_TICKS_PER_SECsched.s
After assembly and disassembly
Dump WITHOUT the padding. Some offsets are adjust accordingly.
marc-hb commentedon Feb 17, 2023
I asked people in my team to run the
xt-objdumpcommand above. Most of them have the padding but (at least) one other person was also missing the padding in the default configuration.As soon as he lower the debug level to
-g1the padding was restored, exactly likesofbld07andsofbld08. So these two CI systems are "special" but no so special.marc-hb commentedon Feb 18, 2023
I compared all the toolchain files between
sofbld07which drops the padding by default and my workstation which does not. A number of text files in the toolchain have absolute installation paths in them. After a mass search/replace of these installation paths, all the files in the toolchain were byte for byte identical across the two machines.lgirdwood commentedon Feb 20, 2023
@marc-hb XCC bugs go to vendor.
marc-hb commentedon Feb 21, 2023
The differences between build machines is actually just a difference in (debug) path lengths. I think I can now reproduce the object code change on any system as long as
west topdir | wcis higher than 27 characters. Again lowering the debug level to-g1or-g0avoids the object code variation.[-]XCC RG-2017.8-linux: generated object code is affected by the -gN level on Intel CI build systems[/-][+]XCC RG-2017.8-linux: generated object code is affected by the -g2 level + longer source paths[/+][-]XCC RG-2017.8-linux: generated object code is affected by the -g2 level + longer source paths[/-][+]XCC RG-2017.8-linux: generated object code is affected by -g2 level (the default level) + longer source paths[/+]andyross commentedon Feb 21, 2023
FWIW: I have a setup internally where I have xt-xcc/clang version RI-2021.6 built for all the SOF targets (and a handful of other tool versions, e.g. the original targets for all the devices). If you have a simple-ish test case it wouldn't be much of a hardship to try it across the cross product of the version/device space to try to isolate the problem.
marc-hb commentedon Feb 21, 2023
I tried again to reproduce without SOF but still could not. Maybe because the number of ELF sections is much lower? RI-2021.6 does not seem affected but I did not try to "stress" it.
Reproducing with
RG-2017.8-linuxand SOF is now extremely simple. I understand it's time-consuming to install that toolchain if you don't have it yet.andyross commentedon Feb 21, 2023
I guess what I was thinking of as a minimal test was "here's an preprocessed C file and some other files read by the toolchain in a tarball, unpack it into /opt/foo and it works, but /opt/very___long___name/ fails".
marc-hb commentedon Feb 21, 2023
Never a dull moment! sof commit a942f10 ("west.yml: upgrade Zephyr to 0c0d73721ed") added ~1000 new zephyr commits which seems to change everything; now the padding seems always dropped?!?
Let's ignore that, too many variables already. Let's focus on the original bug description which is unchanged in sof commit 7fdc623.
Also, while I could never caught it "red-handed", I recommended temporarily uninstalling
ccacheto stay on the safe side. This issue has too many variables already.The attached
sched.*files were created with the following commands ran in two directories. The two directories have strictly identical content, the only difference is one has a longer name.The object code padding is found when compiling SOF in the short path
/tmp/zprj/and it is missing when compiling in the long path/tmp/long_enough_directory_name/.This cannot be anything but a toolchain bug.
I'm attaching
sched.i,sched.sandsched.ofor both directories. They were generated thanks tocd build-tgl; touch ../zephyr/kernel/sched.c,ninja -j1 -vand modifying and re-running thext-xcc sched.ccommand displayed.missing_padding_sched.zip
Expected debug path differences aside, the two
sched.ifiles are strictly identical.The two
sched.shave only.byteand.filedifferences, the actual assembly code is the same. They seem enough to reproduce the issue anywhere else, see below.Yet the two
sched.ohave object code differences, notably this missing padding (but not just)This very short
xt-xcccommand is enough to reproduce on another system the twosched.ofiles from their respectivesched.ssources without playing any path game:valgrind --trace-children=yesreports only small leaks inxt-xccandxt-as, nothing really worrying.andyross commentedon Feb 22, 2023
Nice, that's a smoking gun. I'll run these against all my local toolchain variants tomorrow to see if we can figure out which version(s) and/or core(s) are at fault.
Note that the ".byte" directive is the offending bit for sure, that's an instruction to assemble the list of numbers that follow into the stream, probably that's how the compiler has decided to express padding. The ".file" directive just specifies the name of the file, which obviously shouldn't change the generated code; GNU as documents it as a noop; my guess is xt-as does so too.
marc-hb commentedon Feb 22, 2023
The
.bytedirectives are found only at the very end of the assembly, very far away from thek_ticks_to_us_floor64code and.section, and only inside the.debug_xxx.sections.marc-hb commentedon Feb 22, 2023
I filed new case # 46682483 at https://support.cadence.com
marc-hb commentedon Mar 2, 2023
... and the reply is unsurprisingly "upgrade your toolchain". Which will likely never happen for TGL, @lgirdwood confirm?
marc-hb commentedon Mar 7, 2023
After internal discussion we're not going to upgrade the toolchain and perform a full round of validation for a reproducibility issue "only". We hope there is no other side effect.
Moreover there is a (funny!) workaround for the reproducibility issue: make your source directory longer or shorter.
Closing as wontfix.
marc-hb commentedon Mar 24, 2023
I tested again with the same sof commit 64fe2ea and I confirm that there is a
west topdirlength threshold between 27 and 28 characters. There are other thresholds above 28 characters but none below 27.Then I tested again with much later sof commit ac91071 and I could not reproduce this bug at all! west topdir lengths below 50 all produce the same output! Go figure...
For v2.5, the west topdir length threshold is between 35 and 36.
marc-hb commentedon Mar 24, 2023
The recent introduction of CONFIG_ASSERT in commit 4d67d2f means that
./scripts/xtensa-build-zephyr.py --debug ...will now always be different across systems and users, even for the stripped output. SeeTotally different issue and not really a "bug".