Repository navigation
Windows/Linux disassembly differences when using #include ../something.h? #9034
Description
Originally posted by @marc-hb in #9021 (comment)
Same, previously seen disassembly difference in https://github.com/thesofproject/sof/actions/runs/8630426621?pr=9021 already observed in #8863
--- win/build-sof-staging/sof-info/imx8x/zephyr.lst 2024-04-10 19:30:27
+++ lin/build-sof-staging/sof-info/imx8x/zephyr.lst 2024-04-04 09:12:34
@@ -81134,6 +81134,12 @@
92433b10: 833441 l32r a4, 924147e0 <_stext+0x1598> (92413034 <s_lpuartClock>)
92433b13: a03340 addx4 a3, a3, a4
92433b16: 03a8 l32i.n a10, a3, 0
+ * @param name Which clock to enable, see \ref clock_ip_name_t.
+ * @return true for success, false for failure.
+ */
+static inline bool CLOCK_EnableClock(clock_ip_name_t name)
+{
+ return CLOCK_EnableClockExt(name, 0);
92433b18: 31e9 s32i.n a14, a1, 12
92433b1a: 00a0b2 movi a11, 0
92433b1d: ffcfa5 call8 92433818 <CLOCK_EnableClockExt>from your experience, do you happen to know what would cause differences between windows and linux builds?
I downloaded the files, did some inspection, jogged my memory and searched archives and I think I may have found something... Do you have any .. in some #include paths by any chance? We could have a situation similar to these:
- audio: crossover: crossover code refactor #8587 (comment)
- zephyr/CMakeLists.txt: remove '..' in include paths #7428
So the inline C code seems correct in both cases, but for some reason the same zephyr-sdk/xtensa-...-objdump -S is "not as good" when running on Windows and can't find the source of the inline function CLOCK_EnableClock() when disassembling LPUART_Init()... even though both Linux and Windows find that same source earlier in the binary when disassembling mcux_ccm_on()!
There's one notable difference here: mcux_ccm_on() is located in Zephyr whereas LPUART_Init() is located in modules/hal/nxp! Which is why I'm suspecting #include paths?
I think the simple reason it is appearing now is because you're now enabling some lpuart in this commit.
Activity
marc-hb commentedon Apr 11, 2024
You have to click on the "Summary" / home button in the top-left to access the build artefacts. Don't ask me why.
Then don't forget to run "dos2unix" before comparing text files.
I don't think there is a lot CI can do at this point.
Based on this example and the previous ones I referenced above, I have a strong suspicion that some debug sections are "not as good" on Windows, likely because of some funky
#include "../something.h". So the next step is to compile on both Linux and Windows and manually compare the debug sections forLPUART_Init()to confirm that suspicion.We can't automate the comparison of debug sections because they're almost always full of absolute paths.
IF I'm guessing this correctly, then we have 2 approaches to solve this problem:
Either we decide that a slightly inferior debug inferior on Windows is OK. Who uses Windows anyway? :-) Debug sections don't actually matter for reproducible builds; now they seem to just "get in the way". Then the best course is probably a new Zephyr CONFIG_OUTPUT_DISASSEMBLE_WITH_SOURCE that defaults to "yes" but that we can turn off in this test. That new Kconfig would just turn off
disassembly_flag_inline_sourceand problem solved! It should be a small CMake change; there is already CONFIG_OUTPUT_DISASSEMBLE_ALL from @tejlmand that does something somewhat similar.Or you care about the debug experience on Windows and you think reproducible builds is a surprising and indirect but effective way to keep the debug experience on par with the Linux one. Then you need to find why debug sections are not as good on Windows. Hopefully just some relative
#includeto convert to absolute?But again: the first thing is to prove my educated guess about debug sections. EDIT: I would start with comparing
xtensa-...-objdump -S LPUART_Init.obetween Linux and Windows.[-]Disassembly differences when using #include ../something.h?[/-][+]Windows/Linux disassembly differences when using #include ../something.h?[/+]LaurentiuM1234 commentedon Apr 15, 2024
Strangely enough, dropping all of the
static inlinestuff fromCLOCK_EnableClock()'s definition seems to fix this issue?i.e: Inside
fsl_clock.hwe changeto
and move the definition of
CLOCK_EnableClock()tofsl_clock.c.(note:
fsl_clock.his included byfsl_common_dsp.h, which is included byfsl_common.h, which, in turn is included byfsl_lpuart.h. This header is used by the LPUART driver)Results in https://github.com/thesofproject/sof/actions/runs/8689213734/job/23827370722?pr=9033 (ignore 8ULP which doesn't have this change).
HAL change in https://github.com/zephyrproject-rtos/hal_nxp/pull/374/files.
So far I wasn't able to find any ".." include paths in the sources/headers used for SOF.
The main issue that I have with 2) is that it's time consuming and I'm not sure it's worth the effort (like you said, who uses Windows?). These issues are (to me) somewhat annoying to debug (I don't have a Windows environment so I have to rely on the CI to test stuff out) and very unintuitive (I can't really tell you why the aforementioned fix works).
marc-hb commentedon Apr 15, 2024
It could also be a bug somewhere in the toolchain that gets triggered only on Windows somehow. So a 3rd option would be to try to update the SDK? As @dbaluta did in earlier and small commit 45cbf04. It's a minor release so it's unlikely to hurt anyway.
Otherwise can you help with option 1. CONFIG_OUTPUT_DISASSEMBLE_WITH_SOURCE? It's CMake and Kconfig which very few enjoy but it looks pretty trivial.
marc-hb commentedon Apr 15, 2024
Small tangent: after optimization it's extremely hard to connect machine code with source code, anyone who has ever tried to debug optimized code knows this.
https://gcc.gnu.org/onlinedocs/gcc-4.9.0/gcc/Debugging-Options.html
Disassembly runs into the same issue. So maybe using -S on optimized code is too complex to be reliable? Now we're seeing a Windows-only bug, but how many "portable" -S bugs have we ignored so far? Maybe a lot.
LaurentiuM1234 commentedon Apr 16, 2024
Yep, created zephyrproject-rtos/zephyr#71535. Results in: #9051 (ignore
zmainfailures, should be caused by the fact that this change is not upstream yet). 8ULP failures and the 8QXP failures introduced by #9021 seem fixed.cmake: add control over inline source code disassembly
.github/zephyr: disable disassembly with inline source code
cmake: add control over inline source code disassembly
.github/zephyr: disable disassembly with inline source code
cmake: add control over inline source code disassembly