Repository navigation
[BUG] [APL] FW core dump has empty oops data if FW built with GCC #1346
Description
Describe the bug
In case DSP hits an exception, it dumps oops data for the driver to read out.
What actually happens is that I get a dump file that is partially ok. I can find "struct sof_ipc_panic_info" correctly filled. E.g. I can find SOF_IPC_PANIC_MAGIC and the rest of the struct seems ok. Start of the dump however is just zeroes and there doesn't seem to be any valid values for "struct sof_ipc_dsp_oops_xtensa". Stack dump seems ok and I can find correct functions my manually looking up symbols from the stack dump, but the coredumper python scripts cannot make sense out of this dump as many key fields are just zeroes.
To Reproduce
Cause FW to hit oops. I did this by adding following code to volume.c:volume_copy()
» » panic(1234);
» » while(1) {}
Expected behavior
I can extract the oops file by doing:
scp root@dut:/sys/kernel/debug/sof/exception oops.bin
And feed data to coredumper tool.
Impact
If DSP oops cannot be succesfully saved, debugging hard-to-reproduce bugs is severely impacted.
Environment
-
Branch name and commit hash of 3 repositories: sof (firmware), linux (kernel driver) and soft (tools & topology).
linux sofdev 2e94569
sof master e14ab70 -
Name of the topology file
n/a -
Name of the platform(s) on which the bug is observed.
APL UP2 -
Reproducibility Rate. If you can only reproduce it randomly, it’s useful to report how many times the bug has been reproduced vs. the number of attempts it’s taken to reproduce the bug.
100%
Screenshots or console output
Two example dumps attached.
Highlights from the comments below:
problem is definitely in how the dump routine handles WINDOWBASE updates. If ROTW is called even once (like happens with 1 iteration of store_register_loop), the result is an invalid dump. Code looks current and I fail to see how a single ROTW can have such impact (there are only a few ops on the core after this
Got basic gdb working at least do a degree within QEMU and it seems ROTW causes another exception and we end up in DoubleExceptionVector handler. But that's probably just a symptom, the same code works when compiled on XT-CC. I
I now got an OK exception dump (for another bug) on WHL (cnl image), built with GCC, so at least this is not happening in all cases. Rootcause still unknown.
Fwiw, the ABI between GCC and XCC is slightly different wrt calling convention and registers windows hence there are some incompatibilities with some of the dump data.
Activity
kv2019i commentedon Apr 29, 2019
@mmaka1 @abonislawski it seems part of the dump is correct (panic header and stack dump), so basic procedure I follow seems to be ok. Can you take a look at the exception dumps and see if yo notice anything obviously wrong that could explain this?
Btw, I compiled now with GCC, I'll try with Xtensa toolchain as well and see if that makes a difference.
[-][BUG] [APL] FW core dump has empty oops data[/-][+][BUG] [APL] FW core dump has empty oops data if FW built with GCC[/+]kv2019i commentedon Apr 29, 2019
Ok, bingo! When I rebuilt FW with XT toolchain, ta-daa, dump copied verbatim from /sys/kernel/debug/sof/exception looks perfectly fine. Exactly the same dump compiled with GCC results in the weird partially filled oops file I copied as attachment to this bug. Relabelled the subject, so this applies only to gcc.
lgirdwood commentedon Apr 29, 2019
@kv2019i Can you attach both exception binaries, this will help root cause the issue. It may be the exception window size is different between xcc and gcc (and we may have to pad if using gcc).
kv2019i commentedon Apr 30, 2019
@lrgirdwo wrote:
I was looking at that but it seems there is really a lot of data missing. E.g. I can' t find a byte sequence that would look like a valid "struct sof_ipc_dsp_oops_arch_hdr" anywhere in the GCC dump. I'll attach the binary files and attach a hexdump diff below. The oops data seems completely zeroed out and I can't find e.g. the oops config codes anywhere in the GCC-built FW oops, so it seems it's more than an offset.
The diff looks a bit hard to read in mark down, so I'll attach the full binaries plus my hexdump-diff as well.
dsp-oops-20190430.zip
lgirdwood commentedon Apr 30, 2019
@kv2019i the other thing I can think of is calling convention between C and assembler. Can you try again and comment out arch_dump_regs_a below and see if the header is correct.
and also check that a2 is used by fill_core_dump() for oops.
kv2019i commentedon May 2, 2019
@lgirdwood wrote:
Calling code seems to be ok, but something in arch_dump_regs_a is definitely causing this.
Currently I've isolated the problem to "store_register_loop:" segment of exc-dump.S. If I jump over the register store loop, the dump looks ok -- both header and the special registers saved after the register store loop look ok. The code does look correct, so not yet figured out why this is wrong. The callstack seems to slightly differ between GCC and XT builds, so maybe this triggers some issue w.r.t register windowing. Continuing to debug this.
lgirdwood commentedon May 2, 2019
@kv2019i I assume store_register_loops works if it only does 1 iteration ? Maybe worth running it through qemu and trace the registers.
kv2019i commentedon May 2, 2019
@lgirdwood wrote:
The strange thing it doesn't. Even if I force iterations to one, the oops get completely messed up. :P The disassembly looks completely same as well, so pretty weird. I'll have to try in QEMU as well -- I was actually assuming coredump works under QEMU, but in fact I have not tested that myself.
Update: problem is definitely in how the dump routine handles WINDOWBASE updates. If ROTW is called even once (like happens with 1 iteration of store_register_loop), the result is an invalid dump. Code looks current and I fail to see how a single ROTW can have such impact (there are only a few ops on the core after this). 'll try to reproduce in QEMU to understand what happens with ROTW.
lgirdwood commentedon May 3, 2019
@kv2019i debuging another issue and here are some qemu dumps
Memory dump of
hexdump -C /dev/shm/qemu-bridge-hp-sram-mem | lessQemu registers
lgirdwood commentedon May 3, 2019
@kv2019i AR regs start at offset 0xa8cc (base is 0xa800), header looks OK. A regs appear to be missing, but are duplicated in AR36 - AR51 , except AR38
lgirdwood commentedon May 3, 2019
and here is stack
kv2019i commentedon May 7, 2019
@lgirdwood
Got basic gdb working at least do a degree within QEMU and it seems ROTW causes another exception and we end up in DoubleExceptionVector handler. But that's probably just a symptom, the same code works when compiled on XT-CC. If I skip the register dump loop that uses ROTW, and dump register state, it looks suspicious (WINDOWBASE and WINDOWSTART), compared to succesful DSP oops dump when compiling with XT-CC.
kv2019i commentedon May 10, 2019
I now got an OK exception dump (for another bug) on WHL (cnl image), built with GCC, so at least this is not happening in all cases. Rootcause still unknown.
3 remaining items
tlauda commentedon Mar 12, 2020
@kv2019i Still reproduced?
kv2019i commentedon Mar 20, 2020
@tlauda I'm stuck in a continuous stream of P1 bugs, so I haven't had time to test this in recent times. As nobody else seems to have hit this, let's downgrade to P3 and upon next triage round, let's close if no new findings (and I've not reproduce by that time).
tlauda commentedon Mar 20, 2020
@kv2019i OK, thanks for the update.
marc-hb commentedon Jun 25, 2021
After banging my head really hard and pulling a lot of hair I had the funny idea of hitting https://www.google.com/search?q=arch_dump_regs_a There is a single hit on the entire Internet: this page.
To avoid someone getting hurt again I submitted panic comments and a workaround suggestion in PR #4401
panic: add comments, especially link to dump corruption bug #1346
plbossart commentedon Jun 25, 2021
https://en.wikipedia.org/wiki/Googlewhack !
github-actions commentedon Jun 20, 2025
This issue has been marked as stale because it has been open (more than) 60 days with no activity. Remove the stale label or add a comment saying that you would like to have the label removed otherwise this issue will automatically be closed in 14 days. Note, that you can always re-open a closed issue at any time.