Repository navigation
[BUG] DSP panic on XRUN storm #2214
Description
Describe the bug
If I do arecord | aplay (full command in "reproduce" section), then I do something on the system known to trigger Xruns (incoming SSH connections do the trick for this system in particular, at least when netbooted). Usually after a number of Xruns (which varies between 2 at a minimum to 20 for much of the commit range, though some fail after more than 100 Xruns) I get DSP panic with no report.
This seems to be a regression. I can reproduce the bug on 07bf590 (@akloniex can you check that other platforms are affected?) but not on its parent commit (200+ xruns in one series, aplay/arecord did give up at some point from the Xruns but I was able to restart them; no panic).
To Reproduce
Steps to reproduce the behavior: (e.g. list commands or actions used to reproduce the bug)
The below command should happen on the board:
arecord -Dhw:0,0 --buffer-size=384 -vv -f S24_LE -c 2 -r 48000 | aplay -Dhw:0,0 --buffer-size=96 -f S24_LE -c 2 -r 48000 -t raw
From the computer which may access the board, do, ssh audioboard -t alsamixer in a loop. On my board in particular this triggers an underrun or overrun almost every time; if it doesn't find something else to load up the system so it XRuns.
Reproduction Rate
Number of Xruns varies, but I'm able to consistently reproduce it in all attempts.
Did not check whether disabling the mailbox trace affects it.
Expected behavior
A clear and concise description of what you expected to happen.
Impact
What impact does this issue have on your progress (e.g., annoyance, showstopper)
Annoyance.
Environment
- Branch name and commit hash of the 2 repositories: sof (firmware/topology) and linux (kernel driver).
- Kernel: (internal build, though shouldn't affect)
- SOF: 62dbfcf (master)
- Name of the topology file
- Topology: sof-imx8qxp-cs42888.m4 (except backend is renamed to align with kernel)
- Name of the platform(s) on which the bug is observed.
- Platform: imx8 (note: DSP has only one Xtensa core)
- Other specific changes to the configuration
- Agent is disabled (agent-triggered panics mask this issue)
Activity
lgirdwood commentedon Dec 16, 2019
@paulstelian97 it's probably also worth scripting and upstreaming a version of this test so we can have a general script to force xrun storms for future test/debug/CI.
Could we be in an IRQ storm here (since we are continually rescheduling new pipeline copies) ? This would explain why there is no exception.
paulstelian97 commentedon Dec 17, 2019
@lgirdwood I don't know of any variant that is consistent to cause many Xruns (as in once in 2 seconds, like in my manual test case), do you have any idea? For me simply loading up the system using new SSH connections worked but I'm not sure this works on your test boards.
I don't think there is any IRQ storm (at least my cascaded IRQ controller driver doesn't claim any; not sure about IPC IRQ storms though but given the rate of Xruns that I'm able to generate to trigger this error I doubt it)
The panic code is 63 (forced panic) but I see no trace related to that, not even those I manually inserted in the __panic function. Since the kernel on my platform doesn't react to panics (I have started a thread on the mailing list to discuss about how we can implement that) I cannot actually look at a good stack trace and am limited to
tools/coredumper/sof-coredump-reader.pywhich does dump the 64 registers but has no good stack dump. I need to figure out where that panic is actually triggered though.lgirdwood commentedon Dec 18, 2019
@paulstelian97 Fwiw, the system agent will force a panic iff the DSP does not go into the idle state (not sure if this is 63 though without checking)
paulstelian97 commentedon Dec 18, 2019
@lgirdwood Yeah, I already mentioned that the agent is explicitly disabled. With the agent enabled, the problem is quite a bit older (as in since the last refactors to the agent itself a few months ago).
I wonder if this 63 (explicitly triggered panic) comes from there actually being a hang and the last code not being cleared. I could try adding a memset(0) of the exception mailbox at boot to test this possibility.
paulstelian97 commentedon Dec 19, 2019
Found new repro, simply closing arecord while aplay is still running is enough to cause DSP crashes. Need more to investigate, when I have a good repro I will reopen and investigate. Will check my bisect against this variant too.
paulstelian97 commentedon Dec 19, 2019
Closing, much simpler repro at #2233 (same regression). Apparently the Xruns actually cause arecord to stop and they aren't the direct cause of the crash but arecord stopping is.