Repository navigation
[BUG] [CNL] IPC timeout during suspend-resume tests #2498
Description
I've been getting an "IPC timed out" error during my weekly Mantis testing:
[ 7339.996692] [93112] sof-audio-pci 0000:00:1f.3: ipc tx succeeded: 0x80010000
[ 7340.010861] [93112] sof-audio-pci 0000:00:1f.3: FW Poll Status: reg=0x20140000 successful
[ 7340.010874] [93112] sof-audio-pci 0000:00:1f.3: ipc tx: 0x40010000
[ 7340.514720] sof-audio-pci 0000:00:1f.3: error: ipc timed out for 0x40010000 size 76
[ 7340.514724] sof-audio-pci 0000:00:1f.3: info: preventing DSP entering D3 state to preserve context
[ 7340.514732] [93112] sof-audio-pci 0000:00:1f.3: unknown ROM status value ffffffff
[ 7340.514775] sof-audio-pci 0000:00:1f.3: invalid header size 0xffffffff. FW oops is bogus
[ 7340.514779] sof-audio-pci 0000:00:1f.3: error: unexpected fault 0xffffffff trace 0xffffffff
[ 7340.514789] sof-audio-pci 0000:00:1f.3: error: hda irq intsts 0x00000000 intlctl 0xc0000000 rirb 00
[ 7340.514792] sof-audio-pci 0000:00:1f.3: error: dsp irq ppsts 0x00000000 adspis 0x00000000
[ 7340.514799] sof-audio-pci 0000:00:1f.3: error: host status 0x00000000 dsp status 0x00000000 mask 0x00000003
[ 7340.514803] sof-audio-pci 0000:00:1f.3: ctx_save ipc error -110, proceeding with suspend
[ 7340.514906] [93112] sof-audio-pci 0000:00:1f.3: FW Poll Status: reg=0x1010f0f successful
[ 7340.514913] [93112] sof-audio-pci 0000:00:1f.3: FW Poll Status: reg=0xf0f successful
[ 7340.514917] [93112] sof-audio-pci 0000:00:1f.3: DSP core(s) enabled? 0 : core_mask f
[ 7340.514967] [93112] sof-audio-pci 0000:00:1f.3: Debug PCIR: 00000010 at 00000044
[ 7340.515009] [93112] sof-audio-pci 0000:00:1f.3: Current DSP power state: D3
We've had similar issues with different codes or on different platforms: #2334 thesofproject/sof#3541 and it's unclear whether they all have the same cause or not. This specific one triggers during "check-suspend-resume-with-audio.sh -m capture" on "HDA Analog"
Activity
xiulipan commentedon Oct 14, 2020
@lyakh I just tried to reproduce this test on some CML platforms, no luck with 3 tries. Will do more test
lyakh commentedon Oct 14, 2020
@xiulipan you mean 3 single tests or 3 loops of N tests? For me it typically takes between 10 and 15 repetitions.
xiulipan commentedon Oct 14, 2020
@lyakh Thanks for the clarify, I will try more time.
wwittbrx commentedon Oct 15, 2020
@lyakh What configs are you using when building the FW (cannonlake_defconfig, cannonlake_chrome_overridedefconfig)?
Also, what commits are you building for FW? I can reproduce similar issue on slim driver when running f97e9f0 with Chrome config. No reproduction on e5d17f3.
lyakh commentedon Oct 20, 2020
@wwittbrx I'm using cannonlake_defconfig. The newest commit with which I was able to reproduce the problem was 80ef3cf Even easier it seems to be to reproduce it with 1b17974
lyakh commentedon Oct 21, 2020
Update:
keyonjie commentedon Oct 26, 2020
@lyakh since @xiulipan can't reproduce with the same recipe, can you help to debug it more once you can reproduce it stablly.
lyakh commentedon Oct 26, 2020
Triggered again in a kmod-load-unload test as documented in #2535 which is closed as duplicate of this issue.
lyakh commentedon Nov 16, 2020
update 2: as of last Friday's weekly test the problem didn't occur. Also wasn't able to reproduce it manually earlier during last week. How about I check again end of this week, and if the bug doesn't show up again, we close this bug (until it re-appears)?
mengdonglin commentedon Nov 23, 2020
@lyakh Okay. Thank you!
lyakh commentedon Nov 23, 2020
Last Friday same result: no more reproduction, closing this bug as agreed
mengdonglin commentedon Nov 30, 2020
@lyakh Thank you for double check!
lyakh commentedon Dec 4, 2020
I think it is the same issue, but maybe it is not. We've had several bugs filed with different IPC fields reading 0xffffffff or 0xff000000 or something similar, this is one of them. Now I came across a failure of thesofproject/sof#3642 on Mantis (again) with no complete dmesg or firmware logs, but the test quotes a single kernel log line:
which means a trace position IPC has been read with
.overflowand.messagesset to 0xffffffff. Checking the code this cannot be done by the firmware, so, this leads me to think, that it's again mailbox content flipping to 0xffffffff possibly again while suspending the DSP.plbossart commentedon Dec 4, 2020
@lyakh The Mantis issue also happened for one of my SoundWire PRs, see https://sof-ci.01.org/linuxpr/PR2601/build4953/devicetest/?model=CML_MANTIS_HDA&testcase=check-kmod-load-unload
It doesn't look random any longer, I can't recall having see two reports on the same day? This also happens smack while we are doing PM-related IPC...
plbossart commentedon Jun 16, 2021
@lyakh if this isn't a known issue any longer please close. We can always reopen a new issue later.
lyakh commentedon Jun 17, 2021
Closing for a lack of recent activity or evidence