Repository navigation
[BUG]DSP panic when suspend-resume on cml-sku0983-sdw and cml-sku0955-hda #4538
Description
Describe the bug
in inner daily /5449?model=CML_SKU0983_SDW&testcase=check-suspend-resume-50
DSP panic when suspend-resume
To Reproduce
TPLG=/lib/firmware/intel/sof-tplg/sof-cml-rt711-rt1308-mono-rt715.tplg ~/sof-test/test-case/check-suspend-resume.sh -l 50
Reproduction Rate
rather low maybe , cannot reproduce it yet (once I can , I'll update this comment)
Environment
Kernel Branch: topic/sof-dev
Kernel Commit: d54b7803 storage: d54b7803-2
SOF Branch: main
SOF Commit: 52bc39f storage: 52bc39f
Topology: /lib/firmware/intel/sof-tplg/sof-cml-rt711-rt1308-mono-rt715.tplg
Platform: cml-sku0983-sdw
Screenshots or console output
[console]
2021-07-22 23:04:39 UTC [REMOTE_INFO] ===== Round(47/50) =====
2021-07-22 23:04:39 UTC [REMOTE_COMMAND] Run the command: rtcwake -m mem -s 5
rtcwake: assuming RTC uses UTC ...
rtcwake: wakeup from "mem" using /dev/rtc0 at Thu Jul 22 23:04:45 2021
2021-07-22 23:04:46 UTC [REMOTE_COMMAND] sleep for 5
2021-07-22 23:04:51 UTC [REMOTE_INFO] Check for the kernel log status
declare -- cmd="journalctl_cmd --since=@1626995074"
2021-07-22 23:04:51 UTC [ERROR] Caught kernel log error
===========================>>
[ 5561.663293] kernel: sof-audio-pci-intel-cnl 0000:00:1f.3: error : DSP panic!
[ 5561.663306] kernel: sof-audio-pci-intel-cnl 0000:00:1f.3: ------------[ DSP dump start ]------------
[ 5561.663320] kernel: sof-audio-pci-intel-cnl 0000:00:1f.3: status: fw entered - code 00000005
[ 5561.663392] kernel: sof-audio-pci-intel-cnl 0000:00:1f.3: error: extended rom status: 0x5 0xdead00b 0x41c0 0x0 0x0 0x0 0x1811102 0x0
[ 5561.663398] kernel: sof-audio-pci-intel-cnl 0000:00:1f.3: ------------[ DSP dump end ]------------
<<===========================
Activity
keyonjie commentedon Aug 16, 2021
@ujfalusi looks we have very limited dmesg log when DSP panic happen now?
ujfalusi commentedon Aug 16, 2021
@keyonjie, by default we are not flooding the kernel log with repeated dumps when the firmware crashes, we print the dsp dump only once and we also print the ipc dump once (separately counted).
All optional dumps and the dump flood can be re-enabled via module parameter
sof_debugwith bit 4 is set.The dumps have not been changed, we print the same information as before but we have markers around to make them easier to spot.
lgirdwood commentedon Aug 16, 2021
@ujfalusi does the FW stack and regs get dumped by default (it should not be an option) ?
ujfalusi commentedon Aug 16, 2021
The content of the dump has not been changed and there is no option to change it. The only change is that by default we print the dump (DSP / IPC) once.
The (HDA) registers will be dumped if the fw_state is SOF_FW_BOOT_COMPLETE, if the dump is called during any other state then the registers are not printed and they were not printed in the past either.
XiaoyunWu6666 commentedon Aug 20, 2021
happen in inner daily 5971 on model CML_SKU0983_SDW in testcase check-suspend-resume-50
keqiaozhang commentedon Aug 23, 2021
@fredoh9 , can you help to follow up this issue? since we only have one device of this model in JF and it's hard for us reproduce the suspend/resume issue via ssh.
lgirdwood commentedon Aug 27, 2021
@XiaoyunWu6666 @fredoh9 @aiChaoSONG does our inner daily testing have the capability to dump FW crashes and run the crash dump script on each FW crash. This would be really useful information to attache to the issues. @mengdonglin fyi.
This FW crash dump script should also be usable in CI too (and also a great help in root causing bugs).
fredoh9 commentedon Aug 27, 2021
I can reproduce this on CML_SKU0983_SDW. I think if we use iteration of 50, high chance to reproduce.
@lgirdwood Where is the crash dump script? And I can try to run the script manually then I can try to integrate to SOF CI test. Do you recommend sof_debug to set bit 4 also?
lgirdwood commentedon Aug 28, 2021
sof-coredump-reader.py is in sof/tools/coredumper/
Great :)
What will this do ?
38 remaining items
ranj063 commentedon Oct 15, 2021
@keyonjie #4874 fixes the symptom not the root cause.
keyonjie commentedon Oct 18, 2021
@ranj063 I had the same feeling. But when I am trying it think it louder, maybe the place "assert(task_next);" should be a normal path (Not error path I mean), so actually there is no error if we went there and the "assert" is a false positive.
keyonjie commentedon Oct 18, 2021
Thanks, @XiaoyunWu6666 the PR is open now.
miRoox commentedon Nov 22, 2021
cannot be reproduced, close.
lyakh commentedon Nov 23, 2021
I think the problem is not about whether or not the task list is empty, but that an EDF software interrupt triggers after resume. IIUC, EDF scheduling is done from software interrupts. As long as an EDF interrupt triggers,
edf_scheduler_run()will run. And if the task list is empty, or if tasks on the list aren't suitable to be run, we get anassert(). So the real question seems to be why we're getting a software EDF interrupt after resume? Has it been scheduled before suspend or after resume or is it spurious (unlikely)? Can it be, that it was scheduled before suspend and somehow survived the suspend? If it was correctly scheduled before suspend but just couldn't be processed, it should be safe to drop it after resume. Can we clear the interrupt inscheduler_restore_edf()?lyakh commentedon Nov 24, 2021
An important question: do we know on which core the panic happens? I assume this isn't happening when restoring a secondary core after a D0ix?
After looking at the code some more, I don't see a good way to clear the EDF (software) interrupt. I think
interrupt_mask()would would clear it, as described in an Architecture Specification document, so we could add apair of calls to
scheduler_init_edf()anywhere before the call tointerrupt_enable(), but I'm not 100% sure this would work. E.g. as in #5026 . If that doesn't work, I'd propose to remove theassert()call at least temporarily to win us more time to investigate this properly.mengdonglin commentedon Nov 29, 2021
This issue can be reproduced on an ADL-P device with https://github.com/thesofproject/sof/releases/tag/v1.9.2, but cannot reproduce with latest main branch
miRoox commentedon Dec 6, 2021
It was fixed by #4874, the patch was also cherry-picked to stable-v1.9 and stable-v2.0.