Repository navigation
[BUG] TGLU_RVP_NOCODEC DSP panic while suspend-resume test #4480
Description
Describe the bug
DSP panic logs seems different from the others DSP panic, and after seeing DSP panic, following FW status and IPC are working fine.
[12428.728341] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: DSP core(s) enabled? 1 : core_mask 1
[12428.731876] kernel: serial 00:03: activated
[12428.737923] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: FW Poll Status: reg[0xd4]=0x80000000 successful
[12428.738009] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: error : DSP panic!
[12428.738012] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: panic: dsp_oops_offset 788480 offset 788480
[12428.738030] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: status: fw entered - code 00000005
[12428.738041] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: extended rom status: 0x5 0xdead00b 0x41c0 0x0 0x0 0x0 0x25101c2 0x0
[12428.959487] kernel: atkbd serio0: Failed to deactivate keyboard on isa0060/serio0
[12428.995064] kernel: nvme nvme0: 8/0/0 default/read/poll queues
[12429.030091] kernel: ata1: SATA link down (SStatus 4 SControl 300)
[12429.038399] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: FW Poll Status: reg[0x80000]=0x5 timedout
[12429.038419] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: status: fw entered - code 00000005
Found in internal weekly stress test, test result 5213.
model=TGLU_RVP_NOCODEC&testcase=check-suspend-resume-100
To Reproduce
TPLG=sof-tgl-nocodec.tplg ~/sof-test/test-case/check-suspend-resume.sh -l 100
Reproduction Rate
not sure but not 100% need high iteration.
In this test, it is found in 71/100 loop.
Expected behavior
suspend/resume test return success
Environment
- Kernel Branch: topic/sof-dev
- Kernel Commit: 62368fe1
- SOF Branch: main
- SOF Commit: 5374da8
- tplg: sof-tplg-nocodec.tplg
Attached full dmesg from the test.
dmesg_tgl_nocodec_dsp_panic.zip
Activity
fredoh9 commentedon Jul 12, 2021
check-suspend-resume-with-capture-25 in same weekly test found same DSP panic.
plbossart commentedon Jul 12, 2021
@fredoh9 are you saying that the same commits pass or fail depending on how long the tests are?
plbossart commentedon Jul 12, 2021
Answering to my own question
PASS: https://sof-ci.sh.intel.com/#/result/planresultdetail/5216?model=TGLU_RVP_NOCODEC&testcase=check-suspend-resume-50
TPLG=sof-tgl-nocodec.tplg ~/sof-test/test-case/check-suspend-resume.sh -l 50
Start Time: 2021-07-11 21:20:54 UTC
End Time: 2021-07-11 23:45:30 UTC
Kernel Branch: topic/sof-dev
Kernel Commit:
62368fe1
SOF Branch: main
SOF Commit:
5374da8
FAIL: https://sof-ci.sh.intel.com/#/result/planresultdetail/5213?model=TGLU_RVP_NOCODEC&testcase=check-suspend-resume-100
TPLG=sof-tgl-nocodec.tplg ~/sof-test/test-case/check-suspend-resume.sh -l 100
Start Time: 2021-07-10 18:23:05 UTC
End Time: 2021-07-11 00:18:36 UTC
Kernel Branch: topic/sof-dev
Kernel Commit:
62368fe1
SOF Branch: main
SOF Commit:
5374da8
Same exact commits, same platform, different duration. Gah.
ranj063 commentedon Jul 12, 2021
@plbossart @fredoh9 I think this issue is related to #4419
There's a temporary workaround for it in #4466
plbossart commentedon Jul 12, 2021
@ranj063 we are not changing the firmware as described in #4419. What makes you think it might be related?
ranj063 commentedon Jul 12, 2021
The other issue can be reproduced even without changing the FW @plbossart. @slawblauciak confirmed it in one of the comments
slawblauciak commentedon Jul 13, 2021
As mentioned before, can you try this PR and see if it helps? #4466
ranj063 commentedon Jul 13, 2021
@fredoh9 would it be possible to check if #4466 helps with this issue?
keqiaozhang commentedon Sep 15, 2021
Same issue observed on ADLP-RVP-NOCODEC:
https://sof-ci.sh.intel.com/#/result/planresultdetail/6549?model=ADLP_RVP_NOCODEC&testcase=check-suspend-resume-with-capture-100
ranj063 commentedon Sep 15, 2021
@keqiaozhang are you testing with the same commit in SOF as the original issue? or are you testing with latest main?
lgirdwood commentedon Sep 20, 2021
@keqiaozhang are you able to do a FW stack trace when it next occurs. @keyonjie knows how to do this if you have any questions.
XiaoyunWu6666 commentedon Sep 29, 2021
The DSP panic reproduced again on TGLU_RVP_NOCODEC in inner daily 6914 (sorry I forgot to do a FW stack trace dump , but tried to reproduce it manually ... So the error in dmesg just went away)
Add 'options snd_sof sof_debug=0x41' in /etc/modprobe.d on the DUT [related issue comment #4538 ]
Let's see what this debug option will bring to us next time the DSP panic happened in daily test
@1994lwz will do similar trial on CML platforms which encountered DSP panic before also
lgirdwood commentedon Oct 12, 2021
@XiaoyunWu6666 @1994lwz any update ?
XiaoyunWu6666 commentedon Oct 13, 2021
have not seen it for 2 weeks