Repository navigation
[BUG] timeout on STREAM_SD_OFFSET read when suspend-resume on CML Helios #4558
Copy link
Copy link
Closed
Labels
CMLApplies to Comet Lake platformApplies to Comet Lake platformI2SApplies to I2S bus for codec connectionApplies to I2S bus for codec connectionP3Low-impact bugs or featuresLow-impact bugs or featuresbugSomething isn't working as expectedSomething isn't working as expected
Milestone
Description
Describe the bug
timeout on STREAM_SD_OFFSET read when suspend-resume on CML
inner daily /5515?model=CML_HEL_RT5682&testcase=check-suspend-resume-with-capture-5
To Reproduce
TPLG=/lib/firmware/intel/sof-tplg/sof-cml-rt1011-rt5682.tplg \
sof-test/test-case/check-suspend-resume-with-audio.sh -S 1 -w 3 -l 300 -m capture
-S 1 -w 3 save some time while preserving an acceptable ssh experience. For faster reproduction on a local system try -S 1 -w 1?
-m capture is critical: -m playback never reproduced even after 100 iterations.
Environment
Start Time: 2021-07-26 21:24:47 UTC
End Time: 2021-07-27 04:58:29 UTC
Kernel Branch: topic/sof-dev
Kernel Commit: 2113dc7f
SOF Branch: main
SOF Commit: 57ee04f
Topology:sof-cml-rt1011-rt5682.tplg
Platform:CML_HEL_RT5682
Screenshots or console output
[console]
2021-07-26 23:04:00 UTC Sub-Test: [REMOTE_INFO] ===== Round(2/5) =====
2021-07-26 23:04:00 UTC Sub-Test: [REMOTE_COMMAND] Run the command: rtcwake -m mem -s 5
rtcwake: assuming RTC uses UTC ...
rtcwake: wakeup from "mem" using /dev/rtc0 at Mon Jul 26 23:04:07 2021
2021-07-26 23:04:10 UTC Sub-Test: [REMOTE_COMMAND] sleep for 5
arecord: suspend:1691: suspend: prepare error: Connection timed out
2021-07-26 23:04:15 UTC Sub-Test: [REMOTE_INFO] Check for the kernel log status
declare -- cmd="journalctl_cmd --since=@1627340635"
2021-07-26 23:04:15 UTC [ERROR] Caught kernel log error
===========================>>
[ 5378.823476] kernel: sof-audio-pci-intel-cnl 0000:00:1f.3: error: hda_dsp_stream_trigger: cmd 5: timeout on STREAM_SD_OFFSET read
[ 5382.615214] kernel: sof-audio-pci-intel-cnl 0000:00:1f.3: error: cl_copy_fw: timeout HDA_DSP_SRAM_REG_ROM_STATUS read
[ 5382.616259] kernel: sof-audio-pci-intel-cnl 0000:00:1f.3: error: hda_dsp_stream_trigger: cmd 0: timeout on STREAM_SD_OFFSET read
[ 5382.616266] kernel: sof-audio-pci-intel-cnl 0000:00:1f.3: error: DMA trigger stop failed
[ 5382.616272] kernel: sof-audio-pci-intel-cnl 0000:00:1f.3: ------------[ DSP dump start ]------------
[ 5382.616305] kernel: sof-audio-pci-intel-cnl 0000:00:1f.3: error: extended rom status: 0x5000001 0x0 0x0 0x0 0x0 0x0 0x1811102 0x0
[ 5382.616311] kernel: sof-audio-pci-intel-cnl 0000:00:1f.3: ------------[ DSP dump end ]------------
[ 5382.616315] kernel: sof-audio-pci-intel-cnl 0000:00:1f.3: error: load fw failed ret: -110
[ 5382.616373] kernel: sof-audio-pci-intel-cnl 0000:00:1f.3: error: failed to start DSP
[ 5382.616378] kernel: sof-audio-pci-intel-cnl 0000:00:1f.3: error: failed to boot DSP firmware after resume -110
[ 5382.616382] kernel: PM: dpm_run_callback(): pci_pm_resume+0x0/0x80 returns -110
[ 5382.616513] kernel: sof-audio-pci-intel-cnl 0000:00:1f.3: PM: failed to resume async: error -110
[ 5383.132839] kernel: sof-audio-pci-intel-cnl 0000:00:1f.3: error: ipc timed out for 0x30100000 size 48
[ 5383.132909] kernel: sof-audio-pci-intel-cnl 0000:00:1f.3: ------------[ IPC dump start ]------------
[ 5383.132967] kernel: sof-audio-pci-intel-cnl 0000:00:1f.3: error: hda irq intsts 0x00000000 intlctl 0x00001388 rirb 00
[ 5383.133019] kernel: sof-audio-pci-intel-cnl 0000:00:1f.3: error: dsp irq ppsts 0x00000000 adspis 0x00000003
[ 5383.133077] kernel: sof-audio-pci-intel-cnl 0000:00:1f.3: error: host status 0x00000000 dsp status 0x00040000 mask 0x00000000
[ 5383.133131] kernel: sof-audio-pci-intel-cnl 0000:00:1f.3: ------------[ IPC dump end ]------------
[ 5383.133175] kernel: sof-audio-pci-intel-cnl 0000:00:1f.3: error: load pipeline ipc failure
[ 5383.133216] kernel: sof-audio-pci-intel-cnl 0000:00:1f.3: error: failed to load widget PIPELINE.11.DMIC1.IN
[ 5383.133264] kernel: sof-audio-pci-intel-cnl 0000:00:1f.3: error: failed widget list set up for pcm 7 dir 1
[ 5383.133313] kernel: sof-audio-pci-intel-cnl 0000:00:1f.3: error: set pcm hw_params after resume
[ 5383.133356] kernel: sof-audio-pci-intel-cnl 0000:00:1f.3: ASoC: error at snd_soc_pcm_component_prepare on 0000:00:1f.3: -110
[ 5383.133496] kernel: DMIC16kHz: ASoC: soc_pcm_prepare() failed (-110)
[ 5383.133535] kernel: DMIC16kHz: ASoC: dpcm_fe_dai_prepare() failed (-110)
<<===========================
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
CMLApplies to Comet Lake platformApplies to Comet Lake platformI2SApplies to I2S bus for codec connectionApplies to I2S bus for codec connectionP3Low-impact bugs or featuresLow-impact bugs or featuresbugSomething isn't working as expectedSomething isn't working as expected
Activity
[-][BUG] timeout on STREAM_SD_OFFSET read when suspend-resume on CML[/-][+][BUG] timeout on STREAM_SD_OFFSET read when suspend-resume on CML Helios[/+]plbossart commentedon Jul 27, 2021
@slawblauciak what makes you think it's a kernel issue? There's nothing that changed recently.
@lgirdwood Can you please look into this?
plbossart commentedon Jul 27, 2021
@XiaoyunWu6666 The same problem affects other CML devices, it's not just CML_HEL_RT5682
http://sof-ci.sh.intel.com/#/result/planresultdetail/5515?model=CML_SKU0983_SDW&testcase=check-suspend-resume-with-capture-5
http://sof-ci.sh.intel.com/#/result/planresultdetail/5515?model=CML_SKU0955_HDA&testcase=check-suspend-resume-with-capture-5
It's an across-the-board problem for all CML devices using I2S/HDAudio or SoundWire links. That's very very suspicious.
plbossart commentedon Jul 27, 2021
Reproduced in 10 iterations on CML SKU 09C6 (same as 0983)
kernel SHA1: 85bfa34a88d6
SOF SHA1: 9a7a5ce (HEAD -> main, origin/main) pipeline: trigger START from the task context
plbossart commentedon Jul 27, 2021
plbossart commentedon Jul 27, 2021
@XiaoyunWu6666 @marc-hb @fredoh9 @bardliao I would appreciate it if you can try to reproduce the issue on a CML device. My results show that there is a firmware regression since I don't see any issue with the latest kernel/older firmware.
More details for the record - @lgirdwood FYI
test 5515 on July 27 - BAD
Kernel Commit: 2113dc7f
SOF Commit: 57ee04f
test 5464 July 26 GOOD
Kernel Commit: d54b7803
SOF Commit: 9648fb6
so basically I bisected from yesterday's tested SOF commit
good: [9648fb6] alloc: fix heap_trace() block start index
until today's HEAD
bad: [9a7a5ce] pipeline: trigger START from the task context
lgirdwood commentedon Jul 27, 2021
@plbossart thanks for bisect , can you confirm if the trace patch or the pipeline patch is bad ?
plbossart commentedon Jul 27, 2021
I don't understand the question @lgirdwood. My tests show what the first bad commit was, I don't know if the last patches on pipelines are correct or not. It could also be that my 'good' tests are 'green failures', and the issue happens with an earlier commit.
lgirdwood commentedon Jul 27, 2021
Sorry, was not clear. I mean is the bisect clear cut or vague - I agree though this is a FW issue.
OR
I guess what you are saying it could be either above two commits (and needs further investigation) ?
plbossart commentedon Jul 27, 2021
no, I am saying: the problem started with 0def905 or earlier. I don't have a 100% confidence in the 'good' results, but I trust the tests when they tell me the DSP boot failed.
lgirdwood commentedon Jul 27, 2021
@XiaoyunWu6666 are you able to do some further investigation.
OR redo the bisect with 20 iterations (based on @plbossart being able to reproduce with 10). Thanks !
@marc-hb @lyakh fyi
plbossart commentedon Jul 27, 2021
Note that sometimes 10-20 iterations is not enough. I just ran a check which didn't fail in 50, and my last took 36 with the SOF main branch
plbossart commentedon Jul 27, 2021
With these two commands, I was able to run 100 iterations without issues.
test command:
42 remaining items
XiaoyunWu6666 commentedon Aug 2, 2021
Close it now since relevant feature was reverted in #4578
Since this issue takes only several rounds to reproduce in testcase check-suspend-resume-with-capture-5 , we'll test how long it will take additionally to add this case in PR test.
If the extra time is acceptable ,we can track how it goes in PR test.
bardliao commentedon Aug 3, 2021
BTW, I can still reproduce the issue with #4578 merged in GCC build. That could be a different issue.
XiaoyunWu6666 commentedon Sep 7, 2021
happen in inner daily 6362
on model CML_RVP_SDW
in testcase check-suspend-resume-with-capture-5
lgirdwood commentedon Sep 7, 2021
@XiaoyunWu6666 what's the reproducability now and is this fatal ?
XiaoyunWu6666 commentedon Sep 9, 2021
@lgirdwood
The reproduction rate is rather low , see issue summary below
Concerning the number of testcases that this issue affects , it is not that fatal (about one each time). But its appearance causes the testcase to fail
lgirdwood commentedon Sep 9, 2021
@XiaoyunWu6666 thanks for the update.
@plbossart fwiw, there is a speculative HW channel for the DMA that switched ON/OFF to save power. @kv2019i and I were discussing this week and we could have a race between the DMA channel PM switching during the driver/FW DMA/stream shutdown logic. I wonder if there is a status bit for this that could be polled as part of the shutdown logic......
marc-hb commentedon Sep 10, 2021
The failure in 6362 looks different to me:
This could mean this issue hasn't happened since revert #4578
ranj063 commentedon Sep 18, 2021
@XiaoyunWu6666 @1994lwz could you please check if thesofproject/linux#3166 helps fix this issue?