Repository navigation
[CML] [BUG] "suspend-resume-with-capture" test failed with kernel errors #2761
Description
A weekly test failed on Mantis with:
[ 7390.700426] PM: suspend entry (s2idle)
[ 7390.701376] Filesystems sync: 0.000 seconds
[ 7390.702276] Freezing user space processes ... (elapsed 0.004 seconds) done.
[ 7390.707111] OOM killer disabled.
[ 7390.707117] Freezing remaining freezable tasks ... (elapsed 0.001 seconds) done.
[ 7390.708851] printk: Suspending console(s) (use no_console_suspend to debug)
[ 7390.710935] [124064] sof-audio-pci 0000:00:1f.3: pcm: trigger stream 7 dir 1 cmd 5
[ 7390.710958] [124064] sof-audio-pci 0000:00:1f.3: ipc tx: 0x60050000: GLB_STREAM_MSG: TRIG_STOP
[ 7390.711200] [124064] sof-audio-pci 0000:00:1f.3: ipc tx succeeded: 0x60050000: GLB_STREAM_MSG: TRIG_STOP
[ 7390.711973] [124064] sof-audio-pci 0000:00:1f.3: FW Poll Status: reg=0x240002 timedout
[ 7390.711980] sof-audio-pci 0000:00:1f.3: error: hda_dsp_stream_trigger: cmd 5: timeout on STREAM_SD_OFFSET read
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[ 7390.711993] [124064] sof-audio-pci 0000:00:1f.3: ipc tx: 0x60030000: GLB_STREAM_MSG: PCM_FREE
[ 7390.712218] [124064] sof-audio-pci 0000:00:1f.3: ipc tx succeeded: 0x60030000: GLB_STREAM_MSG: PCM_FREE
[ 7390.712854] [124064] sof-audio-pci 0000:00:1f.3: FW Poll Status: reg=0x20140000 successful
[ 7390.712882] [124064] sof-audio-pci 0000:00:1f.3: ipc tx: 0x40010000: GLB_PM_MSG: CTX_SAVE
[ 7390.713092] [124064] sof-audio-pci 0000:00:1f.3: ipc tx succeeded: 0x40010000: GLB_PM_MSG: CTX_SAVE
after which the DSP failed to boot:
[ 7397.314533] [124064] sof-audio-pci 0000:00:1f.3: booting DSP firmware
[ 7397.314559] [124064] sof-audio-pci 0000:00:1f.3: FW Poll Status: reg=0x40000 successful
[ 7397.314573] [124064] sof-audio-pci 0000:00:1f.3: FW Poll Status: reg=0x40000 successful
[ 7397.314575] [124064] sof-audio-pci 0000:00:1f.3: period_bytes:0x0
[ 7397.314576] [124064] sof-audio-pci 0000:00:1f.3: periods:1
[ 7397.314606] [124064] sof-audio-pci 0000:00:1f.3: Attempting iteration 0 of Core En/ROM load...
[ 7397.314608] [124064] sof-audio-pci 0000:00:1f.3: DSP core(s) enabled? 0 : core_mask f
[ 7397.314612] [124064] sof-audio-pci 0000:00:1f.3: FW Poll Status: reg=0xf0f0f0f successful
[ 7397.314616] [124064] sof-audio-pci 0000:00:1f.3: FW Poll Status: reg=0xf0f0f00 successful
[ 7397.314618] [124064] sof-audio-pci 0000:00:1f.3: unstall/run core: core_mask = f
[ 7397.314621] [124064] sof-audio-pci 0000:00:1f.3: DSP core(s) enabled? 1 : core_mask f
[ 7397.314626] [124064] sof-audio-pci 0000:00:1f.3: FW Poll Status: reg=0xf0f0000 successful
[ 7397.314628] [124064] sof-audio-pci 0000:00:1f.3: unstall/run core: core_mask = 1
[ 7397.314631] [124064] sof-audio-pci 0000:00:1f.3: DSP core(s) enabled? 1 : core_mask 1
[ 7397.315163] [124064] sof-audio-pci 0000:00:1f.3: FW Poll Status: reg=0x80000000 successful
[ 7397.315184] [124064] sof-audio-pci 0000:00:1f.3: FW Poll Status: reg=0xf0f0e0e successful
[ 7397.315200] [124064] sof-audio-pci 0000:00:1f.3: FW Poll Status: reg=0x1010e0e successful
[ 7397.315212] [124064] sof-audio-pci 0000:00:1f.3: DSP core(s) enabled? 0 : core_mask e
[ 7397.384032] [124064] sof-audio-pci 0000:00:1f.3: FW Poll Status: reg=0x5000001 successful
[ 7400.381741] [124064] sof-audio-pci 0000:00:1f.3: FW Poll Status: reg=0x5000001 timedout
[ 7400.381750] sof-audio-pci 0000:00:1f.3: error: cl_copy_fw: timeout HDA_DSP_SRAM_REG_ROM_STATUS read
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[ 7400.382307] [124064] sof-audio-pci 0000:00:1f.3: FW Poll Status: reg=0x140002 timedout
[ 7400.382314] sof-audio-pci 0000:00:1f.3: error: hda_dsp_stream_trigger: cmd 0: timeout on STREAM_SD_OFFSET read
[ 7400.382320] sof-audio-pci 0000:00:1f.3: error: DMA trigger stop failed
[ 7400.382327] [124064] sof-audio-pci 0000:00:1f.3: unknown ROM status value 05000001
[ 7400.382335] sof-audio-pci 0000:00:1f.3: error: status = 0x00000000 panic = 0x00000000
[ 7400.382352] sof-audio-pci 0000:00:1f.3: error: extended rom status: 0x5000001 0x0 0x0 0x0 0x0 0x0 0x1811102 0x0
[ 7400.382359] [124064] sof-audio-pci 0000:00:1f.3: unknown ROM status value 05000001
[ 7400.382365] sof-audio-pci 0000:00:1f.3: error: load fw failed ret: -110
[ 7400.382396] sof-audio-pci 0000:00:1f.3: error: failed to reset DSP
[ 7400.382401] sof-audio-pci 0000:00:1f.3: error: failed to boot DSP firmware after resume -110
[ 7400.382406] PM: dpm_run_callback(): pci_pm_resume+0x0/0xd0 returns -110
[ 7400.382425] PM: Device 0000:00:1f.3 failed to resume: error -110
Attaching the firmware log but I haven't found anything obviously wrong there
slogger.txt
Activity
[-][CNL] [BUG] "suspend-resume-with-capture" test failed with kernel errors[/-][+][CML] [BUG] "suspend-resume-with-capture" test failed with kernel errors[/+]keqiaozhang commentedon Mar 1, 2021
I didn't observe this issue in daily test.
@lyakh , what is the reproduction rate?
lyakh commentedon Mar 8, 2021
@keqiaozhang I've seen it once, a later weekly test, run on 26.02 didn't reproduce it.
keyonjie commentedon Mar 15, 2021
could this be related with the capture DMI_L1 enable? related with #2781?
RanderWang commentedon Mar 15, 2021
is it related with #705. I checked with our partner`s driver and found this solution was only applied to APL
lyakh commentedon Mar 17, 2021
The same bug triggered again last Friday, kernel hash 578778f ("ASoC: SOF: Intel: TGL: fix EHL ops") firmware fa35ee0 ("kpb: fix type for force_copy_type")
libinyang commentedon Mar 18, 2021
@lyakh Could you attach your kernel config?
lyakh commentedon Mar 19, 2021
@libinyang attached. However, I must confess: I just realised, that building the firmware failed last Friday because of the changed cmake command line... So, although the failure was real, it was testing a fresh kernel of March 12th, but the firmware was of February 26th.
linux.config.txt
plbossart commentedon Mar 19, 2021
@lyakh what cmake change are you referring to. I haven't changed anything on my side to wondering if I am also testing obsolete firmware...
lyakh commentedon Mar 19, 2021
@plbossart I'm talking about thesofproject/sof@288e228 but if you're using the standard scripts, e.g. xtensa-build-all.sh then you should be fine...
lyakh commentedon Mar 19, 2021
Test complete: no reproduction.
keyonjie commentedon Mar 22, 2021
@lyakh so do you think we can close it now?
libinyang commentedon Mar 22, 2021
The config is OK. So the bug is not the DMI_L1 issue.
lyakh commentedon Mar 22, 2021
@keyonjie but that should work too, shouldn't it - newest kernel with a 3-week old firmware? Maybe that's what we should test to see if the bug can be reproduced that way?
plbossart commentedon Apr 20, 2021
@lyakh I would suggest we close this bug if no one has been able to reproduce it?
keyonjie commentedon May 10, 2021
@lyakh can you close it and reopen if needed in future?
keqiaozhang commentedon May 17, 2021
close this issue. @lyakh please reopen it if needed.
lyakh commentedon May 17, 2021
@plbossart @keyonjie @keqiaozhang sorry, missed the comments, thanks for closing