Repository navigation
[BUG] [Stress]IPC timed out after resuming form S3 on WHL-I2S #1342
Description
Describe the bug
This issue happens in stress test. IPC timed out occurred after dozens of S3 and trace logger is not available after this.
To Reproduce
$ while rtcwake -m mem -s 5 ; do sleep 2 ; done
Expected behavior
no errors during S3
Impact
IPC times out
Environment
- Kernel: sof-dev/af71673
- SOF/tplg: master/be1331
- tplg name: sof-whl-rt5682.tplg
- WHL-RVP, I2S mode
- Reproducibility Rate: need more tests.
dmesg
[ 323.559329] sof-audio-pci 0000:00:1f.3: ipc tx succeeded: 0x30130000: GLB_TPLG_MSG: PIPE_COMPLETE
[ 323.559336] sof-audio-pci 0000:00:1f.3: tplg: complete pipeline PIPELINE.1.SSP1.OUT id 5
[ 323.559342] sof-audio-pci 0000:00:1f.3: ipc tx: 0x30130000: GLB_TPLG_MSG: PIPE_COMPLETE
[ 323.559478] sof-audio-pci 0000:00:1f.3: ipc tx succeeded: 0x30130000: GLB_TPLG_MSG: PIPE_COMPLETE
[ 323.559493] sof-audio-pci 0000:00:1f.3: ipc tx: 0x50010000: GLB_COMP_MSG: SET_VALUE: SET_VALUE
[ 323.559741] sof-audio-pci 0000:00:1f.3: ipc tx succeeded: 0x50010000: GLB_COMP_MSG: SET_VALUE: SET_VALUE
[ 323.559753] sof-audio-pci 0000:00:1f.3: ipc tx: 0x50010000: GLB_COMP_MSG: SET_VALUE: SET_VALUE
[ 323.559979] sof-audio-pci 0000:00:1f.3: ipc tx succeeded: 0x50010000: GLB_COMP_MSG: SET_VALUE: SET_VALUE
[ 323.559991] sof-audio-pci 0000:00:1f.3: ipc tx: 0x50010000: GLB_COMP_MSG: SET_VALUE: SET_VALUE
[ 323.860405] sof-audio-pci 0000:00:1f.3: error: ipc timed out for 0x50010000 size 108
[ 323.860417] sof-audio-pci 0000:00:1f.3: status: fw entered - code 00000005
[ 323.860710] sof-audio-pci 0000:00:1f.3: error: unexpected fault 0x00000000 trace 0x00004000
[ 323.860713] sof-audio-pci 0000:00:1f.3: error: waking up any trace sleepers
[ 323.860718] sof-audio-pci 0000:00:1f.3: error: set/get ctrl ipc comp 17
[ 323.860721] sof-audio-pci 0000:00:1f.3: error: failed kcontrol value set for widget: 17
[ 323.860724] sof-audio-pci 0000:00:1f.3: error: restoring kcontrols after resume
[ 323.860727] sof-audio-pci 0000:00:1f.3: error: failed to restore pipeline after resume -110
[ 323.860752] sof-audio-pci 0000:00:1f.3: ipc tx: 0x40010000: GLB_PM_MSG: CTX_SAVE
[ 324.059708] sof-audio-pci 0000:00:1f.3: ipc rx: 0x90020000: GLB_TRACE_MSG
[ 324.059740] sof-audio-pci 0000:00:1f.3: ipc rx done: 0x90020000: GLB_TRACE_MSG
[ 324.164417] sof-audio-pci 0000:00:1f.3: error: ipc timed out for 0x40010000 size 76
[ 324.164430] sof-audio-pci 0000:00:1f.3: status: fw entered - code 00000005
[ 324.164701] sof-audio-pci 0000:00:1f.3: error: unexpected fault 0x00000000 trace 0x00004000
[ 324.164705] sof-audio-pci 0000:00:1f.3: error: ctx_save ipc error during suspend -110
[ 324.164719] pci_pm_suspend(): snd_sof_suspend+0x0/0x10 [snd_sof] returns -110
[ 324.164727] dpm_run_callback(): pci_pm_suspend+0x0/0x120 returns -110
[ 324.164732] PM: Device 0000:00:1f.3 failed to suspend: error -110
[ 324.164736] PM: Some devices failed to suspend, or early wake event detected
[ 324.164810] HDMI HDA Codec ehdaudio0D2: hdac_hdmi_present_sense: disconnect for pin:port 5:0
[ 324.164817] HDMI HDA Codec ehdaudio0D2: hdac_hdmi_present_sense: disconnect for pin:port 7:0
[ 324.402411] acpi LNXPOWER:08: Turning OFF
Activity
mengdonglin commentedon Apr 28, 2019
@stevyan please check if this is the same issue you observed on UP2 in stress S3 test
stevyan commentedon Apr 28, 2019
@mengdonglin @keqiaozhang , according to the log info, it seems be same issue with thesofproject/linux#876.
bkokoszx commentedon May 7, 2019
@keqiaozhang
Do you still have a reproduction of this issue? I performed d0->d3 python tests on WHL platform and I did not have reproduction.
Does the driver for sure not try to read from mailbox after sending CTX_SAVE? Otherwise driver can try to read from disabled memory.
keqiaozhang commentedon May 9, 2019
@bkokoszx
Yes, this issue is still reproducible to me, I tested it with today's code base and failed after dozens of iterations.
Please do the system S3->S0 test. D0/D3 is also working for me.
lgirdwood commentedon May 9, 2019
@bkokoszx python test is great for functional testing but should not be used for integration stress testing, it's just a waste of time. You are not comparing like for like. Use the driver and reproduce. How else to you expect to root cause ?
lgirdwood commentedon May 27, 2019
@mengdonglin any update on this, is it a duplicate ? Looks like this belongs to topology load and init after S3 category.
keqiaozhang commentedon Jun 3, 2019
The failure is changed to system hang with latest code base and no kmsg and trace available.
Test recipe:
kernel: sof-dev/0a747dc
firmeware: master/8901e0
Platforme: WHL-I2S
TPLG: sof-cml-rt5682.tplg
wenqingfu commentedon Jun 19, 2019
not reproducible after thesofproject/linux#1013 et al.