Repository navigation
Suspend stress test failure due to IPC timeouts on v0.6 #370
Description
Chrome PV target for S0ix is 2500 cycles and the stress test fails on multiple boards due to IPC timeouts.
SoF v0.6 OS Recipe:
Flash the latest ww48 BKC OS and Coreboot image(only) from path:
ChromeOS-test-R72-11284 or later OS and Coreboot
• Remove the rootfs verification (/usr/share/vboot/bin/make_dev_ssd.sh --remove_rootfs_verification --partition 2)
-
Kernel change - checkout to CL:1352860 to get all the v0.6 patches, compile & update kernel
OR update from pre-built kernel using steps:
~/trunk/src/scripts $ download_kernel_image.sh octopus
password: auto
~/trunk/src/scripts $ update_pre_build_kernel.sh --remote= octopus -
Firmware change
DSP FW Copy the sof-glk.ri-1f79920 to /lib/firmware/intel/sof-glk.ri
SOF FW source: Commit: 1f799207f4d7eb7108d5e5602a4a4c15ed9c49f6
Md5sum: cd5bdaf4bca100f6186e4e2d93a4a352 /lib/firmware/intel/sof-glk.ri
Topology FW copy the sof-glk-da7219.tplg-8cc1dad to /lib/firmware/intel/sof-glk-da7219.tplg
Topology FW source: Commit: : 8cc1dad
Md5sum: 74dcc70 /lib/firmware/intel/sof-glk-da7219.tplg
How to validate:
Description: Run 2500 consecutive suspend resume cycles on all SKU.
Duration: 24 hours
Setup Steps:
- Check device mode is set to normal mode.
- Reinstall OS to ensure you have a fresh OS and all logs are cleared.
- Connect AC power.
- Remove all external USB devices.
- Connect to WiFi network.
Test Steps:
- Open browse as Guest in VT1
- Press Ctrl+Alt+T to start a crosh
- Type shell to launch a shell
- Start the test with
sudo suspend_stress_test -c 2500 | tee /home/root/suspend.log
Attached logs.
DUT1:
[ 2858.019887] sof-audio sof-audio: error: ipc timed out for 0x80010000 size 0x48
[ 2858.019925] sof-audio sof-audio: status: fw entered - code 00000005
[ 2858.020113] sof-audio sof-audio: error: unexpected fault 0x00000000 trace 0x00004000
[ 2858.020114] sof-audio sof-audio: error: waking up any trace sleepers
[ 2858.020123] sof-audio sof-audio: error: failed to set dai config for iDisp3
[ 2858.020125] sof-audio sof-audio: error: failed to restore pipeline after resume -110
[ 2858.020131] dpm_run_callback(): pci_pm_resume+0x0/0xbe returns -110
[ 2858.020134] call 0000:00:0e.0+ returned -110 after 409461 usecs
[ 2858.020144] PM: Device 0000:00:0e.0 failed to resume: error -110
DUT2
[ 4599.345157] sof-audio sof-audio: error: waiting for HIPCIE done, reg: 0x0
[ 4599.345172] sof-audio sof-audio: unknown ROM status value 80000000
[ 4599.345176] sof-audio sof-audio: error: status = 0xecc00301 panic = 0x00000000
[ 4599.345179] sof-audio sof-audio: unknown ROM status value 80000000
[ 4599.367322] sof-audio sof-audio: DSP core(s) enabled? 0 : core_mask 3
[ 4599.367332] sof-audio sof-audio: error: could not get stream with stream tag0
[ 4599.367334] sof-audio sof-audio: error: load fw failed err: -19
[ 4599.367350] sof-audio sof-audio: unknown ROM status value ffffffff
[ 4599.367366] sof-audio sof-audio: error: status = 0xffffffff panic = 0xffffffff
[ 4599.367381] sof-audio sof-audio: unknown ROM status value ffffffff
[ 4599.367397] sof-audio sof-audio: error: load fw failed err: -19
[ 4599.367399] sof-audio sof-audio: error: failed to reset DSP
[ 4599.367402] sof-audio sof-audio: error: failed to boot DSP firmware after resume -19
S0ix_failure.zip
Activity
[-]GLK: BOBBA: SOF: Stress: S0ix fail to meet PV target(2500 cycles) w/o HDMI connected[/-][+]Stress: S0ix fail to meet PV target(2500 cycles) w/o HDMI connected[/+]ranj063 commentedon Dec 6, 2018
#373 didnt help solve the problem
[-]Stress: S0ix fail to meet PV target(2500 cycles) w/o HDMI connected[/-][+]Suspend stress test failure due to IPC timeouts[/+][-]Suspend stress test failure due to IPC timeouts[/-][+]Suspend stress test failure due to IPC timeouts on v0.6[/+]jocelyn-li commentedon Dec 7, 2018
@bardliao ran the suspend_stress_test overnight on a Bobba DUT, has passed 3725+ iterations and the test is still going.
@keyonjie @xiulipan Any ideas about this?
xiulipan commentedon Dec 11, 2018
test_13_pass.log
test_12_pass.log
Reproduced on an UP2 borad with more debug message.
Please see 12 as a success log, and 13 as a fail log.
Note the
ipc rx: 0x90020000: GLB_TRACE_MSGfrom DSP when we loading the FW and first[ 2291.436852] PM: Some devices failed to suspend, or early wake event detectedranj063 commentedon Dec 12, 2018
This issue has been root caused to a race condition caused by the timing of when the host gets a done interrupt and when the ipc task state in the DSP is set to "completed". A workaround to set the task state to completed before notifying the host works to pass 2500 iterations on 6 boards so far.
Work in progress to finalize the fix
mengdonglin commentedon Dec 31, 2018
Random firmware loading failure is observed in the stress test thesofproject/sof#785
ranj063 commentedon Jan 11, 2019
@mengdonglin @naveen-manohar can we close this bug now that we have proved there are no issues with stress test?
mengdonglin commentedon Jan 14, 2019
Closing this issue since we can no longer reproduce it no matter with SOF V0.55 or V1.0 release for GLK with all recent fixes.
ext4: fix use-after-free in ext4_orphan_cleanup
ext4: fix use-after-free in ext4_orphan_cleanup
Merge branch 'arm32-bpf-add-support-for-cpuv4-insns'