Repository navigation
[BUG] multiple-pause-resume.sh sof-test fails to FW_GEN_MSG failed with err 3018 #8792
Description
Describe the bug
The multiple-pause-resume.sh sof-test started to fail with latest Zephyr on Intel cAVS2.5 based platforms (TGL, ADL). First seen on
#8764
To Reproduce
Build SOF with recent Zephyr, run sof-test.
Reproduction Rate
50+%
Expected behavior
Chain-dma host dma stop fails
Impact
SOF CI PR tests are failing with high rate
Environment
- Branch name and commit hash of the 2 repositories: sof (firmware/topology) and linux (kernel driver).
See [DNM] Zephyr smp rework #8764 - Name of the topology file
- Topology: any Intel cavss2.5 tplg with HDMI (HDMI uses chain-dma) , see [DNM] Zephyr smp rework #8764 test results
- Name of the platform(s) on which the bug is observed.
- Platform: tgl, adl, rpl
Screenshots or console output
See https://sof-ci.01.org/sofpr/PR8764/build2195/devicetest/index.html?model=TGLU_UP_HDA-ipc4&testcase=multiple-pause-resume-50
[ 825.139933] <inf> ipc: ipc_cmd: rx : 0xe050000|0x0
[ 825.140968] <inf> chain_dma: chain_link_stop: comp:128 0x80 chain_link_stop(): dma_stop() link chan_index = 0
[ 825.142083] <inf> ll_schedule: zephyr_ll_task_done: task complete 0xbe0bf810 0x204b0U
[ 825.142095] <inf> ll_schedule: zephyr_ll_task_done: num_tasks 3 total_num_tasks 3
[ 825.142133] <err> ipc: ipc_cmd: ipc4: FW_GEN_MSG failed with err 3018
[ 825.142873] <inf> ipc: ipc_cmd: rx : 0xe040000|0x0
[ 825.142901] <inf> dma: dma_put: dma_put(), dma = 0x9e09f650, sref = 1
[ 825.142915] <inf> dma: dma_put: dma_put(), dma = 0x9e09f6f0, sref = 1
Activity
kv2019i commentedon Jan 25, 2024
zephyrproject-rtos/zephyr#68101 didn't work. 5000us is a REALLY long time to wait already and we still had fails, so something else goes wrong. On other platforms using chain-dma, this is not seeing, making this quite curious. Seems to be specific to cAVS2.5.
ASoC: SOF: ipc4: change trigger order for chain DMA
kv2019i commentedon Jan 26, 2024
I now suspect the DMA stop problem the Zephyr update uncovers, is actually a problem in our host DMA stop sequence. With current kernel and SOF, the order of stop seems wrong:
As far as I can tell, this is against the recommended programming flow. With lcoal stress tests thesofproject/linux#4798 seems to fix the issue (GBUSY no longer stuck), but more stress testing is needed.
FYI @jsarha @ujfalusi @plbossart @ranj063 @RanderWang
chain_dma: workaround for failed DMA stop on pause/stop
chain_dma: workaround for failed DMA stop on pause/stop
chain_dma: workaround for failed DMA stop on pause/stop
chain_dma: workaround for failed DMA stop on pause/stop
chain_dma: workaround for failed DMA stop on pause/stop
kv2019i commentedon Jan 30, 2024
Tried a combination of zephyrproject-rtos/zephyr#68304 and thesofproject/linux#4798 but still fails
chain_dma: workaround for failed DMA stop on pause/stop
kv2019i commentedon Jan 31, 2024
This kernel PR thesofproject/linux#4801 seem to help. One test plan (#37589) has passed, err 3018 not seen at all.
kv2019i commentedon Feb 1, 2024
A test PR with a Zephyr-side fix for the issue -> #8826
kv2019i commentedon Mar 6, 2024
Issue closed with zephyrproject-rtos/zephyr#68415 merged via #8903 today.
drivers: dma: intel-adsp-hda: increase stop timeout to 5000us