Repository navigation
[BUG] IPC flood test caused IPC error on TGLU_RVP_NOCODEC #2844
Description
Describe the bug
CI detected this issue on TGL_RVP_NOCODEC device when testing ipc flood. Looks like this is a kernel regression, will try to bisect it next.
To Reproduce
$ /sof-test/test-case/check-ipc-flood.sh
Reproduction Rate
100%
Environment
Kernel Branch: topic/sof-dev
Kernel Commit: f8e3b2b
SOF Branch: main
SOF Commit: 095d1d4
Platform: tgl_rvp nocodec mode
tplg:sof-tgl-nocodec-ci.tplg
Test report ID: 3264
kernel log
kernel: [ 92.787595] sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx: 0x30100000: GLB_TPLG_MSG: PIPE_NEW
kernel: [ 92.787759] sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx succeeded: 0x30100000: GLB_TPLG_MSG: PIPE_NEW
kernel: [ 92.787763] sof-audio-pci-intel-tgl 0000:00:1f.3: widget PIPELINE.4.SSP1.IN setup complete
kernel: [ 92.787781] sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx: 0x30010000: GLB_TPLG_MSG: COMP_NEW
kernel: [ 92.787890] sof-audio-pci-intel-tgl 0000:00:1f.3: error: ipc error for 0x30010000 size 20
kernel: [ 92.787898] sof-audio-pci-intel-tgl 0000:00:1f.3: error: failed to load widget PCM2P
kernel: [ 92.787903] sof-audio-pci-intel-tgl 0000:00:1f.3: error: failed to restore pipeline after resume -13
kernel: [ 92.787993] sof-audio-pci-intel-tgl 0000:00:1f.3: error: debugfs write failed to resume -5
kernel: [ 93.280592] sof-audio-pci-intel-tgl 0000:00:1f.3: ipc rx: 0x90020000: GLB_TRACE_MSG
kernel: [ 93.280637] sof-audio-pci-intel-tgl 0000:00:1f.3: ipc rx done: 0x90020000: GLB_TRACE_MSG
Activity
plbossart commentedon Apr 15, 2021
@keyonjie didn't you change something in the nocodec topology?
[-][BUG] IPC flood test caused IPC error on TGL_RVP_NOCODEC [/-][+][BUG] IPC flood test caused IPC error on TGLU_RVP_NOCODEC [/+]XiaoyunWu6666 commentedon Apr 20, 2021
Looks like caused by tplg , I'll do bisecting this morning
XiaoyunWu6666 commentedon Apr 20, 2021
Confirmed that https://github.com/thesofproject/sof/commit/cd2444adbf8821d33e098b071b40dab2931d6a8e is the commit that causes this issue.
[test on ubuntu@sh-tglu-rvp-nocodec-02]
Commit Details
commit cd2444a
Author: Keyon Jie yang.jie@linux.intel.com
Date: Tue Apr 13 16:27:35 2021 +0800
keyonjie commentedon Apr 20, 2021
yes, I changed to use core one on the TGL nocodec platform, so this issue is under expected, working on this these days.
plbossart commentedon Apr 20, 2021
@keyonjie it's not a good practice to break CI while you are working on a fix. You need to use separate commits for topology/firmware and driver, and only ask that both are committed when the solution actually works.
@lgirdwood please consider reverting the commit above until we have a clear and agreed on solution.
keyonjie commentedon Apr 21, 2021
@plbossart to this multi-core support specific, we committed and did release that multi-core had been supported from half a year back(tgl-010), it is basic feature and should be verified at least daily, but since we are lack of CI coverage, regression was introduced after that.
IMHO, bug drives the development sometimes, if we don't prioritize it, we could keep wandering there for some more months.
plbossart commentedon Apr 21, 2021
NAK. This impact CI and the work in your team. CI results shall pass, or everyone will stop looking at the results.
If you detected a bug somewhere that can only be exposed by a change, then fix whatever this change exposes BEFORE committing this change.
We have enough surprises already with CI, the last thing we need is a developer trying to prove a point. If you do, then mark your PR as [TEST] and don't ask for it to be merged.
marc-hb commentedon Apr 21, 2021
You can use pull requests or any other branch than main to share work in progress and have tests run on it. Don't dump it on the main branch, negatively affect others' work and slow everyone else down. Others have priorities too and I'm tired of having to look at problems that are completely unrelated to what I'm supposed to work on (assuming we're lucky and they fail in our too limited test coverage, not just locally in some obscure corner case)
Yes, obviously. Everyone can make mistakes: then immediate revert to stop affecting others, test harder and submit again. Life goes on.
keyonjie commentedon Apr 22, 2021
@plbossart @marc-hb the sof-tgl-nocodec-ci.m4 was added with intention of experiment and development only, but yes, let me submit a reversion if that really impact you, considering the multi-core support won't be merged in several working days.
Here it is: thesofproject/sof#4075
fredoh9 commentedon Apr 26, 2021
thesofproject/sof#4075 merged, unblocked
marc-hb commentedon Apr 27, 2021
Test failures impact every single developer because every single developer looks at test results all the time to make sure they did not break anything recently. If some feature or configuration stops working for some random reason then no one can tell whether they just broke it because it's already broken.