You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[SoundWire] intel_resume: MCP_CONTROL_HW_RST not cleared at iteration (dev_err logs) #3012
description
sdw_cdns_clock_stop failed: MCP_CONTROL_HW_RST is not cleared when check-suspend-resume-50 on CML_SKU0983_SDW
happen in inner daily 4778 model=CML_SKU0983_SDW testcase=check-suspend-resume-50
[console log]
[ 4876.595463] kernel: soundwire_intel soundwire_intel.link.0: intel_resume: MCP_CONTROL_HW_RST is not cleared at iteration 0
……
[ 4876.608536] kernel: soundwire_intel soundwire_intel.link.0: intel_resume: MCP_CONTROL_HW_RST is not cleared at iteration 9
[ 4876.610117] kernel: soundwire_intel soundwire_intel.link.0: intel_resume failed: MCP_CONTROL_HW_RST is not cleared
[ 4880.767350] kernel: soundwire_intel soundwire_intel.link.0: sdw_cdns_clock_stop failed: MCP_CONTROL_HW_RST is not cleared
This is not a new issue, the dev_err logs above were added on purpose to help root-cause #2606 which has occurred randomly for the last 6 months, so far only on CML_RVP.
Weekly stress test(test result 4894) found same problem with CML_RVP_SDW.
===========================>>
[ 9449.385740] kernel: soundwire_intel soundwire_intel.link.1: intel_resume_runtime BUS_RESET: MCP_CONTROL_HW_RST is not cleared at iteration 0
[ 9449.387130] kernel: soundwire_intel soundwire_intel.link.1: intel_resume_runtime BUS_RESET: MCP_CONTROL_HW_RST is not cleared at iteration 1
[ 9449.388635] kernel: soundwire_intel soundwire_intel.link.1: intel_resume_runtime BUS_RESET: MCP_CONTROL_HW_RST is not cleared at iteration 2
[ 9449.390185] kernel: soundwire_intel soundwire_intel.link.1: intel_resume_runtime BUS_RESET: MCP_CONTROL_HW_RST is not cleared at iteration 3
[ 9449.391714] kernel: soundwire_intel soundwire_intel.link.1: intel_resume_runtime BUS_RESET: MCP_CONTROL_HW_RST is not cleared at iteration 4
[ 9449.393157] kernel: soundwire_intel soundwire_intel.link.1: intel_resume_runtime BUS_RESET: MCP_CONTROL_HW_RST is not cleared at iteration 5
[ 9449.394707] kernel: soundwire_intel soundwire_intel.link.1: intel_resume_runtime BUS_RESET: MCP_CONTROL_HW_RST is not cleared at iteration 6
[ 9449.396156] kernel: soundwire_intel soundwire_intel.link.1: intel_resume_runtime BUS_RESET: MCP_CONTROL_HW_RST is not cleared at iteration 7
[ 9449.397253] kernel: soundwire_intel soundwire_intel.link.1: intel_resume_runtime BUS_RESET: MCP_CONTROL_HW_RST is not cleared at iteration 8
[ 9449.398875] kernel: soundwire_intel soundwire_intel.link.1: intel_resume_runtime BUS_RESET: MCP_CONTROL_HW_RST is not cleared at iteration 9
[ 9449.400241] kernel: soundwire_intel soundwire_intel.link.1: intel_resume_runtime BUS_RESET failed: MCP_CONTROL_HW_RST is not cleared
<<===========================
changed the title [-][SoundWire] MCP_CONTROL_HW_RST not cleared dev_err logs[/-][+][SoundWire] MCP_CONTROL_HW_RST not cleared at iteration (dev_err logs)[/+]on Jun 28, 2021
changed the title [-][SoundWire] MCP_CONTROL_HW_RST not cleared at iteration (dev_err logs)[/-][+][SoundWire] intel_resume: MCP_CONTROL_HW_RST not cleared at iteration (dev_err logs)[/+]on Jun 28, 2021
[ 3757.004272] kernel: soundwire_intel soundwire_intel.link.0: intel_start_bus_after_reset: MCP_CONTROL_HW_RST is not cleared at iteration 0
[ 3757.005828] kernel: soundwire_intel soundwire_intel.link.0: intel_start_bus_after_reset: MCP_CONTROL_HW_RST is not cleared at iteration 1
[ 3757.007261] kernel: soundwire_intel soundwire_intel.link.0: intel_start_bus_after_reset: MCP_CONTROL_HW_RST is not cleared at iteration 2
[ 3757.008806] kernel: soundwire_intel soundwire_intel.link.0: intel_start_bus_after_reset: MCP_CONTROL_HW_RST is not cleared at iteration 3
[ 3757.010309] kernel: soundwire_intel soundwire_intel.link.0: intel_start_bus_after_reset: MCP_CONTROL_HW_RST is not cleared at iteration 4
[ 3757.011854] kernel: soundwire_intel soundwire_intel.link.0: intel_start_bus_after_reset: MCP_CONTROL_HW_RST is not cleared at iteration 5
[ 3757.013310] kernel: soundwire_intel soundwire_intel.link.0: intel_start_bus_after_reset: MCP_CONTROL_HW_RST is not cleared at iteration 6
[ 3757.014856] kernel: soundwire_intel soundwire_intel.link.0: intel_start_bus_after_reset: MCP_CONTROL_HW_RST is not cleared at iteration 7
[ 3757.016357] kernel: soundwire_intel soundwire_intel.link.0: intel_start_bus_after_reset: MCP_CONTROL_HW_RST is not cleared at iteration 8
[ 3757.017927] kernel: soundwire_intel soundwire_intel.link.0: intel_start_bus_after_reset: MCP_CONTROL_HW_RST is not cleared at iteration 9
[ 3757.019355] kernel: soundwire_intel soundwire_intel.link.0: intel_start_bus_after_reset failed: MCP_CONTROL_HW_RST is not cleared
[ 3762.647781] kernel: soundwire_intel soundwire_intel.link.0: sdw_cdns_clock_stop failed: MCP_CONTROL_HW_RST is not cleared
I tried to reproduce this manually on one of the test devices jf-cml-rvp-sdw-3. So far no luck. I am already at 150 cycles of multiple-pipelines + pm_runtime suspend/resume and nothing logged.
Activity
[-]dev_err logs for root-cause[/-][+][SoundWire] MCP_CONTROL_HW_RST not cleared dev_err logs[/+]fredoh9 commentedon Jun 27, 2021
Weekly stress test(test result 4894) found same problem with CML_RVP_SDW.
dmesg_test_4894_cml.txt
[-][SoundWire] MCP_CONTROL_HW_RST not cleared dev_err logs[/-][+][SoundWire] MCP_CONTROL_HW_RST not cleared at iteration (dev_err logs)[/+][-][SoundWire] MCP_CONTROL_HW_RST not cleared at iteration (dev_err logs)[/-][+][SoundWire] intel_resume: MCP_CONTROL_HW_RST not cleared at iteration (dev_err logs)[/+]plbossart commentedon Jul 20, 2021
same warnings seen on Intel daily test 5356 (started on July 19).
The next daily tests after that will rely on #3006 which was merged on July 20.
If we see this error again then my theory of a clock stop issue was wrong... Let's see!
plbossart commentedon Jul 27, 2021
Seems my theory was wrong, this error still happens with the latest kernel, however this is the first time I managed to see it on Dell SKU 09c6
dmesg.txt
plbossart commentedon Aug 20, 2021
Seen by @bardliao (c.f. #3098 (comment))
plbossart commentedon Oct 12, 2021
Issue seen in https://sof-ci.sh.intel.com/#/result/planresultdetail/7312?model=CML_SKU0983_SDW&testcase=multiple-pause-resume-25
plbossart commentedon Nov 30, 2021
Observed this issue again when running suspend/resume stress testing on CML_SKU0983_SDW. Test ID:8054.
21 remaining items
plbossart commentedon Apr 17, 2023
PR #4225 change the way the reset is done, so it's super interesting if we see this error after April 17, 2023 in our test results.
plbossart commentedon Mar 25, 2024
Seen again in Intel daily tests
planresultdetail/39159?model=TGLU_SKU0A32_SDCA-ipc3&testcase=check-runtime-pm-status-15
plbossart commentedon Apr 4, 2024
the report from last week is due to other problems and fixed by #4896
marc-hb commentedon Jul 11, 2024
Just seen again, re-opening to be on the safe side.
https://sof-ci.01.org/softestpr/PR1220/build635/devicetest/index.html?model=CML_RVP_SDW-ipc3&testcase=check-runtime-pm-status
Details
marc-hb commentedon Jul 11, 2024
plbossart commentedon Jul 15, 2024
Humm, just one report on CML_RVP_SDW in over 14 months, that is going to be tricky to solve....
marc-hb commentedon Jul 19, 2024
Another one:
https://sof-ci.01.org/linuxpr/PR5106/build4064/devicetest/index.html?model=CML_RVP_SDW-ipc3&testcase=multiple-pipeline-all
It's coming back!
plbossart commentedon Jul 19, 2024
from https://sof-ci.01.org/softestpr/PR1220/build635/devicetest/index.html?model=CML_RVP_SDW-ipc3&testcase=check-runtime-pm-status, it looks like the first error started with the AMD machine driver rework
8245a14
I am not sure how this commit might have interfered with the low-level reset sequence on CML.
@bardliao @marc-hb @ssavati can you double-check if this bisect information seems correct?
plbossart commentedon Jul 19, 2024
I tried to reproduce this manually on one of the test devices jf-cml-rvp-sdw-3. So far no luck. I am already at 150 cycles of multiple-pipelines + pm_runtime suspend/resume and nothing logged.
plbossart commentedon Jul 19, 2024
I will stop the tests for now, unless we can reproduce it with a simple sequence we are not going to make progress or be able to bisect. Gah.
marc-hb commentedon Jul 19, 2024
This always had a low reproduction rate, didn't it? So I doubt this can be bisected.