Repository navigation
[BUG][TGL-012-drop-stable] DSM Multicore Stream Stall #3947
Description
Describe the bug
In DSM test where DSM component is run on CORE 1 and rest of topology on default CORE 0 after triggering pipelines start, playback pipeline stops copying data.
Topology
+----------------------------------------------+
| +------+ +---+ +- -+ +---+ +-------+ |
| | Host |-->|Buf|-->|Dsm|-->|Buf|-->|SSP Dai|-------+
| +------+ +---+ +---+ +---+ +-------+ | |
+------------------+ ^ +-------------------+ |
| | | |
| +---+ | |
| |Buf| | |
| +---+ | |
+---^---+ |
+----------------------|------------------------+ |
| +------+ +---+ +-|--+ +---+ +-------+ | |
| | Host |<--|Buf|<--|Dmux|<--|Buf|<--|SSP Dai|<-----+
| +------+ +---+ +----+ +---+ +-------+ |
+-----------------------------------------------+
To Reproduce
Run Python test:
22_05_TestDsmMulticoreSsp48000Hz24b32b2ch or 22_05_TestDsmMulticoreSsp48000Hz32b32b2ch
with parameters:
--playback-iterations=100
Reproduction Rate
~1/10
Environment
- Name of the platform(s) on which the bug is observed.
- Platform: TGL B0 RVP
- Firmware branch name and commit
- Branch: tgl-012-drop-stable
- Hash: fa35ee0
- Python tests branch name and commit
- Branch: master
- Hash: 237e3511f3787aa9b45c53b43bea33cfa2646114
Activity
slawblauciak commentedon Apr 2, 2021
I was able to bisect and find which change introduces this behavior:
5104f5e
However, the same change exists on tgl-011-drop-stable branch, and the problem does not occur there.
This leads me to believe that this behavior is a result of at least two changes.
I suspect the problem might have its source in the scheduler again, as such I will continue debugging from that side.
keyonjie commentedon Apr 6, 2021
Hi @slawblauciak @zrombel can you try if reverting 1fa1001 will help?
lgirdwood commentedon Apr 13, 2021
@keyonjie @slawblauciak any updates here ?
slawblauciak commentedon Apr 13, 2021
@lgirdwood I was only able to eliminate some possibilities: it's most likely not related to IDC or spin locks. I had some trace points set up around those and there were no oddities there. Additionally, when the issue occurs I can see that none of the pipelines actually work, the pipeline tasks are not called.
I'll be debugging it further from scheduler side.
slawblauciak commentedon Apr 13, 2021
Also, neglected to mention that this issue is only reproduced with Python tests under slim driver. No attempt of reproduction under full stack was successful.
lgirdwood commentedon Apr 13, 2021
@slawblauciak thanks, btw how is core 1 work scheduled and synchronized with core 0 in this case ?
slawblauciak commentedon Apr 14, 2021
@lgirdwood good question, I'm not exactly familiar with the way SMP is implemented in the firmware.
lgirdwood commentedon Apr 14, 2021
@slawblauciak the issue I'm thinking here if core 1 work takes too long for one iteration and how this could spill over into the core0 work (and how we could detect/report and recover)..
slawblauciak commentedon Apr 14, 2021
@lgirdwood that was my initial though as well, but I believe the work on the cores is scheduled entirely independently and without synchronization, except for the locks in the buffer access. But again, I'm not 100% sure.
slawblauciak commentedon Apr 22, 2021
Upon further debug I was able to partly determine source of the problem. There was a small discrepancy between latest changes made to the scheduler, between tgl-011 and tgl-012. Upon aligning the changes, I was able to get the scheduler to run somewhat properly now. However, the issue still occurs, which suggests that the task might be being incorrectly cancelled or erroneously handled otherwise. Unfortunately I lack the required understanding of the scheduler as a whole and the changes made to be able to tell why aligning them improves the behavior at this moment.
slawblauciak commentedon Apr 23, 2021
@zrombel please verify the effectiveness of the linked PR
zrombel commentedon Apr 27, 2021
Fixed by #4088, similar bug with different root cause found: #4101