[BUG] DSM standalone Multicore Stream Stall

Describe the bug
In DSM test where DSM component is run on CORE 1 and rest of topology on default CORE 0 after triggering pipelines start, playback pipeline stops copying data. Test were performed to check if PR #4088 fixes issue #3947. Root cause seems to be different - error is caused by pipeline XRUN visible in attached trace_logs.

Topology

    +----------------------------------------------+
    | +------+   +---+   +- -+   +---+   +-------+ |
    | | Host |-->|Buf|-->|Dsm|-->|Buf|-->|SSP Dai|-------+
    | +------+   +---+   +---+   +---+   +-------+ |     |
    +------------------+   ^   +-------------------+     |
                       |   |   |                         |
                       | +---+ |                         |
                       | |Buf| |                         |
                       | +---+ |                         |
                       +---^---+                         |
    +----------------------|------------------------+    |
    | +------+   +---+   +-|--+   +---+   +-------+ |    |
    | | Host |<--|Buf|<--|Dmux|<--|Buf|<--|SSP Dai|<-----+
    | +------+   +---+   +----+   +---+   +-------+ |
    +-----------------------------------------------+

To Reproduce
Run Python test:
22_05_TestDsmMulticoreSsp48000Hz24b32b2ch or 22_05_TestDsmMulticoreSsp48000Hz32b32b2ch
with parameters:
--playback-iterations=100

Reproduction Rate
~2/100

Environment

Name of the platform(s) on which the bug is observed.
- Platform: TGL B0 RVP
Firmware branch name and commit
- Branch: main
- Hash: 3500d0e
- Config: defconfig + chromeconfig
- Config: defconfig
Python tests branch name and commit
- Branch: master
- Hash: 825533f4e0d3ff5e2425abb50e9785916d7d1044

Logs
22_05_TestDsmMulticoreSsp_StreamStall.zip

@zrombel

@zrombel do you mean PR #4088 is not enough to fix this? Can you help to do a quick check if PR #4089 help?

#4089 can't address this issue, I can reproduce this and have some clue about it, but need to debug it after May vacation.

Description updated, bug is also present in build with def_config only.

This is an asynchronous scheduling issue. We have everything running synchronously on core 0 and DSM asynchronously (wrt core 0 pipeline copy status).
The fix is to make sure that

DSM source/sink buffers are large enough to cope with scheduling differences.
DSM should continue if reference buffer is empty- treat as 0s
Demux should continue if reference buffer is full.

Just to share quick tests results:
Unfortunately #4089 doesn't fix DSM multicore issue. Results look very similar as main branch results.
Test info: Simple - 1 iteration, Stress 20 - 20 playback iterations, Stress 200 - 200 playback iterations

		sof_main_c2fe4ff10ba_defconfig	sof_main_c2fe4ff10ba_chromeconfig	sof_keyon_ll_37f11e33_defconfig	sof_keyon_ll_37f11e33_chromeconfig
22_00_TestDsmSsp48000Hz24b32b2ch	Simple	PASS	PASS	PASS	PASS
	Stress 20	PASS	PASS	PASS	PASS
	Stress 200	PASS	Glitch 1/200	PASS	Silent begginig 1/200
22_00_TestDsmSsp48000Hz32b32b2ch	Simple	PASS	PASS	PASS	PASS
	Stress 20	PASS	PASS	PASS	PASS
	Stress 200	Glitch 1/200	PASS	PASS	PASS
22_05_TestDsmMulticoreSsp48000Hz24b32b2ch	Simple	PASS	PASS	PASS	PASS
	Stress 20	Glitch 4/20	Stream Stall	Stream Stall	Stream Stall
	Stress 200	Stream Stall	Stream Stall	Stream Stall	Stream Stall
22_05_TestDsmMulticoreSsp48000Hz32b32b2ch	Simple	PASS	PASS	PASS	PASS
	Stress 20	Minor Glitch 1/20	Glitch 4/20	Stream Stall	Stream Stall
	Stress 200	Stream Stall	Stream Stall	Stream Stall	Stream Stall

@zrombel

@zrombel thanks for the valuable info. This looks like a fix for rc2 and TGL12 is needed. @keyonjie we need to fix this in topology to account for the async nature of the work running on core0 and core1 with respect to each other.

@lgirdwood

@lgirdwood @zrombel @slawblauciak I just root caused this today, will submit fix tomorrow.

@zrombel

@zrombel I just updated my PR #4089 which address the stream stall issue at my end, can you help to try it?

@keyonjie

Hi @keyonjie unfortunately stress test shows that #4089 doesn't fix this bug and stream stall still occurs.
Here are logs from the tests. Tests runs 20 playback iterations. Single core tests passes but in multicore tests pipelines stalls.
I attached single and multicore tests logs for comparison.

Keyon_ll_DSM_multicore_error.zip

@zrombel

Thanks @zrombel as talked this only happen when only the DSM component is put on core1 (and all other components of the pipeline in Core 0), we should add logic to invalidate buffers when the producer and consumer of it are run on different cores.

I'm moving this to v1.9 the DSM feature works when the DSM pipeline is on core 1 but not the DSM component. i.e. the synchronization between DSM as a standalone component and its source and sink buffers needs aligned.

This is blocking the upcoming TGL release, bumping to P1.

This is fixed by #4089.

@zrombel

@zrombel @slawblauciak can we close now ?

Yes, closing.

Navigation Menu

Repository navigation

[BUG] DSM standalone Multicore Stream Stall #4101

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[BUG] DSM standalone Multicore Stream Stall #4101

Description

Activity

keyonjie commented on Apr 30, 2021

keyonjie commented on Apr 30, 2021

zrombel commented on Apr 30, 2021

lgirdwood commented on Apr 30, 2021

zrombel commented on May 5, 2021

lgirdwood commented on May 5, 2021

keyonjie commented on May 6, 2021

keyonjie commented on May 7, 2021

zrombel commented on May 7, 2021

keyonjie commented on May 7, 2021

lgirdwood commented on May 10, 2021

slawblauciak commented on May 11, 2021

zrombel commented on May 11, 2021

lgirdwood commented on May 24, 2021

zrombel commented on May 24, 2021

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions