[BUG] [TGL Chrome] Sporadic playback failure on speaker, headset and HDMI

Describe the bug
All the boards ran for 1 hour without any issue for YouTube streaming, but if we pause and play few times, then there is no sound coming.

To Reproduce
Play youtube for few seconds, pause for few seconds to 5 mins, then resume playback and observe no sound.

Reproduction Rate
100% (The issue can happen on speaker, headset and HDMI with equal rate.)

Expected behavior
Should not lose sound with pause/resume.

Environment
Kernel Branch: https://chromium.googlesource.com/chromiumos/third_party/kernel/+/refs/heads/chromeos-5.4
SOF Branch: tgl-013-drop-stable
SOF Commit: 9a35f65
Device: Chrome VOXEL
TPLG:sof-tgl-max98373-rt5682-xperi.tplg

Note
The SOF FW include Google KWD and Maxiam DSM IP.

dmesg.txt

@RDharageswari

cc @RDharageswari @ranj063

@yongzhi1

Hi @yongzhi1 can you attach sof-logger logs also?

@yongzhi1

@yongzhi1 can you also confirm if the DSM is in the output path here. I know we have the same pause test in CI but it may not be holding for 5 secs and it wont be using DSM.

@yongzhi1

Hi @yongzhi1 can you attach sof-logger logs also?

Sure, the dmesg.txt in the issue description session was provided by validation, here are the logs I captured:

dmesg-003.txt
dsp-log-003.txt

@yongzhi1

@yongzhi1 can you also confirm if the DSM is in the output path here. I know we have the same pause test in CI but it may not be holding for 5 secs and it wont be using DSM.

Yes, but the same issue can be reproduced on headset and HDMI as well:
dmesg-hdmi.txt
dmesg-hdmi-02.txt

From the DSP log:

[783510433.293176] (10420.000000) c0 sa  src/lib/agent.c:79  WARN validate(), ll drift detected, delta = 112955
[783515376.522146] ( 4943.229004) c0 sa  src/lib/agent.c:79  WARN validate(), ll drift detected, delta = 95011
[783520980.428174] ( 5603.906250) c0 idc src/idc/idc.c:113   ERROR idc_wait_in_blocking_mode() error: timeout

that's the first occurrence of the ERROR idc_wait_in_blocking_mode() error: timeout error and 112955 and 95011 deltas are huge. On TGL the timer ticks with 38.4MHz, i.e. those deltas are like 3 and 2.5ms roughly. Looks like some code path is blocking interrupts for too long.

@lyakh

Thanks @lyakh for the insight , so it seems the kernel side IPC error is because idc_do_cmd() get blocked to process the pipeline param cmd.

With the following proof-of-concept, the issue seems to be fixed, but I am sure you can provide the right fix :)
yongzhi1@f61f493

idc-temp-fix-log.txt

@lyakh

Thanks @lyakh for the insight , so it seems the kernel side IPC error is because idc_do_cmd() get blocked to process the pipeline param cmd.

With the following proof-of-concept, the issue seems to be fixed, but I am sure you can provide the right fix :)
yongzhi1@f61f493

Unfortunately I don't have a simple correct fix. We have some clues which code paths are blocking interrupts for a long time in some scenarios and it would be easy enough to find out which one is doing that here, but fixing them is often rather difficult.

@bkokoszx

Agreed with Karol to move this P1 ADL issue to @bkokoszx .

@bkokoszx

Could it be that we dont restore the secondary cores after a D0i3 cycle?

Yes, we dont even power up secondary cores on wake up :) Even after additional power up there is no restore flow for them.
With @bkokoszx we managed to fully restore secondary cores, we will share debug code soon for testing purposes.

@abonislawski

Here is debug branch mentioned by @abonislawski :
https://github.com/bkokoszx/sof/tree/tgl-013-drop-stable-secondary_core_restore

@yongzhi1 Are you able to try to reproduce issue with above code?

@sathyap-chrome

FYI @sathyap-chrome @Vamshigopal

@bkokoszx

@bkokoszx In ADL we saw little different issue, logs
wov_spk.txt
from logs Error signature is matching with original TGL issue.

Test : if we try speaker playback after GHW(s0ix)
Observation : speaker playback is muted.
fw_branch: origin/adl-003-drop-stable

We have not reported this issue because we are seeing this behavior only with google kpd, with intel clap issue is not observed.
with your debug branch https://github.com/bkokoszx/sof/tree/tgl-013-drop-stable-secondary_core_restore
the issue is no longer reproduced with google kpd.

@bkokoszx

@bkokoszx In ADL we saw little different issue, logs
wov_spk.txt
from logs Error signature is matching with original TGL issue.

Test : if we try speaker playback after GHW(s0ix)
Observation : speaker playback is muted.
fw_branch: origin/adl-003-drop-stable

We have not reported this issue because we are seeing this behavior only with google kpd, with intel clap issue is not observed.
with your debug branch https://github.com/bkokoszx/sof/tree/tgl-013-drop-stable-secondary_core_restore
the issue is no longer reproduced with google kpd.

@Vamshigopal I think adl-003-drop-stable is already aligned with the latest tgl-013-drop-stable so the issue you mentioned should already been fixed, can you check the top commit of your adl-003-drop-stable branch?

@bkokoszx please correct me if I am wrong.

@keyonjie

@keyonjie we saw the issue even when tgl013 FW resigned for ADL.
with or without the WA submitted in TGL.

@keyonjie

@keyonjie we saw the issue even when tgl013 FW resigned for ADL. with or without the WA submitted in TGL.

got it, then looks @bkokoszx and @abonislawski 's fix for the LPS should be the correct way to go, we should get them for all TGL+ WoV related platforms.

@keyonjie

@keyonjie
adl-003-drop-stable is based on tgl-013-drop-stable. Additionally, it contains only two more topology commits, and minor change in FW referring to check buffer allocation in comp_prepare() function. So basically, there are mostly the same.

@Vamshigopal

Tested the series with below topology config on ADL Brya platform:

SPk pipe -> dump codec (no algo running) - CORE 0
DMIC -> Intelligo - CORE1
HS pipe -> no algo running - CORE2
HDMI pipelines -> no algo running - CORE3
Results here:
Audio_multicore_testing.xlsx

We also tested known issues raised in TGL when multicore was enabled, attached results here:
sof_issues_on_ADL.xlsx

Expect the known alsa conformance faulire, rest all use cases are PASS
kindly help merge this and also make a adl-003-hotfix branch to include them

cc @Vamshigopal @kv2019i @bkokoszx

@yongzhi1

@yongzhi1 @bkokoszx , ok to close now?

Can reopen if needed.

Navigation Menu

Repository navigation

[BUG] [TGL Chrome] Sporadic playback failure on speaker, headset and HDMI #4382

32 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[BUG] [TGL Chrome] Sporadic playback failure on speaker, headset and HDMI #4382

Description

Activity

yongzhi1 commented on Jun 21, 2021

keyonjie commented on Jun 22, 2021

lgirdwood commented on Jun 22, 2021

yongzhi1 commented on Jun 22, 2021

yongzhi1 commented on Jun 22, 2021

lyakh commented on Jun 23, 2021

yongzhi1 commented on Jun 23, 2021

lyakh commented on Jun 24, 2021

32 remaining items

kv2019i commented on Aug 17, 2021

abonislawski commented on Sep 27, 2021

bkokoszx commented on Sep 27, 2021

kv2019i commented on Sep 27, 2021

Vamshigopal commented on Sep 28, 2021

keyonjie commented on Sep 30, 2021

sathyap-chrome commented on Sep 30, 2021

keyonjie commented on Sep 30, 2021

bkokoszx commented on Sep 30, 2021

sathyap-chrome commented on Oct 4, 2021

kv2019i commented on Nov 25, 2021

lgirdwood commented on Dec 21, 2021

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions