[BUG][TGL] [DSM] [I2S] ctx_save ipc time out and errors are observed while speaker playback stop

@ranj063

Scenario:
Below ipc time out and ctx_save ipc errors are observed at speaker playback stop.
Note:
Even though we see these errors, we did not see any functionality impact.
The next playback goes though without any issues
This issue is 100% reproducible with the DSM algorithm integrated FW.
But this issue may not be related to DSM only as it is observed in the other platforms as well.
It could also be a timing issue, because i did not hit into this issue, when i added more logs to the suspend() routine in kernel and in the FW ipc handlers.(handler.c). without even the below mentioned change.

Errors (enabled Audio dynamic logs):

=====log snippet====
[ 342.545884] sof-audio-pci 0000:00:1f.3: pcm: trigger stream 0 dir 0 cmd 0 //stream 0 is speaker
[ 342.545893] sof-audio-pci 0000:00:1f.3: ipc tx: 0x60050000
[ 342.546107] sof-audio-pci 0000:00:1f.3: ipc tx succeeded: 0x60050000
[ 342.546113] sof-audio-pci 0000:00:1f.3: FW Poll Status: reg=0x140000 successful
[ 342.546117] sof-audio-pci 0000:00:1f.3: ipc tx: 0x60030000
[ 342.546224] sof-audio-pci 0000:00:1f.3: ipc tx succeeded: 0x60030000
[ 342.546282] sof-audio-pci 0000:00:1f.3: pcm: free stream 0 dir 0
[ 342.546324] sof-audio-pci 0000:00:1f.3: pcm: close stream 0 dir 0
[ 345.040458] sof-audio-pci 0000:00:1f.3: ipc tx: 0x40010000
[ 345.542258] sof-audio-pci 0000:00:1f.3: error: ipc timed out for 0x40010000 size 76
[ 345.542281] sof-audio-pci 0000:00:1f.3: unknown ROM status value ffffffff
[ 345.542328] sof-audio-pci 0000:00:1f.3: invalid header size 0xffffffff. FW oops is bogus
[ 345.542330] sof-audio-pci 0000:00:1f.3: error: unexpected fault 0xffffffff trace 0xffffffff
[ 345.542354] sof-audio-pci 0000:00:1f.3: error: hda irq intsts 0x00000000 intlctl 0xc0000000 rirb 00
[ 345.542355] sof-audio-pci 0000:00:1f.3: error: dsp irq ppsts 0x00000000 adspis 0x00000000
[ 345.542377] sof-audio-pci 0000:00:1f.3: error: host status 0x00000000 dsp status 0x00000000 mask 0x00000003
[ 345.542378] sof-audio-pci 0000:00:1f.3: ctx_save ipc error -110, proceeding with suspend
[ 345.542479] sof-audio-pci 0000:00:1f.3: FW Poll Status: reg=0x9010f0f successful
[ 345.542486] sof-audio-pci 0000:00:1f.3: FW Poll Status: reg=0xf0f successful
[ 345.542500] sof-audio-pci 0000:00:1f.3: DSP core(s) enabled? 0 : core_mask 1
[ 345.542565] sof-audio-pci 0000:00:1f.3: Debug PCIR: 00000010 at 00000044
[ 345.542586] sof-audio-pci 0000:00:1f.3: Turning i915 HDAC power 0
[ 345.542609] sof-audio-pci 0000:00:1f.3: Current DSP power state: D3

With DSM library integrated, we see that the context save IPC fails.
The context save IPC gets called after the stream closure.
3 things happens during context save.
• disable HPSRAM
• disable LPSRAM
• send IPC response to the host
This IPC is timing out, as we couldn’t power down LPSRAM.

the following FW change helped to triage the issue:

--- a/src/platform/intel/cavs/lib/power_down.S
+++ b/src/platform/intel/cavs/lib/power_down.S
@@ -54,6 +54,7 @@ sram_dis_loop_cnt:
#define pfl_reg a15

power_down:
+ j _PD_DISABLE_HPSRAM
entry sp, 32
// effectively executes:
// xthal_dcache_region_lock(&literals, 128);

chromeaudio@workstation312:~/work/sof/sof/exp2$
with this change there is no timeout. So timeout is when we try to power down LPSRAM with DSM.
Thanks @ranj063 : For all the help to debug the issue so far.

FW:tgl-010-drop-stable branch + DSM Library

@mwasko

@mwasko @fkaczmar @slawblauciak fiy
@RDharageswari good info here, can you confirm if the DSM algo was ever active prior to ctx_save ? Do you know if DSM uses LPSRAM ?
@ryans-lee does DSM do anything with LPSRAM memory ?

Hi Liam,
Yes DSM is integrated in the speaker pb path.
The algo was active while speaker pb, and this issue is observed after that.

@RDharageswari

Thanks @RDharageswari - so the issue only happens after the algo is active.

should all the memory be freed before poweroff? The LPSRAM utilization is it a reason for power off not happening?
Tried one other experiment, to free memory in STOP, may not be the actual solution
src/audio/smart_amp/smart_amp.c
case COMP_TRIGGER_STOP:

          smart_amp_free(dev);
          break;
  default:

And with this change the issue is not reproducible.
Also Ranjani gave a test kernel patch to destroy the pipelines, while suspend and the issue is not seen in that case as well.

@mwasko

@mwasko what's the FW opinion here wrt LPSRAM and poweroff hangs ?

@naveen-manohar

@naveen-manohar are you saying this is a duplicate of #3541?

Hi Marc,
There are 2 issues as a part of #3541.
One is the IPC timeout issue:
sof-audio-pci 0000:00:1f.3: error: ipc timed out for 0x40010000 size 76
And other is the regmap issue, i believe #3541 is for the clean exit when the IPC error happens(Error handling)
This bug tracks the IPC timeout issue.

@RDharageswari

@RDharageswari do you still need inputs here on LPSRAM from @mwasko @slawblauciak ?

@bkokoszx

Hi Liam,
Yes we are still debugging on this issue.
debugging with @bkokoszx

@RDharageswari

@RDharageswari
Could you try following patch?

diff --git a/src/platform/intel/cavs/lib/power_down.S b/src/platform/intel/cavs/lib/power_down.S
index 018c47f76e0c..9422e8770de0 100644
--- a/src/platform/intel/cavs/lib/power_down.S
+++ b/src/platform/intel/cavs/lib/power_down.S
@@ -51,6 +51,7 @@ sram_dis_loop_cnt:
 #define temp_reg1                    a7
 #define temp_reg2                    a8
 #define temp_reg3                    a9
+#define host_base                   a10
 #define pfl_reg                      a15

 power_down:
@@ -78,6 +79,8 @@ _PD_DISABLE_LPSRAM:
  *  cavs_lpsram_power_down_entire();
  * }
  */
+       movi host_base, IPC_HOST_BASE
+
        beqz b_enable_lpsram, _PD_DISABLE_HPSRAM
        m_cavs_lpsram_power_down_entire temp_reg0, temp_reg1, temp_reg2,\
                                        sram_dis_loop_cnt
@@ -138,16 +141,15 @@ _PD_SEND_IPC:
  * bit by writing IPC_DIPCTDR_BUSY to IPC_DIPCTDR
  * and writing IPC_DIPCTDA_DONE to IPC_DIPCTDA
  */
-       movi temp_reg0, IPC_HOST_BASE
-       l32i temp_reg1, temp_reg0, IPC_DIPCTDR
+       l32i temp_reg1, host_base, IPC_DIPCTDR
        movi temp_reg2, ipc_flag
        l32i temp_reg2, temp_reg2, 0
        or temp_reg1, temp_reg1, temp_reg2
-       s32i temp_reg1, temp_reg0, IPC_DIPCTDR
+       s32i temp_reg1, host_base, IPC_DIPCTDR

-       l32i temp_reg1, temp_reg0, IPC_DIPCTDA
+       l32i temp_reg1, host_base, IPC_DIPCTDA
        or temp_reg1, temp_reg1, temp_reg2
-       s32i temp_reg1, temp_reg0, IPC_DIPCTDA
+       s32i temp_reg1, host_base, IPC_DIPCTDA

 _PD_SLEEP:
 /* effecfively executes:

We see similar issue on WHL and it helps.

Hi Bartsoz,
This fix is helping here as well. I tested the same scenario for around 5 times with this patch and don't see the issue happening

@RDharageswari

@RDharageswari
Thanks. That's good. After you confirm this patch with stress tests, I will make a PR.

As a word of explanation, the root cause of the problem seem to be the duplicated literal definition that was linked during compilation optimization to the memory location which was disabled earlier during power down process.

@bkokoszx

@bkokoszx good job - I will include this in v1.6 too

@naveen-manohar

@naveen-manohar : Hi Naveen can you help to test if this fix helps to solve the IPC timeout issue here:
#3541

@bkokoszx

@bkokoszx : Thanks for the fix.. I did verify your suggested patch on Soundwire SKU.. #3541 Issue Isn't observed with 20 iterations I tried.

Also tried basic Speaker functionality between suspend-resume & with my limited test, it worked fine

I've made a PR for this issue: #3582

Navigation Menu

Repository navigation

[BUG][TGL] [DSM] [I2S] ctx_save ipc time out and errors are observed while speaker playback stop #3538

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[BUG][TGL] [DSM] [I2S] ctx_save ipc time out and errors are observed while speaker playback stop #3538

Description

Activity

lgirdwood commented on Oct 21, 2020

RDharageswari commented on Oct 21, 2020

lgirdwood commented on Oct 22, 2020

RDharageswari commented on Oct 22, 2020

lgirdwood commented on Oct 23, 2020

marc-hb commented on Oct 24, 2020

RDharageswari commented on Oct 26, 2020

lgirdwood commented on Oct 29, 2020

RDharageswari commented on Oct 29, 2020

bkokoszx commented on Oct 31, 2020

RDharageswari commented on Nov 2, 2020

bkokoszx commented on Nov 3, 2020

mwasko commented on Nov 3, 2020

lgirdwood commented on Nov 3, 2020

RDharageswari commented on Nov 3, 2020

naveen-manohar commented on Nov 4, 2020

bkokoszx commented on Nov 4, 2020

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions