[BUG] [TGL] [tgl-012-drop-stable] Audio corruption and unexpected fault with codec adapter

Describe the bug

SOF crashes on boot with 'unexpected fault' when playback from speakers – as a result no audio (see dmesg output in unexpected_fault.txt).
To reproduce: reboot device and start playback, for example from youtube
Reproduction rate: 1/10.
Audio is corrupted when playback from speakers.
To reproduce: start playback on speakers, inject HP, eject HP, restore playback. Corrupted audio sounds like a bitrate mismatch.
Reproduction rate: close to 1
To Reproduce
Steps to repro:
Play video through youtube and check the playback
The playback might be
a. correct
b. white noise only
c. no sound can be heard
pause playback and close lid for 30 seconds
open lid, go back to step 1, the playback would be either a or b or c

The probabilities of symptoms b and c are around:
b: 50%
c: 25%

uploading white noise sample video clip:
white noise came out since 12 second.

Reproduction Rate
How often does the issue happen ? i.e. 1/10 (once in ten attempts), 1/1000 or all the time.
Does the reproduction rate vary with any other configuration or user action, if so please describe and show the new reproduction rate.

Expected behavior
A clear and concise description of what you expected to happen.

Impact
Showstopper
blocks integration

Environment
This issue is not observed without codec adapter
This issue is seen while CONFIG_DUMMY_CODEC is enabled
The above observation and attached logs are with codec adapter + waves binary

Name of the platform(s) on which the bug is observed.
- Platform: TGL
- Branch: https://github.com/thesofproject/sof/tree/tgl-012-drop-stable

Screenshots or console output
Please find attached
unexpected_fault.txt
crash-sof-log.txt
dmesg_crash.log
crash-dmesg-log.txt
corruption-sof-log.txt

IMG_5164.MOV

@RDharageswari

@RDharageswari, just to clarify, are steps 5,6 (pause + close/open lid ) necessary to trigger this issue?

also, you mention CONFIG_DUMMY_CODEC while logs clearly shows waves_codec_configure is being used. Missing info about the topology and cores utilized

@RDharageswari

@RDharageswari are you using CONFIG_DUMMY_CODEC? can you submit a PR with the tplg you are using?

@RDharageswari

@RDharageswari, just to clarify, are steps 5,6 (pause + close/open lid ) necessary to trigger this issue?

@mrajwa , steps 5/6 are not necessary.

When system booted into desktop, the very first audio playback could be:
a. correct/normal
b. white noise only (see enclosed video)
c. no sound can be heard

There is a chance to recover by:
step 5. close lid for around 30 secs
step 6. open lid and play again and hear the playback
it still could be a, b or c

@lancedkhou

@lancedkhou, ok thanks. Could you please show the topology settings including scheduled cores?

@RDharageswari

@RDharageswari this is very strange. So, you say that with CONFIG_DUMMY_CODEC=y this bug happens but with CONFIG_DUMMY_CODEC=n this bug doesn't happen?

I think the problem is because init_process function is missing from waves codec. It is also missing with dummy, I will fix that once @cujomalainey latest patches are in.

Can you please try this patch:

diff --git a/src/audio/codec_adapter/codec/waves.c b/src/audio/codec_adapter/codec/waves.c
index 497c81d9c..54a465ed2 100644
--- a/src/audio/codec_adapter/codec/waves.c
+++ b/src/audio/codec_adapter/codec/waves.c
@@ -648,6 +648,19 @@ int waves_codec_prepare(struct comp_dev *dev)
        return ret;
 }
 
+
+int waves_codec_init_process(struct comp_dev *dev)
+{
+       int ret;
+       struct codec_data *codec = comp_get_codec(dev);
+       struct waves_codec_data *waves_codec = codec->private;
+
+       codec->cpd.consumed = 0;
+       codec->cpd.init_done = 1;
+
+       return 0;
+}
+
 int waves_codec_process(struct comp_dev *dev)
 {
        int ret;
diff --git a/src/include/sof/audio/codec_adapter/interfaces.h b/src/include/sof/audio/codec_adapter/interfaces.h
index 4787ef4d7..b4a279c96 100644
--- a/src/include/sof/audio/codec_adapter/interfaces.h
+++ b/src/include/sof/audio/codec_adapter/interfaces.h
@@ -60,6 +60,7 @@ static struct codec_interface interfaces[] = {
                .id = WAVES_ID,
                .init  = waves_codec_init,
                .prepare = waves_codec_prepare,
+               .init_process = waves_codec_init_process,
                .process = waves_codec_process,
                .apply_config = waves_codec_apply_config,
                .reset = waves_codec_reset,

Sure, will try the attached patch.
There is an update from waves:
SOF crashes on boot with 'unexpected fault' when playback from speakers – as a result, no audio.
There is no Waves processing, only the dummy codec (passthrough with memcpy) added to the topology.
sof-logger_crash.log
dmesg_crash.log
The reproduction rate can be 1/50.
Please see the logs attached.

@RDharageswari

@RDharageswari thanks! Indeed the logs you provided do not contain anything related to 'waves' codec as opposed to first logs. So, I guess it the same problem reported above but related to dummy codec.

Dummy was renamed to passthrough so depending on what sources you have please try this:

--- a/src/audio/codec_adapter/codec/passthrough.c
+++ b/src/audio/codec_adapter/codec/passthrough.c
@@ -36,6 +36,7 @@ int passthrough_codec_prepare(struct comp_dev *dev)
                return -ENOMEM;
        }
        codec->cpd.out_buff_size = cd->period_bytes;
+       codec->cpd.init_done = 1;

@RDharageswari

@RDharageswari one more thing. Can you please attach the topology used (.m4 files). And also how to test. Are they using ALSA PCM interface or ALSA Compress interface?

I guess it's ALSA PCM as support for ALSA SOF compress is not merged yet. I have only tested ~~dummy~~ passthrough codec with my implementation of ALSA compress but it should work fine with PCM.

I'm concerned about the fact that is happens only 2% of the time.

@stolx

Ignore above link, it fails the cmd, not prepare like i originally thought. Though @stolx I did find something interesting unique to waves and not to the passthrough. If you play 32Le to passthrough you get normal playback but waves reduces rate from 48k to 32k according to alsa conformance. Firmware also vomits a bunch of xrun errors. I have been trying to hunt down the issue but no luck yet so far.

Note I cannot observe this when using only core 0. Hmm I'm starting to think non-homogenous pipeline core processing is a big factor here. I am going to produce new test images with the entire pipeline on a single core and we can see if the issues persist.

Helper question - what kernel branch and commit do you use to reproduce this issue?

This is the chrome os 5.4 fork we are working on

Note I cannot observe this when using only core 0.

same, when only core 0 is used everything works fine

@lyakh

@lyakh FYI you worked on issues with multicore...

@stolx

Today I have verified tgl-012-drop-stable with DOLBY DAP lib and sof-hda-generic topology on linux kernel, the results are as follows:

Parallel playback and capture using aplay and arecord respectively works OK ~~(except barely audible, low amplitude white noise in the background)~~
Playback using YT also works fine, pause/resume too ~~(low amp noise is also present + sporadic glitches occurred)~~

Results are the same no matter if codec adapter is on master or slave core.
I have never experienced any crash or no audio output issue.

Update: as it turns out YT playback format on my ubuntu is S16_LE while the topology was set to S32_LE - this mismatch may cause white noise... @stolx @RDharageswari please adjust the format in the topology.

@stolx

@stolx, to address the noise issue, I believe you need to update waves_effect_config in waves.c according to #4008

@mrajwa

@mrajwa @keyonjie can you give us an update on what has been ruled out and the current state and avenues of investigation ?

I'm going to be taking a look at how this adapter handles the buffers this week to see if there is an issue there.

@cujomalainey

@cujomalainey I have a suspicion that perhaps we are not producing/consuming data on cache line size and aligned chunks. @slawblauciak @keyonjie have we double checked this ? Something could have slipped in here that means we are not aligned with cache.

@lgirdwood

@lgirdwood I was only able to reproduce the noise issue with dolby codec I have. After applying the patch from #4008 (will be updated further soon) the problem is gone. I also double checked with Keyon, we did a live debug session on friday testing post processing on multicore. We also tried to stress it a bit adding more parallel pipelines - no issue observed. On the other hand, @zrombel run full scope tests and the results revealed stream stalls on multicore scenario (note he uses slim driver for test) this issue is common for all features: DSM, post processing, etc.
Is there anybody else who noticed this issue on full stack device?

@RDharageswari

@RDharageswari Do you think we can close this issue?

Navigation Menu

Repository navigation

[BUG] [TGL] [tgl-012-drop-stable] Audio corruption and unexpected fault with codec adapter #3965

11 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[BUG] [TGL] [tgl-012-drop-stable] Audio corruption and unexpected fault with codec adapter #3965

Description

Activity

mrajwa commented on Mar 29, 2021

mrajwa commented on Mar 29, 2021

keyonjie commented on Mar 29, 2021

lancedkhou commented on Mar 29, 2021

mrajwa commented on Mar 29, 2021

dbaluta commented on Mar 29, 2021

RDharageswari commented on Mar 29, 2021

dbaluta commented on Mar 29, 2021

dbaluta commented on Mar 29, 2021

11 remaining items

cujomalainey commented on Mar 31, 2021

mrajwa commented on Apr 1, 2021

cujomalainey commented on Apr 1, 2021

stolx commented on Apr 1, 2021

juimonen commented on Apr 1, 2021

mrajwa commented on Apr 6, 2021

mrajwa commented on Apr 7, 2021

lgirdwood commented on Apr 13, 2021

cujomalainey commented on Apr 13, 2021

lgirdwood commented on Apr 13, 2021

mrajwa commented on Apr 13, 2021

mengdonglin commented on Jul 20, 2021

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions