Skip to content

[BUG][TGL][Chrome] DSP Panic when PCM99 Capture start during Playback with IGO integrated firmware #4584

Closed
Closed
[BUG][TGL][Chrome] DSP Panic when PCM99 Capture start during Playback with IGO integrated firmware#4584
Assignees
bkokoszx
Labels
DSP panicDSP panic observedIGO NCObserved in IGO NC integrationP1Blocker bugs or important featuresTGLApplies to Tiger LakebugSomething isn't working as expectedmulticoreIssues observed when not only core#0 is used.

Description

@sathya-nujella
sathya-nujella
opened on Aug 2, 2021

Describe the bug
With IGO integrated firmware and topology, During playback, when PCM99 capture starts, DSP panic issue is observed.

To Reproduce

  1. Youtube playback
  2. suspend resume DUT (in Chrome: suspend_stress_test -c 5)
  3. Suspend/resume is done Successfully
  4. Playback Continue
  5. arecord -Dhw:0,99 -f S32_LE -r 48000 -c 4 -d 20 /tmp/test.wav
  6. As soon we start arecord, DSP panic shown in dmesg

Reproduction Rate
DSP panic issue is reproduced without suspend/resume in above steps.
To consistently/easily reproduce, above steps were found out.
issue happens 60% of the times - 3 out of 5 times

Expected behavior
Playback and Capture should work.

Impact
Blocker and affects basic use cases

Environment

  1. Kernel:
    Chrome 5.4 kernel (branch: cros/release-R93-14092.B-chromeos-5.4)
  2. SOF fw + tplg
    Tgl_13release + private patches in google repo+ Intelligo module(
    have shared all the binaries through email as they are closed source.

This PR patches included: #4575

Please also include the relevant sections from the firmware log and kernel log in the report (and attach the full logs for complete reference). Kernel log is taken from dmesg and firmware log from sof-logger. See https://thesofproject.github.io/latest/developer_guides/debugability/logger/index.html

part of dmesg:
[ 197.893703] sof-audio-pci 0000:00:1f.3: error : DSP panic!
[ 197.893755] sof-audio-pci 0000:00:1f.3: status: fw entered - code 00000005
[ 197.893859] sof-audio-pci 0000:00:1f.3: error: runtime exception
[ 197.893862] sof-audio-pci 0000:00:1f.3: error: trace point 00004000
[ 197.893866] sof-audio-pci 0000:00:1f.3: error: panic at :0
[ 197.893869] sof-audio-pci 0000:00:1f.3: error: DSP Firmware Oops
(full dmesg attached)

Activity

  1. sathya-nujella
    added
    bugSomething isn't working as expected
    TGLApplies to Tiger Lake
    P1Blocker bugs or important features
    on Aug 2, 2021

  2. sathya-nujella commented on Aug 2, 2021

    @sathya-nujella
    sathya-nujella
    on Aug 2, 2021
    ContributorAuthor

    dmesg_sof_issue_4584_dsp_panic.txt

  3. sathya-nujella commented on Aug 2, 2021

    @sathya-nujella
    sathya-nujella
    on Aug 2, 2021
    ContributorAuthor

    @RDharageswari , @yongzhi1 , @ranj063 , @keyonjie , @slawblauciak - FYI

  4. sathya-nujella commented on Aug 3, 2021

    @sathya-nujella
    sathya-nujella
    on Aug 3, 2021
    ContributorAuthor

    As @ranj063 mentioned enabled all kernel dynamic logs.

    We can see, kernel driver sent PCM Capture99 START and it is successful.

    [ 252.753079] sof-audio-pci 0000:00:1f.3: pcm: hw params stream 99 dir 1
    [ 252.753084] sof-audio-pci 0000:00:1f.3: generating page table for 00000000642fc1c2 size 0x10000 pages 16
    [ 252.753092] sof-audio-pci 0000:00:1f.3: FW Poll Status: reg=0x40000 successful
    [ 252.753127] sof-audio-pci 0000:00:1f.3: FW Poll Status: reg=0x40000 successful
    [ 252.753133] sof-audio-pci 0000:00:1f.3: period_bytes:0x4000
    [ 252.753134] sof-audio-pci 0000:00:1f.3: periods:4
    [ 252.753162] sof-audio-pci 0000:00:1f.3: stream_tag 4
    [ 252.753173] sof-audio-pci 0000:00:1f.3: ipc tx: 0x60010000
    [ 252.754647] sof-audio-pci 0000:00:1f.3: ipc tx succeeded: 0x60010000
    [ 252.754653] sof-audio-pci 0000:00:1f.3: pcm: stream dir 1, posn mailbox offset is 790756
    [ 252.757309] sof-audio-pci 0000:00:1f.3: pcm: trigger stream 99 dir 1 cmd 1
    [ 252.757478] sof-audio-pci 0000:00:1f.3: FW Poll Status: reg=0x2044001e successful
    [ 252.757483] sof-audio-pci 0000:00:1f.3: ipc tx: 0x60040000
    [ 252.757784] sof-audio-pci 0000:00:1f.3: ipc tx succeeded: 0x60040000
    [ 252.764529] sof-audio-pci 0000:00:1f.3: error : DSP panic!

  5. sathya-nujella commented on Aug 3, 2021

    @sathya-nujella
    sathya-nujella
    on Aug 3, 2021
    ContributorAuthor

    Tried this patch, still issue is reproducible:

    SOF fw: tgl_13 + IGO private patches + PR:#4575 + Below patch ----> still panic issue happens.

    b91818b

  6. mengdonglin
    added
    multicoreIssues observed when not only core#0 is used.
    on Aug 3, 2021
  7. bkokoszx
    self-assigned this
    on Aug 3, 2021
  8. mengdonglin
    added
    DSP panicDSP panic observed
    on Aug 3, 2021

  9. sathya-nujella commented on Aug 3, 2021

    @sathya-nujella
    sathya-nujella
    on Aug 3, 2021
    ContributorAuthor

    with @ranj063 , tried the below change to move capture buffers to core-1 ----> we still hit the DSP panic issue.

    diff --git a/tools/topology/sof/pipe-igonr-capture.m4 b/tools/topology/sof/pipe-igonr-capture.m4
index 21926373448b..aa3aa52d6f4f 100644
--- a/tools/topology/sof/pipe-igonr-capture.m4
+++ b/tools/topology/sof/pipe-igonr-capture.m4
@@ -62,11 +62,11 @@ W_IGO_NR(0, PIPELINE_FORMAT, 2, DAI_PERIODS, SCHEDULE_CORE, LIST(`         ', "I
 # Capture Buffers
 W_BUFFER(0, COMP_BUFFER_SIZE(DAI_PERIODS,
        COMP_SAMPLE_SIZE(PIPELINE_FORMAT), PIPELINE_CHANNELS, COMP_PERIOD_FRAMES(PCM_MAX_RATE, SCHEDULE_PERIOD)),
-       PLATFORM_PASS_MEM_CAP)
+       PLATFORM_PASS_MEM_CAP, SCHEDULE_CORE)

 W_BUFFER(1, COMP_BUFFER_SIZE(DAI_PERIODS,
        COMP_SAMPLE_SIZE(PIPELINE_FORMAT), PIPELINE_CHANNELS, COMP_PERIOD_FRAMES(PCM_MAX_RATE, SCHEDULE_PERIOD)),
-       PLATFORM_PASS_MEM_CAP)
+       PLATFORM_PASS_MEM_CAP, SCHEDULE_CORE)
 #
 # Pipeline Graph
 #

  10. lgirdwood commented on Aug 4, 2021

    @lgirdwood
    lgirdwood
    on Aug 4, 2021
    Member

    @sathya-nujella is there anything in the FW error log or sof-logger prior to the panic ?

  11. lgirdwood commented on Aug 4, 2021

    @lgirdwood
    lgirdwood
    on Aug 4, 2021
    Member

    @sathya-nujella do the buffer core change or #4575 have any impact on the reproduction rate or reproduction point ?

  12. sathya-nujella commented on Aug 4, 2021

    @sathya-nujella
    sathya-nujella
    on Aug 4, 2021
    ContributorAuthor

    @sathya-nujella do the buffer core change or #4575 have any impact on the reproduction rate or reproduction point ?

    Hi @lrgirdwo , same reproduction point in both experiments.

  13. sathya-nujella commented on Aug 4, 2021

    @sathya-nujella
    sathya-nujella
    on Aug 4, 2021
    ContributorAuthor

    @sathya-nujella is there anything in the FW error log or sof-logger prior to the panic ?

    Nothing specific error in sof-logs.
    @keyonjie is able to repro too with same steps. Any observations please?

  14. 4 remaining items

  15. sathya-nujella commented on Aug 6, 2021

    @sathya-nujella
    sathya-nujella
    on Aug 6, 2021
    ContributorAuthor

    Adding @fuyuntsuo to this issue -FYI

  16. fuyuntsuo commented on Aug 8, 2021

    @fuyuntsuo
    fuyuntsuo
    on Aug 8, 2021
    Contributor

    @sathya-nujella I have problem to reproduce your environment (tgl-013 in google private repo + PR #4575). The image build fails. Seems like the memory doesn't fit anymore. Log here.
    build_log.txt

    Can you push your working commit to private repo as detached head so that I can align with yours. Thanks.

  17. sathya-nujella commented on Aug 8, 2021

    @sathya-nujella
    sathya-nujella
    on Aug 8, 2021
    ContributorAuthor

    @sathya-nujella I have problem to reproduce your environment (tgl-013 in google private repo + PR #4575). The image build fails. Seems like the memory doesn't fit anymore. Log here.
    build_log.txt

    Can you push your working commit to private repo as detached head so that I can align with yours. Thanks.

    Sure @fuyuntsuo.

    FYI - Build works with this:
    1)checkout to this one: (private google repo)
    https://chrome-internal-review.googlesource.com/c/chromeos/third_party/sound-open-firmware-private/+/3982238
    2)
    wget https://patch-diff.githubusercontent.com/raw/thesofproject/sof/pull/4575.patch
    git am 4575.patch
    3)
    IGO library in respective folder

    Can you please share build error ?
    Also do 'git submodule update' before build and try again. Thank you!

  18. fuyuntsuo commented on Aug 9, 2021

    @fuyuntsuo
    fuyuntsuo
    on Aug 9, 2021
    Contributor

    thanks @sathya-nujella I am able to build the image. Trying to reproduce issue now.

  19. keyonjie commented on Aug 9, 2021

    @keyonjie
    keyonjie
    on Aug 9, 2021
    Contributor

    @fuyuntsuo ping me if you can't reproduce the issue.

  20. fuyuntsuo commented on Aug 9, 2021

    @fuyuntsuo
    fuyuntsuo
    on Aug 9, 2021
    Contributor

    @keyonjie. Following @sathya-nujella 's comment here, I am still not able to see any error in dmesg at this moment.

    May I know what does cras_test_client --playback_f /dev/zero and suspend_stress_test -c 1 do? The latter command shows lots of logs during each iteration.

  21. keyonjie commented on Aug 9, 2021

    @keyonjie
    keyonjie
    on Aug 9, 2021
    Contributor

    @keyonjie. Following @sathya-nujella 's comment here, I am still not able to see any error in dmesg at this moment.

    May I know what does cras_test_client --playback_f /dev/zero and suspend_stress_test -c 1 do? The latter command shows lots of logs during each iteration.

    @fuyuntsuo let's discuss details in email thread or even call directly, it is inefficient here.

  22. mengdonglin
    added
    IGO NCObserved in IGO NC integration
    and removed
    stress CI multicore tplgUsing stressful multicore CI topology sof-tgl-nocodec-ci.m4
    on Aug 19, 2021

  23. bkokoszx commented on Aug 31, 2021

    @bkokoszx
    bkokoszx
    on Aug 31, 2021
    Collaborator

    @sathya-nujella Do you still have a reproduction of this issue? Today, I've tried to reproduce it on top of tgl-013-drop-stable and did not hit it.

  24. sathya-nujella commented on Aug 31, 2021

    @sathya-nujella
    sathya-nujella
    on Aug 31, 2021
    ContributorAuthor

    @sathya-nujella Do you still have a reproduction of this issue? Today, I've tried to reproduce it on top of tgl-013-drop-stable and did not hit it.

    Hi Bartosz,

    We don't observe "DSP panic issue now", but please see details below:

    We are not testing using tgl-13 branch as is. We are applying all the below PR's on previous tgl-13 CL with IGO patches:

    #4575
    #4612
    #4640 <<< this PR has more changes than the one merged in tgl-13. We are using the patches from this PR during tests. So, we need your help to push delta changes to tgl-13 too. Thank you!
    #4696

    ==
    Also, for ref:
    @lancedkhou and @fuyuntsuo observed some IPC errors during their tests. We have requested them to raise new GH issues.

  25. lgirdwood commented on Aug 31, 2021

    @lgirdwood
    lgirdwood
    on Aug 31, 2021
    Member

    @sathya-nujella thanks, can you also link the new GH issues here too so we can track.

  26. XiaoyunWu6666 commented on Jan 24, 2022

    @XiaoyunWu6666
    XiaoyunWu6666
    on Jan 24, 2022
    Contributor

    @sathya-nujella Could you please provide other relevant GH issue links? thanks

  27. lgirdwood commented on Jan 24, 2022

    @lgirdwood
    lgirdwood
    on Jan 24, 2022
    Member

    @XiaoyunWu6666 this was fixed last year so should be closed. I'll close, we can always reopen if needed.

  28. lgirdwood
    closed this as completedon Jan 24, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

DSP panicDSP panic observedIGO NCObserved in IGO NC integrationP1Blocker bugs or important featuresTGLApplies to Tiger LakebugSomething isn't working as expectedmulticoreIssues observed when not only core#0 is used.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions