Skip to content

Speaker-test hangs on close if rmbox -t open then dsp panic #401

Closed
Closed
Speaker-test hangs on close if rmbox -t open then dsp panic#401
Assignees
ranj063keyonjie
Labels
GLKApplies to Gemini Lake platformP2Critical bugs or normal featuresbugSomething isn't working as expectedlinuxProved to be a Linux kernel driver issueverifiedQA/Bug reporter verifies solution worked.

Description

@cujomalainey
cujomalainey
opened on Sep 20, 2018

Open rmbox -t before running speaker-test then try to close speaker-test via ctrl-c
Speaker-test will hang until rmbox is closed.

Observed on GLK v0.4 drop

If you try and use the speaker again after closing speaker test and rmbox then the DSP panics

[  101.283585] sof-audio sof-audio: error : DSP panic!
[  101.283592] sof-audio sof-audio: status: fw entered - code 00000005
[  101.283711] sof-audio sof-audio: error: runtime exception
[  101.283714] sof-audio sof-audio: error: trace point 00004000
[  101.283717] sof-audio sof-audio: error: DSP Firmware Oops
[  101.283720] sof-audio sof-audio: error: Exception Cause: LoadStorePIFDataErrorCause, Synchronous PIF data error during LoadStore access
[  101.283723] sof-audio sof-audio: EXCCAUSE 0x0000000d EXCVADDR 0x00002340 PS       0x00070025 SAR     0x00000020
[  101.283726] sof-audio sof-audio: EPC1     0xbe02041d EPC2     0xbe010bee EPC3     0x00000000 EPC4    0x00000000
[  101.283729] sof-audio sof-audio: EPC5     0x00000000 EPC6     0x00000000 EPC7     0x00000000 DEPC    0x00000000
[  101.283732] sof-audio sof-audio: EPS2     0x00060720 EPS3     0x00000000 EPS4     0x00000000 EPS5    0x00000000
[  101.283735] sof-audio sof-audio: EPS6     0x00000000 EPS7     0x00000000 INTENABL 0x000101d8 INTERRU 0x00000222
[  101.283738] sof-audio sof-audio: stack dump from 0xbe07fdb0
[  101.283741] sof-audio sof-audio: 0xbe07fdb0: 0xbe02041d 0x00060d35 0x00000020 0xbe059c04
[  101.283745] sof-audio sof-audio: 0xbe07fdb4: 0x04000020 0xffffffed 0xffffffb4 0xbe0420a8
[  101.283748] sof-audio sof-audio: 0xbe07fdb8: 0x0000000d 0x00000000 0xbe0230c2 0xbe0230d8
[  101.283751] sof-audio sof-audio: 0xbe07fdbc: 0x00212ef2 0x00000000 0x04444469 0x00000000
[  101.283754] sof-audio sof-audio: 0xbe07fdc0: 0x00212e2c 0x00000000 0x04654443 0x00000000
[  101.283757] sof-audio sof-audio: 0xbe07fdc4: 0x00212ecf 0x00000000 0x04726573 0x00000000
[  101.283760] sof-audio sof-audio: 0xbe07fdc8: 0x00212e45 0x00000000 0x03654f70 0x00000000
[  101.283763] sof-audio sof-audio: 0xbe07fdcc: 0xbe01de68 0xbe07fe70 0xbe07fe90 0xbe059700
[  101.283766] sof-audio sof-audio: error: waking up any trace sleepers
[  101.591314] sof-audio sof-audio: error: ipc timed out for 0x60010000 size 0x60
[  101.591342] sof-audio sof-audio: status: fw entered - code 00000005
[  101.591507] sof-audio sof-audio: error: runtime exception
[  101.591510] sof-audio sof-audio: error: trace point 00004000
[  101.591512] sof-audio sof-audio: error: DSP Firmware Oops
[  101.591515] sof-audio sof-audio: error: Exception Cause: LoadStorePIFDataErrorCause, Synchronous PIF data error during LoadStore access
[  101.591519] sof-audio sof-audio: EXCCAUSE 0x0000000d EXCVADDR 0x00002340 PS       0x00070025 SAR     0x00000020
[  101.591521] sof-audio sof-audio: EPC1     0xbe02041d EPC2     0xbe010bee EPC3     0x00000000 EPC4    0x00000000
[  101.591524] sof-audio sof-audio: EPC5     0x00000000 EPC6     0x00000000 EPC7     0x00000000 DEPC    0x00000000
[  101.591527] sof-audio sof-audio: EPS2     0x00060720 EPS3     0x00000000 EPS4     0x00000000 EPS5    0x00000000
[  101.591530] sof-audio sof-audio: EPS6     0x00000000 EPS7     0x00000000 INTENABL 0x000101d8 INTERRU 0x00000222
[  101.591532] sof-audio sof-audio: stack dump from 0xbe07fdb0
[  101.591535] sof-audio sof-audio: 0xbe07fdb0: 0xbe02041d 0x00060d35 0x00000020 0xbe059c04
[  101.591538] sof-audio sof-audio: 0xbe07fdb4: 0x04000020 0xffffffed 0xffffffb4 0xbe0420a8
[  101.591541] sof-audio sof-audio: 0xbe07fdb8: 0x0000000d 0x00000000 0xbe0230c2 0xbe0230d8
[  101.591544] sof-audio sof-audio: 0xbe07fdbc: 0x00212ef2 0x00000000 0x04444469 0x00000000
[  101.591547] sof-audio sof-audio: 0xbe07fdc0: 0x00212e2c 0x00000000 0x04654443 0x00000000
[  101.591550] sof-audio sof-audio: 0xbe07fdc4: 0x00212ecf 0x00000000 0x04726573 0x00000000
[  101.591552] sof-audio sof-audio: 0xbe07fdc8: 0x00212e45 0x00000000 0x03654f70 0x00000000
[  101.591555] sof-audio sof-audio: 0xbe07fdcc: 0xbe01de68 0xbe07fe70 0xbe07fe90 0xbe059700
[  101.591558] sof-audio sof-audio: error: waking up any trace sleepers
[  101.591570] sof-audio sof-audio: error: got wrong posn offset 0x1867b80 for PCM -110
[  101.591573] sof-audio sof-audio: error: set pcm hw_params after resume
[  101.591578]  Speakers: ASoC: trigger FE failed -110

Activity

  1. mengdonglin
    added
    GLKApplies to Gemini Lake platform
    bugSomething isn't working as expected
    on Sep 29, 2018

  2. mengdonglin commented on Sep 29, 2018

    @mengdonglin
    mengdonglin
    on Sep 29, 2018
    Collaborator

    @keqiaozhang Could you reproduce this issue on the code base for GLK v0.5 drop?

  3. keqiaozhang commented on Sep 29, 2018

    @keqiaozhang
    keqiaozhang
    on Sep 29, 2018
    Collaborator

    @mengdonglin
    Yes, I can reproduce this issue on the code base of GLK v0.5 drop. I also found if we run speaker-test first then open rmbox, this issue cannot be reproduced, but if we open rmbox first, then run speaker-test and kill it, this issue will be happened.

  4. mengdonglin
    added
    P2Critical bugs or normal features
    on Sep 29, 2018
  5. mengdonglin
    assigned
    mengdonglin
    and
    keqiaozhang
    on Sep 29, 2018

  6. mengdonglin commented on Sep 29, 2018

    @mengdonglin
    mengdonglin
    on Sep 29, 2018
    Collaborator

    @akloniex @mmaka1 Could you help to evaluate if this issue can be fixed in GLK v0.5 release that still uses rmbox or next release that uses the trace logger?

  7. mengdonglin
    added
    exported
    on Sep 30, 2018

  8. lgirdwood commented on Oct 1, 2018

    @lgirdwood
    lgirdwood
    on Oct 1, 2018
    Member

    @mengdonglin looks like a driver issue (hang until rmbox closed) and fw issues (panic). Will need someone from driver team to check. However @bardliao trace update may fix driver part.

  9. cujomalainey commented on Oct 10, 2018

    @cujomalainey
    cujomalainey
    on Oct 10, 2018
    ContributorAuthor

    Initial tests show the new logger is not susceptible to this bug. Will keep investigating.

  10. keqiaozhang commented on Oct 10, 2018

    @keqiaozhang
    keqiaozhang
    on Oct 10, 2018
    Collaborator

    @cujomalainey
    Could you share your test branch/commit?
    I also tried logger with latest sof master and this issue still exists.

  11. cujomalainey commented on Oct 10, 2018

    @cujomalainey
    cujomalainey
    on Oct 10, 2018
    ContributorAuthor

    There is a chance that it might be working correctly because if the bugs I'm seeing in #473. I'll verify tomorrow when I'm in the office.

  12. cujomalainey commented on Oct 10, 2018

    @cujomalainey
    cujomalainey
    on Oct 10, 2018
    ContributorAuthor

    Yea I'm unable to reproduce on my system and I'm working off the same location as in #473 (so 4.18 kernel, not the 4.14) I'll check this again on 4.14 once we get V0.5.

  13. keqiaozhang commented on Oct 12, 2018

    @keqiaozhang
    keqiaozhang
    on Oct 12, 2018
    Collaborator

    I still can reproduce this issue with kernel 4.14 +sof-master.

    1. ChromeOS:R71-11041
    2. sof-master: cc3fb1e
    3. kernel-cros-sof-v4.14-rebase: 274bfc60
    4. soft-master: 7f47792

  14. 19 remaining items

  15. Jiangxinx commented on Nov 1, 2018

    @Jiangxinx
    Jiangxinx
    on Nov 1, 2018
    Contributor

    Not reproduced with kernel branch debug/v4.19-backport, on APL UP2 platfrom.

    STEP:
    1.sudo sof-logger -l sof-apl.ldc -t -o logger.log
    2.aplay -Dhw:0,0 audio.wav
    3.close aplay via ctrl-c
    4.aplay -Dhw:0,0 audio.wav

    Test env:
    hardware version: UP2
    sof-master: 43da169
    soft-master: 4ac9248
    kernel debug/v4.19-backport: 925637e3
    tplg: sof-apl-pcm512x.tplg

  16. lgirdwood commented on Nov 1, 2018

    @lgirdwood
    lgirdwood
    on Nov 1, 2018
    Member

    @keqiaozhang can you provide dmesg log starting prior to oops and logger to with latest code from today. Thanks

  17. keyonjie commented on Nov 1, 2018

    @keyonjie
    keyonjie
    on Nov 1, 2018
    Contributor

    looks trigger stop ipc is not recieved by FW at this case:
    apl-ipc.c: 82 IPC Pen

  18. xiulipan commented on Nov 1, 2018

    @xiulipan
    xiulipan
    on Nov 1, 2018
    Contributor

    @keyonjie
    This Pen is actually pretty tricky here. It is caused by DSP->HOST IPC reply IRQ.
    There are very strict rules to forbidden HOST to send an IPC to DSP when DSP is not finished the last one, and we also enable a scheduler to handle IPC.
    You can add more trace to print the IPC header, to see if we really send a STOP IPC to DSP and the DSP did not handle it due to this IPC Pen

  19. keyonjie commented on Nov 1, 2018

    @keyonjie
    keyonjie
    on Nov 1, 2018
    Contributor

    @ranj063 this issue is related with runtime PM, you can run 'echo on > /sys/bus/pci/devices/0000:00:0e.0/power/control' then you won't see this issue.

    we may need to revisit & align on how should we handle trace during suspend/resume, involving both FW and driver side.

  20. keqiaozhang commented on Nov 1, 2018

    @keqiaozhang
    keqiaozhang
    on Nov 1, 2018
    Collaborator

    In the first run, speaker-test just hang in there, need to stop sof-logger, then speaker-test will stop and exit. there's no obvious error log in demsg, below is the sof-logger output:
    sof-logger:

      0      2      IPC    103068.697917       104.218750            apl-ipc.c:111        Cmd
    0      2      IPC    103073.020833         4.322917            handler.c:571        PMr
    0      2      IPC    117700.156250     14627.135742            apl-ipc.c:111        Cmd
    0      2      IPC    117704.739583         4.583333            handler.c:182        SAl
    0      2     PIPE    117711.302083         6.562500           pipeline.c:803        Par
    0      2     HOST    117715.729167         4.427083               host.c:446        host-params
    0      2     HOST    117724.427083         8.697917            hda-dma.c:341        Dgt
    0      2     HOST    117728.750000         4.322917            hda-dma.c:516        Dsc
    0      2   VOLUME    117734.531250         5.781250             volume.c:234        par
    0      2      DAI    117738.437500         3.906250                dai.c:355        par
    0      2      DMA    117742.500000         4.062500             dw-dma.c:322        Dgt
    0      2     PIPE    117751.979167         9.479167           pipeline.c:598        pre
    0      2     HOST    117755.729167         3.750000               host.c:547        pre
    0      2   VOLUME    117759.583333         3.854167             volume.c:495        pre
    0      2      DAI    117765.156250         5.572917                dai.c:412        pre
    0      2      IPC    131058.385417     13293.229492            apl-ipc.c:111        Cmd
    0      2      IPC    131063.281250         4.895833            handler.c:395        tri
    0      2     PIPE    131069.895833         6.614583           pipeline.c:753        cmd
    0      2     HOST    131074.322917         4.427083               host.c:277        trg
    0      2     HOST    131078.437500         4.114583            hda-dma.c:397        DEn
    0      2     HOST    131082.656250         4.218750            hda-dma.c:280        hda-dma-init 0xbe042468 ch 0
    0      2   VOLUME    131121.354167        38.697918             volume.c:429        trg
    0      2      DAI    131125.260417         3.906250                dai.c:520        trg
    0      2      DAI    131128.437500         3.177083                dai.c:529        tsa
    0      2      DMA    131136.145833         7.708333             dw-dma.c:1139       dw-dma-int-register
    0      2      SSP    131146.458333        10.312500                ssp.c:824        tri
    0      2      SSP    131150.833333         4.375000                ssp.c:764        sta
    0      2      IPC   2319170.052083   2188019.250000            apl-ipc.c:111        Cmd
    0      2      IPC   2319174.791667         4.739583            handler.c:395        tri
    0      2     PIPE   2319180.989583         6.197917           pipeline.c:753        cmd
    0      2     HOST   2319185.000000         4.010417               host.c:277        trg
    0      2     HOST   2319189.270833         4.270833            hda-dma.c:472        DDi
    0      2   VOLUME   2319193.697917         4.427083             volume.c:429        trg
    0      2      DAI   2319197.343750         3.645833                dai.c:520        trg
    0      2      DAI   2319200.729167         3.385417                dai.c:591        tsp
    0      2      DMA   2319204.427083         3.697917             dw-dma.c:548        DDi
    0      2      SSP   2319214.583333        10.156250                ssp.c:824        tri
    0      2      SSP   2319345.781250       131.197922                ssp.c:805        Ss1
    0      2      SSP   2319350.416667         4.635417                ssp.c:814        Ss2
    0      2      IPC   2319581.666667       231.250000            apl-ipc.c:111        Cmd
    0      2      IPC   2319585.989583         4.322917            handler.c:293        SFr
    0      2     PIPE   2319592.291667         6.302083           pipeline.c:836        PRe
    0      2     HOST   2319596.354167         4.062500               host.c:609        res
    0      2     HOST   2319600.312500         3.958333            hda-dma.c:472        DDi
    0      2   VOLUME   2319609.010417         8.697917             volume.c:571        res

    DSP panic happens at the second run.
    sof-logger:

    0      2     HOST     89760.885417     89760.882812            hda-dma.c:341        Dgt
    0      2     HOST     89782.656250        21.770834            hda-dma.c:516        Dsc
    0      2     HOST     89797.083333        14.427083            hda-dma.c:397        DEn
    0      2     HOST     89801.041667         3.958333            hda-dma.c:280        hda-dma-init 0xbe042430 ch 0
    0      2      IPC     89935.520833       134.479172            apl-ipc.c:111        Cmd
    0      2      IPC     89940.312500         4.791667            handler.c:810        tcn
    0      2     HOST     89945.520833         5.208333               host.c:334        new
    0      2      DMA     89957.239583        11.718750                dma.c:116        dma-get 0xbe042468 sref 2
    0      2      IPC     90081.145833       123.906250            apl-ipc.c:111        Cmd
    0      2      IPC     90085.364583         4.218750            handler.c:810        tcn
    0      2   VOLUME     90090.156250         4.791667             volume.c:176        new
    0      2      IPC     90228.072917       137.916672            apl-ipc.c:111        Cmd
    0      2      IPC     90232.447917         4.375000            handler.c:834        Ibn
    0      2   BUFFER     90237.031250         4.583333             buffer.c:53         new
    0      2      IPC     90378.645833       141.614578            apl-ipc.c:111        Cmd
    0      2      IPC     90382.864583         4.218750            handler.c:834        Ibn
    0      2   BUFFER     90387.552083         4.687500             buffer.c:53         new
    0      2      IPC     90483.697917        96.145836            apl-ipc.c:111        Cmd
    0      2      IPC     90487.916667         4.218750            handler.c:810        tcn
    0      2      DAI     90493.906250         5.989583                dai.c:171        new
    0      2    POWER     90505.468750        11.562500         pm_runtime.c:83         dis-ssp-clk-gating index 1 CLKCTL 00200002
    0      2      DAI     90520.000000        14.531250                dai.c:86         dai-get 0xbe045fbc sref 1
    0      2    POWER     90524.739583         4.739583         pm_runtime.c:136        dis-dwdma-clk-gating index 0 CLKCTL 04200002

    dmesg:

    [  838.229643] sof-audio sof-audio: pcm: hw params stream 1 dir 0
[  838.229646] sof-audio sof-audio: generating page table for ffff9d6783093000 size 0xff00 pages 16
[  838.229672] sof-audio sof-audio: period_bytes:0xd80
[  838.229673] sof-audio sof-audio: periods:18
[  838.229683] sof-audio sof-audio: stream_tag 1
[  838.229712] sof-audio sof-audio: ipc tx: 0x60010000
[  838.229957] sof-audio sof-audio: ipc tx succeeded: 0x60010000
[  838.232778] sof-audio sof-audio: pcm: trigger stream 1 dir 0 cmd 1
[  838.232781] sof-audio sof-audio: PCM: resume stream 1 dir 0 DMA position 0
[  838.232783] sof-audio sof-audio: pcm: hw params stream 1 dir 0
[  838.232785] sof-audio sof-audio: generating page table for ffff9d6783093000 size 0xff00 pages 16
[  838.232809] sof-audio sof-audio: period_bytes:0xd80
[  838.232811] sof-audio sof-audio: periods:18
[  838.232820] sof-audio sof-audio: stream_tag 1
[  838.232841] sof-audio sof-audio: ipc tx: 0x60010000
[  838.232984] sof-audio sof-audio: error : DSP panic!
[  838.232990] sof-audio sof-audio: panic: dsp_oops_offset 788480 offset 788480
[  838.232996] sof-audio sof-audio: status: fw entered - code 00000005
[  838.233156] sof-audio sof-audio: error: runtime exception
[  838.233159] sof-audio sof-audio: error: trace point 00004000
[  838.233162] sof-audio sof-audio: error: panic happen at :0
[  838.233165] sof-audio sof-audio: error: DSP Firmware Oops
[  838.233168] sof-audio sof-audio: error: Exception Cause: LoadStorePIFDataErrorCause, Synchronous PIF data error during LoadStore access
[  838.233171] sof-audio sof-audio: EXCCAUSE 0x0000000d EXCVADDR 0x00002340 PS       0x00060025 SAR     0x00000010
[  838.233174] sof-audio sof-audio: EPC1     0xbe024307 EPC2     0xbe01f68e EPC3     0xbe013dc0 EPC4    0x00000000
[  838.233177] sof-audio sof-audio: EPC5     0x00000000 EPC6     0x00000000 EPC7     0x00000000 DEPC    0x00000000
[  838.233179] sof-audio sof-audio: EPS2     0x00060d20 EPS3     0x00060722 EPS4     0x00000000 EPS5    0x00000000
[  838.233182] sof-audio sof-audio: EPS6     0x00000000 EPS7     0x00000000 INTENABL 0x000101d0 INTERRU 0x00000222
[  838.233185] sof-audio sof-audio: stack dump from 0xbe07fcc0
[  838.233188] sof-audio sof-audio: 0xbe07fcc0: 0xbe024307 0x00060935 0x00000010 0x00000000
[  838.233191] sof-audio sof-audio: 0xbe07fcc4: 0xbe042468 0xffffffed 0x9e064978 0xbe04248c
[  838.233194] sof-audio sof-audio: 0xbe07fcc8: 0x0000000d 0x00000000 0xbe026e65 0xbe026e6b
[  838.233197] sof-audio sof-audio: 0xbe07fccc: 0x00000000 0x00234d14 0x00000000 0x2000683c
[  838.233199] sof-audio sof-audio: 0xbe07fcd0: 0x00000000 0x00000000 0xbe026e49 0xbe026e4f
[  838.233202] sof-audio sof-audio: 0xbe07fcd4: 0x00000000 0x00234ccb 0x00000000 0x2000490c
[  838.233205] sof-audio sof-audio: 0xbe07fcd8: 0x00000000 0x00234c5f 0x00000000 0xbe0135a4
[  838.233208] sof-audio sof-audio: 0xbe07fcdc: 0xbe020273 0xbe07fd80 0xbe07fda0 0xbe05d700

  21. ranj063 commented on Nov 1, 2018

    @ranj063
    ranj063
    on Nov 1, 2018
    Collaborator

    @keyonjie sorry I think I'm missing your point about runtime PM. once we start playback with aplay, the device should be active.

    Are you talking about the device being suspended even before we start the logger?

  22. keyonjie commented on Nov 2, 2018

    @keyonjie
    keyonjie
    on Nov 2, 2018
    Contributor

    @ranj063 yes, suspend -> start logger -> start playback.

  23. keyonjie commented on Nov 6, 2018

    @keyonjie
    keyonjie
    on Nov 6, 2018
    Contributor

    thesofproject/linux#248 should be able to fix this.

  24. keyonjie commented on Nov 20, 2018

    @keyonjie
    keyonjie
    on Nov 20, 2018
    Contributor

    thesofproject/linux#248 is closed.
    @ranj063 please backport kernel #287 and #237 and then we can revisit this issue, this #401 issue is already fixed on my observation after applied them manually.

  25. mengdonglin commented on Nov 20, 2018

    @mengdonglin
    mengdonglin
    on Nov 20, 2018
    Collaborator

    @ranj063 please @keqiaozhang when the backporting is done, so he can verify the solution.

  26. mengdonglin
    assigned
    ranj063
    on Nov 22, 2018

  27. keqiaozhang commented on Nov 27, 2018

    @keqiaozhang
    keqiaozhang
    on Nov 27, 2018
    Collaborator

    Confirmed that this issue is fixed with top of cros-sof-v4.14-rebase branch.
    Tested with:
    kernel-cros-sof-v4.14-rebase: f8f388ef
    sof-master: 443233
    tplg-master: ad3f3e4

  28. keqiaozhang
    closed this as completedon Nov 27, 2018
  29. mengdonglin
    added
    verifiedQA/Bug reporter verifies solution worked.
    on Nov 27, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

GLKApplies to Gemini Lake platformP2Critical bugs or normal featuresbugSomething isn't working as expectedlinuxProved to be a Linux kernel driver issueverifiedQA/Bug reporter verifies solution worked.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions