Skip to content

[DUT][ICL] random error: cl_dsp_init: timeout HDA_DSP_SRAM_REG_ROM_STATUS read #2890

Closed
Closed
[DUT][ICL] random error: cl_dsp_init: timeout HDA_DSP_SRAM_REG_ROM_STATUS read#2890
Assignees
xiulipan
Labels
BootFirmware boot or code signing related.HDAApplies to HD-Audio bus for codec connectionICLApplies to IceLake platformP2Critical bugs or normal featuresbugSomething isn't working as expectedstatus = 0x00000000boot status
Milestone
v1.6

Description

@xiulipan
xiulipan
opened on Jan 14, 2020

random dsp error message will appear on ICL RVP when FW need to be loaded(boot/runtime_pm_resume/s3_resume)

dmesg example 
kernel: [  409.510390] sof-audio-pci 0000:00:1f.3: FW Poll Status: reg=0xf0f0f0f successful
kernel: [  409.510423] sof-audio-pci 0000:00:1f.3: FW Poll Status: reg=0xf0f0f0e successful
kernel: [  409.510428] sof-audio-pci 0000:00:1f.3: unstall/run core: core_mask = 1
kernel: [  409.510435] sof-audio-pci 0000:00:1f.3: DSP core(s) enabled? 1 : core_mask 1
kernel: [  409.510981] sof-audio-pci 0000:00:1f.3: FW Poll Status: reg=0x80000000 successful
kernel: [  409.510987] sof-audio-pci 0000:00:1f.3: FW Poll Status: reg=0xf010e0e successful
kernel: [  409.511140] sof-audio-pci 0000:00:1f.3: error: no reply expected, received 0x0
kernel: [  409.811376] sof-audio-pci 0000:00:1f.3: FW Poll Status: reg=0x6000010 timedout
kernel: [  409.811379] sof-audio-pci 0000:00:1f.3: error: cl_dsp_init: timeout HDA_DSP_SRAM_REG_ROM_STATUS read
kernel: [  409.811403] sof-audio-pci 0000:00:1f.3: unknown ROM status value 06000010
kernel: [  409.811426] sof-audio-pci 0000:00:1f.3: error: status = 0x00000000 panic = 0x00000000
kernel: [  409.811447] sof-audio-pci 0000:00:1f.3: unknown ROM status value 06000010
kernel: [  409.811456] sof-audio-pci 0000:00:1f.3: FW Poll Status: reg=0x1010f0f successful
kernel: [  409.811464] sof-audio-pci 0000:00:1f.3: FW Poll Status: reg=0x1000f0f successful
kernel: [  409.811469] sof-audio-pci 0000:00:1f.3: DSP core(s) enabled? 0 : core_mask f
kernel: [  409.811472] sof-audio-pci 0000:00:1f.3: iteration 0 of Core En/ROM load failed: -110
kernel: [  409.811493] sof-audio-pci 0000:00:1f.3: Error code=0xffffffff: FW status=0xffffffff
kernel: [  409.812051] sof-audio-pci 0000:00:1f.3: FW Poll Status: reg=0xf0f0f0f successful
kernel: [  409.812083] sof-audio-pci 0000:00:1f.3: FW Poll Status: reg=0xf0f0f0e successful
kernel: [  409.812088] sof-audio-pci 0000:00:1f.3: unstall/run core: core_mask = 1
kernel: [  409.812095] sof-audio-pci 0000:00:1f.3: DSP core(s) enabled? 1 : core_mask 1
kernel: [  409.812887] sof-audio-pci 0000:00:1f.3: FW Poll Status: reg=0x80000000 successful
kernel: [  409.812909] sof-audio-pci 0000:00:1f.3: FW Poll Status: reg=0xf010e0e successful
kernel: [  409.813063] sof-audio-pci 0000:00:1f.3: error: no reply expected, received 0x0
kernel: [  409.887374] sof-audio-pci 0000:00:1f.3: FW Poll Status: reg=0x5000001 successful
kernel: [  409.897918] sof-audio-pci 0000:00:1f.3: FW Poll Status: reg=0x5 successful
kernel: [  409.898383] sof-audio-pci 0000:00:1f.3: FW Poll Status: reg=0x140000 successful
kernel: [  409.898395] sof-audio-pci 0000:00:1f.3: Firmware download successful, booting...
kernel: [  409.905888] sof-audio-pci 0000:00:1f.3: ipc rx: 0x70000000: FW_READY
kernel: [  409.905893] sof-audio-pci 0000:00:1f.3: ipc: DSP is ready 0x70000000 offset 0x81000
kernel: [  409.905901] sof-audio-pci 0000:00:1f.3: ipc rx done: 0x70000000: FW_READY
kernel: [  409.905990] sof-audio-pci 0000:00:1f.3: firmware boot complete

previous discussion in thesofproject/sof-test#16
@bardliao @ranj063

Activity

  1. stripes416
    changed the title [-][BUT][ICL] random error: cl_dsp_init: timeout HDA_DSP_SRAM_REG_ROM_STATUS read[/-] [+][DUT][ICL] random error: cl_dsp_init: timeout HDA_DSP_SRAM_REG_ROM_STATUS read[/+] on Jan 28, 2020

  2. stripes416 commented on Jan 28, 2020

    @stripes416
    stripes416
    on Jan 28, 2020

    Update: I've been watching for this during manual testing on ICL for the last several days and I haven't seen it.

  3. Liviali155 commented on Apr 1, 2020

    @Liviali155
    Liviali155
    on Apr 1, 2020

    On 20200401 daily test reproduced this issue again when do module load/unload test,sof/dev(3bdd89d)+master(1e39cee),I reproduced it manually,but can not reproduce it

    Removing snd_soc_core
Removing snd_hda_codec
Removing snd_hda_core
Removing snd_hwdep
Removing snd_compress
Removing snd_pcm
2020-03-31 23:46:11 UTC [REMOTE_INFO] checking for general errors after kmod unload with sof-kernel-log-check tool
2020-03-31 23:46:11 UTC [ERROR] Caught dmesg error
===========================>>
Mar 31 16:46:08 jf-icl-rvp-hda-1 kernel: [ 6623.273917] sof-audio-pci 0000:00:1f.3: error: cl_dsp_init: timeout HDA_DSP_SRAM_REG_ROM_STATUS read
<<===========================
2020-03-31 23:46:11 UTC [REMOTE_ERROR] error found after kmod unload is real error, failing
2020-03-31 23:46:11 UTC [REMOTE_INFO] Test Result: FAIL!

    check-kmod-load-unload-50.sh.txt
    cmd.txt
    dmesg.txt

  4. Liviali155 commented on Apr 7, 2020

    @Liviali155
    Liviali155
    on Apr 7, 2020

    On icl reproduced this issue with latest sof/dev(052e45d) +master(b4e8b1e)

    f-icl-rvp-hda-4

    Reproduction Rate
    2/2

    To Reproduce
    1.sudo reboot
    2.cd sof-test
    3.cd test-case
    4.export TPLG=/lib/firmware/intel/sof-tplg/sof-hda-generic-4ch.tplg
    5../check-kmod-load-unload-after-playback.sh -l 50

    dmesg_error.log 
    jf-icl-rvp-hda-4 kernel: [  805.732942] sof-audio-pci 0000:00:1f.3: FW Poll Status: reg=0xf0f0f0e successful
jf-icl-rvp-hda-4 kernel: [  805.732948] sof-audio-pci 0000:00:1f.3: unstall/run core: core_mask = 1
jf-icl-rvp-hda-4 kernel: [  805.732955] sof-audio-pci 0000:00:1f.3: DSP core(s) enabled? 1 : core_mask 1
jf-icl-rvp-hda-4 kernel: [  805.733754] sof-audio-pci 0000:00:1f.3: FW Poll Status: reg=0x80000000 successful
jf-icl-rvp-hda-4 kernel: [  805.733763] sof-audio-pci 0000:00:1f.3: FW Poll Status: reg=0xf010e0e successful
jf-icl-rvp-hda-4 kernel: [  806.033805] sof-audio-pci 0000:00:1f.3: FW Poll Status: reg=0x6000010 timedout
jf-icl-rvp-hda-4 kernel: [  806.033808] sof-audio-pci 0000:00:1f.3: error: cl_dsp_init: timeout HDA_DSP_SRAM_REG_ROM_STATUS read
jf-icl-rvp-hda-4 kernel: [  806.033835] sof-audio-pci 0000:00:1f.3: unknown ROM status value 06000010
jf-icl-rvp-hda-4 kernel: [  806.033857] sof-audio-pci 0000:00:1f.3: error: status = 0x00000000 panic = 0x00000000
jf-icl-rvp-hda-4 kernel: [  806.033878] sof-audio-pci 0000:00:1f.3: unknown ROM status value 06000010
jf-icl-rvp-hda-4 kernel: [  806.033887] sof-audio-pci 0000:00:1f.3: FW Poll Status: reg=0x1010f0f successful
jf-icl-rvp-hda-4 kernel: [  806.033895] sof-audio-pci 0000:00:1f.3: FW Poll Status: reg=0x1000f0f successful
jf-icl-rvp-hda-4 kernel: [  806.033901] sof-audio-pci 0000:00:1f.3: DSP core(s) enabled? 0 : core_mask f
jf-icl-rvp-hda-4 kernel: [  806.033903] sof-audio-pci 0000:00:1f.3: iteration 0 of Core En/ROM load failed: -110
jf-icl-rvp-hda-4 kernel: [  806.033924] sof-audio-pci 0000:00:1f.3: Error code=0xffffffff: FW status=0xffffffff
jf-icl-rvp-hda-4 kernel: [  806.034212] sof-audio-pci 0000:00:1f.3: FW Poll Status: reg=0xf0f0f0f successful
jf-icl-rvp-hda-4 kernel: [  806.034230] sof-audio-pci 0000:00:1f.3: FW Poll Status: reg=0xf0f0f0e successful
jf-icl-rvp-hda-4 kernel: [  806.034236] sof-audio-pci 0000:00:1f.3: unstall/run core: core_mask = 1
jf-icl-rvp-hda-4 kernel: [  806.034243] sof-audio-pci 0000:00:1f.3: DSP core(s) enabled? 1 : core_mask 1
jf-icl-rvp-hda-4 kernel: [  806.034776] sof-audio-pci 0000:00:1f.3: FW Poll Status: reg=0x80000000 successful
jf-icl-rvp-hda-4 kernel: [  806.034800] sof-audio-pci 0000:00:1f.3: FW Poll Status: reg=0xf010e0e successful
jf-icl-rvp-hda-4 kernel: [  806.109817] sof-audio-pci 0000:00:1f.3: FW Poll Status: reg=0x5000001 successful
jf-icl-rvp-hda-4 kernel: [  806.120735] sof-audio-pci 0000:00:1f.3: FW Poll Status: reg=0x5 successful
jf-icl-rvp-hda-4 kernel: [  806.121255] sof-audio-pci 0000:00:1f.3: FW Poll Status: reg=0x140000 successful
jf-icl-rvp-hda-4 kernel: [  806.121260] sof-audio-pci 0000:00:1f.3: Firmware download successful, booting...

    dmesg.txt
    slogger.txt

  5. mengdonglin
    transferred this issue fromthesofproject/linuxon Apr 28, 2020
  6. mengdonglin
    added
    BootFirmware boot or code signing related.
    bugSomething isn't working as expected
    ICLApplies to IceLake platform
    on Apr 28, 2020

  7. mengdonglin commented on Apr 28, 2020

    @mengdonglin
    mengdonglin
    on Apr 28, 2020
    Collaborator

    Transfer to SOF repo from linux issue No. 1703, to group with other DSP boot failure.

  8. mengdonglin
    added
    status = 0x00000000boot status
    HDAApplies to HD-Audio bus for codec connection
    on Apr 28, 2020

  9. Liviali155 commented on Apr 28, 2020

    @Liviali155
    Liviali155
    on Apr 28, 2020

    On icl run multiple-pipeline-playback also can reproduce this issue with latest sof/dev(8f1c6ad) +master(de2dcbb)

    After error occured ,all the pipeline can work

    1.sudo reboot
    2.cd sof-test
    3.cd test-case
    4.export TPLG=/lib/firmware/intel/sof-tplg/sof-hda-generic-4ch.tplg
    5../multiple-pipeline-playback.sh -l 50

    Reproduction rate
    Run 50times,failed at 44th

    Capture

    dmesg_error.log 
       [  777.984804] sof-audio-pci 0000:00:1f.3: error: cl_dsp_init: timeout HDA_DSP_SRAM_REG_ROM_STATUS read
[  777.984834] sof-audio-pci 0000:00:1f.3: error: status = 0x00000000 panic = 0x00000000

    dmesg0428.log

    sof-logger0428.log,

  10. tlauda commented on Apr 28, 2020

    @tlauda
    tlauda
    on Apr 28, 2020
    Contributor

    0x6000010 indicates that ROM is still waiting for response from CSE regarding IMR allocation. What is the value of the timeout? It looks like timing issue.

  11. Liviali155 commented on Apr 29, 2020

    @Liviali155
    Liviali155
    on Apr 29, 2020

    On icl run multiple-pipeline-playback also can reproduce this issue with latest sof/dev(8f1c6ad) +master(de2dcbb)

    After error occured ,all the pipeline can work

    1.sudo reboot
    2.cd sof-test
    3.cd test-case
    4.export TPLG=/lib/firmware/intel/sof-tplg/sof-hda-generic-4ch.tplg
    5../multiple-pipeline-playback.sh -l 50

    Reproduction rate
    Run 50times,failed at 44th

    Capture

    dmesg_error.log
    dmesg0428.log

    sof-logger0428.log,

    Add the log complete
    dmesg042.log
    sof-logger0429.log

  12. Liviali155 commented on May 22, 2020

    @Liviali155
    Liviali155
    on May 22, 2020

    On latest sof-dev(2de2b2a)+master(11ee3b8 ) run
    ./check-kmod-load-unload-after-playback.sh -l 1000
    ./multiple-pipeline-playback.sh -l 1000
    error not seen

  13. Liviali155
    added
    P2Critical bugs or normal features
    on May 26, 2020

  14. Liviali155 commented on May 26, 2020

    @Liviali155
    Liviali155
    on May 26, 2020

    On 20200526 for-next(e0b2db3)+master(136b48c) ,error occured again when run Check Runtime PM status

    2020-05-25 22:59:15 UTC [REMOTE_INFO] aplay killed
2020-05-25 22:59:15 UTC [REMOTE_INFO] wait dsp power status to become suspended
2020-05-25 22:59:17 UTC [REMOTE_INFO] dsp suspended in 3s
2020-05-25 22:59:17 UTC [REMOTE_INFO] runtime status: suspended
2020-05-25 22:59:17 UTC [ERROR] Caught dmesg error
===========================>>
[  492.621937] sof-audio-pci 0000:00:1f.3: error: cl_dsp_init: timeout HDA_DSP_SRAM_REG_ROM_STATUS read
<<===========================
2020-05-25 22:59:17 UTC [REMOTE_ERROR] Catch error in dmesg
2020-05-25 22:59:20 UTC [REMOTE_INFO] Test Result: FAIL!

    dmesg.txt
    slogger.txt

  15. Liviali155 commented on Jul 8, 2020

    @Liviali155
    Liviali155
    on Jul 8, 2020

    On 20200708 for-next(cce047aa)+master(adc8cc2) ,error occured again when run module reload test manually
    1."sudo reboot" to reboot system
    2.cd sof-test
    3.cd test-case
    4.export TPLG=/lib/firmware/intel/sof-tplg/sof-hda-generic-4ch.tplg
    5../check-kmod-load-unload.sh -l 50

    Failed at 28/50
    Capture

    dmesg_error.log 
    [  360.465179] sof-audio-pci 0000:00:1f.3: FW Poll Status: reg=0xf010e0e successful
[  360.765342] sof-audio-pci 0000:00:1f.3: FW Poll Status: reg=0x6000010 timedout
[  360.765345] sof-audio-pci 0000:00:1f.3: error: cl_dsp_init: timeout HDA_DSP_SRAM_REG_ROM_STATUS read
[  360.765371] sof-audio-pci 0000:00:1f.3: unknown ROM status value 06000010
[  360.765378] sof-audio-pci 0000:00:1f.3: error: status = 0x00000000 panic = 0x00000000
[  360.765400] sof-audio-pci 0000:00:1f.3: unknown ROM status value 06000010
[  360.765409] sof-audio-pci 0000:00:1f.3: FW Poll Status: reg=0x1010f0f successful
[  360.765417] sof-audio-pci 0000:00:1f.3: FW Poll Status: reg=0x1000f0f successful
[  360.765423] sof-audio-pci 0000:00:1f.3: DSP core(s) enabled? 0 : core_mask f
[  360.765426] sof-audio-pci 0000:00:1f.3: iteration 0 of Core En/ROM load failed: -110
[  360.765447] sof-audio-pci 0000:00:1f.3: Error code=0xffffffff: FW status=0xffffffff
[  360.765815] sof-audio-pci 0000:00:1f.3: FW Poll Status: reg=0xf0f0f0f successful
[  360.765832] sof-audio-pci 0000:00:1f.3: FW Poll Status: reg=0xf0f0f0e successful
[  360.765838] sof-audio-pci 0000:00:1f.3: unstall/run core: core_mask = 1
[  360.765845] sof-audio-pci 0000:00:1f.3: DSP core(s) enabled? 1 : core_mask 1

    dmesg0708.log

    sof-logger0708.log

  16. Liviali155 commented on Jul 8, 2020

    @Liviali155
    Liviali155
    on Jul 8, 2020

    On 20200708 sof-dev(ef5a473d)+master(adc8cc2),error not seen when run module reload test 50 times manually

  17. slawblauciak
    assigned
    xiulipan
    on Aug 10, 2020

  18. slawblauciak commented on Aug 10, 2020

    @slawblauciak
    slawblauciak
    on Aug 10, 2020
    Collaborator

    @xiulipan can you please verify if you can still see the error?

  19. xiulipan commented on Aug 11, 2020

    @xiulipan
    xiulipan
    on Aug 11, 2020
    ContributorAuthor

    @slawblauciak Yes, we can still see this issue on for-next branch. I think this maybe related to some kernel patches not up-streamed. Will double check .

  20. mengdonglin
    added this to the v1.6 milestone on Aug 20, 2020

  21. mengdonglin commented on Sep 1, 2020

    @mengdonglin
    mengdonglin
    on Sep 1, 2020
    Collaborator

    Close as recent CI daily test with sof-dev kernel and FW master branch cannot reproduce this issue

  22. mengdonglin
    closed this as completedon Sep 1, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

BootFirmware boot or code signing related.HDAApplies to HD-Audio bus for codec connectionICLApplies to IceLake platformP2Critical bugs or normal featuresbugSomething isn't working as expectedstatus = 0x00000000boot status

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions