Skip to content

[BUG] MTL firmware does not resume any more #9308

Closed
Closed
[BUG] MTL firmware does not resume any more#9308
Assignees
lyakhtmleman
Labels
MTLApplies to Meteor Lake platformP1Blocker bugs or important featuresbugSomething isn't working as expected

Description

@marc-hb
marc-hb
opened on Jul 18, 2024

As of July 22nd, the current status is: something (IMR heap?) overwrites the firmware code stored in the IMR and used for resuming. That's why not booting from IMR with sof_debug=0x80 avoids the crash.
 

To Reproduce

Use audio. Wait. Try to use it again.

Reproduction Rate

100%

Impact

Show stopper

cc:

Screenshots or console output

Screenshot 2024-07-17 at 17 45 08

https://sof-ci.01.org/sofpr/PR9305/build6552/devicetest/index.html
https://sof-ci.01.org/softestpr/PR1220/build662/devicetest/index.html

[  161.664059] kernel: snd_sof:snd_sof_pci_update_bits_unlocked: sof-audio-pci-intel-mtl 0000:00:1f.3: Debug PCIR: 00000000 at  00000044
[  161.664078] kernel: snd_sof:snd_sof_pci_update_bits_unlocked: sof-audio-pci-intel-mtl 0000:00:1f.3: Debug PCIR: 00000000 at  00000048
[  161.667119] kernel: snd_sof:snd_sof_pci_update_bits_unlocked: sof-audio-pci-intel-mtl 0000:00:1f.3: Debug PCIR: 00000000 at  00000048
[  161.667125] kernel: snd_sof:snd_sof_pci_update_bits_unlocked: sof-audio-pci-intel-mtl 0000:00:1f.3: Debug PCIW: 00000040 at  00000048
[  161.667133] kernel: snd_sof_intel_hda_common:hda_dsp_state_log: sof-audio-pci-intel-mtl 0000:00:1f.3: Current DSP power state: D0I0
[  161.667138] kernel: snd_sof:sof_set_fw_state: sof-audio-pci-intel-mtl 0000:00:1f.3: fw_state change: 0 -> 2
[  161.667143] kernel: snd_sof:snd_sof_load_firmware: sof-audio-pci-intel-mtl 0000:00:1f.3: loading firmware
[  161.667146] kernel: snd_sof:sof_set_fw_state: sof-audio-pci-intel-mtl 0000:00:1f.3: fw_state change: 2 -> 3
[  161.668181] kernel: snd_sof_pci_intel_mtl:mtl_dsp_pre_fw_run: sof-audio-pci-intel-mtl 0000:00:1f.3: FW Poll Status: reg[0x1000]=0x1010000 successful
[  161.669221] kernel: snd_sof_pci_intel_mtl:mtl_dsp_pre_fw_run: sof-audio-pci-intel-mtl 0000:00:1f.3: FW Poll Status: reg[0x1d1c]=0x8151 successful
[  161.669227] kernel: snd_sof:snd_sof_run_firmware: sof-audio-pci-intel-mtl 0000:00:1f.3: booting DSP firmware
[  161.669233] kernel: snd_sof_intel_hda_common:hda_dsp_cl_boot_firmware: sof-audio-pci-intel-mtl 0000:00:1f.3: IMR restore supported, booting from IMR directly
[  161.670274] kernel: snd_sof_pci_intel_mtl:mtl_dsp_core_power_up: sof-audio-pci-intel-mtl 0000:00:1f.3: FW Poll Status: reg[0x178d04]=0x2000101 successful
[  161.670281] kernel: snd_sof_pci_intel_mtl:mtl_dsp_cl_init: sof-audio-pci-intel-mtl 0000:00:1f.3: Primary core power up successful
[  161.670290] kernel: snd_sof_pci_intel_mtl:mtl_dsp_cl_init: sof-audio-pci-intel-mtl 0000:00:1f.3: FW Poll Status: reg[0x73214]=0x80000000 successful
[  161.670301] kernel: snd_sof_pci_intel_mtl:mtl_enable_interrupts: sof-audio-pci-intel-mtl 0000:00:1f.3: FW Poll Status: reg[0x1800]=0x41 successful
[  161.670310] kernel: snd_sof_pci_intel_mtl:mtl_enable_interrupts: sof-audio-pci-intel-mtl 0000:00:1f.3: FW Poll Status: reg[0x1140]=0x1 successful
[  161.670320] kernel: snd_sof_pci_intel_mtl:mtl_dsp_cl_init: sof-audio-pci-intel-mtl 0000:00:1f.3: FW Poll Status: reg[0x163200]=0x50000005 successful
[  163.714303] kernel: sof-audio-pci-intel-mtl 0000:00:1f.3: ------------[ DSP dump start ]------------
[  163.714323] kernel: sof-audio-pci-intel-mtl 0000:00:1f.3: Firmware boot failure due to timeout
[  163.714330] kernel: sof-audio-pci-intel-mtl 0000:00:1f.3: fw_state: SOF_FW_BOOT_IN_PROGRESS (3)
[  163.714343] kernel: sof-audio-pci-intel-mtl 0000:00:1f.3: 0x50000005: module: ROM_EXT, state: FW_ENTERED, running
[  163.714356] kernel: sof-audio-pci-intel-mtl 0000:00:1f.3: Firmware state: 0x5, status/error code: 0x0
[  163.714438] kernel: sof-audio-pci-intel-mtl 0000:00:1f.3: Unknown toolchain is used
[  163.714446] kernel: sof-audio-pci-intel-mtl 0000:00:1f.3: error: DSP Firmware Oops
[  163.714451] kernel: sof-audio-pci-intel-mtl 0000:00:1f.3: error: Exception Cause: AllocaCause, MOVSP instruction, if caller’s registers are not in the register file
[  163.714458] kernel: sof-audio-pci-intel-mtl 0000:00:1f.3: EXCCAUSE 0x00000005 EXCVADDR 0x00000004 PS       0x00060020 SAR     0x00000004
[  163.714466] kernel: sof-audio-pci-intel-mtl 0000:00:1f.3: EPC1     0xa0088553 EPC2     0x00000000 EPC3     0x00000000 EPC4    0x00000000
[  163.714474] kernel: sof-audio-pci-intel-mtl 0000:00:1f.3: EPC5     0x00000000 EPC6     0x00000000 EPC7     0x00000000 DEPC    0x00000000
[  163.714480] kernel: sof-audio-pci-intel-mtl 0000:00:1f.3: EPS2     0x00000000 EPS3     0x00000000 EPS4     0x00000000 EPS5    0x00000000
[  163.714486] kernel: sof-audio-pci-intel-mtl 0000:00:1f.3: EPS6     0x00000000 EPS7     0x00000000 INTENABL 0x00000000 INTERRU 0x00000000
[  163.714492] kernel: sof-audio-pci-intel-mtl 0000:00:1f.3: stack dump from 0x00000000
[  163.714498] kernel: sof-audio-pci-intel-mtl 0000:00:1f.3: AR registers:
[  163.714506] kernel: sof-audio-pci-intel-mtl 0000:00:1f.3: 0x0: a0041e68 a0102688 00000000 a0065c84
[  163.714512] kernel: sof-audio-pci-intel-mtl 0000:00:1f.3: 0x10: a0066370 00000001 0000000a a01025c8
[  163.714519] kernel: sof-audio-pci-intel-mtl 0000:00:1f.3: 0x20: a0065d65 a0102628 00000001 40150478
[  163.714525] kernel: sof-audio-pci-intel-mtl 0000:00:1f.3: 0x30: a0065d65 a0102628 00000001 40150478
[  163.714532] kernel: sof-audio-pci-intel-mtl 0000:00:1f.3: ------------[ DSP dump end ]------------
[  163.714538] kernel: sof-audio-pci-intel-mtl 0000:00:1f.3: error: failed to boot DSP firmware after resume -5
[  163.714547] kernel: snd_sof:sof_set_fw_state: sof-audio-pci-intel-mtl 0000:00:1f.3: fw_state change: 3 -> 4
[  163.714558] kernel: sof-audio-pci-intel-mtl 0000:00:1f.3: ASoC: error at snd_soc_pcm_component_pm_runtime_get on 0000:00:1f.3: -5
[  163.714569] kernel:  NoCodec-0: ASoC: error at __soc_pcm_open on NoCodec-0: -5
[  163.714578] kernel:  Port0: ASoC: error at dpcm_be_dai_startup on Port0: -5
[  163.714588] kernel:  Port0: ASoC: error at dpcm_fe_dai_startup on Port0: -5

Activity

  1. marc-hb
    added
    bugSomething isn't working as expected
    urgent
    on Jul 18, 2024

  2. marc-hb commented on Jul 18, 2024

    @marc-hb
    marc-hb
    on Jul 18, 2024
    CollaboratorAuthor

    Last working daily test run 43924

    Start Time: 2024-07-16 13:08:34 UTC
    Linux Commit: 1998ade4783a
    KConfig Commit: 8189104a4f38
    SOF Commit: 3051607efb4f
    Zephyr Commit: 650227d8c47f

    First failing daily test run 43972

    Start Time: 2024-07-17 13:21:12 UTC
    Linux Commit: c3ecd35f66c1
    KConfig Commit: 8189104a4f38
    SOF Commit: d629e521020c
    Zephyr Commit: 740d7f735e23

    d629e52 serhiy.katsyuba@intel.com // (origin/main) ipc4: copier: Add IPC4 channel map handler
    ffce2cb serhiy.katsyuba@intel.com // ipc4: copier: Extend get_convertion_func() to support remapping
    eda6029 serhiy.katsyuba@intel.com // dai-zephyr: Prioritize HW params channels over base config params
    8082006 serhiy.katsyuba@intel.com // ipc4: pcm_converter: Add channel remapping conversion functions
    3199786 serhiy.katsyuba@intel.com // pcm_converter: Add channel map parameter
    8b927ad serhiy.katsyuba@intel.com // dai-zephyr: Use frames, not samples, for DMA copy bytes calculation
    6e183cb serhiy.katsyuba@intel.com // dai-zephyr: Fix to avoid using buffers with uninitialized stream params
    ee66620 serhiy.katsyuba@intel.com // ipc4: dai-zephyr: Do not reuse process func as multi-gateway channel copy
    5a000e8 tomasz.m.leman@intel.com // west.yml: update zephyr to 740d7f735e2
    3051607 guennadi.liakhovetski@linux.intel.com // kcps: fix 0 module CPC case

    _

    c3ecd35f66c1 Vijendar.Mukunda@amd.com // !fixup ASoC: amd: acp: fix for unused-but-set-variable warning for amp_num
1998ade4783a pierre-louis.bossart@linux.intel.com // ASoC: SOF: sof-audio.h: optimize snd_sof_pcm_stream_pipeline_list

    I checked all the corresponding Pull Requests and they all passed separately.

    cc: @serhiy-katsyuba-intel

  3. lyakh commented on Jul 18, 2024

    @lyakh
    lyakh
    on Jul 18, 2024
    Collaborator

    @marc-hb the last passing test of 16.07 already includes LLEXT patches as you can also see by following 3051607 and by checking that daily test's logs

  4. lyakh commented on Jul 18, 2024

    @lyakh
    lyakh
    on Jul 18, 2024
    Collaborator

    can it be the 5a000e8 Zephyr upgrade?

  5. fredoh9 commented on Jul 18, 2024

    @fredoh9
    fredoh9
    on Jul 18, 2024
    Contributor

    I only reverted LLEXT PR while keeping zephyr upgrade.
    FW re-loading worked fine. This is my revert branch, https://github.com/fredoh9/sof/tree/fix/revert

  6. lyakh commented on Jul 18, 2024

    @lyakh
    lyakh
    on Jul 18, 2024
    Collaborator

    I only reverted LLEXT PR while keeping zephyr upgrade. FW re-loading worked fine. This is my revert branch, https://github.com/fredoh9/sof/tree/fix/revert

    @fredoh9 thanks for checking that! And that isn't surprising, since every PR, merged after LLEXT passed its testing too - on a base that didn't include LLEXT. But apparently one of those PRs conflicts with LLEXT.

  7. kv2019i
    assigned
    lyakh
    and
    tmleman
    on Jul 18, 2024

  8. kv2019i commented on Jul 18, 2024

    @kv2019i
    kv2019i
    on Jul 18, 2024
    Collaborator

    @lyakh @tmleman can you check this, it's seems it's a combination of LLEXT and Zephyr update. Given where it fails, this could be a new side-effect of the power_down.S changes.

  9. kv2019i
    added
    P1Blocker bugs or important features
    MTLApplies to Meteor Lake platform
    on Jul 18, 2024
  10. marc-hb
    added a commit that references this issue on Jul 18, 2024

    intel_adsp_ace15_mtpm.conf: temporarily disable CONFIG_MODULES

    b8a6d93

  11. marc-hb commented on Jul 18, 2024

    @marc-hb
    marc-hb
    on Jul 18, 2024
    CollaboratorAuthor

    Partial quote from #9291 (comment)

    We can have cases where two PRs pass the tests independetly, but when they are both in the tree, the test fails

    This is a well-known CI issue and Gitlab and Github have a solution for it: ...

  12. kv2019i
    added a commit that references this issue on Jul 18, 2024

    intel_adsp_ace15_mtpm.conf: temporarily disable CONFIG_MODULES

    8847de0

  13. 19 remaining items

  14. coreboot-bot
    added a commit that references this issue on Jul 27, 2024

    intel: adsp: fix firmware image in IMR overwriting

    3260346

  15. marc-hb commented on Jul 29, 2024

    @marc-hb
    marc-hb
    on Jul 29, 2024
    CollaboratorAuthor

    Tentative Zephyr upgrade in sof/west.yml with the temporary fix:

  16. kv2019i commented on Aug 5, 2024

    @kv2019i
    kv2019i
    on Aug 5, 2024
    Collaborator

    With #9338 merged, @marc-hb @tmleman can we close this?

  17. tmleman commented on Aug 5, 2024

    @tmleman
    tmleman
    on Aug 5, 2024
    Contributor

    With #9338 merged, @marc-hb @tmleman can we close this?

    @marc-hb opened a separate issue on the Zephyr side zephyrproject-rtos/zephyr#76247. I think this one can be closed unless the FW grows faster than someone manages to fix it.

  18. marc-hb commented on Aug 5, 2024

    @marc-hb
    marc-hb
    on Aug 5, 2024
    CollaboratorAuthor

    can we close this?

    Once #9347 is passing and merged.

  19. kv2019i commented on Aug 6, 2024

    @kv2019i
    kv2019i
    on Aug 6, 2024
    Collaborator

    @marc-hb wrote:

    Once #9347 is passing and merged.

    Ack, now done, closing this one. Thanks all!

  20. kv2019i
    closed this as completedon Aug 6, 2024
  21. thedjnK
    added a commit that references this issue on Oct 27, 2024

    intel: adsp: fix firmware image in IMR overwriting

    84e38c3
  22. mariucker
    added a commit that references this issue on Dec 12, 2024

    intel: adsp: fix firmware image in IMR overwriting

    972bd91
  23. intel-lab-lkp
    added 2 commits that reference this issue on Feb 14, 2026

    ASoC: SOF: Add platform ops callback for DAI link hardware readiness

    e800788

    ASoC: SOF: Add platform ops callback for DAI link hardware readiness

    53f881b
  24. ujfalusi
    added a commit that references this issue on Feb 16, 2026

    ASoC: SOF: Add platform ops callback for DAI link hardware readiness

    358f19d
  25. intel-lab-lkp
    added 2 commits that reference this issue on Mar 4, 2026

    ASoC: SOF: Add platform ops callback for DAI link hardware readiness

    46a5f66

    ASoC: SOF: Add platform ops callback for DAI link hardware readiness

    2a8b266
  26. jma-phytec
    added a commit that references this issue on Jun 9, 2026

    intel: adsp: fix firmware image in IMR overwriting

    2996182
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

MTLApplies to Meteor Lake platformP1Blocker bugs or important featuresbugSomething isn't working as expected

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions