Skip to content

[BUG] driver crash after hibernation on kernel 6.11.0-8 #9572

Closed
Closed
[BUG] driver crash after hibernation on kernel 6.11.0-8#9572
Labels
MTLApplies to Meteor Lake platformbugSomething isn't working as expected

Description

@md0-code
md0-code
opened on Oct 11, 2024

Describe the bug
Upon returning from hibernation on kernel 6.11.0.-8 the sof-audio-pci-intel-mtl driver crashes. Does not happen with kernel version 6.8.0-45.

To Reproduce

  • hibernate the system and resume
  • check dmesg errors

Reproduction Rate
1/1

Expected behavior
Driver should survive hibernation

Impact
Major - no audio available after resuming from hibernation

Environment
OS: Ubuntu oracular 24.10 x86_64
Host: ASUS Zenbook 14 UX3405MA_Q415MA (1.0)
Kernel: Linux 6.11.0-8-generic
SOF: 2024-09

[    2.477315] sof-audio-pci-intel-mtl 0000:00:1f.3: enabling device (0000 -> 0002)
[    2.477547] sof-audio-pci-intel-mtl 0000:00:1f.3: DSP detected with PCI class/subclass/prog-if 0x040100
[    2.572705] sof-audio-pci-intel-mtl 0000:00:1f.3: bound 0000:00:02.0 (ops i915_audio_component_bind_ops [i915])
[    2.580124] sof-audio-pci-intel-mtl 0000:00:1f.3: use msi interrupt mode
[    2.601737] sof-audio-pci-intel-mtl 0000:00:1f.3: hda codecs found, mask 5
[    2.601744] sof-audio-pci-intel-mtl 0000:00:1f.3: using HDA machine driver skl_hda_dsp_generic now
[    2.601748] sof-audio-pci-intel-mtl 0000:00:1f.3: DMICs detected in NHLT tables: 2
[    2.607251] sof-audio-pci-intel-mtl 0000:00:1f.3: Firmware paths/files for ipc type 1:
[    2.607256] sof-audio-pci-intel-mtl 0000:00:1f.3:  Firmware file:     intel/sof-ipc4/mtl/sof-mtl.ri
[    2.607257] sof-audio-pci-intel-mtl 0000:00:1f.3:  Firmware lib path: intel/sof-ipc4-lib/mtl
[    2.607258] sof-audio-pci-intel-mtl 0000:00:1f.3:  Topology file:     intel/sof-ace-tplg/sof-hda-generic-2ch.tplg
[    2.608181] sof-audio-pci-intel-mtl 0000:00:1f.3: Loaded firmware library: ADSPFW, version: 2.11.1.1
[    2.728728] sof-audio-pci-intel-mtl 0000:00:1f.3: Booted firmware version: 2.11.1.1
[    2.751069] sof-audio-pci-intel-mtl 0000:00:1f.3: Topology: ABI 3:29:1 Kernel ABI 3:23:1

Screenshots or console output
Relevant crash log:

[   53.320065] sof-audio-pci-intel-mtl 0000:00:1f.3: Code loader DMA did not complete
[   53.320078] sof-audio-pci-intel-mtl 0000:00:1f.3: ------------[ DSP dump start ]------------
[   53.320079] sof-audio-pci-intel-mtl 0000:00:1f.3: Firmware download failed
[   53.320081] sof-audio-pci-intel-mtl 0000:00:1f.3: fw_state: SOF_FW_BOOT_READY_OK (6)
[   53.320105] sof-audio-pci-intel-mtl 0000:00:1f.3: 0x50000005: module: ROM_EXT, state: FW_ENTERED, running
[   53.320113] sof-audio-pci-intel-mtl 0000:00:1f.3: Firmware state: 0x5, status/error code: 0x0
[   53.320129] sof-audio-pci-intel-mtl 0000:00:1f.3: Core dump is not available due to invalid separator 0xc0de
[   53.320132] sof-audio-pci-intel-mtl 0000:00:1f.3: ------------[ DSP dump end ]------------
[   53.320203] sof-audio-pci-intel-mtl 0000:00:1f.3: Failed to start DSP
[   53.320205] sof-audio-pci-intel-mtl 0000:00:1f.3: error: failed to boot DSP firmware after resume -110
[   53.320208] sof-audio-pci-intel-mtl 0000:00:1f.3: PM: dpm_run_callback(): pci_pm_restore returns -110
[   53.320226] sof-audio-pci-intel-mtl 0000:00:1f.3: PM: failed to restore async: error -110
[   55.098996] sof-audio-pci-intel-mtl 0000:00:1f.3: failed to disable Host IPC and/or SOUNDWIRE
[   55.099007] sof-audio-pci-intel-mtl 0000:00:1f.3: error: failed to power down DSP during suspend -110
[   55.099011] sof-audio-pci-intel-mtl 0000:00:1f.3: can't suspend (snd_sof_runtime_suspend [snd_sof] returned -110)

Activity

  1. md0-code
    added
    bugSomething isn't working as expected
    on Oct 11, 2024
  2. abonislawski
    added
    MTLApplies to Meteor Lake platform
    on Oct 15, 2024

  3. lgirdwood commented on Oct 15, 2024

    @lgirdwood
    lgirdwood
    on Oct 15, 2024
    Member

    @ujfalusi did we have a recent fix around code loader that can be tried by @md0-code

  4. ujfalusi commented on Oct 15, 2024

    @ujfalusi
    ujfalusi
    on Oct 15, 2024
    Contributor

    I think the fix for this is sent to upstream and will be back ported to 6.11 as soon as it hits 6.12:
    https://lore.kernel.org/linux-sound/20241008060710.15409-1-peter.ujfalusi@linux.intel.com/

    Original issue: thesofproject/linux#5135

  5. md0-code commented on Oct 15, 2024

    @md0-code
    md0-code
    on Oct 15, 2024
    Author

    @ujfalusi Any relatively easy way to test this fix now? I don't have too much experience with kernel patches...

  6. ujfalusi commented on Oct 16, 2024

    @ujfalusi
    ujfalusi
    on Oct 16, 2024
    Contributor

    @md0-code, you would need to build your own kernel, if you are not comfortable doing that then I would wait for the patch to be backported to stable.
    Meanwhile disable the hibernate and use suspend, I believe that is working fine for you?

    The official guide for ubuntu is: https://wiki.ubuntu.com/Kernel/BuildYourOwnKernel
    You would need to apply the patch before building the kernel.

    The other way would be to use our development tree and test it. You need the packages to be able to build the kernel

    sudo apt install build-dep git libncurses-dev gawk flex bison openssl libssl-dev dkms libelf-dev libudev-dev libpci-dev libiberty-dev autoconf llvm

    (I'm not sure what you actually need on ubuntu) then:

    mkdir ~/kernel_test && cd ~/kernel_test
git clone https://github.com/ujfalusi/sof-kconfig.git
git clone https://github.com/thesofproject/linux.git
cd linux
../sof-kconfig/kconfig-sof-default.sh
make -j `nproc` bindeb-pkg
sudo dpkg -i ../linux-image--6.12.0-rc2*

    Then reboot to use this kernel:

    awk -F\' '/menuentry / {print $2}' /boot/grub/grub.cfg
# observe the list and note the one which is something "Ubuntu, with Linux 6.12.0-rc2-g2e6038a1d377"
sudo grub-reboot "the string you have noted above"
# reboot the laptop

    Check if you really booted the new kernel:

    uname -a

    test the hibernate, then you can remove the kernel to use the distro one:

    apt list --installed | grep linux-image
# note again the SOF kernel line and then
dpkg -r linux-image-6.12.0-rc2-g2e6038a1d377
# reboot

    I'm not sure if ubuntu uses Grub, if not then this might not work, so, I would only try it if I would know what I'm doing..

  7. md0-code commented on Oct 16, 2024

    @md0-code
    md0-code
    on Oct 16, 2024
    Author

    @ujfalusi Thank you for your instructions. They were helpful, and I was able to compile the kernel with the suggested patch. However, the problem still persists - I continue to encounter the same errors when resuming from hibernate. Is it possible that this issue has a different cause? Please let me know if you need any additional information to help diagnose the problem.

  8. ujfalusi commented on Oct 16, 2024

    @ujfalusi
    ujfalusi
    on Oct 16, 2024
    Contributor

    @md0-code, oh, I'm really surprised that the patch did not helped!
    Can you copy sof-dyndbg.conf.txt as /etc/modprobe.d/sof-dyndbg.conf, reboot and attach the full kernel log with the kernel that has the patch applied?
    Which contains one hibernate cycle

  9. md0-code commented on Oct 16, 2024

    @md0-code
    md0-code
    on Oct 16, 2024
    Author

    Here it is, a hibernate cycle with a patched 6.12.0-rc3+ kernel:
    kernel_log-mtl_hibernate_problem.txt

  10. ujfalusi commented on Oct 16, 2024

    @ujfalusi
    ujfalusi
    on Oct 16, 2024
    Contributor

    @md0-code, I see:

    [   38.516640] sof-audio-pci-intel-mtl 0000:00:1f.3: Code loader DMA did not complete
[   38.516652] sof-audio-pci-intel-mtl 0000:00:1f.3: ------------[ DSP dump start ]------------
[   38.516655] sof-audio-pci-intel-mtl 0000:00:1f.3: Firmware download failed
[   38.516657] sof-audio-pci-intel-mtl 0000:00:1f.3: fw_state: SOF_FW_BOOT_READY_OK (6)
[   38.516682] sof-audio-pci-intel-mtl 0000:00:1f.3: 0x50000005: module: ROM_EXT, state: FW_ENTERED, running
[   38.516690] sof-audio-pci-intel-mtl 0000:00:1f.3: Firmware state: 0x5, status/error code: 0x0
[   38.516705] sof-audio-pci-intel-mtl 0000:00:1f.3: Core dump is not available due to invalid separator 0xc0de
[   38.516709] sof-audio-pci-intel-mtl 0000:00:1f.3: ------------[ DSP dump end ]------------
[   38.516782] sof-audio-pci-intel-mtl 0000:00:1f.3: Failed to start DSP
[   38.516785] sof-audio-pci-intel-mtl 0000:00:1f.3: error: failed to boot DSP firmware after resume -110

    Which is not right since the patch [1] removed the Code loader DMA did not complete print from the kernel, so this kernel does not have the patch applied..

    [1] https://lore.kernel.org/linux-sound/20241008060710.15409-1-peter.ujfalusi@linux.intel.com/

  11. ujfalusi commented on Oct 16, 2024

    @ujfalusi
    ujfalusi
    on Oct 16, 2024
    Contributor

    @md0-code, and this is indeed the issue that the patch is fixing:

    [   38.191728] snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-mtl 0000:00:1f.3: ipc rx      : 0x1b080000|0x0
[   38.191740] snd_sof:sof_set_fw_state: sof-audio-pci-intel-mtl 0000:00:1f.3: fw_state change: 3 -> 6
[   38.191744] snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-mtl 0000:00:1f.3: ipc rx done : 0x1b080000|0x0

    this is the FW_READY notification from firmware that it has booted up.

  12. md0-code commented on Oct 16, 2024

    @md0-code
    md0-code
    on Oct 16, 2024
    Author

    Thank you for the confirmation. Most probably I did something wrong then when compiling the kernel - I saved your patch as patch.diff then did a patch -p1 < patch.diff against the latest mainline 6.12.0-rc3 before issuing a make -j nproc bindeb-pkg. I even checked to see that the hda-loader.c file was actually modified before installing the resulting deb file... I obviously need to learn more about the whole kernel patching procedure before doing this kind of thing...

  13. ujfalusi commented on Oct 16, 2024

    @ujfalusi
    ujfalusi
    on Oct 16, 2024
    Contributor

    If you cloned the tree with git, then you can download the patch and git am <the.patch>, but the patch -p1 < patch.diff should work as well.
    Are you sure that the modified kernel is installed and is the one that booted up?

  14. ujfalusi commented on Oct 16, 2024

    @ujfalusi
    ujfalusi
    on Oct 16, 2024
    Contributor

    If in doubt, do a make menuconfig navigate to General setup press ENTER at Local version - append to kernel release and enter something like -bugtesting, then exit and save the config and re-build the kernel.
    Note also that there will be multiple debian packages and you need to install the latest.

  15. md0-code commented on Oct 16, 2024

    @md0-code
    md0-code
    on Oct 16, 2024
    Author

    Yes, I did a git clone git://git.launchpad.net/~ubuntu-kernel-test/ubuntu/+source/linux/+git/mainline-crack cod/mainline/v6.12-rc3,
    make menuconfig, patch -p1 < patch.diff and make -j \'nproc\' bindeb-pkg, and then dpkg -t <linux-image*.deb>

    uname -a returns now Linux laptop 6.12.0-rc3+ #2 SMP PREEMPT_DYNAMIC Wed Oct 16 14:32:07 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

    I'll try your suggestion and see if it makes any difference.

  16. ujfalusi commented on Oct 16, 2024

    @ujfalusi
    ujfalusi
    on Oct 16, 2024
    Contributor

    you can check with git diff if the patch did modified the kernel, the git am will apply the patch cleanly, you can check the topmost commit with git show (but you can only apply to a clean tree, so before git am you need to do a git reset --hard).

    I'm not sure what goes wrong, it looks like that you done everything correctly, the computer just says no ;)

  17. md0-code commented on Oct 17, 2024

    @md0-code
    md0-code
    on Oct 17, 2024
    Author

    Found the problem! I was using a wildcard to copy the resulting .deb file and forgot that I still had an unpatched version in the output folder which always cam up first :) My bad :) Can confirm that the patch works and I can finally hibernate my laptop! Thank you again @ujfalusi

  18. md0-code
    closed this as completedon Oct 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    MTLApplies to Meteor Lake platformbugSomething isn't working as expected

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions