Repository navigation
Firmware recovery #1675
Description
We really need to have a watchdog-like mechanism where the host driver can detect the DSP is no longer responding and recover by reinitializing the hardware and notify apps.
{{ message }}
We really need to have a watchdog-like mechanism where the host driver can detect the DSP is no longer responding and recover by reinitializing the hardware and notify apps.
Activity
lgirdwood commentedon Jan 9, 2020
ACK. This can be first tied into the IPC timeout handler code. We already have a developer Kconfig option that prevents D3 upon FW crash (via IPC timeout) so would likely need the default action to be recover if this Kconfig option is not enabled.
plbossart commentedon Jan 9, 2020
Right, this can work as a kernel-side detection only.
But I was thinking here of an explicit watchdog or something that shows a sign of activity on the firmware side.
kv2019i commentedon Jan 18, 2020
@plbossart This is a duplicate of #452
Still valid. Some work was done for #452 but delayed due to other work.
plbossart commentedon May 24, 2021
@ujfalusi the Skylake driver implemented a recovery mechanism that was used on ApolloLake, but I am not sure if that part was ever upstreamed. @mwasko would you happen to recall?
IIRC there's a need to let applications know the card was restarted, I just don't recall what mechanism (uevent?) was used.
ujfalusi commentedon May 25, 2021
@plbossart, what have worked for me in the past is to call snd_pcm_stop_xrun(struct snd_pcm_substream *substream) to escalate the event up to the application.
I need to take a look at the options, but instant detection of firmware crash can only be done if the crashed firmware would give us an interrupt/event that it crashed, which I don't think it is going to do.
The only option is to set up a timer matching the period_elapsed (well 1.2x of it to be safe) and restart it when we have got the elapsed. If the timer fires then the firmware might have been crashed.
mwasko commentedon May 25, 2021
In Skylake driver we had implemented the FW recovery for specific releases for ApolloLake but I don't think it was upstream to Linux kernel. However the recovery itself was not done based on watchdog-like mechanism but on critical errors like IPC timeout or FW errors like memory issues during FW boot.
ujfalusi commentedon May 25, 2021
@mwasko, thanks for the details. IPC timeout is a good indication of firmware error with the additional watchdog (when it applies) we can also catch crashes which happen without much IPC traffic, like audio operation when we just receive the periodic elapsed events.
I'll keep these in mind.
kv2019i commentedon May 25, 2021
@ujfalusi And to further complicate, SOF supports SNDRV_PCM_HW_PARAMS_NO_PERIOD_WAKEUP, so you might not get any regular IPCs nor period-elapsed IRQs. If a DSP stops in this configuration, you can detect this at XRUN, but it's not very straighforward -- some hw_ptr liveliness check could work, but needs to differentiate from application side errors. I believe starting with distinct FW failure events (a DSP panic or error to IPC), is a good starting point.
ujfalusi commentedon May 25, 2021
@kv2019i, IPC timout can be used to check NO_PERIOD_WAKEUP from a periodic timer (read position, status, hearthbeat, something small and trivial) but it would just defeat the whole purpose of NO_PERIOD_WAKEUP. I would leave this for now.
The other possibility would be to handle the underflow/overflow interrupts from the DAI HW directly in Linux. That would indicate FW issue (unless the firmware is crashed but the DMA is still happily servicing the peripheral in cyclic mode).
In that case we would notice it when user space asks for a position.
marc-hb commentedon Jan 11, 2024
This suspend/resume failure below did not just TIMEOUT, for a change the system stayed alive enough to send logs. What I found interesting and relevant is:
So I suspect the kernel successfully rebooted the DSP but didn't properly clean-up something in its internal state.
https://sof-ci.01.org/sofpr/PR8571/build1764/devicetest/index.html?model=TGLU_RVP_NOCODEC-ipc4&testcase=check-suspend-resume-with-playback
Internal TEST run #36728.
CI build start time: 2024-01-10T15:53:36-08:00
Linux Branch: topic/sof-dev
Commit: 623c00b5e977
Kconfig Branch: master
Kconfig Commit: 40725018b505
PR ID: 8571
Commit: Merge 6978c7c7e421 into 1cc7a4c5b1b1 ⇒ 73327103315f
Zephyr Commit: d7af6f371034
Kernel logs
EDIT: same again in https://sof-ci.01.org/sofpr/PR8743/build1905/devicetest/index.html?model=TGLU_RVP_NOCODEC-ipc4&testcase=check-suspend-resume-with-playback / thesofproject/sof#8743
ujfalusi commentedon Jan 15, 2024
@marc-hb, #4780 should in theory will fix the kernel state in case of this dsp panic. I cannot reproduce the issue locally.