[BUG]Attempt of Core En/ROM load failed before booting DSP firmware on ICL_RVP_HDA and GLK_BOB_DA7219 #4873

Activity

1. XiaoyunWu6666

   added

   bugSomething isn't working as expectedSomething isn't working as expected

   GLKApplies to Gemini Lake platformApplies to Gemini Lake platform

   ICLApplies to IceLake platformApplies to IceLake platform

   Intel Linux Daily testsThis issue can be found in internal Linux daily testsThis issue can be found in internal Linux daily tests

   on Oct 14, 2021
2. plbossart commented on Oct 14, 2021

   

   plbossart

   on Oct 14, 2021

   Member

   More actions

   Not a real error. IIRC in the past we had an error log only when the last iteration failed, was this modified @ujfalusi @ranj063 ?
3. ujfalusi commented on Oct 14, 2021

   

   ujfalusi

   on Oct 14, 2021

   Contributor

   More actions

   I think we should drop the SOF_DBG_PRINT_ALL_DUMPS flag for sof_debug, that will filter this out.
4. XiaoyunWu6666 commented on Oct 15, 2021

   

   XiaoyunWu6666

   on Oct 15, 2021

   ContributorAuthor

   More actions

   hi @plbossart @ujfalusi
   thanks for advice , @1994lwz has removed the SOF_DBG_PRINT_ALL_DUMPS flag from /etc/modprobe.d on ICL and GLK platforms , manually tried it and found error had gone
   Will observe daily test more and close this issue if it's good
5. lgirdwood commented on Oct 15, 2021

   

   lgirdwood

   on Oct 15, 2021

   Member

   More actions

   @XiaoyunWu6666 this is the ROM reporting an ECC error IIUC. @mwasko ?

   [ 4564.288212] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: fw_state: SOF_FW_BOOT_IN_PROGRESS (2)
   [ 4564.288231] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: unknown ROM status value 80000000
   [ 4564.288257] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: extended rom status:  0x80000000 0xecc00301 0x0 0x0 0x0 0x0 0x0 0x0
   
6. marc-hb commented on Oct 25, 2021

   

   marc-hb

   on Oct 25, 2021

   Collaborator

   More actions

   Note this was already discussed one year ago in #3395 and other pages linked from there.

   When filing a bug like please also share the sof-test output (copied below)

   Not a real error. IIRC in the past we had an error log only when the last iteration failed, was this modified @ujfalusi @ranj063 ?

   sof-test trusts the kernel log level. If this not an error but just a warning, why is it logged at the error level?

   so errors in initial phases should be demoted to warnings or just debug messages.

   +1

   I think we should drop the SOF_DBG_PRINT_ALL_DUMPS flag for sof_debug, that will filter this out.

   Wouldn't we miss logs when there is actually an issue then?

   declare -- cmd="journalctl_cmd --since=@1634164731"
   2021-10-13 22:39:02 UTC [ERROR] Caught kernel log error
   ===========================>>
   [ 4564.288196] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: ------------[ DSP dump start ]------------
   [ 4564.288212] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: fw_state: SOF_FW_BOOT_IN_PROGRESS (2)
   [ 4564.288257] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: extended rom status:  0x80000000 0xecc00301 0x0 0x0 0x0 0x0 0x0 0x0
   [ 4564.288268] kernel: sof-audio-pci-intel-apl 0000:00:0e.0: ------------[ DSP dump end ]------------
   <<===========================
   2021-10-13 22:39:02 UTC [REMOTE_ERROR] Caught error in kernel log
   2021-10-13 22:39:02 UTC [REMOTE_ERROR] Starting func_exit_handler(), exit status=1, FUNCNAME stack:
   2021-10-13 22:39:02 UTC [REMOTE_ERROR]  die()  @  /home/ubuntu/sof-test/test-case/../case-lib/lib.sh
   2021-10-13 22:39:02 UTC [REMOTE_ERROR]  main()  @  /home/ubuntu/sof-test/test-case/check-runtime-pm-status.sh:129
   
7. aiChaoSONG commented on Oct 26, 2021

   

   aiChaoSONG

   on Oct 26, 2021

   Collaborator

   More actions

   potential fix: thesofproject/linux#3238
   The concept here is to log warning at booting retry that are not fatal.

   👍React with 👍1Reacted by Marc Herbert
8. ujfalusi commented on Oct 26, 2021

   

   ujfalusi

   on Oct 26, 2021

   Contributor

   More actions

   @marc-hb, yes, we will if the SOF_DBG_PRINT_ALL_DUMPS is not set, which was the whole point, to hide the non fatal debug prints from a normal user.
9. marc-hb commented on Oct 26, 2021

   

   marc-hb

   on Oct 26, 2021

   Last edited by marc-hb

   Collaborator

   More actions

   @marc-hb, yes, we will if the SOF_DBG_PRINT_ALL_DUMPS is not set, which was the whole point, to hide the non fatal debug prints from a normal user.

   SOF_DBG_PRINT_ALL_DUMPS seems to control how much gets printed. This issue is not about how much gets printed. This issue is about an incorrect log level where a retry warning is incorrectly reported as a final error. The verbosity of the logs should not affect their log level, these must be independent from each other.

   Moreover, BOTH warnings and errors should be kept after turning off debug logs.
10. ujfalusi commented on Oct 27, 2021

    

    ujfalusi

    on Oct 27, 2021

    Contributor

    More actions

    @marc-hb, yes, we will if the SOF_DBG_PRINT_ALL_DUMPS is not set, which was the whole point, to hide the non fatal debug prints from a normal user.

    SOF_DBG_PRINT_ALL_DUMPS seems to control how much gets printed. This issue is not about how much gets printed. This issue is about an incorrect log level where a retry warning is incorrectly reported as a final error. The verbosity of the logs should not affect their log level, these must be independent from each other.

    Right, the panics and dumps level is determined from the fw pow, a DSP crash is fatal from the DSP pow, it is another thing that from Linux (in case of HDA) we will give it another run in hopes that this time it is going to make it.
    Having said that, I agree that the levels for these should be refocused from the kernel's pow, so move it lower.

    Moreover, BOTH warnings and errors should be kept after turning off debug logs.

    You have lost me here. Where do you want to keep never printed logs and how long?
11. marc-hb commented on Oct 27, 2021

    

    marc-hb

    on Oct 27, 2021

    Collaborator

    More actions

    Right, the panics and dumps level is determined from the fw pow,

    Is the kernel aware of FW log levels?

    You have lost me here. Where do you want to keep never printed logs and how long?

    It's very confusing to use symbols with DBG in their name to decide between WARN and ERROR. Log level jam.
12. marc-hb commented on Oct 28, 2021

    

    marc-hb

    on Oct 28, 2021

    Collaborator

    More actions

    Observed again in unrelated thesofproject/sof-test#794 (comment)

    Why are we observing this suddenly again? It's been months and months since #2890. @ujfalusi is this related to some recent change of yours?
13. keqiaozhang commented on Nov 22, 2021

    

    keqiaozhang

    on Nov 22, 2021

    Collaborator

    More actions

    Closing... this issue is not reproducible.
14. keqiaozhang

    closed this as completedon Nov 22, 2021
15. marc-hb commented on Dec 6, 2021

    

    marc-hb

    on Dec 6, 2021

    Collaborator

    More actions

    Important, November 2021 refactoring: thesofproject/linux#3250