Repository navigation
[BUG] Signature issue on TGL UP Extreme i11 - openssl3 #99
Description
Describe the bug
The firmware generated locally on my NUC does not boot on UpExtreme 11. The daily-build firmware and the 2.1 version boot fine, so it's not a hardware/CSME issue, rather an undocumented or missing environment for the signature to work.
Standard Ubuntu 22.04 on my side. Same scripts work fine on Up Extreme (WHL)
To Reproduce
Steps to reproduce the behavior: (e.g. list commands or actions used to reproduce the bug)
scripts/xtensa-build-all.sh -d tgl (debug version)
scripts/xtensa-build-all.sh tgl
Reproduction Rate
100%
Expected behavior
no errors on boot
Impact
showstopper
Environment
-
Branch name and commit hash of the 2 repositories: sof (firmware/topology) and linux (kernel driver).
-
Kernel: 9eaf67f8c40cc (sof/topic/sof-dev) soundwire: intel: set dev_num_ida_min
-
SOF: e9442e04c (HEAD -> main, origin/main) module_adpater: change the buffer size of waves_codec
-
-
Name of the topology file
- Topology: irrelevant
-
Name of the platform(s) on which the bug is observed.
- Platform: TGL UpExtreme i11
Screenshots or console output
[ 7.008951] snd_sof_intel_hda_common:hda_cl_copy_fw: sof-audio-pci-intel-tgl 0000:00:1f.3: FW Poll Status: reg[0x80000]=0x80000012 timedout
[ 7.008957] sof-audio-pci-intel-tgl 0000:00:1f.3: hda_cl_copy_fw: timeout with rom_status_reg (0x80000) read
[ 7.008969] snd_sof_intel_hda_common:hda_dsp_stream_trigger: sof-audio-pci-intel-tgl 0000:00:1f.3: FW Poll Status: reg[0x160]=0x140000 successful
[ 7.008973] sof-audio-pci-intel-tgl 0000:00:1f.3: ------------[ DSP dump start ]------------
[ 7.008974] sof-audio-pci-intel-tgl 0000:00:1f.3: Firmware download failed
[ 7.008975] sof-audio-pci-intel-tgl 0000:00:1f.3: fw_state: SOF_FW_BOOT_IN_PROGRESS (2)
[ 7.008978] snd_sof_intel_hda_common:hda_dsp_get_status: sof-audio-pci-intel-tgl 0000:00:1f.3: unknown ROM status value 80000012
[ 7.008991] sof-audio-pci-intel-tgl 0000:00:1f.3: extended rom status: 0x80000012 0x2c 0x0 0x0 0x0 0x0 0x2510113 0x0
[ 7.008992] sof-audio-pci-intel-tgl 0000:00:1f.3: ------------[ DSP dump end ]------------
[ 7.009016] sof-audio-pci-intel-tgl 0000:00:1f.3: Failed to start DSP
[ 7.009017] sof-audio-pci-intel-tgl 0000:00:1f.3: error: failed to boot DSP firmware -110
[ 7.009020] snd_sof:sof_set_fw_state: sof-audio-pci-intel-tgl 0000:00:1f.3: fw_state change: 2 -> 3
[ 7.061361] snd_sof_intel_hda_common:hda_dsp_core_reset_enter: sof-audio-pci-intel-tgl 0000:00:1f.3: FW Poll Status: reg[0x4]=0x1d003c timedout
[ 7.061368] sof-audio-pci-intel-tgl 0000:00:1f.3: error: hda_dsp_core_reset_enter: timeout on HDA_DSP_REG_ADSPCS read
[ 7.061370] sof-audio-pci-intel-tgl 0000:00:1f.3: error: dsp core reset failed: core_mask 1
[ 7.062148] snd_sof:sof_set_fw_state: sof-audio-pci-intel-tgl 0000:00:1f.3: fw_state change: 3 -> 0
[ 7.062173] sof-audio-pci-intel-tgl 0000:00:1f.3: error: sof_probe_work failed err: -110
Activity
lgirdwood commentedon Jun 14, 2022
@plbossart can you attach your FW. Would be nice to diff it's headers against v2.1.
Btw, I assume you are signing with community key.? Do both v2.1 community and Intel keys work for you ?
lgirdwood commentedon Jun 14, 2022
One more thing, is your rimage submodule up to date ? (and in $PATH)
plbossart commentedon Jun 14, 2022
git log --oneline
b4886bebb (HEAD -> main, origin/main) module_adapter: add zephyr logging support
cd rimage/
git log --oneline
9d45332 (HEAD) Write firmware file micro version to manifest for cAVS platforms
rimage is not in $PATH. It was not needed before and must be set by scripts.
plbossart commentedon Jun 14, 2022
compilation log
log.txt
firmware
sof-tgl.ri.gz
marc-hb commentedon Jun 14, 2022
This reminds me this bug fix: 95d887251ee40397300
fredoh9 commentedon Jun 14, 2022
Interesting, I compared with mine
./sof_ri_info.py ~/Downloads/sof-tgl.ri --erase_vars ~/Downloads/sof-tgl.ri_no_varsmarc-hb commentedon Jun 14, 2022
I implemented
--erase-varsso the image after--erase_varsis 100% identical when using the same toolchain (which you obviously did, otherwise the differences would be much bigger). So I do find these few bytes difference worrying, can you please share thehexdiff?Also, please share the manifest differences as shown by
diff -u <(sof/tools/sof_ri_info.py image1.ri) <(sof/tools/sof_ri_info.py image2.ri). Some randomness is expected becauserimageuses a salt (that's why--erase_varsexists), other differences are not.BTW
--erase-varsis used in CI to test every PR catch any__TIMESTAMP__(in addition to checkpatch)fredoh9 commentedon Jun 14, 2022
Don't know the root cause, but from the build log, Source content hash is different.
This is mine,
This is Pierre's,
plbossart commentedon Jun 14, 2022
@keqiaozhang reported a similar issue worked-around with the "scripts/xtensa-build-all.sh -d tgl" option. In my case the debug option doesn't solve anything.
marc-hb commentedon Jun 14, 2022
This is the very first difference and the one we must focus on first. All other differences could be impacted by this. The source hash is not affected by any of these:
-doption does not affect the source hash.So there is really, absolutely no reason for @plbossart's source hash to be different. I have the same
1c511e77source hash as @fredoh9 with any toolchain.fredoh9 commentedon Jun 14, 2022
if
-dmake difference, this is more serious bug.Good thing is it worked for both for me, it didn't work for both for Pierre. I like the results, consistency, at least.
marc-hb commentedon Jun 14, 2022
@plbossart can you please first make sure that
git status --ignoredis clean (sorry for asking this but we're grasping at straws now), then run the following commands and report which ones don't matchgit ls-files src/ scripts/ | md5sum 72028df5e905505089a10e92e4d94781 -fredoh9 commentedon Jun 14, 2022
@marc-hb, I built without -d, I have 100% same with yours above
plbossart commentedon Jun 14, 2022
@marc-hb I removed all ignored files and rebuilt, same SHA1
37 remaining items
aiChaoSONG commentedon Jun 16, 2022
@marc-hb I think I signed the wrong binary (build_tgl_xcc/sof), that's why I have assertion failure. if I use the correct binary (build_tgl_xcc/src/arch/xtensa/sof-tgl), no failure at all. thanks Jaska to correct me.
juimonen commentedon Jun 16, 2022
@aiChaoSONG & @plbossart can you try: #97
aiChaoSONG commentedon Jun 16, 2022
Manually tried #97, firmware can boot on UpExtreme, file a SOF test PR to test the rimage PR on other platforms except ACE: thesofproject/sof#5928
plbossart commentedon Jun 16, 2022
Confirmed fix, works for me as well. Thanks @juimonen
marc-hb commentedon Jun 16, 2022
@plbossart , @aiChaoSONG and anyone with openssl3, could you run
sof_ri_info.pyagain with the tentative rimage fix and compare the output withopenssl2openssl1 again?For
openssl2openssl1 you can either download it from here or use the Docker container.cmake: make sure sha1 is computed without filters
aiChaoSONG commentedon Jun 17, 2022
SOF commit: 1083bc2a39fbfcf4056ece5983ff00532e370179
Without #97, rimage commit: 9d45332
xcc_ri_info_with_ssl3.txt
gcc_ri_info_with_ssl3.txt
gcc_ri_info_with_ssl1.txt
With #97
xcc_ri_info_with_ssl3_and_rimage_fix.txt
gcc_ri_info_with_ssl3_and_rimage_fix.txt
[-][BUG] Signature issue on TGL UP Extreme i11[/-][+][BUG] Signature issue on TGL UP Extreme i11 - openssl3[/+]marc-hb commentedon Jun 17, 2022
The difference between
... is not bigger than the difference between:
So maybe everything is OK? I wish I knew or remembered why there is so much
cavs0015.met (ADSP Metadata File Extension)difference but life is too short... @juimonen , @lgirdwood , any idea?lgirdwood commentedon Jun 20, 2022
@plbossart rimage fix now merged. Can you try and we can close if good ?
aiChaoSONG commentedon Jun 27, 2022
#97 fix confirmed
marc-hb commentedon Jan 11, 2023
I'm reverting the
--no-filtersin thesofproject/sof/pull/6920 because it's not compatible with Windows. Long story in thesofproject/sof/pull/6920