Repository navigation
[FEATURE] remove SOF build time dependency to rimage headers #8178
Description
Is your feature request related to a problem? Please describe.
rimage was originally moved to a different repository as it is an independent tool from SOF.
With changes since SOF2.4, we have more and more direct dependencies from SOF to rimage. IPC4 protocol requires metadata that is defined in firmware manifest (data maintained in toml files, rimage encodes this to firmware binary file).
Most recently, loadable module support was added to SOF firmware and this code depends on firmware file structure definitions that are needed both by rimage and the new SOF module loader.
The build dependencies are causing issues to downstream SOF users (e.g. zephyrproject-rtos/zephyr#62262 ).
Having the rimage manifest changes (like thesofproject/rimage#177) not tested by full SOF CI, is causing operational issues (problems caused by rimage changes are detected too late).
Describe the solution you'd like
Merge rimage back to sof repository as it is now more integrate part of SOF.
Describe alternatives you've considered
- alternative: duplicate manifest headers in SOF (these are part of the loadable modules ABI)
- cons: does not help with SOF CI woes with rimage PRs
cc:
- ipc4: add get &set large config support #4808
- [RFC] rimage: Add support for multiple toml file rimage#179
- copy zephyr based sof image toml files from rimage to sof and split to platform and module toml #8243
- [FEATURE] Move logger/trace (and debug tools?) to other repository #2480
- [FEATURE] use west instead of submodules. #7901
Activity
kv2019i commentedon Sep 7, 2023
FYI @nashif @lgirdwood @marc-hb @softwarecki @aiChaoSONG
marc-hb commentedon Sep 7, 2023
cc: @andyross , @dbaluta , @iuliana-prodan , @dcpleung
Anyone remembers the rationale for a separate rimage repo? We don't want to go back and forth and back. I know booting Zephyr tests on Intel/NXP hardware is possible with rimage and without SOF. Could have this been a reason?
Having more git repos for no good reason makes everything more complicated. Among others and as mentioned above, it makes CI especially more complicated: how do you submit an rimage change (or Zephyr) to SOF testing? You can:
... BUT it's tedious and experience shows most people don't take the time.
EDIT: the complexity of splitting rimage was not missed in #2086 but it happened anyway.
https://en.wikipedia.org/wiki/Conway%27s_law aside (does not apply here), the ability to make arbitrary combinations is probably the only good reason to have multiple, independent repos. For instance we want the latest SOF-TEST branch to be able to test most SOF branches out of the box. We don't want to branch SOF-TEST every time that we branch SOF, that would be much more work. Obviously, you also don't want SOF-TEST code to change while bisecting SOF. So: a different repo for SOF-TEST. It's different for lower level CMocka unit tests which are in the same repo.
Similarly, we want the latest SOF docker image to be able to test any SOF branch, which shows
sof/scripts/docker_build/should be in a separate repo:Back to rimage: do we need the ability to freely combine any rimage version with any SOF version? I guess not.
PS: https://en.wikipedia.org/wiki/Monorepo (@ Google, Facebook, Microsoft)
cc:
plbossart commentedon Sep 7, 2023
2019 change in 9495677
At some point there was a decision that tools don't belong in the SOF tree. That was after we merged them into SOF from https://github.com/thesofproject/soft to avoid the sort of problems we now have.
This clearly means we don't have a clear idea of what we are doing :-)
marc-hb commentedon Sep 7, 2023
Below is a gitarcheology of the past moves with links to the corresponding discussions. I hope I did not miss anything.
The most relevant link is #2086 , this is where most of the rimage split discussion took place.
October 2016: creation of new sof-tools.git with rimage import as the first commit
February 2018: rimage is moved from sof-tools.git/rimage/ to sof.git/rimage/ to "remove a cyclic dependency between these two repos"
Git history is not transferred... yet! See next step.
The sof-tools.git/tools/rimage/keys/ subdirectory is copied but not removed and also left behind for some unknown reason.
November 2018: the rest of sof-tools.git follows and is also merged into sof.git to "reduce complexity".
This time the sof-tools git history is merged into sof.git thanks to @cujomalainey. This funnily enough resurrects the old sof-tools.git/rimage/ history in sof.git: https://github.com/thesofproject/sof/commits/72d797d9ddae7
sof-tools.git is frozen/archived.
--- SHORT-LIVED, SINGLE SOF REPO NIRVANA !? --
Nov 2019 - May 2020: rimage split from sof.git to a new rimage submodule (ick) for (at least) some Zephyr reason:
rimage history is not transferred.
In the links above I think I spotted some "plbossart" guy who already said that at the time.
marc-hb commentedon Sep 13, 2023
In a meeting it was noted that rimage has recently gained more features which has made it more SOF-dependent than it was when it was split in May 2020.
So I ran some basic
git log --oneline ... | wccommands on https://github.com/thesofproject/sof/commits/main/rimage and https://github.com/thesofproject/rimage/commits/main to quickly gauge therimageactivity. It has roughly doubled since January 2023:Between June 2020 (right after it was split) and January 2023
Since January 2023:
Of course these are just commit numbers, they don't directly measure dependencies.
cujomalainey commentedon Sep 15, 2023
Thanks for the history @marc-hb I agree, anytime we are expressing the same information across multiple repos we are in for a bad time. Now that the manifest contains data from the modules I think the split is not acceptable anymore. In reality I would rather the manifest derive from the C definition of the component so it's a single file but that is a different problem entirely.
kv2019i commentedon Sep 19, 2023
FYI @pjdobrowolski
aiChaoSONG commentedon Sep 19, 2023
IMHO: rimage could be in a separate repo, or we could move it under sof/tools, just like other tools, but those toml files should stay with sof.
I have implemented the code to copy Intel IPC4 toml files to sof/config, split to platform related toml to
sof/config/platform/, module related toml to sof/config/module/`, files are still named as PLATFORM.toml and update xtensa-build-zephyr.py build script.Further split could be done for module toml, we could have one toml for each module, and only include the toml if it is enabled in kconfig. I could raise an RFC once #8212 got merged.
abonislawski commentedon Sep 19, 2023
Looks like a good idea, rimage repo + in SOF toml per module, just like all the rest module files in SOF, no need to maintain one giant toml in another repo
marc-hb commentedon Sep 20, 2023
More .toml flexibility sounds good no matter what. So it's hopefully off-topic here.
EDIT: I take "off-topic" back :-(
kv2019i commentedon Sep 25, 2023
I'm tagging this as "unclear" as we don't have a consensus whether this is a problem to begin with.
lgirdwood commentedon Sep 29, 2023
@nashif do you need rimage moved into SOF with unified headers ?
19 remaining items
cmake: add new ${RIMAGE_TOP} constant
zephyr/cmake: add new ${RIMAGE_TOP} constant
rimage: move everything down to subdir tools/rimage/, prepare move to…
Move complete rimage git history back into sof/tools/rimage
Switch Zephyr build to new sof/tools/rimage location
Switch smex and sof-logger to new sof/tools/rimage location
cmake: switch XTOS build to sof/tools/rimage
Switch Zephyr build to new sof/tools/rimage location
Switch smex and sof-logger to new sof/tools/rimage location
cmake: switch XTOS build to sof/tools/rimage
marc-hb commentedon Oct 9, 2023
rimage transfer ready for prime time in:
kv2019i commentedon Oct 11, 2023
#8297 merged, so this feature request is now moot (build time dependency within one repository is ok).
cmake: block compilation of the 'main' rimage branch
cmake: block compilation of the 'main' rimage branch