buildsystem: kconfig and cmake by jajanusz · Pull Request #845 · thesofproject/sof

jajanusz · 2019-01-15T21:47:05Z

Tools subdir has its own buildsystem and this PR doesn't change it.

Please do not merge until CIs are ready for new buildsystem.
If I missed something please let me know in comments.

Cmake version 3.10 or higher is required, it comes with ubuntu 18.04

Docs are coming soon.
Examples below show how to use new buildsystem.
CMake unlike autotools is designed for out-of-tree builds, that's why you should make separate dirs for your configurations (you can manage builds for many confs/arch/platforms from the one source that way).

All snippets below assume that at beginning your working dir is repo root.

Build fw for apollolake with xtensa toolchain:

mkdir build_fw && cd build_fw
cmake -DTOOLCHAIN=xt -DROOT_DIR=$CONFIG_PATH/xtensa-elf -DMEU_PATH=$MEU_PATH -DMEU_PRIVATE_KEY=$PRIVATE_KEY_PATH ..
make apollolake_defconfig
# if you want to customize FW you can type make menuconfig to launch Kconfig editor
make bin -j4
# sof_apl.ri will be in the root of build dir, rimage is built in the background and used to build fw

Build and run unit tests:

mkdir build_ut && cd build_ut
cmake -DTOOLCHAIN=xt -DROOT_DIR=$CONFIG_PATH/xtensa-elf -DBUILD_UNIT_TESTS=ON ..
make apollolake_defconfig
make -j4 && ctest -j8
# ctest is a part of CMake toolset
# cmocka is fetched + built in background and used to run unit tests

Build and install testbench:

mkdir build_host && cd build_host
cmake -DBUILD_HOST=ON -DCMAKE_INSTALL_PREFIX=install ..
make -j4
make install
# testbench should be in <path_to_repo>/build/install/bin
# all supported libs should be in <path_to_repo>/build/install/lib

If you have ninja installed you can use it instead of make. Just type cmake -GNinja ... while doing configuration step. Ninja is usually faster than make, especially for incremental builds.

Implementation for #228
Tested with CMake versions:
3.10.2 (default for Ubuntu 18.04)
3.13.3 (current)

jajanusz · 2019-01-15T21:51:22Z

@lgirdwood
I decided against subplatforms choices in initial versions for KBL and SKL, they differ only in machine name passed to rimage, so I moved FW_NAME to src/platform/Kconfig.
IMHO We should focus more on building for features instead of platforms/subplatforms. Anyway I can always add this subplatform choice if you still want it, just left it out for now to make things easier.

jajanusz · 2019-01-15T21:53:33Z

Missing: make tarball

@lgirdwood
Missing: something like make install_headers ?
Should FW install some headers? If so then which one?
ATM install target works only for testbench build for bin and libs installation.

jajanusz · 2019-01-15T21:55:34Z

@xiulipan
If I want to make it work on travis CI do I have to just edit xtensa-build-all.sh or something more?
What about docker stuff? Will you take care of it?

xiulipan · 2019-01-16T04:21:20Z

@jajanusz
Actually sof-ci and travis-ci are all using the scripts. I suggest you to just make the change to scripts and readme in this PR as well to make sure CI can check.

Only cmake is needed?
PS: please give me the apt dependency I will update the docker image first. Then could you help me try the docker image in your local first?

jajanusz · 2019-01-16T05:24:22Z

@jajanusz
Actually sof-ci and travis-ci are all using the scripts. I suggest you to just make the change to scripts and readme in this PR as well to make sure CI can check.

Only cmake is needed?
PS: please give me the apt dependency I will update the docker image first. Then could you help me try the docker image in your local first?

You need Python 3 and Cmake 3.10+.

Something like sudo apt-get install cmake=3.10.2-1ubuntu2 should work. If you cannot get it from apt you can also build newest version from source: https://cmake.org/install/

xiulipan · 2019-01-16T06:47:10Z

@jajanusz
Thanks for info. I will re-check travis.yml and update the docker image.
Then you can update scripts in this PR to see if the travis could work.

xiulipan · 2019-01-16T10:46:32Z

@jajanusz
I have updated the dokcer image used for travis-ci as #852
You can add the change to scripts and update this PR to let travis-ci pre check.

lgirdwood

Good work, looks like it's almost ready to merge.

lgirdwood · 2019-01-16T10:20:56Z

So I think the 6 above options are not user configurable but should be enabled/disabled by the platform selection (that same way Linux does this). e.g.

config MEM_WND bool

and the any platforms that use memory windows would select it

config PLATFORM_APL bool "build APL blah" select CONFIG_MEM_WND etc....

I will make it like you wrote. It looks like these selects in Linux are used mostly for HAVE_* configs, because they declare capabilites of arch/platform, however I think in the future we should make Kconfig files less platform-oriented.

Problem is in how you define user. What you wrote is good for someone that want to run feature-customized version of FW on already defined platform, but for adding new platform with different HW capabilites it may become unmanagable.

Maybe we should aim for definitions based on multiple configs where you have base+diffs (something more like zephyr than linux).

@jajanusz Might be easier to have a complete derived config for a platform/board, easier to read complete recipe. Base rather does not change, so no issue with multiple edits.

Another note: how about storing platform/board defs in /platforms, or /boards, not that deep under arch (what about heterogeneous plaforms)?

@mmaka1 is right that storing it in arch may not be the best idea (even if it's how it's done in kernel)

@mmaka1 @lgirdwood We can always easily change things like these when PR is merged. Kconfig/Cmake are just tools, we can use them however we want for our project.

Fwiw, Linux does store it's def_configs under arch//configs/ but I don't think we need to do this atm (as Linux easily has hundreds).
@jajanusz yeah, I'm keen to merge and sort out any integration issues after merging, that way all the others can try/test and provide feedback.
@mmaka1 do you approve ? if so I will merge.

@jajanusz if you are also happy for merging now pls change the PR title to remove the DO NOT MERGE. Thanks.

@lgirdwood
Still waiting for person responsible for CI to update Internal CI.
I'll let you know when it's done.

lgirdwood · 2019-01-16T10:22:28Z

menu "Xtensa Architecture" ?

lgirdwood · 2019-01-16T10:23:01Z

depends on CONFIG_CAVS ?

lgirdwood · 2019-01-16T11:15:43Z

We probably want to prefix platform with PLATFORM_

I can do this, but I'd have also to change all #ifdefs from CONFIG_APOLLOLAKE to CONFIG_PLATFORM_APOLLOLAKE etc (and create even more mess) . I wanted more direct replacement of autotools for initial support. Cannot we keep it as it is ATM?

ok, we can refine this later.

lgirdwood · 2019-01-16T11:19:23Z

Do we have a menu to select SRC, volume, mixer components ?

No because there were no CONFIG_ define for it in old buildsystem (or I missed it?).
Anyway you can always easily add it in src/audio/Kconfig

ok, we will ad that later.

jajanusz · 2019-01-16T23:50:15Z

@lgirdwood
Updated with your suggestions.
Added some fixes to remove possible warnings in CMake 3.13.3
Changed defconfigs to be minimal (like result of make savedefconfig in linux)

ranj063 · 2019-01-17T07:47:19Z

@singalsu i havent had the time to test the testbench with this change. have you tried it?

Signed-off-by: Janusz Jankowski <janusz.jankowski@linux.intel.com>

This commit doesn't touch tools buildsystem. Signed-off-by: Janusz Jankowski <janusz.jankowski@linux.intel.com>

overrideconfig is custom target that can be used to set config values from scripts Signed-off-by: Janusz Jankowski <janusz.jankowski@linux.intel.com>

Signed-off-by: Janusz Jankowski <janusz.jankowski@linux.intel.com>

Change DEBUG_BUILD define to use Kconfig's CONFIG_DEBUG Signed-off-by: Janusz Jankowski <janusz.jankowski@linux.intel.com>

Signed-off-by: Janusz Jankowski <janusz.jankowski@linux.intel.com>

xiulipan · 2019-01-22T05:34:05Z

@jajanusz

First question here, is kconfig used by cmake?
Or kconfig is something work with cmake.
Why not use the pure cmake style build system?

Then some small question:

1.For some old platforms rom build is not supported, in old build system we can ignore the rom flag is the platform is not support. Could this be support to make -a -r options work for xtensa-build-all.sh.
I think some check is needed in CMake.

make[3]: *** No rule to make target '../src/platform/baytrail/rom.x.in', needed by 'rom.x'.  Stop.
CMakeFiles/Makefile2:718: recipe for target 'src/arch/xtensa/CMakeFiles/ld_script_rom.x.dir/all' failed
make[2]: *** [src/arch/xtensa/CMakeFiles/ld_script_rom.x.dir/all] Error 2
make[2]: *** Waiting for unfinished jobs....

update build will fail due to lots of reason. git clean will fix but make clean won't.
It seems the cmake still use some old build system cache.
Please check:

[  6%] Building ASM object src/arch/xtensa/up/xtos/CMakeFiles/xtos.dir/core-restore.S.obj
In file included from /home/pxl/work/sof/sof/src/arch/xtensa/include/xtensa/config/core.h:50,
                 from /home/pxl/work/sof/sof/src/arch/xtensa/include/xtensa/coreasm.h:52,
                 from /home/pxl/work/sof/sof/src/arch/xtensa/up/xtos/core-restore.S:25:
/home/pxl/work/sof/sof/src/platform/baytrail/include/arch/xtensa/config/core-isa.h:8:2: error: #error "No ISA configuration selected"
 #error "No ISA configuration selected"
  ^~~~~
In file included from /home/pxl/work/sof/sof/src/arch/xtensa/up/xtos/core-restore.S:32:
/home/pxl/work/sof/sof/src/platform/baytrail/include/arch/xtensa/config/core-isa.h:8:2: error: #error "No ISA configuration selected"
 #error "No ISA configuration selected"
  ^~~~~
In file included from /home/pxl/work/sof/sof/src/arch/xtensa/up/xtos/core-restore.S:34:
/home/pxl/work/sof/sof/src/arch/xtensa/up/xtos/xtos-internal.h:375:2: error: #error CALL8-only is not supported!
 #error CALL8-only is not supported!
  ^~~~~

/home/pxl/work/sof/sof/src/arch/xtensa/boot_loader.c: In function 'boot_master_core':
/home/pxl/work/sof/sof/src/arch/xtensa/boot_loader.c:251:11: error: implicit declaration of function 'hp_sram_init'; did you mean 'list_init'? [-Werror=implicit-function-declaration]
  result = hp_sram_init();
           ^~~~~~~~~~~~
           list_init
cc1: all warnings being treated as errors
src/arch/xtensa/CMakeFiles/bootloader.dir/build.make:78: recipe for target 'src/arch/xtensa/CMakeFiles/bootloader.dir/boot_loader.c.obj' failed

A little modification to split xcc and gcc bulid with different build folder will better.

xiulipan · 2019-01-22T05:35:41Z

+	USES_TERMINAL
+)
+
+if(CONFIG_BUILD_VM_ROM)


Please check the platform and compiler used here.
To avoid wrong use and build all use chaos.

Just change config. It shouldn't be in buildsystem, if you don't want ROM just dont set CONFIG_BUILD_VM_ROM to 'y'.

Oh, I mean if people mis-used the CONFIG_BUILD_VM_ROM there will be an error. So have another check in CMake will be better.

Then it should be set in Kconfig files (not cmake) that BUILD_VM_ROM depends on HAVE_ROM and for example platform APOLLOLAKE selects HAVE_ROM. However for me it looks that it's not something bound to platform, more like it's not supported by our CI/qemu/something external yet. Or am I wrong?

xiulipan · 2019-01-22T05:36:22Z

 	fi

 	# only delete binary related to this build
 	rm -fr src/arch/xtensa/sof-$j.*


This line is not needed.

More like HOST is not needed, i need toolchain name that is 'xt' or 'xtensa-*-elf'

I mean rm -fr src/arch/xtensa/sof-$j.* is not needed.
The build result is in the build_*, right?

Yes you are right.

xiulipan · 2019-01-22T05:37:00Z

-		--with-root-dir=$ROOT --host=$HOST --enable-debug=$BUILD_DEBUG \
-		CC=$XCC OBJCOPY=$XTOBJCOPY OBJDUMP=$XTOBJDUMP \
-		--with-dsp-core=$XTENSA_CORE $BUILD_ROM
+	rm -fr build_$j


can we also include gcc or xcc in the build dir?

I don't understand, you specify which one to use with TOOLCHAIN, it's like CROSS_COMPILE in kernel that sets tools in gcc convention (+ in our case sets some cmake specific things).

yes, so can we parse the name and make the build dir for gcc looks like build_byt_gcc?
In this case, we can compare gcc and xcc build very easily.

Yes you can do that. However I'm not sure what you want to do, so maybe it would be better if you just update scripts when this buildsystem is merged.

jajanusz · 2019-01-22T09:32:41Z

First question here, is kconfig used by cmake?
Or kconfig is something work with cmake.
Why not use the pure cmake style build system?

Cmake and Kconfig are 2 different things. Cmake is buildsystem, kconfig is configuration tool (+ defines generator etc), just like in linux/zephyr.

1.For some old platforms rom build is not supported, in old build system we can ignore the rom flag is the platform is not support. Could this be support to make -a -r options work for xtensa-build-all.sh.
I think some check is needed in CMake.

It should be done in script then if you want to disable it. It's user fault if he passes wrong config.

update build will fail due to lots of reason. git clean will fix but make clean won't.
It seems the cmake still use some old build system cache.

You usually don't use make clean in cmake, take advantage of out-of-tree build and just remove build dir.

A little modification to split xcc and gcc bulid with different build folder will better.

Just use differnt build directory.

I'll prevent users from doing in-tree build in next PR, to avoid issues exactly like these (other cmake projects do it also).

xiulipan · 2019-01-22T09:35:07Z

@jajanusz OK, I will update the scripts later.

jajanusz · 2019-01-22T10:06:36Z

@jajanusz OK, I will update the scripts later.

Thanks

lgirdwood · 2019-01-22T14:42:03Z

@jajanusz any update ? Are we good to merge ?

jajanusz · 2019-01-22T15:34:19Z

@jajanusz any update ? Are we good to merge ?

Yes please, only jenkins ci fails, but @xiulipan will take care of scripts after this PR

jajanusz requested review from RanderWang, bardliao, lbetlej, lgirdwood, libinyang, mmaka1, plbossart, ranj063, singalsu, tlauda and xiulipan as code owners January 15, 2019 21:47

jajanusz modified the milestone: 1.3 Jan 15, 2019

xiulipan mentioned this pull request Jan 16, 2019

scripts: pre update sof docker for cmake covert #852

Merged

lgirdwood reviewed Jan 16, 2019

View reviewed changes

jajanusz force-pushed the kconfig-py-cmake branch from 650706f to fbe0f75 Compare January 16, 2019 23:47

lgirdwood approved these changes Jan 17, 2019

View reviewed changes

mmaka1 approved these changes Jan 17, 2019

View reviewed changes

jajanusz force-pushed the kconfig-py-cmake branch 3 times, most recently from 73c9186 to 33acee7 Compare January 18, 2019 13:06

jajanusz added 13 commits January 21, 2019 17:24

cmake: add testbench host build

d64f9bd

Signed-off-by: Janusz Jankowski <janusz.jankowski@linux.intel.com>

cmake: add CMakeLists for unit tests

2361c7d

Signed-off-by: Janusz Jankowski <janusz.jankowski@linux.intel.com>

test: debugability: fix for full paths

b1c022c

Signed-off-by: Janusz Jankowski <janusz.jankowski@linux.intel.com>

cmake: add doc CMakeLists

342bf32

Signed-off-by: Janusz Jankowski <janusz.jankowski@linux.intel.com>

cmake: defconfigs support

cff2cf2

Signed-off-by: Janusz Jankowski <janusz.jankowski@linux.intel.com>

buildsystem: remove autotools from sof

7e29a01

This commit doesn't touch tools buildsystem. Signed-off-by: Janusz Jankowski <janusz.jankowski@linux.intel.com>

cmake: add overrideconfig

c201b59

overrideconfig is custom target that can be used to set config values from scripts Signed-off-by: Janusz Jankowski <janusz.jankowski@linux.intel.com>

cmake: vm rom build

c7590d5

Signed-off-by: Janusz Jankowski <janusz.jankowski@linux.intel.com>

cmake: debug build support

3059d46

Change DEBUG_BUILD define to use Kconfig's CONFIG_DEBUG Signed-off-by: Janusz Jankowski <janusz.jankowski@linux.intel.com>

cmake: update xtensa-build-all.sh

3bb75f9

Signed-off-by: Janusz Jankowski <janusz.jankowski@linux.intel.com>

cmake: update host-build-all.sh

d2c4f8f

Signed-off-by: Janusz Jankowski <janusz.jankowski@linux.intel.com>

cmake: update qemu-check.sh

e80713b

Signed-off-by: Janusz Jankowski <janusz.jankowski@linux.intel.com>

icl: increase text section size

fa3bc5a

Signed-off-by: Janusz Jankowski <janusz.jankowski@linux.intel.com>

jajanusz force-pushed the kconfig-py-cmake branch from b01d9a0 to fa3bc5a Compare January 21, 2019 16:26

This was referenced Jan 21, 2019

build: update for cmake thesofproject/sof-docs#79

Merged

unit tests: update for cmake thesofproject/sof-docs#80

Merged

xiulipan suggested changes Jan 22, 2019

View reviewed changes

xiulipan approved these changes Jan 22, 2019

View reviewed changes

jajanusz changed the title ~~[DO NOT MERGE] buildsystem: kconfig and cmake~~ buildsystem: kconfig and cmake Jan 22, 2019

lgirdwood merged commit 297958f into thesofproject:master Jan 22, 2019

xiulipan mentioned this pull request Jan 23, 2019

Build result changes from sof-apl.ri to sof_apl.ri #869

Closed

This was referenced Jan 23, 2019

Make building xtensa(UP) on SMP platforms possible #850

Merged

readme: update for cmake #876

Merged

Replace autotools with Kconfig #228

Closed

Conversation

jajanusz commented Jan 15, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jajanusz commented Jan 15, 2019

Uh oh!

jajanusz commented Jan 15, 2019

Uh oh!

jajanusz commented Jan 15, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

xiulipan commented Jan 16, 2019

Uh oh!

jajanusz commented Jan 16, 2019

Uh oh!

xiulipan commented Jan 16, 2019

Uh oh!

xiulipan commented Jan 16, 2019

Uh oh!

lgirdwood left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jajanusz Jan 16, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jajanusz commented Jan 16, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ranj063 commented Jan 17, 2019

Uh oh!

xiulipan commented Jan 22, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jajanusz Jan 22, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

jajanusz commented Jan 15, 2019 •

edited

Loading

jajanusz commented Jan 15, 2019 •

edited

Loading

jajanusz Jan 16, 2019 •

edited

Loading

jajanusz commented Jan 16, 2019 •

edited

Loading

jajanusz Jan 22, 2019 •

edited

Loading

jajanusz commented Jan 22, 2019 •

edited

Loading