[BUG] Zephyr baseline update blocked

Describe the bug
Updating SOF main to Zephyr main shows issues in SOF CI tests, blocking merging.

Last good #8797
First failed attempt: #8804 (Zephyr zephyrproject-rtos/zephyr@6a0b1da )
Latest failed attempt: #8813

To Reproduce
Update Zephyr version in SOF west.yml and submit PR to SOF CI.

Reproduction Rate
100%

Expected behavior
SOF test suite passed with latest Zephyr.

Impact
Blocks updates of Zephyr in SOF main.

Environment
See above

Screenshots or console output

Notes
Discovery of this was delayed by an earlier regression spotted in #8747 and later fixed on SOF side (by #8790 ).

Bisect is slowed down by interface change decsribed in zephyrproject-rtos/zephyr#64755

Fixed done on Zephyr side:

Revert "pm: Remove CURRENT_CPU macro" zephyrproject-rtos/zephyr#68234 (UPDATE: replaced by intel_adsp: Fix interruption locks in power management code zephyrproject-rtos/zephyr#67863 )
soc: xtensa: intel_adsp: cavs: fix XCC build zephyrproject-rtos/zephyr#68108
ace: fix DSP panic during startup zephyrproject-rtos/zephyr#67617

Status as of 31th Jan, we have two blocking issues:

failure to power down non-primary core, example log showing SET_DX IPC failure -> https://sof-ci.01.org/sofpr/PR8813/build2413/devicetest/index.html?model=MTLP_RVP_NOCODEC&testcase=check-pause-resume-playback-100
failure in chain-dma stop on Intel cAVS2.5 platforms, SOF-side workaround exists 386a525 , but tracked as [BUG] multiple-pause-resume.sh sof-test fails to FW_GEN_MSG failed with err 3018 #8792

The issues with SMP interface change are still unresolved, but I took a quick peek at latest Zephyr 3.6.0-rc2 and we have more problems:

-fno-lto added -> xtensa toolchain fails
various union initializers that break -> break Intel TGL builds with xtensa toolchain in SOF CI
the cache interface change (to sys_cache) -> requires SOF side changes

I'll look at the toolchain issues.

-fno-lto added -> xtensa toolchain fails

Fix submitted:

cmake/compiler/gcc: use -flto/-fno-lto only for gcc >= 4.5.0 zephyrproject-rtos/zephyr#69058

various union initializers that break -> break Intel TGL builds with xtensa toolchain in SOF CI

Bisected to big commit zephyrproject-rtos/zephyr@25173f71cda630d4fb (zephyrproject-rtos/zephyr#67424)

Déjà vu?

Revert "init: add missing initialization of dev pointer in SYS_INIT (alternative to PR68125) zephyrproject-rtos/zephyr#68118

various union initializers that break -> break Intel TGL builds with xtensa toolchain in SOF CI

Fix submitted:

pm: use braces for .pm_base inside anonymous union initializer zephyrproject-rtos/zephyr#69270

@LaurentiuM1234

@LaurentiuM1234 @dbaluta Any preference how we handle this for SOF2.9? We are still struggling with the SMP interface change on Intel platforms (merged in Zephyr upstream), but the problems only affect multicore configurations. The 2.9 branch point is next week, so we have two options to proceed:
a) fix all the remain issues and update to 3.6-rc for SOF2.9
b) branch SOF2.9 with the current version of Zephyr used by SOF and merge new Zephyr baseline ASAP after stable-v2.9 branch is created

Option (a) is obviously preferrable, but do you see any showstopper with (b)? We can backport Zephyr changes to stable-v2.9 branch, but of course if there's a lot, that's going to be hard.

FYI @marc-hb @lgirdwood .

@dbaluta

Option (a) is obviously preferrable, but do you see any showstopper with (b)? We can backport Zephyr changes to stable-v2.9 branch, but of course if there's a lot, that's going to be hard.

as far as the NXP native switch is concerned there shouldn't be an issue with option (b). Also, as far as I'm aware the only changes to Zephyr (that can impact SOF) we've had lately are all native-related. @dbaluta and @iuliana-prodan can correct me if I'm wrong here.

a) fix all the remain issues and update to 3.6-rc for SOF2.9 b) branch SOF2.9 with the current version of Zephyr used by SOF and merge new Zephyr baseline ASAP after stable-v2.9 branch is created

Option (a) is obviously preferrable, but do you see any showstopper with (b)? We can backport Zephyr changes to stable-v2.9 branch, but of course if there's a lot, that's going to be hard.

Lets try a) and extend code freeze by 1 week. If too many problems then b) will be backup.

#8732 is getting close, but there are still issues (plus 8732 uses unmerged commits from Zephyr). So let's go with option (b) for SOF2.9. We can backport Zephyr upstream fixes to the stable branch used by SOF.

west.yml: update Zephyr to 9d1df132b1c + one revert

Test results look good now #8903 . After code reviews, we can merge this.

We do need one revert, filed following bug in Zephyr to track zephyrproject-rtos/zephyr#69807

west.yml: update Zephyr to 9d1df132b1c + one revert

Baseline now updated via #8903 so we can close this. I filed a new bug #8908 to track the need for the revert.

Navigation Menu

Repository navigation

[BUG] Zephyr baseline update blocked #8818

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[BUG] Zephyr baseline update blocked #8818

Description

Activity

kv2019i commented on Jan 30, 2024

kv2019i commented on Jan 31, 2024

kv2019i commented on Feb 14, 2024

marc-hb commented on Feb 14, 2024

marc-hb commented on Feb 16, 2024

marc-hb commented on Feb 16, 2024

marc-hb commented on Feb 21, 2024

kv2019i commented on Feb 23, 2024

LaurentiuM1234 commented on Feb 23, 2024

lgirdwood commented on Feb 23, 2024

kv2019i commented on Mar 4, 2024

kv2019i commented on Mar 5, 2024

kv2019i commented on Mar 6, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions