Skip to content

[BUG] compare-linux-win CI build check fails due to zephyr_version.h difference #9797

Closed
Closed
[BUG] compare-linux-win CI build check fails due to zephyr_version.h difference#9797
Assignees
kv2019i
Labels
Known PR FailuresIssues seen in SOF github pull-request checksP3Low-impact bugs or featuresbugSomething isn't working as expected

Description

@kv2019i
kv2019i
opened on Jan 27, 2025

Status update:

Starting with daily build 795, https://github.com/thesofproject/sof/actions/runs/12940934767
the Linux + mnft (manifest) builds started to lose git tags in the git describe output that goes into their BUILD_VERSION. For instance v4.0.0-2813-g42701fdb2729 became 42701fdb2729. The Windows builds kept it and the Linux "zmain" (Zephyr main branch) kept it too.
This correlate with a Ubuntu VM upgrade, see below.

Describe the bug
Starting today 27th Jan 2025, multiple pull-requests are showing a failure in build reproducibility between Linux and Windows builds with
error:

https://github.com/thesofproject/sof/actions/runs/12947318306/job/36114439673?pr=9794

 Files linux-build  imx8 imx8x imx8m imx8ulp/build-sof-staging/sof-info/imx8/stripped-zephyr.elf and windows-build  imx8 imx8x imx8m imx8ulp/build-sof-staging/sof-info/imx8/stripped-zephyr.elf differ
Files linux-build  imx8 imx8x imx8m imx8ulp/build-sof-staging/sof-info/imx8/zephyr.lst and windows-build  imx8 imx8x imx8m imx8ulp/build-sof-staging/sof-info/imx8/zephyr.lst differ
Files linux-build  imx8 imx8x imx8m imx8ulp/build-sof-staging/sof-info/imx8/zephyr_version.h and windows-build  imx8 imx8x imx8m imx8ulp/build-sof-staging/sof-info/imx8/zephyr_version.h differ
Files linux-build  imx8 imx8x imx8m imx8ulp/build-sof-staging/sof-info/imx8m/stripped-zephyr.elf and windows-build  imx8 imx8x imx8m imx8ulp/build-sof-staging/sof-info/imx8m/stripped-zephyr.elf differ
Files linux-build  imx8 imx8x imx8m imx8ulp/build-sof-staging/sof-info/imx8m/zephyr.lst and windows-build  imx8 imx8x imx8m imx8ulp/build-sof-staging/sof-info/imx8m/zephyr.lst differ
Files linux-build  imx8 imx8x imx8m imx8ulp/build-sof-staging/sof-info/imx8m/zephyr_version.h and windows-build  imx8 imx8x imx8m imx8ulp/build-sof-staging/sof-info/imx8m/zephyr_version.h differ
Files linux-build  imx8 imx8x imx8m imx8ulp/build-sof-staging/sof-info/imx8ulp/stripped-zephyr.elf and windows-build  imx8 imx8x imx8m imx8ulp/build-sof-staging/sof-info/imx8ulp/stripped-zephyr.elf differ

This is shown in multiple open PRs:

To add to the mystery, I can't see the failure in any of the pull requests failed last week.

To Reproduce
Submit a pull request to SOF repository.

Reproduction Rate
100%

Expected behavior
Linux and Windows host builds should create identical binaries.

Impact
Failure seen on every CI request, can mask other errors.

Activity

  1. kv2019i
    added
    bugSomething isn't working as expected
    Known PR FailuresIssues seen in SOF github pull-request checks
    on Jan 27, 2025

  2. kv2019i commented on Jan 27, 2025

    @kv2019i
    kv2019i
    on Jan 27, 2025
    CollaboratorAuthor

    @thesofproject/nxp Rings any bells to you? I checked multiple PRs merged last week, but I couldn't find a PR where this failure would have been seen (already pre merge). I could have missed some.

  3. dbaluta commented on Jan 27, 2025

    @dbaluta
    dbaluta
    on Jan 27, 2025
    Collaborator

    @LaurentiuM1234 @marc-hb didn't we have this issue couple of months ago? Will need to check on that.

  4. dbaluta commented on Jan 27, 2025

    @dbaluta
    dbaluta
    on Jan 27, 2025
    Collaborator

    @kv2019i looks like this is not only for imx builds.

     Files linux-build  tgl tgl-h/build-sof-staging/sof-info/tgl-h/stripped-zephyr.elf and windows-build  tgl tgl-h/build-sof-staging/sof-info/tgl-h/stripped-zephyr.elf differ
Files linux-build  tgl tgl-h/build-sof-staging/sof-info/tgl-h/zephyr.lst and windows-build  tgl tgl-h/build-sof-staging/sof-info/tgl-h/zephyr.lst differ
Files linux-build  tgl tgl-h/build-sof-staging/sof-info/tgl-h/zephyr_version.h and windows-build  tgl tgl-h/build-sof-staging/sof-info/tgl-h/zephyr_version.h differ
Files linux-build -d mtl/build-sof-staging/sof-info/mtl/boot.mod and windows-build -d mtl/build-sof-staging/sof-info/mtl/boot.mod differ
Files linux-build -d mtl/build-sof-staging/sof-info/mtl/stripped-main.elf and windows-build -d mtl/build-sof-staging/sof-info/mtl/stripped-main.elf differ
Files linux-build -d mtl/build-sof-staging/sof-info/mtl/stripped-zephyr.elf and windows-build -d mtl/build-sof-staging/sof-info/mtl/stripped-zephyr.elf differ
Files linux-build -d mtl/build-sof-staging/sof-info/mtl/zephyr.lst and windows-build -d mtl/build-sof-staging/sof-info/mtl/zephyr.lst differ
Files linux-build -d mtl/build-sof-staging/sof-info/mtl/zephyr_version.h and windows-build -d mtl/build-sof-staging/sof-info/mtl/zephyr_version.h differ
  5. kv2019i
    changed the title [-][BUG] compare-linux-win CI build check fails to binary difference in imx8/imx8 zephyr.elf[/-] [+][BUG] compare-linux-win CI build check fails to binary difference in zephyr.elf for multiple platforms[/+] on Jan 27, 2025

  6. dbaluta commented on Jan 27, 2025

    @dbaluta
    dbaluta
    on Jan 27, 2025
    Collaborator

    @kv2019i Relevant issue in the past #9034 was solved by opting out on adding the source code on the assembly listing file.

  7. LaurentiuM1234 commented on Jan 27, 2025

    @LaurentiuM1234
    LaurentiuM1234
    on Jan 27, 2025
    Contributor

    FYI test also failed when trying to add build job for imx95 #9546 a while ago. If I recall correctly, one of the compilers would add some comments on one of the OSes but not the other. Superficially looking at the 2 .lst's from #9794 it would seem like a dif. issue though.

  8. marc-hb commented on Jan 27, 2025

    @marc-hb
    marc-hb
    on Jan 27, 2025
    Collaborator

    Files linux-build -d mtl/build-sof-staging/sof-info/mtl/zephyr_version.h and windows-build -d mtl/build-sof-staging/sof-info/mtl/zephyr_version.h differ

    This is the easiest bit to focus on first.

    You can downloads artefacts from every PR page, please compare these.

    Relevant issue in the past #9034 was solved by opting out on adding the source code on the assembly listing file.

    zephyr_version.h looks like a very different problem this time.

  9. marc-hb commented on Jan 27, 2025

    @marc-hb
    marc-hb
    on Jan 27, 2025
    Collaborator

    You can downloads artefacts from every PR page,

    Actually, daily tests are more convenient for this: because zero PR interference.

    Started failing in daily 795 , 4 days ago: https://github.com/thesofproject/sof/actions/runs/12940934767

    please compare these.

    Don't wait because artefacts expire after some time.

  10. marc-hb commented on Jan 27, 2025

    @marc-hb
    marc-hb
    on Jan 27, 2025
    Collaborator

    There's been a progressive rollout of a new Windows image ( Version: 20250113.1.0 -> 20250120.2.0) around the same time but it does not correlate with the failure. EDIT: it's unrelated. The problem is on the Linux side.

    For instance, the IMX build got the newer Windows image in 794 but 794 was still all green:
    https://github.com/thesofproject/sof/actions/runs/12898787138/job/35966473504

    As you can tell from the version numbers, minor upgrades like these happen all the time.

    The first step is really to download and compare zephyr_version.h.

  11. marc-hb
    changed the title [-][BUG] compare-linux-win CI build check fails to binary difference in zephyr.elf for multiple platforms[/-] [+][BUG] compare-linux-win CI build check fails to binary difference in zephyr_version.h for multiple platforms[/+] on Jan 27, 2025
  12. marc-hb
    changed the title [-][BUG] compare-linux-win CI build check fails to binary difference in zephyr_version.h for multiple platforms[/-] [+][BUG] compare-linux-win CI build check fails due to zephyr_version.h difference[/+] on Jan 27, 2025
  13. abonislawski
    assigned
    kv2019i
    on Jan 28, 2025
  14. abonislawski
    added
    P3Low-impact bugs or features
    on Jan 28, 2025

  15. 7 remaining items

  16. marc-hb
    added 9 commits that reference this issue on Jan 28, 2025

    .github/zephyr: simplify and fix the git describe performance hack

    a8cb0ad

    .github/zephyr: simplify and fix the git describe performance hack

    42403a1

    .github: extract new sof-docs.yml from pull-request.yml

    8a9c895

    .github/zephyr: simplify and fix the git describe performance hack

    6748d1c

    .github: extract new sof-docs.yml from pull-request.yml

    685c087

    .github/zephyr: simplify and fix the git describe performance hack

    4c3d692

    .github: extract new sof-docs.yml from pull-request.yml

    429c30d

    .github: extract new sof-docs.yml from pull-request.yml

    ff3f6d2

    .github: extract new sof-docs.yml from pull-request.yml

    5e51eb3

  17. marc-hb commented on Jan 28, 2025

    @marc-hb
    marc-hb
    on Jan 28, 2025
    Collaborator

    Fix submitted. This should also turn the daily tests back to green.

  18. kv2019i
    added 2 commits that reference this issue on Jan 30, 2025

    .github/zephyr: simplify and fix the git describe performance hack

    92d9cca

    .github: extract new sof-docs.yml from pull-request.yml

    31146f5

  19. kv2019i commented on Jan 30, 2025

    @kv2019i
    kv2019i
    on Jan 30, 2025
    CollaboratorAuthor

    Closed via #9801 . Thanks @marc-hb !

  20. kv2019i
    closed this as completedon Jan 30, 2025

  21. marc-hb commented on Jan 31, 2025

    @marc-hb
    marc-hb
    on Jan 31, 2025
    Collaborator

    After dropping sof-docs from them, daily tests are back to green!

    https://github.com/thesofproject/sof/actions/workflows/daily-tests.yml

    Image
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Known PR FailuresIssues seen in SOF github pull-request checksP3Low-impact bugs or featuresbugSomething isn't working as expected

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions