Skip to content

[FEATURE] Make github actions build docker images and push to ghcr.io #5073

Open
Open
[FEATURE] Make github actions build docker images and push to ghcr.io#5073
Assignees
fredoh9
Labels
enhancementNew feature or request

Description

@afq984
afq984
opened on Dec 11, 2021

Is your feature request related to a problem? Please describe.
docker.io/thesofproject/sof AFAIK is built manually and can become stale from time to time

Describe the solution you'd like
make GitHub Actions build docker images on: schedule and push GitHub Action built docker images to ghcr.io (GitHub container registry)

Describe alternatives you've considered

  • Do nothing.
  • Build on: push: branches: [main]. sof_builder takes too long to build and most build dependencies are actually not pinned by thesofproject/sof so I think it doesn't make too much sense to tie the action run to thesofproject/sof commits.

Additional context
I tried the setup on my account, highlights of steps below:

Allow the repository actions to push to ghcr.io containers

Using GITHUB_TOKEN is the preferred method of accessing ghcr.io in actions (instead of using a personal access token). However the docs are a bit stale. What I did was:

  1. Get github personal access token here. This is used just to create the container package.

  2. Create ghcr.io package

    docker login ghcr.io -u afq984  # enter token when prompted with password
docker tag hello-world ghcr.io/afq984/sof
docker push ghcr.io/afq984/sof

  3. Go to the package settings page of the created container package. Add the sof repository to the package to allow it to push to the registry

    image

  4. Repeat the above for sofqemu

  5. Maybe need to tweak https://github.com/thesofproject/sof/settings/actions. The defaults worked for me but YMMV.

Setup github actions

Demos:

Activity

  1. afq984
    added
    enhancementNew feature or request
    on Dec 11, 2021

  2. afq984 commented on Dec 11, 2021

    @afq984
    afq984
    on Dec 11, 2021
    ContributorAuthor

    cc @cujomalainey @marc-hb

  3. marc-hb commented on Dec 11, 2021

    @marc-hb
    marc-hb
    on Dec 11, 2021
    Collaborator

    Build on: push: branches: [main]. sof_builder takes too long to build and most build dependencies are actually not pinned > by thesofproject/sof so I think it doesn't make too much sense to tie the action run to thesofproject/sof commits.

    It is possible to trigger some actions only when certain files/directories are modified, the Zephyr project uses this:

    https://github.com/trilom/file-changes-action

    https://github.com/zephyrproject-rtos/test_results/blob/ef9cc5a9387649/.github/workflows/check_results.yml#L33
    uses: trilom/file-changes-action@v1.2.3

    EDIT: this project looks orphaned. Also, does it have 1000+ dependencies!?
    https://github.com/trilom/file-changes-action/graphs/contributors
    https://github.com/trilom/file-changes-action/network/dependencies

  4. marc-hb commented on Dec 11, 2021

    @marc-hb
    marc-hb
    on Dec 11, 2021
    Collaborator

    Automated builds are nice, but without testing they're not worth much. That's why Docker images are built manually right now: because images have to be manually tested anyway. So for now I don't see the value of storing in some cloud all these docker images (one for every PR?) if they're not tested at all.

    By the way how does ghcr.io compare with Docker Hub that we use currently, why do you seem to recommend a switch to it? Is its pricing there: https://github.com/features/packages#pricing ?

  5. afq984 commented on Dec 11, 2021

    @afq984
    afq984
    on Dec 11, 2021
    ContributorAuthor

    It is possible to trigger some actions only when certain files/directories are modified

    yes that's possible, but I think we should also build periodically since I don't think we can detect xtensa-overlay, crosstool-ng & friends' changes easily now.

    Also nit about uses: trilom/file-changes-action@v1.2.3: we should use trilom/file-changes-action@96385226e7e8530961a9bfebcd319aa054e8012c instead to avoid supply chain attacks as tags are mutable, unless we treat github.com/trilom as a trusted source.

    So for now I don't see the value of storing in some cloud all these docker images (one for every PR?) if they're not tested at all.

    I once wanted to use new toolchains added in the overlays but the it was not in the docker.io image. Would be nice if there's a grab & go container to use, as building the toolchain takes a long time.

    For the testing concern, I think we can instead only tag verified images as latest

    Or alternatively still use docker as source of truth, and let github just host unstable images as a first step.

    Is its pricing there: https://github.com/features/packages#pricing ?

    Yes, the page says GitHub Packages is free for public repositories

    One more thing I'd like to mention is last time (~1 yrs) I tried ghcr.io's network bandwidth is not very good, which needs to be re-evaluated.

  6. afq984 commented on Dec 11, 2021

    @afq984
    afq984
    on Dec 11, 2021
    ContributorAuthor

    One more thing I'd like to mention is last time (~1 yrs) I tried ghcr.io's network bandwidth is not very good, which needs to be re-evaluated.

    A clean pull from

    • docker.io/thesofproject/sof took 60.13 seconds
    • ghcr.io/afq984/sof took 67.60 seconds

    While the docker.io registry download is much faster (I don't have a good way to measure it though), the time is dominated by image extraction so the difference isn't that much.

  7. marc-hb commented on Dec 11, 2021

    @marc-hb
    marc-hb
    on Dec 11, 2021
    Collaborator

    yes that's possible, but I think we should also build periodically since I don't think we can detect xtensa-overlay, crosstool-ng & friends' changes easily now.
    I once wanted to use new toolchains added in the overlays but the it was not in the docker.io image. Would be nice if there's a grab & go container to use, as building the toolchain takes a long time.

    Thanks for asking very good questions and starting this discussion. There is one simple way to solve most of these problems: 1. Extract sof/scripts/docker_build/ into a new git repo / github project. There is no reason to have a very tight coupling of the versions of our Docker image (or the toolchains) with the sof git repo as we have now. The former are external and optional dependencies. I don't think this tight coupling was by design. 2. In this new docker-image repo, add a new manifest / simple config file with the git versions of the overlays cloned by default by CI. Right now the build scripts just fetch the latest versions which is not reproducible and not good. They should still be able to fetch the latest versions but not in CI by default.

    For the testing concern, I think we can instead only tag verified images as latest

    I'm not familiar with docker image versioning but I'm afraid that's not enough separation. I mean when I type docker image history I wouldn't want to see a mix of officially tested and releases images mixed up with images built for random PRs from random users and it would be very inconvenient when trying to "bisect" a Docker image regression (and there are security concerns here too). How about different image names? How do other projects solve this problem, what are the best practices here? Maybe https://github.com/zephyrproject-rtos/docker-image have solved this already (I haven't looked at it yet)

  8. marc-hb commented on Dec 11, 2021

    @marc-hb
    marc-hb
    on Dec 11, 2021
    Collaborator

    While the docker.io registry download is much faster

    There is a good chance the Github registery is very fast when used from Github-hosted Github Actions.

    (I don't have a good way to measure it though),

    Maybe this if you really really want to test
    https://devops.stackexchange.com/questions/2731/downloading-docker-images-from-docker-hub-without-using-docker

    the time is dominated by image extraction so the difference isn't that much.

    Right, a large optimization of a small part is actually a small optimization. As long as the performance is comparable I would focus on all the other aspects instead: features and integration, reliability, ease of use, rate limits, price...

  9. afq984 commented on Dec 12, 2021

    @afq984
    afq984
    on Dec 12, 2021
    ContributorAuthor

    Extract sof/scripts/docker_build/ into a new git repo / github project

    Yeah I imagine that'd make a lot of things easier

    add a new manifest / simple config file with the git versions of the overlays cloned by default by CI

    maybe git submodule is a good candidate?

    docker image history

    The docs isn't very clear about what this command does, but it seems to list what each image layer adds on top of the previous layer, rather than the history of different builds:

    % docker image history docker.io/thesofproject/sof
IMAGE          CREATED        CREATED BY                                      SIZE      COMMENT
f16acc0f8076   3 months ago   /bin/sh -c #(nop) WORKDIR /home/sof/work/sof…   0B        
<missing>      3 months ago   /bin/sh -c #(nop)  USER sof                     0B        
<missing>      3 months ago   |9 CLONE_DEFAULTS=--depth 5 CT_NG_REPO=https…   0B        
<missing>      3 months ago   |9 CLONE_DEFAULTS=--depth 5 CT_NG_REPO=https…   410MB     
<missing>      3 months ago   /bin/sh -c #(nop)  ARG NEWLIB_REPO=https://g…   0B        
<missing>      3 months ago   /bin/sh -c #(nop)  ENV PATH=/home/sof/work/x…   0B        
<missing>      3 months ago   /bin/sh -c #(nop)  ENV PATH=/home/sof/work/x…   0B        
...

    Given that I think having a lot of builds won't actually pollute its output.

    "bisect" a Docker image regression

    docker images similarly to git can be tagged. It looks like this in docker hub's UI and this in ghcr's UI.

    Assuming we have a dedicated sof-docker repository and use submodules to track dependencies,
    We can map git branches/tags to docker tags:

    • Each git tag v0.1, v0.2 etc is mapped to a docker image with the same tag.
    • Docker tag latest is the default tag. We can tag a commit as latest to produce the corresponding image or just use the main/master branch for that.
    • Build nightly tagged images from main. The difference is dependencies are sync'd to HEAD before building.

    This way, we'll have view of latest, nightly, v0.3, v0.2, v0.1 on dockerhub/ghcr's web UI.
    zephyrproject-rtos/docker-image's publish workflow is very similar. I imagine other CI jobs will only ever want to use v-* tagged images or latest. When a regression happens, we just need to update the latest tag to point to a good version.

  10. marc-hb commented on Dec 12, 2021

    @marc-hb
    marc-hb
    on Dec 12, 2021
    Collaborator

    maybe git submodule is a good candidate?

    Submodules have very few fans, I think we should move to west instead which is what Zephyr uses. Long story in #3517.

    In this particular case though I think a "stable" config file with few revisions would be much simpler and do the job. Either submodules or west would be overkill and unnecessarily require people to learn a new tool (git submodules have been "new" to me for 15 years, I have to refresh my memories every single time)

    The docs isn't very clear about what this command [docker image history] does, but it seems to list what each image layer adds on top of the previous layer, rather than the history of different builds:

    My bad. I think I finally found why I was ignorant about versioning Docker images: because Docker provides no such feature at all, everyone must roll their own versioning. It's very time consuming to search documentation of a feature that does not exist; fortunately there is stackoverflow :-)

    Considering Docker's "roll your own" versioning approach, using different image names for official versus PR images seems like a bare minimum to me.

    Each git tag v0.1, v0.2 etc is mapped to a docker image with the same tag.

    Sounds good. For PR images we could use the PR number + the git sha.

    Build nightly tagged images from main.

    Considering how infrequently our docker image has been moving so far, I don't see the point of rebuilding the same image every night over and over again, sounds like noise. Nightly builds are useful there are a significant number of relatively fast moving parts. In such fast-paced environments you cannot build every combination of every git push or merge in every repo, so nightly builds provide useful checkpoints. However in the very different and slow moving case of our docker image and toolchains we can build every incoming code change. Building for every docker image change will result in much fewer images in total and more images different from each other.

    Unless there are plans for some new, fast-paced toolchain developments that I'm not aware of? Scary :-)

    When a regression happens, we just need to update [move back] the latest tag to point to a good version

    Right, and not call a tag that moves back latest but something else like stable for instance.

  11. afq984
    self-assigned this
    on Dec 16, 2021

  12. greg-intel commented on Jan 5, 2022

    @greg-intel
    greg-intel
    on Jan 5, 2022
    Contributor

    Thanks for the information and verification @afq984. After working with some docker images today, I can see how there'd be a benefit to potentially leveraging some of this.

  13. afq984
    removed their assignment
    on Apr 18, 2022

  14. afq984 commented on Apr 18, 2022

    @afq984
    afq984
    on Apr 18, 2022
    ContributorAuthor

    un-assigning myself as not having cycles to work on this

  15. lgirdwood commented on Apr 20, 2022

    @lgirdwood
    lgirdwood
    on Apr 20, 2022
    Member

    @greg-intel is this something you can own ?

  16. marc-hb commented on Apr 21, 2022

    @marc-hb
    marc-hb
    on Apr 21, 2022
    Collaborator

    After using the same Docker image for a long period we had a recent burst of upgrades because of topologies v2 and there will be a couple more coming (see #5323 + links, maybe #5386 some day) but I think this was unusual, I expect the tempo to slow down again. This means the return on investment of automating some testing will be low, more effort than testing manually.

    tl;dr: I think this is a P3 right now.

  17. greg-intel commented on Apr 21, 2022

    @greg-intel
    greg-intel
    on Apr 21, 2022
    Contributor

    Yeah, I agree with Marc. Also, automated means someone's not actively vetting the image, which means we'd maybe want more framework than running the docker build and upload commands.

    TBH, it's entirely possible, but I haven't given it too much thought.

    But if someone finds themselves with the bandwidth to work on this, I'd support their effort.

  18. marc-hb commented on Apr 26, 2022

    @marc-hb
    marc-hb
    on Apr 26, 2022
    Collaborator

    The Docker image with the Zephyr SDK is available on both docker.io and ghcr.io. So I submitted a switch of our Zephyr build to ghcr:

    It's not faster. I still think it's valuable, see why in #5738

  19. afq984 commented on Apr 27, 2022

    @afq984
    afq984
    on Apr 27, 2022
    ContributorAuthor

    Reading #5738, I do feel this FR is essentially two independent things combined together:

    1. Use CI to build docker images
    2. Host the docker image also on ghcr.io

    While there's no great demand on (1), how about hosting the docker image on both docker.io & ghcr.io? The benefits would be the same as #5738, not sure if it's worth the maintenance burden though.

  20. marc-hb
    assigned
    fredoh9
    on Nov 10, 2022

  21. marc-hb commented on Dec 21, 2022

    @marc-hb
    marc-hb
    on Dec 21, 2022
    Collaborator

    This could be useful https://github.com/marketplace/actions/docker-setup-buildx

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions