Repository navigation
Including docker image in standard release #14190
Description
@nodejs/release @nodejs/docker ... Given the number of users who depend on the Node.js docker images, I think it's time that we strongly consider integrating generation of the docker images as part of our regular release CI process -- that way, the updated images would be available immediately when a new release is cut -- which is particularly helpful when cutting security releases.
Activity
evanlucas commentedon Jul 12, 2017
+1 from me. This would be quite useful I would think
cjihrig commentedon Jul 12, 2017
+1
SimenB commentedon Jul 12, 2017
+1
I don't think the images necessarily can be available immediately, as it has to go through a PR to docker hub (see docker-library/official-images#3181). But at least skipping waiting for a build and merge on Node's side would be really nice!
LaurentGoderre commentedon Jul 12, 2017
@jasnell can we get a musl pre-compiled version of node as well for alpine Linux?
chorrell commentedon Jul 12, 2017
Just to set some expectations:
Updating the the relevant Docker files in https://github.com/nodejs/docker-node as part of the CI release process is certainly possible. It's just a matter of updating some text files. Even generating the PR for the Docker Hub could be part of an automated release process. However, there will always be some lag between when that PR is accepted and when the actual images are available on the Docker Hub. See the comment here for what that is: docker-library/official-images#3181 (comment)
The PR to the Docker Hub also involves a review process on the Docker Hub side. They are very responsive, but it does involve human intervention so things can get complicated depending on the day/time of a given Node.js release. A while ago we established a kind of fast-track process for security updates where the Hub PR would have [security] in the title. The idea being that would give it priority for review/merging but the actual build times in the Hub will still vary. I'm not sure everyone on @nodejs/docker is aware of this though...
The whole process can be automated, sure, but "available immediately" is not something that's currently possible. It might be worth having a separate conversation with the Docker Hub team about what's possible there.
A that being said, and after working on the docker-node for a while, I think automating as much of the update process is worthwhile and long overdue.
LaurentGoderre commentedon Jul 12, 2017
It would be nice to have auromation points at the edge of each systems. for example, if the NodeJs group has a bot, then the release build can create a PR for the Docker node repo. This would prevent having to do these manually every time. I am not familiar on how the docker hub wortks yet but there may be some ways to do something similar there too.
pesho commentedon Jul 12, 2017
Definitely. There is already nodejs/docker-node#431 which attempts to automate the second part (submitting the Docker Hub PR).
mhdawson commentedon Jul 12, 2017
This is a good idea from my perspective and the automation would also help as we expand docker support to other platforms.
PeterDaveHello commentedon Jul 12, 2017
Or maybe we could just directly maintain those images using our own organization so the process would definitely be shorter and faster, the only concern may be side effect on the migration from the old docker official ones to our new ones.
chorrell commentedon Jul 13, 2017
Yeah, we could, but it wouldn't be an official Docker image and the migration path would be awkward.
Having our own org and images is an interesting idea though. We could do that anyway as an alternative to the official images, kind of like a clean slate after all we've learned maintaining the current images. That might be worth a separate discussion as a docker-node issue.
yhwang commentedon Jul 14, 2017
@PeterDaveHello Currently, different architecture images are categorized into different organizations in docker hub, i.e. ppc64le arch goes to ppc64le, s390x arch goes to s390x. It's kind of convention for now. If we want to put all images under one organization, I don't know if it's proper or not but definitely is different from others.
SimenB commentedon Oct 24, 2017
Seeing as we'll be getting security patches later today - if one of us in @nodejs/docker could be pinged, or a PR submitted to the repo (even better, of course), when the new binaries become available so that we can start the process of getting a new release on docker hub as soon as possible, that would be great!
The process for a PR (as long as it's just a patch release and not a minor) is just to run
./update.shin the root of the repo.(Unrelated to my post, but related to the posts above mine, I'm -1 on our own org on docker hub)
LaurentGoderre commentedon Oct 24, 2017
Is there any way we can leverage GitHub hook to have a bot create the PR automatically?
billinghamj commentedon Apr 25, 2018
Thanks @SimenB for directing me here.
What is the path to making this happen? Is it just a question of the relevant PRs to add the support? Or is there still some consideration as to whether or not it is acceptable to the relevant Node leadership people?
LaurentGoderre commentedon Apr 26, 2018
The docker images need the official packages to be released prior to the build. I think the best of our time is to investigate creating a PR with new node versions to the docker PR, just like we do when our build creates a PR to the official images.
https://github.com/nodejs/docker-node/blob/master/generate-stackbrew-pr.sh
rvagg commentedon Apr 26, 2018
This is a bit of a problem because release builds require manual promotion by the releaser. The builds could sit in staging for hours before they are promoted, and may even be overwritten by other builds before they are promoted. So any automation here would need to hook into the release process. I don't think it's something we can trigger from Jenkins during the build.
There's two possibilities for hooking something in releases:
Another one is to simply have a cron job somewhere that watches for releases. We do that for a few things @ NodeSource, like building the
nodesnaps that get published to snapcraft.io. It's not hard to watch index.tab or index.json and trigger something when it changes. In fact, the snap process is kind of similar to what you want (snaps are essentially like containerized software packages):So I have a bunch of crontab entries that have this in them:
cd /path/to/snap-node && ./snapcraft.yaml.sh -r10 -gnode10, one for each version we're releasing.SimenB commentedon Apr 26, 2018
When a new major comes out is sort of a special case, as that's the time when we're able to make breaking changes to the images (such as this time, when we upgraded from alpine 3.6 to 3.7 for node 10).
But all other releases just needs to run the
update.shscript, so that part would be awesome to automate.That script works by
curling https://nodejs.org/dist/ (and the same for chakracore) for the latest of the majors in the repo, so as long as the new release is available there, the script should succeed.I'm not sure which of the paths forward mentioned above would be best, but it sounds like at least the first one (running something as part of the script ran by the releaser) would be too early, right?
LaurentGoderre commentedon Apr 26, 2018
I would prefer the cron route
jasnell commentedon Apr 26, 2018
We (nearForm) have a process similar to what @rvagg describes... Essentially a process that watches the release json file and generates a new set of docker images automatically within 30 minutes of the release being posted. Works rather well.
jasnell commentedon Oct 25, 2018
Closing due to lack of follow up discussion and activity.
SimenB commentedon Oct 25, 2018
There has been setup a process for security releases over the summer: nodejs/security-wg#306
Automating it more would be sweet, but IMo the most important part (getting a security release out) has been addressed