Repository navigation
build: provide pre-built nodejs/node images or cache #39672
Description
Activity
Also, FWIW, I am happy to help make this happen however I can. I assume if we go down the route of images, the auto-generation will be the... harder part for me to meaningfully contribute to purely because I don't have access/am not familiar with the infrastructure we'd likely use to do that compute.
The challenge I see with a Docker container is how to "connect" it to the user's GitHub credentials (so they can easily push to their fork after working on a change).
I also think that's a much better problem to have than having to compile Node.js on your own computer 😅
I suppose I should ask, assuming we do images:
- are there any strong opinions on what needs to be included excluded outside of the most basic requirements?
- how would we build and publish nightly? do we want to Do It Ourselves or do we want to see if Actions will be "good enough"?
I think we should really explore two options and see what is feasible:
-
have one docker image generated every night compiled with everything we need.
-
have a platform-specific (Linux/Intel, Mac/Intel, Mac/Apple, Win/Intel) archive created from the
build/folder that is donwloaded
A few stats:
tar czftime: 30star xzftime: 5s.tar.gzsize: 284MB
If we consider a 100MBit line, this bundle will download in around 20-30s.
-
have one docker image generated every night compiled with everything we need.
I'm experimenting with this in https://github.com/targos/node-dev-docker
how would we build and publish nightly? do we want to Do It Ourselves or do we want to see if Actions will be "good enough"?
GitHub actions might be enough, but I don't know where we would push the image. I would try with the GitHub container registry, but I don't know if it's designed to hold short-lived images (we don't want to keep all nightly builds in storage forever).
Reacted by Matteo Collinahave one docker image generated every night compiled with everything we need.
I'm experimenting with this in https://github.com/targos/node-dev-docker
After 41 minutes on my mac, the image is built. Its size is 2.86 GB
After 41 minutes on my mac, the image is built. Its size is 2.86 GB
Compressed or uncompressed? Usually those are transferred in a compressed fashion.
I guess uncompressed. It's the size reported in the Docker dashboard:
You could always remove
out. Although that would remove the compiled output, the ccache cache should still exist to speed up build times.I going to try again with
--ninjabecause I don't like the output with the normal build and subsequent calls tomakeare not no-ops.You could always remove
out. Although that would remove the compiled output, the ccache cache should still exist to speed up build times.This might be really interesting.
Still building, but FWIW the image is already 1.86 GB before compilation, so there's probably a lot of room for improvement (like removing the apt cache)
22 remaining items
oooh I did miss it, my apologies. I'll comb through it and see what's similar and what's different - I'm sure there's some things you thought of that I didn't 👍🏻
Reacted by Michaël Zasso and Matteo CollinaI've been working on this over the past few weeks. Today I made the final bit of progress needed (thanks @mhdawson and @bmeck for the advice) to get it in a workable state.
Progress
So far, I've got a Docker image that:
- sets up a user in
ubuntu:latest - installs all of the necessary dependencies to build Node.js
- creates the needed directories for building and cachingt
- clones Node.js (shallowly - this can be tweaked to be more expansive later if we'd like)
- builds Node.js with Ninja instructions
- installs the built Node.js into the system, making it usable along with all of the globals we'd provide (
node,npm, etc.) - installs node-core-utils globally
This is all done in
bnb/devenvand is presently published to bitandbang/devenv:latest on Docker Hub. Taking this approach (publishing an image) lends a few benefits, with one of the most impactful (imo) being that it can be set up to run as a VM completely agnostically with whatever specs you can afford to throw at it.Additionally, I've got an example devcontainer config (available at
bnb/node-devcontainerthat includes a minimal-ish.devcontainer.jsonconfiguration and Dockerfile that allow GitHub Codespaces to build out a nice developer environment that can be used from GitHub.dev, from VS Code, or from the command line (thanks to thegh codespaces sshcommand released yesterday!) directly within GitHub or from the Docker Desktop app. There are a few extensions I've set up that I think greatly enhance the experience when using Codespaces and VS Code. YMMV, we can always change this.My proposal with this would be to include the configuration files (those in
./.devcontainer/) innodejs/node. I'd of course expect some iteration/additions as we get a bit further along, but this works as-is now.Further Work
I've set this up with building and publishing nightly in mind. Theoretically this should be doable (and ~relatively trivial) with GitHub Actions, but I've been focusing on the Docker side of things until the breakthrough progress I had today.
If we do decide to proceed with this, I'd presumably move the
devenvrepo tonodejs/(the name was absolutely temp, I don't have a strong opinion on what it should be named... yet) and have it run/be maintained in the org. I'd also PR the.devcontainerdirectory tonodejs/node, which would allow people to launch the developer environment directly fromnodejs/nodeon GitHub.com.Ideally, to proceed, we'd need a few things:
- GitHub Actions building the Docker image nightly
- GitHub Actions publishing the Docker image nightly
And to land:
- move
bnb/devenvto the Node.js org. - PR
./.devcontainerfrombnb/node-devcontainerintonodejs/node
If this is something you'd like to see, I'd appreciate hearing that.
Reacted by Colin Ihrig, Matteo Collina and Michaël Zasso- sets up a user in
I'm +1 on trying this out and make it part of the Node.js infrastructure. I have a few questions.
Could the docker image be part of our docker team help maintaining?
Could the image be used to develop Node.js outside of CodeSpaces? In that case, how would the flow work?
Would it be possible we get some level of sponsorship from GitHub to let developers use CodeSpaces for free in that environment/as part of contributing to Node.js?
I'm +1 on trying this out and make it part of the Node.js infrastructure.
+1 from me as well.
Could the docker image be part of our docker team help maintaining?
It could be maintained by anyone, so yes if they want to. I'd definitely love to continue contributing, of course :)
Could the image be used to develop Node.js outside of CodeSpaces? In that case, how would the flow work?
Yep! The image is your typical Docker image, just perhaps a bit more chonky than you'd normally want a production service Docker image to be. Spin it up on your local machine or in a cloud, SSH in and you're good to go. We should definitely have instructions for this!
Would it be possible we get some level of sponsorship from GitHub to let developers use CodeSpaces for free in that environment/as part of contributing to Node.js?
I'm not sure on this, since I don't think this is how Codespaces works in terms of the infrastructure/billing. Specifically, it's presently tied to org membership with the org being responsible for the bill. What I'd assume is more likely than public entitlements, given the billing structure, is getting an entitlement for the org and its members. I'll be happy to ask on both fronts, though.
Is this relevant?
This issue has been marked as stale due to 210 days of inactivity.
It will be automatically closed in 30 days if no further activity occurs. If this is still relevant, please leave a comment or update it to keep it open.This issue has been automatically closed after 30 days of inactivity following its stale status (no activity for a total of 120 days).
If this is still relevant, feel free to reopen it or leave a comment with additional details so we can continue the discussion.
Is your feature request related to a problem? Please describe.
Presently, making a small contribution to node.js core is particularly challenging. There are a number of contributing factors, but a primary one is simply having to build Node.js from scratch. For someone unfamiliar/new to Node.js, this can be particularly challenging compared to basically anything else in the JavaScript ecosystem.
Describe the solution you'd like
It would be a nice contributor experience enhancement to provide a pre-built image (containerd or Docker?) or some kind of cache that would help alleviate the pain of needing to build completely from scratch every single time.
One theoretical manifestation of how this could work:
Containers provide some nice benefits:
Or, we could do something in the vein of a cloud cache (in the vein of goma). While these require immense work, they are extremely useful and don't require you to run Docker. If there's more interest in this, I'm happy to see what resources I can pull from Microsoft (including potentially open sourcing some things) to enable it.
Either way, this is a build feature that seems to go in a positive direction that helps both new and existing contributors. Would love thoughts, if there are any.
Describe alternatives you've considered
Doing nothing: I mean this works but also retains an unnecessary bad experience.