Repository navigation
SLOW OR FAILED (500 ERROR) NODE.JS DOWNLOADS #4495
Description
Activity
For some reason, iojs.org redirected to nodejs.org, causing errors with
nvm(https://github.com/SheetJS/sheetjs/runs/5564331772 test failed when trying to fetch https://iojs.org/dist/v3.3.1/iojs-v3.3.1-linux-x64.tar.xz)I wonder if the site/Cloudflare are just getting overwhelmed with traffic for people downloading the latest release?
Node v16.14.1 was released about an hour ago... #4494 / https://github.com/nodejs/node/releases/tag/v16.14.1
It's the latest LTS release, so everybody in the world looking for the default version of Node is downloading uncached right now, basically. (Other than as apparently cached by Cloudflare).
Should settle down in a few hours, or by tomorrow??
Note: I have nothing to do with Node's web servers or site, I'm just a regular person speculating at this point.
Closing as the issue isn't happening right now, but also pinging @nodejs/build in case there's anything to add here or something to be aware of.
People are still reporting this over in the OpenJS Slack, so I'm going to re-open this until that gets sorted.
I have CloudFlare LB error notices spanning from ~2:40 am UTC to ~3:40 am UTC, which I suppose correlates with the latest 16.x release. Overloaded primary server, switching to the backup. Users may have got the 500 while hitting the primary before it switched.
I don't have a good explanation beyond that, I'm not sure what's caused the overload, I don't believe that server actually has trouble serving, but we have witnessed some weird I/O issues connected with rsync that maybe all happened at the same time. Perhaps during the next release someone should be on the server watching for weirdness.
Reacted by Rich Trott, Jordan Harband and jwigleyThis seems to be happening again; https://iojs.org/dist/index.json is 500ing, and v12.22.11 just went out.
(after about 5-10 minutes, the 500s seemed to stop, but it's still worth looking into)
Reacted by DeeDeeGFWIW I am still seeing either "500 internal server error" or very slow file downloads (less than 50 kilobytes a second. Often less than 10 kilobytes a second, especially at the beginning of a download.)
More details (click to expand):
The "slow download" symptom is less obvious for small files, because they complete quickly anyway.
I have seen a tarball download start, take a long time (over a minute), and ultimately fail to download mid-way through.
I hope that's useful to diagnose the problem. (Results may vary across the globe, since the path through the CDN is probably not identical everywhere?)
Edit to add: My experience is basically identical (as described in this comment) for nodejs.org/dist/ and iojs.org/dist. And identical experience today compared to 2 days ago.
After promoting 12.22.11 I went to rerun the same release script to promote 14.19.1 and it just hung during the bit when it fetches the promotable builds. Same behaviour from another machine on a different network. Weirdly I was able to ssh into the machine in an interactive session and manually run the command the promotion script was trying to run 😕.
I don't have a good explanation beyond that, I'm not sure what's caused the overload, I don't believe that server actually has trouble serving, but we have witnessed some weird I/O issues connected with rsync that maybe all happened at the same time. Perhaps during the next release someone should be on the server watching for weirdness.
I logged into the machine, ran
ps -efand saw we had two rsyncs in progressroot 12174 12137 0 00:08 ? 00:00:00 rsync --server --sender -logDtprze.iLsfxC . /home/dist/iojs/ nodejs 27646 27638 0 00:00 ? 00:00:00 ssh benchmark rsync --server --sender -logDtprze.iLsfx . coverage-out/out/The first of those, running as root, is from the backup server -- I've left that alone. I killed the second one, which is the coverage data, and suddenly my other terminal running the promotion script got unstuck. This would kind of suggest the coverage data syncing is partly, or wholly, responsible. In the past either @mhdawson or I would do an annoying manual clear up of old coverage data to reduce the volume of stuff being synced but I'm going to recommend now we turn off running coverage on Jenkins and the associated data shuffling to populate coverage.nodejs.org and switch exclusively to codecov.io.
Actually as I typed the above paragraph my promotion execution has broken 😢 so there's still something up with the machine:
Are you sure you want to promote the v14.19.1 assets? [y/n] y client_loop: send disconnect: Broken pipeCurrently there are no rsync processes -- but running the promotion script is "hung" again for me 😞.
These are the top things viatop:top - 00:33:22 up 195 days, 16:59, 1 user, load average: 0.72, 0.80, 0.84 Tasks: 165 total, 2 running, 163 sleeping, 0 stopped, 0 zombie %Cpu(s): 4.7 us, 3.3 sy, 0.0 ni, 83.6 id, 0.1 wa, 0.0 hi, 8.2 si, 0.1 st KiB Mem : 16432136 total, 1251072 free, 1222728 used, 13958336 buff/cache KiB Swap: 0 total, 0 free, 0 used. 14391196 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 25151 www-data 20 0 595256 456632 375512 S 9.0 2.8 510:58.65 nginx 25149 www-data 20 0 595796 456892 375408 S 8.3 2.8 529:56.28 nginx 25152 www-data 20 0 597244 458740 375596 S 8.0 2.8 524:21.40 nginx 25150 www-data 20 0 594556 456176 375488 S 7.3 2.8 505:23.46 nginx 25153 www-data 20 0 596728 458372 375512 S 7.0 2.8 517:06.30 nginx 25154 www-data 20 0 595496 456944 375552 S 6.3 2.8 515:04.23 nginx 23 root 20 0 0 0 0 R 5.0 0.0 238:39.12 ksoftirqd/3 12273 telegraf 20 0 6316820 38820 7492 S 3.0 0.2 17:58.94 telegraf 58 root 20 0 0 0 0 S 2.0 0.0 2773:53 kswapd0 7 root 20 0 0 0 0 S 0.7 0.0 595:13.23 rcu_sched 18 root 20 0 0 0 0 S 0.3 0.0 12:58.42 ksoftirqd/2 16257 root 20 0 42092 3784 3128 R 0.3 0.0 0:00.13 topI think I'm going to bounce nginx.
Reacted by Aron Hegyi and Jordan HarbandHappening same here. Know this type of comments are not useful in most cases, but in this case at least serves to know that there are users still being affected 😬
Edit: worked now for me
I did
systemctl restart nginx. I'm still seeing rsync processes from the backup server still running.44 remaining items
Seen on an Azure DevOps agent:
Downloading: https://nodejs.org/dist/v18.17.1/node-v18.17.1-linux-x64.tar.gz ##[error]Aborted ##[warning]Content-Length (44417128 bytes) did not match downloaded file size (8204861 bytes).Reacted by Martin Hochel and jfuinsuregyp ERR! stack Error: 500 status code downloading checksum
Reacted by Gary WrightPlease reopen as it still happens multiple times a day!
Same, still happening
we were facing this as well for quite some time. only solution to mitigate this is to use cached nodejs that ships with container :-/ not the best thing in the world, but it is what it is microsoft/fluentui#29552
Reacted by Gary WrightHave been facing this intermittently too:
Downloading: https://nodejs.org/dist/v16.16.0/node-v16.16.0-win-x64.7z
##[error]Aborted
##[warning]Content-Length (17121358 bytes) did not match downloaded file size (5534315 bytes).and happening again :(
Downloading: https://nodejs.org/dist/v16.16.0/node-v16.16.0-darwin-x64.tar.gz
##[error]Aborted
##[warning]Content-Length (30597385 bytes) did not match downloaded file size (5075560 bytes).Reacted by Gary WrightThis was happening continually today at https://nodejs.org/dist/v20.8.1/node-v20.8.1-darwin-arm64.tar.gz
Any update on why this is closed?
Reacted by Petrisor LacatusThis is closed because it's a known issue, and we're working on it.
I've been getting this on GitHub Actions (
ubuntu-latest) for the last couple days:> Could not get resource 'https://nodejs.org/dist/v18.14.0/node-v18.14.0-linux-x64.tar.gz'. > Read timed outThis is closed because it's a known issue, and we're working on it.
I think it would better communicate your intent if you closed this issue after you have solved it.
Reacted by Ian Currie, jballschneider, Thibaud Guillaume-Gentil, Jems, Hugh Gallagher, kyled526, Eduardo Pinho and Markus Szumovski- unpinned this issue
on Mar 5, 2024 - added a commit that references this issue
on Sep 8, 2025 - added a commit that references this issue
on Sep 8, 2025
Edited by the Node.js Website Team
Learn more about this incident at https://nodejs.org/en/blog/announcements/node-js-march-17-incident
tl;dr: The Node.js website team is aware of ongoing issues with intermittent download instability.More Details: nodejs/build#1993 (comment)
Original Issue Below
When trying to get files off of
nodejs.org/dist/...ornodejs.org/download/..., I get a server error.(error page served by nginx)
Full error message page (HTML snippet, click to expand)
Browsing around the dirs, like https://nodejs.org/dist/latest-v16.x/, seems to work. Also, Downloading really small files such as https://nodejs.org/dist/latest-v16.x/SHASUMS256.txt seems to work sporadically, whereas downloading tarballs doesn't seem to work.
Given that the outage seems sporadic: Maybe it's a resource exhaustion issue over at the server? Running out of RAM or something?? I don't know.
Edit to add: The error message page seems to be served by Cloudflare. (According to theActually that's probably not what that means.server: cloudflareresponse header, when looking in browser dev tools). So I guess this is a Cloudflare issue?