Windows builds broken #3929
Description
Activity
Happy to help if I can, this is blocking the release.
I did an investigation. Everything was broken between
Oct 9, 2024, 3:23 AMandOct 9, 2024, 5:20 AMJenkins local time. One noticeable thing that happened between those 2 moments is windows-update-reboot trigger, which updated git from v2.46.2 to 2.47.0 (from logYou have git.install v2.46.2 installed. Version 2.47.0 is available based on your source(s).).I'll try taking one machine offline downgrading git and seeing if it fixes the issue. If it does I'll go ahead and downgrade all of the machines and pin Git to 2.46.2 until we find a way for it to work with v2.47.0.
Reacted by Antoine du Hamel, Matteo Collina and Richard Lau@StefanStojanovic did you get some results from the investigation?
I made changes in Windows 10 machines (downgrading git to 2.46.2) since there are only 4 of them, so was the least work to get them all done. However now I get different error log with
Permission denied, please try again.when trying to dogit fetch .... Currently, I'm not sure what to make of it.The machines doing the compilation are on the latest git and they seem not to have issues pushing code.
An update - after reinstalling Git,
cygpath, which is required, was missing fromPATHfor some reason (C:\Program Files\Git\usr\binwasn't added). After adding it manually, canceling all test jobs (as they are waiting a lot because of the timeouts) and retrying a random one, I have this run and I can see that Win10 tests are running (the others will timeout again).So until now, based on my investigation, I think this needs to be done on all Windows CI machines:
choco uninstall gitchoco install git --version=2.46.2- Check if reinstall was successful as 1/4 machines still kept v2.47.1,
choco pin add -n gitchoco pin add -n git.install- Add
C:\Program Files\Git\usr\binto PATH - Reboot machines
After that is done I think the test jobs will be back to normal.
P.S. While I was writing an update Win10 test jobs finished, some of them failed, but because of test failures.
Looking at https://ci.nodejs.org/job/node-test-binary-windows-js-suites/30776/, the problem seems resolved for
win10-COMPILED_BY-vs2022machines, but still ongoing for all the other onesLooking at https://ci.nodejs.org/job/node-test-binary-windows-js-suites/30776/, the problem seems resolved for
win10-COMPILED_BY-vs2022machines, but still ongoing for all the other onesExactly, I'll apply same fix to the other machines too
Any chance you'd be able to share a timeline for when the Windows CI would be ready?
In an hour or two max, I'm already working on it (halfway there I'd say). Will let you know once it's ready.
Reacted by Antoine du HamelReacted by Antoine du Hamel and Richard LauFixed all of the machines except 2 Windows 2022 in Rackspace with VS 2022 (the ones we use for both compilation and tests) which I marked as temporary offline (the other 4 are fixed). I'll deal with them first thing in the morning tomorrow CET.
Anyway, here are the daily master test jobs I retried (https://ci.nodejs.org/job/node-test-binary-windows-js-suites/30782/ and https://ci.nodejs.org/job/node-test-binary-windows-native-suites/25168/), as you can see, all passed, so should be all fine except for the 2 missing machines. That is just temporary, I'll reenable them tomorrow. Regards.
Reacted by Richard Lau and Antoine du HamelThe last 2 machines were fixed and reenabled a few hours ago. From what I see, everything is back to normal. I'll keep this issue open until Monday just in case.
Reacted by Richard LauReacted by Richard LauClosing this issue now since it didn't repeat over the weekend.
Reacted by Richard Lau
It looks like since this morning the Windows test jobs started failing
https://ci.nodejs.org/job/node-test-binary-windows-js-suites/
https://ci.nodejs.org/job/node-test-binary-windows-native-suites/
It looks like they're timing out trying to connect to the binary_tmp git repo:
e.g. https://ci.nodejs.org/job/node-test-binary-windows-js-suites/30748/RUN_SUBSET=0,nodes=win10-COMPILED_BY-vs2022/console
cc @StefanStojanovic