Skip to content

Parallel node-gyp builds failing due to concurrent file access (Windows only) #3095

Closed

Description

@MRayermannMSFT

Hi! I'm looking for help solving an issue we're starting to see more often on Windows, when npm install builds our native node modules.

The issue is that because npm install runs native module builds in parallel, and each needs exclusive access to some files, they end up failing due to not being able to get that exclusive access. We believe we're seeing this issue more often because we recently standardized on using node-addon-api for all five of our native modules. Thus, we have five parallel builds, all trying to access a lot of the same files. I've provided a series of example errors below (if someone wants full logs, I can provide those as well).

To be clear, I don't think this is necessarialy node-gyp's fault, however it may be part of the solution. For now, I am here to see if anyone in the node-gyp community has any ideas/thoughts/suggestions on how to proceed next. So far here's some ideas I've had:

  1. Use --install-strategy=nested to de-deduplicate node-addon-api, thus reducing (maybe even eliminating) the chance of conflicting file access. Our project is ultimately bundled, so this will probably work out ok, but there are likely going to be other complications. Would be nice if we could only de-deduplicate node-addon-api.
  2. Combine some of the five native modules, thus reducing the total number. This won't fix the problem, but it'll reduce the chance of it happening.
  3. Solicit npm to not do parallel native node module installs.
  4. Add a retry mechanism to node-gyp.
  5. Prebuild the binaries for as many of our native node_modules as we can, thus avoiding having to rebuild them for the main project.
  6. Write our own build-time code to de-deduplicate node-addon-api.
  7. Use --ignore-scripts, and then use npm rebuild for any package which has an installScript: true in package-lock. Currently using this.
  8. Use --foreground-scripts. Need to test this.
Details

The specified task executable "CL.exe" could not be run. System.IO.IOException: The process cannot access the file 'PATH_TO_NODE_MODULES\node-addon-api\Release\obj\nothing\nothing.tlog\CL.command.1.tlog' because it is being used by another process. [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]
   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath) [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]
   at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost) [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost) [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]
   at System.IO.StreamReader..ctor(String path, Encoding encoding, Boolean detectEncodingFromByteOrderMarks, Int32 bufferSize, Boolean checkHost) [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]
   at System.IO.StreamReader..ctor(String path) [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]
   at System.IO.File.OpenText(String path) [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]
   at Microsoft.Build.CPPTasks.TrackedVCToolTask.MapSourcesToCommandLines() [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]
   at Microsoft.Build.CPPTasks.TrackedVCToolTask.GenerateSourcesOutOfDateDueToCommandLine() [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]
   at Microsoft.Build.CPPTasks.CL.ComputeOutOfDateSources() [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]
   at Microsoft.Build.CPPTasks.TrackedVCToolTask.SkipTaskExecution() [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]
   at Microsoft.Build.Utilities.ToolTask.Execute() [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]

Details

node-gyp\src\win_delay_load_hook.cc(1,1): error C1083: Cannot open compiler generated file: 'PATH_TO_NODE_MODULES\node-addon-api\Release\obj\nothing\win_delay_load_hook.obj': Permission denied [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]

Details

npm error LINK : fatal error LNK1181: cannot open input file 'Release\obj\nothing\\node-addon-api\nothing.obj' [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]

Details

The specified task executable "CL.exe" could not be run. System.IO.IOException: The process cannot access the file 'PATH_TO_NODE_MODULES\node-addon-api\Release\obj\nothing\nothing.tlog\CL.write.1.tlog' because it is being used by another process. [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]
   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath) [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]
   at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost) [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost) [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]
   at System.IO.StreamWriter.CreateFile(String path, Boolean append, Boolean checkHost) [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]
   at System.IO.StreamWriter..ctor(String path, Boolean append, Encoding encoding, Int32 bufferSize, Boolean checkHost) [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]
   at System.IO.File.InternalWriteAllText(String path, String contents, Encoding encoding, Boolean checkHost) [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]
   at System.IO.File.WriteAllText(String path, String contents, Encoding encoding) [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]
   at Microsoft.Build.Utilities.CanonicalTrackedOutputFiles.SaveTlog(DependencyFilter includeInTLog) [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]
   at Microsoft.Build.CPPTasks.CL.PostExecuteTool(Int32 exitCode) [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]
   at Microsoft.Build.CPPTasks.TrackedVCToolTask.ExecuteTool(String pathToTool, String responseFileCommands, String commandLineCommands) [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]
   at Microsoft.Build.CPPTasks.CL.ExecuteTool(String pathToTool, String responseFileCommands, String commandLineCommands) [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]
   at Microsoft.Build.Utilities.ToolTask.Execute() [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]

Details

npm error LINK : fatal error LNK1181: cannot open input file 'Release\obj\nothing\win_delay_load_hook.obj' [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]

Details

npm error C:\Program Files\Microsoft Visual Studio\2022\Enterprise\MSBuild\Microsoft\VC\v170\Microsoft.CppBuild.targets(386,5): error MSB3374: The last access/last write time on file "Release\obj\nothing\nothing.tlog\unsuccessfulbuild" cannot be set. Could not find file 'PATH_TO_NODE_MODULES\node-addon-api\Release\obj\nothing\nothing.tlog\unsuccessfulbuild'.  [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]

Details

npm error FileTracker : error FTK1011: could not create the new file tracking log file: PATH_TO_NODE_MODULES\node-addon-api\Release\obj\nothing\nothing.tlog\CL.write.1.tlog. The file exists. [PATH_TO_NODE_MODULES\node-addon-api\nothing.vcxproj]

Activity

  1. lukekarrys commented on Dec 2, 2024

    @lukekarrys
    Member

    @MRayermannMSFT I think 3 is likely the best option since npm is in charge of how and when those scripts get run.

    There is also a hack/workaround that uses the npm flag --foreground-scripts which forces npm scripts to run sequentially: npm/cli#3034 (comment). Can you try and see if this flag does the same to native module builds?

  2. MRayermannMSFT commented on Dec 2, 2024

    @MRayermannMSFT
    Author

    There is also a hack/workaround that uses the npm flag --foreground-scripts which forces npm scripts to run sequentially:

    Oh nice! I had actually just implemented a workaround using --ignore-scripts, where we use that flag, and then we use npm rebuild for any package which has an installScript: true in package-lock. I'll give --foreground-scripts a try as well though.

  3. tmm1 commented on Dec 29, 2024

    @tmm1
    Contributor

    Does this manifest as follows?

    node-addon-api\Release\obj\node_addon_api_except\node_add.508A2FFA.tlog\node_addon_api_except.lastbuildstate' because it is being used by another process.

    Does the --foreground-scripts workaround help?

    EDIT: Missed the detailed logs above. I can confirm --foreground-scripts runs in serial and works around the issue.

  4. added a commit that references this issue on Dec 29, 2024
  5. tmm1 commented on Dec 30, 2024

    @tmm1
    Contributor

    An alternative to --foreground-scripts is to use a simple retry loop around npm ci. However, this requires a taskkill /im vctip.exe /f within the loop to work (or the patch from #3112)

  6. lukekarrys commented on Apr 1, 2025

    @lukekarrys
    Member

    I think with #3112 landing and any other fix being outside of what node-gyp is likely responsible for, I'm going to close this issue. Please let me know if this should be reopened if there's anything here I'm missing.

  7. MRayermannMSFT commented on Apr 1, 2025

    @MRayermannMSFT
    Author

    Thanks for making an effort at fixing this! I know it's a bit of a power user scenario. Will let you know if it comes up again.

  8. tmm1 commented on Jun 23, 2025

    @tmm1
    Contributor

    @MRayermannMSFT I'm curious if you noticed any difference after #3112?

    I had a hard time upgrading to the latest node-gyp, but figured it out in #3172

      $nodePath = Split-Path -Parent (Get-Command node).Path
      npm install --global --prefix $nodePath\\node_modules\\npm node-gyp@latest
    

    However I still see some errors like:

    npm error C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\MSBuild\Microsoft\VC\v160\Microsoft.CppBuild.targets(382,5): error MSB3491: Could not write lines to file "Release\obj\node_addon_api_except\node_add.A3690B4C.tlog\node_addon_api_except.lastbuildstate". The process cannot access the file 'D:\a_work\1\s\vscode\node_modules@vscode\node-addon-api\Release\obj\node_addon_api_except\node_add.A3690B4C.tlog\node_addon_api_except.lastbuildstate' because it is being used by another process. [D:\a_work\1\s\vscode\node_modules@vscode\node-addon-api\node_addon_api_except.vcxproj]

    as well as warnings:

    npm warn cleanup 'D:\a\_work\1\s\vscode\node_modules\node-pty',
    npm warn cleanup [Error: EBUSY: resource busy or locked, rmdir 'D:\a_work\1\s\vscode\node_modules\node-pty'] {

    These go away by adding --foreground-scripts, but that makes execution unbearably slow.

  9. tmm1 commented on Jun 23, 2025

    @tmm1
    Contributor

    Yea my logs clearly show /nodeReuse:false added, but the errors continue.

    It seems that flag will stop lingering processes, but currently tlog files are always generated and node-gyp running in parallel will cause failures. See dotnet/msbuild#10814

  10. MRayermannMSFT commented on Jun 23, 2025

    @MRayermannMSFT
    Author

    I'm curious if you noticed any difference after #3112?

    Unfortunately I no longer work on the project that was haivng these issues. However, I do see that the project is still using the strategy of "Use --ignore-scripts, and then use npm rebuild for any package which has an installScript: true in package-lock". Guess they haven't gotten around to trying out the fixes made in April.

  11. tmm1 commented on Jun 27, 2025

    @tmm1
    Contributor

    @lukekarrys @cclauss can you re-open this issue?

    Seems like we would need some type of retry at the msbuild invocation layer to solve this.

  12. cclauss commented on Jun 28, 2025

    @cclauss
    Contributor
  13. tmm1 commented on Jul 18, 2025

    @tmm1
    Contributor

    I'm still working on trying to make a standalone repro here. It has been surprisingly difficult, mostly because the problems are so intermittent. The standard github runners have low cpu counts so its hard to achieve much concurrency.

    Here's another type of recent failure:

    npm error FileExistsError: [WinError 183] Cannot create a file when that file already exists: 'C:\\a\\vscode\\node_modules\\@vscode\\node-addon-api\\node_addon_api.sln.gyp.u3xg9ttn.tmp' -> 'C:\\a\\vscode\\node_modules\\@vscode\\windows-mutex\\build\\..\\..\\node-addon-api\\node_addon_api.sln'
    
    npm error os.rename(self.tmp_path, filename)
    npm error     sln = MSVSNew.MSVSSolution(
    
  14. tmm1 commented on Jul 18, 2025

    @tmm1
    Contributor

    However I still see some errors like:

    npm error C:\Program Files (x86)\Microsoft Visual Studio\2019

    Interesting, here I was still using MSVC 2019, but since then the windows-2019 images have been deprecated. It seems like the race condition specific to msbuild holding files open (or perhaps respecting /nodeReuse:false correctly) is much improved in newer versions of MSVC.

  15. tmm1 commented on Jul 29, 2025

    @tmm1
    Contributor

    Here is a failing run: https://github.com/anysphere/node-gyp-win32-test/actions/runs/16587954733/job/46916898348

    npm error Traceback (most recent call last):
    npm error   File "C:\hostedtoolcache\windows\node\20.18.2\x64\node_modules\npm\node_modules\node-gyp\gyp\gyp_main.py", line 45, in <module>
    npm error     sys.exit(gyp.script_main())
    npm error              ~~~~~~~~~~~~~~~^^
    npm error   File "C:\hostedtoolcache\windows\node\20.18.2\x64\node_modules\npm\node_modules\node-gyp\gyp\pylib\gyp\__init__.py", line 700, in script_main
    npm error     return main(sys.argv[1:])
    npm error   File "C:\hostedtoolcache\windows\node\20.18.2\x64\node_modules\npm\node_modules\node-gyp\gyp\pylib\gyp\__init__.py", line 692, in main
    npm error     return gyp_main(args)
    npm error   File "C:\hostedtoolcache\windows\node\20.18.2\x64\node_modules\npm\node_modules\node-gyp\gyp\pylib\gyp\__init__.py", line 677, in gyp_main
    npm error     generator.GenerateOutput(flat_list, targets, data, params)
    npm error     ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    npm error   File "C:\hostedtoolcache\windows\node\20.18.2\x64\node_modules\npm\node_modules\node-gyp\gyp\pylib\gyp\generator\msvs.py", line 2152, in GenerateOutput
    npm error     sln = MSVSNew.MSVSSolution(
    npm error         sln_path,
    npm error     ...<3 lines>...
    npm error         version=msvs_version,
    npm error     )
    npm error   File "C:\hostedtoolcache\windows\node\20.18.2\x64\node_modules\npm\node_modules\node-gyp\gyp\pylib\gyp\MSVSNew.py", line 228, in __init__
    npm error     self.Write()
    npm error     ~~~~~~~~~~^^
    npm error   File "C:\hostedtoolcache\windows\node\20.18.2\x64\node_modules\npm\node_modules\node-gyp\gyp\pylib\gyp\MSVSNew.py", line 365, in Write
    npm error     f.close()
    npm error     ~~~~~~~^^
    npm error   File "C:\hostedtoolcache\windows\node\20.18.2\x64\node_modules\npm\node_modules\node-gyp\gyp\pylib\gyp\common.py", line 405, in close
    npm error     os.rename(self.tmp_path, filename)
    npm error     ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
    npm error FileExistsError: [WinError 183] Cannot create a file when that file already exists: 'C:\\a\\node-gyp-win32-test\\node-gyp-win32-test\\node_modules\\@vscode\\node-addon-api\\node_addon_api.sln.gyp.2673whas.tmp' -> 'C:\\a\\node-gyp-win32-test\\node-gyp-win32-test\\node_modules\\@vscode\\windows-mutex\\build\\..\\..\\node-addon-api\\node_addon_api.sln'
    
  16. tmm1 commented on Jul 29, 2025

    @tmm1
    Contributor

    Another failure here: https://github.com/anysphere/node-gyp-win32-test/actions/runs/16587962641/job/46916919457

    npm error path C:\a\node-gyp-win32-test\node-gyp-win32-test\node_modules\@vscode\spdlog
    npm error command failed
    npm error command C:\Windows\system32\cmd.exe /d /s /c node-gyp rebuild
    npm error C:\Program Files\Microsoft Visual Studio\2022\Enterprise\MSBuild\Microsoft\VC\v170\Microsoft.CppBuild.targets(385,5): error MSB3491: Could not write lines to file "Release\obj\node_addon_api_except\node_add.F7E42C4A.tlog\node_addon_api_except.lastbuildstate". The process cannot access the file 'C:\a\node-gyp-win32-test\node-gyp-win32-test\node_modules\@vscode\node-addon-api\Release\obj\node_addon_api_except\node_add.F7E42C4A.tlog\node_addon_api_except.lastbuildstate' because it is being used by another process.  [C:\a\node-gyp-win32-test\node-gyp-win32-test\node_modules\@vscode\node-addon-api\node_addon_api_except.vcxproj]
    

    And another: https://github.com/anysphere/node-gyp-win32-test/actions/runs/16587963490/job/46916922118

    npm error C:\Program Files\Microsoft Visual Studio\2022\Enterprise\MSBuild\Microsoft\VC\v170\Microsoft.CppBuild.targets(385,5): error MSB3491: Could not write lines to file "Release\obj\node_addon_api_except\node_add.F7E42C4A.tlog\node_addon_api_except.lastbuildstate". The process cannot access the file 'C:\a\node-gyp-win32-test\node-gyp-win32-test\node_modules\@vscode\node-addon-api\Release\obj\node_addon_api_except\node_add.F7E42C4A.tlog\node_addon_api_except.lastbuildstate' because it is being used by another process.  [C:\a\node-gyp-win32-test\node-gyp-win32-test\node_modules\@vscode\node-addon-api\node_addon_api_except.vcxproj]
    

    In order to reproduce, I had to create a larger github runner with more CPU cores:

    Windows Latest (2022) · 16-cores · 64 GB RAM · 600 GB SSD

  17. tmm1 commented on Jul 29, 2025

    @tmm1
    Contributor

    One thing I noticed is gyp sets TrackerLogDirectory. Its possible using a distinct dir here would help.

    _MSBuildOnly(_compile, "TrackerLogDirectory", _folder_name)

    There is also another option TrackFileAccess that might help.

  18. tmm1 commented on Jul 29, 2025

    @tmm1
    Contributor

    It appears that even after npm ci exits due to error, there are many node/MSBuild processes still running. This explains why retry loops often don't work.

    https://github.com/anysphere/node-gyp-win32-test/actions/runs/16590860920/job/46925908613

    Image

    In the build above I used npm ci || npm ci, but the second invocation fails right away with:

    npm error code EBUSY
    npm error syscall rmdir
    npm error path C:\a\node-gyp-win32-test\node-gyp-win32-test\node_modules\kerberos
    npm error errno -4082
    npm error EBUSY: resource busy or locked, rmdir 'C:\a\node-gyp-win32-test\node-gyp-win32-test\node_modules\kerberos'
    npm error A complete log of this run can be found in: C:\npm\cache\_logs\2025-07-29T08_34_26_597Z-debug-0.log
    
  19. tmm1 commented on Jul 29, 2025

    @tmm1
    Contributor

    FWIW if you use pnpm it works fine. So the issue here may be that npm itself is not waiting on or cleaning up parallel postinstalls when one fails. There also seems to be some difference in how the two build node-addon-api, which multiple packages depend on and cause the race in the npm case.

    https://github.com/anysphere/node-gyp-win32-test/actions/runs/16591354158/job/46927553476

  20. tmm1 commented on Jul 29, 2025

    @tmm1
    Contributor

    There also seems to be some difference in how the two build node-addon-api, which multiple packages depend on and cause the race in the npm case.

    Yep, it looks like pnpm is building them all in separate dirs so the file race doesn't occur.

    node_modules/.pnpm/@vscode+windows-ca-certs@0.3.3/node-addon-api@8.5.0/node_modules/node-addon-api/Release/obj/node_addon_api_except/node_add.5648AC9A.tlog/node_addon_api_except.lastbuildstate
    
    node_modules/.pnpm/@vscode+sqlite3@5.1.8-vscode/node-addon-api@8.5.0/node_modules/node-addon-api/Release/obj/node_addon_api_except/node_add.0A8EC9E5.tlog/node_addon_api_except.lastbuildstate
    
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions