Skip to content

build: make build faster (at least on Windows) #12281

Closed
Closed
build: make build faster (at least on Windows)#12281
Assignees
refack
Labels
buildIssues and PRs related to Node.js builds or CI infrastructure.windowsIssues and PRs related to the Windows platform.

Description

@refack
refack
opened on Apr 8, 2017
  • Version: *
  • Platform: Windows
  • Subsystem: build

The problem: even for minor changes building on Windows can take 10s of minutes.
This issue is for tracking several approaches to mitigate that.

  • make GYP less aggressive in recreating *.*proj files
  • better signaling to VS of the minimal change set
  • use compiler cache
  • make using ninja simpler
  • allow for debug build to use /lib/ files as is (skip mksnapshot)
  • shortcut the ICU toolchain
  • allow decoupling of /deps/ from main executable compilation & linking
  • easy way to cloud-source parts of the compilation

Activity

  1. refack
    added
    buildIssues and PRs related to Node.js builds or CI infrastructure.
    discussIssues opened for discussion and feedback.
    windowsIssues and PRs related to the Windows platform.
    on Apr 8, 2017

  2. refack commented on Apr 8, 2017

    @refack
    refack
    on Apr 8, 2017
    ContributorAuthor

    /cc @nodejs/build @nodejs/platform-windows

  3. refack
    self-assigned this
    on Apr 8, 2017

  4. gibfahn commented on Apr 8, 2017

    @gibfahn
    gibfahn
    on Apr 8, 2017
    Member

    vcbuild.bat could definitely do with some love, thanks for picking this up @refack!

  5. refack commented on Apr 8, 2017

    @refack
    refack
    on Apr 8, 2017
    ContributorAuthor

    🙏may the powers that be protect us from the evil that is cmd scripting

  6. refack commented on Apr 8, 2017

    @refack
    refack
    on Apr 8, 2017
    ContributorAuthor

    vcbuild.bat could definitely do with some love

    Was thinking of rewriting it PowerShell

  7. gibfahn commented on Apr 8, 2017

    @gibfahn
    gibfahn
    on Apr 8, 2017
    Member

    Was thinking of rewriting it PowerShell

    Hmm, not sure what the rest of @nodejs/build and @nodejs/platform-windows would think of that, but it sounds like it might be a good idea to me. Fundamentally I think people would be more willing to learn and contribute to a Powershell build script than a batch one, and the tangled mess of gotos in vcbuild.bat doesn't seem particularly contributor-friendly.

    Or maybe we should write it in js...

  8. refack commented on Apr 8, 2017

    @refack
    refack
    on Apr 8, 2017
    ContributorAuthor

    Actually could be js more specifically JScript. Windows does have a built in JScript runner

  9. refack commented on Apr 19, 2017

    @refack
    refack
    on Apr 19, 2017
    ContributorAuthor

    Timing example, after applying #12511 (a change in /lib/path.js)

    C:\code\node>time < nul&vcbuild > nul&time < nul
The current time is: 14:07:53.76
Enter the new time: The current time is: 14:23:41.10
Enter the new time:

    ~16 minutes

  10. sam-github commented on Apr 19, 2017

    @sam-github
    sam-github
    on Apr 19, 2017
    Contributor

    Is that faster or slower than the before time?

  11. refack commented on Apr 19, 2017

    @refack
    refack
    on Apr 19, 2017
    ContributorAuthor

    Just an example of a recompile after just changing a .js file :(

    Mean while on "Ubuntu on Windows"

    refael@refaelux:/mnt/d/code/node$ time BUILDTYPE=Debug make -C out -j3
make: Entering directory '/mnt/d/code/node/out'
  ACTION Regenerating Makefile
-fmake --ignore-environment --toplevel-dir=. -I/mnt/d/code/node/common.gypi -I/mnt/d/code/node/config.gypi --depth=. -Goutput_dir=/mnt/d/code/node/out --generator-output=/mnt/d/code/node/out -Dcomponent=static_library -Dlibrary=static_library -Dlinux_use_bundled_binutils=0 -Dlinux_use_bundled_gold=0 -Dlinux_use_gold_flags=0 node.gyp /mnt/d/code/node/node.gyp -I /mnt/d/code/node/common.gypi -I /mnt/d/code/node/config.gypi --depth=. --generator-output /mnt/d/code/node/out -Goutput_dir=/mnt/d/code/node/out -Dcomponent=static_library -Dlibrary=static_library -Dlinux_use_bundled_binutils=0 -Dlinux_use_bundled_gold=0 -Dlinux_use_gold_flags=0
  ACTION _mnt_d_code_node_node_gyp_node_js2c_host_node_js2c /mnt/d/code/node/out/Debug/obj/gen/node_javascript.cc
  TOUCH 77ac9600461e1dfdf3594f5e11b3cd135d0cbfd6.intermediate
  ACTION Generating inspector protocol sources from protocol json 77ac9600461e1dfdf3594f5e11b3cd135d0cbfd6.intermediate
  CXX(target) /mnt/d/code/node/out/Debug/obj.target/node/gen/node_javascript.o
  LINK(target) /mnt/d/code/node/out/Debug/node
rm 77ac9600461e1dfdf3594f5e11b3cd135d0cbfd6.intermediate
make: Leaving directory '/mnt/d/code/node/out'

real    1m25.091s
user    0m45.625s
sys     0m33.406s

    [linux envy]

  12. sam-github commented on Apr 19, 2017

    @sam-github
    sam-github
    on Apr 19, 2017
    Contributor

    Not that much faster for me on Linux. C++ linking of Debug symbols is brutal. I worked with a program that had a 45 minute non-debug linktime once. Debug couldn't be linked, not enough memory on our machines.

  13. refack commented on Apr 19, 2017

    @refack
    refack
    on Apr 19, 2017
    ContributorAuthor

    Same scenario, just changing a .js file, but build with ninja

    c:\code\node\out.ninja\out\Debug$ time < nul &echo.""&ninja -j 3 -v -f build.ninja node&time < nul
The current time is: 18:50:11.05
Enter the new time: ""
[1/5] C:\bin\dev\python27\python.exe gyp-win-tool action-wrapper environment.x64 node_js2c_target_node_js2c_bce0facc3e66b523ccf35ab2d859c2d9..rsp ..\..\..\
[2/5] C:\bin\dev\python27\python.exe gyp-win-tool stamp obj\node.actions_depends.stamp
[3/5] C:\bin\dev\python27\python.exe gyp-win-tool stamp obj\node.compile_depends.stamp
[4/5] ninja -t msvc -e environment.x64 -- "D:\bin\dev\VS\2017\BuildTools\VC\Tools\MSVC\14.10.25017\bin\HostX64\x64\cl.exe" /nologo /showIncludes /FC @obj\gen\node.node_javascript.obj.rsp /c gen\node_javascript.cc /Foobj\gen\node.node_javascript.obj /Fdobj\node.cc.pdb
[5/5] C:\bin\dev\python27\python.exe gyp-win-tool link-with-manifests environment.x64 True node.exe "C:\bin\dev\python27\python.exe gyp-win-tool link-wrapper environment.x64 False link.exe /nologo /OUT:node.exe @node.exe.rsp" 1 mt.exe rc.exe "obj\node.node.exe.intermediate.manifest" obj\node.node.exe.generated.manifest ..\..\..\src\res\node.exe.extra.manifest
The current time is: 18:51:07.85
Enter the new time:

    56 seconds. I'm in love with ninja

  14. 6 remaining items

  15. Trott commented on Aug 2, 2017

    @Trott
    Trott
    on Aug 2, 2017
    Member

    Should this remain open? If so, does it make sense to make it a tracking issue with some check boxes so it's obvious at a glance what parts are done and what parts aren't?

  16. refack commented on Aug 2, 2017

    @refack
    refack
    on Aug 2, 2017
    ContributorAuthor

    Don't triagathon me 👼
    Yes, I'll break break this up. I have several better defined sub-tasks in mind.

  17. refack commented on Aug 2, 2017

    @refack
    refack
    on Aug 2, 2017
    ContributorAuthor

    /cc @nodejs/platform-windows now that we've grown in numbers, and at @Trott's behest, I've broken this issue down to several more tractable tasks. I'd be happy if we all pitched in (I've given most of the tasks some thought, but not enough work, so I'd be happy to discuss each of them).

  18. tniessen commented on Aug 3, 2017

    @tniessen
    tniessen
    on Aug 3, 2017
    Member

    I would prefer Powershell over JScript.

  19. refack commented on Aug 3, 2017

    @refack
    refack
    on Aug 3, 2017
    ContributorAuthor

    I would prefer Powershell over JScript.

    Yep, that's #12310 (I'm 80% there, just 80% more left) - https://gist.github.com/refack/969ecc6642981bfd7caef7d948d78076

  20. Trott
    removed
    discussIssues opened for discussion and feedback.
    on Mar 11, 2018

  21. jasnell commented on Jun 25, 2020

    @jasnell
    jasnell
    on Jun 25, 2020
    Member

    There's been no further discussion in years with no clear action here. Closing.

  22. jasnell
    closed this as completedon Jun 25, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

buildIssues and PRs related to Node.js builds or CI infrastructure.windowsIssues and PRs related to the Windows platform.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions