Skip to content

Performance regression after updating to V8 11.3 #47350

Closed
Closed
Performance regression after updating to V8 11.3#47350
Labels
performanceIssues and PRs related to the performance of Node.js.

Description

@anonrig
anonrig
opened on Mar 31, 2023

I realized that after V8 11.3 the benchmarks changed a lot. As far as I know, Ada performs worse right now, and I'm researching the root cause of this.

Prior to V8 here is the benchmark CI result of Ada: https://ci.nodejs.org/view/Node.js%20benchmark/job/benchmark-node-micro-benchmarks/1307/

Comparing main branch before V8 11.3 merging with Ada branch (with V8 11.3)

➜  node git:(update-ada-v2) ✗ node-benchmark-compare url-comparing-v10-with-ada.csv
                                                                                         confidence improvement accuracy (*)   (**)  (***)
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='auth' withBase='false'              ***     13.79 %       ±2.04% ±2.72% ±3.54%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='auth' withBase='true'               ***     12.54 %       ±2.60% ±3.46% ±4.51%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='dot' withBase='false'               ***    102.67 %       ±2.20% ±2.93% ±3.82%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='dot' withBase='true'                ***     91.53 %       ±2.84% ±3.79% ±4.93%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='file' withBase='false'              ***    -37.76 %       ±1.57% ±2.09% ±2.73%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='file' withBase='true'               ***    -36.73 %       ±1.49% ±1.98% ±2.59%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='idn' withBase='false'               ***     13.55 %       ±1.63% ±2.17% ±2.82%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='idn' withBase='true'                ***     12.03 %       ±3.29% ±4.38% ±5.70%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='javascript' withBase='false'        ***      6.59 %       ±2.42% ±3.22% ±4.19%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='javascript' withBase='true'         ***      5.79 %       ±1.66% ±2.22% ±2.90%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='long' withBase='false'              ***     25.56 %       ±2.65% ±3.53% ±4.60%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='long' withBase='true'               ***     22.16 %       ±2.59% ±3.44% ±4.48%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='percent' withBase='false'           ***     32.07 %       ±1.58% ±2.11% ±2.75%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='percent' withBase='true'            ***     25.25 %       ±1.65% ±2.19% ±2.85%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='short' withBase='false'             ***     11.55 %       ±3.28% ±4.39% ±5.76%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='short' withBase='true'              ***      8.60 %       ±3.71% ±4.96% ±6.50%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='wpt' withBase='false'               ***     35.73 %       ±1.46% ±1.94% ±2.55%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='wpt' withBase='true'                ***     14.72 %       ±2.33% ±3.12% ±4.09%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='ws' withBase='false'                ***     17.45 %       ±2.89% ±3.85% ±5.01%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='ws' withBase='true'                 ***     15.45 %       ±1.94% ±2.59% ±3.39%

Be aware that when doing many comparisons the risk of a false-positive result increases.
In this case, there are 20 comparisons, you can thus expect the following amount of false-positive results:
  1.00 false positives, when considering a   5% risk acceptance (*, **, ***),
  0.20 false positives, when considering a   1% risk acceptance (**, ***),
  0.02 false positives, when considering a 0.1% risk acceptance (***)

Comparing current main branch with Ada branch (both on V8 11.3)

➜  node git:(update-ada-v2) ✗ node-benchmark-compare url.csv
                                                                                         confidence improvement accuracy (*)   (**)  (***)
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='auth' withBase='false'              ***     11.85 %       ±1.05% ±1.40% ±1.82%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='auth' withBase='true'               ***      9.00 %       ±0.79% ±1.05% ±1.36%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='dot' withBase='false'               ***    100.67 %       ±2.00% ±2.68% ±3.50%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='dot' withBase='true'                ***     81.76 %       ±1.47% ±1.97% ±2.58%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='file' withBase='false'              ***      7.12 %       ±1.26% ±1.68% ±2.19%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='file' withBase='true'               ***      4.10 %       ±0.80% ±1.07% ±1.39%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='idn' withBase='false'               ***      9.33 %       ±0.91% ±1.21% ±1.58%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='idn' withBase='true'                ***      7.89 %       ±0.81% ±1.08% ±1.41%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='javascript' withBase='false'        ***      5.47 %       ±1.31% ±1.75% ±2.29%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='javascript' withBase='true'         ***      3.72 %       ±1.07% ±1.42% ±1.86%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='long' withBase='false'              ***     22.05 %       ±1.22% ±1.63% ±2.13%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='long' withBase='true'               ***     18.64 %       ±0.95% ±1.26% ±1.64%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='percent' withBase='false'           ***    103.22 %       ±1.39% ±1.86% ±2.43%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='percent' withBase='true'            ***     88.36 %       ±1.07% ±1.42% ±1.86%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='short' withBase='false'             ***      7.02 %       ±1.11% ±1.48% ±1.93%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='short' withBase='true'              ***      4.21 %       ±1.08% ±1.44% ±1.88%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='wpt' withBase='false'               ***     29.17 %       ±1.28% ±1.71% ±2.24%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='wpt' withBase='true'                ***      5.96 %       ±1.49% ±1.99% ±2.62%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='ws' withBase='false'                ***     14.88 %       ±1.10% ±1.47% ±1.92%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='ws' withBase='true'                 ***     11.53 %       ±0.95% ±1.26% ±1.64%

Be aware that when doing many comparisons the risk of a false-positive result increases.
In this case, there are 20 comparisons, you can thus expect the following amount of false-positive results:
  1.00 false positives, when considering a   5% risk acceptance (*, **, ***),
  0.20 false positives, when considering a   1% risk acceptance (**, ***),
  0.02 false positives, when considering a 0.1% risk acceptance (***)

cc @targos @nodejs/performance

Activity

  1. anonrig commented on Mar 31, 2023

    @anonrig
    anonrig
    on Mar 31, 2023
    MemberAuthor

    This is definitely not a regression on the recent changes I've did to Ada branch. Reverting all of those changes still didn't make a difference.

  2. anonrig commented on Mar 31, 2023

    @anonrig
    anonrig
    on Mar 31, 2023
    MemberAuthor

    Comparing Node 19.8.1 with current main branch

                                                                                             confidence improvement accuracy (*)   (**)  (***)
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='auth' withBase='false'                       1.51 %       ±2.04% ±2.73% ±3.58%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='auth' withBase='true'                        2.44 %       ±2.56% ±3.41% ±4.44%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='dot' withBase='false'               ***    -41.41 %       ±1.42% ±1.89% ±2.47%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='dot' withBase='true'                ***    -37.82 %       ±4.29% ±5.76% ±7.62%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='file' withBase='false'              ***    -43.22 %       ±1.63% ±2.17% ±2.83%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='file' withBase='true'               ***    -39.59 %       ±1.74% ±2.33% ±3.07%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='idn' withBase='false'                        2.57 %       ±2.63% ±3.49% ±4.55%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='idn' withBase='true'                         1.21 %       ±2.30% ±3.06% ±3.98%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='javascript' withBase='false'        ***    -42.30 %       ±1.20% ±1.61% ±2.11%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='javascript' withBase='true'         ***    -39.76 %       ±1.11% ±1.48% ±1.93%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='long' withBase='false'                *      3.37 %       ±2.64% ±3.52% ±4.59%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='long' withBase='true'                **      3.22 %       ±2.23% ±2.99% ±3.92%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='percent' withBase='false'           ***    -35.73 %       ±1.65% ±2.20% ±2.87%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='percent' withBase='true'            ***    -33.85 %       ±1.75% ±2.33% ±3.04%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='short' withBase='false'             ***    -40.80 %       ±3.16% ±4.24% ±5.59%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='short' withBase='true'              ***    -38.83 %       ±1.20% ±1.60% ±2.08%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='wpt' withBase='false'               ***     -9.28 %       ±3.97% ±5.34% ±7.06%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='wpt' withBase='true'                ***      5.65 %       ±2.36% ±3.14% ±4.09%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='ws' withBase='false'                        -0.15 %       ±2.97% ±3.95% ±5.14%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='ws' withBase='true'                          0.58 %       ±3.00% ±3.99% ±5.20%

Be aware that when doing many comparisons the risk of a false-positive result increases.
In this case, there are 20 comparisons, you can thus expect the following amount of false-positive results:
  1.00 false positives, when considering a   5% risk acceptance (*, **, ***),
  0.20 false positives, when considering a   1% risk acceptance (**, ***),
  0.02 false positives, when considering a 0.1% risk acceptance (***)

  3. jasnell commented on Mar 31, 2023

    @jasnell
    jasnell
    on Mar 31, 2023
    Member

    /cc @nodejs/v8 @targos

  4. anonrig commented on Mar 31, 2023

    @anonrig
    anonrig
    on Mar 31, 2023
    MemberAuthor

    I'm waiting on merging this PR in order to understand it better: #47340

  5. anonrig commented on Mar 31, 2023

    @anonrig
    anonrig
    on Mar 31, 2023
    MemberAuthor

    Once noticable change I saw on the V8 update is: we enabled turbofan compiler, and disabled v8_enable_static_roots

    0d75c05#diff-d53f68b29a1c48c958c2e6779cc25c916a986357c6010dd01421c17adcf2f09bR282

  6. anonrig commented on Mar 31, 2023

    @anonrig
    anonrig
    on Mar 31, 2023
    MemberAuthor

    Ada microbenchmarks still show the same result (295ns/iter) prior to V8 update, but the node benchmarks show bad results.

    cpu: Apple M1 Max
runtime: node v20.0.0-pre (arm64-darwin)

benchmark         time (avg)             (min … max)       p75       p99      p995
---------------------------------------------------- -----------------------------
noop          316.15 ps/iter     (300 ps … 11.49 ns)  312.5 ps  337.5 ps  345.8 ps
noop            4.08 ns/iter    (4.04 ns … 10.52 ns)   4.06 ns   4.55 ns   4.94 ns
basic domain  295.26 ns/iter (290.07 ns … 309.39 ns) 298.61 ns 307.27 ns 309.39 ns
ipv4          262.41 ns/iter (257.75 ns … 276.33 ns) 265.86 ns 270.04 ns 270.13 ns
ipv6          292.75 ns/iter (284.33 ns … 354.96 ns)  295.3 ns 326.28 ns 354.96 ns
complex       438.23 ns/iter  (433.1 ns … 451.02 ns) 441.98 ns 449.62 ns 451.02 ns
  7. VoltrexKeyva
    added
    performanceIssues and PRs related to the performance of Node.js.
    on Mar 31, 2023

  8. mscdex commented on Mar 31, 2023

    @mscdex
    mscdex
    on Mar 31, 2023
    Contributor

    we enabled turbofan compiler,

    TurboFan has been enabled by default for years now ever since Crankshaft was removed. The addition of the v8_enable_turbofan flag in the V8 gypi is just making that more explicit, so there should be no effective difference.

  9. anonrig commented on Apr 1, 2023

    @anonrig
    anonrig
    on Apr 1, 2023
    MemberAuthor

    cc @joyeecheung

  10. anonrig commented on Apr 1, 2023

    @anonrig
    anonrig
    on Apr 1, 2023
    MemberAuthor

    My local benchmarks show really bad results, but benchmark CI seems to persist with the current results. I'm closing this issue and open a new one if the situation happens again.

    CI: https://ci.nodejs.org/view/Node.js%20benchmark/job/benchmark-node-micro-benchmarks/1314/consoleText

    url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='auth' withBase='false'              ***    106.73 %      ±19.04% ±25.55% ±33.70%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='auth' withBase='true'               ***    111.98 %      ±18.10% ±24.30% ±32.05%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='dot' withBase='false'               ***    112.52 %      ±16.28% ±21.83% ±28.73%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='dot' withBase='true'                ***     91.22 %      ±17.19% ±23.05% ±30.35%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='file' withBase='false'              ***     31.72 %      ±12.53% ±16.68% ±21.72%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='file' withBase='true'               ***     31.05 %      ±12.37% ±16.46% ±21.43%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='idn' withBase='false'               ***     83.54 %      ±15.31% ±20.49% ±26.90%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='idn' withBase='true'                ***     81.76 %      ±16.08% ±21.57% ±28.44%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='javascript' withBase='false'        ***    137.30 %      ±18.20% ±24.43% ±32.23%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='javascript' withBase='true'         ***    111.40 %      ±16.83% ±22.53% ±29.58%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='long' withBase='false'              ***     25.11 %       ±8.80% ±11.73% ±15.28%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='long' withBase='true'               ***     23.16 %       ±8.63% ±11.52% ±15.06%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='percent' withBase='false'           ***     41.12 %      ±14.86% ±19.79% ±25.79%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='percent' withBase='true'            ***     37.21 %      ±12.93% ±17.23% ±22.48%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='short' withBase='false'             ***    123.39 %      ±18.93% ±25.36% ±33.34%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='short' withBase='true'              ***    103.11 %      ±17.88% ±23.97% ±31.54%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='wpt' withBase='false'               ***     35.04 %       ±2.25%  ±3.02%  ±3.98%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='wpt' withBase='true'                ***      9.76 %       ±4.21%  ±5.62%  ±7.36%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='ws' withBase='false'                ***    133.03 %      ±19.31% ±25.91% ±34.16%
url/legacy-vs-whatwg-url-parse.js method='whatwg' e=1 type='ws' withBase='true'                 ***    113.69 %      ±18.69% ±25.08% ±33.08%

Be aware that when doing many comparisons the risk of a false-positive
result increases. In this case, there are 40 comparisons, you can thus
expect the following amount of false-positive results:
  2.00 false positives, when considering a   5% risk acceptance (*, **, ***),
  0.40 false positives, when considering a   1% risk acceptance (**, ***),
  0.04 false positives, when considering a 0.1% risk acceptance (***)
  11. anonrig
    closed this as completedon Apr 1, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    performanceIssues and PRs related to the performance of Node.js.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions