Repository navigation
Run octane benchmarks as part of nightly benmarks #75
Description
Activity
@mhdawson Absolutely, I'll check how we're doing the Octane currently and see how this can be adapted for the community builds
If it helps, here is the script I use to run Octane locally:
'use strict'; const fs = require('fs'); const vm = require('vm'); global.load = load; global.read = read; global.print = console.log; load('run.js'); function load(filename) { vm.runInThisContext(read(filename)); } function read(filename) { return fs.readFileSync(filename, 'utf8'); }
I suppose we want something that writes a file (JSON?) with benchmark numbers?
As a small aside, it's interesting how much slower a new VM context is because of how global property loads/stores are intercepted. Code:
'use strict'; const fs = require('fs'); const vm = require('vm'); const context = vm.createContext({ load, read, print: console.log }); load('run.js'); function load(filename) { vm.runInContext(read(filename), context); } function read(filename) { return fs.readFileSync(filename, 'utf8'); }
Baseline, running in the main context with the code from #75 (comment):
Richards: 16977 DeltaBlue: 30242 Crypto: 18320 RayTrace: 40033 EarleyBoyer: 5395 RegExp: 2965 Splay: 10561 SplayLatency: 17355 NavierStokes: 20545 PdfJS: 9680 Mandreel: 14538 MandreelLatency: 35356 Gameboy: 32829 CodeLoad: 9013 Box2D: 13528 zlib: 45012 Typescript: 19904 ---- Score (version 9): 16300Running in a new context:
Richards: 288 DeltaBlue: 635 Crypto: 3192 RayTrace: 1813 EarleyBoyer: 765 RegExp: 2901 Splay: 2714 SplayLatency: 9947 NavierStokes: 19886 PdfJS: 2873 Mandreel: 71.8 MandreelLatency: 377 Gameboy: 11405 CodeLoad: 7358 Box2D: 3300 zlib: 41335 Typescript: 16536 ---- Score (version 9): 2676Performance relative to the baseline:
0.49 Mandreel 1.07 MandreelLatency 1.70 Richards 2.10 DeltaBlue 4.53 RayTrace 14.18 EarleyBoyer 17.42 Crypto 24.39 Box2D 25.70 Splay 29.68 PdfJS 34.74 Gameboy 57.31 SplayLatency 81.64 CodeLoad 83.08 Typescript 91.83 zlib 96.79 NavierStokes 97.84 RegExpIOW, RegExp is almost as fast while Mandreel runs 200x slower.
@CurryKitten provided scripts to run through #79 which has now landed.
I've updated the jobs to run and publish results
Kicked off jobs to validate across the versions, when we do manual chart generation run
Seems like it does not run on 0.12.x:
HEAD is now at 4852334... Merge pull request #41 from chromium/revert-40-revert-39-fix-richards-box + cd .. + cp /home/iojs/build/workspace/benchmark-footprint-experimental-0.12.x/benchmarking/benchmarks/octane/octane.js . + node octane.js+ tee octane.log /home/iojs/build/workspace/benchmark-footprint-experimental-0.12.x/octane.js:2 const fs = require('fs'); ^^^^^ SyntaxError: Use of const in strict mode. at exports.runInThisContext (vm.js:73:16) at Module._compile (module.js:443:25) at Object.Module._extensions..js (module.js:478:10) at Module.load (module.js:355:32) at Function.Module._load (module.js:310:12) at Function.Module.runMain (module.js:501:10) at startup (node.js:129:16) at node.js:814:3Seems like the issue is in our script. Can you submit a PR to change it to use var instead and validate it runs ok on 0.12.X
Everything should be in place now, but the www for benchmarking is not being mirrored as expected. Raised this issue: nodejs/build#556.
You can see the chart directly at: https://benchmarking.nodejs.org/charts/octane.png
Reacted by Timothy GuStill need help to get it showing up on bechmarking.nodejs.org
Ok octane benchmarks now showing up on benchmarking.nodejs.org
We had a a recent case where one of our floating patches on v8 caused a performance issue. We believe this could have been caught with octane runs.
I'm thinking this means we should run octane nightly on master and publish the results.
@CurryKitten I know we run octane internally, would you be able to help putting together something that we can use to run in the community builds ?