Repository navigation
Add necessary files to support running the octane benchmark - #79
CurryKitten wants to merge 3 commits into
Conversation
|
I've not got an environment to hand to check, however if my memory serves Octane throws out individual scores for each of the different tests, rather than one overall score. We then have a decision to make whether we take a geomean of all the scores (in which case we need to include some calculation of a geomean in the jenkins job or have a wrapper script in the repo), or store each score seperately (in which case we need to add more lines to the octane.json file / or perhaps multipe files). The skipping on certain hardware platforms was a big endian vs little endian thing, that some tests don't work properly due to binary little endian data in the tests. I'm not sure what the plan going forward is - there was some talk of extra hardware for benchmarking in the community - if this is going to be exclusively little endian, then we don't need to worry - if there's a chance of there being big endian hardware too, we may need to point out that a geomean is the little endian geomean. |
|
@gareth-ellis Octane does put out a score at the end, which is a Geomean. Although this did also appear in our jobs, for some reason we were calculating the results manually from the scores of all the individual benchmarks... although the results are at least the same. |
|
@CurryKitten , I wonder if that was due to the big endian / little endian calculation.... |
PR-URL: #79 Reviewed-by: Michael Dawson <michael_dawson@ca.ibm.com> Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
|
@CurryKitten as a note for next time the commit message was longer than the allowed 52 chars. I shortened it as part of landing. Landed as: 15fdf84 |
In support of #75 this PR adds the .js file to run Octane and the .json file to support the chart generation.
Our internal version of Octane had several layers of script wrappers, which I've removed to keep things as simple as possible. Our .js wrapper also split out the tests individually in order to allow us to bypass certain tests on certain hardware platforms - so I'm using the code that @bnoordhuis pasted in the original issue - with a very small mod.
In order to run this via Jenkins, the process would be -
The normal clone/make for node.js
git clone nodejs/benchmarking
git clone chromium/octane (https://github.com/chromium/octane.git)
In order to make sure we always test on a consistent version of octane I suggest we checkout the (as of now) latest commit, as there's only one release/tag from 2013
git checkout 4852334fea692f02ce1504f16c625d63abb15183Run the benchmark, keeping a log to extract the score from and allowing the metrics to echo to STDOUT so jenkins can capture them, then grab the score with awk