Skip to content
This repository was archived by the owner on Nov 28, 2020. It is now read-only.

Add necessary files to support running the octane benchmark - #79

Closed
CurryKitten wants to merge 3 commits into
nodejs:masterfrom
CurryKitten:master
Closed

CurryKitten wants to merge 3 commits into
nodejs:masterfrom
CurryKitten:master

Conversation

@CurryKitten

Copy link
Copy Markdown
Contributor

In support of #75 this PR adds the .js file to run Octane and the .json file to support the chart generation.

Our internal version of Octane had several layers of script wrappers, which I've removed to keep things as simple as possible. Our .js wrapper also split out the tests individually in order to allow us to bypass certain tests on certain hardware platforms - so I'm using the code that @bnoordhuis pasted in the original issue - with a very small mod.

In order to run this via Jenkins, the process would be -

The normal clone/make for node.js
git clone nodejs/benchmarking
git clone chromium/octane (https://github.com/chromium/octane.git)

In order to make sure we always test on a consistent version of octane I suggest we checkout the (as of now) latest commit, as there's only one release/tag from 2013

git checkout 4852334fea692f02ce1504f16c625d63abb15183

Run the benchmark, keeping a log to extract the score from and allowing the metrics to echo to STDOUT so jenkins can capture them, then grab the score with awk

node octane | tee octane.log
cat octane.log | awk -F: '/Score/ {print $2}' | sed 's/ //g'

@gareth-ellis

Copy link
Copy Markdown
Member

I've not got an environment to hand to check, however if my memory serves Octane throws out individual scores for each of the different tests, rather than one overall score.

We then have a decision to make whether we take a geomean of all the scores (in which case we need to include some calculation of a geomean in the jenkins job or have a wrapper script in the repo), or store each score seperately (in which case we need to add more lines to the octane.json file / or perhaps multipe files).

The skipping on certain hardware platforms was a big endian vs little endian thing, that some tests don't work properly due to binary little endian data in the tests. I'm not sure what the plan going forward is - there was some talk of extra hardware for benchmarking in the community - if this is going to be exclusively little endian, then we don't need to worry - if there's a chance of there being big endian hardware too, we may need to point out that a geomean is the little endian geomean.

bnoordhuis

This comment was marked as off-topic.

@CurryKitten

Copy link
Copy Markdown
Contributor Author

@gareth-ellis Octane does put out a score at the end, which is a Geomean. Although this did also appear in our jobs, for some reason we were calculating the results manually from the scores of all the individual benchmarks... although the results are at least the same.

@gareth-ellis

Copy link
Copy Markdown
Member

@CurryKitten , I wonder if that was due to the big endian / little endian calculation....

mhdawson

This comment was marked as off-topic.

mhdawson

This comment was marked as off-topic.

mhdawson pushed a commit that referenced this pull request Nov 24, 2016
PR-URL: #79
Reviewed-by: Michael Dawson <michael_dawson@ca.ibm.com>
Reviewed-By: Ben Noordhuis <info@bnoordhuis.nl>
@mhdawson

Copy link
Copy Markdown
Member

@CurryKitten as a note for next time the commit message was longer than the allowed 52 chars. I shortened it as part of landing.

Landed as: 15fdf84

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants