Repository navigation
coverage.nodejs.org misreporting #19912
Description
Activity
9th: https://ci.nodejs.org/job/node-test-commit-linux-coverage/600/nodes=benchmark/console
10th: https://ci.nodejs.org/job/node-test-commit-linux-coverage/601/nodes=benchmark/consoleDoes the output help? it has %'s that match what we're seeing on the table.
Maybe we just added a ton of code yesterday and didn't catch up with enough new tests ...
Those look really useful, thanks.
I had found these:
9th: https://ci.nodejs.org/job/node-test-commit-linux-coverage/600/
10th: https://ci.nodejs.org/job/node-test-commit-linux-coverage/601/
... but yours are better.I noticed from the job status pages there that job 601 took just 16 mins instead of the usual ~31 mins.
I reviewed the diff of the two outputs. Aside from the obvious size difference (601 is a shorter log), there are fewer files covered and the coverage for each file is generally worse.
It seems to point towards some missing files (both from the test suite and the source files).
Is it possible to re-run the job on the CI server?
maybe we just wait for today's run and see how it goes, perhaps we should delete yesterday's record if it's a simple anomaly
Today's run has the same numbers as yesterdays .. something's up with the codebase I think.
I just checked on the machine and the last login was Apr 3 so I don't think this is about a software update or anything on the system.Reacted by chrismillerukTrying to repro the issue locally is proving difficult. I'm not familiar with the codebase and build setup so any pointers appreciated. I found some commands in the build repo but I can't confirm that these are the same as used by
node-test-commit-linux-coveragejob.
https://github.com/nodejs/build/tree/master/jenkins/scripts/coverage#coverage-job./configure --coverage make coverage-clean NODE_TEST_DIR=${HOME}/node-tmp PYTHON=python COVTESTS=test-ci make coverage -j $(getconf _NPROCESSORS_ONLN)The above fails locally (macOS 10.13.4) with the following error:
error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool: unknown option character `-' in: --coverageThese are the commits between the last known good and the first known failure:
244af7a...d1156daThe OpenSSL changes look like a sensible suspect, there were some CI failures that have already been reviewed and discounted #19794 (comment) but it might be worth a second look?
@nodejs/testing
Looking at the console outputs, it seems like the issue is that
make testseems to fail during/after addon compilation. This is the original error message from https://ci.nodejs.org/job/node-test-commit-linux-coverage/601/nodes=benchmark/consoleFull:npm ERR! weird error structured-stack:1 npm ERR! weird error (function (){cov_12264jx1gp.f[26]++;cov_12264jx1gp.s[202]++;Error.prepareStackTrace=function(err,trace){cov_12264jx1gp.f[27]++;cov_12264jx1gp.s[203]++;err.stack=trace;};cov_12264jx1gp.s[204]++;Error.stackTraceLimit=Infinity;cov_12264jx1gp.s[205]++;return function structuredStack(){cov_12264jx1gp.f[28]++;cov_12264jx1gp.s[206]++;// eslint-disable-next-line no-restricted-syntax npm ERR! weird error ^ npm ERR! weird error npm ERR! weird error ReferenceError: cov_12264jx1gp is not defined npm ERR! weird error at structured-stack:1:14 npm ERR! weird error at structured-stack:2:30 npm ERR! weird error at Script.runInContext (vm.js:24:1196) npm ERR! weird error at Script.runInNewContext (vm.js:24:1416) npm ERR! weird error at runInNewContext (vm.js:28:1085) npm ERR! weird error at isInsideNodeModules (internal/util.js:36:44) npm ERR! weird error at showFlaggedDeprecation (buffer.js:26:744) npm ERR! weird error at new Buffer (buffer.js:41:94) npm ERR! weird error at Object.<anonymous> (/home/iojs/build/workspace/node-test-commit-linux-coverage/nodes/benchmark/deps/npm/node_modules/mississippi/node_modules/duplexify/index.js:6:20) npm ERR! weird error at Module._compile (internal/modules/cjs/loader.js:109:892)Aaand https://coverage.nodejs.org/ is back to normal! 🎉
Reacted by Richard Lau, Michaël Zasso and chrismillerukReacted by Richard Lau
https://coverage.nodejs.org/ is reporting a significant drop in code coverage in the most recent nightly.
JS Coverage: 94.42% -> 28.80%
C++ Coverage: 92.50% -> 31.10%
Note that my local coverage results seem OK so this looks like a build / job failure.