Repository navigation
test_runner_describe_it failures break TAP output #43773
Description
Version
main
Platform
No response
Subsystem
test
What steps will reproduce the bug?
No response
How often does it reproduce? Is there a required condition?
No response
What is the expected behavior?
No response
What do you see instead?
Seen on the CI. When test_runner_describe_it (formerly test_runner_destribe_it (#43764)) fails it breaks the tap2junit TAP parser we have in the CI:
e.g.
- https://ci.nodejs.org/job/node-test-commit-linuxone/32996/nodes=rhel8-s390x/consoleFull
- https://ci.nodejs.org/job/node-test-commit-osx/46125/nodes=osx11-x64/consoleFull
15:47:26 not ok 3127 message/test_runner_desctibe_it
15:47:26 ---
15:47:26 duration_ms: 3.801
15:47:26 severity: fail
15:47:26 exitcode: 1
15:47:26 stack: |-
15:47:26 TAP version 13
15:47:26 # Subtest: sync pass todo
15:47:26 ok 1 - sync pass todo # TODO
15:47:26 ---
15:47:26 duration_ms: 0.017320097
15:47:26 ...
15:47:26 # Subtest: sync pass todo with message
15:47:26 ok 2 - sync pass todo with message # TODO this is a passing todo
15:47:26 ---
15:47:26 duration_ms: 0.00010036
15:47:26 ...
15:47:26 # Subtest: sync fail todo
15:47:26 not ok 3 - sync fail todo # TODO
...
16:00:46 + tap2junit -i test.tap -o test.xml
16:00:48 Traceback (most recent call last):
16:00:48 File "/usr/local/bin/tap2junit", line 10, in <module>
16:00:48 sys.exit(main())
16:00:48 File "/Library/Python/3.8/site-packages/tap2junit/__main__.py", line 64, in main
16:00:48 convert(args.input, args.output)
16:00:48 File "/Library/Python/3.8/site-packages/tap2junit/__main__.py", line 43, in convert
16:00:48 result = parse(input_file, data)
16:00:48 File "/Library/Python/3.8/site-packages/tap2junit/__main__.py", line 35, in parse
16:00:48 tap_parser.parse(data)
16:00:48 File "/Library/Python/3.8/site-packages/tap2junit/tap13.py", line 174, in parse
16:00:48 self._parse(StringIO(source))
16:00:48 File "/Library/Python/3.8/site-packages/tap2junit/tap13.py", line 144, in _parse
16:00:48 raise ValueError("Descending test id on line: %r" % line)
16:00:48 ValueError: Descending test id on line: 'not ok 1 - +sync throw fail'Such failures are also breaking ncu-ci from node-core-utils (nodejs/node-core-utils#639 (comment)) which breaks the CI Reliability reports in https://github.com/nodejs/reliability.
Additional information
No response
Activity
MoLow commentedon Jul 11, 2022
Taking a look. might take me a few days
MoLow commentedon Jul 11, 2022
@richardlau do we know if this affects anything else besides node-core-utils?
richardlau commentedon Jul 11, 2022
@MoLow https://github.com/nodejs/tap2junit, as described above. In the Jenkins CI this is used to convert the TAP output of Node.js'
tools/test.pyto XML which Jenkins can parse -- we do this because the tap plugin for Jenkins has very bad performance (nodejs/build#453).F3n67u commentedon Jul 11, 2022
I experience multiple time when jenkins job say test.xml is empty, but I don't know what is wrong and ignored. If a job say test.xml is empty, then there must be some problem when run tap2junit right?
richardlau commentedon Jul 11, 2022
@F3n67u Yes, either tap2junit has failed to parse the tap output or the build failed to compile and no tests were run.
MoLow commentedon Jul 12, 2022
@richardlau please see my PR - the output is valid, it is that
tap2junitparser that cannot handle it