Repository navigation
Running the test suite with --junit-xml and -R incorrectly reports refleaks #83434
Description
| BPO | 39253 |
|---|---|
| Nosy | @vstinner, @zooba, @pablogsal |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2020-01-07.23:54:53.841>
labels = ['tests', '3.9']
title = 'Running the test suite with --junit-xml and -R incorrectly reports refleaks'
updated_at = <Date 2020-01-08.17:31:56.794>
user = 'https://github.com/pablogsal'bugs.python.org fields:
activity = <Date 2020-01-08.17:31:56.794>
actor = 'pablogsal'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Tests']
creation = <Date 2020-01-07.23:54:53.841>
creator = 'pablogsal'
dependencies = []
files = []
hgrepos = []
issue_num = 39253
keywords = []
message_count = 9.0
messages = ['359561', '359562', '359563', '359606', '359608', '359610', '359611', '359612', '359615']
nosy_count = 3.0
nosy_names = ['vstinner', 'steve.dower', 'pablogsal']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue39253'
versions = ['Python 3.9']
Activity
pablogsal commentedon Jan 7, 2020
./python -m test test_list -R : --junit-xml something
0:00:00 load avg: 0.52 Run tests sequentially
0:00:00 load avg: 0.52 [1/1] test_list
beginning 9 repetitions
123456789
.........
test_list leaked [798, 798, 798, 798] references, sum=3192
test_list leaked [345, 345, 345, 345] memory blocks, sum=1380
test_list failed
== Tests result: FAILURE ==
1 test failed:
test_list
Total duration: 3.4 sec
Tests result: FAILURE
pablogsal commentedon Jan 7, 2020
This is going to be very noisy now that we are using --junit-xml in the buildbots...
vstinner commentedon Jan 8, 2020
Oh. That explains these strange results on AMD64 RHEL7 Refleaks 3.8:
https://buildbot.python.org/all/#/builders/219/builds/41
argv: [b'make', b'buildbottest', b'TESTOPTS=-R 3:3 -u-cpu --junit-xml test-results.xml -j10 ${BUILDBOT_TESTOPTS}', b'TESTPYTHONOPTS=', b'TESTTIMEOUT=11700']
...
test_quopri leaked [168, 168, 168] references, sum=504
test_quopri leaked [75, 75, 75] memory blocks, sum=225
1:50:20 load avg: 1.06 Re-running test_future5 in verbose mode
test_future5 leaked [42, 42, 42] references, sum=126
test_future5 leaked [20, 20, 20] memory blocks, sum=60
...
The previous build is successful and it didn't use --junit-xml option:
https://buildbot.python.org/all/#/builders/219/builds/40
zooba commentedon Jan 8, 2020
We can stop using --junit-xml if we want. GitHub Actions doesn't support displaying test results anyway, so once Azure Pipelines is switched off we'll just be reading the console all the time anyway (unless Buildbot has a way of rendering test results that I haven't noticed?)
zooba commentedon Jan 8, 2020
My guess would be that the reference tracking needs some kind of awareness for logging results. The results are collected as regular Python objects and formatted later, as we need to calculate totals before writing the XML file.
I guess keeping those objects around looks like a leak?
pablogsal commentedon Jan 8, 2020
We are using --junit-xml on the buildbots to analyze failures and in the future we can use it to better display the summary, as is more reliable than parsing the longs.
Yeah, because it seems that we collect those objects in the innermost runner, while the refleak runner runs the whole "test suite" (the test file, like test_list.py) as one unit.
To be honest, the easiest fix here maybe is just deactivating the --junit and make a warning as in bpo-27103
pablogsal commentedon Jan 8, 2020
I mean deactivating the --junit *if* is used together with -R
vstinner commentedon Jan 8, 2020
FYI test_io is skipped in Python 2.7 when using -R, because test_io is not reliable. Extract of Lib/test/test_io.py:
I'm not shocked by disabling JUnit feature when using -R. But I suggest to raise an error and tune the buildbot configuration to not use it, rather than silently ignore the option.
pablogsal commentedon Jan 8, 2020
I already configured the buildbots to not use --junit when running with =R
5 remaining items
vstinner commentedon Mar 25, 2024
The problem is that writing XML is done outside the code checking for refleaks, and creating the XML stores objects in memory. Pseudo-code:
Maybe a solution would be to write partial XML in run_test() and then cumulate results in a final XML. Pseudo-code:
Or partial results can be Python objects written with pickle.
zooba commentedon Mar 25, 2024
Disabling result collection during leak tests is probably best. I don't think we collect the number of leaked objects in the results, so why would you turn it on? By mistake.
pythongh-83434: Disable XML in regrtest when -R option is used
vstinner commentedon Mar 25, 2024
@zooba:
Ok, I wrote PR gh-117232 for that.
gh-83434: Disable XML in regrtest when -R option is used (#117232)
pythongh-83434: Disable XML in regrtest when -R option is used (pytho…
pythongh-83434: Disable XML in regrtest when -R option is used (pytho…
vstinner commentedon Mar 26, 2024
Fixed by change d52bdfb.
[3.12] gh-83434: Sync libregrtest and test_regrtest with the main bra…
[3.12] pythongh-83434: Sync libregrtest and test_regrtest with the ma…
[3.11] [3.12] gh-83434: Sync libregrtest and test_regrtest with the m…
Squashed commit of the following:
Squashed commit of the following:
pythongh-83434: Disable XML in regrtest when -R option is used (pytho…