Repository navigation
Possible race condition with psutil #71
Description
Hi again,
I think that I may have found a possible race condition when counting the memory with psutil of a process using the include_children option. The problem (I think) is in this piece of code in _get_memory:
if include_children:
for p in process.get_children(recursive=True):
mem += p.get_memory_info()[0] / _TWO_20The method get_childrenreturns a list that is used to iterate over and calculate the total memory. It may happen though that one of the child processes dies or finishes before the sum has finished, resulting on an error like this:
Reading configuration from '/pica/h1/guilc/repos/facs/tests/data/bin/fastq_screen.conf'
Using 1 threads for searches
Adding database phiX
Processing /pica/h1/guilc/repos/facs/tests/data/synthetic_fastq/simngs_phiX_100.fastq
Output file /pica/h1/guilc/repos/facs/tests/data/tmp/simngs_phiX_100_screen.txt already exists - skipping
Processing complete
Process MemTimer-2:
Traceback (most recent call last):
File "/sw/comp/python/2.7_kalkyl/lib/python2.7/multiprocessing/process.py", line 232, in _bootstrap
self.run()
File "/pica/h1/guilc/.virtualenvs/facs/lib/python2.7/site-packages/memory_profiler.py", line 124, in run
include_children=self.include_children)
File "/pica/h1/guilc/.virtualenvs/facs/lib/python2.7/site-packages/memory_profiler.py", line 52, in _get_memory
mem += p.get_memory_info()[0] / _TWO_20
File "/pica/h1/guilc/.virtualenvs/facs/lib/python2.7/site-packages/psutil/__init__.py", line 758, in get_memory_info
return self._platform_impl.get_memory_info()
File "/pica/h1/guilc/.virtualenvs/facs/lib/python2.7/site-packages/psutil/_pslinux.py", line 470, in wrapper
raise NoSuchProcess(self.pid, self._process_name)
NoSuchProcess: process no longer exists (pid=17442)
It happens randomly, and can be solved encapsulating the sum on a try except statement:
if include_children:
for p in process.get_children(recursive=True):
try:
mem += p.get_memory_info()[0] / _TWO_20
except NoSuchProcess:
passI'm not sure that this is the best solution though... any comments/ideas? @fabianp @brainstorm
Thanks!
Activity
brainstorm commentedon Feb 7, 2014
That clearly biases the measurement since the information is lost (together with @guillermo-carrasco we saw some strange numbers after this tweak).
I guess the next step would be reproducing the error (in tests?), which might prove tricky. @fabianp, we are having this issue while we run this particular test:
https://github.com/SciLifeLab/facs/blob/master/tests/test_fastqscreen.py#L112
If you see something clearly wrong in the way we call it, please, let us know.
brainstorm commentedon Feb 10, 2014
@guillermo-carrasco I guess you meant
OSErroras the exception, since:guillermo-carrasco commentedon Feb 11, 2014
Not really, I meant NoSuchProcess, you have to import it from psutil
kblin commentedon Jan 10, 2017
This just happened to me on a profiling run. Seeing how the uncaught exception causes the profiling to stop completely, I'd prefer to just miss out on the short-running job's memory usage.
Fix for issue #71
fabianp commentedon Jan 10, 2017
I pushed a fix in a9e5a02 . Feel free to reopen/send a pull request if it doesn't solve the issue.
kblin commentedon Jan 10, 2017
Didn't try this yet, but it looks like that won't work. I'm clearly getting a
psutil.NoSuchProcess: psutil.NoSuchProcess process no longer exists (pid=1363), not anOSError. But I can try a similar patch.fabianp commentedon Jan 10, 2017
Catch the right exception for issue #71