Possible race condition with psutil

@fabianp

Hi again,

I think that I may have found a possible race condition when counting the memory with psutil of a process using the include_children option. The problem (I think) is in this piece of code in _get_memory:

if include_children:
    for p in process.get_children(recursive=True):
        mem += p.get_memory_info()[0] / _TWO_20

The method get_childrenreturns a list that is used to iterate over and calculate the total memory. It may happen though that one of the child processes dies or finishes before the sum has finished, resulting on an error like this:

Reading configuration from '/pica/h1/guilc/repos/facs/tests/data/bin/fastq_screen.conf'
Using 1 threads for searches
Adding database phiX
Processing /pica/h1/guilc/repos/facs/tests/data/synthetic_fastq/simngs_phiX_100.fastq
Output file /pica/h1/guilc/repos/facs/tests/data/tmp/simngs_phiX_100_screen.txt already exists - skipping
Processing complete
Process MemTimer-2:
Traceback (most recent call last):
  File "/sw/comp/python/2.7_kalkyl/lib/python2.7/multiprocessing/process.py", line 232, in _bootstrap
    self.run()
  File "/pica/h1/guilc/.virtualenvs/facs/lib/python2.7/site-packages/memory_profiler.py", line 124, in run
    include_children=self.include_children)
  File "/pica/h1/guilc/.virtualenvs/facs/lib/python2.7/site-packages/memory_profiler.py", line 52, in _get_memory
    mem += p.get_memory_info()[0] / _TWO_20
  File "/pica/h1/guilc/.virtualenvs/facs/lib/python2.7/site-packages/psutil/__init__.py", line 758, in get_memory_info
    return self._platform_impl.get_memory_info()
  File "/pica/h1/guilc/.virtualenvs/facs/lib/python2.7/site-packages/psutil/_pslinux.py", line 470, in wrapper
    raise NoSuchProcess(self.pid, self._process_name)
NoSuchProcess: process no longer exists (pid=17442)

It happens randomly, and can be solved encapsulating the sum on a try except statement:

if include_children:
    for p in process.get_children(recursive=True):
        try:
            mem += p.get_memory_info()[0] / _TWO_20
        except NoSuchProcess:
            pass

I'm not sure that this is the best solution though... any comments/ideas? @fabianp @brainstorm

Thanks!

@guillermo-carrasco

That clearly biases the measurement since the information is lost (together with @guillermo-carrasco we saw some strange numbers after this tweak).

I guess the next step would be reproducing the error (in tests?), which might prove tricky. @fabianp, we are having this issue while we run this particular test:

https://github.com/SciLifeLab/facs/blob/master/tests/test_fastqscreen.py#L112

If you see something clearly wrong in the way we call it, please, let us know.

@guillermo-carrasco

@guillermo-carrasco I guess you meant OSError as the exception, since:

  File "/home/roman/.pyenv/versions/2.7.6/lib/python2.7/site-packages/memory_profiler.py", line 53, in _get_memory
    except NoSuchProcess:
NameError: global name 'NoSuchProcess' is not defined

Not really, I meant NoSuchProcess, you have to import it from psutil

This just happened to me on a profiling run. Seeing how the uncaught exception causes the profiling to stop completely, I'd prefer to just miss out on the short-running job's memory usage.

Fix for issue #71

I pushed a fix in a9e5a02 . Feel free to reopen/send a pull request if it doesn't solve the issue.

Didn't try this yet, but it looks like that won't work. I'm clearly getting a psutil.NoSuchProcess: psutil.NoSuchProcess process no longer exists (pid=1363), not an OSError. But I can try a similar patch.

It would indeed be great if you can send a pull request for that

…

On Tue, Jan 10, 2017 at 3:10 PM, Kai Blin ***@***.***> wrote: Didn't try this yet, but it looks like that won't work. I'm clearly getting a psutil.NoSuchProcess: psutil.NoSuchProcess process no longer exists (pid=1363), not an OSError. But I can try a similar patch. — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <https://github.com/fabianp/memory_profiler/issues/71#issuecomment-271584350>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAQ8hwdisHB0f2IK6Sajep0HT9qZuhW4ks5rQ5E6gaJpZM4Bfo5e> .

Catch the right exception for issue #71

Navigation Menu

Repository navigation

Possible race condition with psutil #71

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Possible race condition with psutil #71

Description

Activity

brainstorm commented on Feb 7, 2014

brainstorm commented on Feb 10, 2014

guillermo-carrasco commented on Feb 11, 2014

kblin commented on Jan 10, 2017

fabianp commented on Jan 10, 2017

kblin commented on Jan 10, 2017

fabianp commented on Jan 10, 2017

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions