large negative increment in %mprun on ipython #219

Problem

The %mprun line profiler reported an unexpected big negative increment values in a for loop. I came across this problem when running it on Google Colaboratory, and also another user said the same problem occurred on IPython on macOS.

I used most recent version of HEAD of master branch (memory_profiler 0.54.0 + PR #202 + something) by pip install -U https://github.com/pythonprofilers/memory_profiler/archive/master.zip, but the problem was reproduced.

Code

The original source code is from https://jakevdp.github.io/PythonDataScienceHandbook/01.07-timing-and-profiling.html by Jake VanderPlas.

!pip install -U https://github.com/pythonprofilers/memory_profiler/archive/master.zip
%load_ext memory_profiler

%%file mprun_demo.py
def sum_of_lists(N):
    total = 0
    for i in range(5):
        L = [j ^ (j >> i) for j in range(N)]
        total += sum(L)
        del L # remove reference to L
    return total

from mprun_demo import sum_of_lists
%mprun -f sum_of_lists sum_of_lists(1000000)

See the actual notebook here. You can try and re-run the above example.

Result

Filename: /content/mprun_demo.py

Line #    Mem usage    Increment   Line Contents
================================================
     1    150.3 MiB    150.3 MiB   def sum_of_lists(N):
     2    150.3 MiB      0.0 MiB       total = 0
     3    150.3 MiB      0.0 MiB       for i in range(5):
     4    181.1 MiB -46807065.7 MiB           L = [j ^ (j >> i) for j in range(N)]
     5    181.1 MiB      0.0 MiB           total += sum(L)
     6    150.3 MiB   -123.0 MiB           del L # remove reference to L
     7    150.3 MiB      0.0 MiB       return total

Environment

• IPython on Google Colaboratory
• memory_profiler 0.54.0 + 4 commits (959d9fc)

Related?

#9, #195, #211

Also the same problem is reported on Stack Overflow: "Strange increment value reported during IPython memory profiling".