Excessive memory usage

I got a report about excessive memory usage. At first I thought it's wrongly deployed because I haven't seen it in a long time, but after upgrading to latest code I could trigger it as well. I'm now doing a poor man's bisect to see where it broke.

Perhaps the objgraph can help?

I first need to find at least approximately when it was introduced and how to trigger it. So far I was only able to trigger it once, and only after the original reporter insisted he's running the current code. Is this a new bug that was introduced last year (as I can't reproduce it with older code) or a continuation of the old leaks that I thought were fixed?

I did indeed used objgraph last time I was tracing the leaks.

Did you check this: 6ca3460? It comes to my mind first when I think about memleak.

@g1itch

@g1itch that was a workaround for a weird bug in Python's threading, the automatic gc was occasionally triggeigr a RecursionError.

Yes, I remember. But it still may be a good starting point for bisect.

I can't reproduce it even with more recent code so it's unlikely a good starting point

I was able to narrow it down a bit, between 7e1f1d2 (works ok) and 0331649 (can be triggered). That leaves only about 4 potential sources as most of the commits in between are code quality. Continuing testing.

Narrowed it futher, it's most likely the first one of these two:

a69732f ( Addrthread finish )
2a16538 ( Restrict outbound connections on network groups )

I'll narrow it down and try to write a minimal patch that stops the memory leak (even if it disables the functionality), and then I'll see how to fix the leak properly.

Pretty sure now it's a69732f

Don't upload addresses as they appear

If there is a memleak, it's barely visible. With 10 connections (2 inbound) memory_profiler.memory_usage() returns 105.36. objgraph.show_growth(limit=10) shows 1695 Peer objects.

diff --git a/src/bitmessagemain.py b/src/bitmessagemain.py
index d6cb289b..42fcb305 100755
--- a/src/bitmessagemain.py
+++ b/src/bitmessagemain.py
@@ -238,7 +238,7 @@ class Main(object):
         if daemon:
             with shared.printLock:
                 print('Running as a daemon. Send TERM signal to end.')
-            self.daemonize()
+            # self.daemonize()
 
         self.setSignalHandler()
 
diff --git a/src/class_singleCleaner.py b/src/class_singleCleaner.py
index b9fe3d1c..14c99f9a 100644
--- a/src/class_singleCleaner.py
+++ b/src/class_singleCleaner.py
@@ -23,6 +23,9 @@ import gc
 import os
 import time
 
+import objgraph
+from memory_profiler import memory_usage
+
 import knownnodes
 import queues
 import shared
@@ -31,7 +34,7 @@ import tr
 from bmconfigparser import BMConfigParser
 from helper_sql import sqlExecute, sqlQuery
 from inventory import Inventory
-from network import BMConnectionPool, StoppableThread
+from network import BMConnectionPool, StoppableThread, stats
 
 
 class singleCleaner(StoppableThread):
@@ -150,6 +153,15 @@ class singleCleaner(StoppableThread):
 
             gc.collect()
 
+            objgraph.show_growth(limit=10)
+
+            self.logger.warning('Memory used: %sM', memory_usage().pop())
+
+            connections = stats.connectedHostsList()
+            self.logger.warning('Connections (%s):', len(connections))
+            for c in connections:
+                self.logger.warning('-> %s', c.destination)
+
             if state.shutdown == 0:
                 self.stop.wait(singleCleaner.cycleLength)

$ BITMESSAGE_HOME=/tmp/bmtest mprof run --include-children python2 src/bitmessagemain.py
$ mprof plot --output /tmp/bmtest/memory-profile.png

Don't upload addresses as they appear

Don't put addresses into queue

No much difference even than ran overnight:

Navigation Menu

Repository navigation

Excessive memory usage #1598

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Excessive memory usage #1598

Description

Activity

g1itch commented on Jan 31, 2020

PeterSurda commented on Jan 31, 2020

g1itch commented on Jan 31, 2020

PeterSurda commented on Jan 31, 2020

g1itch commented on Jan 31, 2020

PeterSurda commented on Jan 31, 2020

PeterSurda commented on Feb 2, 2020

PeterSurda commented on Feb 2, 2020

PeterSurda commented on Feb 3, 2020

g1itch commented on Feb 3, 2020

g1itch commented on Feb 5, 2020

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions