TLS/SSL asyncio leaks memory

Bug report

Bug description:

python3.9 without uvloop doesn't leaks memory (or noticeably smaller).
python3.11+ (and others?) leaks memory A LOT under load (with or without uvloop) - up to +2gb per every test!

test commands:
ab -n50000 -c15000 -r https://127.0.0.1/
(apt install apache2-utils)

import asyncio, ssl, uvloop

class HTTP(asyncio.Protocol):
    
    def __init__(self):
        self.transport = None
        
    def connection_made(self, transport):
        self.transport = transport

    def data_received(self, data):
        self.transport.write(
            b'HTTP/1.1 200 OK\r\nContent-Length: 0\r\nConnection: keep-alive\r\n\r\n'
        )
        self.transport.close()
        
def make_tls_context():
    ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
    ctx.load_cert_chain('cert.crt', 'cert.key')
    return ctx

tls_context = make_tls_context()
loop = uvloop.new_event_loop()

async def start_server(loop):
    return await loop.create_server(
        HTTP, '127.0.0.1', 443, backlog=65535,
        ssl=tls_context)

loop.run_until_complete(start_server(loop))
loop.run_forever()

CPython versions tested on:

3.9, 3.11, 3.12

Operating systems tested on:

Debian Linux

Linked PRs

Can confirm that the issue exists - Python 3.11, and uvloop 0.17.0, when trying to set up many client SSL connections that sometimes reconnects - even directly with asyncio.create_connection, and it's cleaned up not properly in some place.

Sometimes it leaked several megabytes per second for me.

I was able to track this by attaching to a running python process via gdb, and when it leaked, I put a catchpoint for brk syscalls, and examined 40 sequential breakpoint events, when all were related to /usr/lib/python3.10/ssl.py.

(gdb) py-bt
Traceback (most recent call first):
  File "/usr/lib/python3.10/ssl.py", line 917, in read
    v = self._sslobj.read(len)
  <built-in method buffer_updated of uvloop.loop.SSLProtocol object at remote 0xffff758f90c0>
  <built-in method run of _contextvars.Context object at remote 0xffff758c7340>
  File "/home/ubuntu/py-core/env/lib/python3.10/site-packages/aiohttp/web.py", line 544, in run_app
    loop.run_until_complete(main_task)
  File "/home/ubuntu/py-core/server_app.py", line 39, in <module>
    web.run_app(app_factory(), loop=LOOP, host=args.host, port=args.port)
(gdb)

Also, a gc.get_objects() count was not actually growing, that's why I suppose that this is related a lower-level native implementation.

@1st1

@1st1, @asvetlov, @graingert, @gvanrossum, @kumaraditya303 (as asyncio module experts)

Unfortunately I am really bad at this type of low level stuff. :-( If you suspect a leak in the C accelerator, you can disable it and see if the leak goes away. (Or at least becomes less severe.)

OTOH if the problem is in the SSL code, we need a different kind of expert.

Finally please figure out if this is in uvloop or not. If it’s tied to uvloop this is the wrong tracker.

Unfortunately, this bug was reproduced only in our production, and we weren't able to cause it in other environments.

It's already mitigated, as the root cause was in poor stability of connections.

I don't think I would be able to reproduce it again in production without uvloop, because it caused a lot of harm for our system.

But ready to assist and provide more details, if anyone wants to ask me some clarifying questions about it.

Also, there are several related tickets, and I'm not sure if they are all about uvloop:

asyncio: potential leak of TLS connections #106684 - very similar cpython issue, already closed though.
Memory leak with Python 3.11.2 aio-libs/aiohttp#7252 - also, an already closed, but recent aiohttp issue, they were able to mitigate it somehow, but not sure if it was the same bug.

I actually was able to reproduce this leak isolated several days ago, but it leaked for only ~5 megabytes in several hours. I adopted a repro script from this comment to the aiohttp bug mentioned above to be ready for run, and ran it for about 4 hours.

The initial memory consumption (RSS) had initially stabilised on 258120 bytes, but after several hours increased to ~264000. Hope it's the same bug.

#!/usr/bin/env python3
import aiohttp
import tracemalloc
import ssl
import asyncio

def init():
    timeout = aiohttp.ClientTimeout(connect=5, sock_read=5)
    ssl_ctx = ssl._create_unverified_context()
    conn = aiohttp.TCPConnector(ssl=ssl_ctx, enable_cleanup_closed=True)
    session = aiohttp.ClientSession(connector=conn, timeout=timeout, cookie_jar=aiohttp.CookieJar(unsafe=True))
    return session

async def fetch(client):
    try:
        async with client.request('GET', url=f'https://api.pro.coinbase.com/products/BTC-USD/ticker') as r:
            msg = await r.text()
            print(msg)
    except asyncio.CancelledError:
        raise
    except Exception as err:
        print("error", err)
        pass

async def main(): 
    requests = 600
    clients = [init() for _ in range(requests)]
    tracemalloc.start()
    try:
        while True:
            await asyncio.gather(*[fetch(client) for client in clients])
            await asyncio.sleep(5)
    except asyncio.CancelledError:
        pass  # end and clean things up
    finally:
        memory_used = tracemalloc.get_traced_memory()
        snapshot = tracemalloc.take_snapshot()
        stats = snapshot.statistics('lineno')
        for stat in stats[:10]:
            print(stat)
        try:
            await asyncio.gather(*[client.close() for client in clients.values()])
        except:
            pass

asyncio.run(main())

Could you find a reproducer without aiohttp?

@EBazarov

Unsure if this is helpful at all or related but I came across this issue Kludex/uvicorn#2078 in which in the thread it is discussed/concluded that the issue of memory not being released is not isolated to uvicorn but seen in granian/gunicorn/hypercorn and as a result could be interpreter level (apologies for butchering the summary). Thread has some great charts/analysis though it is on the server level, however the different implementations w.r.t granian and uvicorn can help approximate where the issue might be surfacing if it's related. Example repo: https://github.com/Besedo/memory_issue/tree/main by @EBazarov Apologies in advance if it is unrelated and will remove the comment/create one in the right place.

gi0baro:

Given everything we stated in the upper replies, I suspect there's no much to do about this. Given the number of frameworks and servers tested, it just seems the Python interpreter is the culprit here, as even if the heap memory gets released, the RSS remains high.

This should be highlighted as I (and many others) could've avoided a lot of headache had I not upgraded my applications to the latest Python (or used a different language altogether).

I can absolutely relate to this because we ended up rewriting the whole app in Rust. One of the main reasons was this bug (and async libs being quite buggy in general). It took me days to debug what's going on and finally landed in here.

In my case MALLOC_TRIM_THRESHOLD_=131072 seems to work. Those vertical lines are project updates. I'm using Python 3.11 and Django 5.0.
I've not seen memory consumption higher than 450 MB, but I got rid of media passing through Django server, just JSON requests and responses.

Are there any updates on this issue? Did anyone open up a bug report with the glibc maintainers?

This issue has made trying to run a long-running Python server nearly impossible.

@nikumar1206

@nikumar1206 I did attempt a few moments ago https://sourceware.org/pipermail/libc-help/2024-October/006791.html but not sure.

@stalkerg

This is helpful @stalkerg assuming the glibc maintainers actually get to reading the thread they will eventually come up with something, hopefully not a won't-fix

@fantix

@fantix

Hello,
we have the same problem. Did anyone find a solution or workaround in the meantime? Many thanks for your efforts.

Hello, we have the same problem. Did anyone find a solution or workaround in the meantime? Many thanks for your efforts.

Ditching glibc entirely seems to be a solution after all...

@geraldog

@geraldog Many thanks for your reply.
It looks like a great workaround but to be fully honest I am not familiar enough with the low level package dependencies to completely understand the consequences.
I assume most python environments (docker containers like f.e. python-3.11-slim etc) and many python packages depend on c support.
How exactly do you recommend to replace glibc then? is there a compatible package as drop-in replacement that you can recommend? Or do you use other base images to create the runtime environments?

Again many thanks for your help. I really appreciate it a lot.

How exactly do you recommend to replace glibc then? is there a compatible package as drop-in replacement that you can recommend? Or do you use other base images to create the runtime environments?

Try an Alpine Linux image. People in this thread have had success in working around this bug with it. It's built around musl libc instead of GNU libc.

@geraldog

Hi!
@geraldog could you please summarize what's happening there and if we should close this issue if the problem is not on CPython's end? Thank you in advance.

@Eclips4

Hi @Eclips4

This is definitely not a CPython issue and I believe the issue should therefore be closed.

The observed "memory leak" is actually a glibc quirk or optimization as discovered by @stalkerg where memory allocated is never reclaimed with a free(), keeping the previously allocated memory in a pool for later re-use.

Some users like me have reported success with setting the env var MALLOC_TRIM_THRESHOLD_ to a low value as a workaround, while other users proved the issue is not reproducible with musl libc based system like Alpine.

@geraldog

Thank you for the answer, @geraldog!
I'm closing this as not planned, since the problem is not at CPython's end.
Please ping me and I'll reopen this issue(or maybe it would be better to create a new one, because this one has had quite big discussion..) if you find something on CPython's side related to this issue.

Thank you all for spending time finding the root cause!

Navigation Menu

Repository navigation

TLS/SSL asyncio leaks memory #109534

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

83 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

TLS/SSL asyncio leaks memory #109534

Description

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

Activity

selevit commented on Nov 17, 2023

arhadthedev commented on Nov 18, 2023

gvanrossum commented on Nov 18, 2023

gvanrossum commented on Nov 18, 2023

gvanrossum commented on Nov 18, 2023

selevit commented on Nov 19, 2023

selevit commented on Nov 19, 2023

gvanrossum commented on Nov 19, 2023

Faolain commented on Nov 27, 2023

83 remaining items

gyula-lakatos commented on Jul 2, 2024

fifdee commented on Jul 3, 2024

nikumar1206 commented on Sep 29, 2024

stalkerg commented on Oct 7, 2024

geraldog commented on Oct 7, 2024

1st1 commented on Oct 8, 2024

CodeToFreedom commented on Feb 21, 2025

geraldog commented on Feb 22, 2025

CodeToFreedom commented on Feb 22, 2025

geraldog commented on Feb 22, 2025

Eclips4 commented on Mar 11, 2025

geraldog commented on Mar 11, 2025

Eclips4 commented on Mar 12, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions