You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
how to use memory_profiler to test async function #289
@ZoidbergPi , using @profile decorator with async def syntax works as well since my PR has been merged as this is essentially the same thing as using @asyncio.coroutine decorator. The test of asyncio support added in my PR is using asyncio.couroutine because to support python3.4, because async def-syntax has been introduced in python.3.5
@leiyugithub , even though my PR has been merged, there were no new releases of this library since. To use this functionality you either have to install the package from github, or ask the maintainer to create and publish a new release to PyPI.
I just tried profiling an async function with @memory_profiler.profile just as in @Hazhzeng 's example above and the output also looks like above. I'm getting something like:
Line # Mem usage Increment Occurrences Line Contents
=============================================================
16 301.0 MiB 301.0 MiB 1 @memory_profiler.profile
17 async def calculate_metrics(sizes):
18 301.0 MiB 0.0 MiB 1 metrics = dict(index_time_s=[], memory_bytes=[], query_time_s=[])
19 301.0 MiB 0.0 MiB 1 threshold = 0.8
20
.... more function body ...
Increment 0.0 everywhere.
Where I put .... more function body ... is the actual code which allocates memory heavily and which I would like to profile. Now I just have the 301MB on the line with the decorator and no increment below.
If I comment out the rest of the function body the number stays the same.
In the example by @Hazhzeng it seems to be just the same issue.
Hello @chr1st1ank. First of all, please, note that @Hazhzeng's issue was related to decorated functions not returning the correct value.
Also, please, note that the amount of allocated memory has been 301.0 MiB starting from line 16, before the function execution started. Lines 18 and 19 are displayed to add 0 memory only because their impact is minimal. Only large memory increments are tracked and displayed.
Here is an example that I used to test locally (Python 3.9.9, memory_profiler==0.60.0):
Line # Mem usage Increment Occurrences Line Contents
=============================================================
6 18.1 MiB 18.1 MiB 1 @profile
7 async def foo():
8 25.9 MiB 7.7 MiB 1 a = [1] * (10 ** 6)
9 178.2 MiB 152.4 MiB 1 b = [2] * (2 * 10 ** 7)
10 178.2 MiB 0.0 MiB 2 await asyncio.sleep(1)
11 25.9 MiB -152.3 MiB 1 del b
12 25.9 MiB 0.0 MiB 1 return a
Ok, thank you. If there is a filter on too small changes, that might be the issue. I moved the @profile one level up the call stack and now I can see some numbers. The inner functions did apparently not make enough of a difference. Still the numbers are too small now, but that might be an issue with dereferencing the complex objects which are allocated. I think I need to find something else to figure out how much memory these objects take. But thanks for the clarification.
Activity
ZoidbergPi commentedon Oct 17, 2020
see PR: #255 it seems added support @asyncio.coroutine in python3.4 by @d-ryzhikov, but it didn't work for async def mark in python3.5+
d-ryzhykau commentedon Oct 17, 2020
@ZoidbergPi , using
@profiledecorator withasync defsyntax works as well since my PR has been merged as this is essentially the same thing as using@asyncio.coroutinedecorator. The test of asyncio support added in my PR is usingasyncio.couroutinebecause to support python3.4, becauseasync def-syntax has been introduced in python.3.5@leiyugithub , even though my PR has been merged, there were no new releases of this library since. To use this functionality you either have to install the package from github, or ask the maintainer to create and publish a new release to PyPI.
fabianp commentedon Oct 19, 2020
Hazhzeng commentedon Mar 22, 2021
Hi @d-ryzhikov, I just tested it with async def syntax with the latest release 0.58.0 on Python 3.8.1, and the return value is still None.
Result as below:
Commenting the line
@memory_profiler.profileyields the proper return value2.Is there any way to mitigate this issue? Thanks.
d-ryzhykau commentedon Mar 22, 2021
@Hazhzeng , this is a bug. Thank you for catching it. I will prepare a PR fixing this a bit later today.
vertillo commentedon Sep 20, 2021
If you try to await an async function that awaits for other async functions it doesn't work
d-ryzhykau commentedon Sep 25, 2021
Hello @anphetamina. Could you please provide steps to replicate your problem?
chr1st1ank commentedon Feb 2, 2022
How is this supposed to work?
I just tried profiling an async function with
@memory_profiler.profilejust as in @Hazhzeng 's example above and the output also looks like above. I'm getting something like:Increment 0.0 everywhere.
Where I put
.... more function body ...is the actual code which allocates memory heavily and which I would like to profile. Now I just have the 301MB on the line with the decorator and no increment below.If I comment out the rest of the function body the number stays the same.
In the example by @Hazhzeng it seems to be just the same issue.
Edit:
Python 3.9.9
memory_profiler 0.60.0
d-ryzhykau commentedon Feb 2, 2022
Hello @chr1st1ank. First of all, please, note that @Hazhzeng's issue was related to decorated functions not returning the correct value.
Also, please, note that the amount of allocated memory has been 301.0 MiB starting from line 16, before the function execution started. Lines 18 and 19 are displayed to add 0 memory only because their impact is minimal. Only large memory increments are tracked and displayed.
Here is an example that I used to test locally (Python 3.9.9, memory_profiler==0.60.0):
chr1st1ank commentedon Feb 3, 2022
Ok, thank you. If there is a filter on too small changes, that might be the issue. I moved the
@profileone level up the call stack and now I can see some numbers. The inner functions did apparently not make enough of a difference. Still the numbers are too small now, but that might be an issue with dereferencing the complex objects which are allocated. I think I need to find something else to figure out how much memory these objects take. But thanks for the clarification.