Repository navigation
How much is my memory? #31641
Description
- Version: master
- Platform: linux / all
- Subsystem: process
This is opened for 3 reasons:
- help progress Investigate flaky parallel/test-worker-memory #23277
- gather intelligence on internal memory management where I may have lack of clarity
- help clearly define process's memory usage - for future support and documentation (specifically for cloud workloads where usage has bearings on billing)
$ cat foo.cc
#include <stdlib.h>
#include <unistd.h>
#include <stdio.h>
#include <malloc.h>
void *list[1024 * 1024];
int r() {
// add a one so as to never return 0
return 1 + random() % 137;
}
int main() {
// use calloc so that the pages are 'touched'.
for(int i=0;i< 1024 * 1024;i++)
list[i] = calloc(r(), 1);
for(int i=0;i< 1024 * 1024;i++)
free(list[i]);
sleep(60 * 1000);
}A linux (RHEL 8) observation:
The static memory usage (without the callocs) of this code is 12 MB
- 8 MB for the array (64-bit, so 8 * 1MB)
- 4 MB for a.out + libc + ...
top output after the program runs and sleeps for enough time:
$ top -p 6777
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6777 root 20 0 99772 96344 932 S 0.0 5.1 0:00.16 a.out
$ cat /proc/meminfo | grep MemFree
MemFree: 1637932 kB
$
top output after the system is heavily loaded:
top -p 6777
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6777 root 20 0 99772 0 0 S 0.0 0.0 0:00.16 a.out
$ cat /proc/meminfo | grep MemFree
MemFree: 70444 kB
pmap shows 85 MB is still mapped into the process, matching with top.
0000000002c7c000 87384K rw--- [ anon ]
Observations:
rssstays high (~ virtual memory) for ever, if there is no load in the system- virtual memory stays high even after all the allocated memory is freed.
- all numbers tally: 85 for the dynamic memory + 12 for the static part = 97, as shown in top
Problems / questions:
- what is a meaningful measure of program's memory usage? we can say these are the values we get from the OS, and leave it for the user to interpret those, but that does not help them enough.
rssis a function of not only on the program's activity with memory, but also to other parts of the system.virtual memoryon the other hand was apparently the sum of allocations (malloc/calloc/mmap...) in the program, but looks like it is not, and depends on the glibc's memory management implementations and its discretion as to when to retain and when to release chunks back to OS, and as to what size of chunks qualify for removal, and may be also depend on the memory usage hueristics as well! When I used 4K chunks for calloc/free, I got different results.
In Cloud workloads this has implications. A user would want to minimize memory tagged against their process and would do everything in their application to achieve that, but due the dependency on the system load and allocation caching behavior, this would not translate well?
-
memory leak definition becomes loose and detection becomes less comprehensive. Plurality of tools profiling the app from different angles would bring-in mutually contradicting results? For example: in a batch process, a code that genuinely leaks an MB versus an MB that is cached in malloc arenas - what is the difference from outside, and why should I worry more about the former and not worry about the later?
-
I read about wss (working set size) - that is a true measure of how much memory the program is actively engaged in. The idea looks like tracking paging-in; as that would be a true reflection of the program's needs. I don't find an API for this, nor a platform-neutral way to obtain this. Even if we could, this has an issue - the value is transient, as it depends on the history of usage, not the current state?
Activity
gireeshpunathil commentedon Feb 19, 2020
I tested in mac and windows, and saw the same behavior - with minor differences w.r.t numbers (probably owing to the system configurations and implementation differences in memory management at glibc / kernel), but the pattern is same.
I found references that confirm my theory:
http://www.brendangregg.com/wss.html
https://www.networkworld.com/article/2298360/kernel-space--how-much-memory-am-i-really-using-.html
relevant excerpt:
gireeshpunathil commentedon Feb 19, 2020
with that, I would like to convert this into a (libuv?) feature request:
use case: ability for applications to figure out its memory usage in a reasonable manner - one of:
with a preference on the former.
/cc @nodejs/libuv
affected module: @nodejs/workers
bnoordhuis commentedon Feb 19, 2020
I'm not sure what you're asking for here. What's a "malloc cache"?
How would you measure that? The approach in Gregg's article is... creative.... but not something that can be generalized across platforms, possibly not even architectures.
gireeshpunathil commentedon Feb 19, 2020
@bnoordhuis -
malloc cacheis a term I coined, for lack of a better qualifier. If you look at my calculation in the first comment, 94 MB is seen as allocated to the process, after themalloc/freebusiness is long over - out of which only 12 MB belongs to the program, the rest has to be released and un-accounted from virtual memory, but it hasn't - so my guess is that it is cached by malloc subsystem, for future requests.I agree that it is complicated, and might need OS support to achieve that; and hence I realistically opted for the first one.
bnoordhuis commentedon Feb 19, 2020
Oh, you mean arena allocations. Libuv has no insight into libc's bookkeeping though.
You could write an add-on that calls glibc's
mtrace()but at the risk of stating the obvious: that's specific to glibc.gireeshpunathil commentedon Feb 19, 2020
@bnoordhuis - ok, but Libuv has malloc wrappers that can potentially keep a track of active allocations?
https://github.com/libuv/libuv/blob/18ff13c71e5e68066729bc90664f9550bfe5ead4/src/uv-common.c#L75-L79
I am not saying the entire app's memory demand funnel through Libuv. My take is that given Node.js is a language runtime that abstracts many implementation details from the underlying system, one or more or all of
libuv / v8 / nodeshould be able to measure the real memory allocation either pro-actively (through malloc wrapping) or reactively (through relecting on process's memory, with help of underlying platform if need be).In the absence of that, the closest what we get is
rssandvirtual mem. As illustrated in the linux example, those aretoo farfrom the real memory (96 instead of 12) to be of any meaningful accounting purpose?bnoordhuis commentedon Feb 19, 2020
It does, but those only track what libuv itself allocates, which isn't much compared to Node or V8.
An LD_PRELOAD wrapper will let you track all malloc/realloc/posix_memalign/free calls, however.
gireeshpunathil commentedon Feb 20, 2020
@bnoordhuis - unfortunately, the LD_PRELOAD approach does not help on the immediate issue (#23277) or the wider need (accurate metering in Cloud deployments)
An in-built, callable API will be the best way, and for the long term.
mmarchini commentedon Feb 24, 2020
I'm not sure we can influence metering in cloud deployments. Even if we find a way to provide working set size instead of resident set size, cloud providers would probably still consider rss for metering since the memory allocated in rss can't be allocated for other processes. While I agree it would be awesome to have a better way to visualize actual memory usage, we should also be aware that introducing yet another metric can lead to more confusion.
There's no guarantee native modules will use it, which can also lead to further confusion (as wss will be imprecise). If a native module depends on a third-party library (not uncommon), the library will definitely not use this callable API.
Have you looked into how other languages and runtimes deal with this? It's hard to believe we're the first runtime trying to incorporate a more precise memory reporting solution.
gireeshpunathil commentedon Feb 25, 2020
@mmarchini: thanks for the views!
Sorry for the confusion. I meant here about an API to get the accurate memory usage, not an allocation wrapper. Agree that we can't enforce third party native code to invoke the wrappers, so we might need to override the allocators.
openj9 shows memory layout of the program to this level of granularity:
and I guess it wraps malloc to achieve that:
addaleax commentedon Feb 26, 2020
@gireeshpunathil Node.js could provide allocation wrappers to libuv, but V8 (which most likely performs more native-heap allocations than libuv in a typical Node.js scenario) and Node.js itself use
operator newa lot to allocate memory. I think that currently means that heap snapshots are the only way to track memory usage in a very detailed way.mhdawson commentedon Feb 27, 2020
@addaleax in the report above I think only this section
is memory that would be reported in a heap snapshot. The rest is off-heap memory. Having said that it might not change your point that a lot of the memory is being allocated by V8 versus Node.js. Information on the Node.js allocations might be interesting anyway but would only be a subset of the overall.
github-actions commentedon Mar 8, 2022
There has been no activity on this feature request for 5 months and it is unlikely to be implemented. It will be closed 6 months after the last non-automated comment.
For more information on how the project manages feature requests, please consult the feature request management document.
gireeshpunathil commentedon Mar 8, 2022
thinking further on this, while this is reported by many users as seen from the linked issues, I don't see a practical way to implement this cross-platform. Let us may be keep it open for the next one month to see if any one has got ideas, or else let it close through the default bot action!
benjamingr commentedon Mar 8, 2022
( @gireeshpunathil note your comment opted-out of the bot behavior :) (I agree with what you wrote btw))
gireeshpunathil commentedon Mar 8, 2022
that is fine, I will keep an eye on this for a while to see any ideas on implementation, or else manual close
github-actions commentedon Sep 15, 2022
There has been no activity on this feature request for 5 months and it is unlikely to be implemented. It will be closed 6 months after the last non-automated comment.
For more information on how the project manages feature requests, please consult the feature request management document.
github-actions commentedon Oct 16, 2022
There has been no activity on this feature request and it is being closed. If you feel closing this issue is not the right thing to do, please leave a comment.
For more information on how the project manages feature requests, please consult the feature request management document.