Repository navigation
Large virtual memory (13-20 GB) when running Peertube without --jitless #45557
Description
Activity
@pbeyssac Could you provide the output of
cat /proc/74144/statusplease, replacing74144with the actual PID of thenodeprocess when its memory usage appears to be high?Thanks for your reply.
Here goes:
% cat /proc/94670/status node 94670 67081 94670 67081 pts/5 ctty 1669029587,527265 4,154345 0,557096 kqread 1001 1001 1001,1001 ioc-tubeNote that it is FreeBSD's /proc, which is different from the Linux output, so here's the relevant bit in the man page:
status The process status. This file is read-only and returns a single line containing multiple space-separated fields as follows: • command name • process id • parent process id • process group id • session id • device name of the controlling terminal, or a minus sign (“-”) if there is no controlling terminal. • a list of process flags: ctty if there is a controlling terminal, sldr if the process is a session leader, noflags if neither of the other two flags are set. • the process start time in seconds and microseconds, comma separated. • the user time in seconds and microseconds, comma separated. • the system time in seconds and microseconds, comma separated. • the wait channel message • the process credentials consisting of the effective user id and the list of groups (whose first member is the effective group id) all comma separated. • the hostname of the jail in which the process runs, or ‘-’ to indicate that the process is not running within a jail.Please find attached the memory map of the process as returned by /proc/${pid}/map.
map.txtAnother remark (I'm not sure this is relevant, but just in case): the process I started with --jitless died on an error which looks like a Peertube error, but which never happens when running with JIT. Perhaps a race condition due to the code running at a different speed.
[...:443] 2022-11-21 12:06:17.761 error: uncaughtException: Cannot read properties of undefined (reading 'timeoutType') TypeError: Cannot read properties of undefined (reading 'timeoutType') at _resume (/var/www/peertube/versions/peertube-v4.3.1/node_modules/undici/lib/client.js:1167:29) at resume (/var/www/peertube/versions/peertube-v4.3.1/node_modules/undici/lib/client.js:1131:3) at connect (/var/www/peertube/versions/peertube-v4.3.1/node_modules/undici/lib/client.js:1116:3) { "error": { "stack": "TypeError: Cannot read properties of undefined (reading 'timeoutType')\n at _resume (/var/www/peertube/versions/peertube-v4.3.1/node_modules/undici/lib/client.js:1167:29)\n at resume (/var/www/peertube/versions/peertube-v4.3.1/node_modules/undici/lib/client.js:1131:3)\n at connect (/var/www/peertube/versions/peertube-v4.3.1/node_modules/undici/lib/client.js:1116:3)", "message": "Cannot read properties of undefined (reading 'timeoutType')" }, (...)% cat /proc/94670/status node 94670 67081 94670 67081 pts/5 ctty 1669029587,527265 4,154345 0,557096 kqread 1001 1001 1001,1001 ioc-tubeNote that it is FreeBSD's /proc
This looks like
/stat, not/status./statdoes not contain the accumulated memory info.This looks like
/stat, not/status./statdoes not contain the accumulated memory info.I confirm it is really /proc/${pid}/status on FreeBSD (see the "cat", directly copied from my terminal). FreeBSD /proc differs from Linux and has no /proc/*/stat anyway.
Here /rlimit just in case (but these are static limits, not usage):
% cat /proc/94670/rlimit cpu -1 -1 fsize -1 -1 data 34359738368 34359738368 stack 536870912 536870912 core -1 -1 rss -1 -1 memlock 65536 65536 nproc 63680 63680 nofile 1882368 1882368 sbsize -1 -1 vmem -1 -1 npts -1 -1 swap -1 -1 kqueues -1 -1 umtx -1 -1man page excerpt:
rlimit This is a read-only file containing the process current and maximum limits. Each line is of the format rlimit current max, with -1 indicating infinity.Excerpt from ps:
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND peertube 94670 0,0 0,3 13254356 203024 5 S+J 12:19 0:12,45 node: peertube (node)Sorry, I was looking at the man page for CentOS, not FreeBSD.
The RSS looks reasonable. Maybe the JIT compiler creates a bunch of large memory mappings thus blowing up the reserved address space in virtual memory, but that's usually not a problem. (For example, V8 might decide to reserve a 4GiB address range for every single WebAssembly instance for performance reasons.)
Maybe the JIT compiler creates a bunch of large memory mappings
That's what happens. I'm going to guess peertube loads wasm files (undici does, of that I'm sure) and V8 makes large reservations for wasm modules, basically XL guard pages. That's virtual memory though, not RSS, so it's harmless.
- changed the title
[-]Extremely high node18/node14 memory use (13-20 GB) when running Peertube, fixed by running with --jitless[/-][+]Large virtual memory (13-20 GB) when running Peertube without --jitless[/+]on Nov 22, 2022
Version
v18.7.0 (also tested on node14)
Platform
FreeBSD tube 12.3-RELEASE FreeBSD 12.3-RELEASE GENERIC amd64 (also tested on Linux)
Subsystem
No idea (but no idea how to check for this)
What steps will reproduce the bug?
How often does it reproduce? Is there a required condition?
Same behaviour at every run. Seems easy to reproduce.
What is the expected behavior?
13-20 GB seems to be a lot more than necessary.
Starting with --jitless seems to shrink the memory size significantly to ~1 GB, which looks more reasonable:
Perhaps there's a specific Javascript module with specific code which triggers excessive memory use in the JIT compiler, but I have no idea how to check for this.
What do you see instead?
nodejs grows up to 13 GB - 20 GB memory (depending on node version/OS -- same observations on node14/FreeBSD, Linux, and on other Peertube instances)
Additional information
Posted here on the Peertube forum, where I was advised to come here and ask for help.
https://framacolibri.org/t/high-memory-use-for-node-process-v4-3-0-v4-3-1/16155/7