Repository navigation
zlib: memory leak with gunzipSync #1479
Description
Activity
- changed the title
[-]memory leak with zlib.gunzipSync[/-][+]zlib: memory leak with gunzipSync[/+]on Apr 21, 2015 It also happens on Windows :
node test.js > NUL # # Fatal error in ..\..\src\heap\mark-compact.cc, line 2137 # CHECK(success) failed #
Can you get some deeper info on it? Maybe run it through valgrind?
Problem in your code. GC isn't called, because you use only blocking methods.
Run "node --expose_gc test.js"
'use strict'; var zlib = require('zlib'); var data = 'abcdefghijklmnopqrstuvwxyz'; var gzipped = zlib.gzipSync(data); var step = 0; while (true) { step++; var contents = zlib.gunzipSync(gzipped); process.stdout.write(contents.toString() + '\n'); if (step % 1000) { gc(); } }
it's true. Using
while (true)without exit is very bad, and makes it easier to cause your own errors.Yeah of course,
while(true)was for the example. It is not what I was using in my real script.I had something like this:
glob('**/*.gz', { cwd: config.data }, function (err, files) { files.forEach(treatFile); // about 100k files to unzip and treat });
Now I am using the async version without any issue. The code was just simpler using gunzipSync...
Thanks for your answers :)
https://github.com/iojs/io.js/blob/v2.0.1/lib/zlib.js#L458 — this is the line that causes it.
It delays emitting thecloseevent until the next tick, lockingthis. That's why that objects are not collected, because in sync code everything is running in a single tick.Looks like even the
*Syncmethods were created with async in mind.@dchusovitin You are incorrect. Both incremental and full GC runs are automatically called even in synchronous code. Moreover, your code sample doesn't even change anything (with and without fixing the mistype) except for slowing things down.
Looks like even the *Sync methods were created with async in mind.
I think it's more that everything
*Syncwas created after their async counterparts. If something async is happening in a sync call then it's a bug.But the reason to defer the event is to prevent infinite recursion. I'm not sure if this is fixable without breaking something else.
Same bug with
zlib.gzipSyncused in a loop - process out of memory.
I was running a script to read sequentially a lot of gzipped files, and print on stdout the result of a computation for each file.
After about 16'000 files, the process just stopped and
Killedwas printed on the terminal.I guess that the memory used by the process increases until the kernel decides to kill it.
I could reduce the code to this testcase:
This is what I see with top command just before the process disappears:
> top 15363 mzasso 20 0 10,240g 3,716g 11776 R 111,0 24,2 0:53.59 nodeNB: a similar code, that would just write
datain the while loop keeps a stable memory consumption