ungzipping buffer with trailing garbage throws error

One site sends a gzipped html page and I try to ungzip page for further processing but zlib throws error incorrect header check. Error happens due to extra data after gzipped page. Yes, gzipped data is not valid, since have trailing data, but in other hand, browser eat that normally.

Minimal example:

var zlib = require('zlib')

// from test-zlib-from-gzip-with-trailing-garbage.js
bufOk = Buffer.concat([
    zlib.gzipSync("abc"),
    Buffer(10).fill(0)
]);

bufWrong = Buffer.concat([
    zlib.gzipSync("abc"),
    Buffer("\n \n\n\n\n \n")
]);

// wil not raise Error
console.log("bufOk:", zlib.gunzipSync(bufOk).toString());

// will raise Error('incorrect header check')
console.log("bufWrong:", zlib.gunzipSync(bufWrong).toString());

Also live example:

var req = require('request')
req.get('http://irsural.ru', {gzip: true}, (err, res, body) => {
    if (err != null) {
        console.log('Error:', err);
        return 
    }
    console.log(body);
});

In other languages (for example, python - requests) I can set windowBits to 16 + zlib.Z_MAX_WINDOWBITS but node strictly restricted windowBits from 0 to 15 even in addon.

Version: v6.2.2
Platform: linux, x86_64
Subsystem: zlib

@addaleax

See PR #5883, node.js used to silently ignore trailing garbage but it no longer does. Zero padding is still allowed but everything else will raise an error.

cc @addaleax

In other languages (for example, python - requests) I can set windowBits to 16 + zlib.Z_MAX_WINDOWBITS but node strictly restricted windowBits from 0 to 15 even in addon.

Python’s gzip module seems to behave exactly like nodes does, btw.

Yes, gzipped data is not valid, since have trailing data, but in other hand, browser eat that normally.

There is, unfortunately, no common behaviour for different kinds of trailing data in browsers; often, trailing garbage will be ignored, but e.g. if it looks like the start of another gzip member, Chrome will just bail out completely (I don’t know why it does that, though).

@addaleax

Is there anything to do here @addaleax and @bnoordhuis ?

It's the bad feature, but not the bug.

@addaleax

Sorry for resurrecting this after 4 years @addaleax, but I wasn't able to find any solution elsewhere. Is there a way to ignore the trailing garbage with the currently available options? I tried various combinations of zlib.contstants, but with no luck.

Example website:

https://www.ebay.com/sch/sis.html?_nkw=Beechcraft

Browser works fine, gunzip works fine.

curl https://www.ebay.com/sch/sis.html\?_nkw\=Beechcraft -H "accept-encoding: gzip" | gunzip

But I can't convince Node.js to decompress this payload, aside from buffer.slice the end of it, but since the HTML size is dynamic, it doesn't really help, because I could never know in advance how much to slice.

Navigation Menu

Repository navigation

ungzipping buffer with trailing garbage throws error #7502

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

ungzipping buffer with trailing garbage throws error #7502

Description

Activity

bnoordhuis commented on Jun 30, 2016

addaleax commented on Jun 30, 2016

jasnell commented on Aug 9, 2016

horpto commented on Aug 9, 2016

mnmkng commented on Jun 6, 2020

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions