Skip to content

HTTPS Server crashing randomly (With solution) #13170

Closed
Closed
HTTPS Server crashing randomly (With solution)#13170
Labels
tlsIssues and PRs related to the tls subsystem.

Description

@YindSoft
YindSoft
opened on May 23, 2017

I've been experiencing some issues with a cluster https server.

It throws the following error:

net.js:729
  if (req.async && this._handle.writeQueueSize != 0)
                               ^

TypeError: Cannot read property 'writeQueueSize' of null
    at TLSSocket.Socket._writeGeneric (net.js:729:32)
    at TLSSocket.Socket._writev (net.js:737:8)
    at doWrite (_stream_writable.js:327:12)
    at clearBuffer (_stream_writable.js:416:5)
    at onwrite (_stream_writable.js:368:7)
    at WriteWrap.afterWrite [as oncomplete] (net.js:824:12)
Worker 33233 died -> 33245 born.

I went ahead and change that line for:
if (req.async && this._handle && this._handle.writeQueueSize != 0)
Recompile node and it seems to have solve the problem.
But I don't really know why it ocurrs and how to replicate it, I just see it almost every 10 minutes in my logs.

Activity

  1. cjihrig commented on May 23, 2017

    @cjihrig
    cjihrig
    on May 23, 2017
    Contributor

    Please include the information that was asked of you in the issue template:

    Please fill in as much of the template below as you're able.

Version: output of `node -v`
Platform: output of `uname -a` (UNIX), or version and 32 or 64-bit (Windows)
Subsystem: if known, please specify affected core module name

If possible, please provide code that demonstrates the problem, keeping it as
simple and free of external dependencies as you are able.

  2. YindSoft commented on May 23, 2017

    @YindSoft
    YindSoft
    on May 23, 2017
    Author

    Version: node 7.10.0
    Platform: Linux ip-172-30-0-191 3.13.0-48-generic #80-Ubuntu SMP Thu Mar 12 11:16:15 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
    Subsystem: net.js module.

    try {
    var options = {
        SNICallback: function (domain, cb) {
            if (!secureContext[domain]) {
                domain = 'default';
            }
            cb(null, tls.createSecureContext(secureContext[domain]));
        },
        key: secureContext['default'].key,
        cert: secureContext['default'].cert,
    };
    global.serverHTTPS = https.createServer(options, app);
} catch (err){
    console.error(err.message);
    console.error(err.stack);
}


global.serverHTTPS.on('connection', onConnection);
global.serverHTTPS.on('error', onError);


global.serverHTTPS.listen(global.config.httpsPort);

    This is the code I use to open the HTTPS server and it works.. but after a while it give the error above.

  3. YindSoft commented on May 23, 2017

    @YindSoft
    YindSoft
    on May 23, 2017
    Author

    I'm currently running the server with the fix I put above and it hasn't have any issue so far, the problem seems to have stopped. But I had to recompile node.. so it's just a temporal solution or if you can find the root of the problem.

  4. mscdex
    added
    tlsIssues and PRs related to the tls subsystem.
    on May 23, 2017

  5. mscdex commented on May 23, 2017

    @mscdex
    mscdex
    on May 23, 2017
    Contributor

    /cc @nodejs/crypto

  6. cjihrig commented on May 23, 2017

    @cjihrig
    cjihrig
    on May 23, 2017
    Contributor

    @YindSoft are you able to trace through the application, perhaps with the inspector, and find out where this._handle becomes falsey (or provide a full code sample that can be executed)? There is a check a few lines up in that same function that should prevent what you're seeing, but is not apparently.

  7. YindSoft commented on May 23, 2017

    @YindSoft
    YindSoft
    on May 23, 2017
    Author

    @cjihrig I can't trace it as it occurs in my production server, It's really strange indeed as you mention it has a check above, but somehow somewhere in the middle this._handle turns to NULL and throws up that error.
    The issue is that I couldn't replicate the issue in my local server or test servers, I think it occurs when there is quite a bit of traffic.

  8. santigimeno commented on May 23, 2017

    @santigimeno
    santigimeno
    on May 23, 2017
    Member

    Maybe it's not related but being the cluster module involved and _handle set to NULL sounds a lot like: #3072

  9. YindSoft commented on May 24, 2017

    @YindSoft
    YindSoft
    on May 24, 2017
    Author

    After a while, it doesn't throw the errors, but found out that the workers keep on dying without any error message: (After several hours running)
    Worker 52933 died -> 62684 born.
    Worker 52738 died -> 62690 born.
    Worker 52956 died -> 64368 born.
    Worker 52962 died -> 64369 born.
    Worker 52683 died -> 64380 born.

    Any thoughts?

  10. cjihrig commented on May 24, 2017

    @cjihrig
    cjihrig
    on May 24, 2017
    Contributor

    Without a more complete reproduction, it's hard to say.

  11. Trott commented on Aug 13, 2017

    @Trott
    Trott
    on Aug 13, 2017
    Member

    @YindSoft Do you know if you still experience this with Node.js 8.3.0?

    Should this remain open?

  12. bnoordhuis commented on Aug 14, 2017

    @bnoordhuis
    bnoordhuis
    on Aug 14, 2017
    Member

    I'm reasonably sure this has been fixed by #14588. Not yet released but will be soon.

  13. bnoordhuis
    closed this as completedon Aug 14, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    tlsIssues and PRs related to the tls subsystem.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions