Skip to content

Memory leak on Node 19/20 with includeLocalVariables enabled #14162

Closed as not planned

Description

@seeARMS

Is there an existing issue for this?

How do you use Sentry?

Sentry Saas (sentry.io)

Which SDK are you using?

@sentry/node

SDK Version

8.36.0

Framework Version

Node 19.9.0, also tested on Node 20.18.0

Link to Sentry event

N/A

Reproduction Example/SDK Setup

import * as Sentry from "@sentry/node"

import api from "@opentelemetry/api"



function trimData(data) {
  const size = JSON.stringify(data || "").length

  // Arbitrarily cut off massive payloads. Sentry silently drops events that are too large,
  // so trim them.
  // See https://github.com/getsentry/sentry-javascript/issues/3036#issuecomment-1066491190
  if (size > 200000) {
    return data?.slice(0, 200000)
  }

  return data
}

function checkIfSentryIgnore(url) {
  if (!url) return false

  for (const route of SENTRY_IGNORED_ROUTES) {
    if (typeof route === "string" && url.endsWith(route)) {
      return true
    } else if (route instanceof RegExp && route.test(url)) {
      return true
    }
  }
}

/**
 * Need to ignore long-running websockets.
 *
 * See: https://github.com/getsentry/sentry-javascript/issues/13412#issuecomment-2340434248
 */
const IGNORE_WEBSOCKETS = ["infura", "ws", "alchemy", "ankr", "zora"]

Sentry.init({
  dsn: "snip",

  // Disable entirely on non-production environments.
  enabled: true,
  debug: !!process.env.SENTRY_DEBUGGING,
  registerEsmLoaderHooks: { exclude: [/openai/] },
  includeLocalVariables: true,
  integrations: [
    Sentry.onUncaughtExceptionIntegration({
      exitEvenIfOtherHandlersAreRegistered: false,
    }),
    Sentry.expressIntegration(),
    Sentry.httpIntegration({
      ignoreOutgoingRequests(url, request) {
        console.log("SENTRY: should we ignore outgoing request?", url)
        for (const route of IGNORE_WEBSOCKETS) {
          if (url.includes(route)) {
            console.log("SENTRY: Ignoring outgoing request", url)
            return true
          }
        }

        console.log("SENTRY: Not ignoring outgoing request", url)
        return false
      },
    }),
  ],

  release: process.env.COMMIT,
  environment: process.env.FLY_APP_NAME || "development",

  beforeSendTransaction(event) {
    console.log("SENTRY: beforeSendTransaction", event)

    if (event.transaction === "redis-connect") {
      // Don't send the event to Sentry
      return null
    }

    console.log("SENTRY: Sending transaction", event)

    return event
  },
  beforeSend: (event) => {
    console.log("SENTRY: beforeSend", event)
    try {
      if (event.request?.data) {
        event.request.data = trimData(event.request.data)
      }
      if (event.message) {
        event.message = trimData(event.message)
      }

      return event
    } catch (e) {
      console.error("Error in Sentry beforeSend", e, event)
      return event
    }
  },
})

Steps to Reproduce

Run Node 19 or Node 20 with Sentry enabled in my specific setup. The memory gradually increases until an OOM occurs.

This graph shows when Node was updated from 18 to 19.9.0. No other code changes occurred.
Image

Expected Result

Memory does not gradually increase until an OOM occurs.

This graph shows when I disabled includeLocalVariables in the above Sentry.init: memory stops increasing.
Image

Actual Result

Memory increases gradually until it OOMs, if includeLocalVariables is enabled.

If Sentry is disabled entirely, or if the includeLocalVariables flag is disabled, then memory looks as expected.

I originally suspected this was the issue, since we also use websockets and long-running requests, but after ignoring the requests and confirming via the console.logs that none of the long-running requests got spans created (with the issue still persisting), I tried disabling local variables and that fixed it.

These are (truncated) logs with trace GC enabled. It's not particularly showing much other than memory growth. This shows with local variables enabled, and this is with it disabled.

Activity

  1. moved this to Waiting for: Product Owner in GitHub Issues with 👀 3on Nov 2, 2024
  2. self-assigned this
    on Nov 4, 2024
  3. timfish commented on Nov 4, 2024

    @timfish
    Collaborator

    Thanks for reporting this. I tested the recent changes around local variable capture for memory leaks since this has been an issue before. I did see an increase in memory usage with includeLocalVariables enabled but after running for a couple of hours I concluded that it wasn't leaking. I'll take another look!

  4. moved this from Waiting for: Product Owner to No status in GitHub Issues with 👀 3on Nov 4, 2024
  5. timfish commented on Nov 5, 2024

    @timfish
    Collaborator

    I've been unable to reproduce a memory leak with express and includeLocalVariables enabled.

    If you have a minimal reproduction that I could debug that would help.

  6. vitor-ramalho-deel commented on Nov 27, 2024

    @vitor-ramalho-deel

    Faced the same issue on my project using node 20.17.0, nestjs 10.4.3, sentry/nestjs 8.35.0

    Image

  7. moved this to Waiting for: Product Owner in GitHub Issues with 👀 3on Nov 27, 2024
  8. AbhiPrasad commented on Nov 28, 2024

    @AbhiPrasad
    Contributor

    @vitor-ramalho-deel are you able to provide a heap snapshot or reproduction to help us dig in further? It's hard to tell what might be causing this (might be another dependency playing bad with Sentry for example).

  9. moved this from Waiting for: Product Owner to No status in GitHub Issues with 👀 3on Nov 28, 2024
  10. moved this to Waiting for: Community in GitHub Issues with 👀 3on Dec 10, 2024
  11. getsantry commented on Jan 1, 2025

    @getsantry

    This issue has gone three weeks without activity. In another week, I will close it.

    But! If you comment or otherwise update it, I will reset the clock, and if you remove the label Waiting for: Community, I will leave it alone ... forever!


    "A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀

  12. tellisnz commented on Apr 12, 2025

    @tellisnz

    Can confirm this is in at least 9.1.0 node 22

    Image

    I got a heap dump and nothing looked out of the ordinary - it seems like it was a native memory issue?

    ChatGPT thoughts:
    Image

    Apologies but i am unable to provide a reproduction.

  13. AbhiPrasad commented on Apr 13, 2025

    @AbhiPrasad
    Contributor

    @tellisnz could you open a new GH issue and fill out the issue template? We'll take a look!

  14. tellisnz commented on Apr 14, 2025

    @tellisnz

    @AbhiPrasad - created #16041 but to be honest we're not going to chase it, we just simply disabled in prod and can work around it in staging/dev

    I can't create a good reproduction case with our setup, and company policy precludes providing dependency set, sorry.

  15. ghost added on Aug 12, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions