DamianBlog

Updated and republished by Dara Hayes on May 24 2017

 

Tracking down memory leaks with Node.js has always been a challenge. The following discusses how to track memory leaks in a Node application using Node’s --inspect flag and the two awesome node modules – memwatch and heapdump.

 

memory leak

 

First, a trivial sample leak:

const http = require('http');

var server = http.createServer((req, res) => {
 for (var i=0; i<1000; i++) {
   server.on('request', function leakyfunc() {});
 }

 res.end('Hello World\n');
}).listen(1337, '127.0.0.1');
server.setMaxListeners(0);

console.log('Server running at http://127.0.0.1:1337/. Process PID: ', process.pid);

Every request adds another 1,000 leaky listeners. We are going to simulate requests using autocannon – A HTTP benchmarking tool written in node. It can be installed with npm i -g autocannon. If we run autocannon -c 1 -d 60 http://localhost:1337 in one shell, and in another shell view our process: top -pid <process pid> we will see very high and erratic memory usage for this node process. Our node process has gone wild. So how do you diagnose from here?

Memory leak detection

The memwatch-next module is great for detecting memory leaks. Let’s install into our app as follows: npm install --save memwatch-next. Then, in our code, add:

const memwatch = require('memwatch-next');

and also a listener for the ‘leak’ event:

memwatch.on('leak', (info) => {
  console.error('Memory leak detected:\n', info);
});

Now when we run our test again, we see the following emitted:

Memory leak detected:
{
  growth: 14280616,
  reason: 'heap growth over 5 consecutive GCs (3s) - -2147483648 bytes/hr'
}

Memwatch has detected a memory leak! Memwatch defines a leak event as:

A leak event will be emitted when your heap usage has increased for five consecutive garbage collections.’

See memwatch for more details.

Memory leak analysis

So we’ve detected that we have a memory leak – great! But what now? The next step is to find where your leak is. The example leak is pretty obvious, but the process for finding one is always the same:

  1. Create heap dumps at different time intervals.
  2. Compare the dumps to see what’s growing.

Let’s see two ways of doing this.

Node’s –inspect Flag

Node’s --inspect flag landed in node version 6. This feature lets you debug and inspect your node process from within Chrome’s DevTools. Simply start the application passing the --inspect flag:

$ node --inspect index.js
Debugger listening on port 9229.
Warning: This is an experimental feature and could change at any time.
To start debugging, open the following URL in Chrome:
    chrome-devtools://devtools/bundled/inspector.html?experiments=true&v8only=true&ws=127.0.0.1:9229/9fe96182-d16e-4028-9e17-3c1d20d8398a

Follow the URL to the inspector and navigate to the memory tab.

DevTools Memory Tab

DevTools Memory Tab

Here’s what we will do:

  • Hit the application with autocannon -c 1 -d 60 http://localhost:1337
  • Take a heap snapshot after roughly 10 seconds, and again after 30 seconds.

 

Heap Snapshot Comparison

Heap Snapshot Comparison

The comparison view shows us what has happened between snapshots. We can see a huge number of closure objects have been created. The delta columns also give us an idea of how much growth there has been. Let’s take a deeper look:

Found our Memory Leak

Found our Memory Leak

Now it is clear that our leakyfunc() is the root of the problem. The Retainers view shows where the reference to our leakyfunc() object is held in memory. We can also find exactly where it was created in our code.

Memory Leak Details

Memory Leak Details

For a great overview of memory profiling in DevTools, see Taming The Unicorn: Easing JavaScript Memory Profiling In Chrome DevTools.

Heapdump

node-heapdump is a great module. If the --inspect option is not available, or you need to create snapshots programmatically, heapdump can be used to create a snapshot from within the application code. For more on heapdump, see the following StrongLoop blog post. Let’s use heapdump in our code now, so that every time a memory leak is detected, we write out a snapshot of the V8 stack to disk:

memwatch.on('leak', (info) => {
  console.error('Memory leak detected:\n', info);
  heapdump.writeSnapshot((err, filename) => {
    if (err) console.error(err);
    else console.error('Wrote snapshot: ' + filename);
})

Run our test again, and you should see some ‘.heapsnapshot’ files created in our application folder. Now in Chrome, launch DevTools (alt-cmd-i on Mac), hit the Memory tab and hit Load to load in our snapshots. Now we can find the leak as we did in the previous section.

Concluding observations

The above outlines one method of ‘self detecting’ memory leaks in Node and these are my concluding observations:

  • heapdump has some caveats about snapshot size, etc.
  • Read the fine print carefully. It’s also CPU intensive to take a snapshot.
  • There are other ways of generating snapshots with heapdump. Pick whichever way suits your application best
  • e.g. sending sigusr2 to the process, etc. This is also possible with memwatch-sigusr2.
  • Consider where and when you enable memwatch/heapdump.
  • Only enable in testing and not in production. Also consider how many heapdumps to write out before possibly bailing out of the process. Again, do whatever best suits your application.
  • Consider the other ways of detecting memory growth from within your application
  • i.e. monitoring the size of process.memoryUsage().rss as an alternative to memwatch.
  • When a memory leak is detected, you should also make sure it gets picked up by your monitoring and alerting tools so humans are told about it.
  • Beware of false positives: short memory spikes can appear as leaks to memwatch.
  • If your app suddenly gets busy and uses a big chunk of memory which it takes a while to process, this processing time can span multiple garbage collections and memwatch will report a leak. However, there may be nothing wrong here per se. When your app is done processing, the memory usage should settle down again. What you’re really looking for is persistent reporting of memory leaks. One off ‘blips’ may actually be ok.

The gist for source code referred to in this article can be found here. Having trouble with detecting memory leaks in node.js or want to know more on self-detection of memory leaks? You can tweet me here and connect with me on LinkedIn. If you’d like to know more about how nearForm can help you, contact us here or on any of the communication methods below.

 

By: Damian Beresford

Damian is a software architect here at nearForm, with an interest in all things Node.js, Docker, microservices, mobile and enterprise. Damian has over 20 years’ experience with all stages of software development, across many technologies and platforms. He is passionate about software development and how next generation mobile and cloud are transforming the enterprise.
  • Shubhra Kar

    Great blog post. As a follow up, folks should be looking at the Garbage collection within V8 for understanding how heap dump results map to the managed memory segments in Node. Here is a link : https://strongloop.com/strongblog/node-js-performance-garbage-collection/

  • simon

    Looks like memwatch is broken, last commit was from 2013 and doesn’t work on node 0.12

    • benxamin

      Marco Minetti has been patching it. https://github.com/marcominetti/node-memwatch

      His fork runs on 0.12 (and Heroku).

      • Kelly Miyashiro

        Your link to the 0.12 branch is just a link to the original npm module.

        `memwatch.setup()` is not a method.

    • https://metalabs.io/ Vlad Miller

      Also try memwatch-next, seems they’re the ones who has the latest commit to memwatch fork at this time :D

      https://github.com/appcelerator/node-memwatch

      • simon

        Ok thanks Vlad I will check it out

  • Chaff

    I’d be interested in your unit test integration of this approach and how we could repurpose it to detecting memory leaks during continuous integration of client-side apps running in V8. Have you/can you publish your setup and a unit test example on GitHub?

  • Toby

    The IBM SDK for Node.js http://www.ibm.com/developerworks/web/nodesdk/ which is community node with support for IBM’s free tooling, ships with the Health Center monitoring agent npm pre installed. This gives you live GC monitoring, CPU usage and method profiling via the Health Center client http://www.ibm.com/developerworks/java/jdk/tools/healthcenter/ and available from the eclipse marketplace http://marketplace.eclipse.org/. We are looking at making this available outside of the IBM SDK for Node.js as this is extremely useful for detecting memory leaks and performance problems.

  • Kyle Davis

    In the article you have a line, ‘memwatch.setup();’ – it throws an error for me and it is missing in the gist. Is that a leftover from a previous version of memwatch?

  • Joao Paulo Aires Borras

    Hi, I am trying to use the code as is to learn about how to tackle memory leak problems in my own application but when I run the curl command to cause memory leak, the memwatch module gives the following output:

    *********************************

    Uncaught Error: attempt to end() a HeapDiff that was already ended

    { start: Sun Aug 09 2015 22:22:47 GMT+0900 (JST),

    end: Sun Aug 09 2015 22:22:48 GMT+0900 (JST),

    growth: 1805272,

    reason: ‘heap growth over 5 consecutive GCs (1s) – -2147483648 bytes/hr’ }

    *********************************************************

    And I never get the detailed information that is in this article.

    Is there anything that I have to do besides what is in the article?

    Thank you,
    Joao Paulo

  • Cory Robinson

    **** PLEASE UPDATE THIS BLOG/POST TO REFERENCE https://github.com/marcominetti/node-memwatch **** The current version `memwatch` is broken on current versions on Node.js and no longer maintained — install `memwatch-next`

    ****************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************