Figure Out Where Your Memory Went With dotnet-dump

Another problem we may uncover with performance testing is a memory leak. Those nasty problems hide well inside our application, and they may only show up when our application runs for weeks. How can we force them out in the open without waiting for a month?

Install dotnet-dump

If you not already have dotnet-dump installed, you should do that now with this command:

dotnet tool install --global dotnet-dump

For a refresh on how to use dotnet-dump I suggest you read my post on the .Net Diagnostic Tools.

Prepare the application

To find the memory leaks we can run our application in the debug mode. If we are lucky, this will add enough context to the dump file so that we can pinpoint the source of the memory leak much faster.

dotnet run -c Debug --launch-profile https

First create the load

Even when memory leaks only show up after the application runs for a long time, the runtime is not the deciding factor. The load we have on the application is usually much more important – then if no one uses the application, we hardly notice any memory leaks. Therefore, the more load we can put on the application, the faster we should see the memory leak.

Luckily for us, we already have the tooling to produce load. We can use Bombardier or NBomber and hit the part of the application we expect to have the memory leak. We do not need to find the highest throughput, a steady high load that the application can handle works best. For Bombardier we can reduce the concurrent clients and expand the duration of our load test:

bombardier -c 100 -d 5m https://localhost:7285/Speed/LeakLib

Then dump the memory

Contrary to our experience with dotnet-trace, we must wait before we can run dotnet-dump. We want to create a snapshot of the memory as it is when we run the command. But first our application needs to allocate enough memory so that the memory leak starts to take effect. We wait a few minutes and then run this command:

dotnet-dump collect -n WebApp

This produces a *.dmp file containing all the memory currently used by the application. If the application uses 5GB of RAM, the resulting dump file will also be 5GB.

Analyse the dump file

Now starts the hard part: we need to find the needle in the haystack.

There are different tools we can use to work with the dump file:

WinDdb
The analyze option of dotnet-dump (which uses WinDbg)
JetBrains dotMemory

Of these tools, JetBrains dotMemory offers the most comfort. We cover this tool in its own post in two weeks.

If you want to dive into the two other options, I suggest you watch Diagnose Memory Leaks in .NET Applications with WinDbg to gain a basic understanding of WinDbg. Adam Driscoll does an excellent job in explaining how to navigate WinDbg to figure out where your memory went.