Skip to content

The Strange Memory Leak in .Net 8

Last year we moved our applications from .Net 6 to .Net 8. It was much less work that it was to move from .Net 4.8 to .Net 6. The migration itself was straightforward, and we could put our updated application into production without any issues. That was until a week later when we got this strange error:

Application '/LM/W3SVC/11/ROOT' with physical root '...'

hit unexpected managed exception, exception code = '0xe0434352'.

First 30KB characters of captured stdout and stderr logs:

Out of memory.

Dump the memory

To figure out what was going on, we created a dump file. First, we tried dotnet-dump, but that only gave us the access denied error. ProcDump could create a dump file, but that did not help us much either. The allocated memory was mostly empty. Even WinDbg showed us that 99.9% of the memory was free. Yet we could see that the memory only grew. Something was wrong.

Pinned objects

When we analysed the dump file in dotMemory, we saw that we hat many allocated blocks, but they were to 99.9% free:

The memory is to the largest extend free.

The only things that look a bit strange are the many orange bars that seem to be everywhere. But what are they?

As it turns out, each orange bar stands for a pinned object. Those are objects like file handles that cannot be moved by the garbage collector. And since they cannot be moved, they stay where they are and block the memory from being released.

That explains the memory leak, but why do we have so many pinned objects?

Is the problem application-specific?

The whole application was too large and too complex to find the memory leak by chance. Therefore, we decided to look at our other applications and see if we spot the same problem in a smaller one. We noticed that this problem was widespread and flew under the radar because we did not have enough users to blow up sooner. It was not only our web applications, but background services showed the same symptoms. The only common denominator was .Net 8. Was it a .Net 8 problem?

Create a minimalistic example

Google did not find any useful results, but we found many issues with .Net Core going back to 3.0 where developers reported similar memory leaks – but none of those issues resulted in any insight on what was the source of the problem. All we found were unreproducible problems.

We tried our luck with an empty ASP.NET application that only had the official template in it. But there we could create as much load as we wanted, the memory leak did not show up. Something was off, but as our search suggested, it was at a non-obvious place that required a lot of bad luck to show up.

Tear apart an application

Since the empty application did not have the problem, we started with one of our smallest applications where we could reproduce the problem. We removed one feature after another, but the memory leak persisted.

The only thing we learned was how to reproduce the fragmented memory within 30 seconds. That was a massive reduction for our feedback loop, but we were nowhere closer to find the source of our problem.

As we finally removed the dependency injection configuration, we no longer could reproduce the memory leak. Finally! The problem had to be with the DI container.

Hold the culprit

We remembered from our Google search that many reported a problem with files, so we dived through our dependency injection configuration to find anything that used a file. What we found at the end was this code block:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
var config = new ConfigurationBuilder()
                .SetBasePath(AppContext.BaseDirectory)
                .AddJsonFile("appsettings.json", optional: false, reloadOnChange: true)
                .AddEnvironmentVariables();

if (Environment.GetEnvironmentVariable("ASPNETCORE_ENVIRONMENT") != null
    && Environment.GetEnvironmentVariable("ASPNETCORE_ENVIRONMENT") == "Development")
{
    config.AddJsonFile("appsettings.Development.json", optional: true, reloadOnChange: true);
}

return config.Build();

This code block reads the appsettings.json file and merges it with the appsettings.Development.json file, should we run in the development environment. There was nothing obviously wrong about it. To start somewhere, we set the parameter reloadOnChange to false and rerun our performance test.

1
2
3
4
5
.AddJsonFile("appsettings.json", optional: false, reloadOnChange: false)

...

config.AddJsonFile("appsettings.Development.json", optional: true, reloadOnChange: false);
Lo and behold, the leak had disappeared.

Conclusion

The option reloadOnChange for the AddJsonFile() method creates a pinned object in the memory to keep a handle on the configuration file. That way it can detect a change and reload the configuration.

Unfortunately, that handle gets created whenever this part of the dependency injection configuration runs. While it looked in the code that this part should only be run once, it was run frequently. That gave us not only one handle to check if the configuration file has changed, but hundreds – and none of them could be moved out of the memory block they were created in.

In the end it was not a bug in .Net 8, but rather a side effect of a not so clearly documented behaviour in the dependency injection container.