Investigating memory issues. OutOfMemoryException.

General information about OOM.

Let's first grasp, how do we usually get that scary OOM exception.

In case of 32bit process (doesn't matter, if it runs on 32bit or 64bit system) our process has only 2Gb of available virtual address space. Actually, it has 4, but 2 of them are used by system needs. Maximum you may extend it to 3Gb using a specific flag.

In case of 64bit process our address space is almost unlimited (8Tb). Real memory, that can be committed, is limited only with size of RAM plus size of page file - all this memory may be addressed with our process.

OutOfMeowmory exception :)

OOM exception appears, when GC tries to allocate some contiguous part of space for new object, but fails with that. It may easily appear in 32bit process, as 2Gb is not that big deal, especially when we have, for example, large caches. OOM exception may hardly appear on 64bit systems (there were no such issues in my practice!), but I can assume it is possible, when your process has addressed and committed all available physical space or when virtual address space of the process is so fragmented by allocating/deallocating LOH, that there is absolutely no place for some new object.



Making a dump on OOM exception

Hello! Remember this blog post?

Make sure that you have a good dump by loading it to WinDBG and running !threads command.
In the "Exception" column of !threads output you should see OutOfMemoryException stored in one of the threads. If you do not have it, then dump was not created correctly. For example, customer may have set DebugDiag to catch every exception that it sees, or dump was created via TaskManager by hands - in this case success cannot be guaranteed too.

Important notice

Make sure you read next portion of text carefully, as it will save you from spending a couple of days on a dump that has no point of interest in it. I had this awful experience because of one nasty mistake :(

If you run !dumpheap -type Exception, it will always show you OutOfMemoryException and StackOverflowException at the end.


000007feef6004a8        1           24 System.Text.DecoderExceptionFallback
000007feef600430        1           24 System.Text.EncoderExceptionFallback
000007feef5f6ee8        1          136 System.ExecutionEngineException
000007feef5f6dd8        1          136 System.StackOverflowException
000007feef5f6cc8        1          136 System.OutOfMemoryException
000007ff0046bd78        1          144 Sitecore.Diagnostics.PerformanceCounters.InstanceNameIsNotAvailableException
000007feee2391f0        1          152 System.Net.WebException
000007feef5f6ff8        2          272 System.Threading.ThreadAbortException
000007feefdd2970        3          408 System.Threading.ThreadStateException
000007feef627b70        7          448 System.UnhandledExceptionEventHandler
Total 19 objects

This is simple output from a dump of a process, that is not experiencing any OOMs etc.
But...having 1 object of each of those types doesn't mean that you have the exception thrown. Why do you have it anyway?
That's just because of a nature or OOM and StackOverflowException. If application gets to one of those exceptional situations, it simply won't have enough memory to create exception object. That's why process has them already stored in kind of "emergency store", and in case of urgency will simply generate a link to one of these objects.


So how to work on OOM dumps?


Surprise :) nothing new to investigating OOMs comparably to other memory issues. You need to find most memory-consuming objects, blame them for being a culprit and make appropriate changes to code or configuration.
One small difference that I can recall, is that if you have 32bit process, and if you find nothing specific in dumps, or result of !dumpheap varies from one dump to another, this may just mean that you need more memory for your application. Switching to 64bit architecture is one of the acceptable solutions.

Wish you always have enough healthy memory! :)

Credits for images to:

http://images.dailydawdle.com

http://www.sisense.com