Investigating memory issues. Additional info.

I'll add here some additional information about memory issues from time to time.

This post is only for those who have already read first and second posts.

I often receive questions if !dumpheap shows real size of the objects. Surely not - it shows only "own" size of the objects, without counting the size of its members. But if you need to know real object size, it can be done by using !objsize command. Only remember, that you need to pass some object address to this command:


0:058> !objsize 13c00824
sizeof(13c00824) =    156261460 (   0x9505c54) bytes (Sitecore.Data.Items.Item)

Without setting a parameter, !objsize will go through all objects in managed memory and count their size. This will take a looooong time. Maybe even a couple of days.

Only note, that all these result sizes may overlap each other, just because we have singletones and (I assume) static objects. So if a lot of objects of some type have huge size, and you have a lot of such objects  - check their members, most of all they have some common member.

One more useful thing. If you want to take a look through all gcroots of objects of some type, it would be great to use Windbg loops:

.foreach (obj { !dumpheap -type System.SomeType -short }) { !gcroot ${obj} }

Investigating memory issues. OutOfMemoryException.

General information about OOM.

Let's first grasp, how do we usually get that scary OOM exception.

In case of 32bit process (doesn't matter, if it runs on 32bit or 64bit system) our process has only 2Gb of available virtual address space. Actually, it has 4, but 2 of them are used by system needs. Maximum you may extend it to 3Gb using a specific flag.

In case of 64bit process our address space is almost unlimited (8Tb). Real memory, that can be committed, is limited only with size of RAM plus size of page file - all this memory may be addressed with our process.

OutOfMeowmory exception :)

OOM exception appears, when GC tries to allocate some contiguous part of space for new object, but fails with that. It may easily appear in 32bit process, as 2Gb is not that big deal, especially when we have, for example, large caches. OOM exception may hardly appear on 64bit systems (there were no such issues in my practice!), but I can assume it is possible, when your process has addressed and committed all available physical space or when virtual address space of the process is so fragmented by allocating/deallocating LOH, that there is absolutely no place for some new object.



Making a dump on OOM exception

Hello! Remember this blog post?

Make sure that you have a good dump by loading it to WinDBG and running !threads command.
In the "Exception" column of !threads output you should see OutOfMemoryException stored in one of the threads. If you do not have it, then dump was not created correctly. For example, customer may have set DebugDiag to catch every exception that it sees, or dump was created via TaskManager by hands - in this case success cannot be guaranteed too.

Important notice

Make sure you read next portion of text carefully, as it will save you from spending a couple of days on a dump that has no point of interest in it. I had this awful experience because of one nasty mistake :(

If you run !dumpheap -type Exception, it will always show you OutOfMemoryException and StackOverflowException at the end.


000007feef6004a8        1           24 System.Text.DecoderExceptionFallback
000007feef600430        1           24 System.Text.EncoderExceptionFallback
000007feef5f6ee8        1          136 System.ExecutionEngineException
000007feef5f6dd8        1          136 System.StackOverflowException
000007feef5f6cc8        1          136 System.OutOfMemoryException
000007ff0046bd78        1          144 Sitecore.Diagnostics.PerformanceCounters.InstanceNameIsNotAvailableException
000007feee2391f0        1          152 System.Net.WebException
000007feef5f6ff8        2          272 System.Threading.ThreadAbortException
000007feefdd2970        3          408 System.Threading.ThreadStateException
000007feef627b70        7          448 System.UnhandledExceptionEventHandler
Total 19 objects

This is simple output from a dump of a process, that is not experiencing any OOMs etc.
But...having 1 object of each of those types doesn't mean that you have the exception thrown. Why do you have it anyway?
That's just because of a nature or OOM and StackOverflowException. If application gets to one of those exceptional situations, it simply won't have enough memory to create exception object. That's why process has them already stored in kind of "emergency store", and in case of urgency will simply generate a link to one of these objects.


So how to work on OOM dumps?


Surprise :) nothing new to investigating OOMs comparably to other memory issues. You need to find most memory-consuming objects, blame them for being a culprit and make appropriate changes to code or configuration.
One small difference that I can recall, is that if you have 32bit process, and if you find nothing specific in dumps, or result of !dumpheap varies from one dump to another, this may just mean that you need more memory for your application. Switching to 64bit architecture is one of the acceptable solutions.

Wish you always have enough healthy memory! :)

Credits for images to:

http://images.dailydawdle.com

http://www.sisense.com

Issues on opening dumps

and this day come, and you can't open a dump in WinDBG. Dump is erring, refuses to load dlls and shakes its head each time when you are trying to run any well-known command.


Don't be threatened by this situation, because everything can be fixed. Everything. I need to believe in this :)

No gcroots?

I was pretty much wrong in my previous post, when said that if your object has no gcroots, you are probably experiencing a memory leak. My colleagues has corrected me at the meeting, so I decided to go through this theme once again.

And this is what I found:

If a lot of objects do not have gcroots, it just means that they possibly will be collected with next GC run. But! they may have unmanaged roots, and therefore will never be collected. Here is a cool article, explaining how this situation may happen: http://adavesh.blogspot.com/2012/02/memory-leaks-in-net-application-dont.html

Investigating memory issues. High memory usage.

I divide memory issues in 2 types:

• customer complains about OOM exception that sometimes occur.  For OOM exception, dump should be gathered with DebugDiag set for catching OOM (explained here)
• customer complains about high memory usage, however OOM exception is not thrown. For high memory usage - with DebugDiag set for high performance counters value (explained here)

Let's first grasp how to handle high memory usage.

After you've got a dump, what are our next steps? First of all, you should get main idea: find objects that consume most of the memory, then find where these objects come from ... and at last find a solution, how to reduce memory used.

I'll explain this main idea on one dump sample.

How to install WinDBG

When preparing a first lecture for my fellow colleagues, I occasionally understood that first you need to install   this WinDBG somehow.

WinDBG comes as a part of Debugging Tools for Windows. If you have Windows 7/Windows Server 2003,  it is the first quest - to find a version of debugging tools on this page:
Download and Install Debugging Tools for Windows

I came to the conclusion that this one is what you need:
Microsoft Windows SDK for Windows 7 and .NET Framework 4

When installing, the only thing you really need is definitely Debugging Tools themselves:

Make sure to choose your components

Here it is, our brilliant tool

Do you need x86 or x64 version of the Debugging Tools?

The computer that runs the debugger is called the host computer, and the computer being debugged is called the target computer.

...and more blablabla on this page: Choosing the 32-Bit or 64-Bit Debugging Tools

However, I can say that in my practice I always use 32-bit WinDBG for 32-bit dumps and 64-bit Windbg for 64-bit dumps. Otherwise I've got these awful "%1 is not a valid Win32 application" messages, or some other issues. Correct versions of WinDBG always gracefully opened correct versions of dumps for my investigations. So this is what I can advise. Will be glad to hear somebody's else opinion.

Download link above will install x64 version of WinDBG (if you have x64 system, I suppose). If you want to install x86 version of Windbg, go to this page:
http://www.microsoft.com/en-us/download/details.aspx?id=8442
and download full SDK iso of needed version. Not that convenient, but at least possible.

Gather dump on OOM exception

Gathering dump on exactly OOM exception. I didn't make this post by myself, as I found a pretty good one on the internet. Possibly I'll make some concretized repost later, if there is a need in it.

http://blogs.msdn.com/b/kaushal/archive/2012/05/09/using-debugdiag-to-capture-a-dump-on-first-chance-exception.aspx

For most situations you should follow exactly these steps EXCEPT OF Step 5:

Action type for unconfigured first chance exceptions: Full Userdump Log Stack Trace
Action Limit for unconfigured first chance exceptions: 10 1 

You do not need full user dump for every exception, especially for handled exception. We need full dump only for OOM case, and possibly a log for other exceptions.

Gather performance counters log

And one more post about gathering information for debugging. Along with dumps, it would be always great to have performance counters log - to see how process' memory and processor time has changed over the time, and compare this log with things we can dig in dumps.

First of all, press Windows' Start -> Run and call perfmon (Performance Monitor):

0. Start perfmon

And step-by-step screenshots:

How to gather dump with DebugDiag. Perfcounters.

As I often need to explain, how to create dumps, I'm going to create a small tutorial for each kind of them. Here we start.
If you experience kind of memory leak, when process is just eating memory without giving OOM or StackOverflow, it would be good to create a dump at the moment when used memory is already high, but not that high to induce an application shutdown or other unwanted consequences.

First of all, you need to download DebugDiag tool: http://www.microsoft.com/en-us/download/details.aspx?id=26798

Then open DebugDiag 1.2 from your Start menu and here are your next steps: