Meet Git

Last days I was struggling with Git and GitHub, as after work I sometimes make some custom projects and try to learn something new. So I'm going to keep in this post my own piece of advice on using git, as I've already stepped in one and the same situation several times :)

Permission denied (publickey):

If you try to clone a repository from the server to your Mac, and it says "Permission denied (publickey)", all you need to do is:
1. Show hidden files on Mac:

defaults write com.apple.finder AppleShowAllFiles TRUE

killall Finder

2. Check what you have in /Users/youruser/.ssh folder. In my case there was a github_rsa file.
3. Run

ssh-add ~/.ssh/github_rsa

It will say:

Enter passphrase for /Users/sitecore/.ssh/github_rsa:

But when you typing something, nothing appears. Don't worry - it's ok, just type your passphrase (password) and press Enter.

Voila! Identity added.

Clone only one branch:

git init
git remote add -t $BRANCH -f origin $REMOTE_REPO
git checkout $BRANCH

Standard git clone -b branchname - will clone full repository and just checkout you to needed branch. The same as if you did git clone (repository) and git checkout (branch)

Picture by https://github.com/billyroh

Investigating issues with locks

Tadammm

Last post from my "Windbg bible" series :) Let's finish it and then I'll go for something new and more complicated.

What does lock means? In simple words. I call "Lock" a situation, when several threads are requesting one and the same resource (object, file) when this resouce is already locked by some other thread. While this thread is holding a lock on a resource, while it is working on it - other threads are not able to touch it and just spend time in waiting mode.

What are the sympthoms of having lock-issue?

Investigating High Processor Usage issues

If you only know how much time it took for me to make this post! First of all, it was hard to start it, as sometimes I am a bit bored of writing about things that I have done many times already. And in addition, first version of the post was washed away by incorrectly closed browser. Oh no!.. Let's start again...

Today we are talking about heavy CPU load. How it all begins?  Usually customers are complaining about the fact that his solution suddenly started to eat 100% CPU. Actually, it can be not only 100, but any value that is higher than standard CPU consumption for this process. Requests can be performed slowly or are not served at all.

How to gather dumps in these situations, is described here. But often it will be even better to gather dumps simply by clicking context menu on w3wp process in TaskManager.

Investigating issues with Unmanaged Memory. First steps.

I want to write this blogpost really fast, until I forget everything, despite I have absolutely no time and need to work on other cases :( Let's count this as part of my work, okay?

Prologue.

You can scroll it down to scene 1, if you want to concentrate on concrete steps.

This case had a long story - starting from OOM troubles on 32-bit process, when process 32-bit mode was considered as main boundary for a large solution. BTW, my troubles with opening dumps at that point resulted in this blog post, if you remember it.
After switching to 64bit solution was running smoothly for some time, we even closed the support case. Suddenly (oh, I love this word!) our poor process had eaten 66Gb of memory, which is unbelievably huge. Actually, it was all of the memory available - 16Gb of RAM and 50Gb of page file.


Customer managed to gather a dump when process memory usage was at something like 6Gb. I was almost sure, that issue solution is struggling because of was this bad guy - Lucene analyzer. Luckily, we have script that counts all related to this issue data structures (and also counts cache sizes), so I thought that I will just run this script on the dump and confirm my assumptions.


I was a bit shocked when script has shown only 140Mb of Lucene-related things in memory. Cache sizes were in healthy range. So this is something new. I loaded our good old friend WinDBG and started analyzing memory as usually.

Investigating dumps with exception

Dump that contains concrete exception is useful in case of "Sometimes"-issues. "Sometimes"-issue is an issue, which description is started from one of the words - sometimes, rarely, suddenly etc. Do you recall some of such cases, and how often they are closed as not-reproducible? Memory dump is your friend here :)

In case of any exception I would suggest to gather dump the same as in case for OOM exception, which is described here. Do not forget to load needed modules and symbols, and after that you are ready to start investigation.


Actually, dump with concrete exception is simplest situation.

Pocket-size vocabulary

In this post I'm going to keep some useful WinDBG commands and tips. Simply and short :) I'll update it from time to time.

• .time 

           Shows date and time when dump was created:

• Opening a new dump. If you need to open another dump in same windbg window, choose Debug -> Stop debugging, or press Shift+F5.

• Question mark makes conversion from hexadecimal to decimal and vice versa:

0:082> ?1e74

Evaluate expression: 7796 = 00001e74

0:082> ?0n7796

Evaluate expression: 7796 = 00001e74

0n is prefix for decimal numbers

0x is prefix for hexadecimal numbers

• There is an autocompletion option in WinDBG: you may start writing !ana, then press Tab and it will be completed to !analyze

• .chain is a command to look currently loaded extensions:

• .unload is a command used to unload one of the currently loaded extensions. For example, to change it with another version of this extension. Make sure that you pass full name of the extension as a parameter - exactly in a form specified in .chain output:

• !sam c:\temp This command will export all dlls from current dump to the location you've specified as a parameter. Note, that this is a command from psscor dlls, so if you have loaded sos instead of psscor, it won't work.

• .logopen /t c:\temp\Output.txt - if you want to save your next commands output to a file.

Types of "I need your dump"-issues

So, you have installed WinDBG, you know how to create a dump (one, two and in the middle of three) , how to load it to WinDBG and how to load necessary modules and symbols. What's next?

There are lots of different commands to work with WinDBG, and at first you can feel lost in all this information. What should be your next step?

that's how I see the dump
investigation process :)

First of all, you need to categorize problem that you have. I can divide "dump"-issues into several categories:

1. Exception. Simply exception.

2. High processor load. Means that your w3wp process takes up to 100% of processor time. From my point of view, any number higher than 20% requires attention.

3. Memory issues. Out of memory exceptions and generally high memory usage. Additional ideas.

4. Locks. Problem with locks appear when you have several threads that are trying to access some resource, that is locked by another thread.




For each of these problem I'm going to create a separate post and add links to this kind of "Contents" page... Push me somebody, pleeeeease :)

UPD. Finished, yeeeey :)

Investigating memory issues. Additional info.

I'll add here some additional information about memory issues from time to time.

This post is only for those who have already read first and second posts.

I often receive questions if !dumpheap shows real size of the objects. Surely not - it shows only "own" size of the objects, without counting the size of its members. But if you need to know real object size, it can be done by using !objsize command. Only remember, that you need to pass some object address to this command:


0:058> !objsize 13c00824
sizeof(13c00824) =    156261460 (   0x9505c54) bytes (Sitecore.Data.Items.Item)

Without setting a parameter, !objsize will go through all objects in managed memory and count their size. This will take a looooong time. Maybe even a couple of days.

Only note, that all these result sizes may overlap each other, just because we have singletones and (I assume) static objects. So if a lot of objects of some type have huge size, and you have a lot of such objects  - check their members, most of all they have some common member.

One more useful thing. If you want to take a look through all gcroots of objects of some type, it would be great to use Windbg loops:

.foreach (obj { !dumpheap -type System.SomeType -short }) { !gcroot ${obj} }

Investigating memory issues. OutOfMemoryException.

General information about OOM.

Let's first grasp, how do we usually get that scary OOM exception.

In case of 32bit process (doesn't matter, if it runs on 32bit or 64bit system) our process has only 2Gb of available virtual address space. Actually, it has 4, but 2 of them are used by system needs. Maximum you may extend it to 3Gb using a specific flag.

In case of 64bit process our address space is almost unlimited (8Tb). Real memory, that can be committed, is limited only with size of RAM plus size of page file - all this memory may be addressed with our process.

OutOfMeowmory exception :)

OOM exception appears, when GC tries to allocate some contiguous part of space for new object, but fails with that. It may easily appear in 32bit process, as 2Gb is not that big deal, especially when we have, for example, large caches. OOM exception may hardly appear on 64bit systems (there were no such issues in my practice!), but I can assume it is possible, when your process has addressed and committed all available physical space or when virtual address space of the process is so fragmented by allocating/deallocating LOH, that there is absolutely no place for some new object.



Making a dump on OOM exception

Hello! Remember this blog post?

Make sure that you have a good dump by loading it to WinDBG and running !threads command.
In the "Exception" column of !threads output you should see OutOfMemoryException stored in one of the threads. If you do not have it, then dump was not created correctly. For example, customer may have set DebugDiag to catch every exception that it sees, or dump was created via TaskManager by hands - in this case success cannot be guaranteed too.

Important notice

Make sure you read next portion of text carefully, as it will save you from spending a couple of days on a dump that has no point of interest in it. I had this awful experience because of one nasty mistake :(

If you run !dumpheap -type Exception, it will always show you OutOfMemoryException and StackOverflowException at the end.


000007feef6004a8        1           24 System.Text.DecoderExceptionFallback
000007feef600430        1           24 System.Text.EncoderExceptionFallback
000007feef5f6ee8        1          136 System.ExecutionEngineException
000007feef5f6dd8        1          136 System.StackOverflowException
000007feef5f6cc8        1          136 System.OutOfMemoryException
000007ff0046bd78        1          144 Sitecore.Diagnostics.PerformanceCounters.InstanceNameIsNotAvailableException
000007feee2391f0        1          152 System.Net.WebException
000007feef5f6ff8        2          272 System.Threading.ThreadAbortException
000007feefdd2970        3          408 System.Threading.ThreadStateException
000007feef627b70        7          448 System.UnhandledExceptionEventHandler
Total 19 objects

This is simple output from a dump of a process, that is not experiencing any OOMs etc.
But...having 1 object of each of those types doesn't mean that you have the exception thrown. Why do you have it anyway?
That's just because of a nature or OOM and StackOverflowException. If application gets to one of those exceptional situations, it simply won't have enough memory to create exception object. That's why process has them already stored in kind of "emergency store", and in case of urgency will simply generate a link to one of these objects.


So how to work on OOM dumps?


Surprise :) nothing new to investigating OOMs comparably to other memory issues. You need to find most memory-consuming objects, blame them for being a culprit and make appropriate changes to code or configuration.
One small difference that I can recall, is that if you have 32bit process, and if you find nothing specific in dumps, or result of !dumpheap varies from one dump to another, this may just mean that you need more memory for your application. Switching to 64bit architecture is one of the acceptable solutions.

Wish you always have enough healthy memory! :)

Credits for images to:

http://images.dailydawdle.com

http://www.sisense.com

Issues on opening dumps

and this day come, and you can't open a dump in WinDBG. Dump is erring, refuses to load dlls and shakes its head each time when you are trying to run any well-known command.


Don't be threatened by this situation, because everything can be fixed. Everything. I need to believe in this :)

No gcroots?

I was pretty much wrong in my previous post, when said that if your object has no gcroots, you are probably experiencing a memory leak. My colleagues has corrected me at the meeting, so I decided to go through this theme once again.

And this is what I found:

If a lot of objects do not have gcroots, it just means that they possibly will be collected with next GC run. But! they may have unmanaged roots, and therefore will never be collected. Here is a cool article, explaining how this situation may happen: http://adavesh.blogspot.com/2012/02/memory-leaks-in-net-application-dont.html

Investigating memory issues. High memory usage.

I divide memory issues in 2 types:

• customer complains about OOM exception that sometimes occur.  For OOM exception, dump should be gathered with DebugDiag set for catching OOM (explained here)
• customer complains about high memory usage, however OOM exception is not thrown. For high memory usage - with DebugDiag set for high performance counters value (explained here)

Let's first grasp how to handle high memory usage.

After you've got a dump, what are our next steps? First of all, you should get main idea: find objects that consume most of the memory, then find where these objects come from ... and at last find a solution, how to reduce memory used.

I'll explain this main idea on one dump sample.

How to install WinDBG

When preparing a first lecture for my fellow colleagues, I occasionally understood that first you need to install   this WinDBG somehow.

WinDBG comes as a part of Debugging Tools for Windows. If you have Windows 7/Windows Server 2003,  it is the first quest - to find a version of debugging tools on this page:
Download and Install Debugging Tools for Windows

I came to the conclusion that this one is what you need:
Microsoft Windows SDK for Windows 7 and .NET Framework 4

When installing, the only thing you really need is definitely Debugging Tools themselves:

Make sure to choose your components

Here it is, our brilliant tool

Do you need x86 or x64 version of the Debugging Tools?

The computer that runs the debugger is called the host computer, and the computer being debugged is called the target computer.

...and more blablabla on this page: Choosing the 32-Bit or 64-Bit Debugging Tools

However, I can say that in my practice I always use 32-bit WinDBG for 32-bit dumps and 64-bit Windbg for 64-bit dumps. Otherwise I've got these awful "%1 is not a valid Win32 application" messages, or some other issues. Correct versions of WinDBG always gracefully opened correct versions of dumps for my investigations. So this is what I can advise. Will be glad to hear somebody's else opinion.

Download link above will install x64 version of WinDBG (if you have x64 system, I suppose). If you want to install x86 version of Windbg, go to this page:
http://www.microsoft.com/en-us/download/details.aspx?id=8442
and download full SDK iso of needed version. Not that convenient, but at least possible.

Gather dump on OOM exception

Gathering dump on exactly OOM exception. I didn't make this post by myself, as I found a pretty good one on the internet. Possibly I'll make some concretized repost later, if there is a need in it.

http://blogs.msdn.com/b/kaushal/archive/2012/05/09/using-debugdiag-to-capture-a-dump-on-first-chance-exception.aspx

For most situations you should follow exactly these steps EXCEPT OF Step 5:

Action type for unconfigured first chance exceptions: Full Userdump Log Stack Trace
Action Limit for unconfigured first chance exceptions: 10 1 

You do not need full user dump for every exception, especially for handled exception. We need full dump only for OOM case, and possibly a log for other exceptions.

Gather performance counters log

And one more post about gathering information for debugging. Along with dumps, it would be always great to have performance counters log - to see how process' memory and processor time has changed over the time, and compare this log with things we can dig in dumps.

First of all, press Windows' Start -> Run and call perfmon (Performance Monitor):

0. Start perfmon

And step-by-step screenshots:

How to gather dump with DebugDiag. Perfcounters.

As I often need to explain, how to create dumps, I'm going to create a small tutorial for each kind of them. Here we start.
If you experience kind of memory leak, when process is just eating memory without giving OOM or StackOverflow, it would be good to create a dump at the moment when used memory is already high, but not that high to induce an application shutdown or other unwanted consequences.

First of all, you need to download DebugDiag tool: http://www.microsoft.com/en-us/download/details.aspx?id=26798

Then open DebugDiag 1.2 from your Start menu and here are your next steps:

Timeouts when debugging in VS

If you are debugging a  w3wp process in VisualStudio, but it always breaks after some short time of inactivity, go to inetmgr, open Application Pools, select your pool, Advanced properties, and set Ping property to False:

WinDBG bible. Chapter one.

And here I am, starting bunch of posts (I hope there will be more than one!) about simple windbg usage. All you need to know if you are new to memory debugging, but want to step in it.

First of all. What is memory dump? Memory dump can be simply explained as "photo" taken of all memory consumed by our process at some point of time. Looking at this photo, we can everybody pictured in it, and blame somebody for taking too much space or jumping into the foreground scene :)

How to create memory dump?

I've just got in the situation where simple code like:

using (new SecurityDisabler()){
//publish item
}

didn't work.

I was so sure in fact that it should work because of my previous experience with Sitecore, that it took me some time to find the problem. And here it is: there is a setting Publishing.CheckSecurity in web.config. When it is set to true, Sitecore uses SecurityEnabler when publishes, so turning on SecurityDisabler before it makes no sense.

If you run into same question, just use UserSwitcher or disable Publishing.CheckSecurity.

To filter comments like <!-- smth with linebreaks --> in WinMerge:

1. Go to C:\Program Files (x86)\WinMerge, open IgnoreSectionMarkers.ini file
2. Delete everything that is there.
3. Put next lines in the file:


[set0]
StartMarker=<!--
EndMarker=-->
InlineMarker=<!--
FileType0=xml
FileType1=config

4. Save file and restart WinMerge.

To make these changes take effect, you need also next changes to WinMerge options:

"Compare" section: 

• disable "Match similar lines" checkbox
• enable "Filter comments" checkbox

Simple WinMerge filter:


---
def: include ##will select all files without [filter below] in name

f: (2) ## Match all files with (2) in name
f: \.disable$
f: \.example$
f: \.bak$

Very useful for Reflector:


If you use auto-properties instead of traditional ones they are named <Name>k__BackingField instead something like _Name.  

And I'm still discovering ECM :)

There is a new version of ECM that is going to be released soon. Well, now, when I'm going to publish this post, new version is officially released as ECM 1.3.2

Yesterday I had an informative meeting with Ivan Korshun about "what's new" in this upcoming version. Two main features are Emulation mode for concrete mails and DispatchSummary.aspx page.

Tip 1. First thing to remember is that DispatchSummary shows only current dispatch tasks, and doesn't show test task.

Tip 2. Another fun thing that I want to notice is understanding on generating and sending threads. Actually, there are 2 settings - NumberThreads and MaxGenerationThreads. Surprisingly, they are tied up with each other. For example, we can have 8 generating threads and 100 threads at all. This means, that we can generate 8 emails simultaneously. As soon as thread finishes generating email, it becomes a sending thread. Next free thread for generation is taken from the thread pool. And so on, until job is finished or until we reach end of pool.

Tip 3. Required Bandwidth = V/T. Amount of mails divided by time of sending (if only not local MTA is used).

All other information that we have discuseed, and consequently all dispatch real-time info that we can oversee, is described thoroughly in Sitecore ECM Tuning guide (http://sdn.sitecore.net/Products/ECM/ECM%201,-d-,3/Documentation.aspx). Looks like really powerful feature. Happy tuning!

Following up ECM theme: Mystic "RedirectUrlPage.aspx" is added to links when "href" property starts from url not equal to BaseUrl. RedirectUrlPage does nothing but returns 302 redirect code and adds "Location" header with value that is taken from real url. I've also found out how really ECM works when sends messages - it creates a link to download (see previous post), then this link is just processed as each standard request to Sitecore, with all pipelines, and returned output is being processed by several methods like replacing tokens and link modifications. Then output is being sent to subscribers.

Last couple of days I was buried in ECM code debug.

Some tips for future:

0. Processing is different for HTML mail and Newsletter. First one is regulated by HtmlMail, second one by WebPageMail. Both have one base class HtmlMailBase.

1. "Message Preview" composes an url and downloads it - to present you an actual email. Actions are the same as if you dispatch this email. This is relevant for WebPageMail, where HtmlMail just takes "Body" field value from message item.

2. Link to download an email is very easy to understand:

http://hostname/?sc_itemid=%7b9121BA1E-619A-4B84-BCB9-A1806F117810%7d&sc_lang=en&sc_device=%7bFE5D7FDF-89C0-4D99-9AA3-B5FBD009C9F3%7d&sc_mode=normal&sc_database=master&ec_recipient=sitecore%5cAdmin&ec_id=9033D12313D74836A7D5CECD015A3E82

http://hostname - equal to WebUtil.GetServerUrl(Context.Request.Url). You may check it in httpRequestBegin to make sure that correct hostname is resolved.

sc_itemid,sc_lang,sc_device,sc_mode,sc_database - properties of item that is chosen in "Body" ("Web Page") field of newsletter item. By default it is Newsletter Root under your newsletter item.

ec_recipient, ec_id - ECM properties, name of mail's PersonalizationContact and global EcmId.

3. When you put some links into the content, they are also transformed - sc_camp and ec_as are added there. sc_camp is email campaign ID, ec_as is ID of automation state (one of stored in AutomationStates table in Analytics db). More in Sitecore.Modules.EmailCampaign.HtmlMailBase.ModifyHrefLink

Lately, when user clicks some links, ec_as is processed for Analytics purposes, and as a nice tip, you may get current username using AnalyticsHelper.GetRecepient.

4. Tokens like $email$, $phone$ can be placed anywhere, even in links. Links are processed lately then tokens are replaced with actual values. One may define its own tokens in item, that is defined by ECM Manager root, Default Subscriber profile.

5. There are different URLs used by ECM. One is the url that you are launching Sitecore from - Context.Request.Url, and other is ECM Manager root setting BaseUrl. First is used to create this magic "DownloadLink" for message, second is used to create links inside a message.

6. Magical RegisterEmailOpened.aspx page, that is a source for 1-pixel image inserted in message automatically, returns 404 anyway and is used only for tracking purposes.