Friday, 22 June 2012

Issues on opening dumps

and this day come, and you can't open a dump in WinDBG. Dump is erring, refuses to load dlls and shakes its head each time when you are trying to run any well-known command.


Don't be threatened by this situation, because everything can be fixed. Everything. I need to believe in this :)


1. First, you should find out if you use correct version of WinDBG. For example, you see this message in WinDBG output, when running simple commands to load modules:

.loadby sos mscorwks
The call to LoadLibrary(C:\Windows\Microsoft.NET\Framework\v2.0.50727\sos) failed, Win32 error 0n193
    "%1 is not a valid Win32 application."
Please check your debugger configuration and/or network access.

This means that dump was created for 32-bit process, and we are trying to load 32-bit dlls into 64-bit debugger. Just to say, my PC is AMD64 architecture, 64bit OS.

Despite Microsoft says that we should be OK with that, I haven't found a way to open a 32-bit dump in 64-bit WinDBG even after a long play with symbols :)

So I've opened 32-bit WinDBG:

0:000> .loadby sos mscorwks

Looks like OK, but...

2. ...here comes another issue, that can be seen very often. This issue means that you have wrong version of mscordawks, and most possibly wrong version of sos. "Wrong" means different from what our dump's owner had when created a dump.

0:000> !threads
Failed to load data access DLL, 0x80004005
Verify that 1) you have a recent build of the debugger (6.2.14 or newer)
            2) the file mscordacwks.dll that matches your version of mscorwks.dll is 
                in the version directory
            3) or, if you are debugging a dump file, verify that the file 
                mscordacwks_<arch>_<arch>_<version>.dll is on your symbol path.
            4) you are debugging on the same architecture as the dump file.
                For example, an IA64 dump file must be debugged on an IA64
                machine.

You can also run the debugger command .cordll to control the debugger's
load of mscordacwks.dll.  .cordll -ve -u -l will do a verbose reload.
If that succeeds, the SOS command should work on retry.

If you are debugging a minidump, you need to make sure that your executable
path is pointing to mscorwks.dll as well.

Upsy.. But don't be upset, run cordll command!

0:000> .cordll -ve -u -l
CLR DLL status: No load attempts

That's ok that you've got nothing, to look at verbose output run any sos command:

0:000> !threads
CLRDLL: C:\Windows\Microsoft.NET\Framework\v2.0.50727\mscordacwks.dll:2.0.50727.5456 f:0
doesn't match desired version 2.0.50727.5448 f:0
CLRDLL: Unable to find mscordacwks_x86_x86_2.0.50727.5448.dll by mscorwks search
CLRDLL: Unable to find 'mscordacwks_x86_x86_2.0.50727.5448.dll' on the path
CLRDLL: Unable to find mscorwks.dll by search
CLRDLL: ERROR: Unable to load DLL mscordacwks_x86_x86_2.0.50727.5448.dll, Win32 error 0n2

Here we go. Let's take a closer look at the error message:

ERROR: Unable to load DLL mscordacwks_x86_x86_2.0.50727.5448.dll, Win32 error 0n2

  • First "x86" means architecture of your debugger - if you see this message in WinDBGx64, it will say "Unable to find 'mscordacwks_AMD64_x86_2.0.50727.5448.dll' on the path".
  • Second "x86" means mode that our process was running on, when dump was created. In this case it will be x86, in case of 64-bit process it will be x64.
  • Finally  2.0.50727.5448 means version of mscordacwks that we need.  Our version of mscordacwks is 2.0.50727.5456 (see first error message in the output), when we need 2.0.50727.5448.

The best thing that you may do now is to ask customer for his mscordacwks and sos. In this concrete case customer should take mscordacwks from "Framework" folder (as we need x86), from v.2.0.50727 folder - this all can be found from our error messages.
If you do not have the possibility to ask for the dlls, you'll have to find this dlls on the internet, but this is not that easy. I know, I tried :(


After you find needed dlls, put them to c:\symbols folder. Then we should tell our debugger to find symbols in this folder, if you haven't done it yet, and try to load everything again:


.symfix c:\symbols


0:000> .loadby sos mscorwks
0:000> !threads
CLRDLL: C:\Windows\Microsoft.NET\Framework\v2.0.50727\mscordacwks.dll:2.0.50727.5456 f:0
doesn't match desired version 2.0.50727.5448 f:0
CLRDLL: Unable to find mscordacwks_x86_x86_2.0.50727.5448.dll by mscorwks search
CLRDLL: Unable to find 'mscordacwks_x86_x86_2.0.50727.5448.dll' on the path
CLRDLL: Unable to get version info for 'c:\symbols\mscorwks.dll\4E154BCE5ab000\mscordacwks_x86_x86_2.0.50727.5448.dll', Win32 error 0n87
CLRDLL: ERROR: Unable to load DLL mscordacwks_x86_x86_2.0.50727.5448.dll, Win32 error 0n87


Error. Again. But this is better now :) Look at the last error message but one:

CLRDLL: Unable to get version info for 'c:\symbols\mscorwks.dll\4E154BCE5ab000\mscordacwks_x86_x86_2.0.50727.5448.dll', Win32 error 0n87

What we need to do now, is to go to c:\symbols\mscorwks.dll\4E154BCE5ab000\, rename our mscordacwks.dll to mscordacwks_x86_x86_2.0.50727.5448.dll, and put it to this folder.


3. One more possible issue. After actions above !threads now gives you an output, but ... kind of encrypted in addresses:

Index TID TEB StackBase StackLimit DeAlloc StackSize ThreadProc
0 0000000000000000 0x00000000fffdb000 0x0000000000000000 0x000000000022fd20 0x0000000000000000 0xffffffffffdd02e0 0xffffffff80004002


Looks like sos is not loaded. Remember that we already have needed version of SOS? I assume you asked it along with mscordacwks! Just put it anywhere, like in that c:\symbols folder, and load it directly:

.load c:\symbols\sos.dll

Good, no errors so far...



0:000> !threads

If you've got correct output, that's good. If not...

4. And one more issue. Mscordawks is OK, SOS is OK, but then what does it mean?

Failed to load data access DLL, 0x80004005
Verify that 1) you have a recent build of the debugger (6.2.14 or newer)


etc.etc.


Pfff. In this case the most possible is that you should switch to 32-bit mode. Yes, switch to 32-bit mode debug, debugging 32-bit dump in 32-bit version of WinDBG. Sounds totally crazy, but that's what we need to do:


.load wow64exts
!sw

You should receive a message:


Switched to 32bit mode

And now check your !threads:


ThreadCount: 103
UnstartedThread: 7
BackgroundThread: 76
PendingThread: 0
DeadThread: 13
Hosted Runtime: no
                                      PreEmptive   GC Alloc           Lock
       ID OSID ThreadOBJ    State     GC       Context       Domain   Count APT Exception
  18    1 1a14 0000000000f1a270      8220 Enabled  0000000000000000:0000000000000000 0000000000320bc0     0 Ukn
  55    2 3564 00000000028e9828      b220 Enabled  ffffffffe68d0160:ffffffffe68d210c 0000000000320bc0     0 Ukn (Finalizer)
  56    3 25c4 00000000028ffe80   180b220 Enabled  0000000000000000:0000000000000000 0000000000320bc0     0 Ukn (Threadpool Worker)

Hoooooray!


2 comments:

  1. Really informative blog, Keep blogging!

    God Bless you!

    ReplyDelete
  2. Tess recommends to use a tool for making correct dump instead of using wow64 extensions. Neat trick is to use task manager running in 32 bit mode for creating 32 bit dump (regular taskmanager will embed wow64 abstraction layer information in the dump).

    ReplyDelete