This idea started with a question I posted to StackOverflow.com a few weeks ago.
Basically, I have been writing WinDbg extensions the ‘normal’ way for a while. I found them extremely useful for parsing data from large DMP files. Specifically, I work on a large application, and being able to pull information from data structures in the minidump (with full memory) is awesome. I would write code like this:
I was checking out the newish website StackOverflow.com, where you can post up technical questions, and other programmers answer them. Its actually fun, cause you can earn reputation points and ‘badges’ for answering questions and whatnot.
At any rate, I posted up this question about how to modify the age of a pdb file. I asked because our build process has been broken for a number of years, such that we were generating .PDB file that didn’t match the corresponding DLLs. It was an unfortuate side effect of having bad dependencies causing our application to relink when it didn’t need to. After each release engineering build target there is a packaging step that gathers the images (dlls and exes) needed for that target. However, the PDBs aren’t gathered until the end. Therefore, if an image was relinked between those two times, the newly relinked pdb and the original dll would be off by a few hours.
This could be worked around by using the .symopt +0×40 option in windbg. It forced the pdbs to work properly. However, they would not work in VisualStudio. Also, it meant we couldn’t run a symbol server, because with +0×40 it would just grab the first pdb of that name it found.
The great news is that I got an answer on stackoverflow! I was pointed to this site http://undocumented.rawol.com/ which contains the pdfs of the fantastic out of print book called “Undocumented Windows 2000 Secrets: A Programmers Cookbook”. The author goes into great detail about the PDB file format, and includes code samples. He even updated the website to include information about version 7.0 (what VisualStudio 2003 uses).
So how do you do it? There are potentially 3 places that contain age & signature information that need to match the corresponding image (dll or exe) file. Two of them are easy to find, because the signature is very recognizable. It is in the header of the file, and easily picked out with a hex editor. I even found this before I was pointed to the book website. In fact, the second reference, near the end of the file, is also a signature&age combination, so it is fairly easy to find as well. However, the last one is TRICKY. Each PDB file is a compound file, and is split into ’streams’. Each stream contains specific information. The 3rd stream contains a reference to the age, but not the signature. Since the age is only a simple 4byte integer value, it is very difficult to pick out on its own! But once I knew where to look, it was easy. I have modified a few releases by hand, but I hope to write a program to automate the process.
Lets say you want to set a break point, but only break into the debugger when you havn’t gotten to this point via a given code path. You essentially want to only break when a given function is NOT in the stack trace. How do you do this? In VisualStudio (2003 at least), you don’t. The magic of windbg!
I like to use scripts cause I can be very verbose, but you could do this on one line as well. First, just set a breakpoint:
bu `mymodule!myfile.cpp:92` "$$><C:\\windbgscripts\\myscript.txt"
That command sets a breakpoint that will save in a workspace, on the file/line specified. Further, it executes the script specified in the quotes whenever you hit this breakpoint.
Now comes the fun part. Whats in the script?
.if ( @@c++(this->;;m_cRef) != 0){ .foreach (line {k100}) { .if ($SPAT("line","*MyClass::MyFunc*")) {gc} };k } .else {gc}
Look at that madness!!!!! Basically, all I am doing here is checking if the refcount of ‘this’ is zero. If it is, I just continue with the ‘gc’ command (gc is ‘go from conditional breakpoint’). If it isn’t zero, I am interested in it. Then I use the .foreach command. .foreach tokenizes anything in windbg. It can take command or external files as input. In this case, I give it the ‘k100′ command. This dumps a stacktrace with a depth of 100. Each token is assigned to the variable ‘line’ and the code is executed. The .if statements use the $SPAT command which check if 2 strings are equal. The cool thing is that you can put wildcards in there. So I just put in some strings from code paths which I don’t want to break, and then do a ‘gc’. This essentially causes me to only break in the debugger when those strings are NOT in my stacktrace!
WOW!
I am currently traveling to Memphis on business, but I am trying to figure out a couple difficult mysteries of the debugging world. I know someone out there can help!
1) What happened to the ‘Stack Trace Depth’ setting in GFlags? Why was it removed? Is the functionality just gone? I have a memory leak plaguing our customers, but our application is very complex and the level at which the allocation happens is much much much more than 16 levels. All the documentation I can find says to ‘change the Stack Trace depth edit control’ in gflags, but it doesn’t seem to be there!!!
2) How can I set a windbg breakpoint that breaks only when a certain path hasn’t been taken through the code to reach this point. In other words, if module!object::functionX is in the call stack, I do NOT want to break, but rather continue, until I reach this point from a different path. Do I have to do a ‘k’ combined with a .foreach command, and parse each stack? That will slow things down, but may work if I can get the syntax correct.
If I figure these out, I will post what I find…..or maybe someone out there can post something in the comments.
I was tracking down a memory leak using DMP files using a method outlined here. Windbg is great for this kind of memory leak debugging because you don’t have to worry about symbols at the time of the dmp gathering. Thus, you can start your debugging back home, after you get back from the customer site! If you are debugging in the office, it would probably be easier to just use a program like UMDH.
Anyway, after I had gathered my .DMP files (with user mode stack trace databases turned on), I loaded them into windbg and started to take a look. Unfortunately as I did a !heap -a, I kept getting errors like:
heap entry extra at 000a24a0
000a24a8: 01818 . 00030 [07] – busy (18), tail fill – unable to read heap
Vostokov also discusses this error here. Being the good debugger I used the win2k version of !heap as he suggests, and it allowed me to !heap -a, and get some addresses. Great!
Now I want to dump out some stacks!
!heap -p -a
ERRRROOOORR. No pageheap information? But I used gflags to +ust, why do i need pageheap enabled? I just want stack traces! All the webpages and books seem to suggest you can do this. WTF?
Turns out, the once you use the win2k version of !heap ONCE, it will always use that version during that run of windbg! So simply reload your dump file, and you should be able to get stacktrace information again.