SOS from your production environment
Developing large enterprise applications is a complex and difficultundertaking. Writing the code is just one of many tasks we have to do. Weworry about requirements, designs, architecture, unit testing, daily builds,release builds to QC and many more things. All this effort is spent tocreate a reliable, scalable, well performing and functioning application.Then comes the day where we move it in production (if you are lucky and itis hosted by your own organization) or customer start installing it on theirservers. This is a big day, a day of celebration. We see the fruits of allour labor and we are excited to see users using the application, gettingtheir feedback and improving the application. But too often it starts tohaunt you. The customer reports crashes, instability or unpredictablebehavior. You tell yourself, but it is working on our environments. What isdifferent between our test environments and the customer environments?
This is one of the most difficult challenges a development team can face.Your options are suddenly limited. On your development environment you fireup the VS.NET debugger, set breakpoints, look at the application state, etc.Through that you are finally able to figure out what is going on and thenmake your code change. But tell the customer that you need to install VS.NETto be able to debug this issue. Watch out for the reaction, it might bepretty nasty. Production environments are very locked down and only approvedapplications can be installed. Very often any change applied to productionneeds to go through a stringent test process, which takes time. All thiswhile the end users have to bear with the stability, performance orfunctional problems. If this goes on for too long then users will abandonthe application and the organization has to fight an uphill battle toconvince end users to come back. This creates lots of frustration, noise,problems and can result in large losses. This is ultimate hell for everydeveloper. You have no idea what is going on while everyone expects aresolution by yesterday.
Gather data to make informed decisions
Applications can behave very different in various environments and underload. First stop worrying about all the shouting. Concentrate on gatheringthe right data so you can narrow down what is going on. Start with basicinformation, like which OS and Windows patches are installed. Look at theevent log to find out if there are system or application errors reported. Ifnot done automatically, run a virus check too make sure there is no virusinfection going on. Enable your custom application logs and comb throughthem to find out what is happening. If all that does not uncover anything,then understand how the application is used. Which features are used heavilyby users, how many concurrent users are on the system, etc. Then replicate asimilar environment in house and run a load test against it, which simulatesa usage scenario as close as possible (see myarticle about concurrent users stress testing).
If all that does not bring you closer to a resolution then you need to takea snapshot of the application in production and analyze it. This articlewill introduce you to the basic approach for this and then point you to moreadvanced articles. It is easier then most people believe. Microsoft hasbuilt a very nice debugging story ? in the unmanaged as well as managedworld.
The "Debugging Tools for Windows"
Microsoft provides debugging tools for Windows NT 4.0, Windows 2000, WindowsXP and Windows 2003. The homepage for the "Debugging Tools for Windows" canbe found here. Follow the link "Install Debugging Tools for Windows 32-bit Version" todownload the latest version of them (this article uses the version 188.8.131.52).The tools by default are installed in the folder "c:\program files\debuggingtools for windows". The install also adds a menu group "Debugging Tools forWindows" under "All Programs". This includes a "Debugging Help" whichprovides some very good information.
There are a number of debuggers which you can use to debug your application.This article will concentrate on how you can take a dump of your applicationand then analyze these dumps on another environment and not the productionenvironment itself. You will see how you can take a dump when theapplication hangs, crashes or just while it is running. These dumps includea complete memory dump so you can see all the threads executing, all theobjects on the stack, etc. This is the least intrusive approach in reallyunderstanding what is happening in your application while used inproduction. This does also not require any files to be registered, whichmakes it easier to get permission to use it in production and also to removeagain when no longer needed (which the customer might request). Install thedebugging tools on any machine you want and then copy the following fivefiles from the "c:\program files\debugging tools for windows" folder to theproduction environment:
You don?t need to register the DLL?s. The file cdb.exe is the "MicrosoftConsole Debugger" and the file adsplus.vbs is a windows scripting file whichis used to automate the CDB debugger. This requires the Windows ScriptingHost 5.6 to be installed (run cscript.exe to check the version number). Ifrequired download the version from here and install it on the production server.
Always create the symbol files for your binaries
Symbol files are needed by a debugger to be able to show you more then justclass, method and object addresses. Symbols enable debuggers to show you theclass names, variable names, etc. You can debug an application withoutsymbols, but it is much harder and needs a lot of experience. You want tomake your life as easy as possible, therefore always generate the symbolfiles. When you compile your application in debug mode you will see in thesame folder where the DLL or EXE gets generated also a PDB file. The PDBfile is the symbol file which you need for debugging purpose. Of course youdo not want to release the debugging version of your binaries. You can tellthe compiler also to generate these symbol files when compiling in releasemode. Open the project settings in your Visual Studio .NET IDE (menu Project| Settings). Select the Build tab, select in the Configuration drop down box"Release" if not already selected and then click on the Advanced button. Inthe "Debug Info" drop down box select "PDB-only". Close your projectsettings and rebuild your project. You need to do that for all projectfiles. Make it a habit that when you release your application you not justrelease the binaries (DLL?s and EXE?s) but also all its symbols. Thereforeyou have the symbols ready anytime you need them for debugging purpose.
Symbol files contain information like all the class names, method names,global and local variable names as well as source line numbers. They arekept separate so that your binaries are smaller and faster when running.Later in the article we explain how you can load these symbols into thedebugger. You can also obtain all the symbols for the Windows OS, the .NETframework and
her Microsoft products. You can tell the debugger todownload it as needed from the internet or if you do not have access to theInternet while debugging you can download them from the Microsoft site (Windows symbols). The article will explain how to set up your debugger to downloadMicrosoft symbols files as needed.
Using ADPlus to take application dumps
Now you are ready to take dumps. First start your application. The articlehas a ThrowException .NET sample application attached which allows you togenerate two unhandled exceptions. We will use this sample application towalk through all the examples in this article. Next open the task managerand go to the "Process" tab. Select the check box "Show processes from allusers" at the bottom so you can see all processes running. Next find theprocess named "ThrowException.exe" and note down the process ID (shown inthe PID column).
ADPlus has a number of command line operations. First you need to decide ifyou want to perform a crash dump or hang dump. A crash dump is forsituations when your application unexpectedly terminates. Hang dumps can beused to take a dump when your application hangs or any time while it isrunning. ADPlus can not be used in scenarios where your application crasheswhile starting up. It can only be used for applications which are runningand then crash. Use the CDB or WinDbg debuggers for scenarios where yourapplication crashes during startup. ADPlus is automating the CDB debuggerand attaches it to your process. It can also be used to attach it tomultiple processes, for example your application runs under IIS and usesalso COM+. When CDB kicks in it freezes all processes it has been attachedto, takes a dump for each asynchronously and then lets these processescontinue to run.
Running ADPlus in crash mode
Open a command prompt and go to the folder where you installed or copied thedebugging files. You need to provide at a minimum the following command linearguments when running ADPlus:
- Mode ? The mode you want the CDB debugger to run in. Add "-crash"for crash mode or "-hang" for hang mode.
- Process to monitor ? Add "-p <process id>" to tell CDBwhich process to attach to. You can repeat that option for each process youwant to monitor. For each process it spawns a separate instance of CDB.
- Quiet mode ? When you run ADPlus it will show at the beginning adialog box telling you which mode has been chosen and where the log fileswill be created. When you run ADPlus on a remote machine then you need tosuppress this dialog box otherwise ADPlus itself will hang (see later in thearticle). Add the option "-quiet".
- Location of log files ? With the option "-o <log filepath>" you can specify the path where the log file will be created. TheCDB debugger creates a unique folder each time it runs under that log filepath. The folder name will be a combination of the mode and date and timethe CDB has been started, for example:
This guarantees that no dump will be overwritten with another dump. In thatfolder you find the actual memory dump as well as a number of log files. Thefile "ADPlus_report.txt" contains information about the configuration theCDB debugger has been started up with. The file "Process_List.txt" listsinformation about all the processes running when CDB started. The file"PID-<process id>__<processname>__<date>__<time>.log" contains all the output of the CDBdebugger while running. The actual dump generated by CDB gets placed in thefile "PID-<process id>__<process name>__<…>.dmp".
- Symbol path ? The option "-y <path> specifies the pathwhere the symbol files can be found. The path contains three pieces ofinformation:
- Symbol server ? The symbol server to use. This should always be"srv" unless you have a custom symbol server you utilize.
- Downstream store ? The downstream symbol store, e.g."c:\symbols". CDB will cache symbols from the upstream store to thedownstream store, providing a cascading symbol store cache.
- Upstream store ? the upstream symbol store. This can be a localpath, a network path or a URL.
All three pieces of the path should be separated by a "*". The followingexample points to the public symbol store from Microsoft and uses a localdownstream store:
This allows to download CDB the symbols to your local store which makes itmuch faster for any subsequent access to the symbol file. Symbols are copiedto the downstream store as CDB requires it. So it doesn?t just go ahead andcopy every symbol file. You can also list multiple symbol stores byseparating each with a semicolon. The next example points to the Microsoftpublic symbol store as well as the symbol files of your application:
You can also use the environment variable "_NT_SYMBOL_PATH" instead of usingthe "-y" option. As mentioned earlier in the article, you can download allthe Microsoft symbols if the production environment does not have internetaccess. This also means that all your application symbols should be copiedto a folder on the production environment. The following article provides a much more comprehensive explanation ofthe symbol stores and symbol server.
- Exception mode ? Any exception can be raised to the debugger asfirst-chance exception or second-chance exception. First chance exceptionsare non fatal exceptions which are handled by the application. If a firstchance exception is not handled by the application then it gets raised assecond chance exception. Only debuggers can handle second chance exceptions.Second chance exceptions normally cause the application to shut down, unlessa debugger is attached to it. By default ADPlus takes a minim dump for allfirst chance exceptions except unknown and EH exceptions (these are quitecommon and would generate too much overhead). This pauses the thread, logsin the log file the exception, thread ID and call stack of the thread whichraised the exception as well as the date and time when the exceptionoccurred. Finally it takes the mini dump and then resumes the process. Thefollowing four command line options control what action is taken when afirst chance or second chance exception happens:
- Full dump on first chance exceptions ? The option "-FullOnFirst"tells ADPlus to take a full dump for first chance exceptions.
- No dump on first chance exceptions ? The option "-NoDumpOnFirst"tells ADPlus to take no dumps at all for first chance exceptions.
- Mini dump for second chance exceptions ? By default ADPlus takesa full dump for second chance exceptions. The option "-MiniOnSecond" tellsADPlus to only take mini dumps at second chance exceptions. This is usefulwhen you need to send the dump to someone to look at. These are small dumpswhile full dumps can be hundreds of mega bytes and are difficult to sendaround.
- No dump on second exceptions ? The option "NoDumpOnSecond" tellsADPlus not to generate any dumps on second chance exceptions.
- < b>Notification ? The option "-notify <machine name> willsend an alert to the machine when a crash dump is taken. This will bring upa message box on the machine and is useful so you don?t have to wait till acrash happens.
For a complete list of all the ADPlus command line arguments, please referto the topic "ADPlus Command-Line Options" in the "Debugging Help". It alsoexplains how you can create a configuration file with all these settings andtell ADPlus with the option "-c <configuration>" to use theconfiguration file instead. Assuming that the application ThrowExceptionruns under the process ID 2828, here is how to start ADPlus in crash mode,logging all information in the "c:\crashlogs" folder.
ADPlus ?crash ?p 2828 ?o c:\crashlogs ?y"srv*c:\symbols*c:\ThrowException;srv*c:\symbols*http://msdl.microsoft.com/download/symbols" ?quiet -FullOnFirst
This spawns a new window which shows the CDB debugger attached to yourapplication. You can press Ctrl+C in that window anytime to take a hang dumpif no crash happens. But this will terminate the process. ADPlus can not berun in crash mode through Terminal Server on Windows NT 4.0 and Windows2000. The following article explains how to run in crash mode remotely. Italso contains more detailed information about how to use ADPlus.
Running ADPlus in hang mode
A hang dump will be taken the moment you run it. The CDB debugger attachesto the process, freezes the process, takes a full dump, detaches again andthen resumes the process again. This does not terminate the process at all.The hang mode can be run locally or remotely through Terminal Server. Allcommand line options explained in the previous section apply to the hangmode, except the Exception mode and Notification. Here is a sample of a hangdump:
ADPlus ?hang ?p 2828 ?o c:\crashlogs ?y"srv*c:\symbols*c:\ThrowException;srv*c:\symbols*http://msdl.microsoft.com/download/symbols" ?quiet
Analyzing the dump file
Taking the dump file is only half the work. Now that we have the dump filewe need to learn how to read it. The remainder of this article will assumethat we have taken a full dump. We will also assume that we are using the.NET application attached to this article. Refer to the "Debugger Help" ifyou need to analyze a dump from an unmanaged application. First copy it offthe production environment to another machine so you can analyze it withoutdisturbing the production environment itself. The machine you use to analyzeneeds to have the "Debugging Tools for Windows" installed.
Through the start menu "All Programs | Debugging Tools for Windows" startthe WinDbg debugger. Everything we will walk through can also be donethrough CDB, which is console based while WinDbg as a Windows UI. First weset again the Symbol path. Go to the menu "File | Symbol File Path". Enterthe same as you passed along to ADPlus using the "-y option". In our casethis is as follows:
This means you should also have the symbol files of your applicationavailable on the machine where you analyze the dump file. Next you open thedump file through the menu "File | Open Crash Dump". This can be either acrash or hang dump. Select the "DMP" file created by ADPlus. When you load acrash dump it will show that there was a second chance exception.
Loading the SOS debugger extension
Before we can start digging into the dump we need to load the SOS (Son ofStrike) extension for .NET. This extension provides an easy way to analyzemanaged data structures and look at the managed world. You can find the SOSextension at "%windir%\Microsoft.NET\Framework\<.NET version>\sos.dll". Thedebugging tools include a newer version of the SOS extension for .NET 1.0and 1.1 which is located at "c:\program files\debugging tools forwindows\clr10\sos.dll".
At the bottom of the WinDbg window you see a single text box, which allowsyou to type in commands for the debugger. Commands can be started with anexclamation mark or a dot. This article will use the dot notation.
- .load ? The load command can be used to load debuggingextensions. If you have an environment variable which has a path to the .NETframework files then you can simply use ".load sos". You can also beexplicit and type in the full path ".load c:\program files\debugging toolsfor windows\clr10\sos". We will debug a .NET 2.0 Beta 1 application so weuse instead ".load c:\windows\Microsoft.NET\ Framework\v2.0.40607\sos".
- .unload ? Is used to unload a debugger extension. You can providethe name of the extension, for example sos, or if no name is provided itwill unload the last extension loaded. You can also use the unloadallcommand to unload any extension loaded.
- .chain ? Used to display the chain of loaded debuggerextensions.
Now that we have loaded the SOS extension we can start using its debuggercommands to look at the .NET application dump. All the SOS debugger commandsneed to be started with the exclamation mark. Please note that the commandsand options will differ depending on which version of the SOS you load. Ofcourse the SOS provided as part of the latest version of the debugging toolswill provide the most recent commands. Note that the latest debugging toolsdo not yet contain an updated version of the .NET 2.0 SOS extension. Thisone should become available with the Beta 2 of .NET soon.
Digging into the .NET dump file using SOS
Use the "!help" command do display a list of all the available debuggercommands provided by the SOS extension. First we want to see a list of allthreads and their status. You can use the "~" (without the exclamation mark)command to list all the unmanaged threads. We of course want to look at themanaged threads, therefore we use the "!threads" command. You will see alisting like this:
ID ThreadOBJ State Domain APT Exception
. 0 1 001530f8 6020 00149ff8 STA System.IO.DirectoryNotFoundException
2 2 00161248 b220 00149ff8 MTA (Finalizer)
You can see that thread number zero is a STA thread and it has thrown theexception. Thread number two is a MTA thread which is used by the garbagecollector to run any Finalize method if implemented by an object before itgets destroyed. Next we want to find out more about the exception itself. Weuse the "!PrintException" command and pass along the address of theexception as shown by the "!threads" command. You will see the followinginformation:
0:000> !PrintException 00c01c3c
Exception type: System.IO.DirectoryNotFoundException
Message: Could not find a part of the path 'w:\MyFile.txt'.
0012f140 77e649d3 [Frame: 12f140]
0012f180 78cb9da9 System.IO.__Error.WinIOError(Int32, System.String),
0012f1ac 78a8add4 System.IO.FileStream.Init(System.String, …), mdToken:
0012f258 78a8aa13 System.IO.FileStream..ctor(System.String, …), mdToken:
0012f288 78a8d295 System.IO.FileStream..ctor(System.String, …), mdToken:
0012f2b0 78b66e59 System.IO.File.Create(System.String, Int32, Boolean),
0012f2c4 78b66e05 System.IO.File.Create(System.String), mdToken: 0600355c
0012f2c8 78b68250 System.IO.FileInfo.Create(), mdToken: 0600359b
0012f2d8 7b3b5f15 System.Windows.Forms.Control.OnClick(System.EventArgs),
0012f2e8 7b3ed65d System.Windows.Forms.Button.OnClick(System.EventArgs),
0012f2f4 7b3ed7a9 System.Windows.Forms.Button.OnMouseUp(…), mdToken:
0012f31c 7b3ba2f7 System.Windows.Forms.Control.WmMouseUp(…), mdToken:
0012f358 7b363775 System.Windows.Forms.Control.WndProc(…), mdToken:
0012f374 7b371345 [Frame: 12f374]
Please note that this is an abbreviated list. You can see the individualstack frames as well as all the calls. You see the class and method namesfor each call, the complete method signature (please note the list is againabbreviated in some cases) and a meta-data token ID. Next you can use themeta-data token and obtain the method table and method description of aspecific method call shown above. We want to do that for the last methodcalled from within our own code ? UnhandledException2_Click. We use the"!Token2EE" command, passing along the name of the module and the meta-datatoken shown from the stack trace. The information you get is like this:
0:000> !Token2EE ThrowException.exe 0600000d
JITTED Code Address: 05200378
Please note that the list is again abbreviated. You see the module address(module where the class is loaded from), the meta-data token, the methoddescription address and the method name. Next we want to find out more aboutthe module itself, which we can do with the "!DumpModule" command passingalong the module address. The information you get looks like this:
0:000> !DumpModule 00187990
MetaData start address: 004023d0 (3760 bytes)
You see a wealth of information about the module including the assemblyaddress. Next you can use the "!DumpAssembly" command passing along theassembly address. This provides you more information about the assemblyitself. The information looks like following:
0:000> !DumpAssembly 00187850
Parent Domain: 00149ff8
You see the assembly file name, the class loader and a list of all themodules in the assembly as well as their addresses. You also see the domainaddress, which of course is the same as the one shown from the "!threads"command. We found our way from the thread to the stack call list, to themeta-data token description, to the module and finally to the assembly. Nextwe want to find out more about the domain using the "!DumpDomain" commandpassing along the domain address. The information looks like this:
0:000> !DumpDomain 00149ff8
Domain 1: 00149ff8
Assembly: 00164df0 [C:\WINDOWS\assembly\GAC_32\mscorlib\...\mscorlib.dll]
Assembly: 00187850 [C:\ThrowException\bin\Release\ThrowException.exe]
Assembly: 0018a708 [C:\WINDOWS\assembly\GAC_MSIL\System\...\System.dll]
Please note that the list is again abbreviated. You see a list of allassemblies loaded into the domain. For each assembly you see the classloader address as well as a list of modules. You could now use again"!DumpAssembly" as well as "!DumpModule" to find out more information abouteach assembly and module loaded. Let?s go back to the meta-data tokendescription. You can also see a method description address which we can nowuse to find out more information about the method itself. Use the "!DumpMD"command and pass along the method description address. The information lookslike this:
0:000> !DumpMD 03783320
Method Name: ThrowException.ThrowException.UnhandledException2_Click(…)
The information is again abbreviated. You see now the module this method islocated in, whether it has been already JIT?ed or not and the class andmethod table address. Next we use the "!DumpMT ?MD" command to find all themethods this class has. Pass along the method table address and you will geta list like this:
0:000> !DumpMT -MD 03783384
mdToken: 02000006 (C:\ThrowException\b
Number of IFaces in IFaceMap: 16
Slots in VTable: 348
Entry MethodDesc JIT Name
7b6d5583 7b6d5588 PreJIT System.Windows.Forms.Form.ToString()
78a67578 78cee968 PreJIT System.Object.Equals(System.Object)
78a5cb28 78cee998 PreJIT System.Object.GetHashCode()
This list is again abbreviated. You see again all the methods and for eachthe method description address. You can use again the "!DumpMD" to find outmore information about each method. The command "!DumpMT ?MD" shows you aEEClass address, which is the same as the Class address shown with"!DumpMD". Use the "!DumpClass" command passing along the class address toget a list of all class attributes and fields. The list will look like this:
0:000> !DumpClass 04a416b8
Class Name: ThrowException.ThrowException
mdToken: 02000006 (C:\ThrowException\bin\Release\ThrowException.exe)
Parent Class: 7b177c18
Method Table: 03783384
Vtable Slots: 158
Total Method Slots: 15c
Class Attributes: 100000
MT Field Offset Type Attr Value Name
788ef7e4 4000162 4 CLASS instance __identity
7a6e50a4 400077b 8 CLASS instance site
7a6e50a4 400077c c CLASS instance events
7a6e50a4 400077a 120 CLASS static 00bed5fc EventDisposed
7b178104 40016c7 7f8 CLASS static 00bf4084 defaultIcon
This list is again abbreviated. You see how many methods the class has, howmany instance and static fields it has and much more. It also shows thevalue of any static field. If you want to find more out about the classinstance stored in a static field then you can use the "!DumpObj" commandpassing along the value of the static field. The information will look likethis:
0:000> !DumpObj 00bf4084
Size: 40(0×28) bytes
MT Field Offset Type Attr Value Name
788ef7e4 4000162 4 CLASS instance 00000000 __identity
7af77784 40002da 8 CLASS instance 00bf4310 iconData
7af77784 40002db c System.Int32 instance 38 bestImageOffset
7af77784 40002dc 10 System.Int32 instance 16 bestBitDepth
7af767c4 40002dd 1c VALUETYPE instance 00bf40a0 iconSize
This is again an abbreviated list. You can see the object size, the modulefile name as well as all instance fields and their values. If the fieldcontains again a class instance then you can use again the "!DumpObj"command to find out more about that object. You can also find all objectsstored on the stack (of course these are references to the objects whichitself are on the heap) by using the "!DumpStackObjects" command. The listwill look like this:
ESP/REG Object Name
0012f08c 788ed6c0 System.Exception
0012f098 00c01c3c System.IO.DirectoryNotFoundException
0012f0b8 00c01c3c System.IO.DirectoryNotFoundException
0012f0c0 00c014a0 System.IO.FileStream
0012f0c4 00c01dc0 System.Char
This is again an abbreviated list. You see all the objects, the object typesand their addresses. Use again the "!DumpObj" command passing along theobject address to find more out about the object itself. There are many morecommands SOS supports. Use the "!help" command to get the list of commandsand then "!help" passing along the command itself to get a more detaileddescription about the command.
Using SOS from within your Visual Studio.NET IDE
You can also use SOS from within your Visual Studio.NET IDE. First you needto enable also the "unmanaged code" debugging option in Visual Studio. In VS2005 open the project settings, go to the debugging tab and select theoption "Unmanaged code debugging". Next set break points in yourapplication, run it and when a breakpoint is hit open the "immediate window"(in VS 2005 through the menu "Debug | Windows | Immediate"). In the"immediate window" load the SOS debugger extension through the command".load sos". Now you can execute the same commands as in WinDbg and you seethe same output. For example the command "!threads" shows you a list ofmanaged threads running.
This article provides a good introduction to the "live debugging" storyMicrosoft has. There is a robust debugging framework present butunfortunately it is not widely known in the developer community. Too oftendevelopers as well as managers take a shot in the dark approach and hope tofind a resolution this way. It is easy to take crash and hang dumps inproduction without having to disturb the production environment. It willtake some experience to be able to read these dump files. Nevertheless it iseasy to find out a wealth of information about the application at the timeof crash or hang. This article is just an introduction. There are many morecommands available. You can find more out about each by using the "!help"command. The following blog by Mike Taulty provides more examples as well as morecommand descriptions. There are two more articles by John Robbins (article one and article two) which give also a good introduction into this topic.Finally there is a very comprehensive guide by Microsoft how to debug production .NET applications. If you have comments onthis article or this topic, please contact me @ email@example.com. I want to hear if you learned something new. Contact me if you havequestions about this topic or article.
About the author
Klaus Salchner has worked for 14 years in the industry, nine years in Europeand another five years in North America. As a Senior Enterprise Architectwith solid experience in enterprise software development, Klaus spendsconsiderable time on performance, scalability, availability,maintainability, globalization/localization and security. The projects hehas been involved in are used by more than a million users in 50 countrieson three continents.
Klaus calls Vancouver, British Columbia his home at the moment. His next biggoal is doing the New York marathon in 2005. Klaus is interested in guestspeaking opportunities or as an author for .NET magazines or Web sites. Hecan be contacted at firstname.lastname@example.org or http://www.enterprise-minds.com.
Enterprise application architecture and design consulting services areavailable. If you want to hear more about it contact me! Involve me in yourprojects and I will make a difference for you. Conta
ct me if you have anidea for an article or research project. Also contact me if you want toco-author an article or join future research projects!