Resolving "The agent process was stopped while the test was running" on a TFS build agent

I'm currently working on a project that uses TFS for automated builds. Recently, our builds started failing with the error "The agent process was stopped while the test was running" while running unit tests. It wouldn't always happen during the same test, though it did frequently.

After inspecting the Event Viewer on the build machines, we found an error in the .NET Runtime that was taking down the build agent:

Application: QTAgent32_40.exe

Framework Version: v4.0.30319

Description: The process was terminated due to an unhandled exception.

Exception Info: Moq.MockException

Stack:

   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.<ThrowAsync>b__5(System.Object)

   at System.Threading.QueueUserWorkItemCallback.WaitCallback_Context(System.Object)

   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)

   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)

   at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()

   at System.Threading.ThreadPoolWorkQueue.Dispatch()

   at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()

The exception that was being thrown on a ThreadPoolThread (always extremely bad, never good) was a MockException: our unit testing framework was throwing an exception that was taking down the build agent.  But where?

How We Solved It

By some miracle, one of our developers decided to run our problematic tests with mstest on the command line on his local machine. As it turns out, this was great because it showed the stack traces for the loose threads on the command line. Turns out we had a lot more loose threads than just the one that was taking down our test agent on the build machine. The command the developer used was (similar to) the following:

mstest.exe /testcontainer:"C:\path\to\my\test.dll" /noisolation

After auditing all of the errors by rooting them out with command line runs of mstest, the solutions to our problems all boiled down to one thing:

*ALWAYS* wrap the contents of 'async void' methods with a try-catch block!

Journey to robust windows services: Service could not be installed. Verify that you have sufficient privileges to install system services

This is an annoying and almost entirely useless error. It can be caused by any number of things. However, in my case today, it was caused by the fact that the EventLog Source hadn't been created when I tried to install my Windows Service using the WiX installer that I had developed (which it turned out wasn't quite complete). Fortunately, WiX (at least as of 3.5) has built-in support for creating Windows Event Log sources as part of the installation process. Fortunately, I was able to rely on this article on stackoverflow.com to guide me through the process. The gist of the article:

1. Add the "http://schemas.microsoft.com/wix/UtilExtension" namespace to your .wxs file root XML element
2. Add a util:EventSource Log="Application" Name="MyServiceLogSourceName" EventMessageFile="%SystemRoot%\Microsoft.NET\Framework\v2.0.50727\EventLogMessages.dll" element to each component which requires an Event Log source
3. Problem solved

Journey to robust windows services: debugging your WiX custom actions

I've recently had some trouble getting WiX custom actions to work, and after searching around for a while, I found this article. It gives the reader two options for how to debug WiX custom actions, but I had to use both in combination to get it to work. I'll repeat the steps here, just in case the link goes stale:

1. Follow all the necessary steps to create a WiX setup project, along with a separate Custom Action project (available as a project type in the New Project dialog in Visual Studio 2010+ with WiX 3.7)
2. After you've created your custom action method and gotten the project building, add System.Diagnostics.Debugger.Launch(); at the beginning of your custom action method. This will kick the custom action into the debugger so that you can debug it (assuming you've got the wixpdb in the same directory from which you launched the installer).
3. Go to Control Panel -> System -> Advanced System Settings -> Advanced (tab) -> Environment Settings (button) -> System Variables (group) -> New .... (button) 
4. In the dialog that comes up, create a new variable called 'MMsiBreak' (without the quotes) and give it the value of your custom action method (e.g. MyCustomActionMethod)
5. Now run the debug version of your installer and it should get kicked right into your method and allow you to debug your custom action.

Journey to robust web services: debugging services and logging messages

One of the quickest and best ways to debug issues with the services you're developing is to log and trace all activity with WCF along with the messages being sent back and forth. This page on MSDN will provide you with the instructions required to setup logging and tracking for WCF.

Debugging multiple processes and following the children in Eclipse Juno

I guess Eclipse has made it much easier in recent versions of Eclipse and CDT to debug multiple processes and have the debugger follow children. It was actually easier than I had thought it would be to get this functionality in Eclipse Juno. To do so, follow these steps:

1. Create your debug configuration, pointing to the project and program you require. 
2. Ensure that you setup your LD_LIBRARY_PATH in your environment variables as necessary.
3. (and here's the really important part) In the Debug Configuration, go to the 'Debugger' tab. You should see a groupbox called 'Debugger Options' containing (at least) two tabs: 'Main' and 'Shared Libraries'. On the 'Main' tab, ensure that you have 'Non-stop mode' and 'Automatically debug forked processes'. In order to get these abilities, you'll need to have both non-stop gdb and multi-process gdb
4. As a side note, you may need to also have your debug configuration pointed to a custom gdbinit command file with the following line:
• set follow-fork-mode child

This is admittedly a pretty high-level overview of how to get things going, but if I ever have any problems with this in the future, I'll post more details. Or if people ask me :P

Journey to robust Windows Services

Because I have the need for it in several projects, I'm going to be working with Windows Services implemented with .NET quite a bit in the next little while. That said, I'd like to refer to some resources for working with Windows Services:

• The MSDN page on creating windows services
• The MSDN page on debugging a Windows Service
• The MSDN page on installing and uninstalling Windows Services

Also, there are some issues that I ran into while working on my first service. I had originally planned to expose a WCF endpoint through WS-HTTP, however, I can into an unpleasant exception when I tried to do so: AddressAccessDeniedException. Apparently, the cause of this is the fact that http bindings are reserved for processes run as administrators. Fool me once, shame on me. In light of that fact, I'm switching over to using net.tcp for my local Windows Service-hosted WCF service. I'll post more here as I learn it.

Visual Studio (2010?) Debugging Gotchas - Part I

I recently ran across a problem where I was trying to debug some code and the Visual Studio debugger wouldn't stop in my class. It would stop in the test class from which I was debugging, but not the class I really wanted to debug. I looked on the breakpoint and noticed that it was transparent, so I hovered over it to view the tooltip, and the tooltip informed me that my class had a DebuggerStepThroughAttribute assigned to it. My first thought was "what ? that's bullshit", and then it occurred to me that I hadn't looked at the rest of the (partial) class which was defined in another file that was generated from an XML schema by xsd.exe. Sure enough, xsd.exe places DebuggerStepThroughAttribute attributes on the classes it generates. Why is this default behaviour ? This was an annoying as hell bug that took time out of my day that's better devoted to other things, like being productive so that I don't get shit from my boss. It also took more time than I'd care to admit to, largely due to my inexperience with Visual Studio and .NET. I've had yet to see anything like this coming from a Java / scripting background.