MSDeploy: The underlying connection was closed: An unexpected error occurred on a send.

Our company has started deploying all of its numerous web applications via automated (or at least scripted) deployment with MSDeploy. This has worked fine, up until very recently where one machine in particular seemed to be cursed and simply didn't want to work. Even when we nuked the VM and completely recommissioned it from scratch from our VM web server template, we still kept getting the following error:

Error: Could not complete the request to remote agent URL 'https://myserver:10987/MSDeployAgentService'.
Error: The underlying connection was closed: An unexpected error occurred on a send.
Error: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.
Error: An existing connection was forcibly closed by the remote host
Error count: 1.

The problem turned out to be caused by the fact that we hadn't correctly configured the Management Service for MSDeploy (even though we use a custom port for the MsDepSvc deployment service, the Web Management Service (WMSvc) wasn't correctly configured): 

1. Because we were behind a load balancer, we needed to specifically set the IP address of the server
2. We were attempting to use SSL, but we didn't have a certificate specified in the Management Service configuration.

Here's what our configuration looked like in the end, obfuscated to protect the guilty:

Step 1:

Step 2:

Setting up continuous Webjobs in Azure

First things first: you need to properly setup your dashboard:

This means that in the 'Application settings' area of the panel for your Web app, you need to go into the 'Connection strings' setting and add blob storage connection strings with the names 'AzureWebJobsDashboard' and 'AzureWebJobsStorage', with a type of 'Custom'.

Second: if you have continuous web jobs associated with a Web app, you need to ensure that the Web app setting 'Always On' is toggled *on*.

Parameterizing ApplicationInsights.config

I've been working on a lot of web applications in Azure recently, and we're using Application Insights to analyze logs and instrument events in the application. Additionally, we're using MSDeploy to deploy all of our applications as we're trying to get to the point of having Continuous Delivery on-premise. Because we use multiple environments, we need to use the parameters.xml file and SetParameters files to set parameters at deployment time, meaning that we also need to parameterize the InstrumentationKey used by Application Insights to determine which instance of App Insights gets the instrumentation for which application.

This is a bit of a challenge because the XPath support for the parameters.xml file is incomplete and leaves something to be desired. At first glance, I tried to use the naive XPath expression to parameterize our InstrumentationKey:

Match="/ApplicationInsights/InstrumentationKey"

I realized that this wasn't working after checking our deployments, so after some Googling, I found the recommendation that to parameterize an *element* rather than an attribute, you need to use "/text()" in your XPath expression (which makes sense if you know enough about XPath and XQuery). So I tried it:

Match="/ApplicationInsights/InstrumentationKey/text()"

Still didn't work. Then I was stumped for a while and let things simmer.

After a couple more hours and getting frustrated, it hit me: when parameterizing another element a few weeks prior, I ran into issues because a parent element of the one that I wanted to parameterize overrode the default namespace. I checked the ApplicationInsights.config file, and sure enough, the default namespace was the non-empty namespace. I had to use a wildcard to solve the previous problem, so I tried the following:

Match="/*/InstrumentationKey/text()"

Still didn't work, much to my surprise this time. After a little bit more Googling around, I found out that this time it was because the *root* element itself used a non-empty namespace, and overrode the default. I stumbled across this response by Vishal Joshi to a comment on one of his blog posts, which included a suggestion for it using something I didn't even know you could do with XPath (and I've actually got quite a bit of experience using XPath): use the local-name query operator to match only on local name rather than fully qualified name. If you're querying a document that uses an empty default namespace, they're both the same, hence why the simple match expressions work 99% of the time. Updating my query to this:

Match="//*[local-name()='InstrumentationKey']/text()"

... worked like a charm!

Logging your application with App Insights instead of Windows Azure Diagnostics

We've recently started deploying applications to Microsoft Azure. Unfortunately, we don't use Cloud Apps for our web apps, and instead use MSDeploy on the command line to enable automated deployment. Consequently, we don't have any .csdef files in which to implement Windows Azure Diagnostics (WAD).

Fortunately, there's a suitable replacement for applications in Azure App Service (rather than Cloud Services): Application Insights. Not only does this provide a TraceListener for tracing your application, but also provides a whole suite of other useful diagnostics, making this something that you should really be using anyway.  Using the links here and here, I was able to get up to speed with App Insights within an hour and a half, and implement a working solution for our application.

Debugging database deployments with MSDeploy and SqlPackage

I recently had problems deploying a database to Azure, and thanks to this article, I now know why. The best part of this experience is that I've learned to debug this in a much easier manner than relying on the unhelpful error message:

*** An error occurred during deployment plan generation. Deployment cannot continue.
Failed to import target model my-database-name. Detailed message Value cannot be null.
Parameter name: conn
Value cannot be null.
Parameter name: conn

The short of it is that:

1) you need to enable a log trace with these commands for dbDacFx and SSDT:

logman create trace -n DacFxDebug -p "Microsoft-SQLServerDataTools" 0x800 -o "%LOCALAPPDATA%\DacFxDebug.etl" -ets

logman create trace -n SSDTDebug -p "Microsoft-SQLServerDataToolsVS" 0x800 -o "%LOCALAPPDATA%\SSDTDebug.etl" -ets

2) Go execute the command that's causing you grief
3) Execute the following commands to stop the logging trace:

logman stop DacFxDebug -ets

logman stop SSDTDebug -ets

4) Open 'eventvwr' in the Run dialog available from the Start Menu
5) Go to the right side 'Actions' pane, and click on 'Open Saved Log'.
6) When prompted to convert the log, click 'No'.

The logs will display under the 'Saved Logs' folder in the left hand navigation tree, and you'll be able to inspect the errors that occurred. Good luck with deployment!